T2V resolves to the same value as R2V; loader shows a single model
- T2V selections (editor default mode, explicit [start-end,t2v] override tags, raw t2v values) now resolve to r2v in every task-type resolver (h3_task_type, _multitrack_task_type, frontend getMultiTrackTaskType); h3_generation_mode maps raw t2v to reference conditioning as belt and braces. Reference-mode with zero reference media renders text-only locally; the cloud path falls back to its native t2va request instead of raising. i2v/l2v/v2v/edit flows unchanged. - MoxieMultimediaLoader now presents a single model picker: LoRA, video VAE, audio VAE, and CLIP widgets removed and auto-selected from the pack's models/ folders (first sorted file, loud error when missing, LoRA applied at strength 1.0 when bundled). Outputs and load order unchanged; saved workflows keep working (extra widget values ignored). Verified: py_compile; scripts/test_t2v_r2v_and_loader.py 50/50; frontend vitest 566/567 (one pre-existing env failure on HEAD); bun run build; scripts/test_rtx_upscale_integration.py all pass.
This commit is contained in:
parent
5ad3859bf7
commit
0c4987d090
@ -6,6 +6,8 @@
|
|||||||
|
|
||||||
### ⚡ Improvements
|
### ⚡ Improvements
|
||||||
|
|
||||||
|
- **MultiTrack Editor / MultiTrack Project**: A **T2V selection now produces the exact same value as the R2V setting**. The shipped Moxie model set is reference-conditioned (ref2v), so text-to-video conditioning produced broken renders; the editor's T2V entry point, explicit `[start-end,t2v]` prompt-override tags, and every task-type resolver (`h3_task_type`, the editor task output, and the frontend type computation) now resolve T2V to `r2v`, and the local sampler treats a raw `t2v` value as reference conditioning. Reference-mode with zero reference media renders as text-only locally; the optional MiniMax cloud path falls back to its native text-only request mode instead of erroring. `i2v`, `l2v`, `v2v`, and edit flows are unchanged.
|
||||||
|
- **Moxie Multimedia Loader**: The loader now **looks like it loads a single model**. Only the diffusion-model picker (plus the attention backend) is visible; the LoRA, video VAE, audio VAE, and CLIP widgets were removed and their files are auto-selected from the pack's `models/loras`, `models/vae`, `models/audio_vae`, and `models/clip` folders (first file in sorted order, matching the bundled model set). The LoRA is applied automatically with strength 1.0; remove the file from `models/loras` to disable it. Missing auxiliary models raise the same clear setup error as before. Workflows saved before this change keep working — extra widget values are ignored.
|
||||||
- **MultiTrack Project / Moxie Project Video Combine**: Single-pass projects now condition and sample at half the editor-selected resolution internally. The project timeline is composed at the segments' own size, and Moxie Project Video Combine restores the exact editor-selected resolution in pixel space before any output leaves the node (RTX Video Super Resolution via the optional `nvidia-vfx` package when available, with an automatic Lanczos fallback otherwise) — so every output carries the full editor resolution regardless of template wiring. The editor resolution therefore stays the final output resolution while the sampling chain runs smaller and faster. Dual-pass, SelfLift, and passthrough modes are unchanged.
|
- **MultiTrack Project / Moxie Project Video Combine**: Single-pass projects now condition and sample at half the editor-selected resolution internally. The project timeline is composed at the segments' own size, and Moxie Project Video Combine restores the exact editor-selected resolution in pixel space before any output leaves the node (RTX Video Super Resolution via the optional `nvidia-vfx` package when available, with an automatic Lanczos fallback otherwise) — so every output carries the full editor resolution regardless of template wiring. The editor resolution therefore stays the final output resolution while the sampling chain runs smaller and faster. Dual-pass, SelfLift, and passthrough modes are unchanged.
|
||||||
- **Moxie Project Video Combine**: Removed the leftover `VIDEO` and `FILENAME_PREFIX` outputs. The node now exposes exactly the components consumed downstream — `images`, `audio`, and `fps`, matching the core `GetVideoComponents` layout — and the output file name is chosen on `easy saveVideo`. The full-featured `easy multitrackProjectVideoCombine` node keeps its `VIDEO` and `FILENAME_PREFIX` outputs. Workflows saved before this change that wired the removed outputs need those links reconnected once; the shipped Moxie-consolidated template is already rewired.
|
- **Moxie Project Video Combine**: Removed the leftover `VIDEO` and `FILENAME_PREFIX` outputs. The node now exposes exactly the components consumed downstream — `images`, `audio`, and `fps`, matching the core `GetVideoComponents` layout — and the output file name is chosen on `easy saveVideo`. The full-featured `easy multitrackProjectVideoCombine` node keeps its `VIDEO` and `FILENAME_PREFIX` outputs. Workflows saved before this change that wired the removed outputs need those links reconnected once; the shipped Moxie-consolidated template is already rewired.
|
||||||
- **MultiTrack Editor / MultiTrack Project**: The `format` combo now displays `Moxie Multimedia` while the internal format value stays `MiniMax`, so saved projects, conditioning caches, and all downstream logic keep working untouched (workflows saved before this change need the format re-picked once after updating). The per-run `Output upscale` console line was removed; the resolution restore now happens silently.
|
- **MultiTrack Editor / MultiTrack Project**: The `format` combo now displays `Moxie Multimedia` while the internal format value stays `MiniMax`, so saved projects, conditioning caches, and all downstream logic keep working untouched (workflows saved before this change need the format re-picked once after updating). The per-run `Output upscale` console line was removed; the resolution restore now happens silently.
|
||||||
|
|||||||
@ -38,7 +38,7 @@ Moxie Multimedia Suite consolidates multi-node setups into a concise pipeline:
|
|||||||
|
|
||||||
| Node | Description |
|
| Node | Description |
|
||||||
|---|---|
|
|---|---|
|
||||||
| **Moxie Multimedia Loader** (`MoxieMultimediaLoader`) | Self-contained loader reading from local `models/` subdirectories (`diffusion`, `loras`, `vae`, `audio_vae`, `clip`). Chains **Diffusion Model → Attention Backend (`comfy_kitchen_int8`) → LoRA Loader** and outputs `model`, `clip`, `video_vae`, and `audio_vae` directly. The model set is MiniMax H3-only: the CLIP recipe is locked to the `minimax` (Qwen3-VL) text encoder with no model-type choices exposed. |
|
| **Moxie Multimedia Loader** (`MoxieMultimediaLoader`) | Self-contained loader reading from local `models/` subdirectories (`diffusion`, `loras`, `vae`, `audio_vae`, `clip`). Presents a **single model picker** (the diffusion model) plus the attention backend; the bundled LoRA, video VAE, audio VAE, and CLIP are auto-selected from their `models/` folders and never shown to end users. Chains **Diffusion Model → Attention Backend (`comfy_kitchen_int8`) → LoRA Loader** and outputs `model`, `clip`, `video_vae`, and `audio_vae` directly. The model set is MiniMax H3-only: the CLIP recipe is locked to the `minimax` (Qwen3-VL) text encoder with no model-type choices exposed. |
|
||||||
| **Moxie Preview Override** (`MoxiePreviewOverride`) | Ported from ComfyUI-KJNodes' Model Preview Override. Intercepts outer-sampling iterations to stream full-resolution animated video/audio preview frames directly to the node canvas widget using `taeh3` / tiny VAE decoders. |
|
| **Moxie Preview Override** (`MoxiePreviewOverride`) | Ported from ComfyUI-KJNodes' Model Preview Override. Intercepts outer-sampling iterations to stream full-resolution animated video/audio preview frames directly to the node canvas widget using `taeh3` / tiny VAE decoders. |
|
||||||
| **Moxie MultiTrack Project** (`MoxieProject`) | Consolidates `easy multitrackProject` + `KSamplerSelect` (euler) + `BasicScheduler` (beta scheduler) + `ManualSigmas`. Takes `model`, `clip`, `video_vae`, and `audio_vae` directly from the Moxie Multimedia Loader. Hardcodes tested single-pass settings while exposing only **`steps`**, **`seed`**, and **`project_name`**. |
|
| **Moxie MultiTrack Project** (`MoxieProject`) | Consolidates `easy multitrackProject` + `KSamplerSelect` (euler) + `BasicScheduler` (beta scheduler) + `ManualSigmas`. Takes `model`, `clip`, `video_vae`, and `audio_vae` directly from the Moxie Multimedia Loader. Hardcodes tested single-pass settings while exposing only **`steps`**, **`seed`**, and **`project_name`**. |
|
||||||
| **Moxie Project Video Combine** (`MoxieCombine`) | Consolidates `easy multitrackProjectVideoCombine` + `GetVideoComponents`. Assembles rendered segments and outputs `images`, `audio`, and `fps` directly into `easy saveVideo` (the file name is set on `easy saveVideo`). |
|
| **Moxie Project Video Combine** (`MoxieCombine`) | Consolidates `easy multitrackProjectVideoCombine` + `GetVideoComponents`. Assembles rendered segments and outputs `images`, `audio`, and `fps` directly into `easy saveVideo` (the file name is set on `easy saveVideo`). |
|
||||||
@ -66,7 +66,7 @@ All model files ship **inside the repo** — cloning it into `custom_nodes` is a
|
|||||||
| Qwen3-VL text encoder (`MM-VL.safetensors`) | `models/clip/` |
|
| Qwen3-VL text encoder (`MM-VL.safetensors`) | `models/clip/` |
|
||||||
| Video VAE (`video_vae.safetensors`) | `models/vae/` |
|
| Video VAE (`video_vae.safetensors`) | `models/vae/` |
|
||||||
| Audio VAE (`audio_vae.safetensors`) | `models/audio_vae/` |
|
| Audio VAE (`audio_vae.safetensors`) | `models/audio_vae/` |
|
||||||
| 3-step turbo LoRA (`MM3step.safetensors`) | `models/loras/` (applied by default, `none` to disable) |
|
| 3-step turbo LoRA (`MM3step.safetensors`) | `models/loras/` (applied automatically; remove the file from the folder to disable) |
|
||||||
| Preview tiny VAE (`MM-preview.safetensors`) | `models/vae_approx/` |
|
| Preview tiny VAE (`MM-preview.safetensors`) | `models/vae_approx/` |
|
||||||
|
|
||||||
If you obtained the pack without the bundled files, the bundled downloader fetches and places them
|
If you obtained the pack without the bundled files, the bundled downloader fetches and places them
|
||||||
|
|||||||
@ -109,7 +109,7 @@ if [[ "$failed" -eq 0 ]]; then
|
|||||||
printf ' %-40s %s\n' "$local_rel" "$(du -h "$MODELS_DIR/$local_rel" 2>/dev/null | cut -f1 || echo missing)"
|
printf ' %-40s %s\n' "$local_rel" "$(du -h "$MODELS_DIR/$local_rel" 2>/dev/null | cut -f1 || echo missing)"
|
||||||
done
|
done
|
||||||
echo
|
echo
|
||||||
echo "Restart ComfyUI so the loader dropdowns pick the files up."
|
echo "Restart ComfyUI so the loader picks the files up."
|
||||||
else
|
else
|
||||||
echo "Some downloads failed - re-run this script to resume." >&2
|
echo "Some downloads failed - re-run this script to resume." >&2
|
||||||
exit 1
|
exit 1
|
||||||
|
|||||||
@ -122,7 +122,9 @@ export function getMultiTrackTaskType(
|
|||||||
if (mode === 'l2v') return 'l2v'
|
if (mode === 'l2v') return 'l2v'
|
||||||
if (mode === 'ref') return hasVideoInRange ? 'rv2v' : 'r2v'
|
if (mode === 'ref') return hasVideoInRange ? 'rv2v' : 'r2v'
|
||||||
if (mode === 'edit') return imageCount > 0 ? 'vi2v' : 'v2v'
|
if (mode === 'edit') return imageCount > 0 ? 'vi2v' : 'v2v'
|
||||||
if (imageCount <= 0) return 't2v'
|
// The Moxie model set is reference-conditioned: a T2V selection must
|
||||||
|
// produce the exact same value as R2V (mirrors utils/h3_project.py).
|
||||||
|
if (imageCount <= 0) return 'r2v'
|
||||||
if (imageCount === 1) return 'i2v'
|
if (imageCount === 1) return 'i2v'
|
||||||
if (imageCount === 2) return 'fl2v'
|
if (imageCount === 2) return 'fl2v'
|
||||||
return 'fmlf2v'
|
return 'fmlf2v'
|
||||||
|
|||||||
@ -292,7 +292,9 @@ describe('TaskSegmentEditor', () => {
|
|||||||
})
|
})
|
||||||
|
|
||||||
it.each([
|
it.each([
|
||||||
[0, 'T2V'],
|
// 0 images: the T2V selection resolves to the exact same value as R2V
|
||||||
|
// (reference-conditioned Moxie model set), so the label reads R2V.
|
||||||
|
[0, 'R2V'],
|
||||||
[1, 'I2V'],
|
[1, 'I2V'],
|
||||||
[2, 'FL2V'],
|
[2, 'FL2V'],
|
||||||
[3, 'FMLF2V'],
|
[3, 'FMLF2V'],
|
||||||
|
|||||||
@ -17,6 +17,7 @@ import {
|
|||||||
getActivePreviewVideoSegment,
|
getActivePreviewVideoSegment,
|
||||||
getActivePreviewAudioSources,
|
getActivePreviewAudioSources,
|
||||||
getMultiTrackTaskModeLabel,
|
getMultiTrackTaskModeLabel,
|
||||||
|
getMultiTrackTaskType,
|
||||||
getSelectedMultiTrackSegment,
|
getSelectedMultiTrackSegment,
|
||||||
frameToSeconds,
|
frameToSeconds,
|
||||||
getSegmentDragPlaceholder,
|
getSegmentDragPlaceholder,
|
||||||
@ -42,6 +43,16 @@ import {
|
|||||||
} from '@/lib/multitrack-utils'
|
} from '@/lib/multitrack-utils'
|
||||||
|
|
||||||
describe('multitrack utilities', () => {
|
describe('multitrack utilities', () => {
|
||||||
|
it('resolves a T2V selection to the exact same task type as R2V', () => {
|
||||||
|
// The Moxie model set is reference-conditioned: text-only default-mode
|
||||||
|
// segments must drive the identical value the R2V setting produces.
|
||||||
|
expect(getMultiTrackTaskType('default', 0, false)).toBe('r2v')
|
||||||
|
expect(getMultiTrackTaskType('ref', 0, false)).toBe('r2v')
|
||||||
|
expect(getMultiTrackTaskType('ref', 0, true)).toBe('rv2v')
|
||||||
|
expect(getMultiTrackTaskType('default', 1, false)).toBe('i2v')
|
||||||
|
expect(getMultiTrackTaskType('l2v', 0, false)).toBe('l2v')
|
||||||
|
})
|
||||||
|
|
||||||
it('merges consecutive selected task segments using the first segment with prompt or active images', () => {
|
it('merges consecutive selected task segments using the first segment with prompt or active images', () => {
|
||||||
const data = createDefaultTrackData()
|
const data = createDefaultTrackData()
|
||||||
const taskTrack = data.tracks[0]
|
const taskTrack = data.tracks[0]
|
||||||
|
|||||||
104
moxie_loader.py
104
moxie_loader.py
@ -60,6 +60,39 @@ def _resolve(subdir, name):
|
|||||||
_loaded_lora = None
|
_loaded_lora = None
|
||||||
|
|
||||||
|
|
||||||
|
def _auto_pick(subdir):
|
||||||
|
"""Return the model file to use for a hidden picker.
|
||||||
|
|
||||||
|
End users only see the diffusion model widget; every auxiliary model
|
||||||
|
(video VAE, audio VAE, CLIP, LoRA) is auto-selected from the pack's
|
||||||
|
models/<subdir> folder - the first file in sorted order, which for the
|
||||||
|
published pack is exactly the bundled file. A missing file keeps the
|
||||||
|
loud, actionable error from _resolve.
|
||||||
|
"""
|
||||||
|
files = _list_files(subdir)
|
||||||
|
if files[0].startswith("<no files found"):
|
||||||
|
raise RuntimeError(
|
||||||
|
"Moxie Multimedia Loader: no model files found in {}. "
|
||||||
|
"Place your models inside this folder and restart ComfyUI.".format(SUBDIR_HINTS[subdir])
|
||||||
|
)
|
||||||
|
selected = files[0]
|
||||||
|
if len(files) > 1:
|
||||||
|
logging.info(
|
||||||
|
"[Moxie Multimedia Loader] Multiple files in %s; using '%s'.",
|
||||||
|
SUBDIR_HINTS[subdir],
|
||||||
|
selected,
|
||||||
|
)
|
||||||
|
return selected
|
||||||
|
|
||||||
|
|
||||||
|
def _auto_lora():
|
||||||
|
"""Bundled LoRA file, or ``None`` when models/loras is empty."""
|
||||||
|
files = _list_files("loras")
|
||||||
|
if files[0].startswith("<no files found"):
|
||||||
|
return None
|
||||||
|
return files[0]
|
||||||
|
|
||||||
|
|
||||||
def _h3_clip_type():
|
def _h3_clip_type():
|
||||||
"""MiniMax H3 CLIP recipe. The Moxie model set is H3-only, so the text
|
"""MiniMax H3 CLIP recipe. The Moxie model set is H3-only, so the text
|
||||||
encoder type is intentionally NOT a user-facing choice."""
|
encoder type is intentionally NOT a user-facing choice."""
|
||||||
@ -74,8 +107,11 @@ def _h3_clip_type():
|
|||||||
|
|
||||||
|
|
||||||
class MoxieMultimediaLoader(io.ComfyNode):
|
class MoxieMultimediaLoader(io.ComfyNode):
|
||||||
"""All-in-one loader: diffusion model -> attention backend -> LoRA, plus
|
"""All-in-one loader that looks like it loads a single model.
|
||||||
video VAE, audio VAE and CLIP, all read from the node pack's models/ folder.
|
|
||||||
|
The only visible picker is the diffusion model; the attention backend is
|
||||||
|
applied to it, the bundled LoRA (if any), the video VAE, the audio VAE
|
||||||
|
and the CLIP are auto-selected from the node pack's models/ folder.
|
||||||
|
|
||||||
The model set is MiniMax H3-only: the CLIP recipe is locked to
|
The model set is MiniMax H3-only: the CLIP recipe is locked to
|
||||||
CLIPType.MINIMAX and outputs are exactly (model, clip, video_vae, audio_vae)
|
CLIPType.MINIMAX and outputs are exactly (model, clip, video_vae, audio_vae)
|
||||||
@ -83,17 +119,14 @@ class MoxieMultimediaLoader(io.ComfyNode):
|
|||||||
|
|
||||||
@classmethod
|
@classmethod
|
||||||
def define_schema(cls) -> io.Schema:
|
def define_schema(cls) -> io.Schema:
|
||||||
# The published pack ships its models inside the repo, so default to
|
|
||||||
# the bundled turbo LoRA out of the box; "none" opts out.
|
|
||||||
lora_files = _list_files("loras")
|
|
||||||
lora_default = "none" if lora_files[0].startswith("<no files found") else lora_files[0]
|
|
||||||
return io.Schema(
|
return io.Schema(
|
||||||
node_id="MoxieMultimediaLoader",
|
node_id="MoxieMultimediaLoader",
|
||||||
display_name="Moxie Multimedia Loader",
|
display_name="Moxie Multimedia Loader",
|
||||||
category="loaders/moxie",
|
category="loaders/moxie",
|
||||||
description="Loads the Moxie diffusion model, applies the attention backend and a LoRA, and loads "
|
description="Loads the Moxie diffusion model (with the attention backend and the bundled LoRA "
|
||||||
"a video VAE, an audio VAE and the Moxie CLIP - all from the models folder bundled "
|
"already applied) plus the video VAE, audio VAE and text encoder that ship with "
|
||||||
"with this custom node. Outputs model, clip, video_vae and audio_vae.",
|
"this custom node - one model picker, everything else is handled automatically. "
|
||||||
|
"Outputs model, clip, video_vae and audio_vae.",
|
||||||
inputs=[
|
inputs=[
|
||||||
io.Combo.Input(
|
io.Combo.Input(
|
||||||
"diffusion_model",
|
"diffusion_model",
|
||||||
@ -113,41 +146,6 @@ class MoxieMultimediaLoader(io.ComfyNode):
|
|||||||
default="comfy_kitchen_attention",
|
default="comfy_kitchen_attention",
|
||||||
tooltip="Dense attention backend. comfy_kitchen_attention uses quantized INT8 attention (Nvidia/AMD GPUs only).",
|
tooltip="Dense attention backend. comfy_kitchen_attention uses quantized INT8 attention (Nvidia/AMD GPUs only).",
|
||||||
),
|
),
|
||||||
io.Combo.Input(
|
|
||||||
"lora_name",
|
|
||||||
options=["none"] + lora_files,
|
|
||||||
default=lora_default,
|
|
||||||
tooltip="LoRA file from the pack's models/loras folder, applied to both the model and "
|
|
||||||
"the CLIP. Defaults to the bundled turbo LoRA; 'none' disables it.",
|
|
||||||
),
|
|
||||||
io.Float.Input("lora_strength_model", default=1.0, min=-100.0, max=100.0, step=0.01,
|
|
||||||
tooltip="How strongly the LoRA modifies the diffusion model."),
|
|
||||||
io.Float.Input("lora_strength_clip", default=1.0, min=-100.0, max=100.0, step=0.01,
|
|
||||||
tooltip="How strongly the LoRA modifies the CLIP model."),
|
|
||||||
io.Combo.Input(
|
|
||||||
"video_vae",
|
|
||||||
options=_list_files("vae"),
|
|
||||||
tooltip="Video/image VAE file (inside models/vae).",
|
|
||||||
),
|
|
||||||
io.Combo.Input(
|
|
||||||
"audio_vae",
|
|
||||||
options=_list_files("audio_vae"),
|
|
||||||
tooltip="Audio VAE file (inside models/audio_vae). Supports plain VAE checkpoints and "
|
|
||||||
"checkpoints with audio_vae./vocoder. key prefixes (e.g. LTX audio VAE).",
|
|
||||||
),
|
|
||||||
io.Combo.Input(
|
|
||||||
"clip",
|
|
||||||
options=_list_files("clip"),
|
|
||||||
tooltip="Text encoder file (inside models/clip). Loaded with the bundled "
|
|
||||||
"Moxie text-encoder recipe.",
|
|
||||||
),
|
|
||||||
io.Combo.Input(
|
|
||||||
"clip_device",
|
|
||||||
options=["default", "cpu"],
|
|
||||||
default="default",
|
|
||||||
advanced=True,
|
|
||||||
tooltip="Force the CLIP onto CPU if needed.",
|
|
||||||
),
|
|
||||||
],
|
],
|
||||||
outputs=[
|
outputs=[
|
||||||
io.Model.Output(display_name="model", tooltip="Diffusion model with attention backend and LoRA applied."),
|
io.Model.Output(display_name="model", tooltip="Diffusion model with attention backend and LoRA applied."),
|
||||||
@ -257,23 +255,19 @@ class MoxieMultimediaLoader(io.ComfyNode):
|
|||||||
diffusion_model,
|
diffusion_model,
|
||||||
weight_dtype,
|
weight_dtype,
|
||||||
attention_backend,
|
attention_backend,
|
||||||
lora_name,
|
|
||||||
lora_strength_model,
|
|
||||||
lora_strength_clip,
|
|
||||||
video_vae,
|
|
||||||
audio_vae,
|
|
||||||
clip,
|
|
||||||
clip_device,
|
|
||||||
) -> io.NodeOutput:
|
) -> io.NodeOutput:
|
||||||
# Chain: Load Diffusion Model -> Model Attention Backend -> Lora Loader (needs both model and clip)
|
# Chain: Load Diffusion Model -> Model Attention Backend -> Lora Loader (needs both model and clip)
|
||||||
model = cls._load_diffusion_model(diffusion_model, weight_dtype)
|
model = cls._load_diffusion_model(diffusion_model, weight_dtype)
|
||||||
model = cls._apply_attention_backend(model, attention_backend)
|
model = cls._apply_attention_backend(model, attention_backend)
|
||||||
|
|
||||||
clip_obj = cls._load_clip(clip, clip_device)
|
# Hidden pickers: auxiliary models are auto-selected from the pack's
|
||||||
model, clip_obj = cls._apply_lora(model, clip_obj, lora_name, lora_strength_model, lora_strength_clip)
|
# models/ folder so end users only ever see the diffusion model.
|
||||||
|
clip_obj = cls._load_clip(_auto_pick("clip"), "default")
|
||||||
|
lora_name = _auto_lora()
|
||||||
|
model, clip_obj = cls._apply_lora(model, clip_obj, lora_name, 1.0, 1.0)
|
||||||
|
|
||||||
video_vae_obj = cls._load_vae(video_vae, "vae")
|
video_vae_obj = cls._load_vae(_auto_pick("vae"), "vae")
|
||||||
audio_vae_obj = cls._load_vae(audio_vae, "audio_vae")
|
audio_vae_obj = cls._load_vae(_auto_pick("audio_vae"), "audio_vae")
|
||||||
|
|
||||||
return io.NodeOutput(model, clip_obj, video_vae_obj, audio_vae_obj)
|
return io.NodeOutput(model, clip_obj, video_vae_obj, audio_vae_obj)
|
||||||
|
|
||||||
|
|||||||
@ -2686,7 +2686,10 @@ def _multitrack_task_type(task: dict, image_count: int, has_video: bool) -> str:
|
|||||||
content = task.get("content", {})
|
content = task.get("content", {})
|
||||||
explicit_task_type = content.get("task_type") if isinstance(content, dict) else None
|
explicit_task_type = content.get("task_type") if isinstance(content, dict) else None
|
||||||
if isinstance(explicit_task_type, str) and explicit_task_type.strip():
|
if isinstance(explicit_task_type, str) and explicit_task_type.strip():
|
||||||
return explicit_task_type.strip()
|
normalized = explicit_task_type.strip().lower()
|
||||||
|
# The Moxie model set is reference-conditioned: T2V must yield the
|
||||||
|
# exact same value as R2V (mirrors utils.h3_project.h3_task_type).
|
||||||
|
return "r2v" if normalized == "t2v" else normalized
|
||||||
mode = content.get("task_mode", "default") if isinstance(content, dict) else "default"
|
mode = content.get("task_mode", "default") if isinstance(content, dict) else "default"
|
||||||
if mode == "l2v":
|
if mode == "l2v":
|
||||||
return "l2v"
|
return "l2v"
|
||||||
@ -2694,7 +2697,7 @@ def _multitrack_task_type(task: dict, image_count: int, has_video: bool) -> str:
|
|||||||
return "rv2v" if has_video else "r2v"
|
return "rv2v" if has_video else "r2v"
|
||||||
if mode == "edit":
|
if mode == "edit":
|
||||||
return "vi2v" if image_count > 0 else "v2v"
|
return "vi2v" if image_count > 0 else "v2v"
|
||||||
return "i2v" if image_count > 0 else "t2v"
|
return "i2v" if image_count > 0 else "r2v"
|
||||||
|
|
||||||
|
|
||||||
# code based on https://github.com/RH-RunningHub/ComfyUI-RH-Bernini/blob/main/nodes_bernini.py
|
# code based on https://github.com/RH-RunningHub/ComfyUI-RH-Bernini/blob/main/nodes_bernini.py
|
||||||
|
|||||||
450
scripts/test_t2v_r2v_and_loader.py
Normal file
450
scripts/test_t2v_r2v_and_loader.py
Normal file
@ -0,0 +1,450 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""Standalone verification: T2V == R2V normalization + one-model loader.
|
||||||
|
|
||||||
|
No ComfyUI required. Checks (matching the standing verification rules):
|
||||||
|
1. utils/h3_project.py - h3_task_type / h3_generation_mode normalize T2V
|
||||||
|
to the exact same value as R2V (functions extracted via AST).
|
||||||
|
2. nodes/basic.py - _multitrack_task_type mirrors the normalization.
|
||||||
|
3. utils/llm_api.py - reference-mode with zero media falls back to the
|
||||||
|
cloud-native text-only mode instead of raising.
|
||||||
|
4. moxie_loader.py - schema shows a single model picker; execute()
|
||||||
|
auto-picks LoRA/VAE/CLIP from the pack's models/ folders and still
|
||||||
|
applies the LoRA to model+clip at strength 1.0.
|
||||||
|
|
||||||
|
Run: python3 scripts/test_t2v_r2v_and_loader.py
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import ast
|
||||||
|
import enum
|
||||||
|
import importlib.util
|
||||||
|
import os
|
||||||
|
import sys
|
||||||
|
import tempfile
|
||||||
|
import types
|
||||||
|
|
||||||
|
FAILED = []
|
||||||
|
|
||||||
|
|
||||||
|
def check(name, condition, detail=""):
|
||||||
|
status = "PASS" if condition else "FAIL"
|
||||||
|
print(f"[{status}] {name}" + (f" - {detail}" if detail and not condition else ""))
|
||||||
|
if not condition:
|
||||||
|
FAILED.append(name)
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# AST extraction helpers
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
def extract_functions(path, names, constants=()):
|
||||||
|
"""Return {name: function} for top-level (or class-level) defs.
|
||||||
|
|
||||||
|
All requested functions plus any requested module-level constants are
|
||||||
|
compiled into ONE shared namespace so they can reference each other.
|
||||||
|
"""
|
||||||
|
with open(path, "r", encoding="utf-8") as handle:
|
||||||
|
tree = ast.parse(handle.read())
|
||||||
|
body = []
|
||||||
|
|
||||||
|
def visit(nodes):
|
||||||
|
for node in nodes:
|
||||||
|
if isinstance(node, ast.FunctionDef) and node.name in names:
|
||||||
|
body.append(ast.FunctionDef(
|
||||||
|
name=node.name, args=node.args, body=node.body,
|
||||||
|
decorator_list=[], returns=node.returns,
|
||||||
|
type_comment=None, type_params=[],
|
||||||
|
))
|
||||||
|
elif isinstance(node, ast.ClassDef):
|
||||||
|
visit(node.body)
|
||||||
|
elif (
|
||||||
|
isinstance(node, ast.Assign) and constants and any(
|
||||||
|
isinstance(target, ast.Name) and target.id in constants
|
||||||
|
for target in node.targets
|
||||||
|
)
|
||||||
|
):
|
||||||
|
body.append(node)
|
||||||
|
|
||||||
|
visit(tree.body)
|
||||||
|
ast.fix_missing_locations(ast.Module(body=body, type_ignores=[]))
|
||||||
|
namespace = {"__name__": "extracted"}
|
||||||
|
exec(compile(ast.Module(body=body, type_ignores=[]), path, "exec"), namespace)
|
||||||
|
missing = set(names) - set(namespace)
|
||||||
|
if missing:
|
||||||
|
raise RuntimeError(f"could not extract from {path}: {sorted(missing)}")
|
||||||
|
return namespace
|
||||||
|
|
||||||
|
|
||||||
|
def extract_constant(path, name):
|
||||||
|
with open(path, "r", encoding="utf-8") as handle:
|
||||||
|
tree = ast.parse(handle.read())
|
||||||
|
for node in tree.body:
|
||||||
|
if isinstance(node, ast.Assign) and any(
|
||||||
|
isinstance(target, ast.Name) and target.id == name for target in node.targets
|
||||||
|
):
|
||||||
|
return ast.literal_eval(node.value)
|
||||||
|
raise RuntimeError(f"constant {name} not found in {path}")
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# 1. utils/h3_project.py - T2V -> R2V at the resolution choke point
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
def verify_h3_project():
|
||||||
|
print("\n== 1. utils/h3_project.py: h3_task_type / h3_generation_mode ==")
|
||||||
|
h3_dir = os.path.join(os.path.dirname(os.path.dirname(os.path.abspath(__file__))), "utils")
|
||||||
|
h3_file = os.path.join(h3_dir, "h3_project.py")
|
||||||
|
|
||||||
|
h3_ns = extract_functions(
|
||||||
|
h3_file,
|
||||||
|
["h3_task_type", "h3_generation_mode", "_normalize_task_type", "_frame_value"],
|
||||||
|
constants=("_T2V_EQUIVALENT_TASK_TYPE",),
|
||||||
|
)
|
||||||
|
equiv = h3_ns["_T2V_EQUIVALENT_TASK_TYPE"]
|
||||||
|
check("t2v equivalent constant is r2v", equiv == "r2v", repr(equiv))
|
||||||
|
|
||||||
|
def muted_image(image):
|
||||||
|
return bool(isinstance(image, dict) and image.get("muted"))
|
||||||
|
|
||||||
|
h3_ns["multitrack_is_muted_image"] = muted_image
|
||||||
|
h3_task_type = h3_ns["h3_task_type"]
|
||||||
|
h3_generation_mode = h3_ns["h3_generation_mode"]
|
||||||
|
|
||||||
|
def make_info(with_video=False):
|
||||||
|
tracks = [{"type": "task", "segments": [{"start_frame": 0, "end_frame": 48, "content": {}}]}]
|
||||||
|
if with_video:
|
||||||
|
tracks.append({
|
||||||
|
"type": "video",
|
||||||
|
"segments": [{"start_frame": 0, "end_frame": 48, "content": {}}],
|
||||||
|
})
|
||||||
|
return {"tracks": tracks}
|
||||||
|
|
||||||
|
def entry_for(mode=None, task_type=None, images=None, info=None):
|
||||||
|
content = {}
|
||||||
|
if mode is not None:
|
||||||
|
content["task_mode"] = mode
|
||||||
|
if task_type is not None:
|
||||||
|
content["task_type"] = task_type
|
||||||
|
if images is not None:
|
||||||
|
content["images"] = images
|
||||||
|
return {"start_frame": 0, "end_frame": 48, "task": {"content": content}}
|
||||||
|
|
||||||
|
check("default mode, no images -> r2v (was t2v)",
|
||||||
|
h3_task_type(entry_for(mode="default"), make_info()) == "r2v")
|
||||||
|
check("no task_mode, no images -> r2v (was t2v)",
|
||||||
|
h3_task_type(entry_for(), make_info()) == "r2v")
|
||||||
|
check("explicit t2v -> r2v",
|
||||||
|
h3_task_type(entry_for(task_type="t2v"), make_info()) == "r2v")
|
||||||
|
check("explicit T2V (uppercase) -> r2v",
|
||||||
|
h3_task_type(entry_for(task_type="T2V"), make_info()) == "r2v")
|
||||||
|
check("explicit custom type passes through",
|
||||||
|
h3_task_type(entry_for(task_type="wan-2.2-fun"), make_info()) == "wan-2.2-fun")
|
||||||
|
check("muted images ignored -> r2v (was t2v)",
|
||||||
|
h3_task_type(entry_for(images=[{"muted": True}]), make_info()) == "r2v")
|
||||||
|
check("one active image -> i2v unchanged",
|
||||||
|
h3_task_type(entry_for(images=[{"id": "a"}]), make_info()) == "i2v")
|
||||||
|
check("ref mode, no video -> r2v",
|
||||||
|
h3_task_type(entry_for(mode="ref"), make_info()) == "r2v")
|
||||||
|
check("ref mode, video in range -> rv2v",
|
||||||
|
h3_task_type(entry_for(mode="ref"), make_info(with_video=True)) == "rv2v")
|
||||||
|
check("l2v mode -> l2v",
|
||||||
|
h3_task_type(entry_for(mode="l2v"), make_info()) == "l2v")
|
||||||
|
check("edit mode, no images -> v2v",
|
||||||
|
h3_task_type(entry_for(mode="edit"), make_info()) == "v2v")
|
||||||
|
|
||||||
|
check("generation_mode(t2v) -> reference (belt and braces)",
|
||||||
|
h3_generation_mode("t2v") == "reference")
|
||||||
|
check("generation_mode(r2v) -> reference",
|
||||||
|
h3_generation_mode("r2v") == "reference")
|
||||||
|
check("generation_mode(rv2v) -> reference",
|
||||||
|
h3_generation_mode("rv2v") == "reference")
|
||||||
|
check("generation_mode(i2v) -> multi_frames",
|
||||||
|
h3_generation_mode("i2v") == "multi_frames")
|
||||||
|
check("generation_mode(l2v) -> last_frame",
|
||||||
|
h3_generation_mode("l2v") == "last_frame")
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# 2. nodes/basic.py - editor task output mirrors the normalization
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
def verify_basic_node():
|
||||||
|
print("\n== 2. nodes/basic.py: _multitrack_task_type ==")
|
||||||
|
basic_file = os.path.join(
|
||||||
|
os.path.dirname(os.path.dirname(os.path.abspath(__file__))), "nodes", "basic.py"
|
||||||
|
)
|
||||||
|
resolver = extract_functions(basic_file, ["_multitrack_task_type"])["_multitrack_task_type"]
|
||||||
|
check("explicit t2v -> r2v",
|
||||||
|
resolver({"content": {"task_type": "t2v"}}, 0, False) == "r2v")
|
||||||
|
check("default mode, no images -> r2v (was t2v)",
|
||||||
|
resolver({"content": {"task_mode": "default"}}, 0, False) == "r2v")
|
||||||
|
check("no content -> r2v (was t2v)", resolver({}, 0, False) == "r2v")
|
||||||
|
check("custom type kept",
|
||||||
|
resolver({"content": {"task_type": "wan-2.2-fun"}}, 0, False) == "wan-2.2-fun")
|
||||||
|
check("ref mode no video -> r2v",
|
||||||
|
resolver({"content": {"task_mode": "ref"}}, 0, False) == "r2v")
|
||||||
|
check("one image -> i2v",
|
||||||
|
resolver({"content": {"task_mode": "default"}}, 1, False) == "i2v")
|
||||||
|
check("edit mode no images -> v2v",
|
||||||
|
resolver({"content": {"task_mode": "edit"}}, 0, False) == "v2v")
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# 3. utils/llm_api.py - reference-mode with zero media -> text-only cloud mode
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
def verify_llm_api():
|
||||||
|
print("\n== 3. utils/llm_api.py: _h3_media_for_task_type ==")
|
||||||
|
llm_file = os.path.join(
|
||||||
|
os.path.dirname(os.path.dirname(os.path.abspath(__file__))), "utils", "llm_api.py"
|
||||||
|
)
|
||||||
|
media_resolver = extract_functions(llm_file, ["_h3_media_for_task_type"])["_h3_media_for_task_type"]
|
||||||
|
check("r2v with zero media -> t2va text-only (no raise)",
|
||||||
|
media_resolver("r2v", [], [], []) == ("t2va", [], [], []))
|
||||||
|
check("rv2v with zero media -> t2va text-only",
|
||||||
|
media_resolver("rv2v", [], [], []) == ("t2va", [], [], []))
|
||||||
|
check("r2v with media -> r2va",
|
||||||
|
media_resolver("r2v", ["img1", "img2"], ["vid"], ["aud"]) == ("r2va", ["img1", "img2"], ["vid"], ["aud"]))
|
||||||
|
check("direct t2v still -> t2va",
|
||||||
|
media_resolver("t2v", [], [], []) == ("t2va", [], [], []))
|
||||||
|
check("i2v with image -> i2va",
|
||||||
|
media_resolver("i2v", ["img"], [], []) == ("i2va", ["img"], [], []))
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# 4. moxie_loader.py - one-model UI + auto-picked auxiliary models
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
def _install_loader_stubs(state):
|
||||||
|
comfy_api = types.ModuleType("comfy_api")
|
||||||
|
comfy_latest = types.ModuleType("comfy_api.latest")
|
||||||
|
|
||||||
|
class _FakeDiffusionModel:
|
||||||
|
def clone(self):
|
||||||
|
return _FakeDiffusionModel()
|
||||||
|
|
||||||
|
def set_model_optimized_attention(self, attention_function):
|
||||||
|
state.calls.append(("set_model_optimized_attention",))
|
||||||
|
return None
|
||||||
|
|
||||||
|
state.model = _FakeDiffusionModel()
|
||||||
|
|
||||||
|
class _Port:
|
||||||
|
def __init__(self, name, **kwargs):
|
||||||
|
self.name = name
|
||||||
|
self.kwargs = kwargs
|
||||||
|
|
||||||
|
class _Schema:
|
||||||
|
def __init__(self, **kwargs):
|
||||||
|
for key, value in kwargs.items():
|
||||||
|
setattr(self, key, value)
|
||||||
|
|
||||||
|
class _NodeOutput:
|
||||||
|
def __init__(self, *values):
|
||||||
|
self.values = values
|
||||||
|
|
||||||
|
io = types.SimpleNamespace(
|
||||||
|
Schema=_Schema,
|
||||||
|
NodeOutput=_NodeOutput,
|
||||||
|
ComfyNode=type("ComfyNode", (), {}),
|
||||||
|
Combo=types.SimpleNamespace(Input=staticmethod(lambda name, **kw: _Port(name, **kw))),
|
||||||
|
Float=types.SimpleNamespace(Input=staticmethod(lambda name, **kw: _Port(name, **kw))),
|
||||||
|
Model=types.SimpleNamespace(Output=staticmethod(lambda name=None, **kw: _Port(name, **kw))),
|
||||||
|
Clip=types.SimpleNamespace(Output=staticmethod(lambda name=None, **kw: _Port(name, **kw))),
|
||||||
|
Vae=types.SimpleNamespace(Output=staticmethod(lambda name=None, **kw: _Port(name, **kw))),
|
||||||
|
)
|
||||||
|
comfy_latest.io = io
|
||||||
|
comfy_api.latest = comfy_latest
|
||||||
|
sys.modules["comfy_api"] = comfy_api
|
||||||
|
sys.modules["comfy_api.latest"] = comfy_latest
|
||||||
|
|
||||||
|
folder_paths = types.ModuleType("folder_paths")
|
||||||
|
folder_paths.get_folder_paths = lambda kind: []
|
||||||
|
sys.modules["folder_paths"] = folder_paths
|
||||||
|
|
||||||
|
comfy = types.ModuleType("comfy")
|
||||||
|
|
||||||
|
class _FakeVAE:
|
||||||
|
def __init__(self, sd=None, metadata=None):
|
||||||
|
self.sd = sd
|
||||||
|
|
||||||
|
def throw_exception_if_invalid(self):
|
||||||
|
return None
|
||||||
|
|
||||||
|
class _CLIPType(enum.Enum):
|
||||||
|
MINIMAX = "minimax"
|
||||||
|
|
||||||
|
def load_diffusion_model(path, model_options=None):
|
||||||
|
state.calls.append(("load_diffusion_model", os.path.basename(path)))
|
||||||
|
return state.model
|
||||||
|
|
||||||
|
def load_clip(ckpt_paths=None, embedding_directory=None, clip_type=None, model_options=None):
|
||||||
|
state.calls.append(("load_clip", clip_type))
|
||||||
|
return state.clip
|
||||||
|
|
||||||
|
def load_torch_file(path, safe_load=False, return_metadata=False):
|
||||||
|
state.calls.append(("load_torch_file", os.path.basename(path)))
|
||||||
|
sd = {"dummy": 1}
|
||||||
|
return (sd, {"meta": True}) if return_metadata else sd
|
||||||
|
|
||||||
|
def load_lora_for_models(model, clip, lora_sd, strength_model, strength_clip, lora_metadata=None):
|
||||||
|
state.calls.append(("load_lora_for_models", strength_model, strength_clip))
|
||||||
|
return state.lora_model, clip
|
||||||
|
|
||||||
|
def get_attention_function(name, *args):
|
||||||
|
state.calls.append(("get_attention_function", name))
|
||||||
|
return state.attention_fn
|
||||||
|
|
||||||
|
def state_dict_prefix_replace(sd, mapping, filter_keys=False):
|
||||||
|
return sd
|
||||||
|
|
||||||
|
comfy.sd = types.SimpleNamespace(
|
||||||
|
load_diffusion_model=load_diffusion_model,
|
||||||
|
load_clip=load_clip,
|
||||||
|
load_lora_for_models=load_lora_for_models,
|
||||||
|
VAE=_FakeVAE,
|
||||||
|
CLIPType=_CLIPType,
|
||||||
|
)
|
||||||
|
comfy.utils = types.SimpleNamespace(
|
||||||
|
load_torch_file=load_torch_file,
|
||||||
|
state_dict_prefix_replace=state_dict_prefix_replace,
|
||||||
|
)
|
||||||
|
comfy_ldm = types.ModuleType("comfy.ldm")
|
||||||
|
comfy_ldm_modules = types.ModuleType("comfy.ldm.modules")
|
||||||
|
comfy_attention = types.ModuleType("comfy.ldm.modules.attention")
|
||||||
|
comfy_attention.get_attention_function = get_attention_function
|
||||||
|
comfy.ldm = comfy_ldm
|
||||||
|
comfy.ldm.modules = comfy_ldm_modules
|
||||||
|
comfy.ldm.modules.attention = comfy_attention
|
||||||
|
comfy.__path__ = [] # make "comfy" importable as a package
|
||||||
|
sys.modules["comfy"] = comfy
|
||||||
|
sys.modules["comfy.sd"] = comfy.sd
|
||||||
|
sys.modules["comfy.utils"] = comfy.utils
|
||||||
|
sys.modules["comfy.ldm"] = comfy_ldm
|
||||||
|
sys.modules["comfy.ldm.modules"] = comfy_ldm_modules
|
||||||
|
sys.modules["comfy.ldm.modules.attention"] = comfy_attention
|
||||||
|
|
||||||
|
|
||||||
|
def _load_loader_module():
|
||||||
|
pack_dir = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
|
||||||
|
loader_file = os.path.join(pack_dir, "moxie_loader.py")
|
||||||
|
spec = importlib.util.spec_from_file_location("moxie_loader_standalone", loader_file)
|
||||||
|
module = importlib.util.module_from_spec(spec)
|
||||||
|
sys.modules[spec.name] = module
|
||||||
|
spec.loader.exec_module(module)
|
||||||
|
return module
|
||||||
|
|
||||||
|
|
||||||
|
def verify_loader():
|
||||||
|
print("\n== 4. moxie_loader.py: single-model UI + auto-picked aux models ==")
|
||||||
|
state = types.SimpleNamespace(
|
||||||
|
calls=[], clip=object(), attention_fn=object(), lora_model=object(),
|
||||||
|
)
|
||||||
|
_install_loader_stubs(state)
|
||||||
|
loader = _load_loader_module()
|
||||||
|
|
||||||
|
schema = loader.MoxieMultimediaLoader.define_schema()
|
||||||
|
input_names = [port.name for port in schema.inputs]
|
||||||
|
output_names = [port.name for port in schema.outputs]
|
||||||
|
check("schema shows exactly the single-model inputs",
|
||||||
|
input_names == ["diffusion_model", "weight_dtype", "attention_backend"],
|
||||||
|
repr(input_names))
|
||||||
|
for hidden in ("lora_name", "lora_strength_model", "lora_strength_clip",
|
||||||
|
"video_vae", "audio_vae", "clip", "clip_device"):
|
||||||
|
check(f"widget hidden: {hidden}", hidden not in input_names)
|
||||||
|
check("outputs still model/clip/video_vae/audio_vae",
|
||||||
|
[port.name or port.kwargs.get("display_name") for port in schema.outputs]
|
||||||
|
== ["model", "clip", "video_vae", "audio_vae"])
|
||||||
|
|
||||||
|
with tempfile.TemporaryDirectory() as tmp:
|
||||||
|
models = os.path.join(tmp, "models")
|
||||||
|
for subdir, name in (
|
||||||
|
("diffusion", "moxie.safetensors"),
|
||||||
|
("vae", "video_vae.safetensors"),
|
||||||
|
("audio_vae", "audio_vae.safetensors"),
|
||||||
|
("clip", "MM-VL.safetensors"),
|
||||||
|
("loras", "MM3step.safetensors"),
|
||||||
|
):
|
||||||
|
os.makedirs(os.path.join(models, subdir), exist_ok=True)
|
||||||
|
with open(os.path.join(models, subdir, name), "wb") as handle:
|
||||||
|
handle.write(b"stub")
|
||||||
|
loader.MODELS_DIR = models
|
||||||
|
loader._loaded_lora = None
|
||||||
|
|
||||||
|
result = loader.MoxieMultimediaLoader.execute(
|
||||||
|
diffusion_model="moxie.safetensors",
|
||||||
|
weight_dtype="default",
|
||||||
|
attention_backend="comfy_kitchen_attention",
|
||||||
|
)
|
||||||
|
|
||||||
|
names = [call[0] for call in state.calls]
|
||||||
|
check("full chain runs: model -> attention -> clip -> lora -> vaes",
|
||||||
|
names == [
|
||||||
|
"load_diffusion_model", "get_attention_function",
|
||||||
|
"set_model_optimized_attention", "load_clip",
|
||||||
|
"load_torch_file", "load_lora_for_models",
|
||||||
|
"load_torch_file", "load_torch_file",
|
||||||
|
], repr(names))
|
||||||
|
lora_calls = [call for call in state.calls if call[0] == "load_lora_for_models"]
|
||||||
|
check("LoRA auto-applied at strength 1.0/1.0",
|
||||||
|
lora_calls == [("load_lora_for_models", 1.0, 1.0)], repr(lora_calls))
|
||||||
|
clip_calls = [call for call in state.calls if call[0] == "load_clip"]
|
||||||
|
check("CLIP auto-picked and loaded with the locked recipe",
|
||||||
|
len(clip_calls) == 1 and clip_calls[0][1] is not None, repr(clip_calls))
|
||||||
|
check("four outputs returned",
|
||||||
|
isinstance(result, loader.io.NodeOutput) and len(result.values) == 4)
|
||||||
|
check("model output carries LoRA",
|
||||||
|
result.values[0] is state.lora_model)
|
||||||
|
check("clip output is the loaded clip", result.values[1] is state.clip)
|
||||||
|
|
||||||
|
# Empty loras folder -> LoRA silently skipped, everything else loads.
|
||||||
|
os.remove(os.path.join(models, "loras", "MM3step.safetensors"))
|
||||||
|
state.calls.clear()
|
||||||
|
loader._loaded_lora = None
|
||||||
|
loader.MoxieMultimediaLoader.execute(
|
||||||
|
diffusion_model="moxie.safetensors",
|
||||||
|
weight_dtype="default",
|
||||||
|
attention_backend="pytorch_attention",
|
||||||
|
)
|
||||||
|
check("empty loras folder skips LoRA without breaking the load",
|
||||||
|
"load_lora_for_models" not in [call[0] for call in state.calls])
|
||||||
|
|
||||||
|
# Missing clip folder -> loud setup error (same message as before).
|
||||||
|
state.calls.clear()
|
||||||
|
os.rename(os.path.join(models, "clip"), os.path.join(models, "clip_backup"))
|
||||||
|
try:
|
||||||
|
loader.MoxieMultimediaLoader.execute(
|
||||||
|
diffusion_model="moxie.safetensors",
|
||||||
|
weight_dtype="default",
|
||||||
|
attention_backend="pytorch_attention",
|
||||||
|
)
|
||||||
|
check("missing clip raises setup error", False)
|
||||||
|
except RuntimeError as error:
|
||||||
|
check("missing clip raises setup error",
|
||||||
|
"no model files found in models/clip" in str(error), str(error))
|
||||||
|
os.rename(os.path.join(models, "clip_backup"), os.path.join(models, "clip"))
|
||||||
|
|
||||||
|
# Multiple files -> deterministic first-sorted pick, no crash.
|
||||||
|
with open(os.path.join(models, "clip", "AAA-other.safetensors"), "wb") as handle:
|
||||||
|
handle.write(b"stub2")
|
||||||
|
picked = loader._auto_pick("clip")
|
||||||
|
check("multiple clip files pick first sorted deterministically",
|
||||||
|
picked == "AAA-other.safetensors", repr(picked))
|
||||||
|
|
||||||
|
|
||||||
|
def main():
|
||||||
|
verify_h3_project()
|
||||||
|
verify_basic_node()
|
||||||
|
verify_llm_api()
|
||||||
|
verify_loader()
|
||||||
|
print()
|
||||||
|
if FAILED:
|
||||||
|
print(f"FAILED CHECKS ({len(FAILED)}): {FAILED}")
|
||||||
|
return 1
|
||||||
|
print("ALL CHECKS PASSED")
|
||||||
|
return 0
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
sys.exit(main())
|
||||||
@ -93,7 +93,24 @@ def test_h3_task_type_ignores_muted_reference_images():
|
|||||||
entry = h3_task_entries(info)[0]
|
entry = h3_task_entries(info)[0]
|
||||||
entry["task"]["content"]["images"] = [{"muted": True}]
|
entry["task"]["content"]["images"] = [{"muted": True}]
|
||||||
|
|
||||||
assert h3_task_type(entry, info) == "t2v"
|
# The Moxie model set is reference-conditioned: T2V resolves to the
|
||||||
|
# exact same value as R2V.
|
||||||
|
assert h3_task_type(entry, info) == "r2v"
|
||||||
|
|
||||||
|
|
||||||
|
def test_h3_task_type_normalizes_explicit_t2v_to_r2v():
|
||||||
|
info = _tracks_info()
|
||||||
|
entry = h3_task_entries(info)[0]
|
||||||
|
entry["task"]["content"]["task_type"] = "t2v"
|
||||||
|
|
||||||
|
assert h3_task_type(entry, info) == "r2v"
|
||||||
|
|
||||||
|
|
||||||
|
def test_h3_generation_mode_maps_t2v_to_reference():
|
||||||
|
# Belt and braces: even a raw t2v value must behave exactly like R2V.
|
||||||
|
assert h3_generation_mode("t2v") == "reference"
|
||||||
|
assert h3_generation_mode("r2v") == "reference"
|
||||||
|
assert h3_generation_mode("i2v") == "multi_frames"
|
||||||
|
|
||||||
|
|
||||||
def test_parse_tracks_info_rejects_invalid_dimensions():
|
def test_parse_tracks_info_rejects_invalid_dimensions():
|
||||||
|
|||||||
@ -340,6 +340,9 @@ def test_video_data_uri_enforces_duration_even_when_file_is_under_size_limit(mon
|
|||||||
("l2v", "i2va", 1, 0, 0),
|
("l2v", "i2va", 1, 0, 0),
|
||||||
("v2v", "r2va", 2, 1, 1),
|
("v2v", "r2va", 2, 1, 1),
|
||||||
("r2v", "r2va", 2, 1, 1),
|
("r2v", "r2va", 2, 1, 1),
|
||||||
|
# Reference-mode with zero media falls back to the cloud's native
|
||||||
|
# text-only mode (T2V selections resolve to r2v upstream).
|
||||||
|
("r2v", "t2va", 0, 0, 0),
|
||||||
("vi2v", "r2va", 2, 1, 1),
|
("vi2v", "r2va", 2, 1, 1),
|
||||||
("rv2v", "r2va", 2, 1, 1),
|
("rv2v", "r2va", 2, 1, 1),
|
||||||
],
|
],
|
||||||
|
|||||||
@ -2305,7 +2305,8 @@ def test_multitrack_h3_project_expands_single_task_sampling_pipeline(monkeypatch
|
|||||||
assert nodes_by_type["KSamplerSelect"]["inputs"]["sampler_name"] == "er_sde"
|
assert nodes_by_type["KSamplerSelect"]["inputs"]["sampler_name"] == "er_sde"
|
||||||
assert "ManualSigmas" in nodes_by_type
|
assert "ManualSigmas" in nodes_by_type
|
||||||
conditioning = nodes_by_type["easy minimaxH3ToVideo"]["inputs"]
|
conditioning = nodes_by_type["easy minimaxH3ToVideo"]["inputs"]
|
||||||
assert conditioning["mode"] == "multi_frames"
|
# T2V selections resolve to the exact same value as R2V (reference).
|
||||||
|
assert conditioning["mode"] == "reference"
|
||||||
assert conditioning["ref_image_size"] == "match"
|
assert conditioning["ref_image_size"] == "match"
|
||||||
assert (conditioning["width"], conditioning["height"]) == (1344, 768)
|
assert (conditioning["width"], conditioning["height"]) == (1344, 768)
|
||||||
assert "SamplerCustomAdvanced" in nodes_by_type
|
assert "SamplerCustomAdvanced" in nodes_by_type
|
||||||
|
|||||||
@ -353,8 +353,25 @@ def test_loader_schema_locks_h3_clip_recipe(moxie):
|
|||||||
|
|
||||||
# The model set is MiniMax H3-only: no CLIP recipe choices are exposed.
|
# The model set is MiniMax H3-only: no CLIP recipe choices are exposed.
|
||||||
assert "clip_type" not in inputs
|
assert "clip_type" not in inputs
|
||||||
assert "clip_device" in inputs
|
|
||||||
assert inputs.index("clip") == len(inputs) - 2 # clip, then clip_device
|
|
||||||
|
def test_loader_schema_shows_a_single_model(moxie):
|
||||||
|
schema = moxie.loader.MoxieMultimediaLoader.define_schema()
|
||||||
|
inputs = _input_names(schema)
|
||||||
|
|
||||||
|
# End users see one model picker plus the attention backend; every
|
||||||
|
# auxiliary model (LoRA, video/audio VAE, CLIP) is auto-selected.
|
||||||
|
assert inputs == ["diffusion_model", "weight_dtype", "attention_backend"]
|
||||||
|
for hidden in (
|
||||||
|
"lora_name",
|
||||||
|
"lora_strength_model",
|
||||||
|
"lora_strength_clip",
|
||||||
|
"video_vae",
|
||||||
|
"audio_vae",
|
||||||
|
"clip",
|
||||||
|
"clip_device",
|
||||||
|
):
|
||||||
|
assert hidden not in inputs
|
||||||
|
|
||||||
|
|
||||||
def test_loader_execute_applies_attention_then_lora(moxie, monkeypatch, tmp_path):
|
def test_loader_execute_applies_attention_then_lora(moxie, monkeypatch, tmp_path):
|
||||||
@ -381,13 +398,6 @@ def test_loader_execute_applies_attention_then_lora(moxie, monkeypatch, tmp_path
|
|||||||
diffusion_model="h3.safetensors",
|
diffusion_model="h3.safetensors",
|
||||||
weight_dtype="default",
|
weight_dtype="default",
|
||||||
attention_backend="comfy_kitchen_attention",
|
attention_backend="comfy_kitchen_attention",
|
||||||
lora_name="turbo.safetensors",
|
|
||||||
lora_strength_model=1.0,
|
|
||||||
lora_strength_clip=1.0,
|
|
||||||
video_vae="video.safetensors",
|
|
||||||
audio_vae="audio.safetensors",
|
|
||||||
clip="qwen.safetensors",
|
|
||||||
clip_device="default",
|
|
||||||
)
|
)
|
||||||
|
|
||||||
sequence = [call[0] for call in state.calls]
|
sequence = [call[0] for call in state.calls]
|
||||||
@ -441,13 +451,6 @@ def test_loader_execute_without_lora_skips_lora_loading(moxie, monkeypatch, tmp_
|
|||||||
diffusion_model="h3.safetensors",
|
diffusion_model="h3.safetensors",
|
||||||
weight_dtype="default",
|
weight_dtype="default",
|
||||||
attention_backend="pytorch_attention",
|
attention_backend="pytorch_attention",
|
||||||
lora_name="<no files found in models/loras>",
|
|
||||||
lora_strength_model=1.0,
|
|
||||||
lora_strength_clip=1.0,
|
|
||||||
video_vae="video.safetensors",
|
|
||||||
audio_vae="audio.safetensors",
|
|
||||||
clip="qwen.safetensors",
|
|
||||||
clip_device="default",
|
|
||||||
)
|
)
|
||||||
|
|
||||||
assert "load_lora_for_models" not in [call[0] for call in state.calls]
|
assert "load_lora_for_models" not in [call[0] for call in state.calls]
|
||||||
@ -477,13 +480,6 @@ def test_loader_requires_minimax_cliptype(moxie, monkeypatch, tmp_path):
|
|||||||
diffusion_model="h3.safetensors",
|
diffusion_model="h3.safetensors",
|
||||||
weight_dtype="default",
|
weight_dtype="default",
|
||||||
attention_backend="pytorch_attention",
|
attention_backend="pytorch_attention",
|
||||||
lora_name="<no files found in models/loras>",
|
|
||||||
lora_strength_model=1.0,
|
|
||||||
lora_strength_clip=1.0,
|
|
||||||
video_vae="video.safetensors",
|
|
||||||
audio_vae="audio.safetensors",
|
|
||||||
clip="qwen.safetensors",
|
|
||||||
clip_device="default",
|
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
@ -1369,7 +1369,9 @@ def test_multitrack_editor_prompt_override_preserves_explicit_task_type():
|
|||||||
|
|
||||||
assert task_content["task_type"] == "t2v"
|
assert task_content["task_type"] == "t2v"
|
||||||
assert task_content["task_mode"] == "default"
|
assert task_content["task_mode"] == "default"
|
||||||
assert task_result.values[2] == "t2v"
|
# Raw content keeps the explicit override tag, but resolved task types
|
||||||
|
# normalize T2V to the exact same value as R2V.
|
||||||
|
assert task_result.values[2] == "r2v"
|
||||||
|
|
||||||
|
|
||||||
def test_multitrack_editor_prompt_override_outputs_custom_task_type_string():
|
def test_multitrack_editor_prompt_override_outputs_custom_task_type_string():
|
||||||
|
|||||||
@ -400,6 +400,19 @@ def h3_second_pass_dimensions(
|
|||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
|
# The shipped Moxie model set is reference-conditioned (r2v): a T2V selection
|
||||||
|
# must drive the exact same value as R2V so the model is never conditioned as
|
||||||
|
# a plain text-to-video task.
|
||||||
|
_T2V_EQUIVALENT_TASK_TYPE = "r2v"
|
||||||
|
|
||||||
|
|
||||||
|
def _normalize_task_type(task_type: str) -> str:
|
||||||
|
normalized = task_type.strip().lower()
|
||||||
|
if normalized == "t2v":
|
||||||
|
return _T2V_EQUIVALENT_TASK_TYPE
|
||||||
|
return normalized
|
||||||
|
|
||||||
|
|
||||||
def h3_task_type(entry: dict[str, Any], info: dict[str, Any]) -> str:
|
def h3_task_type(entry: dict[str, Any], info: dict[str, Any]) -> str:
|
||||||
task = entry.get("task", {})
|
task = entry.get("task", {})
|
||||||
content = task.get("content", {}) if isinstance(task, dict) else {}
|
content = task.get("content", {}) if isinstance(task, dict) else {}
|
||||||
@ -407,7 +420,7 @@ def h3_task_type(entry: dict[str, Any], info: dict[str, Any]) -> str:
|
|||||||
content = {}
|
content = {}
|
||||||
explicit_type = content.get("task_type")
|
explicit_type = content.get("task_type")
|
||||||
if isinstance(explicit_type, str) and explicit_type.strip():
|
if isinstance(explicit_type, str) and explicit_type.strip():
|
||||||
return explicit_type.strip().lower()
|
return _normalize_task_type(explicit_type)
|
||||||
|
|
||||||
start_frame = _frame_value(entry.get("start_frame"))
|
start_frame = _frame_value(entry.get("start_frame"))
|
||||||
end_frame = _frame_value(entry.get("end_frame"))
|
end_frame = _frame_value(entry.get("end_frame"))
|
||||||
@ -439,7 +452,7 @@ def h3_task_type(entry: dict[str, Any], info: dict[str, Any]) -> str:
|
|||||||
return "rv2v" if has_video else "r2v"
|
return "rv2v" if has_video else "r2v"
|
||||||
if mode == "edit":
|
if mode == "edit":
|
||||||
return "vi2v" if image_count > 0 else "v2v"
|
return "vi2v" if image_count > 0 else "v2v"
|
||||||
return "i2v" if image_count > 0 else "t2v"
|
return "i2v" if image_count > 0 else _T2V_EQUIVALENT_TASK_TYPE
|
||||||
|
|
||||||
|
|
||||||
def _h3_locked_track(
|
def _h3_locked_track(
|
||||||
@ -501,7 +514,11 @@ def h3_generation_mode(task_type: str) -> str:
|
|||||||
return "reference"
|
return "reference"
|
||||||
if normalized == "l2v":
|
if normalized == "l2v":
|
||||||
return "last_frame"
|
return "last_frame"
|
||||||
if normalized in {"t2v", "i2v"}:
|
if normalized == "t2v":
|
||||||
|
# Belt and braces: T2V must behave exactly like R2V even if a raw
|
||||||
|
# t2v value bypasses h3_task_type.
|
||||||
|
return "reference"
|
||||||
|
if normalized == "i2v":
|
||||||
return "multi_frames"
|
return "multi_frames"
|
||||||
raise ValueError(f"Unsupported H3 task type: {task_type}")
|
raise ValueError(f"Unsupported H3 task type: {task_type}")
|
||||||
|
|
||||||
|
|||||||
@ -925,7 +925,11 @@ class PromptEnhancerClient:
|
|||||||
return "i2va", selected_images, [], []
|
return "i2va", selected_images, [], []
|
||||||
if normalized_type in {"v2v", "r2v", "vi2v", "rv2v"}:
|
if normalized_type in {"v2v", "r2v", "vi2v", "rv2v"}:
|
||||||
if not (images or videos or audios):
|
if not (images or videos or audios):
|
||||||
raise ValueError(f"MiniMax {normalized_type} requires reference media.")
|
# Reference-mode with zero reference media: the local Moxie
|
||||||
|
# model handles text-only renders in reference conditioning,
|
||||||
|
# but the cloud API has a native text-only mode - use it
|
||||||
|
# instead of failing (T2V selections resolve to r2v).
|
||||||
|
return "t2va", [], [], []
|
||||||
if len(images) > 9:
|
if len(images) > 9:
|
||||||
raise ValueError("MiniMax r2va supports at most 9 reference images.")
|
raise ValueError("MiniMax r2va supports at most 9 reference images.")
|
||||||
if len(videos) > 3:
|
if len(videos) > 3:
|
||||||
|
|||||||
Loading…
Reference in New Issue
Block a user