From 0c4987d0906e0ab0c50f1fb9611ade4794c2aca3 Mon Sep 17 00:00:00 2001 From: Z User Date: Sun, 27 Sep 2026 05:50:35 +0000 Subject: [PATCH] T2V resolves to the same value as R2V; loader shows a single model - T2V selections (editor default mode, explicit [start-end,t2v] override tags, raw t2v values) now resolve to r2v in every task-type resolver (h3_task_type, _multitrack_task_type, frontend getMultiTrackTaskType); h3_generation_mode maps raw t2v to reference conditioning as belt and braces. Reference-mode with zero reference media renders text-only locally; the cloud path falls back to its native t2va request instead of raising. i2v/l2v/v2v/edit flows unchanged. - MoxieMultimediaLoader now presents a single model picker: LoRA, video VAE, audio VAE, and CLIP widgets removed and auto-selected from the pack's models/ folders (first sorted file, loud error when missing, LoRA applied at strength 1.0 when bundled). Outputs and load order unchanged; saved workflows keep working (extra widget values ignored). Verified: py_compile; scripts/test_t2v_r2v_and_loader.py 50/50; frontend vitest 566/567 (one pre-existing env failure on HEAD); bun run build; scripts/test_rtx_upscale_integration.py all pass. --- CHANGELOG.md | 2 + README.md | 4 +- download_models.sh | 2 +- frontend/src/lib/multitrack-utils.ts | 4 +- frontend/src/tests/TaskSegmentEditor.test.tsx | 4 +- frontend/src/tests/multitrack-utils.test.ts | 11 + moxie_loader.py | 104 ++-- nodes/basic.py | 7 +- scripts/test_t2v_r2v_and_loader.py | 450 ++++++++++++++++++ tests/test_h3_project.py | 19 +- tests/test_llm_api.py | 3 + tests/test_minimax_node.py | 3 +- tests/test_moxie_nodes.py | 42 +- tests/test_multitrack_info_output.py | 4 +- utils/h3_project.py | 23 +- utils/llm_api.py | 6 +- 16 files changed, 596 insertions(+), 92 deletions(-) create mode 100644 scripts/test_t2v_r2v_and_loader.py diff --git a/CHANGELOG.md b/CHANGELOG.md index a5b74a2..b53cbb8 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,8 @@ ### ⚡ Improvements +- **MultiTrack Editor / MultiTrack Project**: A **T2V selection now produces the exact same value as the R2V setting**. The shipped Moxie model set is reference-conditioned (ref2v), so text-to-video conditioning produced broken renders; the editor's T2V entry point, explicit `[start-end,t2v]` prompt-override tags, and every task-type resolver (`h3_task_type`, the editor task output, and the frontend type computation) now resolve T2V to `r2v`, and the local sampler treats a raw `t2v` value as reference conditioning. Reference-mode with zero reference media renders as text-only locally; the optional MiniMax cloud path falls back to its native text-only request mode instead of erroring. `i2v`, `l2v`, `v2v`, and edit flows are unchanged. +- **Moxie Multimedia Loader**: The loader now **looks like it loads a single model**. Only the diffusion-model picker (plus the attention backend) is visible; the LoRA, video VAE, audio VAE, and CLIP widgets were removed and their files are auto-selected from the pack's `models/loras`, `models/vae`, `models/audio_vae`, and `models/clip` folders (first file in sorted order, matching the bundled model set). The LoRA is applied automatically with strength 1.0; remove the file from `models/loras` to disable it. Missing auxiliary models raise the same clear setup error as before. Workflows saved before this change keep working — extra widget values are ignored. - **MultiTrack Project / Moxie Project Video Combine**: Single-pass projects now condition and sample at half the editor-selected resolution internally. The project timeline is composed at the segments' own size, and Moxie Project Video Combine restores the exact editor-selected resolution in pixel space before any output leaves the node (RTX Video Super Resolution via the optional `nvidia-vfx` package when available, with an automatic Lanczos fallback otherwise) — so every output carries the full editor resolution regardless of template wiring. The editor resolution therefore stays the final output resolution while the sampling chain runs smaller and faster. Dual-pass, SelfLift, and passthrough modes are unchanged. - **Moxie Project Video Combine**: Removed the leftover `VIDEO` and `FILENAME_PREFIX` outputs. The node now exposes exactly the components consumed downstream — `images`, `audio`, and `fps`, matching the core `GetVideoComponents` layout — and the output file name is chosen on `easy saveVideo`. The full-featured `easy multitrackProjectVideoCombine` node keeps its `VIDEO` and `FILENAME_PREFIX` outputs. Workflows saved before this change that wired the removed outputs need those links reconnected once; the shipped Moxie-consolidated template is already rewired. - **MultiTrack Editor / MultiTrack Project**: The `format` combo now displays `Moxie Multimedia` while the internal format value stays `MiniMax`, so saved projects, conditioning caches, and all downstream logic keep working untouched (workflows saved before this change need the format re-picked once after updating). The per-run `Output upscale` console line was removed; the resolution restore now happens silently. diff --git a/README.md b/README.md index 829ce7b..c4b24ae 100644 --- a/README.md +++ b/README.md @@ -38,7 +38,7 @@ Moxie Multimedia Suite consolidates multi-node setups into a concise pipeline: | Node | Description | |---|---| -| **Moxie Multimedia Loader** (`MoxieMultimediaLoader`) | Self-contained loader reading from local `models/` subdirectories (`diffusion`, `loras`, `vae`, `audio_vae`, `clip`). Chains **Diffusion Model → Attention Backend (`comfy_kitchen_int8`) → LoRA Loader** and outputs `model`, `clip`, `video_vae`, and `audio_vae` directly. The model set is MiniMax H3-only: the CLIP recipe is locked to the `minimax` (Qwen3-VL) text encoder with no model-type choices exposed. | +| **Moxie Multimedia Loader** (`MoxieMultimediaLoader`) | Self-contained loader reading from local `models/` subdirectories (`diffusion`, `loras`, `vae`, `audio_vae`, `clip`). Presents a **single model picker** (the diffusion model) plus the attention backend; the bundled LoRA, video VAE, audio VAE, and CLIP are auto-selected from their `models/` folders and never shown to end users. Chains **Diffusion Model → Attention Backend (`comfy_kitchen_int8`) → LoRA Loader** and outputs `model`, `clip`, `video_vae`, and `audio_vae` directly. The model set is MiniMax H3-only: the CLIP recipe is locked to the `minimax` (Qwen3-VL) text encoder with no model-type choices exposed. | | **Moxie Preview Override** (`MoxiePreviewOverride`) | Ported from ComfyUI-KJNodes' Model Preview Override. Intercepts outer-sampling iterations to stream full-resolution animated video/audio preview frames directly to the node canvas widget using `taeh3` / tiny VAE decoders. | | **Moxie MultiTrack Project** (`MoxieProject`) | Consolidates `easy multitrackProject` + `KSamplerSelect` (euler) + `BasicScheduler` (beta scheduler) + `ManualSigmas`. Takes `model`, `clip`, `video_vae`, and `audio_vae` directly from the Moxie Multimedia Loader. Hardcodes tested single-pass settings while exposing only **`steps`**, **`seed`**, and **`project_name`**. | | **Moxie Project Video Combine** (`MoxieCombine`) | Consolidates `easy multitrackProjectVideoCombine` + `GetVideoComponents`. Assembles rendered segments and outputs `images`, `audio`, and `fps` directly into `easy saveVideo` (the file name is set on `easy saveVideo`). | @@ -66,7 +66,7 @@ All model files ship **inside the repo** — cloning it into `custom_nodes` is a | Qwen3-VL text encoder (`MM-VL.safetensors`) | `models/clip/` | | Video VAE (`video_vae.safetensors`) | `models/vae/` | | Audio VAE (`audio_vae.safetensors`) | `models/audio_vae/` | -| 3-step turbo LoRA (`MM3step.safetensors`) | `models/loras/` (applied by default, `none` to disable) | +| 3-step turbo LoRA (`MM3step.safetensors`) | `models/loras/` (applied automatically; remove the file from the folder to disable) | | Preview tiny VAE (`MM-preview.safetensors`) | `models/vae_approx/` | If you obtained the pack without the bundled files, the bundled downloader fetches and places them diff --git a/download_models.sh b/download_models.sh index 9a6101e..2a1be5b 100755 --- a/download_models.sh +++ b/download_models.sh @@ -109,7 +109,7 @@ if [[ "$failed" -eq 0 ]]; then printf ' %-40s %s\n' "$local_rel" "$(du -h "$MODELS_DIR/$local_rel" 2>/dev/null | cut -f1 || echo missing)" done echo - echo "Restart ComfyUI so the loader dropdowns pick the files up." + echo "Restart ComfyUI so the loader picks the files up." else echo "Some downloads failed - re-run this script to resume." >&2 exit 1 diff --git a/frontend/src/lib/multitrack-utils.ts b/frontend/src/lib/multitrack-utils.ts index 39d78f5..cb49a6e 100644 --- a/frontend/src/lib/multitrack-utils.ts +++ b/frontend/src/lib/multitrack-utils.ts @@ -122,7 +122,9 @@ export function getMultiTrackTaskType( if (mode === 'l2v') return 'l2v' if (mode === 'ref') return hasVideoInRange ? 'rv2v' : 'r2v' if (mode === 'edit') return imageCount > 0 ? 'vi2v' : 'v2v' - if (imageCount <= 0) return 't2v' + // The Moxie model set is reference-conditioned: a T2V selection must + // produce the exact same value as R2V (mirrors utils/h3_project.py). + if (imageCount <= 0) return 'r2v' if (imageCount === 1) return 'i2v' if (imageCount === 2) return 'fl2v' return 'fmlf2v' diff --git a/frontend/src/tests/TaskSegmentEditor.test.tsx b/frontend/src/tests/TaskSegmentEditor.test.tsx index 331d45e..d71675e 100644 --- a/frontend/src/tests/TaskSegmentEditor.test.tsx +++ b/frontend/src/tests/TaskSegmentEditor.test.tsx @@ -292,7 +292,9 @@ describe('TaskSegmentEditor', () => { }) it.each([ - [0, 'T2V'], + // 0 images: the T2V selection resolves to the exact same value as R2V + // (reference-conditioned Moxie model set), so the label reads R2V. + [0, 'R2V'], [1, 'I2V'], [2, 'FL2V'], [3, 'FMLF2V'], diff --git a/frontend/src/tests/multitrack-utils.test.ts b/frontend/src/tests/multitrack-utils.test.ts index 46ff00e..139b693 100644 --- a/frontend/src/tests/multitrack-utils.test.ts +++ b/frontend/src/tests/multitrack-utils.test.ts @@ -17,6 +17,7 @@ import { getActivePreviewVideoSegment, getActivePreviewAudioSources, getMultiTrackTaskModeLabel, + getMultiTrackTaskType, getSelectedMultiTrackSegment, frameToSeconds, getSegmentDragPlaceholder, @@ -42,6 +43,16 @@ import { } from '@/lib/multitrack-utils' describe('multitrack utilities', () => { + it('resolves a T2V selection to the exact same task type as R2V', () => { + // The Moxie model set is reference-conditioned: text-only default-mode + // segments must drive the identical value the R2V setting produces. + expect(getMultiTrackTaskType('default', 0, false)).toBe('r2v') + expect(getMultiTrackTaskType('ref', 0, false)).toBe('r2v') + expect(getMultiTrackTaskType('ref', 0, true)).toBe('rv2v') + expect(getMultiTrackTaskType('default', 1, false)).toBe('i2v') + expect(getMultiTrackTaskType('l2v', 0, false)).toBe('l2v') + }) + it('merges consecutive selected task segments using the first segment with prompt or active images', () => { const data = createDefaultTrackData() const taskTrack = data.tracks[0] diff --git a/moxie_loader.py b/moxie_loader.py index 29be7e2..9cc77bf 100644 --- a/moxie_loader.py +++ b/moxie_loader.py @@ -60,6 +60,39 @@ def _resolve(subdir, name): _loaded_lora = None +def _auto_pick(subdir): + """Return the model file to use for a hidden picker. + + End users only see the diffusion model widget; every auxiliary model + (video VAE, audio VAE, CLIP, LoRA) is auto-selected from the pack's + models/ folder - the first file in sorted order, which for the + published pack is exactly the bundled file. A missing file keeps the + loud, actionable error from _resolve. + """ + files = _list_files(subdir) + if files[0].startswith(" 1: + logging.info( + "[Moxie Multimedia Loader] Multiple files in %s; using '%s'.", + SUBDIR_HINTS[subdir], + selected, + ) + return selected + + +def _auto_lora(): + """Bundled LoRA file, or ``None`` when models/loras is empty.""" + files = _list_files("loras") + if files[0].startswith(" attention backend -> LoRA, plus - video VAE, audio VAE and CLIP, all read from the node pack's models/ folder. + """All-in-one loader that looks like it loads a single model. + + The only visible picker is the diffusion model; the attention backend is + applied to it, the bundled LoRA (if any), the video VAE, the audio VAE + and the CLIP are auto-selected from the node pack's models/ folder. The model set is MiniMax H3-only: the CLIP recipe is locked to CLIPType.MINIMAX and outputs are exactly (model, clip, video_vae, audio_vae) @@ -83,17 +119,14 @@ class MoxieMultimediaLoader(io.ComfyNode): @classmethod def define_schema(cls) -> io.Schema: - # The published pack ships its models inside the repo, so default to - # the bundled turbo LoRA out of the box; "none" opts out. - lora_files = _list_files("loras") - lora_default = "none" if lora_files[0].startswith(" io.NodeOutput: # Chain: Load Diffusion Model -> Model Attention Backend -> Lora Loader (needs both model and clip) model = cls._load_diffusion_model(diffusion_model, weight_dtype) model = cls._apply_attention_backend(model, attention_backend) - clip_obj = cls._load_clip(clip, clip_device) - model, clip_obj = cls._apply_lora(model, clip_obj, lora_name, lora_strength_model, lora_strength_clip) + # Hidden pickers: auxiliary models are auto-selected from the pack's + # models/ folder so end users only ever see the diffusion model. + clip_obj = cls._load_clip(_auto_pick("clip"), "default") + lora_name = _auto_lora() + model, clip_obj = cls._apply_lora(model, clip_obj, lora_name, 1.0, 1.0) - video_vae_obj = cls._load_vae(video_vae, "vae") - audio_vae_obj = cls._load_vae(audio_vae, "audio_vae") + video_vae_obj = cls._load_vae(_auto_pick("vae"), "vae") + audio_vae_obj = cls._load_vae(_auto_pick("audio_vae"), "audio_vae") return io.NodeOutput(model, clip_obj, video_vae_obj, audio_vae_obj) diff --git a/nodes/basic.py b/nodes/basic.py index 66a56b3..e320713 100644 --- a/nodes/basic.py +++ b/nodes/basic.py @@ -2686,7 +2686,10 @@ def _multitrack_task_type(task: dict, image_count: int, has_video: bool) -> str: content = task.get("content", {}) explicit_task_type = content.get("task_type") if isinstance(content, dict) else None if isinstance(explicit_task_type, str) and explicit_task_type.strip(): - return explicit_task_type.strip() + normalized = explicit_task_type.strip().lower() + # The Moxie model set is reference-conditioned: T2V must yield the + # exact same value as R2V (mirrors utils.h3_project.h3_task_type). + return "r2v" if normalized == "t2v" else normalized mode = content.get("task_mode", "default") if isinstance(content, dict) else "default" if mode == "l2v": return "l2v" @@ -2694,7 +2697,7 @@ def _multitrack_task_type(task: dict, image_count: int, has_video: bool) -> str: return "rv2v" if has_video else "r2v" if mode == "edit": return "vi2v" if image_count > 0 else "v2v" - return "i2v" if image_count > 0 else "t2v" + return "i2v" if image_count > 0 else "r2v" # code based on https://github.com/RH-RunningHub/ComfyUI-RH-Bernini/blob/main/nodes_bernini.py diff --git a/scripts/test_t2v_r2v_and_loader.py b/scripts/test_t2v_r2v_and_loader.py new file mode 100644 index 0000000..079438a --- /dev/null +++ b/scripts/test_t2v_r2v_and_loader.py @@ -0,0 +1,450 @@ +#!/usr/bin/env python3 +"""Standalone verification: T2V == R2V normalization + one-model loader. + +No ComfyUI required. Checks (matching the standing verification rules): + 1. utils/h3_project.py - h3_task_type / h3_generation_mode normalize T2V + to the exact same value as R2V (functions extracted via AST). + 2. nodes/basic.py - _multitrack_task_type mirrors the normalization. + 3. utils/llm_api.py - reference-mode with zero media falls back to the + cloud-native text-only mode instead of raising. + 4. moxie_loader.py - schema shows a single model picker; execute() + auto-picks LoRA/VAE/CLIP from the pack's models/ folders and still + applies the LoRA to model+clip at strength 1.0. + +Run: python3 scripts/test_t2v_r2v_and_loader.py +""" + +from __future__ import annotations + +import ast +import enum +import importlib.util +import os +import sys +import tempfile +import types + +FAILED = [] + + +def check(name, condition, detail=""): + status = "PASS" if condition else "FAIL" + print(f"[{status}] {name}" + (f" - {detail}" if detail and not condition else "")) + if not condition: + FAILED.append(name) + + +# --------------------------------------------------------------------------- +# AST extraction helpers +# --------------------------------------------------------------------------- + +def extract_functions(path, names, constants=()): + """Return {name: function} for top-level (or class-level) defs. + + All requested functions plus any requested module-level constants are + compiled into ONE shared namespace so they can reference each other. + """ + with open(path, "r", encoding="utf-8") as handle: + tree = ast.parse(handle.read()) + body = [] + + def visit(nodes): + for node in nodes: + if isinstance(node, ast.FunctionDef) and node.name in names: + body.append(ast.FunctionDef( + name=node.name, args=node.args, body=node.body, + decorator_list=[], returns=node.returns, + type_comment=None, type_params=[], + )) + elif isinstance(node, ast.ClassDef): + visit(node.body) + elif ( + isinstance(node, ast.Assign) and constants and any( + isinstance(target, ast.Name) and target.id in constants + for target in node.targets + ) + ): + body.append(node) + + visit(tree.body) + ast.fix_missing_locations(ast.Module(body=body, type_ignores=[])) + namespace = {"__name__": "extracted"} + exec(compile(ast.Module(body=body, type_ignores=[]), path, "exec"), namespace) + missing = set(names) - set(namespace) + if missing: + raise RuntimeError(f"could not extract from {path}: {sorted(missing)}") + return namespace + + +def extract_constant(path, name): + with open(path, "r", encoding="utf-8") as handle: + tree = ast.parse(handle.read()) + for node in tree.body: + if isinstance(node, ast.Assign) and any( + isinstance(target, ast.Name) and target.id == name for target in node.targets + ): + return ast.literal_eval(node.value) + raise RuntimeError(f"constant {name} not found in {path}") + + +# --------------------------------------------------------------------------- +# 1. utils/h3_project.py - T2V -> R2V at the resolution choke point +# --------------------------------------------------------------------------- + +def verify_h3_project(): + print("\n== 1. utils/h3_project.py: h3_task_type / h3_generation_mode ==") + h3_dir = os.path.join(os.path.dirname(os.path.dirname(os.path.abspath(__file__))), "utils") + h3_file = os.path.join(h3_dir, "h3_project.py") + + h3_ns = extract_functions( + h3_file, + ["h3_task_type", "h3_generation_mode", "_normalize_task_type", "_frame_value"], + constants=("_T2V_EQUIVALENT_TASK_TYPE",), + ) + equiv = h3_ns["_T2V_EQUIVALENT_TASK_TYPE"] + check("t2v equivalent constant is r2v", equiv == "r2v", repr(equiv)) + + def muted_image(image): + return bool(isinstance(image, dict) and image.get("muted")) + + h3_ns["multitrack_is_muted_image"] = muted_image + h3_task_type = h3_ns["h3_task_type"] + h3_generation_mode = h3_ns["h3_generation_mode"] + + def make_info(with_video=False): + tracks = [{"type": "task", "segments": [{"start_frame": 0, "end_frame": 48, "content": {}}]}] + if with_video: + tracks.append({ + "type": "video", + "segments": [{"start_frame": 0, "end_frame": 48, "content": {}}], + }) + return {"tracks": tracks} + + def entry_for(mode=None, task_type=None, images=None, info=None): + content = {} + if mode is not None: + content["task_mode"] = mode + if task_type is not None: + content["task_type"] = task_type + if images is not None: + content["images"] = images + return {"start_frame": 0, "end_frame": 48, "task": {"content": content}} + + check("default mode, no images -> r2v (was t2v)", + h3_task_type(entry_for(mode="default"), make_info()) == "r2v") + check("no task_mode, no images -> r2v (was t2v)", + h3_task_type(entry_for(), make_info()) == "r2v") + check("explicit t2v -> r2v", + h3_task_type(entry_for(task_type="t2v"), make_info()) == "r2v") + check("explicit T2V (uppercase) -> r2v", + h3_task_type(entry_for(task_type="T2V"), make_info()) == "r2v") + check("explicit custom type passes through", + h3_task_type(entry_for(task_type="wan-2.2-fun"), make_info()) == "wan-2.2-fun") + check("muted images ignored -> r2v (was t2v)", + h3_task_type(entry_for(images=[{"muted": True}]), make_info()) == "r2v") + check("one active image -> i2v unchanged", + h3_task_type(entry_for(images=[{"id": "a"}]), make_info()) == "i2v") + check("ref mode, no video -> r2v", + h3_task_type(entry_for(mode="ref"), make_info()) == "r2v") + check("ref mode, video in range -> rv2v", + h3_task_type(entry_for(mode="ref"), make_info(with_video=True)) == "rv2v") + check("l2v mode -> l2v", + h3_task_type(entry_for(mode="l2v"), make_info()) == "l2v") + check("edit mode, no images -> v2v", + h3_task_type(entry_for(mode="edit"), make_info()) == "v2v") + + check("generation_mode(t2v) -> reference (belt and braces)", + h3_generation_mode("t2v") == "reference") + check("generation_mode(r2v) -> reference", + h3_generation_mode("r2v") == "reference") + check("generation_mode(rv2v) -> reference", + h3_generation_mode("rv2v") == "reference") + check("generation_mode(i2v) -> multi_frames", + h3_generation_mode("i2v") == "multi_frames") + check("generation_mode(l2v) -> last_frame", + h3_generation_mode("l2v") == "last_frame") + + +# --------------------------------------------------------------------------- +# 2. nodes/basic.py - editor task output mirrors the normalization +# --------------------------------------------------------------------------- + +def verify_basic_node(): + print("\n== 2. nodes/basic.py: _multitrack_task_type ==") + basic_file = os.path.join( + os.path.dirname(os.path.dirname(os.path.abspath(__file__))), "nodes", "basic.py" + ) + resolver = extract_functions(basic_file, ["_multitrack_task_type"])["_multitrack_task_type"] + check("explicit t2v -> r2v", + resolver({"content": {"task_type": "t2v"}}, 0, False) == "r2v") + check("default mode, no images -> r2v (was t2v)", + resolver({"content": {"task_mode": "default"}}, 0, False) == "r2v") + check("no content -> r2v (was t2v)", resolver({}, 0, False) == "r2v") + check("custom type kept", + resolver({"content": {"task_type": "wan-2.2-fun"}}, 0, False) == "wan-2.2-fun") + check("ref mode no video -> r2v", + resolver({"content": {"task_mode": "ref"}}, 0, False) == "r2v") + check("one image -> i2v", + resolver({"content": {"task_mode": "default"}}, 1, False) == "i2v") + check("edit mode no images -> v2v", + resolver({"content": {"task_mode": "edit"}}, 0, False) == "v2v") + + +# --------------------------------------------------------------------------- +# 3. utils/llm_api.py - reference-mode with zero media -> text-only cloud mode +# --------------------------------------------------------------------------- + +def verify_llm_api(): + print("\n== 3. utils/llm_api.py: _h3_media_for_task_type ==") + llm_file = os.path.join( + os.path.dirname(os.path.dirname(os.path.abspath(__file__))), "utils", "llm_api.py" + ) + media_resolver = extract_functions(llm_file, ["_h3_media_for_task_type"])["_h3_media_for_task_type"] + check("r2v with zero media -> t2va text-only (no raise)", + media_resolver("r2v", [], [], []) == ("t2va", [], [], [])) + check("rv2v with zero media -> t2va text-only", + media_resolver("rv2v", [], [], []) == ("t2va", [], [], [])) + check("r2v with media -> r2va", + media_resolver("r2v", ["img1", "img2"], ["vid"], ["aud"]) == ("r2va", ["img1", "img2"], ["vid"], ["aud"])) + check("direct t2v still -> t2va", + media_resolver("t2v", [], [], []) == ("t2va", [], [], [])) + check("i2v with image -> i2va", + media_resolver("i2v", ["img"], [], []) == ("i2va", ["img"], [], [])) + + +# --------------------------------------------------------------------------- +# 4. moxie_loader.py - one-model UI + auto-picked auxiliary models +# --------------------------------------------------------------------------- + +def _install_loader_stubs(state): + comfy_api = types.ModuleType("comfy_api") + comfy_latest = types.ModuleType("comfy_api.latest") + + class _FakeDiffusionModel: + def clone(self): + return _FakeDiffusionModel() + + def set_model_optimized_attention(self, attention_function): + state.calls.append(("set_model_optimized_attention",)) + return None + + state.model = _FakeDiffusionModel() + + class _Port: + def __init__(self, name, **kwargs): + self.name = name + self.kwargs = kwargs + + class _Schema: + def __init__(self, **kwargs): + for key, value in kwargs.items(): + setattr(self, key, value) + + class _NodeOutput: + def __init__(self, *values): + self.values = values + + io = types.SimpleNamespace( + Schema=_Schema, + NodeOutput=_NodeOutput, + ComfyNode=type("ComfyNode", (), {}), + Combo=types.SimpleNamespace(Input=staticmethod(lambda name, **kw: _Port(name, **kw))), + Float=types.SimpleNamespace(Input=staticmethod(lambda name, **kw: _Port(name, **kw))), + Model=types.SimpleNamespace(Output=staticmethod(lambda name=None, **kw: _Port(name, **kw))), + Clip=types.SimpleNamespace(Output=staticmethod(lambda name=None, **kw: _Port(name, **kw))), + Vae=types.SimpleNamespace(Output=staticmethod(lambda name=None, **kw: _Port(name, **kw))), + ) + comfy_latest.io = io + comfy_api.latest = comfy_latest + sys.modules["comfy_api"] = comfy_api + sys.modules["comfy_api.latest"] = comfy_latest + + folder_paths = types.ModuleType("folder_paths") + folder_paths.get_folder_paths = lambda kind: [] + sys.modules["folder_paths"] = folder_paths + + comfy = types.ModuleType("comfy") + + class _FakeVAE: + def __init__(self, sd=None, metadata=None): + self.sd = sd + + def throw_exception_if_invalid(self): + return None + + class _CLIPType(enum.Enum): + MINIMAX = "minimax" + + def load_diffusion_model(path, model_options=None): + state.calls.append(("load_diffusion_model", os.path.basename(path))) + return state.model + + def load_clip(ckpt_paths=None, embedding_directory=None, clip_type=None, model_options=None): + state.calls.append(("load_clip", clip_type)) + return state.clip + + def load_torch_file(path, safe_load=False, return_metadata=False): + state.calls.append(("load_torch_file", os.path.basename(path))) + sd = {"dummy": 1} + return (sd, {"meta": True}) if return_metadata else sd + + def load_lora_for_models(model, clip, lora_sd, strength_model, strength_clip, lora_metadata=None): + state.calls.append(("load_lora_for_models", strength_model, strength_clip)) + return state.lora_model, clip + + def get_attention_function(name, *args): + state.calls.append(("get_attention_function", name)) + return state.attention_fn + + def state_dict_prefix_replace(sd, mapping, filter_keys=False): + return sd + + comfy.sd = types.SimpleNamespace( + load_diffusion_model=load_diffusion_model, + load_clip=load_clip, + load_lora_for_models=load_lora_for_models, + VAE=_FakeVAE, + CLIPType=_CLIPType, + ) + comfy.utils = types.SimpleNamespace( + load_torch_file=load_torch_file, + state_dict_prefix_replace=state_dict_prefix_replace, + ) + comfy_ldm = types.ModuleType("comfy.ldm") + comfy_ldm_modules = types.ModuleType("comfy.ldm.modules") + comfy_attention = types.ModuleType("comfy.ldm.modules.attention") + comfy_attention.get_attention_function = get_attention_function + comfy.ldm = comfy_ldm + comfy.ldm.modules = comfy_ldm_modules + comfy.ldm.modules.attention = comfy_attention + comfy.__path__ = [] # make "comfy" importable as a package + sys.modules["comfy"] = comfy + sys.modules["comfy.sd"] = comfy.sd + sys.modules["comfy.utils"] = comfy.utils + sys.modules["comfy.ldm"] = comfy_ldm + sys.modules["comfy.ldm.modules"] = comfy_ldm_modules + sys.modules["comfy.ldm.modules.attention"] = comfy_attention + + +def _load_loader_module(): + pack_dir = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) + loader_file = os.path.join(pack_dir, "moxie_loader.py") + spec = importlib.util.spec_from_file_location("moxie_loader_standalone", loader_file) + module = importlib.util.module_from_spec(spec) + sys.modules[spec.name] = module + spec.loader.exec_module(module) + return module + + +def verify_loader(): + print("\n== 4. moxie_loader.py: single-model UI + auto-picked aux models ==") + state = types.SimpleNamespace( + calls=[], clip=object(), attention_fn=object(), lora_model=object(), + ) + _install_loader_stubs(state) + loader = _load_loader_module() + + schema = loader.MoxieMultimediaLoader.define_schema() + input_names = [port.name for port in schema.inputs] + output_names = [port.name for port in schema.outputs] + check("schema shows exactly the single-model inputs", + input_names == ["diffusion_model", "weight_dtype", "attention_backend"], + repr(input_names)) + for hidden in ("lora_name", "lora_strength_model", "lora_strength_clip", + "video_vae", "audio_vae", "clip", "clip_device"): + check(f"widget hidden: {hidden}", hidden not in input_names) + check("outputs still model/clip/video_vae/audio_vae", + [port.name or port.kwargs.get("display_name") for port in schema.outputs] + == ["model", "clip", "video_vae", "audio_vae"]) + + with tempfile.TemporaryDirectory() as tmp: + models = os.path.join(tmp, "models") + for subdir, name in ( + ("diffusion", "moxie.safetensors"), + ("vae", "video_vae.safetensors"), + ("audio_vae", "audio_vae.safetensors"), + ("clip", "MM-VL.safetensors"), + ("loras", "MM3step.safetensors"), + ): + os.makedirs(os.path.join(models, subdir), exist_ok=True) + with open(os.path.join(models, subdir, name), "wb") as handle: + handle.write(b"stub") + loader.MODELS_DIR = models + loader._loaded_lora = None + + result = loader.MoxieMultimediaLoader.execute( + diffusion_model="moxie.safetensors", + weight_dtype="default", + attention_backend="comfy_kitchen_attention", + ) + + names = [call[0] for call in state.calls] + check("full chain runs: model -> attention -> clip -> lora -> vaes", + names == [ + "load_diffusion_model", "get_attention_function", + "set_model_optimized_attention", "load_clip", + "load_torch_file", "load_lora_for_models", + "load_torch_file", "load_torch_file", + ], repr(names)) + lora_calls = [call for call in state.calls if call[0] == "load_lora_for_models"] + check("LoRA auto-applied at strength 1.0/1.0", + lora_calls == [("load_lora_for_models", 1.0, 1.0)], repr(lora_calls)) + clip_calls = [call for call in state.calls if call[0] == "load_clip"] + check("CLIP auto-picked and loaded with the locked recipe", + len(clip_calls) == 1 and clip_calls[0][1] is not None, repr(clip_calls)) + check("four outputs returned", + isinstance(result, loader.io.NodeOutput) and len(result.values) == 4) + check("model output carries LoRA", + result.values[0] is state.lora_model) + check("clip output is the loaded clip", result.values[1] is state.clip) + + # Empty loras folder -> LoRA silently skipped, everything else loads. + os.remove(os.path.join(models, "loras", "MM3step.safetensors")) + state.calls.clear() + loader._loaded_lora = None + loader.MoxieMultimediaLoader.execute( + diffusion_model="moxie.safetensors", + weight_dtype="default", + attention_backend="pytorch_attention", + ) + check("empty loras folder skips LoRA without breaking the load", + "load_lora_for_models" not in [call[0] for call in state.calls]) + + # Missing clip folder -> loud setup error (same message as before). + state.calls.clear() + os.rename(os.path.join(models, "clip"), os.path.join(models, "clip_backup")) + try: + loader.MoxieMultimediaLoader.execute( + diffusion_model="moxie.safetensors", + weight_dtype="default", + attention_backend="pytorch_attention", + ) + check("missing clip raises setup error", False) + except RuntimeError as error: + check("missing clip raises setup error", + "no model files found in models/clip" in str(error), str(error)) + os.rename(os.path.join(models, "clip_backup"), os.path.join(models, "clip")) + + # Multiple files -> deterministic first-sorted pick, no crash. + with open(os.path.join(models, "clip", "AAA-other.safetensors"), "wb") as handle: + handle.write(b"stub2") + picked = loader._auto_pick("clip") + check("multiple clip files pick first sorted deterministically", + picked == "AAA-other.safetensors", repr(picked)) + + +def main(): + verify_h3_project() + verify_basic_node() + verify_llm_api() + verify_loader() + print() + if FAILED: + print(f"FAILED CHECKS ({len(FAILED)}): {FAILED}") + return 1 + print("ALL CHECKS PASSED") + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/tests/test_h3_project.py b/tests/test_h3_project.py index 699b3ff..cac5a5f 100644 --- a/tests/test_h3_project.py +++ b/tests/test_h3_project.py @@ -93,7 +93,24 @@ def test_h3_task_type_ignores_muted_reference_images(): entry = h3_task_entries(info)[0] entry["task"]["content"]["images"] = [{"muted": True}] - assert h3_task_type(entry, info) == "t2v" + # The Moxie model set is reference-conditioned: T2V resolves to the + # exact same value as R2V. + assert h3_task_type(entry, info) == "r2v" + + +def test_h3_task_type_normalizes_explicit_t2v_to_r2v(): + info = _tracks_info() + entry = h3_task_entries(info)[0] + entry["task"]["content"]["task_type"] = "t2v" + + assert h3_task_type(entry, info) == "r2v" + + +def test_h3_generation_mode_maps_t2v_to_reference(): + # Belt and braces: even a raw t2v value must behave exactly like R2V. + assert h3_generation_mode("t2v") == "reference" + assert h3_generation_mode("r2v") == "reference" + assert h3_generation_mode("i2v") == "multi_frames" def test_parse_tracks_info_rejects_invalid_dimensions(): diff --git a/tests/test_llm_api.py b/tests/test_llm_api.py index 9111148..ac7057d 100644 --- a/tests/test_llm_api.py +++ b/tests/test_llm_api.py @@ -340,6 +340,9 @@ def test_video_data_uri_enforces_duration_even_when_file_is_under_size_limit(mon ("l2v", "i2va", 1, 0, 0), ("v2v", "r2va", 2, 1, 1), ("r2v", "r2va", 2, 1, 1), + # Reference-mode with zero media falls back to the cloud's native + # text-only mode (T2V selections resolve to r2v upstream). + ("r2v", "t2va", 0, 0, 0), ("vi2v", "r2va", 2, 1, 1), ("rv2v", "r2va", 2, 1, 1), ], diff --git a/tests/test_minimax_node.py b/tests/test_minimax_node.py index 15ed3fd..ff0f140 100644 --- a/tests/test_minimax_node.py +++ b/tests/test_minimax_node.py @@ -2305,7 +2305,8 @@ def test_multitrack_h3_project_expands_single_task_sampling_pipeline(monkeypatch assert nodes_by_type["KSamplerSelect"]["inputs"]["sampler_name"] == "er_sde" assert "ManualSigmas" in nodes_by_type conditioning = nodes_by_type["easy minimaxH3ToVideo"]["inputs"] - assert conditioning["mode"] == "multi_frames" + # T2V selections resolve to the exact same value as R2V (reference). + assert conditioning["mode"] == "reference" assert conditioning["ref_image_size"] == "match" assert (conditioning["width"], conditioning["height"]) == (1344, 768) assert "SamplerCustomAdvanced" in nodes_by_type diff --git a/tests/test_moxie_nodes.py b/tests/test_moxie_nodes.py index 2c75eb4..9cbf986 100644 --- a/tests/test_moxie_nodes.py +++ b/tests/test_moxie_nodes.py @@ -353,8 +353,25 @@ def test_loader_schema_locks_h3_clip_recipe(moxie): # The model set is MiniMax H3-only: no CLIP recipe choices are exposed. assert "clip_type" not in inputs - assert "clip_device" in inputs - assert inputs.index("clip") == len(inputs) - 2 # clip, then clip_device + + +def test_loader_schema_shows_a_single_model(moxie): + schema = moxie.loader.MoxieMultimediaLoader.define_schema() + inputs = _input_names(schema) + + # End users see one model picker plus the attention backend; every + # auxiliary model (LoRA, video/audio VAE, CLIP) is auto-selected. + assert inputs == ["diffusion_model", "weight_dtype", "attention_backend"] + for hidden in ( + "lora_name", + "lora_strength_model", + "lora_strength_clip", + "video_vae", + "audio_vae", + "clip", + "clip_device", + ): + assert hidden not in inputs def test_loader_execute_applies_attention_then_lora(moxie, monkeypatch, tmp_path): @@ -381,13 +398,6 @@ def test_loader_execute_applies_attention_then_lora(moxie, monkeypatch, tmp_path diffusion_model="h3.safetensors", weight_dtype="default", attention_backend="comfy_kitchen_attention", - lora_name="turbo.safetensors", - lora_strength_model=1.0, - lora_strength_clip=1.0, - video_vae="video.safetensors", - audio_vae="audio.safetensors", - clip="qwen.safetensors", - clip_device="default", ) sequence = [call[0] for call in state.calls] @@ -441,13 +451,6 @@ def test_loader_execute_without_lora_skips_lora_loading(moxie, monkeypatch, tmp_ diffusion_model="h3.safetensors", weight_dtype="default", attention_backend="pytorch_attention", - lora_name="", - lora_strength_model=1.0, - lora_strength_clip=1.0, - video_vae="video.safetensors", - audio_vae="audio.safetensors", - clip="qwen.safetensors", - clip_device="default", ) assert "load_lora_for_models" not in [call[0] for call in state.calls] @@ -477,13 +480,6 @@ def test_loader_requires_minimax_cliptype(moxie, monkeypatch, tmp_path): diffusion_model="h3.safetensors", weight_dtype="default", attention_backend="pytorch_attention", - lora_name="", - lora_strength_model=1.0, - lora_strength_clip=1.0, - video_vae="video.safetensors", - audio_vae="audio.safetensors", - clip="qwen.safetensors", - clip_device="default", ) diff --git a/tests/test_multitrack_info_output.py b/tests/test_multitrack_info_output.py index 4e4922e..95ea7c1 100644 --- a/tests/test_multitrack_info_output.py +++ b/tests/test_multitrack_info_output.py @@ -1369,7 +1369,9 @@ def test_multitrack_editor_prompt_override_preserves_explicit_task_type(): assert task_content["task_type"] == "t2v" assert task_content["task_mode"] == "default" - assert task_result.values[2] == "t2v" + # Raw content keeps the explicit override tag, but resolved task types + # normalize T2V to the exact same value as R2V. + assert task_result.values[2] == "r2v" def test_multitrack_editor_prompt_override_outputs_custom_task_type_string(): diff --git a/utils/h3_project.py b/utils/h3_project.py index 7c15609..50577e2 100644 --- a/utils/h3_project.py +++ b/utils/h3_project.py @@ -400,6 +400,19 @@ def h3_second_pass_dimensions( ) +# The shipped Moxie model set is reference-conditioned (r2v): a T2V selection +# must drive the exact same value as R2V so the model is never conditioned as +# a plain text-to-video task. +_T2V_EQUIVALENT_TASK_TYPE = "r2v" + + +def _normalize_task_type(task_type: str) -> str: + normalized = task_type.strip().lower() + if normalized == "t2v": + return _T2V_EQUIVALENT_TASK_TYPE + return normalized + + def h3_task_type(entry: dict[str, Any], info: dict[str, Any]) -> str: task = entry.get("task", {}) content = task.get("content", {}) if isinstance(task, dict) else {} @@ -407,7 +420,7 @@ def h3_task_type(entry: dict[str, Any], info: dict[str, Any]) -> str: content = {} explicit_type = content.get("task_type") if isinstance(explicit_type, str) and explicit_type.strip(): - return explicit_type.strip().lower() + return _normalize_task_type(explicit_type) start_frame = _frame_value(entry.get("start_frame")) end_frame = _frame_value(entry.get("end_frame")) @@ -439,7 +452,7 @@ def h3_task_type(entry: dict[str, Any], info: dict[str, Any]) -> str: return "rv2v" if has_video else "r2v" if mode == "edit": return "vi2v" if image_count > 0 else "v2v" - return "i2v" if image_count > 0 else "t2v" + return "i2v" if image_count > 0 else _T2V_EQUIVALENT_TASK_TYPE def _h3_locked_track( @@ -501,7 +514,11 @@ def h3_generation_mode(task_type: str) -> str: return "reference" if normalized == "l2v": return "last_frame" - if normalized in {"t2v", "i2v"}: + if normalized == "t2v": + # Belt and braces: T2V must behave exactly like R2V even if a raw + # t2v value bypasses h3_task_type. + return "reference" + if normalized == "i2v": return "multi_frames" raise ValueError(f"Unsupported H3 task type: {task_type}") diff --git a/utils/llm_api.py b/utils/llm_api.py index d3189ed..7b49ad3 100644 --- a/utils/llm_api.py +++ b/utils/llm_api.py @@ -925,7 +925,11 @@ class PromptEnhancerClient: return "i2va", selected_images, [], [] if normalized_type in {"v2v", "r2v", "vi2v", "rv2v"}: if not (images or videos or audios): - raise ValueError(f"MiniMax {normalized_type} requires reference media.") + # Reference-mode with zero reference media: the local Moxie + # model handles text-only renders in reference conditioning, + # but the cloud API has a native text-only mode - use it + # instead of failing (T2V selections resolve to r2v). + return "t2va", [], [], [] if len(images) > 9: raise ValueError("MiniMax r2va supports at most 9 reference images.") if len(videos) > 3: