refactor: remove the LoRA pipeline; distill weights ship baked into the model

The 3-step distill (speedup) LoRA is now baked into the diffusion model
itself, so the Moxie Multimedia Loader no longer applies any LoRA:

- moxie_loader.py: drop the models/loras scan, _auto_lora(), the module
  level LoRA cache and _apply_lora(); the load chain is now
  diffusion model -> attention backend -> CLIP -> VAEs. A leftover file
  in models/loras is silently ignored and the folder can be deleted.
- moxie_project.py / moxie_preview.py: tooltips reference the baked-in
  distill instead of the LoRA.
- tests + standalone loader script: assert no load_lora_for_models call
  happens (the comfy.sd stub now raises if the loader tries), and fix
  the pre-existing MMnode.tiny_vae stub gap (missing pack_vae_approx_files)
  that broke the whole test module at fixture setup.
- README: models table drops the Turbo LoRA row; CHANGELOG entry added.

Node schema, outputs and sampling behavior are unchanged; workflows
saved before this change keep working.
This commit is contained in:
Z User 2026-10-13 03:34:47 +00:00
parent 08229541c3
commit 88849ced65
7 changed files with 34 additions and 131 deletions

View File

@ -6,6 +6,7 @@
### ⚡ Improvements
- **Moxie Multimedia Loader**: Removed the automatic LoRA application entirely. The 3-step distill (speedup) weights are now baked into the diffusion model itself, so the loader no longer reads `models/loras` — a leftover file there is silently ignored, and the `models/loras` folder can be deleted. The node schema, outputs, and load order (diffusion model → attention backend → CLIP → VAEs) are unchanged; workflows saved earlier keep working. The `Moxie Project` steps tooltip now references the baked-in distill instead of the LoRA.
- **FFmpeg resolution & diagnostics**: The pack now finds FFmpeg on far more Windows setups and tells you exactly what is wrong when it cannot. When `ffmpeg` is not on the ComfyUI process's PATH (PATH edits only reach processes started afterwards), it is additionally looked up in the classic `C:\ffmpeg\bin`, the ComfyUI/portable folders (including `ffmpeg\bin` next to the run bat), winget/scoop/chocolatey shim locations, and finally the ffmpeg binary bundled with ComfyUI's own `imageio-ffmpeg` dependency. When composing an H3 project still fails, the error dialog now names the concrete cause instead of a generic message: the missing executable, the exact missing segment/locked-audio file paths, odd canvas sizes, or the tail of FFmpeg's own stderr output (e.g. `Unknown encoder 'libx264'`).
- **Installation**: The pack now ships a **`requirements.txt`**, so ComfyUI Manager installs the dependencies automatically and manual/portable installs have a one-line command (`python_embeded\python.exe -m pip install -r ...` for ComfyUI Portable — see the new "Installing dependencies" section in the README). Because this pack targets **NVIDIA RTX 3060+ GPUs only**, every feature dependency is mandatory: `soundfile`, `jieba`, `huggingface_hub`, `ffmpeg-python`, `opencv-python`, `imageio` (smart split), `nvidia-vfx` (RTX VSR upscale), `openai-whisper` and `qwen-asr` (subtitle recognition), and `voxcpm` (speech). No optional extras remain — previously these were installed on demand with error messages naming the missing package. The file also routes `nvidia-vfx` to NVIDIA's own index (`--extra-index-url https://pypi.nvidia.com/` + `--only-binary=nvidia-vfx`): the PyPI copy is a stub sdist whose `wheel_stub` build backend can abort the whole install (`Cannot import 'wheel_stub.buildapi'`), while NVIDIA's index serves real prebuilt wheels.
- **Installation**: Removed the bundled `download_models.sh` downloader. All model files live in the Hugging Face version of the repository — cloning it downloads the complete model set, so there is no separate downloader step (the README's Models section now explains this).

View File

@ -61,11 +61,10 @@ All model files ship **inside the repo** — cloning it into `custom_nodes` is a
| File | Location |
|---|---|
| Diffusion model (`Moxie-Multimedia.safetensors`) | `models/diffusion/` |
| Diffusion model (`Moxie-Multimedia.safetensors`, distill weights baked in) | `models/diffusion/` |
| Text encoder (`MM-VL.safetensors`) | `models/clip/` |
| Video VAE (bundled file) | `models/vae/` |
| Audio VAE (bundled file) | `models/audio_vae/` |
| Turbo LoRA (`MM3step.safetensors`, applied by default) | `models/loras/` |
| Preview tiny VAE (`MM-preview.safetensors`) | `models/vae_approx/` |
All model files live in the Hugging Face version of this repository — cloning it downloads the complete set, and there is no separate downloader step. If you obtained the pack from somewhere else and the `models/` folders are missing or incomplete, re-clone from Hugging Face to get the bundled files.

View File

@ -19,7 +19,6 @@ SUBDIR_HINTS = {
"vae": "models/vae",
"audio_vae": "models/audio_vae",
"clip": "models/clip",
"loras": "models/loras",
}
@ -53,18 +52,11 @@ def _resolve(subdir, name):
return path
# (path, sd, metadata) cache for the last parsed LoRA, mirroring the native
# LoraLoader. Kept at module level on purpose: newer ComfyUI builds lock node
# classes after registration, so writing class attributes on the node class
# itself raises "Cannot modify class attribute ... on locked class".
_loaded_lora = None
def _auto_pick(subdir):
"""Return the model file to use for a hidden picker.
End users only see the diffusion model widget; every auxiliary model
(video VAE, audio VAE, CLIP, LoRA) is auto-selected from the pack's
(video VAE, audio VAE, CLIP) is auto-selected from the pack's
models/<subdir> folder - the first file in sorted order, which for the
published pack is exactly the bundled file. A missing file keeps the
loud, actionable error from _resolve.
@ -85,14 +77,6 @@ def _auto_pick(subdir):
return selected
def _auto_lora():
"""Bundled LoRA file, or ``None`` when models/loras is empty."""
files = _list_files("loras")
if files[0].startswith("<no files found"):
return None
return files[0]
def _h3_clip_type():
"""MiniMax H3 CLIP recipe. The Moxie model set is H3-only, so the text
encoder type is intentionally NOT a user-facing choice."""
@ -110,12 +94,12 @@ class MoxieMultimediaLoader(io.ComfyNode):
"""All-in-one loader that looks like it loads a single model.
The only visible picker is the diffusion model; the attention backend is
applied to it, the bundled LoRA (if any), the video VAE, the audio VAE
and the CLIP are auto-selected from the node pack's models/ folder.
applied to it, and the video VAE, the audio VAE and the CLIP are
auto-selected from the node pack's models/ folder.
The model set is MiniMax H3-only: the CLIP recipe is locked to
CLIPType.MINIMAX and outputs are exactly (model, clip, video_vae, audio_vae)
with the attention backend and LoRA already applied to the model."""
with the attention backend already applied to the model."""
@classmethod
def define_schema(cls) -> io.Schema:
@ -123,7 +107,7 @@ class MoxieMultimediaLoader(io.ComfyNode):
node_id="MoxieMultimediaLoader",
display_name="Moxie Multimedia Loader",
category="loaders/moxie",
description="Loads the Moxie diffusion model (with the attention backend and the bundled LoRA "
description="Loads the Moxie diffusion model (with the attention backend "
"already applied) plus the video VAE, audio VAE and text encoder that ship with "
"this custom node - one model picker, everything else is handled automatically. "
"Outputs model, clip, video_vae and audio_vae.",
@ -148,8 +132,8 @@ class MoxieMultimediaLoader(io.ComfyNode):
),
],
outputs=[
io.Model.Output(display_name="model", tooltip="Diffusion model with attention backend and LoRA applied."),
io.Clip.Output(display_name="clip", tooltip="CLIP text encoder (with LoRA applied if present)."),
io.Model.Output(display_name="model", tooltip="Diffusion model with attention backend applied."),
io.Clip.Output(display_name="clip", tooltip="CLIP text encoder."),
io.Vae.Output(display_name="video_vae", tooltip="Loaded video/image VAE."),
io.Vae.Output(display_name="audio_vae", tooltip="Loaded audio VAE."),
],
@ -189,36 +173,6 @@ class MoxieMultimediaLoader(io.ComfyNode):
m.set_model_optimized_attention(attention_function)
return m
@classmethod
def _apply_lora(cls, model, clip, lora_name, strength_model, strength_clip):
global _loaded_lora
if (
lora_name is None
or lora_name == "none"
or lora_name.startswith("<no files found")
or (strength_model == 0 and strength_clip == 0)
):
return model, clip
lora_path = _resolve("loras", lora_name)
lora = None
lora_metadata = None
if _loaded_lora is not None:
if _loaded_lora[0] == lora_path:
lora = _loaded_lora[1]
lora_metadata = _loaded_lora[2] if len(_loaded_lora) > 2 else None
else:
_loaded_lora = None
if lora is None:
lora, lora_metadata = comfy.utils.load_torch_file(lora_path, safe_load=True, return_metadata=True)
_loaded_lora = (lora_path, lora, lora_metadata)
model_lora, clip_lora = comfy.sd.load_lora_for_models(
model, clip, lora, strength_model, strength_clip, lora_metadata=lora_metadata
)
return model_lora, clip_lora
@classmethod
def _load_vae(cls, vae_name, subdir="vae"):
path = _resolve(subdir, vae_name)
@ -256,15 +210,13 @@ class MoxieMultimediaLoader(io.ComfyNode):
weight_dtype,
attention_backend,
) -> io.NodeOutput:
# Chain: Load Diffusion Model -> Model Attention Backend -> Lora Loader (needs both model and clip)
# Chain: Load Diffusion Model -> Model Attention Backend
model = cls._load_diffusion_model(diffusion_model, weight_dtype)
model = cls._apply_attention_backend(model, attention_backend)
# Hidden pickers: auxiliary models are auto-selected from the pack's
# models/ folder so end users only ever see the diffusion model.
clip_obj = cls._load_clip(_auto_pick("clip"), "default")
lora_name = _auto_lora()
model, clip_obj = cls._apply_lora(model, clip_obj, lora_name, 1.0, 1.0)
video_vae_obj = cls._load_vae(_auto_pick("vae"), "vae")
audio_vae_obj = cls._load_vae(_auto_pick("audio_vae"), "audio_vae")

View File

@ -1006,7 +1006,7 @@ class ModelPreviewOverrideKJ(io.ComfyNode):
inputs=[
io.Model.Input(
"model",
tooltip="Diffusion model (with attention backend and LoRA already applied).",
tooltip="Diffusion model (with attention backend already applied).",
),
io.Int.Input(
"max_resolution",

View File

@ -52,7 +52,7 @@ class MoxieProject(EasyMultiTrackProject):
not_idempotent=True,
inputs=[
TYPE_TRACKS_INFO.Input("tracks_info"),
io.Model.Input("model", tooltip="Diffusion model from Moxie Multimedia Loader (attention backend and LoRA already applied)."),
io.Model.Input("model", tooltip="Diffusion model from Moxie Multimedia Loader (attention backend already applied)."),
io.Clip.Input("clip", tooltip="CLIP text encoder from Moxie Multimedia Loader."),
io.Vae.Input("video_vae", tooltip="Video VAE from Moxie Multimedia Loader."),
io.Vae.Input("audio_vae", tooltip="Audio VAE from Moxie Multimedia Loader (required to decode Moxie audio)."),
@ -63,8 +63,8 @@ class MoxieProject(EasyMultiTrackProject):
default=3,
min=1,
max=10000,
tooltip="Sampling steps for the beta scheduler (3 for the 3-step turbo LoRA; "
"8 for busy motion with regular LoRAs, 6 low motion, 4 runs/no dialogue).",
tooltip="Sampling steps for the beta scheduler (3 for the baked-in 3-step distill; "
"8 for busy motion, 6 low motion, 4 runs/no dialogue).",
),
io.Int.Input(
"seed",

View File

@ -294,8 +294,7 @@ def _install_loader_stubs(state):
return (sd, {"meta": True}) if return_metadata else sd
def load_lora_for_models(model, clip, lora_sd, strength_model, strength_clip, lora_metadata=None):
state.calls.append(("load_lora_for_models", strength_model, strength_clip))
return state.lora_model, clip
raise AssertionError("MoxieMultimediaLoader must not load LoRAs anymore")
def get_attention_function(name, *args):
state.calls.append(("get_attention_function", name))
@ -352,7 +351,7 @@ def _load_loader_module():
def verify_loader():
print("\n== 4. moxie_loader.py: single-model UI + auto-picked aux models ==")
state = types.SimpleNamespace(
calls=[], clip=object(), attention_fn=object(), lora_model=object(),
calls=[], clip=object(), attention_fn=object(),
)
_install_loader_stubs(state)
loader = _load_loader_module()
@ -377,13 +376,14 @@ def verify_loader():
("vae", "video_vae.safetensors"),
("audio_vae", "audio_vae.safetensors"),
("clip", "MM-VL.safetensors"),
# A leftover loras file must be ignored entirely: the distill
# weights ship baked into the diffusion model now.
("loras", "MM3step.safetensors"),
):
os.makedirs(os.path.join(models, subdir), exist_ok=True)
with open(os.path.join(models, subdir, name), "wb") as handle:
handle.write(b"stub")
loader.MODELS_DIR = models
loader._loaded_lora = None
result = loader.MoxieMultimediaLoader.execute(
diffusion_model="moxie.safetensors",
@ -392,37 +392,21 @@ def verify_loader():
)
names = [call[0] for call in state.calls]
check("full chain runs: model -> attention -> clip -> lora -> vaes",
check("full chain runs: model -> attention -> clip -> vaes (no LoRA)",
names == [
"load_diffusion_model", "get_attention_function",
"set_model_optimized_attention", "load_clip",
"load_torch_file", "load_lora_for_models",
"load_torch_file", "load_torch_file",
], repr(names))
lora_calls = [call for call in state.calls if call[0] == "load_lora_for_models"]
check("LoRA auto-applied at strength 1.0/1.0",
lora_calls == [("load_lora_for_models", 1.0, 1.0)], repr(lora_calls))
clip_calls = [call for call in state.calls if call[0] == "load_clip"]
check("CLIP auto-picked and loaded with the locked recipe",
len(clip_calls) == 1 and clip_calls[0][1] is not None, repr(clip_calls))
check("four outputs returned",
isinstance(result, loader.io.NodeOutput) and len(result.values) == 4)
check("model output carries LoRA",
result.values[0] is state.lora_model)
check("model output is the attention clone (no LoRA on top)",
result.values[0] is not state.model)
check("clip output is the loaded clip", result.values[1] is state.clip)
# Empty loras folder -> LoRA silently skipped, everything else loads.
os.remove(os.path.join(models, "loras", "MM3step.safetensors"))
state.calls.clear()
loader._loaded_lora = None
loader.MoxieMultimediaLoader.execute(
diffusion_model="moxie.safetensors",
weight_dtype="default",
attention_backend="pytorch_attention",
)
check("empty loras folder skips LoRA without breaking the load",
"load_lora_for_models" not in [call[0] for call in state.calls])
# Missing clip folder -> loud setup error (same message as before).
state.calls.clear()
os.rename(os.path.join(models, "clip"), os.path.join(models, "clip_backup"))

View File

@ -7,7 +7,8 @@ Verified contracts:
- MoxieMultimediaLoader: outputs exactly (model, clip, video_vae, audio_vae);
no legacy FAST_MODEL_LOADER bundle output; no clip_type choice (the CLIP
recipe is locked to CLIPType.MINIMAX); the model output has the attention
backend and LoRA applied.
backend applied, and any leftover file in models/loras is ignored (the
distill weights ship baked into the diffusion model).
- MoxieProject: takes model/clip/video_vae/audio_vae as required direct
inputs instead of a bundled model_loader, and packs them into the loader
dict consumed internally by EasyMultiTrackProject.
@ -122,8 +123,6 @@ class _StubState:
self.calls = []
self.attention_fn = lambda pixels, *a, **k: pixels
self.clip = object()
self.lora_model = None
self.lora_clip = None
def _register(state, io, monkeypatch=None):
@ -158,10 +157,7 @@ def _register(state, io, monkeypatch=None):
return state.clip
def load_lora_for_models(model, clip, lora_sd, strength_model, strength_clip, lora_metadata=None):
state.calls.append(("load_lora_for_models", strength_model, strength_clip))
state.lora_model = model
state.lora_clip = clip
return model, clip
raise AssertionError("MoxieMultimediaLoader must not load LoRAs anymore")
comfy_sd.CLIPType = CLIPType
comfy_sd.load_diffusion_model = load_diffusion_model
@ -303,6 +299,7 @@ def _build_moxie():
tiny_vae = types.ModuleType("MMnode.tiny_vae")
tiny_vae.load_tiny_vae_decoder = lambda *a, **k: None
tiny_vae.pack_vae_approx_files = lambda *a, **k: []
added["MMnode.tiny_vae"] = sys.modules.get("MMnode.tiny_vae")
sys.modules["MMnode.tiny_vae"] = tiny_vae
@ -360,7 +357,7 @@ def test_loader_schema_shows_a_single_model(moxie):
inputs = _input_names(schema)
# End users see one model picker plus the attention backend; every
# auxiliary model (LoRA, video/audio VAE, CLIP) is auto-selected.
# auxiliary model (video/audio VAE, CLIP) is auto-selected.
assert inputs == ["diffusion_model", "weight_dtype", "attention_backend"]
for hidden in (
"lora_name",
@ -374,7 +371,7 @@ def test_loader_schema_shows_a_single_model(moxie):
assert hidden not in inputs
def test_loader_execute_applies_attention_then_lora(moxie, monkeypatch, tmp_path):
def test_loader_execute_applies_attention_and_ignores_loras(moxie, monkeypatch, tmp_path):
state = moxie.state
models = tmp_path / "models"
for subdir in ("diffusion", "vae", "audio_vae", "clip", "loras"):
@ -389,9 +386,6 @@ def test_loader_execute_applies_attention_then_lora(moxie, monkeypatch, tmp_path
}[name]
(models / subdir / name).write_bytes(b"stub")
monkeypatch.setattr(moxie.loader, "MODELS_DIR", models)
# LoRA cache lives at module level: node classes are locked by newer
# ComfyUI builds and class attributes can no longer be written at runtime.
monkeypatch.setattr(moxie.loader, "_loaded_lora", None)
state.calls.clear()
result = moxie.loader.MoxieMultimediaLoader.execute(
@ -400,14 +394,15 @@ def test_loader_execute_applies_attention_then_lora(moxie, monkeypatch, tmp_path
attention_backend="comfy_kitchen_attention",
)
# The distill weights ship baked into the diffusion model: a leftover
# file in models/loras must be ignored entirely. The comfy.sd stub keeps
# a raising load_lora_for_models, so an accidental LoRA call fails loudly.
sequence = [call[0] for call in state.calls]
assert sequence == [
"load_diffusion_model",
"get_attention_function",
"load_clip",
"load_torch_file",
"load_lora_for_models",
"load_torch_file",
"load_torch_file",
]
@ -415,50 +410,22 @@ def test_loader_execute_applies_attention_then_lora(moxie, monkeypatch, tmp_path
clip_calls = [call for call in state.calls if call[0] == "load_clip"]
assert clip_calls == [("load_clip", state.CLIPType.MINIMAX)]
# LoRA applied on top of the attention-patched model, after the CLIP load.
# Attention backend applied before the CLIP load.
assert state.calls.index(("get_attention_function", "comfy_kitchen_int8")) < state.calls.index(
("load_clip", state.CLIPType.MINIMAX)
) < state.calls.index(("load_lora_for_models", 1.0, 1.0))
)
# Exactly four direct outputs, model carrying attention + LoRA.
# Exactly four direct outputs, model carrying the attention backend.
assert isinstance(result, _NodeOutput)
assert len(result.values) == 4
model_out, clip_out, video_vae_out, audio_vae_out = result.values
assert model_out is state.lora_model
assert model_out.label == "diffusion.clone"
assert model_out.attention_fn is state.attention_fn
assert clip_out is state.clip
assert isinstance(video_vae_out, _FakeVAE)
assert isinstance(audio_vae_out, _FakeVAE)
def test_loader_execute_without_lora_skips_lora_loading(moxie, monkeypatch, tmp_path):
state = moxie.state
models = tmp_path / "models"
for subdir in ("diffusion", "vae", "audio_vae", "clip", "loras"):
(models / subdir).mkdir(parents=True, exist_ok=True)
for name in ("h3.safetensors", "video.safetensors", "audio.safetensors", "qwen.safetensors"):
subdir = {
"h3.safetensors": "diffusion",
"video.safetensors": "vae",
"audio.safetensors": "audio_vae",
"qwen.safetensors": "clip",
}[name]
(models / subdir / name).write_bytes(b"stub")
monkeypatch.setattr(moxie.loader, "MODELS_DIR", models)
state.calls.clear()
result = moxie.loader.MoxieMultimediaLoader.execute(
diffusion_model="h3.safetensors",
weight_dtype="default",
attention_backend="pytorch_attention",
)
assert "load_lora_for_models" not in [call[0] for call in state.calls]
assert result.values[0] is not state.clip # model output remains a model
assert result.values[0].attention_fn is state.attention_fn
assert result.values[1] is state.clip
def test_loader_requires_minimax_cliptype(moxie, monkeypatch, tmp_path):
state = moxie.state
models = tmp_path / "models"