mmprompt/easy-media-track-gate/references/workflow_wiring.md

101 lines
5.0 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

# Workflow wiring: render ALL scenes in one run + the LLM slot
Verified against ComfyUI-Easy-Media source (`nodes/basic.py`, `nodes/minimax.py`,
`nodes/project.py`, `utils/llm_api.py`) and `example_workflows/MinimaxH3_ForLoop_MultiShot.json`.
## Why only scene 1 renders
The render chain reads the timeline through `easy multiTrackTaskOutput`, which outputs
**one task segment per pass**, picked by its `task_index` int input. Default `0` →
the first clip only. The number of prompts inside the pasted `prompt_override` JSON
does not change this: the JSON is complete (all scenes are in it); the loop lives in
the workflow, not in the JSON. Nothing needs to be added to the payload for multi-shot.
Side facts (same node, verified in source):
- `task_index = -1` outputs the **whole timeline as one merged segment** (all task
contents joined, full frame length). It is for relay/audio-lock cases, NOT a way to
render 3 clips: the local H3 model is trained at ~124–362 frames (~5–15 s); a 30 s
merged pass is out of range.
- `easy multiTrackInfoOutput` (tracks_info in) exposes `TASK_COUNT` = number of task
segments — 3 for the Sam payload. Also WIDTH/HEIGHT/TOTAL_FRAMES/FPS.
## The single-run loop (from the pack's own ForLoop example)
One queue run renders every clip and concatenates them:
```
editor.TRACKS_INFO ─→ multiTrackInfoOutput ─→ TASK_COUNT ─→ forLoopStart.total
editor.TRACKS_INFO ─→ multiTrackTaskOutput (images/audio/video too)
forLoopStart.index ─→ multiTrackTaskOutput.task_index
multiTrackTaskOutput.SYSTEM_PROMPT/USER_PROMPT/TYPE/LENGTH/IMAGES/AUDIO ─→ render chain
render chain → saveVideo.file_path ─→ batchAnything ─→ forLoopEnd.initial_value1
forLoopEnd.value1 ─→ mergeVideosFromPaths ─→ SaveVideo
```
The example adds two math nodes so a single PrimitiveInt `k` (default `-1`) chooses:
- `total = (k<=-1)*TASK_COUNT + (k>-1)*1`
- `task_index = (k<=-1)*loop_index + (k>-1)*k`
`k = -1` → loop all segments (all scenes, one run). `k = 2` → render only segment 2.
Manual fallback without rewiring: queue once per index (0, 1, 2) and join the files.
Per-iteration the task output gives LENGTH ≈ frames for that clip only (240 frames =
10 s — inside the model's trained range), so each loop pass is a normal single-clip
generation.
## The LLM slot (where your own LLM connects)
In the ForLoop example the per-scene prompt may pass through an LLM before rendering:
```
TaskOutput.SYSTEM_PROMPT ─→ [LLM node].system_prompt
TaskOutput.USER_PROMPT ─→ [LLM node].custom_prompt
TaskOutput.IMAGES ─→ [LLM node].images (optional, vision only)
[LLM node].output ─→ ifElse("Using Prompt Enhance") ─→ cleanGpuUsed ─→ render.prompt
ifElse.on_false = raw TaskOutput.USER_PROMPT (bypass switch)
```
The example's LLM is `llama_cpp_instruct_adv` + `llama_cpp_model_loader` (a local GGUF).
That node is replaceable — it is just "system+user (+images) in, one string out".
Three ways to fill the slot:
### Option A — no LLM node at all (default for this skill)
The skill already wrote every scene prompt (identity text, continuity rules, LoRA
trigger) into `prompt_override.json`. Leave the enhance toggle OFF: raw
`USER_PROMPT` flows to the render node each pass. The "LLM that made all the scenes"
is this skill, outside ComfyUI.
### Option B — the pack's built-in enhancer with any OpenAI-compatible server
Add `easy multiTrackPromptEnhancer` inside the loop:
- `system_prompt` ← TaskOutput.SYSTEM_PROMPT, `user_prompt` ← TaskOutput.USER_PROMPT,
`type` ← TaskOutput.TYPE, `length` ← TaskOutput.LENGTH, images/audio/video optional.
- `model` = `openai_compatible (local)`; set `base_url` to ANY OpenAI-compatible
Chat Completions server — Ollama `http://127.0.0.1:11434/v1`, LM Studio
`http://127.0.0.1:1234/v1`, vLLM, llama.cpp server, or a hosted endpoint
(empty base_url reads `OPENAI_LOCAL_BASE_URL` from the pack's config.yaml, else
`http://127.0.0.1:8080/v1`). `apikey`/`model_name` optional depending on server.
Built-in presets also include MiniMax H3-Context-IR (`v2/h3_context_ir`),
bigmodel.cn (GLM) and RunningHub LLM keys from config.yaml.
- Output `PROMPT` → render node's prompt.
### Option C — any ComfyUI LLM node in the loop
Keep the example's wiring shape and swap node 86 for the preferred LLM node
(system_prompt + user_prompt inputs, STRING output). `prompt_format` on TaskOutput
can stay `api`; setting it to `llm` instead makes USER_PROMPT a single ready-made
chat transcript for LLM nodes that have only one prompt input.
## Local project path (alternative, no loop nodes)
`easy multitrackProject` (editor tracks_info in, `segment_count = -1`) builds and runs
every segment itself in one queue — see `MiniMaxH3_v1.3.1_Easy_Project_AllInOne.json`.
It contains no LLM node; prompts come from the timeline JSON. This is the pack's own
agent-driven setup: an external LLM (agent) writes the whole `track_data`, then one
run renders it. The pack even ships `skills/easy-media-multitrack-workflow` +
`patch_workflow.py` for exactly that flow (our SKILL.md step 5B).