mmprompt/easy-media-track-gate/references/workflow_wiring.md

193 lines
10 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

# Workflow wiring: render ALL scenes in one run + the LLM slot
Verified against ComfyUI-Easy-Media source (`nodes/basic.py`, `nodes/minimax.py`,
`nodes/project.py`, `utils/llm_api.py`) and `example_workflows/MinimaxH3_ForLoop_MultiShot.json`.
## Why only scene 1 renders
The render chain reads the timeline through `easy multiTrackTaskOutput`, which outputs
**one task segment per pass**, picked by its `task_index` int input. Default `0` →
the first clip only. The number of prompts inside the pasted `prompt_override` JSON
does not change this: the JSON is complete (all scenes are in it); the loop lives in
the workflow, not in the JSON. Nothing needs to be added to the payload for multi-shot.
Side facts (same node, verified in source):
- `task_index = -1` switches the node to **full-timeline mode**: window 0..720,
LENGTH = all frames snapped to the H3 17k+5 grid (720 → 719), media from all
segments. What the prompt becomes depends on `prompt_format` — see the
single-pass section below (`promptRelay` makes it render all scenes in one pass —
which also puts every frame into ONE sampler call; OOM ceiling below).
- `easy multiTrackInfoOutput` (tracks_info in) exposes `TASK_COUNT` = number of task
segments — 3 for the Sam payload. Also WIDTH/HEIGHT/TOTAL_FRAMES/FPS.
## The prompt_override input requires the `type` field (proven)
`MultiTrackEditor.execute` (`nodes/basic.py` L1949-1957): with `format = "MiniMax"`
it calls `is_minimax_prompt_override(prompt_override)` FIRST. Only a payload with
`"type": "minimax_prompt_override"` and a `prompts` list passes
(`utils/prompt_override.py` `parse_minimax_prompt_override`). Any other non-empty
value — including a full TRACKS_INFO JSON — falls to the `|`-text parser, which
treats the ENTIRE string as one prompt: one segment 0..479 whose `text` is the
literal JSON. Proven by running the repo's parser on the captured workflow
(`scripts/prove_relay_chain.py` STEP A). So: paste the compact payload (with the
type field) into `prompt_override`; full TRACKS_INFO belongs only in `track_data`.
## Single-pass timeline render: task_index=-1 + prompt_format=promptRelay (short timelines only)
This is the pattern the user's own graph (`my timeline ref.json`) is wired for —
no ForLoop, one sampler pass, all scenes in one generation. It renders, but only
when the WHOLE timeline fits in one pass — the OOM ceiling at the end of this
section is a hard limit:
1. Node 186 (String → `prompt_override`): the minimal payload — `type` + `prompts` are
the only required keys (`track-schema.md`); optional keys only when non-default.
2. `easy multiTrackTaskOutput`: `task_index = -1` AND `prompt_format = "promptRelay"`.
3. Everything else stays as-is: TYPE output is the payload's `generation_type` string
verbatim (`task_type` is set from it in the override path, and the parser's code
default is `r2v` — so even the minimal 2-key payload renders TYPE `r2v` → matchLine →
`multi_frames`), LENGTH output becomes 719, one SamplerCustomAdvanced pass renders the
whole timeline.
With `promptRelay` + full-timeline mode, TaskOutput builds the prompt from ALL
task segments (`_format_marker_task_prompt_relay`), each scene suffixed with its
native H3 frame range, joined by ` | `:
```
<scene 1 text> [0-240] | <scene 2 text> [240-480] | <scene 3 text> [480-720]
```
SYSTEM_PROMPT output is empty in this mode; the ranges are the H3 timeline
syntax (`_FRAME_RANGE_RE` parses the same `[s-e]` / `[s-e,s-type]` tokens in the
override parser). Proven by executing the repo's own functions on the Sam
payload (STEP C: 3 ranges, length 719, type t2v).
Both settings are required — `task_index = -1` with `prompt_format = "default"`
keeps only the FIRST task's prompt (STEP D): a 719-frame latent conditioned on
scene 1 alone.
**OOM ceiling (user-verified 2026-11):** one pass = the ENTIRE timeline's frames
in a single sampler call. At the Sam story's 3 × 240 = 720 total frames the
render attempted one ~30 s clip and died with an out-of-memory error — and 720
is also ~2× the trained range (~124–362 frames, node tooltip: "longer is
untested"), so quality was never validated that far either. The pack's own
long-sequence plumbing (`multi_frames`, BlockSparseAttention `min_tokens 12288`)
does not remove the VRAM cost of one 720-frame pass on a consumer card.
Rule: use `task_index = -1` + `promptRelay` only when the whole timeline stays
inside the trained range — total ≤ ~360 frames, i.e. two short scenes. Anything
longer: render per clip (next section) and join the files.
## The single-run loop (from the pack's own ForLoop example)
One queue run renders every clip and concatenates them:
```
editor.TRACKS_INFO ─→ multiTrackInfoOutput ─→ TASK_COUNT ─→ forLoopStart.total
editor.TRACKS_INFO ─→ multiTrackTaskOutput (images/audio/video too)
forLoopStart.index ─→ multiTrackTaskOutput.task_index
multiTrackTaskOutput.SYSTEM_PROMPT/USER_PROMPT/TYPE/LENGTH/IMAGES/AUDIO ─→ render chain
render chain → saveVideo.file_path ─→ batchAnything ─→ forLoopEnd.initial_value1
forLoopEnd.value1 ─→ mergeVideosFromPaths ─→ SaveVideo
```
The example adds two math nodes so a single PrimitiveInt `k` (default `-1`) chooses:
- `total = (k<=-1)*TASK_COUNT + (k>-1)*1`
- `task_index = (k<=-1)*loop_index + (k>-1)*k`
`k = -1` → loop all segments (all scenes, one run). `k = 2` → render only segment 2.
Manual fallback without any rewiring (verified in source, `nodes/basic.py`
`MultiTrackTaskOutput`): keep `prompt_format = "promptRelay"` and just set
`task_index` to 0, 1, 2 — one queue run each. The all-tasks relay fires only when
`output_full_timeline` is set (-1) or the segment's own prompt contains `[s-e]`
markers (`marker_mode`); for a marker-free payload, promptRelay at
`task_index = k` formats the ONE selected clip as `<scene k text> [start-end]`
with LENGTH = that clip's frame count (240 — inside the trained range). `default`
gives the bare scene text if the range suffix is not wanted. Join the files.
Per-iteration the task output gives LENGTH ≈ frames for that clip only (240 frames =
10 s — inside the model's trained range), so each loop pass is a normal single-clip
generation.
## The LLM slot (where your own LLM connects)
In the ForLoop example the per-scene prompt may pass through an LLM before rendering:
```
TaskOutput.SYSTEM_PROMPT ─→ [LLM node].system_prompt
TaskOutput.USER_PROMPT ─→ [LLM node].custom_prompt
TaskOutput.IMAGES ─→ [LLM node].images (optional, vision only)
[LLM node].output ─→ ifElse("Using Prompt Enhance") ─→ cleanGpuUsed ─→ render.prompt
ifElse.on_false = raw TaskOutput.USER_PROMPT (bypass switch)
```
The example's LLM is `llama_cpp_instruct_adv` + `llama_cpp_model_loader` (a local GGUF).
That node is replaceable — it is just "system+user (+images) in, one string out".
Three ways to fill the slot:
### Option A — no LLM node at all (default for this skill)
The skill already wrote every scene prompt (identity text, continuity rules, LoRA
trigger) into `prompt_override.json`. Leave the enhance toggle OFF: raw
`USER_PROMPT` flows to the render node each pass. The "LLM that made all the scenes"
is this skill, outside ComfyUI.
### Option B — the pack's built-in enhancer with any OpenAI-compatible server
Add `easy multiTrackPromptEnhancer` inside the loop:
- `system_prompt` ← TaskOutput.SYSTEM_PROMPT, `user_prompt` ← TaskOutput.USER_PROMPT,
`type` ← TaskOutput.TYPE, `length` ← TaskOutput.LENGTH, images/audio/video optional.
- `model` = `openai_compatible (local)`; set `base_url` to ANY OpenAI-compatible
Chat Completions server — Ollama `http://127.0.0.1:11434/v1`, LM Studio
`http://127.0.0.1:1234/v1`, vLLM, llama.cpp server, or a hosted endpoint
(empty base_url reads `OPENAI_LOCAL_BASE_URL` from the pack's config.yaml, else
`http://127.0.0.1:8080/v1`). `apikey`/`model_name` optional depending on server.
Built-in presets also include MiniMax H3-Context-IR (`v2/h3_context_ir`),
bigmodel.cn (GLM) and RunningHub LLM keys from config.yaml.
- Output `PROMPT` → render node's prompt.
### Option C — any ComfyUI LLM node in the loop
Keep the example's wiring shape and swap node 86 for the preferred LLM node
(system_prompt + user_prompt inputs, STRING output). `prompt_format` on TaskOutput
can stay `api`; setting it to `llm` instead makes USER_PROMPT a single ready-made
chat transcript for LLM nodes that have only one prompt input.
## Local project path (alternative, no loop nodes)
`easy multitrackProject` (editor tracks_info in, `segment_count = -1`) builds and runs
every segment itself in one queue — see `MiniMaxH3_v1.3.1_Easy_Project_AllInOne.json`.
It contains no LLM node; prompts come from the timeline JSON. This is the pack's own
agent-driven setup: an external LLM (agent) writes the whole `track_data`, then one
run renders it. The pack even ships `skills/easy-media-multitrack-workflow` +
`patch_workflow.py` for exactly that flow (our SKILL.md step 5B).
## The built-in LLM system prompts (published, not locked)
The rewrite-LLM contract ships in the open in the pack:
`utils/prompt_builder.py` — `MINIMAX_BASE_PROMPT` (~15.7 KB, "Video Prompt Writing
Guide (T2VA / I2VA / FL2VA / L2VA)") and `MINIMAX_REF_PROMPT` (~23.5 KB, "Full-
Reference Mode Rewrite Output Format Guide" — six-section rewrite with
`<Subject N>` / `<Picture N>` / `<Video N>` / `<Audio N>` labels).
Selection (`build_prompt_request`, MiniMax branch):
- `task_mode` `default` / `l2v` (our t2v payloads) → `MINIMAX_BASE_PROMPT`
- `task_mode` `ref` / `edit` → `MINIMAX_REF_PROMPT`
- a non-empty per-segment `system_prompt` overrides both; with an empty one the
context-segment continuity rules ride the MiniMax rewrite API server-side
(entry rule: not-first segment using `context`; tail rule: next segment uses
`context`) — per the pack's own `multitrack-schema.md`.
Delivery: the system prompt only leaves TaskOutput via `prompt_format`
`api` or `llm` (`default` / `promptRelay` output an empty SYSTEM_PROMPT).
`build_llm_prompt` folds system+user into one transcript for single-input LLM
nodes. To use the user's own LLM with the same behavior: paste
`minimax_base_prompt.txt` (or the ref guide for ref-mode work) into the LLM
node's system prompt field and wire TaskOutput SYSTEM_PROMPT/USER_PROMPT into
it — or leave the LLM's system prompt empty and pass TaskOutput.SYSTEM_PROMPT
through, which already carries the selected guide verbatim.