mmprompt/easy-media-track-gate/references/workflow_wiring.md

173 lines
9.0 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

# Workflow wiring: render ALL scenes in one run + the LLM slot
Verified against ComfyUI-Easy-Media source (`nodes/basic.py`, `nodes/minimax.py`,
`nodes/project.py`, `utils/llm_api.py`) and `example_workflows/MinimaxH3_ForLoop_MultiShot.json`.
## Why only scene 1 renders
The render chain reads the timeline through `easy multiTrackTaskOutput`, which outputs
**one task segment per pass**, picked by its `task_index` int input. Default `0` →
the first clip only. The number of prompts inside the pasted `prompt_override` JSON
does not change this: the JSON is complete (all scenes are in it); the loop lives in
the workflow, not in the JSON. Nothing needs to be added to the payload for multi-shot.
Side facts (same node, verified in source):
- `task_index = -1` switches the node to **full-timeline mode**: window 0..720,
LENGTH = all frames snapped to the H3 17k+5 grid (720 → 719), media from all
segments. What the prompt becomes depends on `prompt_format` — see the
single-pass section below (`promptRelay` makes it render all scenes in one pass).
- `easy multiTrackInfoOutput` (tracks_info in) exposes `TASK_COUNT` = number of task
segments — 3 for the Sam payload. Also WIDTH/HEIGHT/TOTAL_FRAMES/FPS.
## The prompt_override input requires the `type` field (proven)
`MultiTrackEditor.execute` (`nodes/basic.py` L1949-1957): with `format = "MiniMax"`
it calls `is_minimax_prompt_override(prompt_override)` FIRST. Only a payload with
`"type": "minimax_prompt_override"` and a `prompts` list passes
(`utils/prompt_override.py` `parse_minimax_prompt_override`). Any other non-empty
value — including a full TRACKS_INFO JSON — falls to the `|`-text parser, which
treats the ENTIRE string as one prompt: one segment 0..479 whose `text` is the
literal JSON. Proven by running the repo's parser on the captured workflow
(`scripts/prove_relay_chain.py` STEP A). So: paste the compact payload (with the
type field) into `prompt_override`; full TRACKS_INFO belongs only in `track_data`.
## Single-pass timeline render: task_index=-1 + prompt_format=promptRelay
This is the pattern the user's own graph (`my timeline ref.json`) is wired for —
no ForLoop, one sampler pass, all scenes in one generation:
1. Node 186 (String → `prompt_override`): the compact payload WITH `type` field.
2. `easy multiTrackTaskOutput`: `task_index = -1` AND `prompt_format = "promptRelay"`.
3. Everything else stays as-is: TYPE output stays `t2v` (first task's explicit
task_type → matchLine → `multi_frames`), LENGTH output becomes 719, one
SamplerCustomAdvanced pass renders the whole timeline.
With `promptRelay` + full-timeline mode, TaskOutput builds the prompt from ALL
task segments (`_format_marker_task_prompt_relay`), each scene suffixed with its
native H3 frame range, joined by ` | `:
```
<scene 1 text> [0-240] | <scene 2 text> [240-480] | <scene 3 text> [480-720]
```
SYSTEM_PROMPT output is empty in this mode; the ranges are the H3 timeline
syntax (`_FRAME_RANGE_RE` parses the same `[s-e]` / `[s-e,s-type]` tokens in the
override parser). Proven by executing the repo's own functions on the Sam
payload (STEP C: 3 ranges, length 719, type t2v).
Both settings are required — `task_index = -1` with `prompt_format = "default"`
keeps only the FIRST task's prompt (STEP D): a 719-frame latent conditioned on
scene 1 alone.
Caveat from the node's own tooltip: the trained range is ~124–362 frames and
"longer is untested". 719 frames (~30 s) is the intended use of this graph shape
(`multi_frames` mode + BlockSparseAttention `min_tokens 12288` exist for long
sequences), but if quality degrades, fall back to the ForLoop pattern above,
which renders each 240-frame clip inside the trained range.
## The single-run loop (from the pack's own ForLoop example)
One queue run renders every clip and concatenates them:
```
editor.TRACKS_INFO ─→ multiTrackInfoOutput ─→ TASK_COUNT ─→ forLoopStart.total
editor.TRACKS_INFO ─→ multiTrackTaskOutput (images/audio/video too)
forLoopStart.index ─→ multiTrackTaskOutput.task_index
multiTrackTaskOutput.SYSTEM_PROMPT/USER_PROMPT/TYPE/LENGTH/IMAGES/AUDIO ─→ render chain
render chain → saveVideo.file_path ─→ batchAnything ─→ forLoopEnd.initial_value1
forLoopEnd.value1 ─→ mergeVideosFromPaths ─→ SaveVideo
```
The example adds two math nodes so a single PrimitiveInt `k` (default `-1`) chooses:
- `total = (k<=-1)*TASK_COUNT + (k>-1)*1`
- `task_index = (k<=-1)*loop_index + (k>-1)*k`
`k = -1` → loop all segments (all scenes, one run). `k = 2` → render only segment 2.
Manual fallback without rewiring: queue once per index (0, 1, 2) and join the files.
Per-iteration the task output gives LENGTH ≈ frames for that clip only (240 frames =
10 s — inside the model's trained range), so each loop pass is a normal single-clip
generation.
## The LLM slot (where your own LLM connects)
In the ForLoop example the per-scene prompt may pass through an LLM before rendering:
```
TaskOutput.SYSTEM_PROMPT ─→ [LLM node].system_prompt
TaskOutput.USER_PROMPT ─→ [LLM node].custom_prompt
TaskOutput.IMAGES ─→ [LLM node].images (optional, vision only)
[LLM node].output ─→ ifElse("Using Prompt Enhance") ─→ cleanGpuUsed ─→ render.prompt
ifElse.on_false = raw TaskOutput.USER_PROMPT (bypass switch)
```
The example's LLM is `llama_cpp_instruct_adv` + `llama_cpp_model_loader` (a local GGUF).
That node is replaceable — it is just "system+user (+images) in, one string out".
Three ways to fill the slot:
### Option A — no LLM node at all (default for this skill)
The skill already wrote every scene prompt (identity text, continuity rules, LoRA
trigger) into `prompt_override.json`. Leave the enhance toggle OFF: raw
`USER_PROMPT` flows to the render node each pass. The "LLM that made all the scenes"
is this skill, outside ComfyUI.
### Option B — the pack's built-in enhancer with any OpenAI-compatible server
Add `easy multiTrackPromptEnhancer` inside the loop:
- `system_prompt` ← TaskOutput.SYSTEM_PROMPT, `user_prompt` ← TaskOutput.USER_PROMPT,
`type` ← TaskOutput.TYPE, `length` ← TaskOutput.LENGTH, images/audio/video optional.
- `model` = `openai_compatible (local)`; set `base_url` to ANY OpenAI-compatible
Chat Completions server — Ollama `http://127.0.0.1:11434/v1`, LM Studio
`http://127.0.0.1:1234/v1`, vLLM, llama.cpp server, or a hosted endpoint
(empty base_url reads `OPENAI_LOCAL_BASE_URL` from the pack's config.yaml, else
`http://127.0.0.1:8080/v1`). `apikey`/`model_name` optional depending on server.
Built-in presets also include MiniMax H3-Context-IR (`v2/h3_context_ir`),
bigmodel.cn (GLM) and RunningHub LLM keys from config.yaml.
- Output `PROMPT` → render node's prompt.
### Option C — any ComfyUI LLM node in the loop
Keep the example's wiring shape and swap node 86 for the preferred LLM node
(system_prompt + user_prompt inputs, STRING output). `prompt_format` on TaskOutput
can stay `api`; setting it to `llm` instead makes USER_PROMPT a single ready-made
chat transcript for LLM nodes that have only one prompt input.
## Local project path (alternative, no loop nodes)
`easy multitrackProject` (editor tracks_info in, `segment_count = -1`) builds and runs
every segment itself in one queue — see `MiniMaxH3_v1.3.1_Easy_Project_AllInOne.json`.
It contains no LLM node; prompts come from the timeline JSON. This is the pack's own
agent-driven setup: an external LLM (agent) writes the whole `track_data`, then one
run renders it. The pack even ships `skills/easy-media-multitrack-workflow` +
`patch_workflow.py` for exactly that flow (our SKILL.md step 5B).
## The built-in LLM system prompts (published, not locked)
The rewrite-LLM contract ships in the open in the pack:
`utils/prompt_builder.py` — `MINIMAX_BASE_PROMPT` (~15.7 KB, "Video Prompt Writing
Guide (T2VA / I2VA / FL2VA / L2VA)") and `MINIMAX_REF_PROMPT` (~23.5 KB, "Full-
Reference Mode Rewrite Output Format Guide" — six-section rewrite with
`<Subject N>` / `<Picture N>` / `<Video N>` / `<Audio N>` labels).
Selection (`build_prompt_request`, MiniMax branch):
- `task_mode` `default` / `l2v` (our t2v payloads) → `MINIMAX_BASE_PROMPT`
- `task_mode` `ref` / `edit` → `MINIMAX_REF_PROMPT`
- a non-empty per-segment `system_prompt` overrides both; with an empty one the
context-segment continuity rules ride the MiniMax rewrite API server-side
(entry rule: not-first segment using `context`; tail rule: next segment uses
`context`) — per the pack's own `multitrack-schema.md`.
Delivery: the system prompt only leaves TaskOutput via `prompt_format`
`api` or `llm` (`default` / `promptRelay` output an empty SYSTEM_PROMPT).
`build_llm_prompt` folds system+user into one transcript for single-input LLM
nodes. To use the user's own LLM with the same behavior: paste
`minimax_base_prompt.txt` (or the ref guide for ref-mode work) into the LLM
node's system prompt field and wire TaskOutput SYSTEM_PROMPT/USER_PROMPT into
it — or leave the LLM's system prompt empty and pass TaskOutput.SYSTEM_PROMPT
through, which already carries the selected guide verbatim.