173 lines
9.0 KiB
Markdown
173 lines
9.0 KiB
Markdown
# Workflow wiring: render ALL scenes in one run + the LLM slot
|
||
|
||
Verified against ComfyUI-Easy-Media source (`nodes/basic.py`, `nodes/minimax.py`,
|
||
`nodes/project.py`, `utils/llm_api.py`) and `example_workflows/MinimaxH3_ForLoop_MultiShot.json`.
|
||
|
||
## Why only scene 1 renders
|
||
|
||
The render chain reads the timeline through `easy multiTrackTaskOutput`, which outputs
|
||
**one task segment per pass**, picked by its `task_index` int input. Default `0` →
|
||
the first clip only. The number of prompts inside the pasted `prompt_override` JSON
|
||
does not change this: the JSON is complete (all scenes are in it); the loop lives in
|
||
the workflow, not in the JSON. Nothing needs to be added to the payload for multi-shot.
|
||
|
||
Side facts (same node, verified in source):
|
||
|
||
- `task_index = -1` switches the node to **full-timeline mode**: window 0..720,
|
||
LENGTH = all frames snapped to the H3 17k+5 grid (720 → 719), media from all
|
||
segments. What the prompt becomes depends on `prompt_format` — see the
|
||
single-pass section below (`promptRelay` makes it render all scenes in one pass).
|
||
- `easy multiTrackInfoOutput` (tracks_info in) exposes `TASK_COUNT` = number of task
|
||
segments — 3 for the Sam payload. Also WIDTH/HEIGHT/TOTAL_FRAMES/FPS.
|
||
|
||
## The prompt_override input requires the `type` field (proven)
|
||
|
||
`MultiTrackEditor.execute` (`nodes/basic.py` L1949-1957): with `format = "MiniMax"`
|
||
it calls `is_minimax_prompt_override(prompt_override)` FIRST. Only a payload with
|
||
`"type": "minimax_prompt_override"` and a `prompts` list passes
|
||
(`utils/prompt_override.py` `parse_minimax_prompt_override`). Any other non-empty
|
||
value — including a full TRACKS_INFO JSON — falls to the `|`-text parser, which
|
||
treats the ENTIRE string as one prompt: one segment 0..479 whose `text` is the
|
||
literal JSON. Proven by running the repo's parser on the captured workflow
|
||
(`scripts/prove_relay_chain.py` STEP A). So: paste the compact payload (with the
|
||
type field) into `prompt_override`; full TRACKS_INFO belongs only in `track_data`.
|
||
|
||
## Single-pass timeline render: task_index=-1 + prompt_format=promptRelay
|
||
|
||
This is the pattern the user's own graph (`my timeline ref.json`) is wired for —
|
||
no ForLoop, one sampler pass, all scenes in one generation:
|
||
|
||
1. Node 186 (String → `prompt_override`): the compact payload WITH `type` field.
|
||
2. `easy multiTrackTaskOutput`: `task_index = -1` AND `prompt_format = "promptRelay"`.
|
||
3. Everything else stays as-is: TYPE output stays `t2v` (first task's explicit
|
||
task_type → matchLine → `multi_frames`), LENGTH output becomes 719, one
|
||
SamplerCustomAdvanced pass renders the whole timeline.
|
||
|
||
With `promptRelay` + full-timeline mode, TaskOutput builds the prompt from ALL
|
||
task segments (`_format_marker_task_prompt_relay`), each scene suffixed with its
|
||
native H3 frame range, joined by ` | `:
|
||
|
||
```
|
||
<scene 1 text> [0-240] | <scene 2 text> [240-480] | <scene 3 text> [480-720]
|
||
```
|
||
|
||
SYSTEM_PROMPT output is empty in this mode; the ranges are the H3 timeline
|
||
syntax (`_FRAME_RANGE_RE` parses the same `[s-e]` / `[s-e,s-type]` tokens in the
|
||
override parser). Proven by executing the repo's own functions on the Sam
|
||
payload (STEP C: 3 ranges, length 719, type t2v).
|
||
|
||
Both settings are required — `task_index = -1` with `prompt_format = "default"`
|
||
keeps only the FIRST task's prompt (STEP D): a 719-frame latent conditioned on
|
||
scene 1 alone.
|
||
|
||
Caveat from the node's own tooltip: the trained range is ~124–362 frames and
|
||
"longer is untested". 719 frames (~30 s) is the intended use of this graph shape
|
||
(`multi_frames` mode + BlockSparseAttention `min_tokens 12288` exist for long
|
||
sequences), but if quality degrades, fall back to the ForLoop pattern above,
|
||
which renders each 240-frame clip inside the trained range.
|
||
|
||
## The single-run loop (from the pack's own ForLoop example)
|
||
|
||
One queue run renders every clip and concatenates them:
|
||
|
||
```
|
||
editor.TRACKS_INFO ─→ multiTrackInfoOutput ─→ TASK_COUNT ─→ forLoopStart.total
|
||
editor.TRACKS_INFO ─→ multiTrackTaskOutput (images/audio/video too)
|
||
forLoopStart.index ─→ multiTrackTaskOutput.task_index
|
||
multiTrackTaskOutput.SYSTEM_PROMPT/USER_PROMPT/TYPE/LENGTH/IMAGES/AUDIO ─→ render chain
|
||
render chain → saveVideo.file_path ─→ batchAnything ─→ forLoopEnd.initial_value1
|
||
forLoopEnd.value1 ─→ mergeVideosFromPaths ─→ SaveVideo
|
||
```
|
||
|
||
The example adds two math nodes so a single PrimitiveInt `k` (default `-1`) chooses:
|
||
|
||
- `total = (k<=-1)*TASK_COUNT + (k>-1)*1`
|
||
- `task_index = (k<=-1)*loop_index + (k>-1)*k`
|
||
|
||
`k = -1` → loop all segments (all scenes, one run). `k = 2` → render only segment 2.
|
||
Manual fallback without rewiring: queue once per index (0, 1, 2) and join the files.
|
||
|
||
Per-iteration the task output gives LENGTH ≈ frames for that clip only (240 frames =
|
||
10 s — inside the model's trained range), so each loop pass is a normal single-clip
|
||
generation.
|
||
|
||
## The LLM slot (where your own LLM connects)
|
||
|
||
In the ForLoop example the per-scene prompt may pass through an LLM before rendering:
|
||
|
||
```
|
||
TaskOutput.SYSTEM_PROMPT ─→ [LLM node].system_prompt
|
||
TaskOutput.USER_PROMPT ─→ [LLM node].custom_prompt
|
||
TaskOutput.IMAGES ─→ [LLM node].images (optional, vision only)
|
||
[LLM node].output ─→ ifElse("Using Prompt Enhance") ─→ cleanGpuUsed ─→ render.prompt
|
||
ifElse.on_false = raw TaskOutput.USER_PROMPT (bypass switch)
|
||
```
|
||
|
||
The example's LLM is `llama_cpp_instruct_adv` + `llama_cpp_model_loader` (a local GGUF).
|
||
That node is replaceable — it is just "system+user (+images) in, one string out".
|
||
Three ways to fill the slot:
|
||
|
||
### Option A — no LLM node at all (default for this skill)
|
||
|
||
The skill already wrote every scene prompt (identity text, continuity rules, LoRA
|
||
trigger) into `prompt_override.json`. Leave the enhance toggle OFF: raw
|
||
`USER_PROMPT` flows to the render node each pass. The "LLM that made all the scenes"
|
||
is this skill, outside ComfyUI.
|
||
|
||
### Option B — the pack's built-in enhancer with any OpenAI-compatible server
|
||
|
||
Add `easy multiTrackPromptEnhancer` inside the loop:
|
||
|
||
- `system_prompt` ← TaskOutput.SYSTEM_PROMPT, `user_prompt` ← TaskOutput.USER_PROMPT,
|
||
`type` ← TaskOutput.TYPE, `length` ← TaskOutput.LENGTH, images/audio/video optional.
|
||
- `model` = `openai_compatible (local)`; set `base_url` to ANY OpenAI-compatible
|
||
Chat Completions server — Ollama `http://127.0.0.1:11434/v1`, LM Studio
|
||
`http://127.0.0.1:1234/v1`, vLLM, llama.cpp server, or a hosted endpoint
|
||
(empty base_url reads `OPENAI_LOCAL_BASE_URL` from the pack's config.yaml, else
|
||
`http://127.0.0.1:8080/v1`). `apikey`/`model_name` optional depending on server.
|
||
Built-in presets also include MiniMax H3-Context-IR (`v2/h3_context_ir`),
|
||
bigmodel.cn (GLM) and RunningHub LLM keys from config.yaml.
|
||
- Output `PROMPT` → render node's prompt.
|
||
|
||
### Option C — any ComfyUI LLM node in the loop
|
||
|
||
Keep the example's wiring shape and swap node 86 for the preferred LLM node
|
||
(system_prompt + user_prompt inputs, STRING output). `prompt_format` on TaskOutput
|
||
can stay `api`; setting it to `llm` instead makes USER_PROMPT a single ready-made
|
||
chat transcript for LLM nodes that have only one prompt input.
|
||
|
||
## Local project path (alternative, no loop nodes)
|
||
|
||
`easy multitrackProject` (editor tracks_info in, `segment_count = -1`) builds and runs
|
||
every segment itself in one queue — see `MiniMaxH3_v1.3.1_Easy_Project_AllInOne.json`.
|
||
It contains no LLM node; prompts come from the timeline JSON. This is the pack's own
|
||
agent-driven setup: an external LLM (agent) writes the whole `track_data`, then one
|
||
run renders it. The pack even ships `skills/easy-media-multitrack-workflow` +
|
||
`patch_workflow.py` for exactly that flow (our SKILL.md step 5B).
|
||
|
||
## The built-in LLM system prompts (published, not locked)
|
||
|
||
The rewrite-LLM contract ships in the open in the pack:
|
||
`utils/prompt_builder.py` — `MINIMAX_BASE_PROMPT` (~15.7 KB, "Video Prompt Writing
|
||
Guide (T2VA / I2VA / FL2VA / L2VA)") and `MINIMAX_REF_PROMPT` (~23.5 KB, "Full-
|
||
Reference Mode Rewrite Output Format Guide" — six-section rewrite with
|
||
`<Subject N>` / `<Picture N>` / `<Video N>` / `<Audio N>` labels).
|
||
|
||
Selection (`build_prompt_request`, MiniMax branch):
|
||
|
||
- `task_mode` `default` / `l2v` (our t2v payloads) → `MINIMAX_BASE_PROMPT`
|
||
- `task_mode` `ref` / `edit` → `MINIMAX_REF_PROMPT`
|
||
- a non-empty per-segment `system_prompt` overrides both; with an empty one the
|
||
context-segment continuity rules ride the MiniMax rewrite API server-side
|
||
(entry rule: not-first segment using `context`; tail rule: next segment uses
|
||
`context`) — per the pack's own `multitrack-schema.md`.
|
||
|
||
Delivery: the system prompt only leaves TaskOutput via `prompt_format`
|
||
`api` or `llm` (`default` / `promptRelay` output an empty SYSTEM_PROMPT).
|
||
`build_llm_prompt` folds system+user into one transcript for single-input LLM
|
||
nodes. To use the user's own LLM with the same behavior: paste
|
||
`minimax_base_prompt.txt` (or the ref guide for ref-mode work) into the LLM
|
||
node's system prompt field and wire TaskOutput SYSTEM_PROMPT/USER_PROMPT into
|
||
it — or leave the LLM's system prompt empty and pass TaskOutput.SYSTEM_PROMPT
|
||
through, which already carries the selected guide verbatim.
|