101 lines
5.0 KiB
Markdown
101 lines
5.0 KiB
Markdown
# Workflow wiring: render ALL scenes in one run + the LLM slot
|
||
|
||
Verified against ComfyUI-Easy-Media source (`nodes/basic.py`, `nodes/minimax.py`,
|
||
`nodes/project.py`, `utils/llm_api.py`) and `example_workflows/MinimaxH3_ForLoop_MultiShot.json`.
|
||
|
||
## Why only scene 1 renders
|
||
|
||
The render chain reads the timeline through `easy multiTrackTaskOutput`, which outputs
|
||
**one task segment per pass**, picked by its `task_index` int input. Default `0` →
|
||
the first clip only. The number of prompts inside the pasted `prompt_override` JSON
|
||
does not change this: the JSON is complete (all scenes are in it); the loop lives in
|
||
the workflow, not in the JSON. Nothing needs to be added to the payload for multi-shot.
|
||
|
||
Side facts (same node, verified in source):
|
||
|
||
- `task_index = -1` outputs the **whole timeline as one merged segment** (all task
|
||
contents joined, full frame length). It is for relay/audio-lock cases, NOT a way to
|
||
render 3 clips: the local H3 model is trained at ~124–362 frames (~5–15 s); a 30 s
|
||
merged pass is out of range.
|
||
- `easy multiTrackInfoOutput` (tracks_info in) exposes `TASK_COUNT` = number of task
|
||
segments — 3 for the Sam payload. Also WIDTH/HEIGHT/TOTAL_FRAMES/FPS.
|
||
|
||
## The single-run loop (from the pack's own ForLoop example)
|
||
|
||
One queue run renders every clip and concatenates them:
|
||
|
||
```
|
||
editor.TRACKS_INFO ─→ multiTrackInfoOutput ─→ TASK_COUNT ─→ forLoopStart.total
|
||
editor.TRACKS_INFO ─→ multiTrackTaskOutput (images/audio/video too)
|
||
forLoopStart.index ─→ multiTrackTaskOutput.task_index
|
||
multiTrackTaskOutput.SYSTEM_PROMPT/USER_PROMPT/TYPE/LENGTH/IMAGES/AUDIO ─→ render chain
|
||
render chain → saveVideo.file_path ─→ batchAnything ─→ forLoopEnd.initial_value1
|
||
forLoopEnd.value1 ─→ mergeVideosFromPaths ─→ SaveVideo
|
||
```
|
||
|
||
The example adds two math nodes so a single PrimitiveInt `k` (default `-1`) chooses:
|
||
|
||
- `total = (k<=-1)*TASK_COUNT + (k>-1)*1`
|
||
- `task_index = (k<=-1)*loop_index + (k>-1)*k`
|
||
|
||
`k = -1` → loop all segments (all scenes, one run). `k = 2` → render only segment 2.
|
||
Manual fallback without rewiring: queue once per index (0, 1, 2) and join the files.
|
||
|
||
Per-iteration the task output gives LENGTH ≈ frames for that clip only (240 frames =
|
||
10 s — inside the model's trained range), so each loop pass is a normal single-clip
|
||
generation.
|
||
|
||
## The LLM slot (where your own LLM connects)
|
||
|
||
In the ForLoop example the per-scene prompt may pass through an LLM before rendering:
|
||
|
||
```
|
||
TaskOutput.SYSTEM_PROMPT ─→ [LLM node].system_prompt
|
||
TaskOutput.USER_PROMPT ─→ [LLM node].custom_prompt
|
||
TaskOutput.IMAGES ─→ [LLM node].images (optional, vision only)
|
||
[LLM node].output ─→ ifElse("Using Prompt Enhance") ─→ cleanGpuUsed ─→ render.prompt
|
||
ifElse.on_false = raw TaskOutput.USER_PROMPT (bypass switch)
|
||
```
|
||
|
||
The example's LLM is `llama_cpp_instruct_adv` + `llama_cpp_model_loader` (a local GGUF).
|
||
That node is replaceable — it is just "system+user (+images) in, one string out".
|
||
Three ways to fill the slot:
|
||
|
||
### Option A — no LLM node at all (default for this skill)
|
||
|
||
The skill already wrote every scene prompt (identity text, continuity rules, LoRA
|
||
trigger) into `prompt_override.json`. Leave the enhance toggle OFF: raw
|
||
`USER_PROMPT` flows to the render node each pass. The "LLM that made all the scenes"
|
||
is this skill, outside ComfyUI.
|
||
|
||
### Option B — the pack's built-in enhancer with any OpenAI-compatible server
|
||
|
||
Add `easy multiTrackPromptEnhancer` inside the loop:
|
||
|
||
- `system_prompt` ← TaskOutput.SYSTEM_PROMPT, `user_prompt` ← TaskOutput.USER_PROMPT,
|
||
`type` ← TaskOutput.TYPE, `length` ← TaskOutput.LENGTH, images/audio/video optional.
|
||
- `model` = `openai_compatible (local)`; set `base_url` to ANY OpenAI-compatible
|
||
Chat Completions server — Ollama `http://127.0.0.1:11434/v1`, LM Studio
|
||
`http://127.0.0.1:1234/v1`, vLLM, llama.cpp server, or a hosted endpoint
|
||
(empty base_url reads `OPENAI_LOCAL_BASE_URL` from the pack's config.yaml, else
|
||
`http://127.0.0.1:8080/v1`). `apikey`/`model_name` optional depending on server.
|
||
Built-in presets also include MiniMax H3-Context-IR (`v2/h3_context_ir`),
|
||
bigmodel.cn (GLM) and RunningHub LLM keys from config.yaml.
|
||
- Output `PROMPT` → render node's prompt.
|
||
|
||
### Option C — any ComfyUI LLM node in the loop
|
||
|
||
Keep the example's wiring shape and swap node 86 for the preferred LLM node
|
||
(system_prompt + user_prompt inputs, STRING output). `prompt_format` on TaskOutput
|
||
can stay `api`; setting it to `llm` instead makes USER_PROMPT a single ready-made
|
||
chat transcript for LLM nodes that have only one prompt input.
|
||
|
||
## Local project path (alternative, no loop nodes)
|
||
|
||
`easy multitrackProject` (editor tracks_info in, `segment_count = -1`) builds and runs
|
||
every segment itself in one queue — see `MiniMaxH3_v1.3.1_Easy_Project_AllInOne.json`.
|
||
It contains no LLM node; prompts come from the timeline JSON. This is the pack's own
|
||
agent-driven setup: an external LLM (agent) writes the whole `track_data`, then one
|
||
run renders it. The pack even ships `skills/easy-media-multitrack-workflow` +
|
||
`patch_workflow.py` for exactly that flow (our SKILL.md step 5B).
|