track-gate: workflow wiring reference (single-run loop + LLM slot options)
This commit is contained in:
parent
eb090f765d
commit
8a9c79f06e
@ -329,6 +329,14 @@ Deliver it with these instructions:
|
|||||||
- **Manual fallback (no rewiring):** set `task_index` to 0, queue; then 1, queue; then
|
- **Manual fallback (no rewiring):** set `task_index` to 0, queue; then 1, queue; then
|
||||||
2, queue. Three clip files, join them afterwards. (`task_index = -1` outputs the whole
|
2, queue. Three clip files, join them afterwards. (`task_index = -1` outputs the whole
|
||||||
timeline's pre-rendered MEDIA — it is not a 3-clip generation.)
|
timeline's pre-rendered MEDIA — it is not a 3-clip generation.)
|
||||||
|
5. **Optional LLM polish slot** (only if the user asks for it): inside the loop,
|
||||||
|
`TaskOutput.SYSTEM_PROMPT/USER_PROMPT` may feed an LLM node whose output replaces the
|
||||||
|
prompt at the render node, with an ifElse bypass back to raw `USER_PROMPT`. The skill's
|
||||||
|
prompts are already final — leave the bypass OFF unless requested. Ways to fill the
|
||||||
|
slot: any ComfyUI LLM node (system+user in, string out), or the pack's
|
||||||
|
`easy multiTrackPromptEnhancer` with model `openai_compatible (local)` pointed at the
|
||||||
|
user's own endpoint (`base_url` accepts any OpenAI-compatible server: Ollama, LM
|
||||||
|
Studio, vLLM, llama.cpp server). Full wiring: `references/workflow_wiring.md`.
|
||||||
|
|
||||||
**The paste target is the `prompt_override` input and only that payload type.** Never give
|
**The paste target is the `prompt_override` input and only that payload type.** Never give
|
||||||
the user a `plan.json`, an outline, an H3 plan or any other JSON to paste there: without
|
the user a `plan.json`, an outline, an H3 plan or any other JSON to paste there: without
|
||||||
|
|||||||
100
easy-media-track-gate/references/workflow_wiring.md
Normal file
100
easy-media-track-gate/references/workflow_wiring.md
Normal file
@ -0,0 +1,100 @@
|
|||||||
|
# Workflow wiring: render ALL scenes in one run + the LLM slot
|
||||||
|
|
||||||
|
Verified against ComfyUI-Easy-Media source (`nodes/basic.py`, `nodes/minimax.py`,
|
||||||
|
`nodes/project.py`, `utils/llm_api.py`) and `example_workflows/MinimaxH3_ForLoop_MultiShot.json`.
|
||||||
|
|
||||||
|
## Why only scene 1 renders
|
||||||
|
|
||||||
|
The render chain reads the timeline through `easy multiTrackTaskOutput`, which outputs
|
||||||
|
**one task segment per pass**, picked by its `task_index` int input. Default `0` →
|
||||||
|
the first clip only. The number of prompts inside the pasted `prompt_override` JSON
|
||||||
|
does not change this: the JSON is complete (all scenes are in it); the loop lives in
|
||||||
|
the workflow, not in the JSON. Nothing needs to be added to the payload for multi-shot.
|
||||||
|
|
||||||
|
Side facts (same node, verified in source):
|
||||||
|
|
||||||
|
- `task_index = -1` outputs the **whole timeline as one merged segment** (all task
|
||||||
|
contents joined, full frame length). It is for relay/audio-lock cases, NOT a way to
|
||||||
|
render 3 clips: the local H3 model is trained at ~124–362 frames (~5–15 s); a 30 s
|
||||||
|
merged pass is out of range.
|
||||||
|
- `easy multiTrackInfoOutput` (tracks_info in) exposes `TASK_COUNT` = number of task
|
||||||
|
segments — 3 for the Sam payload. Also WIDTH/HEIGHT/TOTAL_FRAMES/FPS.
|
||||||
|
|
||||||
|
## The single-run loop (from the pack's own ForLoop example)
|
||||||
|
|
||||||
|
One queue run renders every clip and concatenates them:
|
||||||
|
|
||||||
|
```
|
||||||
|
editor.TRACKS_INFO ─→ multiTrackInfoOutput ─→ TASK_COUNT ─→ forLoopStart.total
|
||||||
|
editor.TRACKS_INFO ─→ multiTrackTaskOutput (images/audio/video too)
|
||||||
|
forLoopStart.index ─→ multiTrackTaskOutput.task_index
|
||||||
|
multiTrackTaskOutput.SYSTEM_PROMPT/USER_PROMPT/TYPE/LENGTH/IMAGES/AUDIO ─→ render chain
|
||||||
|
render chain → saveVideo.file_path ─→ batchAnything ─→ forLoopEnd.initial_value1
|
||||||
|
forLoopEnd.value1 ─→ mergeVideosFromPaths ─→ SaveVideo
|
||||||
|
```
|
||||||
|
|
||||||
|
The example adds two math nodes so a single PrimitiveInt `k` (default `-1`) chooses:
|
||||||
|
|
||||||
|
- `total = (k<=-1)*TASK_COUNT + (k>-1)*1`
|
||||||
|
- `task_index = (k<=-1)*loop_index + (k>-1)*k`
|
||||||
|
|
||||||
|
`k = -1` → loop all segments (all scenes, one run). `k = 2` → render only segment 2.
|
||||||
|
Manual fallback without rewiring: queue once per index (0, 1, 2) and join the files.
|
||||||
|
|
||||||
|
Per-iteration the task output gives LENGTH ≈ frames for that clip only (240 frames =
|
||||||
|
10 s — inside the model's trained range), so each loop pass is a normal single-clip
|
||||||
|
generation.
|
||||||
|
|
||||||
|
## The LLM slot (where your own LLM connects)
|
||||||
|
|
||||||
|
In the ForLoop example the per-scene prompt may pass through an LLM before rendering:
|
||||||
|
|
||||||
|
```
|
||||||
|
TaskOutput.SYSTEM_PROMPT ─→ [LLM node].system_prompt
|
||||||
|
TaskOutput.USER_PROMPT ─→ [LLM node].custom_prompt
|
||||||
|
TaskOutput.IMAGES ─→ [LLM node].images (optional, vision only)
|
||||||
|
[LLM node].output ─→ ifElse("Using Prompt Enhance") ─→ cleanGpuUsed ─→ render.prompt
|
||||||
|
ifElse.on_false = raw TaskOutput.USER_PROMPT (bypass switch)
|
||||||
|
```
|
||||||
|
|
||||||
|
The example's LLM is `llama_cpp_instruct_adv` + `llama_cpp_model_loader` (a local GGUF).
|
||||||
|
That node is replaceable — it is just "system+user (+images) in, one string out".
|
||||||
|
Three ways to fill the slot:
|
||||||
|
|
||||||
|
### Option A — no LLM node at all (default for this skill)
|
||||||
|
|
||||||
|
The skill already wrote every scene prompt (identity text, continuity rules, LoRA
|
||||||
|
trigger) into `prompt_override.json`. Leave the enhance toggle OFF: raw
|
||||||
|
`USER_PROMPT` flows to the render node each pass. The "LLM that made all the scenes"
|
||||||
|
is this skill, outside ComfyUI.
|
||||||
|
|
||||||
|
### Option B — the pack's built-in enhancer with any OpenAI-compatible server
|
||||||
|
|
||||||
|
Add `easy multiTrackPromptEnhancer` inside the loop:
|
||||||
|
|
||||||
|
- `system_prompt` ← TaskOutput.SYSTEM_PROMPT, `user_prompt` ← TaskOutput.USER_PROMPT,
|
||||||
|
`type` ← TaskOutput.TYPE, `length` ← TaskOutput.LENGTH, images/audio/video optional.
|
||||||
|
- `model` = `openai_compatible (local)`; set `base_url` to ANY OpenAI-compatible
|
||||||
|
Chat Completions server — Ollama `http://127.0.0.1:11434/v1`, LM Studio
|
||||||
|
`http://127.0.0.1:1234/v1`, vLLM, llama.cpp server, or a hosted endpoint
|
||||||
|
(empty base_url reads `OPENAI_LOCAL_BASE_URL` from the pack's config.yaml, else
|
||||||
|
`http://127.0.0.1:8080/v1`). `apikey`/`model_name` optional depending on server.
|
||||||
|
Built-in presets also include MiniMax H3-Context-IR (`v2/h3_context_ir`),
|
||||||
|
bigmodel.cn (GLM) and RunningHub LLM keys from config.yaml.
|
||||||
|
- Output `PROMPT` → render node's prompt.
|
||||||
|
|
||||||
|
### Option C — any ComfyUI LLM node in the loop
|
||||||
|
|
||||||
|
Keep the example's wiring shape and swap node 86 for the preferred LLM node
|
||||||
|
(system_prompt + user_prompt inputs, STRING output). `prompt_format` on TaskOutput
|
||||||
|
can stay `api`; setting it to `llm` instead makes USER_PROMPT a single ready-made
|
||||||
|
chat transcript for LLM nodes that have only one prompt input.
|
||||||
|
|
||||||
|
## Local project path (alternative, no loop nodes)
|
||||||
|
|
||||||
|
`easy multitrackProject` (editor tracks_info in, `segment_count = -1`) builds and runs
|
||||||
|
every segment itself in one queue — see `MiniMaxH3_v1.3.1_Easy_Project_AllInOne.json`.
|
||||||
|
It contains no LLM node; prompts come from the timeline JSON. This is the pack's own
|
||||||
|
agent-driven setup: an external LLM (agent) writes the whole `track_data`, then one
|
||||||
|
run renders it. The pack even ships `skills/easy-media-multitrack-workflow` +
|
||||||
|
`patch_workflow.py` for exactly that flow (our SKILL.md step 5B).
|
||||||
Loading…
Reference in New Issue
Block a user