From 8a9c79f06ea7c331248214bd2f579b09f8c0eb6d Mon Sep 17 00:00:00 2001 From: Z User Date: Sun, 1 Nov 2026 03:30:10 +0000 Subject: [PATCH] track-gate: workflow wiring reference (single-run loop + LLM slot options) --- easy-media-track-gate/SKILL.md | 8 ++ .../references/workflow_wiring.md | 100 ++++++++++++++++++ 2 files changed, 108 insertions(+) create mode 100644 easy-media-track-gate/references/workflow_wiring.md diff --git a/easy-media-track-gate/SKILL.md b/easy-media-track-gate/SKILL.md index 855259b..de6ea73 100755 --- a/easy-media-track-gate/SKILL.md +++ b/easy-media-track-gate/SKILL.md @@ -329,6 +329,14 @@ Deliver it with these instructions: - **Manual fallback (no rewiring):** set `task_index` to 0, queue; then 1, queue; then 2, queue. Three clip files, join them afterwards. (`task_index = -1` outputs the whole timeline's pre-rendered MEDIA — it is not a 3-clip generation.) +5. **Optional LLM polish slot** (only if the user asks for it): inside the loop, + `TaskOutput.SYSTEM_PROMPT/USER_PROMPT` may feed an LLM node whose output replaces the + prompt at the render node, with an ifElse bypass back to raw `USER_PROMPT`. The skill's + prompts are already final — leave the bypass OFF unless requested. Ways to fill the + slot: any ComfyUI LLM node (system+user in, string out), or the pack's + `easy multiTrackPromptEnhancer` with model `openai_compatible (local)` pointed at the + user's own endpoint (`base_url` accepts any OpenAI-compatible server: Ollama, LM + Studio, vLLM, llama.cpp server). Full wiring: `references/workflow_wiring.md`. **The paste target is the `prompt_override` input and only that payload type.** Never give the user a `plan.json`, an outline, an H3 plan or any other JSON to paste there: without diff --git a/easy-media-track-gate/references/workflow_wiring.md b/easy-media-track-gate/references/workflow_wiring.md new file mode 100644 index 0000000..fceea4c --- /dev/null +++ b/easy-media-track-gate/references/workflow_wiring.md @@ -0,0 +1,100 @@ +# Workflow wiring: render ALL scenes in one run + the LLM slot + +Verified against ComfyUI-Easy-Media source (`nodes/basic.py`, `nodes/minimax.py`, +`nodes/project.py`, `utils/llm_api.py`) and `example_workflows/MinimaxH3_ForLoop_MultiShot.json`. + +## Why only scene 1 renders + +The render chain reads the timeline through `easy multiTrackTaskOutput`, which outputs +**one task segment per pass**, picked by its `task_index` int input. Default `0` → +the first clip only. The number of prompts inside the pasted `prompt_override` JSON +does not change this: the JSON is complete (all scenes are in it); the loop lives in +the workflow, not in the JSON. Nothing needs to be added to the payload for multi-shot. + +Side facts (same node, verified in source): + +- `task_index = -1` outputs the **whole timeline as one merged segment** (all task + contents joined, full frame length). It is for relay/audio-lock cases, NOT a way to + render 3 clips: the local H3 model is trained at ~124–362 frames (~5–15 s); a 30 s + merged pass is out of range. +- `easy multiTrackInfoOutput` (tracks_info in) exposes `TASK_COUNT` = number of task + segments — 3 for the Sam payload. Also WIDTH/HEIGHT/TOTAL_FRAMES/FPS. + +## The single-run loop (from the pack's own ForLoop example) + +One queue run renders every clip and concatenates them: + +``` +editor.TRACKS_INFO ─→ multiTrackInfoOutput ─→ TASK_COUNT ─→ forLoopStart.total +editor.TRACKS_INFO ─→ multiTrackTaskOutput (images/audio/video too) +forLoopStart.index ─→ multiTrackTaskOutput.task_index +multiTrackTaskOutput.SYSTEM_PROMPT/USER_PROMPT/TYPE/LENGTH/IMAGES/AUDIO ─→ render chain +render chain → saveVideo.file_path ─→ batchAnything ─→ forLoopEnd.initial_value1 +forLoopEnd.value1 ─→ mergeVideosFromPaths ─→ SaveVideo +``` + +The example adds two math nodes so a single PrimitiveInt `k` (default `-1`) chooses: + +- `total = (k<=-1)*TASK_COUNT + (k>-1)*1` +- `task_index = (k<=-1)*loop_index + (k>-1)*k` + +`k = -1` → loop all segments (all scenes, one run). `k = 2` → render only segment 2. +Manual fallback without rewiring: queue once per index (0, 1, 2) and join the files. + +Per-iteration the task output gives LENGTH ≈ frames for that clip only (240 frames = +10 s — inside the model's trained range), so each loop pass is a normal single-clip +generation. + +## The LLM slot (where your own LLM connects) + +In the ForLoop example the per-scene prompt may pass through an LLM before rendering: + +``` +TaskOutput.SYSTEM_PROMPT ─→ [LLM node].system_prompt +TaskOutput.USER_PROMPT ─→ [LLM node].custom_prompt +TaskOutput.IMAGES ─→ [LLM node].images (optional, vision only) +[LLM node].output ─→ ifElse("Using Prompt Enhance") ─→ cleanGpuUsed ─→ render.prompt +ifElse.on_false = raw TaskOutput.USER_PROMPT (bypass switch) +``` + +The example's LLM is `llama_cpp_instruct_adv` + `llama_cpp_model_loader` (a local GGUF). +That node is replaceable — it is just "system+user (+images) in, one string out". +Three ways to fill the slot: + +### Option A — no LLM node at all (default for this skill) + +The skill already wrote every scene prompt (identity text, continuity rules, LoRA +trigger) into `prompt_override.json`. Leave the enhance toggle OFF: raw +`USER_PROMPT` flows to the render node each pass. The "LLM that made all the scenes" +is this skill, outside ComfyUI. + +### Option B — the pack's built-in enhancer with any OpenAI-compatible server + +Add `easy multiTrackPromptEnhancer` inside the loop: + +- `system_prompt` ← TaskOutput.SYSTEM_PROMPT, `user_prompt` ← TaskOutput.USER_PROMPT, + `type` ← TaskOutput.TYPE, `length` ← TaskOutput.LENGTH, images/audio/video optional. +- `model` = `openai_compatible (local)`; set `base_url` to ANY OpenAI-compatible + Chat Completions server — Ollama `http://127.0.0.1:11434/v1`, LM Studio + `http://127.0.0.1:1234/v1`, vLLM, llama.cpp server, or a hosted endpoint + (empty base_url reads `OPENAI_LOCAL_BASE_URL` from the pack's config.yaml, else + `http://127.0.0.1:8080/v1`). `apikey`/`model_name` optional depending on server. + Built-in presets also include MiniMax H3-Context-IR (`v2/h3_context_ir`), + bigmodel.cn (GLM) and RunningHub LLM keys from config.yaml. +- Output `PROMPT` → render node's prompt. + +### Option C — any ComfyUI LLM node in the loop + +Keep the example's wiring shape and swap node 86 for the preferred LLM node +(system_prompt + user_prompt inputs, STRING output). `prompt_format` on TaskOutput +can stay `api`; setting it to `llm` instead makes USER_PROMPT a single ready-made +chat transcript for LLM nodes that have only one prompt input. + +## Local project path (alternative, no loop nodes) + +`easy multitrackProject` (editor tracks_info in, `segment_count = -1`) builds and runs +every segment itself in one queue — see `MiniMaxH3_v1.3.1_Easy_Project_AllInOne.json`. +It contains no LLM node; prompts come from the timeline JSON. This is the pack's own +agent-driven setup: an external LLM (agent) writes the whole `track_data`, then one +run renders it. The pack even ships `skills/easy-media-multitrack-workflow` + +`patch_workflow.py` for exactly that flow (our SKILL.md step 5B).