mmprompt/easy-media-track-gate/references/workflow_wiring.md

9.0 KiB
Raw Blame History

Workflow wiring: render ALL scenes in one run + the LLM slot

Verified against ComfyUI-Easy-Media source (nodes/basic.py, nodes/minimax.py, nodes/project.py, utils/llm_api.py) and example_workflows/MinimaxH3_ForLoop_MultiShot.json.

Why only scene 1 renders

The render chain reads the timeline through easy multiTrackTaskOutput, which outputs one task segment per pass, picked by its task_index int input. Default 0 → the first clip only. The number of prompts inside the pasted prompt_override JSON does not change this: the JSON is complete (all scenes are in it); the loop lives in the workflow, not in the JSON. Nothing needs to be added to the payload for multi-shot.

Side facts (same node, verified in source):

  • task_index = -1 switches the node to full-timeline mode: window 0..720, LENGTH = all frames snapped to the H3 17k+5 grid (720 → 719), media from all segments. What the prompt becomes depends on prompt_format — see the single-pass section below (promptRelay makes it render all scenes in one pass).
  • easy multiTrackInfoOutput (tracks_info in) exposes TASK_COUNT = number of task segments — 3 for the Sam payload. Also WIDTH/HEIGHT/TOTAL_FRAMES/FPS.

The prompt_override input requires the type field (proven)

MultiTrackEditor.execute (nodes/basic.py L1949-1957): with format = "MiniMax" it calls is_minimax_prompt_override(prompt_override) FIRST. Only a payload with "type": "minimax_prompt_override" and a prompts list passes (utils/prompt_override.py parse_minimax_prompt_override). Any other non-empty value — including a full TRACKS_INFO JSON — falls to the |-text parser, which treats the ENTIRE string as one prompt: one segment 0..479 whose text is the literal JSON. Proven by running the repo's parser on the captured workflow (scripts/prove_relay_chain.py STEP A). So: paste the compact payload (with the type field) into prompt_override; full TRACKS_INFO belongs only in track_data.

Single-pass timeline render: task_index=-1 + prompt_format=promptRelay

This is the pattern the user's own graph (my timeline ref.json) is wired for — no ForLoop, one sampler pass, all scenes in one generation:

  1. Node 186 (String → prompt_override): the compact payload WITH type field.
  2. easy multiTrackTaskOutput: task_index = -1 AND prompt_format = "promptRelay".
  3. Everything else stays as-is: TYPE output stays t2v (first task's explicit task_type → matchLine → multi_frames), LENGTH output becomes 719, one SamplerCustomAdvanced pass renders the whole timeline.

With promptRelay + full-timeline mode, TaskOutput builds the prompt from ALL task segments (_format_marker_task_prompt_relay), each scene suffixed with its native H3 frame range, joined by |:

<scene 1 text> [0-240] | <scene 2 text> [240-480] | <scene 3 text> [480-720]

SYSTEM_PROMPT output is empty in this mode; the ranges are the H3 timeline syntax (_FRAME_RANGE_RE parses the same [s-e] / [s-e,s-type] tokens in the override parser). Proven by executing the repo's own functions on the Sam payload (STEP C: 3 ranges, length 719, type t2v).

Both settings are required — task_index = -1 with prompt_format = "default" keeps only the FIRST task's prompt (STEP D): a 719-frame latent conditioned on scene 1 alone.

Caveat from the node's own tooltip: the trained range is ~124–362 frames and "longer is untested". 719 frames (~30 s) is the intended use of this graph shape (multi_frames mode + BlockSparseAttention min_tokens 12288 exist for long sequences), but if quality degrades, fall back to the ForLoop pattern above, which renders each 240-frame clip inside the trained range.

The single-run loop (from the pack's own ForLoop example)

One queue run renders every clip and concatenates them:

editor.TRACKS_INFO ─→ multiTrackInfoOutput ─→ TASK_COUNT ─→ forLoopStart.total
editor.TRACKS_INFO ─→ multiTrackTaskOutput      (images/audio/video too)
forLoopStart.index ─→ multiTrackTaskOutput.task_index
multiTrackTaskOutput.SYSTEM_PROMPT/USER_PROMPT/TYPE/LENGTH/IMAGES/AUDIO ─→ render chain
render chain → saveVideo.file_path ─→ batchAnything ─→ forLoopEnd.initial_value1
forLoopEnd.value1 ─→ mergeVideosFromPaths ─→ SaveVideo

The example adds two math nodes so a single PrimitiveInt k (default -1) chooses:

  • total = (k<=-1)*TASK_COUNT + (k>-1)*1
  • task_index = (k<=-1)*loop_index + (k>-1)*k

k = -1 → loop all segments (all scenes, one run). k = 2 → render only segment 2. Manual fallback without rewiring: queue once per index (0, 1, 2) and join the files.

Per-iteration the task output gives LENGTH ≈ frames for that clip only (240 frames = 10 s — inside the model's trained range), so each loop pass is a normal single-clip generation.

The LLM slot (where your own LLM connects)

In the ForLoop example the per-scene prompt may pass through an LLM before rendering:

TaskOutput.SYSTEM_PROMPT ─→ [LLM node].system_prompt
TaskOutput.USER_PROMPT   ─→ [LLM node].custom_prompt
TaskOutput.IMAGES        ─→ [LLM node].images        (optional, vision only)
[LLM node].output ─→ ifElse("Using Prompt Enhance") ─→ cleanGpuUsed ─→ render.prompt
ifElse.on_false = raw TaskOutput.USER_PROMPT (bypass switch)

The example's LLM is llama_cpp_instruct_adv + llama_cpp_model_loader (a local GGUF). That node is replaceable — it is just "system+user (+images) in, one string out". Three ways to fill the slot:

Option A — no LLM node at all (default for this skill)

The skill already wrote every scene prompt (identity text, continuity rules, LoRA trigger) into prompt_override.json. Leave the enhance toggle OFF: raw USER_PROMPT flows to the render node each pass. The "LLM that made all the scenes" is this skill, outside ComfyUI.

Option B — the pack's built-in enhancer with any OpenAI-compatible server

Add easy multiTrackPromptEnhancer inside the loop:

  • system_prompt ← TaskOutput.SYSTEM_PROMPT, user_prompt ← TaskOutput.USER_PROMPT, type ← TaskOutput.TYPE, length ← TaskOutput.LENGTH, images/audio/video optional.
  • model = openai_compatible (local); set base_url to ANY OpenAI-compatible Chat Completions server — Ollama http://127.0.0.1:11434/v1, LM Studio http://127.0.0.1:1234/v1, vLLM, llama.cpp server, or a hosted endpoint (empty base_url reads OPENAI_LOCAL_BASE_URL from the pack's config.yaml, else http://127.0.0.1:8080/v1). apikey/model_name optional depending on server. Built-in presets also include MiniMax H3-Context-IR (v2/h3_context_ir), bigmodel.cn (GLM) and RunningHub LLM keys from config.yaml.
  • Output PROMPT → render node's prompt.

Option C — any ComfyUI LLM node in the loop

Keep the example's wiring shape and swap node 86 for the preferred LLM node (system_prompt + user_prompt inputs, STRING output). prompt_format on TaskOutput can stay api; setting it to llm instead makes USER_PROMPT a single ready-made chat transcript for LLM nodes that have only one prompt input.

Local project path (alternative, no loop nodes)

easy multitrackProject (editor tracks_info in, segment_count = -1) builds and runs every segment itself in one queue — see MiniMaxH3_v1.3.1_Easy_Project_AllInOne.json. It contains no LLM node; prompts come from the timeline JSON. This is the pack's own agent-driven setup: an external LLM (agent) writes the whole track_data, then one run renders it. The pack even ships skills/easy-media-multitrack-workflow + patch_workflow.py for exactly that flow (our SKILL.md step 5B).

The built-in LLM system prompts (published, not locked)

The rewrite-LLM contract ships in the open in the pack: utils/prompt_builder.py — MINIMAX_BASE_PROMPT (~15.7 KB, "Video Prompt Writing Guide (T2VA / I2VA / FL2VA / L2VA)") and MINIMAX_REF_PROMPT (~23.5 KB, "Full- Reference Mode Rewrite Output Format Guide" — six-section rewrite with <Subject N> / <Picture N> / <Video N> / <Audio N> labels).

Selection (build_prompt_request, MiniMax branch):

  • task_mode default / l2v (our t2v payloads) → MINIMAX_BASE_PROMPT
  • task_mode ref / edit → MINIMAX_REF_PROMPT
  • a non-empty per-segment system_prompt overrides both; with an empty one the context-segment continuity rules ride the MiniMax rewrite API server-side (entry rule: not-first segment using context; tail rule: next segment uses context) — per the pack's own multitrack-schema.md.

Delivery: the system prompt only leaves TaskOutput via prompt_format api or llm (default / promptRelay output an empty SYSTEM_PROMPT). build_llm_prompt folds system+user into one transcript for single-input LLM nodes. To use the user's own LLM with the same behavior: paste minimax_base_prompt.txt (or the ref guide for ref-mode work) into the LLM node's system prompt field and wire TaskOutput SYSTEM_PROMPT/USER_PROMPT into it — or leave the LLM's system prompt empty and pass TaskOutput.SYSTEM_PROMPT through, which already carries the selected guide verbatim.