mmprompt/easy-media-track-gate/references/workflow_wiring.md

6.5 KiB
Raw Blame History

Workflow wiring: render ALL scenes in one run + the LLM slot

Verified against ComfyUI-Easy-Media source (nodes/basic.py, nodes/minimax.py, nodes/project.py, utils/llm_api.py) and example_workflows/MinimaxH3_ForLoop_MultiShot.json.

Why only scene 1 renders

The render chain reads the timeline through easy multiTrackTaskOutput, which outputs one task segment per pass, picked by its task_index int input. Default 0 → the first clip only. The number of prompts inside the pasted prompt_override JSON does not change this: the JSON is complete (all scenes are in it); the loop lives in the workflow, not in the JSON. Nothing needs to be added to the payload for multi-shot.

Side facts (same node, verified in source):

  • task_index = -1 outputs the whole timeline as one merged segment (all task contents joined, full frame length). It is for relay/audio-lock cases, NOT a way to render 3 clips: the local H3 model is trained at ~124–362 frames (~5–15 s); a 30 s merged pass is out of range.
  • easy multiTrackInfoOutput (tracks_info in) exposes TASK_COUNT = number of task segments — 3 for the Sam payload. Also WIDTH/HEIGHT/TOTAL_FRAMES/FPS.

The single-run loop (from the pack's own ForLoop example)

One queue run renders every clip and concatenates them:

editor.TRACKS_INFO ─→ multiTrackInfoOutput ─→ TASK_COUNT ─→ forLoopStart.total
editor.TRACKS_INFO ─→ multiTrackTaskOutput      (images/audio/video too)
forLoopStart.index ─→ multiTrackTaskOutput.task_index
multiTrackTaskOutput.SYSTEM_PROMPT/USER_PROMPT/TYPE/LENGTH/IMAGES/AUDIO ─→ render chain
render chain → saveVideo.file_path ─→ batchAnything ─→ forLoopEnd.initial_value1
forLoopEnd.value1 ─→ mergeVideosFromPaths ─→ SaveVideo

The example adds two math nodes so a single PrimitiveInt k (default -1) chooses:

  • total = (k<=-1)*TASK_COUNT + (k>-1)*1
  • task_index = (k<=-1)*loop_index + (k>-1)*k

k = -1 → loop all segments (all scenes, one run). k = 2 → render only segment 2. Manual fallback without rewiring: queue once per index (0, 1, 2) and join the files.

Per-iteration the task output gives LENGTH ≈ frames for that clip only (240 frames = 10 s — inside the model's trained range), so each loop pass is a normal single-clip generation.

The LLM slot (where your own LLM connects)

In the ForLoop example the per-scene prompt may pass through an LLM before rendering:

TaskOutput.SYSTEM_PROMPT ─→ [LLM node].system_prompt
TaskOutput.USER_PROMPT   ─→ [LLM node].custom_prompt
TaskOutput.IMAGES        ─→ [LLM node].images        (optional, vision only)
[LLM node].output ─→ ifElse("Using Prompt Enhance") ─→ cleanGpuUsed ─→ render.prompt
ifElse.on_false = raw TaskOutput.USER_PROMPT (bypass switch)

The example's LLM is llama_cpp_instruct_adv + llama_cpp_model_loader (a local GGUF). That node is replaceable — it is just "system+user (+images) in, one string out". Three ways to fill the slot:

Option A — no LLM node at all (default for this skill)

The skill already wrote every scene prompt (identity text, continuity rules, LoRA trigger) into prompt_override.json. Leave the enhance toggle OFF: raw USER_PROMPT flows to the render node each pass. The "LLM that made all the scenes" is this skill, outside ComfyUI.

Option B — the pack's built-in enhancer with any OpenAI-compatible server

Add easy multiTrackPromptEnhancer inside the loop:

  • system_prompt ← TaskOutput.SYSTEM_PROMPT, user_prompt ← TaskOutput.USER_PROMPT, type ← TaskOutput.TYPE, length ← TaskOutput.LENGTH, images/audio/video optional.
  • model = openai_compatible (local); set base_url to ANY OpenAI-compatible Chat Completions server — Ollama http://127.0.0.1:11434/v1, LM Studio http://127.0.0.1:1234/v1, vLLM, llama.cpp server, or a hosted endpoint (empty base_url reads OPENAI_LOCAL_BASE_URL from the pack's config.yaml, else http://127.0.0.1:8080/v1). apikey/model_name optional depending on server. Built-in presets also include MiniMax H3-Context-IR (v2/h3_context_ir), bigmodel.cn (GLM) and RunningHub LLM keys from config.yaml.
  • Output PROMPT → render node's prompt.

Option C — any ComfyUI LLM node in the loop

Keep the example's wiring shape and swap node 86 for the preferred LLM node (system_prompt + user_prompt inputs, STRING output). prompt_format on TaskOutput can stay api; setting it to llm instead makes USER_PROMPT a single ready-made chat transcript for LLM nodes that have only one prompt input.

Local project path (alternative, no loop nodes)

easy multitrackProject (editor tracks_info in, segment_count = -1) builds and runs every segment itself in one queue — see MiniMaxH3_v1.3.1_Easy_Project_AllInOne.json. It contains no LLM node; prompts come from the timeline JSON. This is the pack's own agent-driven setup: an external LLM (agent) writes the whole track_data, then one run renders it. The pack even ships skills/easy-media-multitrack-workflow + patch_workflow.py for exactly that flow (our SKILL.md step 5B).

The built-in LLM system prompts (published, not locked)

The rewrite-LLM contract ships in the open in the pack: utils/prompt_builder.py — MINIMAX_BASE_PROMPT (~15.7 KB, "Video Prompt Writing Guide (T2VA / I2VA / FL2VA / L2VA)") and MINIMAX_REF_PROMPT (~23.5 KB, "Full- Reference Mode Rewrite Output Format Guide" — six-section rewrite with <Subject N> / <Picture N> / <Video N> / <Audio N> labels).

Selection (build_prompt_request, MiniMax branch):

  • task_mode default / l2v (our t2v payloads) → MINIMAX_BASE_PROMPT
  • task_mode ref / edit → MINIMAX_REF_PROMPT
  • a non-empty per-segment system_prompt overrides both; with an empty one the context-segment continuity rules ride the MiniMax rewrite API server-side (entry rule: not-first segment using context; tail rule: next segment uses context) — per the pack's own multitrack-schema.md.

Delivery: the system prompt only leaves TaskOutput via prompt_format api or llm (default / promptRelay output an empty SYSTEM_PROMPT). build_llm_prompt folds system+user into one transcript for single-input LLM nodes. To use the user's own LLM with the same behavior: paste minimax_base_prompt.txt (or the ref guide for ref-mode work) into the LLM node's system prompt field and wire TaskOutput SYSTEM_PROMPT/USER_PROMPT into it — or leave the LLM's system prompt empty and pass TaskOutput.SYSTEM_PROMPT through, which already carries the selected guide verbatim.