# MultiTrack Data and Orchestration Rules (English reference) English reference for the `easy multiTrackEditor` TRACK_DATA and the patch plan, translated from the node pack's own skill doc (`skills/easy-media-multitrack-workflow/references/multitrack-schema.md` in ComfyUI-Easy-Media, which is written in Chinese). Front-end types live in `frontend/src/types/multitrack.ts`; execution logic lives in `nodes/basic.py` of the node pack. Field names are preserved exactly as the schema spells them. ## Resolution (DynamicCombo) `resolution` is a DynamicCombo widget of `easy multiTrackEditor`. It is NOT part of TRACK_DATA. Changing it requires rebuilding the sub-widgets for the chosen mode and syncing the workflow's existing `widgets_values_named`. Write it through `patch_workflow.py`'s `editor.resolution` — never by guessing array positions. | Need | `resolution` label | Required sub-fields | | --- | --- | --- | | Fixed preset output size | e.g. `1344 x 768 (16:9)` | optional `resize_method`, default `stretch` | | Explicit custom size | `width x height (custom)` | `width`, `height`, optional `resize_method` | | Follow first video's ratio | `width x height (auto)` | optional `resize_method` | | Fixed short/long edge | `width x height (shortest)` / `width x height (longest)` | `resize_to_pixel`, optional `resize_method` | | Ratio + total pixel budget | `width x height (megapixels)` | `aspect_ratio`, `megapixels` | Selection order: exact width/height from the user → an existing fixed preset, else custom; only orientation or aspect + quality/pixel budget given → megapixels; keep source video ratio → auto; "short/long edge = N" → shortest/longest. If the user expresses no resolution intent, keep the current setting. The model format's default size is only a reference and never overrides an explicit user requirement. ### MP and ratio in natural language Case-insensitive `mp`, `Mpx`, `megapixel(s)` (and Chinese "百万像素") mean the megapixels unit. `0.9mp`, `1 MP`, `0.8 megapixels` select megapixels mode and the number is taken verbatim. A ratio in the same sentence maps to the full DynamicCombo `aspect_ratio` value: | User ratio | `aspect_ratio` | | --- | --- | | `1:1` | `1:1 (Square)` | | `2:3` | `2:3 (Portrait Photo)` | | `3:2` | `3:2 (Photo)` | | `3:4` | `3:4 (Portrait Standard)` | | `4:3` | `4:3 (Standard)` | | `9:16` | `9:16 (Portrait Widescreen)` | | `16:9` | `16:9 (Widescreen)` | | `21:9` | `21:9 (Ultrawide)` | Example: "generate a 16:9 0.9mp video" normalizes with no follow-up question to: ```json { "resolution": "width x height (megapixels)", "aspect_ratio": "16:9 (Widescreen)", "megapixels": 0.9 } ``` Never convert the MP number into width/height and never round it. MP only: if already in megapixels mode keep its `aspect_ratio`; else pick `16:9`/`9:16` from explicit landscape/portrait wording; still unclear → ask. Contradictory exact size, ratio and MP → ask. Custom example: ```json { "resolution": "width x height (custom)", "width": 1024, "height": 576, "resize_method": "crop" } ``` Valid `resize_method`: `stretch`, `resize`, `pad`, `pad (white)`, `pad_edge`, `pad_edge_pixel`, `crop`, `pillarbox_blur`. Prefer pad/resize to keep the whole picture, crop to fill the canvas; never change an existing resize method the user did not mention. ## Time semantics - `start_frame` is inclusive, `end_frame` exclusive; segment frames = `end_frame - start_frame`. - Seconds → frames: `round(seconds * frame_rate)`; every segment has at least 1 frame. - Segments in one track are sorted by `start_frame` and never overlap; gaps are allowed. Different tracks of the same kind schedule independently. - With segments present, `total_length` should be the maximum `end_frame` over all tracks. With no segments the editor's current default length may stay. - Changing FPS must re-map every segment boundary, `origin_start_frame` and task marker proportionally — not just `frame_rate`. ## Top-level TRACK_DATA object ```json { "muted": false, "volume_db": 0, "task_markers": [], "task_overview": false, "tracks": [], "total_length": 240, "frame_rate": 24 } ``` Track `type` is one of `task`, `video`, `audio`, `subtitle`. Keep unknown fields for forward compatibility. ## Common track and segment shape ```json { "id": "uuid", "name": "Video 0", "type": "video", "color": "var(--primary)", "muted": false, "solo": false, "volume_db": 0, "locked": false, "segments": [ { "id": "uuid", "start_frame": 0, "end_frame": 120, "color": "var(--primary)", "content": {} } ] } ``` New tracks use the project's existing color tokens: task tracks `var(--multitrack-task-bg)`, video tracks `var(--primary)`. Keep an existing track's own color. ## Task segment content ```json { "media_type": "none", "task_mode": "ref", "continuity_mode": "shot", "ref_image_size": "match", "images": [], "user_prompt": "the user prompt", "system_prompt": "", "user_prompt_variant": "a", "user_prompt_b": "", "muted": false, "volume_db": 0 } ``` - `task_mode`: `default`, `l2v`, `ref`, `edit`. - RefMod workflows (identity driven by `MiniMaxH3RefModsLoader` + `MiniMaxH3RefModApply` outside the timeline): author segments as `task_mode: "ref"` even with no per-segment images — the payload then carries `generation_type: "r2v"`, TaskOutput TYPE becomes `r2v`, and the matchLine/promptLine pair switches `minimaxH3ToVideo` to `reference` mode, which pairs with the RefMod conditioning. `ref` rows with `images: []` are legal (refmod pictures ride the conditioning, not the timeline); prompts stay picture-token-free. Verified live by the user: the relay one-pass render works with r2v and refmods "work better with ref2va (r2v)". - `continuity_mode`: `shot` (standalone shot), `context` (normal context join) or `context_drift` (context drift control); the legacy value `context_swap` is still read. - `ref_image_size`: `match` or `max`. - `images` holds at most 9 items; their order defines the picture-reference order. - When editing an existing task keep its `system_prompt`, prompt variant, continuity and reference-image policy unless the user asked to change them. - If the current variant is `b`, edit `user_prompt_b`; otherwise edit `user_prompt`. Never switch the variant unintentionally. Task reference image item: ```json { "id": "uuid", "source_type": "input", "file_path": "codex/session-123/reference.png", "file_name": "reference.png" } ``` `source_type` is one of `input`, `output`, `local`, `url`, `slot`. The path field must match the source: `input`/`output` use `file_path`, `local` uses `local_path`, `url` uses `url`, `slot` uses `slot_name`. Never label a local absolute path as `input`. When several context segments reuse the same reference image, an image item may set `"shared_reference": true`; it becomes a common image that is automatically ordered before each task's private images; common + private images still total at most 9 per task. The current front-end `synchronizeSharedTaskImages` generates new UUIDs for missing common-image metadata on later tasks, so repeatedly normalizing the same sparse raw data yields different IDs; from this code alone the full React maximum-update-depth / widget-unmount trigger chain cannot be confirmed. Until the target front-end is verified to load such data stably, store an explicit private image item per task (`shared_reference: false`) with a fixed ID pointing at the same uploaded file — no duplication or re-upload needed. Switching continuity mode does not fix this. ## Context drift control mode `context_drift` is the "context drift control" join: it carries the previous segment's scene and motion context while letting the model replace subjects, people, objects or局部 regions from a reference video. When the user supplies a video as a subject/person/object/clothing/face replacement reference, orchestrate as follows: - Set `content.continuity_mode` to `context_drift` on ALL task segments in scope — not only the second and later ones. The first segment keeps the same mode too (it has no previous generated context, but consistency matters for project metadata and re-generation). - Put the reference video on a video track covering the target task segments that need motion reference. If the project also has passages that do not depend on the dance/reference video (e.g. a verse), keep one complete project — the full locked audio covers all tasks and the video only covers its part. If the video is also needed across tasks as a model reference, set `shared_reference: true` on the video segment's `content`. - Set track-level `audio_locked: true` on that video track so the source video's timeline drives generation, and explicitly set `audio_locked: false` on every OTHER video track. To replace the original sound, lock a separate audio track, which then wins as the audio source. When only reusing the video's own sound, confirm before locking that the video actually contains a usable audio track and covers all target task segments. - If the user pinned segment lengths or boundaries, the user wins. Otherwise split task segments into consecutive 10-second pieces: each piece is `round(10 * frame_rate)` frames and the last piece takes the reference video's remaining frames — do not pad, do not drop the remainder, do not create an empty tail segment. The reference video itself can stay a single video segment covering the whole range; it does not need to be duplicated per task segment. - If the user explicitly asked to drop/replace/not keep the reference video's original sound, do not auto-lock that video track and follow the user's audio plan instead. ## Context segment prompts Applies to tasks with `task_mode: "ref"` and `continuity_mode: "context"`; `context_drift` (and legacy `context_swap`) do not use this section. Such a segment starts at the previous segment's actually delivered tail (22 frames ≈ 0.92 s at 24 fps) whose framing and camera motion may differ from what the previous prompt imagined — and those frames are locked: text contradicting them tends to render as an overlay, not a replacement. When writing or editing `user_prompt`: - **Keep first, change later**: open with only "the camera keeps the direction and framing it arrived with; the subject keeps the pose, motion and light it arrived with". Never describe the previous segment's INTENDED ending as fact, and never open with a new composition, character layout or scene. - **Give the hold an action**: give the subject a breath, a weight shift or a gaze change so the hold does not render as a freeze-frame. - **Changes go later**: before about `00:02.000` no cut, no spoken line, no new key action; write a required composition or scene change as an explicit cut. - **Start actions mid-motion**: things already in progress at the seam start in motion — never as if completing on frame one. - **Self-contained**: the model never sees the previous prompt; fully define every ``, state scene and light, and never say "the same one" or "as before" for identity or scene. - **Images define identity only**: a reference image is not the first frame or a composition anchor of `[Shot 1]`; context joins use neither image anchoring nor `[video continuation]`. - **When more `context` segments follow, end in motion**: never write "ends", "arrives", "settles", "centers", "slows down"; no cut and no new spoken line in the last second. - **Need a clearly new composition**: use a `shot` segment with keyframes instead of `context`. - **Timestamps**: timestamps in the prompt map onto the generated segment, whose start sits about 0.92 s earlier than the saved segment; the node does not shift them automatically. When `system_prompt` is empty, the task output node automatically appends the same rules to the default MiniMax reference system prompt (entry rule: this segment is not the first and uses `context`; closing rule: the next segment uses `context`). The rules reach the rewriting LLM only through the `api` and `llm` prompt formats; `default` and `promptRelay` emit no system prompt. A custom `system_prompt` disables the auto-append — write the rules yourself. ## Video and audio segment content ```json { "media_type": "video", "source_type": "input", "file_path": "codex/session-123/clip.mp4", "file_name": "clip.mp4", "duration": 5.0, "muted": false, "volume_db": 0, "speed": 1 } ``` Audio sets `media_type` to `audio`. Get source durations with `ffprobe`; the timeline duration follows the user's target, trimming and speed. If media must align with a task segment, reuse the exact same `[start_frame, end_frame)` — never just similar seconds. ## Audio lock decisions `audio_locked` is a track-level field on audio/video tracks. Locking an audio track keeps its audio; locking a video track makes generation follow that video track's timeline and, with no locked audio track, keeps the video's own sound. Both locks can coexist; the audio track then wins as the `audio_lock` audio source while the video track still constrains the picture timeline. It is different from the track's `locked` flag: `locked` only blocks timeline editing and cannot substitute for `audio_locked`. ```json { "id": "music-track", "name": "Master Song", "type": "audio", "audio_locked": true, "muted": false, "segments": [ { "id": "song-segment", "start_frame": 0, "end_frame": 240, "content": { "media_type": "audio", "source_type": "input", "file_path": "codex/mv/song.wav", "file_name": "song.wav" } } ] } ``` A complete locked audio track can coexist with a partial locked video track. At runtime, when the current task has no intersection with the video, the video is simply not passed in and the master audio is still used — never split the project, extend the video, or stretch audio just to satisfy validation. Validation allows a partial video lock only when the locked audio continuously covers the selected tasks (multiple end-to-end audio segments are fine); without a complete master audio it still checks the locked media's intersection with each task. One TRACK_DATA may contain at most one `audio_locked: true` audio track and one `audio_locked: true` video track. When switching the locked source of one type, explicitly set the other tracks of that same type to `false`; never clear the other type's lock. Never write `audio_locked` into a media segment's `content`; the legacy segment-level field only exists for migration. Decide in this order, from context: 1. The user supplies a video for subject/partial replacement and did not ask to remove or replace its sound → lock that video track. 2. The user explicitly says "lock", "master audio", "keep the original sound", "don't regenerate audio" → use the audio/video track they mean. 3. MV, lyric video, beat-sync video, dance video, or visuals generated to a full song's beat and length → lock the song/score master track. 4. Narration, podcast or dialogue is the only basis for video length and shot rhythm → that narration master track may be locked. 5. Ordinary background music, ambience, SFX, character reference audio or temporary scratch audio is NOT auto-treated as locked, unless the user's wording makes it the generation driver. 6. One locked media track already exists and the user did not ask to change it → prefer keeping it; still check it intersects the task range being generated. With multiple audio tracks, disambiguate first by name, file name, source duration, track content and the user's description — e.g. `Master Song` or 完整歌曲 (full song) beats `SFX` or ambience. If two or more plausible candidates remain, STOP and ask the user, listing each candidate's track ID, name, file name and time range. Never decide by track order, first audio track or longest duration. Before locking also check: - The locked track contains at least one media segment matching its type: audio track → `media_type: "audio"`, video track → `media_type: "video"`; when locking a video's own sound, confirm the source video really has an audio track. - The locked media segments intersect every target task segment's `[start_frame, end_frame)`; if not, adjust the timeline or ask first. - As master audio it normally stays `muted: false`. Never touch unrelated solo, volume or edit-lock state just "because locking". - If the context clearly wants no original audio kept, ensure every audio/video track has `audio_locked: false`. ## Shared media `shared_reference` is a boolean on task image items and in audio/video segment `content`. When several context task segments need the same image, audio clip or video as a reference, mark that media shared; it does NOT need to overlap the task segments on the timeline. ```json { "id": "shared-audio-segment", "start_frame": 0, "end_frame": 120, "content": { "media_type": "audio", "source_type": "input", "file_path": "codex/voice/reference.wav", "file_name": "reference.wav", "shared_reference": true } } ``` Decision rules: - Mark media shared only when the user says "make it shared/common media", "all context segments reference this image/audio/video" or equivalent. Never auto-share just because file names match. - Task images are marked on their `content.images[]` items; several shared images are allowed; they are ordered before each task's private images and common + private still total at most 9 per task. - Audio/video are marked on the media segment's `content`; at most one per audio/video track; when switching, explicitly set the track's other segments to `false`. - Shared audio is used from the start of the source file, at most 15 seconds; shared video uses that source video as each task's reference video. - Shared media is a cross-task reference, not timeline-audio reuse. Never set `audio_locked` just because audio is shared; evaluate `audio_locked` separately, only when the user also wants the audio kept as-is and driving the timeline. - If the user wants one audio/video clip BOTH as a shared reference AND as normal timeline output, express the two roles with a separate track or segment and say so in the delivery summary. - `speaker_reference` is the legacy audio-reference field, only for reading and migrating old workflows; new or updated workflows always write `shared_reference`. - With multiple unresolvable candidates, list the media/segment IDs, track names, file names, durations or frame ranges and ask the user; never pick the first or the longest. ## Auto-orchestration decisions - User-given shot/segment lengths win; otherwise derive from source durations or the prompts' timing information, and state the derivation. - Subject/partial video replacement mode with no user-specified lengths always splits task segments into consecutive 10-second pieces, the last taking the remainder. - "N seconds each" means consecutive `[0, N*fps)`, `[N*fps, 2N*fps)`, … — never add one extra frame at the boundaries. - Prompts go into their task segments; reference images into that task segment's `images`; master video and audio into their own media tracks. - For subject-replacement, MV or other master-audio-driven projects, after arranging the timeline lock the single audio/video track and re-check it covers the target task range. - Keep gaps between tracks by default; close gaps and shift later segments only when the user asks for a tight arrangement. - When adjusting existing segments keep their segment IDs; generate new UUIDs only for new segments, tracks and image items. ## MultiTrack Project parameters - `segment_start_number` is the 1-based first task segment index; execution converts with `segment_start_number - 1`. - `segment_count = -1` processes from the start index to the end; a non-negative value limits how many task segments run this time. - `project_save`: `new` or `override`. With `override` and count `-1`, execution deletes saved segments after the start index before generating — say so clearly before changing it. - `sampling_mode`: `single` or `dual`; `sampling_plan` must be a preset available on the current node — never assume values from another workflow's environment. - The editor resolution is the first-pass size; `dual` never scales the first pass down. The second pass is `round(editor_size * upscale_by / 32) * 32` per dimension, Python-round to nearest, ties to even — not a fixed ceiling. `upscale_by` is an independent Project parameter, default `1.250`, three decimals, step `0.001`; old workflows keep their saved value. Both upsample paths and the merged export use the upscaled, aligned size. Example: `1344 x 768` at `1.250` gives a second pass of `1664 x 960`. If the user specifies a final output size, check BOTH the editor size and the multiplier; the editor size alone is not the dual-sampling final size. - Single sampling / first-pass-only preview keeps the editor size; `upscale_by = 1.000` upscales nothing but still allows a second pass; pure audio projects (`32 x 32`) do not upscale. Old first-pass checkpoints and context latents that were shrunk by the old back-calculation should be regenerated before continuing under the new rule. - `1st_pass_only` only applies to explicit one-stage checkpoint flows; never enable it automatically. ## The patch plan The plan contains only actual changes. `node_id` may be omitted (topology locates the nodes); `editor.resolution` is the complete DynamicCombo setting; `editor.track_data` is the complete TRACK_DATA. Never put a whole node or workflow into the plan. ```json { "editor": { "resolution": { "resolution": "width x height (megapixels)", "aspect_ratio": "16:9 (Widescreen)", "megapixels": 0.9 }, "track_data": { "tracks": [], "total_length": 120, "frame_rate": 24 }, "format": "MiniMax" }, "project": { "project_save": "override", "segment_start_number": 1, "segment_count": -1, "sampling_plan": "medium", "sampling_mode": "single" } } ``` Applied with the node pack's own script: ```sh python3 scripts/patch_workflow.py apply SOURCE.json --plan PLAN.json # dry-run python3 scripts/patch_workflow.py apply SOURCE.json --plan PLAN.json --output RESULT.json --write python3 scripts/patch_workflow.py inspect RESULT.json ``` --- ## prompt_override — plain text form: `scene1 | scene2 | scene3` (the simple default) The `prompt_override` input does not require any JSON. Paste plain text — one scene per pipe-separated chunk: ``` | | ``` The editor's execute (`nodes/basic.py`): a non-JSON `prompt_override` with a value goes to `build_multitrack_data_from_prompt_override` → `parse_override_segments` (`utils/prompt_override.py`), which splits on `|`, strips each part and drops empties — one task segment per part, a trailing `|` is a no-op. Proven by executing the pack's code (sandbox `scripts/prove_plain_text_override.py`): - Frames: the editor's existing `total_length` (its track_data / length input) is divided EVENLY across the parts — there are no per-segment durations in this form. 720 total, 3 parts → [0-240] [240-480] [480-720]. - No pipe at all = ONE segment spanning the whole timeline (the "one giant clip" trap). - Task type: parts default to `flf` → task_mode `default` → TaskOutput TYPE `t2v`. - Optional per-part extras (inside the part text): - `[s-e]` frame ranges, seconds form `[1.5s-3s]` (4-frame quanta), optional type suffix `[0-240,r2v]` → that segment becomes task_mode `ref` / TYPE `r2v` (other types: `flf`, `fmlf`, `ref`, `l2v`, `v2v`, `i2v`, `rv2v`, `vi2v`). With explicit ranges the timeline length becomes the last end + 1. The marker sits at the END of the segment, is stripped from the text before it reaches the task, and newlines INSIDE a segment survive — only `|` splits. Worked example: `examples/prompt_override_r2v_refmod.txt` — three six-section r2v scene prompts (each carrying the Sam refmod verbatim in `subject_definitions`, no `` because the story has zero images) with `[0-240,r2v]` / `[240-480,r2v]` / `[480-720,r2v]` markers; proven through the pack's parser: 3 tasks TYPE `r2v`, frames [0-240]/[240-480]/[480-720], task-duration sum 720, text byte-identical to the plan shots. - `@image1` / `@audio1` / `@video1` attach the editor's matching media input slots positionally. The `` bracket form is NOT parsed in this path — it stays literal text; only the JSON payload understands it. - Continuity: parts become independent segments; there is no `continuity_mode` in the text form (context joins are a JSON/TRACK_DATA feature). Use this form when the scenes share one length and t2v (or per-part `[s-e,type]`) is enough. Use the JSON payload below only for per-segment durations, continuity modes, locks or `` media tokens. --- ## prompt_override — JSON payload form (optional: per-segment durations, modes, locks) The `easy multiTrackEditor` node has an optional `prompt_override` input. When the editor's `format` is `MiniMax` and the pasted value is a `minimax_prompt_override` JSON payload, the editor rebuilds its whole task track from that payload: one task segment per `prompts` entry, laid back-to-back on a cursor, each spanning `round(seconds × frame_rate)` frames derived from its duration. **The payload has no frame fields and needs none** — never put `start_frame`/`end_frame` in it. `total_length` becomes the sum of the durations, so a 3-prompt payload at the default 10s each renders as THREE chained ~10-second clips. **The parser requires exactly two keys** (`parse_minimax_prompt_override`, `utils/prompt_override.py`: it checks `type` and that `prompts` is a list — nothing else): ```json {"type":"minimax_prompt_override","prompts":["prompt for segment one ...","prompt for segment two ..."]} ``` Every other field is OPTIONAL with a code default — write a key only when the story needs a non-default value: | key | default | meaning | | --- | --- | --- | | `duration` | `"10"` | SECONDS per clip (never frames), 2–15 enforced at parse time; single value or comma list with tail reuse | | `generation_type` | `"r2v"` | per clip, same single/comma rule: `r2v` (task_mode ref) / `l2v` / `t2v`,`i2v`,`v2v` (default-mode tasks). TaskOutput's TYPE echoes this string verbatim | | `continuity_mode` | `"shot"` | per clip: `shot` / `context` / `context_drift` (legacy `context_swap` is normalized to `context_drift`) | | `system_prompt` | `""` | written into every clip's content when non-empty | | `video_track_lock` | `0` | 1-based index of the video track to lock (0 = none) | | `audio_track_lock` | `0` | 1-based index of the audio track to lock (0 = none) | - `version` — the pack's own `easy minimaxPromptOverride` builder node writes `"version": 1`, but **nothing in the pack reads it**; the parser never touches it. Do not emit it. - The code default for `generation_type` is **`r2v`** (reference mode) — a bare minimal payload already renders as r2v/reference tasks with no key at all, which is why the gate's refmod guidance needs no explicit generation_type. Fallback warning (unchanged): **any other JSON pasted here is not rejected — it falls back to the generic text parser that splits on `|`.** A JSON blob contains no `|`, so it becomes ONE task segment spanning the whole timeline with the JSON dump as its prompt: the classic "rendered the whole story as a single clip" failure. - `prompts` — one entry per clip, in timeline order. The prompt text is the clip's full generation prompt (same standalone composition rules as task segments: no cross-segment references, context joins keep-then-change, only the last clip may come to rest). - Media attaches ONLY through tokens in the prompt text: ``, `