28 KiB
Executable File
MultiTrack Data and Orchestration Rules (English reference)
English reference for the easy multiTrackEditor TRACK_DATA and the patch plan, translated
from the node pack's own skill doc (skills/easy-media-multitrack-workflow/references/multitrack-schema.md
in ComfyUI-Easy-Media, which is written in Chinese). Front-end types live in
frontend/src/types/multitrack.ts; execution logic lives in nodes/basic.py of the node pack.
Field names are preserved exactly as the schema spells them.
Resolution (DynamicCombo)
resolution is a DynamicCombo widget of easy multiTrackEditor. It is NOT part of
TRACK_DATA. Changing it requires rebuilding the sub-widgets for the chosen mode and syncing
the workflow's existing widgets_values_named. Write it through patch_workflow.py's
editor.resolution — never by guessing array positions.
| Need | resolution label |
Required sub-fields |
|---|---|---|
| Fixed preset output size | e.g. 1344 x 768 (16:9) |
optional resize_method, default stretch |
| Explicit custom size | width x height (custom) |
width, height, optional resize_method |
| Follow first video's ratio | width x height (auto) |
optional resize_method |
| Fixed short/long edge | width x height (shortest) / width x height (longest) |
resize_to_pixel, optional resize_method |
| Ratio + total pixel budget | width x height (megapixels) |
aspect_ratio, megapixels |
Selection order: exact width/height from the user → an existing fixed preset, else custom; only orientation or aspect + quality/pixel budget given → megapixels; keep source video ratio → auto; "short/long edge = N" → shortest/longest. If the user expresses no resolution intent, keep the current setting. The model format's default size is only a reference and never overrides an explicit user requirement.
MP and ratio in natural language
Case-insensitive mp, Mpx, megapixel(s) (and Chinese "百万像素") mean the megapixels
unit. 0.9mp, 1 MP, 0.8 megapixels select megapixels mode and the number is taken
verbatim. A ratio in the same sentence maps to the full DynamicCombo aspect_ratio value:
| User ratio | aspect_ratio |
|---|---|
1:1 |
1:1 (Square) |
2:3 |
2:3 (Portrait Photo) |
3:2 |
3:2 (Photo) |
3:4 |
3:4 (Portrait Standard) |
4:3 |
4:3 (Standard) |
9:16 |
9:16 (Portrait Widescreen) |
16:9 |
16:9 (Widescreen) |
21:9 |
21:9 (Ultrawide) |
Example: "generate a 16:9 0.9mp video" normalizes with no follow-up question to:
{
"resolution": "width x height (megapixels)",
"aspect_ratio": "16:9 (Widescreen)",
"megapixels": 0.9
}
Never convert the MP number into width/height and never round it. MP only: if already in
megapixels mode keep its aspect_ratio; else pick 16:9/9:16 from explicit
landscape/portrait wording; still unclear → ask. Contradictory exact size, ratio and MP →
ask. Custom example:
{
"resolution": "width x height (custom)",
"width": 1024,
"height": 576,
"resize_method": "crop"
}
Valid resize_method: stretch, resize, pad, pad (white), pad_edge,
pad_edge_pixel, crop, pillarbox_blur. Prefer pad/resize to keep the whole picture,
crop to fill the canvas; never change an existing resize method the user did not mention.
Time semantics
start_frameis inclusive,end_frameexclusive; segment frames =end_frame - start_frame.- Seconds → frames:
round(seconds * frame_rate); every segment has at least 1 frame. - Segments in one track are sorted by
start_frameand never overlap; gaps are allowed. Different tracks of the same kind schedule independently. - With segments present,
total_lengthshould be the maximumend_frameover all tracks. With no segments the editor's current default length may stay. - Changing FPS must re-map every segment boundary,
origin_start_frameand task marker proportionally — not justframe_rate.
Top-level TRACK_DATA object
{
"muted": false,
"volume_db": 0,
"task_markers": [],
"task_overview": false,
"tracks": [],
"total_length": 240,
"frame_rate": 24
}
Track type is one of task, video, audio, subtitle. Keep unknown fields for
forward compatibility.
Common track and segment shape
{
"id": "uuid",
"name": "Video 0",
"type": "video",
"color": "var(--primary)",
"muted": false,
"solo": false,
"volume_db": 0,
"locked": false,
"segments": [
{
"id": "uuid",
"start_frame": 0,
"end_frame": 120,
"color": "var(--primary)",
"content": {}
}
]
}
New tracks use the project's existing color tokens: task tracks var(--multitrack-task-bg),
video tracks var(--primary). Keep an existing track's own color.
Task segment content
{
"media_type": "none",
"task_mode": "ref",
"continuity_mode": "shot",
"ref_image_size": "match",
"images": [],
"user_prompt": "the user prompt",
"system_prompt": "",
"user_prompt_variant": "a",
"user_prompt_b": "",
"muted": false,
"volume_db": 0
}
task_mode:default,l2v,ref,edit.- RefMod workflows (identity driven by
MiniMaxH3RefModsLoader+MiniMaxH3RefModApplyoutside the timeline): author segments astask_mode: "ref"even with no per-segment images — the payload then carriesgeneration_type: "r2v", TaskOutput TYPE becomesr2v, and the matchLine/promptLine pair switchesminimaxH3ToVideotoreferencemode, which pairs with the RefMod conditioning.refrows withimages: []are legal (refmod pictures ride the conditioning, not the timeline); prompts stay picture-token-free. Verified live by the user: the relay one-pass render works with r2v and refmods "work better with ref2va (r2v)". continuity_mode:shot(standalone shot),context(normal context join) orcontext_drift(context drift control); the legacy valuecontext_swapis still read.ref_image_size:matchormax.imagesholds at most 9 items; their order defines the picture-reference order.- When editing an existing task keep its
system_prompt, prompt variant, continuity and reference-image policy unless the user asked to change them. - If the current variant is
b, edituser_prompt_b; otherwise edituser_prompt. Never switch the variant unintentionally.
Task reference image item:
{
"id": "uuid",
"source_type": "input",
"file_path": "codex/session-123/reference.png",
"file_name": "reference.png"
}
source_type is one of input, output, local, url, slot. The path field must
match the source: input/output use file_path, local uses local_path, url uses
url, slot uses slot_name. Never label a local absolute path as input.
When several context segments reuse the same reference image, an image item may set
"shared_reference": true; it becomes a common image that is automatically ordered before
each task's private images; common + private images still total at most 9 per task. The
current front-end synchronizeSharedTaskImages generates new UUIDs for missing common-image
metadata on later tasks, so repeatedly normalizing the same sparse raw data yields different
IDs; from this code alone the full React maximum-update-depth / widget-unmount trigger chain
cannot be confirmed. Until the target front-end is verified to load such data stably, store
an explicit private image item per task (shared_reference: false) with a fixed ID pointing
at the same uploaded file — no duplication or re-upload needed. Switching continuity mode
does not fix this.
Context drift control mode
context_drift is the "context drift control" join: it carries the previous segment's scene
and motion context while letting the model replace subjects, people, objects or局部 regions
from a reference video. When the user supplies a video as a subject/person/object/clothing/face
replacement reference, orchestrate as follows:
- Set
content.continuity_modetocontext_drifton ALL task segments in scope — not only the second and later ones. The first segment keeps the same mode too (it has no previous generated context, but consistency matters for project metadata and re-generation). - Put the reference video on a video track covering the target task segments that need motion
reference. If the project also has passages that do not depend on the dance/reference video
(e.g. a verse), keep one complete project — the full locked audio covers all tasks and the
video only covers its part. If the video is also needed across tasks as a model reference,
set
shared_reference: trueon the video segment'scontent. - Set track-level
audio_locked: trueon that video track so the source video's timeline drives generation, and explicitly setaudio_locked: falseon every OTHER video track. To replace the original sound, lock a separate audio track, which then wins as the audio source. When only reusing the video's own sound, confirm before locking that the video actually contains a usable audio track and covers all target task segments. - If the user pinned segment lengths or boundaries, the user wins. Otherwise split task
segments into consecutive 10-second pieces: each piece is
round(10 * frame_rate)frames and the last piece takes the reference video's remaining frames — do not pad, do not drop the remainder, do not create an empty tail segment. The reference video itself can stay a single video segment covering the whole range; it does not need to be duplicated per task segment. - If the user explicitly asked to drop/replace/not keep the reference video's original sound, do not auto-lock that video track and follow the user's audio plan instead.
Context segment prompts
Applies to tasks with task_mode: "ref" and continuity_mode: "context";
context_drift (and legacy context_swap) do not use this section. Such a segment starts at
the previous segment's actually delivered tail (22 frames ≈ 0.92 s at 24 fps) whose framing
and camera motion may differ from what the previous prompt imagined — and those frames are
locked: text contradicting them tends to render as an overlay, not a replacement. When
writing or editing user_prompt:
- Keep first, change later: open with only "the camera keeps the direction and framing it arrived with; the subject keeps the pose, motion and light it arrived with". Never describe the previous segment's INTENDED ending as fact, and never open with a new composition, character layout or scene.
- Give the hold an action: give the subject a breath, a weight shift or a gaze change so the hold does not render as a freeze-frame.
- Changes go later: before about
00:02.000no cut, no spoken line, no new key action; write a required composition or scene change as an explicit cut. - Start actions mid-motion: things already in progress at the seam start in motion — never as if completing on frame one.
- Self-contained: the model never sees the previous prompt; fully define every
<Subject N>, state scene and light, and never say "the same one" or "as before" for identity or scene. - Images define identity only: a reference image is not the first frame or a composition
anchor of
[Shot 1]; context joins use neither image anchoring nor[video continuation]. - When more
contextsegments follow, end in motion: never write "ends", "arrives", "settles", "centers", "slows down"; no cut and no new spoken line in the last second. - Need a clearly new composition: use a
shotsegment with keyframes instead ofcontext. - Timestamps: timestamps in the prompt map onto the generated segment, whose start sits about 0.92 s earlier than the saved segment; the node does not shift them automatically.
When system_prompt is empty, the task output node automatically appends the same rules to
the default MiniMax reference system prompt (entry rule: this segment is not the first and
uses context; closing rule: the next segment uses context). The rules reach the rewriting
LLM only through the api and llm prompt formats; default and promptRelay emit no
system prompt. A custom system_prompt disables the auto-append — write the rules yourself.
Video and audio segment content
{
"media_type": "video",
"source_type": "input",
"file_path": "codex/session-123/clip.mp4",
"file_name": "clip.mp4",
"duration": 5.0,
"muted": false,
"volume_db": 0,
"speed": 1
}
Audio sets media_type to audio. Get source durations with ffprobe; the timeline
duration follows the user's target, trimming and speed. If media must align with a task
segment, reuse the exact same [start_frame, end_frame) — never just similar seconds.
Audio lock decisions
audio_locked is a track-level field on audio/video tracks. Locking an audio track keeps
its audio; locking a video track makes generation follow that video track's timeline and,
with no locked audio track, keeps the video's own sound. Both locks can coexist; the audio
track then wins as the audio_lock audio source while the video track still constrains the
picture timeline. It is different from the track's locked flag: locked only blocks
timeline editing and cannot substitute for audio_locked.
{
"id": "music-track",
"name": "Master Song",
"type": "audio",
"audio_locked": true,
"muted": false,
"segments": [
{
"id": "song-segment",
"start_frame": 0,
"end_frame": 240,
"content": {
"media_type": "audio",
"source_type": "input",
"file_path": "codex/mv/song.wav",
"file_name": "song.wav"
}
}
]
}
A complete locked audio track can coexist with a partial locked video track. At runtime, when the current task has no intersection with the video, the video is simply not passed in and the master audio is still used — never split the project, extend the video, or stretch audio just to satisfy validation. Validation allows a partial video lock only when the locked audio continuously covers the selected tasks (multiple end-to-end audio segments are fine); without a complete master audio it still checks the locked media's intersection with each task.
One TRACK_DATA may contain at most one audio_locked: true audio track and one
audio_locked: true video track. When switching the locked source of one type, explicitly set
the other tracks of that same type to false; never clear the other type's lock. Never write
audio_locked into a media segment's content; the legacy segment-level field only exists
for migration.
Decide in this order, from context:
- The user supplies a video for subject/partial replacement and did not ask to remove or replace its sound → lock that video track.
- The user explicitly says "lock", "master audio", "keep the original sound", "don't regenerate audio" → use the audio/video track they mean.
- MV, lyric video, beat-sync video, dance video, or visuals generated to a full song's beat and length → lock the song/score master track.
- Narration, podcast or dialogue is the only basis for video length and shot rhythm → that narration master track may be locked.
- Ordinary background music, ambience, SFX, character reference audio or temporary scratch audio is NOT auto-treated as locked, unless the user's wording makes it the generation driver.
- One locked media track already exists and the user did not ask to change it → prefer keeping it; still check it intersects the task range being generated.
With multiple audio tracks, disambiguate first by name, file name, source duration, track
content and the user's description — e.g. Master Song or 完整歌曲 (full song) beats SFX or
ambience. If two or more plausible candidates remain, STOP and ask the user, listing each
candidate's track ID, name, file name and time range. Never decide by track order, first
audio track or longest duration.
Before locking also check:
- The locked track contains at least one media segment matching its type: audio track →
media_type: "audio", video track →media_type: "video"; when locking a video's own sound, confirm the source video really has an audio track. - The locked media segments intersect every target task segment's
[start_frame, end_frame); if not, adjust the timeline or ask first. - As master audio it normally stays
muted: false. Never touch unrelated solo, volume or edit-lock state just "because locking". - If the context clearly wants no original audio kept, ensure every audio/video track has
audio_locked: false.
Shared media
shared_reference is a boolean on task image items and in audio/video segment content.
When several context task segments need the same image, audio clip or video as a reference,
mark that media shared; it does NOT need to overlap the task segments on the timeline.
{
"id": "shared-audio-segment",
"start_frame": 0,
"end_frame": 120,
"content": {
"media_type": "audio",
"source_type": "input",
"file_path": "codex/voice/reference.wav",
"file_name": "reference.wav",
"shared_reference": true
}
}
Decision rules:
- Mark media shared only when the user says "make it shared/common media", "all context segments reference this image/audio/video" or equivalent. Never auto-share just because file names match.
- Task images are marked on their
content.images[]items; several shared images are allowed; they are ordered before each task's private images and common + private still total at most 9 per task. - Audio/video are marked on the media segment's
content; at most one per audio/video track; when switching, explicitly set the track's other segments tofalse. - Shared audio is used from the start of the source file, at most 15 seconds; shared video uses that source video as each task's reference video.
- Shared media is a cross-task reference, not timeline-audio reuse. Never set
audio_lockedjust because audio is shared; evaluateaudio_lockedseparately, only when the user also wants the audio kept as-is and driving the timeline. - If the user wants one audio/video clip BOTH as a shared reference AND as normal timeline output, express the two roles with a separate track or segment and say so in the delivery summary.
speaker_referenceis the legacy audio-reference field, only for reading and migrating old workflows; new or updated workflows always writeshared_reference.- With multiple unresolvable candidates, list the media/segment IDs, track names, file names, durations or frame ranges and ask the user; never pick the first or the longest.
Auto-orchestration decisions
- User-given shot/segment lengths win; otherwise derive from source durations or the prompts' timing information, and state the derivation.
- Subject/partial video replacement mode with no user-specified lengths always splits task segments into consecutive 10-second pieces, the last taking the remainder.
- "N seconds each" means consecutive
[0, N*fps),[N*fps, 2N*fps), … — never add one extra frame at the boundaries. - Prompts go into their task segments; reference images into that task segment's
images; master video and audio into their own media tracks. - For subject-replacement, MV or other master-audio-driven projects, after arranging the timeline lock the single audio/video track and re-check it covers the target task range.
- Keep gaps between tracks by default; close gaps and shift later segments only when the user asks for a tight arrangement.
- When adjusting existing segments keep their segment IDs; generate new UUIDs only for new segments, tracks and image items.
MultiTrack Project parameters
segment_start_numberis the 1-based first task segment index; execution converts withsegment_start_number - 1.segment_count = -1processes from the start index to the end; a non-negative value limits how many task segments run this time.project_save:neworoverride. Withoverrideand count-1, execution deletes saved segments after the start index before generating — say so clearly before changing it.sampling_mode:singleordual;sampling_planmust be a preset available on the current node — never assume values from another workflow's environment.- The editor resolution is the first-pass size;
dualnever scales the first pass down. The second pass isround(editor_size * upscale_by / 32) * 32per dimension, Python-round to nearest, ties to even — not a fixed ceiling.upscale_byis an independent Project parameter, default1.250, three decimals, step0.001; old workflows keep their saved value. Both upsample paths and the merged export use the upscaled, aligned size. Example:1344 x 768at1.250gives a second pass of1664 x 960. If the user specifies a final output size, check BOTH the editor size and the multiplier; the editor size alone is not the dual-sampling final size. - Single sampling / first-pass-only preview keeps the editor size;
upscale_by = 1.000upscales nothing but still allows a second pass; pure audio projects (32 x 32) do not upscale. Old first-pass checkpoints and context latents that were shrunk by the old back-calculation should be regenerated before continuing under the new rule. 1st_pass_onlyonly applies to explicit one-stage checkpoint flows; never enable it automatically.
The patch plan
The plan contains only actual changes. node_id may be omitted (topology locates the nodes);
editor.resolution is the complete DynamicCombo setting; editor.track_data is the complete
TRACK_DATA. Never put a whole node or workflow into the plan.
{
"editor": {
"resolution": {
"resolution": "width x height (megapixels)",
"aspect_ratio": "16:9 (Widescreen)",
"megapixels": 0.9
},
"track_data": { "tracks": [], "total_length": 120, "frame_rate": 24 },
"format": "MiniMax"
},
"project": {
"project_save": "override",
"segment_start_number": 1,
"segment_count": -1,
"sampling_plan": "medium",
"sampling_mode": "single"
}
}
Applied with the node pack's own script:
python3 scripts/patch_workflow.py apply SOURCE.json --plan PLAN.json # dry-run
python3 scripts/patch_workflow.py apply SOURCE.json --plan PLAN.json --output RESULT.json --write
python3 scripts/patch_workflow.py inspect RESULT.json
prompt_override — plain text form: scene1 | scene2 | scene3 (the simple default)
The prompt_override input does not require any JSON. Paste plain text — one scene per
pipe-separated chunk:
<scene 1 text> | <scene 2 text> | <scene 3 text>
The editor's execute (nodes/basic.py): a non-JSON prompt_override with a value goes to
build_multitrack_data_from_prompt_override → parse_override_segments
(utils/prompt_override.py), which splits on |, strips each part and drops empties —
one task segment per part, a trailing | is a no-op. Proven by executing the pack's code
(sandbox scripts/prove_plain_text_override.py):
- Frames: the editor's existing
total_length(its track_data / length input) is divided EVENLY across the parts — there are no per-segment durations in this form. 720 total, 3 parts → [0-240] [240-480] [480-720]. - No pipe at all = ONE segment spanning the whole timeline (the "one giant clip" trap).
- Task type: parts default to
flf→ task_modedefault→ TaskOutput TYPEt2v. - Optional per-part extras (inside the part text):
[s-e]frame ranges, seconds form[1.5s-3s](4-frame quanta), optional type suffix[0-240,r2v]→ that segment becomes task_moderef/ TYPEr2v(other types:flf,fmlf,ref,l2v,v2v,i2v,rv2v,vi2v). With explicit ranges the timeline length becomes the last end + 1. The marker sits at the END of the segment, is stripped from the text before it reaches the task, and newlines INSIDE a segment survive — only|splits. Worked example:examples/prompt_override_r2v_refmod.txt— three six-section r2v scene prompts (each carrying the Sam refmod verbatim insubject_definitions, no<Picture N>because the story has zero images) with[0-240,r2v]/[240-480,r2v]/[480-720,r2v]markers; proven through the pack's parser: 3 tasks TYPEr2v, frames [0-240]/[240-480]/[480-720], task-duration sum 720, text byte-identical to the plan shots.@image1/@audio1/@video1attach the editor's matching media input slots positionally. The<Picture N>bracket form is NOT parsed in this path — it stays literal text; only the JSON payload understands it.
- Continuity: parts become independent segments; there is no
continuity_modein the text form (context joins are a JSON/TRACK_DATA feature).
Use this form when the scenes share one length and t2v (or per-part [s-e,type]) is
enough. Use the JSON payload below only for per-segment durations, continuity modes,
locks or <Picture N> media tokens.
prompt_override — JSON payload form (optional: per-segment durations, modes, locks)
The easy multiTrackEditor node has an optional prompt_override input. When the editor's
format is MiniMax and the pasted value is a minimax_prompt_override JSON payload, the
editor rebuilds its whole task track from that payload: one task segment per prompts
entry, laid back-to-back on a cursor, each spanning round(seconds × frame_rate) frames
derived from its duration. The payload has no frame fields and needs none — never put
start_frame/end_frame in it. total_length becomes the sum of the durations, so a
3-prompt payload at the default 10s each renders as THREE chained ~10-second clips.
The parser requires exactly two keys (parse_minimax_prompt_override,
utils/prompt_override.py: it checks type and that prompts is a list — nothing else):
{"type":"minimax_prompt_override","prompts":["prompt for segment one ...","prompt for segment two ..."]}
Every other field is OPTIONAL with a code default — write a key only when the story needs a non-default value:
| key | default | meaning |
|---|---|---|
duration |
"10" |
SECONDS per clip (never frames), 2–15 enforced at parse time; single value or comma list with tail reuse |
generation_type |
"r2v" |
per clip, same single/comma rule: r2v (task_mode ref) / l2v / t2v,i2v,v2v (default-mode tasks). TaskOutput's TYPE echoes this string verbatim |
continuity_mode |
"shot" |
per clip: shot / context / context_drift (legacy context_swap is normalized to context_drift) |
system_prompt |
"" |
written into every clip's content when non-empty |
video_track_lock |
0 |
1-based index of the video track to lock (0 = none) |
audio_track_lock |
0 |
1-based index of the audio track to lock (0 = none) |
version— the pack's owneasy minimaxPromptOverridebuilder node writes"version": 1, but nothing in the pack reads it; the parser never touches it. Do not emit it.- The code default for
generation_typeisr2v(reference mode) — a bare minimal payload already renders as r2v/reference tasks with no key at all, which is why the gate's refmod guidance needs no explicit generation_type.
Fallback warning (unchanged): any other JSON pasted here is not rejected — it falls back
to the generic text parser that splits on |. A JSON blob contains no |, so it becomes
ONE task segment spanning the whole timeline with the JSON dump as its prompt: the classic
"rendered the whole story as a single clip" failure.
prompts— one entry per clip, in timeline order. The prompt text is the clip's full generation prompt (same standalone composition rules as task segments: no cross-segment references, context joins keep-then-change, only the last clip may come to rest).- Media attaches ONLY through tokens in the prompt text:
<Picture N>,<Audio N>,<Video N>(the@imageN/@audioN/@videoNforms are accepted and rewritten, but this gate standardizes on the bracket form). N is the 1-based position of the item in the editor's matchingimage/audio/videoinput list — wire the media inventory into those inputs in inventory order. A prompt with no token renders with no reference media. The runtime renumbers tokens compactly for display, but slots are wired from the raw N.
The paste replaces the editor's existing task track (and derives its audio/video tracks
from the tokens), so the payload is self-contained: prompts, durations, modes and media
references all travel inside it. Edit task modes (edit) and full extra tracks are beyond
this payload — use the TRACK_DATA patch plan for those.