Portable self-contained skill (%USERPROFILE%\.agent\skills\easy-media-track-gate\): - references/track-schema.md: full English translation of the node pack's Chinese multitrack-schema.md (resolution DynamicCombo rules, time semantics, task segment content, context/context_drift joins, audio locks, shared media, project params, patch plan shape); field names cross-checked against patch_workflow.py source. - verify_track_rules.py (66 KB, stdlib only): mirrors every check the node pack's patch_workflow.py performs at apply time (ids, sorting, overlap, frames, source_type field matching, <=9 images, continuity enums, one locked audio + one locked video, locked-media task coverage, shared_reference rules, resolution tokens, project fields) and adds the gates upstream leaves to discipline: outline pins (id/frames/ modes/image set diffed per fragment AND at the merge), verbatim media contract (file_path character-for-character vs media_inventory, closest-match hint on rewrite), context prompt rules (self-contained in all modes, keep-then-change opener for joins, end-in-motion tail when another join follows), drift all-or- nothing + consecutive 10s split unless lengths_user_pinned. Subcommands: outline/ segment/trackdata/plan/suite; no-outline runs stay schema-only. Embedded suite 25 cases, prints ~0.7 KB (far under Cline's 8k cap). - build_track_plan.py: merge re-walks every fragment against the guide, assembles TRACK_DATA (task track from fragments; locked master-audio/reference-video tracks built from the outline, never retyped), writes plan.json + track_data.json with stable UUIDv5 ids; --check mode re-validates. - examples/: worked 3-segment rooftop example (outline, 3 fragments, plan.json, track_data.json) - merge is byte-deterministic and the generated plan dry-run- applies to the bundled v1.3.0 blank template with graph_preserved: true. - SKILL.md: five-move procedure with Cline cap workarounds (never read the 66 KB validator, @@CONTINUE@@ split, never read back generated plans), Step 0 media paths (upload only on explicit request; never invent a path), approval pause, prompt craft rules, delivery through the node pack's own patch_workflow.py (dry-run -> --write to a NEW file -> inspect); queueing stays the node pack's own flow on explicit user request.
22 KiB
MultiTrack Data and Orchestration Rules (English reference)
English reference for the easy multiTrackEditor TRACK_DATA and the patch plan, translated
from the node pack's own skill doc (skills/easy-media-multitrack-workflow/references/multitrack-schema.md
in ComfyUI-Easy-Media, which is written in Chinese). Front-end types live in
frontend/src/types/multitrack.ts; execution logic lives in nodes/basic.py of the node pack.
Field names are preserved exactly as the schema spells them.
Resolution (DynamicCombo)
resolution is a DynamicCombo widget of easy multiTrackEditor. It is NOT part of
TRACK_DATA. Changing it requires rebuilding the sub-widgets for the chosen mode and syncing
the workflow's existing widgets_values_named. Write it through patch_workflow.py's
editor.resolution — never by guessing array positions.
| Need | resolution label |
Required sub-fields |
|---|---|---|
| Fixed preset output size | e.g. 1344 x 768 (16:9) |
optional resize_method, default stretch |
| Explicit custom size | width x height (custom) |
width, height, optional resize_method |
| Follow first video's ratio | width x height (auto) |
optional resize_method |
| Fixed short/long edge | width x height (shortest) / width x height (longest) |
resize_to_pixel, optional resize_method |
| Ratio + total pixel budget | width x height (megapixels) |
aspect_ratio, megapixels |
Selection order: exact width/height from the user → an existing fixed preset, else custom; only orientation or aspect + quality/pixel budget given → megapixels; keep source video ratio → auto; "short/long edge = N" → shortest/longest. If the user expresses no resolution intent, keep the current setting. The model format's default size is only a reference and never overrides an explicit user requirement.
MP and ratio in natural language
Case-insensitive mp, Mpx, megapixel(s) (and Chinese "百万像素") mean the megapixels
unit. 0.9mp, 1 MP, 0.8 megapixels select megapixels mode and the number is taken
verbatim. A ratio in the same sentence maps to the full DynamicCombo aspect_ratio value:
| User ratio | aspect_ratio |
|---|---|
1:1 |
1:1 (Square) |
2:3 |
2:3 (Portrait Photo) |
3:2 |
3:2 (Photo) |
3:4 |
3:4 (Portrait Standard) |
4:3 |
4:3 (Standard) |
9:16 |
9:16 (Portrait Widescreen) |
16:9 |
16:9 (Widescreen) |
21:9 |
21:9 (Ultrawide) |
Example: "generate a 16:9 0.9mp video" normalizes with no follow-up question to:
{
"resolution": "width x height (megapixels)",
"aspect_ratio": "16:9 (Widescreen)",
"megapixels": 0.9
}
Never convert the MP number into width/height and never round it. MP only: if already in
megapixels mode keep its aspect_ratio; else pick 16:9/9:16 from explicit
landscape/portrait wording; still unclear → ask. Contradictory exact size, ratio and MP →
ask. Custom example:
{
"resolution": "width x height (custom)",
"width": 1024,
"height": 576,
"resize_method": "crop"
}
Valid resize_method: stretch, resize, pad, pad (white), pad_edge,
pad_edge_pixel, crop, pillarbox_blur. Prefer pad/resize to keep the whole picture,
crop to fill the canvas; never change an existing resize method the user did not mention.
Time semantics
start_frameis inclusive,end_frameexclusive; segment frames =end_frame - start_frame.- Seconds → frames:
round(seconds * frame_rate); every segment has at least 1 frame. - Segments in one track are sorted by
start_frameand never overlap; gaps are allowed. Different tracks of the same kind schedule independently. - With segments present,
total_lengthshould be the maximumend_frameover all tracks. With no segments the editor's current default length may stay. - Changing FPS must re-map every segment boundary,
origin_start_frameand task marker proportionally — not justframe_rate.
Top-level TRACK_DATA object
{
"muted": false,
"volume_db": 0,
"task_markers": [],
"task_overview": false,
"tracks": [],
"total_length": 240,
"frame_rate": 24
}
Track type is one of task, video, audio, subtitle. Keep unknown fields for
forward compatibility.
Common track and segment shape
{
"id": "uuid",
"name": "Video 0",
"type": "video",
"color": "var(--primary)",
"muted": false,
"solo": false,
"volume_db": 0,
"locked": false,
"segments": [
{
"id": "uuid",
"start_frame": 0,
"end_frame": 120,
"color": "var(--primary)",
"content": {}
}
]
}
New tracks use the project's existing color tokens: task tracks var(--multitrack-task-bg),
video tracks var(--primary). Keep an existing track's own color.
Task segment content
{
"media_type": "none",
"task_mode": "ref",
"continuity_mode": "shot",
"ref_image_size": "match",
"images": [],
"user_prompt": "the user prompt",
"system_prompt": "",
"user_prompt_variant": "a",
"user_prompt_b": "",
"muted": false,
"volume_db": 0
}
task_mode:default,l2v,ref,edit.continuity_mode:shot(standalone shot),context(normal context join) orcontext_drift(context drift control); the legacy valuecontext_swapis still read.ref_image_size:matchormax.imagesholds at most 9 items; their order defines the picture-reference order.- When editing an existing task keep its
system_prompt, prompt variant, continuity and reference-image policy unless the user asked to change them. - If the current variant is
b, edituser_prompt_b; otherwise edituser_prompt. Never switch the variant unintentionally.
Task reference image item:
{
"id": "uuid",
"source_type": "input",
"file_path": "codex/session-123/reference.png",
"file_name": "reference.png"
}
source_type is one of input, output, local, url, slot. The path field must
match the source: input/output use file_path, local uses local_path, url uses
url, slot uses slot_name. Never label a local absolute path as input.
When several context segments reuse the same reference image, an image item may set
"shared_reference": true; it becomes a common image that is automatically ordered before
each task's private images; common + private images still total at most 9 per task. The
current front-end synchronizeSharedTaskImages generates new UUIDs for missing common-image
metadata on later tasks, so repeatedly normalizing the same sparse raw data yields different
IDs; from this code alone the full React maximum-update-depth / widget-unmount trigger chain
cannot be confirmed. Until the target front-end is verified to load such data stably, store
an explicit private image item per task (shared_reference: false) with a fixed ID pointing
at the same uploaded file — no duplication or re-upload needed. Switching continuity mode
does not fix this.
Context drift control mode
context_drift is the "context drift control" join: it carries the previous segment's scene
and motion context while letting the model replace subjects, people, objects or局部 regions
from a reference video. When the user supplies a video as a subject/person/object/clothing/face
replacement reference, orchestrate as follows:
- Set
content.continuity_modetocontext_drifton ALL task segments in scope — not only the second and later ones. The first segment keeps the same mode too (it has no previous generated context, but consistency matters for project metadata and re-generation). - Put the reference video on a video track covering the target task segments that need motion
reference. If the project also has passages that do not depend on the dance/reference video
(e.g. a verse), keep one complete project — the full locked audio covers all tasks and the
video only covers its part. If the video is also needed across tasks as a model reference,
set
shared_reference: trueon the video segment'scontent. - Set track-level
audio_locked: trueon that video track so the source video's timeline drives generation, and explicitly setaudio_locked: falseon every OTHER video track. To replace the original sound, lock a separate audio track, which then wins as the audio source. When only reusing the video's own sound, confirm before locking that the video actually contains a usable audio track and covers all target task segments. - If the user pinned segment lengths or boundaries, the user wins. Otherwise split task
segments into consecutive 10-second pieces: each piece is
round(10 * frame_rate)frames and the last piece takes the reference video's remaining frames — do not pad, do not drop the remainder, do not create an empty tail segment. The reference video itself can stay a single video segment covering the whole range; it does not need to be duplicated per task segment. - If the user explicitly asked to drop/replace/not keep the reference video's original sound, do not auto-lock that video track and follow the user's audio plan instead.
Context segment prompts
Applies to tasks with task_mode: "ref" and continuity_mode: "context";
context_drift (and legacy context_swap) do not use this section. Such a segment starts at
the previous segment's actually delivered tail (22 frames ≈ 0.92 s at 24 fps) whose framing
and camera motion may differ from what the previous prompt imagined — and those frames are
locked: text contradicting them tends to render as an overlay, not a replacement. When
writing or editing user_prompt:
- Keep first, change later: open with only "the camera keeps the direction and framing it arrived with; the subject keeps the pose, motion and light it arrived with". Never describe the previous segment's INTENDED ending as fact, and never open with a new composition, character layout or scene.
- Give the hold an action: give the subject a breath, a weight shift or a gaze change so the hold does not render as a freeze-frame.
- Changes go later: before about
00:02.000no cut, no spoken line, no new key action; write a required composition or scene change as an explicit cut. - Start actions mid-motion: things already in progress at the seam start in motion — never as if completing on frame one.
- Self-contained: the model never sees the previous prompt; fully define every
<Subject N>, state scene and light, and never say "the same one" or "as before" for identity or scene. - Images define identity only: a reference image is not the first frame or a composition
anchor of
[Shot 1]; context joins use neither image anchoring nor[video continuation]. - When more
contextsegments follow, end in motion: never write "ends", "arrives", "settles", "centers", "slows down"; no cut and no new spoken line in the last second. - Need a clearly new composition: use a
shotsegment with keyframes instead ofcontext. - Timestamps: timestamps in the prompt map onto the generated segment, whose start sits about 0.92 s earlier than the saved segment; the node does not shift them automatically.
When system_prompt is empty, the task output node automatically appends the same rules to
the default MiniMax reference system prompt (entry rule: this segment is not the first and
uses context; closing rule: the next segment uses context). The rules reach the rewriting
LLM only through the api and llm prompt formats; default and promptRelay emit no
system prompt. A custom system_prompt disables the auto-append — write the rules yourself.
Video and audio segment content
{
"media_type": "video",
"source_type": "input",
"file_path": "codex/session-123/clip.mp4",
"file_name": "clip.mp4",
"duration": 5.0,
"muted": false,
"volume_db": 0,
"speed": 1
}
Audio sets media_type to audio. Get source durations with ffprobe; the timeline
duration follows the user's target, trimming and speed. If media must align with a task
segment, reuse the exact same [start_frame, end_frame) — never just similar seconds.
Audio lock decisions
audio_locked is a track-level field on audio/video tracks. Locking an audio track keeps
its audio; locking a video track makes generation follow that video track's timeline and,
with no locked audio track, keeps the video's own sound. Both locks can coexist; the audio
track then wins as the audio_lock audio source while the video track still constrains the
picture timeline. It is different from the track's locked flag: locked only blocks
timeline editing and cannot substitute for audio_locked.
{
"id": "music-track",
"name": "Master Song",
"type": "audio",
"audio_locked": true,
"muted": false,
"segments": [
{
"id": "song-segment",
"start_frame": 0,
"end_frame": 240,
"content": {
"media_type": "audio",
"source_type": "input",
"file_path": "codex/mv/song.wav",
"file_name": "song.wav"
}
}
]
}
A complete locked audio track can coexist with a partial locked video track. At runtime, when the current task has no intersection with the video, the video is simply not passed in and the master audio is still used — never split the project, extend the video, or stretch audio just to satisfy validation. Validation allows a partial video lock only when the locked audio continuously covers the selected tasks (multiple end-to-end audio segments are fine); without a complete master audio it still checks the locked media's intersection with each task.
One TRACK_DATA may contain at most one audio_locked: true audio track and one
audio_locked: true video track. When switching the locked source of one type, explicitly set
the other tracks of that same type to false; never clear the other type's lock. Never write
audio_locked into a media segment's content; the legacy segment-level field only exists
for migration.
Decide in this order, from context:
- The user supplies a video for subject/partial replacement and did not ask to remove or replace its sound → lock that video track.
- The user explicitly says "lock", "master audio", "keep the original sound", "don't regenerate audio" → use the audio/video track they mean.
- MV, lyric video, beat-sync video, dance video, or visuals generated to a full song's beat and length → lock the song/score master track.
- Narration, podcast or dialogue is the only basis for video length and shot rhythm → that narration master track may be locked.
- Ordinary background music, ambience, SFX, character reference audio or temporary scratch audio is NOT auto-treated as locked, unless the user's wording makes it the generation driver.
- One locked media track already exists and the user did not ask to change it → prefer keeping it; still check it intersects the task range being generated.
With multiple audio tracks, disambiguate first by name, file name, source duration, track
content and the user's description — e.g. Master Song or 完整歌曲 (full song) beats SFX or
ambience. If two or more plausible candidates remain, STOP and ask the user, listing each
candidate's track ID, name, file name and time range. Never decide by track order, first
audio track or longest duration.
Before locking also check:
- The locked track contains at least one media segment matching its type: audio track →
media_type: "audio", video track →media_type: "video"; when locking a video's own sound, confirm the source video really has an audio track. - The locked media segments intersect every target task segment's
[start_frame, end_frame); if not, adjust the timeline or ask first. - As master audio it normally stays
muted: false. Never touch unrelated solo, volume or edit-lock state just "because locking". - If the context clearly wants no original audio kept, ensure every audio/video track has
audio_locked: false.
Shared media
shared_reference is a boolean on task image items and in audio/video segment content.
When several context task segments need the same image, audio clip or video as a reference,
mark that media shared; it does NOT need to overlap the task segments on the timeline.
{
"id": "shared-audio-segment",
"start_frame": 0,
"end_frame": 120,
"content": {
"media_type": "audio",
"source_type": "input",
"file_path": "codex/voice/reference.wav",
"file_name": "reference.wav",
"shared_reference": true
}
}
Decision rules:
- Mark media shared only when the user says "make it shared/common media", "all context segments reference this image/audio/video" or equivalent. Never auto-share just because file names match.
- Task images are marked on their
content.images[]items; several shared images are allowed; they are ordered before each task's private images and common + private still total at most 9 per task. - Audio/video are marked on the media segment's
content; at most one per audio/video track; when switching, explicitly set the track's other segments tofalse. - Shared audio is used from the start of the source file, at most 15 seconds; shared video uses that source video as each task's reference video.
- Shared media is a cross-task reference, not timeline-audio reuse. Never set
audio_lockedjust because audio is shared; evaluateaudio_lockedseparately, only when the user also wants the audio kept as-is and driving the timeline. - If the user wants one audio/video clip BOTH as a shared reference AND as normal timeline output, express the two roles with a separate track or segment and say so in the delivery summary.
speaker_referenceis the legacy audio-reference field, only for reading and migrating old workflows; new or updated workflows always writeshared_reference.- With multiple unresolvable candidates, list the media/segment IDs, track names, file names, durations or frame ranges and ask the user; never pick the first or the longest.
Auto-orchestration decisions
- User-given shot/segment lengths win; otherwise derive from source durations or the prompts' timing information, and state the derivation.
- Subject/partial video replacement mode with no user-specified lengths always splits task segments into consecutive 10-second pieces, the last taking the remainder.
- "N seconds each" means consecutive
[0, N*fps),[N*fps, 2N*fps), … — never add one extra frame at the boundaries. - Prompts go into their task segments; reference images into that task segment's
images; master video and audio into their own media tracks. - For subject-replacement, MV or other master-audio-driven projects, after arranging the timeline lock the single audio/video track and re-check it covers the target task range.
- Keep gaps between tracks by default; close gaps and shift later segments only when the user asks for a tight arrangement.
- When adjusting existing segments keep their segment IDs; generate new UUIDs only for new segments, tracks and image items.
MultiTrack Project parameters
segment_start_numberis the 1-based first task segment index; execution converts withsegment_start_number - 1.segment_count = -1processes from the start index to the end; a non-negative value limits how many task segments run this time.project_save:neworoverride. Withoverrideand count-1, execution deletes saved segments after the start index before generating — say so clearly before changing it.sampling_mode:singleordual;sampling_planmust be a preset available on the current node — never assume values from another workflow's environment.- The editor resolution is the first-pass size;
dualnever scales the first pass down. The second pass isround(editor_size * upscale_by / 32) * 32per dimension, Python-round to nearest, ties to even — not a fixed ceiling.upscale_byis an independent Project parameter, default1.250, three decimals, step0.001; old workflows keep their saved value. Both upsample paths and the merged export use the upscaled, aligned size. Example:1344 x 768at1.250gives a second pass of1664 x 960. If the user specifies a final output size, check BOTH the editor size and the multiplier; the editor size alone is not the dual-sampling final size. - Single sampling / first-pass-only preview keeps the editor size;
upscale_by = 1.000upscales nothing but still allows a second pass; pure audio projects (32 x 32) do not upscale. Old first-pass checkpoints and context latents that were shrunk by the old back-calculation should be regenerated before continuing under the new rule. 1st_pass_onlyonly applies to explicit one-stage checkpoint flows; never enable it automatically.
The patch plan
The plan contains only actual changes. node_id may be omitted (topology locates the nodes);
editor.resolution is the complete DynamicCombo setting; editor.track_data is the complete
TRACK_DATA. Never put a whole node or workflow into the plan.
{
"editor": {
"resolution": {
"resolution": "width x height (megapixels)",
"aspect_ratio": "16:9 (Widescreen)",
"megapixels": 0.9
},
"track_data": { "tracks": [], "total_length": 120, "frame_rate": 24 },
"format": "MiniMax"
},
"project": {
"project_save": "override",
"segment_start_number": 1,
"segment_count": -1,
"sampling_plan": "medium",
"sampling_mode": "single"
}
}
Applied with the node pack's own script:
python3 scripts/patch_workflow.py apply SOURCE.json --plan PLAN.json # dry-run
python3 scripts/patch_workflow.py apply SOURCE.json --plan PLAN.json --output RESULT.json --write
python3 scripts/patch_workflow.py inspect RESULT.json