mmprompt/easy-media-track-gate/references/track-schema.md
Z User ea5c5a71b4 easy-media-track-gate: the h3-plan-gate methodology adapted to ComfyUI-Easy-Media multitrack nodes
Portable self-contained skill (%USERPROFILE%\.agent\skills\easy-media-track-gate\):
- references/track-schema.md: full English translation of the node pack's Chinese
  multitrack-schema.md (resolution DynamicCombo rules, time semantics, task segment
  content, context/context_drift joins, audio locks, shared media, project params,
  patch plan shape); field names cross-checked against patch_workflow.py source.
- verify_track_rules.py (66 KB, stdlib only): mirrors every check the node pack's
  patch_workflow.py performs at apply time (ids, sorting, overlap, frames, source_type
  field matching, <=9 images, continuity enums, one locked audio + one locked video,
  locked-media task coverage, shared_reference rules, resolution tokens, project
  fields) and adds the gates upstream leaves to discipline: outline pins (id/frames/
  modes/image set diffed per fragment AND at the merge), verbatim media contract
  (file_path character-for-character vs media_inventory, closest-match hint on
  rewrite), context prompt rules (self-contained in all modes, keep-then-change
  opener for joins, end-in-motion tail when another join follows), drift all-or-
  nothing + consecutive 10s split unless lengths_user_pinned. Subcommands: outline/
  segment/trackdata/plan/suite; no-outline runs stay schema-only. Embedded suite
  25 cases, prints ~0.7 KB (far under Cline's 8k cap).
- build_track_plan.py: merge re-walks every fragment against the guide, assembles
  TRACK_DATA (task track from fragments; locked master-audio/reference-video tracks
  built from the outline, never retyped), writes plan.json + track_data.json with
  stable UUIDv5 ids; --check mode re-validates.
- examples/: worked 3-segment rooftop example (outline, 3 fragments, plan.json,
  track_data.json) - merge is byte-deterministic and the generated plan dry-run-
  applies to the bundled v1.3.0 blank template with graph_preserved: true.
- SKILL.md: five-move procedure with Cline cap workarounds (never read the 66 KB
  validator, @@CONTINUE@@ split, never read back generated plans), Step 0 media
  paths (upload only on explicit request; never invent a path), approval pause,
  prompt craft rules, delivery through the node pack's own patch_workflow.py
  (dry-run -> --write to a NEW file -> inspect); queueing stays the node pack's
  own flow on explicit user request.
2026-11-01 01:09:26 +00:00

466 lines
22 KiB
Markdown

# MultiTrack Data and Orchestration Rules (English reference)
English reference for the `easy multiTrackEditor` TRACK_DATA and the patch plan, translated
from the node pack's own skill doc (`skills/easy-media-multitrack-workflow/references/multitrack-schema.md`
in ComfyUI-Easy-Media, which is written in Chinese). Front-end types live in
`frontend/src/types/multitrack.ts`; execution logic lives in `nodes/basic.py` of the node pack.
Field names are preserved exactly as the schema spells them.
## Resolution (DynamicCombo)
`resolution` is a DynamicCombo widget of `easy multiTrackEditor`. It is NOT part of
TRACK_DATA. Changing it requires rebuilding the sub-widgets for the chosen mode and syncing
the workflow's existing `widgets_values_named`. Write it through `patch_workflow.py`'s
`editor.resolution` — never by guessing array positions.
| Need | `resolution` label | Required sub-fields |
| --- | --- | --- |
| Fixed preset output size | e.g. `1344 x 768 (16:9)` | optional `resize_method`, default `stretch` |
| Explicit custom size | `width x height (custom)` | `width`, `height`, optional `resize_method` |
| Follow first video's ratio | `width x height (auto)` | optional `resize_method` |
| Fixed short/long edge | `width x height (shortest)` / `width x height (longest)` | `resize_to_pixel`, optional `resize_method` |
| Ratio + total pixel budget | `width x height (megapixels)` | `aspect_ratio`, `megapixels` |
Selection order: exact width/height from the user → an existing fixed preset, else custom;
only orientation or aspect + quality/pixel budget given → megapixels; keep source video
ratio → auto; "short/long edge = N" → shortest/longest. If the user expresses no resolution
intent, keep the current setting. The model format's default size is only a reference and
never overrides an explicit user requirement.
### MP and ratio in natural language
Case-insensitive `mp`, `Mpx`, `megapixel(s)` (and Chinese "百万像素") mean the megapixels
unit. `0.9mp`, `1 MP`, `0.8 megapixels` select megapixels mode and the number is taken
verbatim. A ratio in the same sentence maps to the full DynamicCombo `aspect_ratio` value:
| User ratio | `aspect_ratio` |
| --- | --- |
| `1:1` | `1:1 (Square)` |
| `2:3` | `2:3 (Portrait Photo)` |
| `3:2` | `3:2 (Photo)` |
| `3:4` | `3:4 (Portrait Standard)` |
| `4:3` | `4:3 (Standard)` |
| `9:16` | `9:16 (Portrait Widescreen)` |
| `16:9` | `16:9 (Widescreen)` |
| `21:9` | `21:9 (Ultrawide)` |
Example: "generate a 16:9 0.9mp video" normalizes with no follow-up question to:
```json
{
"resolution": "width x height (megapixels)",
"aspect_ratio": "16:9 (Widescreen)",
"megapixels": 0.9
}
```
Never convert the MP number into width/height and never round it. MP only: if already in
megapixels mode keep its `aspect_ratio`; else pick `16:9`/`9:16` from explicit
landscape/portrait wording; still unclear → ask. Contradictory exact size, ratio and MP →
ask. Custom example:
```json
{
"resolution": "width x height (custom)",
"width": 1024,
"height": 576,
"resize_method": "crop"
}
```
Valid `resize_method`: `stretch`, `resize`, `pad`, `pad (white)`, `pad_edge`,
`pad_edge_pixel`, `crop`, `pillarbox_blur`. Prefer pad/resize to keep the whole picture,
crop to fill the canvas; never change an existing resize method the user did not mention.
## Time semantics
- `start_frame` is inclusive, `end_frame` exclusive; segment frames = `end_frame - start_frame`.
- Seconds → frames: `round(seconds * frame_rate)`; every segment has at least 1 frame.
- Segments in one track are sorted by `start_frame` and never overlap; gaps are allowed.
Different tracks of the same kind schedule independently.
- With segments present, `total_length` should be the maximum `end_frame` over all tracks.
With no segments the editor's current default length may stay.
- Changing FPS must re-map every segment boundary, `origin_start_frame` and task marker
proportionally — not just `frame_rate`.
## Top-level TRACK_DATA object
```json
{
"muted": false,
"volume_db": 0,
"task_markers": [],
"task_overview": false,
"tracks": [],
"total_length": 240,
"frame_rate": 24
}
```
Track `type` is one of `task`, `video`, `audio`, `subtitle`. Keep unknown fields for
forward compatibility.
## Common track and segment shape
```json
{
"id": "uuid",
"name": "Video 0",
"type": "video",
"color": "var(--primary)",
"muted": false,
"solo": false,
"volume_db": 0,
"locked": false,
"segments": [
{
"id": "uuid",
"start_frame": 0,
"end_frame": 120,
"color": "var(--primary)",
"content": {}
}
]
}
```
New tracks use the project's existing color tokens: task tracks `var(--multitrack-task-bg)`,
video tracks `var(--primary)`. Keep an existing track's own color.
## Task segment content
```json
{
"media_type": "none",
"task_mode": "ref",
"continuity_mode": "shot",
"ref_image_size": "match",
"images": [],
"user_prompt": "the user prompt",
"system_prompt": "",
"user_prompt_variant": "a",
"user_prompt_b": "",
"muted": false,
"volume_db": 0
}
```
- `task_mode`: `default`, `l2v`, `ref`, `edit`.
- `continuity_mode`: `shot` (standalone shot), `context` (normal context join) or
`context_drift` (context drift control); the legacy value `context_swap` is still read.
- `ref_image_size`: `match` or `max`.
- `images` holds at most 9 items; their order defines the picture-reference order.
- When editing an existing task keep its `system_prompt`, prompt variant, continuity and
reference-image policy unless the user asked to change them.
- If the current variant is `b`, edit `user_prompt_b`; otherwise edit `user_prompt`. Never
switch the variant unintentionally.
Task reference image item:
```json
{
"id": "uuid",
"source_type": "input",
"file_path": "codex/session-123/reference.png",
"file_name": "reference.png"
}
```
`source_type` is one of `input`, `output`, `local`, `url`, `slot`. The path field must
match the source: `input`/`output` use `file_path`, `local` uses `local_path`, `url` uses
`url`, `slot` uses `slot_name`. Never label a local absolute path as `input`.
When several context segments reuse the same reference image, an image item may set
`"shared_reference": true`; it becomes a common image that is automatically ordered before
each task's private images; common + private images still total at most 9 per task. The
current front-end `synchronizeSharedTaskImages` generates new UUIDs for missing common-image
metadata on later tasks, so repeatedly normalizing the same sparse raw data yields different
IDs; from this code alone the full React maximum-update-depth / widget-unmount trigger chain
cannot be confirmed. Until the target front-end is verified to load such data stably, store
an explicit private image item per task (`shared_reference: false`) with a fixed ID pointing
at the same uploaded file — no duplication or re-upload needed. Switching continuity mode
does not fix this.
## Context drift control mode
`context_drift` is the "context drift control" join: it carries the previous segment's scene
and motion context while letting the model replace subjects, people, objects or局部 regions
from a reference video. When the user supplies a video as a subject/person/object/clothing/face
replacement reference, orchestrate as follows:
- Set `content.continuity_mode` to `context_drift` on ALL task segments in scope — not only
the second and later ones. The first segment keeps the same mode too (it has no previous
generated context, but consistency matters for project metadata and re-generation).
- Put the reference video on a video track covering the target task segments that need motion
reference. If the project also has passages that do not depend on the dance/reference video
(e.g. a verse), keep one complete project — the full locked audio covers all tasks and the
video only covers its part. If the video is also needed across tasks as a model reference,
set `shared_reference: true` on the video segment's `content`.
- Set track-level `audio_locked: true` on that video track so the source video's timeline
drives generation, and explicitly set `audio_locked: false` on every OTHER video track. To
replace the original sound, lock a separate audio track, which then wins as the audio
source. When only reusing the video's own sound, confirm before locking that the video
actually contains a usable audio track and covers all target task segments.
- If the user pinned segment lengths or boundaries, the user wins. Otherwise split task
segments into consecutive 10-second pieces: each piece is `round(10 * frame_rate)` frames
and the last piece takes the reference video's remaining frames — do not pad, do not drop
the remainder, do not create an empty tail segment. The reference video itself can stay a
single video segment covering the whole range; it does not need to be duplicated per task
segment.
- If the user explicitly asked to drop/replace/not keep the reference video's original sound,
do not auto-lock that video track and follow the user's audio plan instead.
## Context segment prompts
Applies to tasks with `task_mode: "ref"` and `continuity_mode: "context"`;
`context_drift` (and legacy `context_swap`) do not use this section. Such a segment starts at
the previous segment's actually delivered tail (22 frames ≈ 0.92 s at 24 fps) whose framing
and camera motion may differ from what the previous prompt imagined — and those frames are
locked: text contradicting them tends to render as an overlay, not a replacement. When
writing or editing `user_prompt`:
- **Keep first, change later**: open with only "the camera keeps the direction and framing it
arrived with; the subject keeps the pose, motion and light it arrived with". Never describe
the previous segment's INTENDED ending as fact, and never open with a new composition,
character layout or scene.
- **Give the hold an action**: give the subject a breath, a weight shift or a gaze change so
the hold does not render as a freeze-frame.
- **Changes go later**: before about `00:02.000` no cut, no spoken line, no new key action;
write a required composition or scene change as an explicit cut.
- **Start actions mid-motion**: things already in progress at the seam start in motion —
never as if completing on frame one.
- **Self-contained**: the model never sees the previous prompt; fully define every `<Subject N>`,
state scene and light, and never say "the same one" or "as before" for identity or scene.
- **Images define identity only**: a reference image is not the first frame or a composition
anchor of `[Shot 1]`; context joins use neither image anchoring nor `[video continuation]`.
- **When more `context` segments follow, end in motion**: never write "ends", "arrives",
"settles", "centers", "slows down"; no cut and no new spoken line in the last second.
- **Need a clearly new composition**: use a `shot` segment with keyframes instead of `context`.
- **Timestamps**: timestamps in the prompt map onto the generated segment, whose start sits
about 0.92 s earlier than the saved segment; the node does not shift them automatically.
When `system_prompt` is empty, the task output node automatically appends the same rules to
the default MiniMax reference system prompt (entry rule: this segment is not the first and
uses `context`; closing rule: the next segment uses `context`). The rules reach the rewriting
LLM only through the `api` and `llm` prompt formats; `default` and `promptRelay` emit no
system prompt. A custom `system_prompt` disables the auto-append — write the rules yourself.
## Video and audio segment content
```json
{
"media_type": "video",
"source_type": "input",
"file_path": "codex/session-123/clip.mp4",
"file_name": "clip.mp4",
"duration": 5.0,
"muted": false,
"volume_db": 0,
"speed": 1
}
```
Audio sets `media_type` to `audio`. Get source durations with `ffprobe`; the timeline
duration follows the user's target, trimming and speed. If media must align with a task
segment, reuse the exact same `[start_frame, end_frame)` — never just similar seconds.
## Audio lock decisions
`audio_locked` is a track-level field on audio/video tracks. Locking an audio track keeps
its audio; locking a video track makes generation follow that video track's timeline and,
with no locked audio track, keeps the video's own sound. Both locks can coexist; the audio
track then wins as the `audio_lock` audio source while the video track still constrains the
picture timeline. It is different from the track's `locked` flag: `locked` only blocks
timeline editing and cannot substitute for `audio_locked`.
```json
{
"id": "music-track",
"name": "Master Song",
"type": "audio",
"audio_locked": true,
"muted": false,
"segments": [
{
"id": "song-segment",
"start_frame": 0,
"end_frame": 240,
"content": {
"media_type": "audio",
"source_type": "input",
"file_path": "codex/mv/song.wav",
"file_name": "song.wav"
}
}
]
}
```
A complete locked audio track can coexist with a partial locked video track. At runtime, when
the current task has no intersection with the video, the video is simply not passed in and
the master audio is still used — never split the project, extend the video, or stretch audio
just to satisfy validation. Validation allows a partial video lock only when the locked audio
continuously covers the selected tasks (multiple end-to-end audio segments are fine); without
a complete master audio it still checks the locked media's intersection with each task.
One TRACK_DATA may contain at most one `audio_locked: true` audio track and one
`audio_locked: true` video track. When switching the locked source of one type, explicitly set
the other tracks of that same type to `false`; never clear the other type's lock. Never write
`audio_locked` into a media segment's `content`; the legacy segment-level field only exists
for migration.
Decide in this order, from context:
1. The user supplies a video for subject/partial replacement and did not ask to remove or
replace its sound → lock that video track.
2. The user explicitly says "lock", "master audio", "keep the original sound", "don't
regenerate audio" → use the audio/video track they mean.
3. MV, lyric video, beat-sync video, dance video, or visuals generated to a full song's beat
and length → lock the song/score master track.
4. Narration, podcast or dialogue is the only basis for video length and shot rhythm → that
narration master track may be locked.
5. Ordinary background music, ambience, SFX, character reference audio or temporary scratch
audio is NOT auto-treated as locked, unless the user's wording makes it the generation
driver.
6. One locked media track already exists and the user did not ask to change it → prefer
keeping it; still check it intersects the task range being generated.
With multiple audio tracks, disambiguate first by name, file name, source duration, track
content and the user's description — e.g. `Master Song` or 完整歌曲 (full song) beats `SFX` or
ambience. If two or more plausible candidates remain, STOP and ask the user, listing each
candidate's track ID, name, file name and time range. Never decide by track order, first
audio track or longest duration.
Before locking also check:
- The locked track contains at least one media segment matching its type: audio track →
`media_type: "audio"`, video track → `media_type: "video"`; when locking a video's own
sound, confirm the source video really has an audio track.
- The locked media segments intersect every target task segment's `[start_frame, end_frame)`;
if not, adjust the timeline or ask first.
- As master audio it normally stays `muted: false`. Never touch unrelated solo, volume or
edit-lock state just "because locking".
- If the context clearly wants no original audio kept, ensure every audio/video track has
`audio_locked: false`.
## Shared media
`shared_reference` is a boolean on task image items and in audio/video segment `content`.
When several context task segments need the same image, audio clip or video as a reference,
mark that media shared; it does NOT need to overlap the task segments on the timeline.
```json
{
"id": "shared-audio-segment",
"start_frame": 0,
"end_frame": 120,
"content": {
"media_type": "audio",
"source_type": "input",
"file_path": "codex/voice/reference.wav",
"file_name": "reference.wav",
"shared_reference": true
}
}
```
Decision rules:
- Mark media shared only when the user says "make it shared/common media", "all context
segments reference this image/audio/video" or equivalent. Never auto-share just because
file names match.
- Task images are marked on their `content.images[]` items; several shared images are
allowed; they are ordered before each task's private images and common + private still
total at most 9 per task.
- Audio/video are marked on the media segment's `content`; at most one per audio/video
track; when switching, explicitly set the track's other segments to `false`.
- Shared audio is used from the start of the source file, at most 15 seconds; shared video
uses that source video as each task's reference video.
- Shared media is a cross-task reference, not timeline-audio reuse. Never set `audio_locked`
just because audio is shared; evaluate `audio_locked` separately, only when the user also
wants the audio kept as-is and driving the timeline.
- If the user wants one audio/video clip BOTH as a shared reference AND as normal timeline
output, express the two roles with a separate track or segment and say so in the delivery
summary.
- `speaker_reference` is the legacy audio-reference field, only for reading and migrating old
workflows; new or updated workflows always write `shared_reference`.
- With multiple unresolvable candidates, list the media/segment IDs, track names, file names,
durations or frame ranges and ask the user; never pick the first or the longest.
## Auto-orchestration decisions
- User-given shot/segment lengths win; otherwise derive from source durations or the
prompts' timing information, and state the derivation.
- Subject/partial video replacement mode with no user-specified lengths always splits task
segments into consecutive 10-second pieces, the last taking the remainder.
- "N seconds each" means consecutive `[0, N*fps)`, `[N*fps, 2N*fps)`, … — never add one extra
frame at the boundaries.
- Prompts go into their task segments; reference images into that task segment's `images`;
master video and audio into their own media tracks.
- For subject-replacement, MV or other master-audio-driven projects, after arranging the
timeline lock the single audio/video track and re-check it covers the target task range.
- Keep gaps between tracks by default; close gaps and shift later segments only when the
user asks for a tight arrangement.
- When adjusting existing segments keep their segment IDs; generate new UUIDs only for new
segments, tracks and image items.
## MultiTrack Project parameters
- `segment_start_number` is the 1-based first task segment index; execution converts with
`segment_start_number - 1`.
- `segment_count = -1` processes from the start index to the end; a non-negative value limits
how many task segments run this time.
- `project_save`: `new` or `override`. With `override` and count `-1`, execution deletes
saved segments after the start index before generating — say so clearly before changing it.
- `sampling_mode`: `single` or `dual`; `sampling_plan` must be a preset available on the
current node — never assume values from another workflow's environment.
- The editor resolution is the first-pass size; `dual` never scales the first pass down. The
second pass is `round(editor_size * upscale_by / 32) * 32` per dimension, Python-round to
nearest, ties to even — not a fixed ceiling. `upscale_by` is an independent Project
parameter, default `1.250`, three decimals, step `0.001`; old workflows keep their saved
value. Both upsample paths and the merged export use the upscaled, aligned size. Example:
`1344 x 768` at `1.250` gives a second pass of `1664 x 960`. If the user specifies a final
output size, check BOTH the editor size and the multiplier; the editor size alone is not
the dual-sampling final size.
- Single sampling / first-pass-only preview keeps the editor size; `upscale_by = 1.000`
upscales nothing but still allows a second pass; pure audio projects (`32 x 32`) do not
upscale. Old first-pass checkpoints and context latents that were shrunk by the old
back-calculation should be regenerated before continuing under the new rule.
- `1st_pass_only` only applies to explicit one-stage checkpoint flows; never enable it
automatically.
## The patch plan
The plan contains only actual changes. `node_id` may be omitted (topology locates the nodes);
`editor.resolution` is the complete DynamicCombo setting; `editor.track_data` is the complete
TRACK_DATA. Never put a whole node or workflow into the plan.
```json
{
"editor": {
"resolution": {
"resolution": "width x height (megapixels)",
"aspect_ratio": "16:9 (Widescreen)",
"megapixels": 0.9
},
"track_data": { "tracks": [], "total_length": 120, "frame_rate": 24 },
"format": "MiniMax"
},
"project": {
"project_save": "override",
"segment_start_number": 1,
"segment_count": -1,
"sampling_plan": "medium",
"sampling_mode": "single"
}
}
```
Applied with the node pack's own script:
```sh
python3 scripts/patch_workflow.py apply SOURCE.json --plan PLAN.json # dry-run
python3 scripts/patch_workflow.py apply SOURCE.json --plan PLAN.json --output RESULT.json --write
python3 scripts/patch_workflow.py inspect RESULT.json
```