583 lines
29 KiB
Markdown
Executable File
583 lines
29 KiB
Markdown
Executable File
# MultiTrack Data and Orchestration Rules (English reference)
|
||
|
||
English reference for the `easy multiTrackEditor` TRACK_DATA and the patch plan, translated
|
||
from the node pack's own skill doc (`skills/easy-media-multitrack-workflow/references/multitrack-schema.md`
|
||
in ComfyUI-Easy-Media, which is written in Chinese). Front-end types live in
|
||
`frontend/src/types/multitrack.ts`; execution logic lives in `nodes/basic.py` of the node pack.
|
||
Field names are preserved exactly as the schema spells them.
|
||
|
||
## Resolution (DynamicCombo)
|
||
|
||
`resolution` is a DynamicCombo widget of `easy multiTrackEditor`. It is NOT part of
|
||
TRACK_DATA. Changing it requires rebuilding the sub-widgets for the chosen mode and syncing
|
||
the workflow's existing `widgets_values_named`. Write it through `patch_workflow.py`'s
|
||
`editor.resolution` — never by guessing array positions.
|
||
|
||
| Need | `resolution` label | Required sub-fields |
|
||
| --- | --- | --- |
|
||
| Fixed preset output size | e.g. `1344 x 768 (16:9)` | optional `resize_method`, default `stretch` |
|
||
| Explicit custom size | `width x height (custom)` | `width`, `height`, optional `resize_method` |
|
||
| Follow first video's ratio | `width x height (auto)` | optional `resize_method` |
|
||
| Fixed short/long edge | `width x height (shortest)` / `width x height (longest)` | `resize_to_pixel`, optional `resize_method` |
|
||
| Ratio + total pixel budget | `width x height (megapixels)` | `aspect_ratio`, `megapixels` |
|
||
|
||
Selection order: exact width/height from the user → an existing fixed preset, else custom;
|
||
only orientation or aspect + quality/pixel budget given → megapixels; keep source video
|
||
ratio → auto; "short/long edge = N" → shortest/longest. If the user expresses no resolution
|
||
intent, keep the current setting. The model format's default size is only a reference and
|
||
never overrides an explicit user requirement.
|
||
|
||
### MP and ratio in natural language
|
||
|
||
Case-insensitive `mp`, `Mpx`, `megapixel(s)` (and Chinese "百万像素") mean the megapixels
|
||
unit. `0.9mp`, `1 MP`, `0.8 megapixels` select megapixels mode and the number is taken
|
||
verbatim. A ratio in the same sentence maps to the full DynamicCombo `aspect_ratio` value:
|
||
|
||
| User ratio | `aspect_ratio` |
|
||
| --- | --- |
|
||
| `1:1` | `1:1 (Square)` |
|
||
| `2:3` | `2:3 (Portrait Photo)` |
|
||
| `3:2` | `3:2 (Photo)` |
|
||
| `3:4` | `3:4 (Portrait Standard)` |
|
||
| `4:3` | `4:3 (Standard)` |
|
||
| `9:16` | `9:16 (Portrait Widescreen)` |
|
||
| `16:9` | `16:9 (Widescreen)` |
|
||
| `21:9` | `21:9 (Ultrawide)` |
|
||
|
||
Example: "generate a 16:9 0.9mp video" normalizes with no follow-up question to:
|
||
|
||
```json
|
||
{
|
||
"resolution": "width x height (megapixels)",
|
||
"aspect_ratio": "16:9 (Widescreen)",
|
||
"megapixels": 0.9
|
||
}
|
||
```
|
||
|
||
Never convert the MP number into width/height and never round it. MP only: if already in
|
||
megapixels mode keep its `aspect_ratio`; else pick `16:9`/`9:16` from explicit
|
||
landscape/portrait wording; still unclear → ask. Contradictory exact size, ratio and MP →
|
||
ask. Custom example:
|
||
|
||
```json
|
||
{
|
||
"resolution": "width x height (custom)",
|
||
"width": 1024,
|
||
"height": 576,
|
||
"resize_method": "crop"
|
||
}
|
||
```
|
||
|
||
Valid `resize_method`: `stretch`, `resize`, `pad`, `pad (white)`, `pad_edge`,
|
||
`pad_edge_pixel`, `crop`, `pillarbox_blur`. Prefer pad/resize to keep the whole picture,
|
||
crop to fill the canvas; never change an existing resize method the user did not mention.
|
||
|
||
## Time semantics
|
||
|
||
- `start_frame` is inclusive, `end_frame` exclusive; segment frames = `end_frame - start_frame`.
|
||
- Seconds → frames: `round(seconds * frame_rate)`; every segment has at least 1 frame.
|
||
- Segments in one track are sorted by `start_frame` and never overlap; gaps are allowed.
|
||
Different tracks of the same kind schedule independently.
|
||
- With segments present, `total_length` should be the maximum `end_frame` over all tracks.
|
||
With no segments the editor's current default length may stay.
|
||
- Changing FPS must re-map every segment boundary, `origin_start_frame` and task marker
|
||
proportionally — not just `frame_rate`.
|
||
|
||
## Top-level TRACK_DATA object
|
||
|
||
```json
|
||
{
|
||
"muted": false,
|
||
"volume_db": 0,
|
||
"task_markers": [],
|
||
"task_overview": false,
|
||
"tracks": [],
|
||
"total_length": 240,
|
||
"frame_rate": 24
|
||
}
|
||
```
|
||
|
||
Track `type` is one of `task`, `video`, `audio`, `subtitle`. Keep unknown fields for
|
||
forward compatibility.
|
||
|
||
## Common track and segment shape
|
||
|
||
```json
|
||
{
|
||
"id": "uuid",
|
||
"name": "Video 0",
|
||
"type": "video",
|
||
"color": "var(--primary)",
|
||
"muted": false,
|
||
"solo": false,
|
||
"volume_db": 0,
|
||
"locked": false,
|
||
"segments": [
|
||
{
|
||
"id": "uuid",
|
||
"start_frame": 0,
|
||
"end_frame": 120,
|
||
"color": "var(--primary)",
|
||
"content": {}
|
||
}
|
||
]
|
||
}
|
||
```
|
||
|
||
New tracks use the project's existing color tokens: task tracks `var(--multitrack-task-bg)`,
|
||
video tracks `var(--primary)`. Keep an existing track's own color.
|
||
|
||
## Task segment content
|
||
|
||
```json
|
||
{
|
||
"media_type": "none",
|
||
"task_mode": "ref",
|
||
"continuity_mode": "shot",
|
||
"ref_image_size": "match",
|
||
"images": [],
|
||
"user_prompt": "the user prompt",
|
||
"system_prompt": "",
|
||
"user_prompt_variant": "a",
|
||
"user_prompt_b": "",
|
||
"muted": false,
|
||
"volume_db": 0
|
||
}
|
||
```
|
||
|
||
- `task_mode`: `default`, `l2v`, `ref`, `edit`.
|
||
- RefMod workflows (identity driven by `MiniMaxH3RefModsLoader` + `MiniMaxH3RefModApply`
|
||
outside the timeline): author segments as `task_mode: "ref"` even with no per-segment
|
||
images — the payload then carries `generation_type: "r2v"`, TaskOutput TYPE becomes
|
||
`r2v`, and the matchLine/promptLine pair switches `minimaxH3ToVideo` to `reference`
|
||
mode, which pairs with the RefMod conditioning. `ref` rows with `images: []` are
|
||
legal (refmod pictures ride the conditioning, not the timeline); prompts stay
|
||
picture-token-free. Verified live by the user: the relay one-pass render works with
|
||
r2v and refmods "work better with ref2va (r2v)".
|
||
- `continuity_mode`: `shot` (standalone shot), `context` (normal context join) or
|
||
`context_drift` (context drift control); the legacy value `context_swap` is still read.
|
||
- `ref_image_size`: `match` or `max`.
|
||
- `images` holds at most 9 items; their order defines the picture-reference order.
|
||
- When editing an existing task keep its `system_prompt`, prompt variant, continuity and
|
||
reference-image policy unless the user asked to change them.
|
||
- If the current variant is `b`, edit `user_prompt_b`; otherwise edit `user_prompt`. Never
|
||
switch the variant unintentionally.
|
||
|
||
Task reference image item:
|
||
|
||
```json
|
||
{
|
||
"id": "uuid",
|
||
"source_type": "input",
|
||
"file_path": "codex/session-123/reference.png",
|
||
"file_name": "reference.png"
|
||
}
|
||
```
|
||
|
||
`source_type` is one of `input`, `output`, `local`, `url`, `slot`. The path field must
|
||
match the source: `input`/`output` use `file_path`, `local` uses `local_path`, `url` uses
|
||
`url`, `slot` uses `slot_name`. Never label a local absolute path as `input`.
|
||
|
||
When several context segments reuse the same reference image, an image item may set
|
||
`"shared_reference": true`; it becomes a common image that is automatically ordered before
|
||
each task's private images; common + private images still total at most 9 per task. The
|
||
current front-end `synchronizeSharedTaskImages` generates new UUIDs for missing common-image
|
||
metadata on later tasks, so repeatedly normalizing the same sparse raw data yields different
|
||
IDs; from this code alone the full React maximum-update-depth / widget-unmount trigger chain
|
||
cannot be confirmed. Until the target front-end is verified to load such data stably, store
|
||
an explicit private image item per task (`shared_reference: false`) with a fixed ID pointing
|
||
at the same uploaded file — no duplication or re-upload needed. Switching continuity mode
|
||
does not fix this.
|
||
|
||
## Context drift control mode
|
||
|
||
`context_drift` is the "context drift control" join: it carries the previous segment's scene
|
||
and motion context while letting the model replace subjects, people, objects or局部 regions
|
||
from a reference video. When the user supplies a video as a subject/person/object/clothing/face
|
||
replacement reference, orchestrate as follows:
|
||
|
||
- Set `content.continuity_mode` to `context_drift` on ALL task segments in scope — not only
|
||
the second and later ones. The first segment keeps the same mode too (it has no previous
|
||
generated context, but consistency matters for project metadata and re-generation).
|
||
- Put the reference video on a video track covering the target task segments that need motion
|
||
reference. If the project also has passages that do not depend on the dance/reference video
|
||
(e.g. a verse), keep one complete project — the full locked audio covers all tasks and the
|
||
video only covers its part. If the video is also needed across tasks as a model reference,
|
||
set `shared_reference: true` on the video segment's `content`.
|
||
- Set track-level `audio_locked: true` on that video track so the source video's timeline
|
||
drives generation, and explicitly set `audio_locked: false` on every OTHER video track. To
|
||
replace the original sound, lock a separate audio track, which then wins as the audio
|
||
source. When only reusing the video's own sound, confirm before locking that the video
|
||
actually contains a usable audio track and covers all target task segments.
|
||
- If the user pinned segment lengths or boundaries, the user wins. Otherwise split task
|
||
segments into consecutive 10-second pieces: each piece is `round(10 * frame_rate)` frames
|
||
and the last piece takes the reference video's remaining frames — do not pad, do not drop
|
||
the remainder, do not create an empty tail segment. The reference video itself can stay a
|
||
single video segment covering the whole range; it does not need to be duplicated per task
|
||
segment.
|
||
- If the user explicitly asked to drop/replace/not keep the reference video's original sound,
|
||
do not auto-lock that video track and follow the user's audio plan instead.
|
||
|
||
## Context segment prompts
|
||
|
||
Applies to tasks with `task_mode: "ref"` and `continuity_mode: "context"`;
|
||
`context_drift` (and legacy `context_swap`) do not use this section. Such a segment starts at
|
||
the previous segment's actually delivered tail (22 frames ≈ 0.92 s at 24 fps) whose framing
|
||
and camera motion may differ from what the previous prompt imagined — and those frames are
|
||
locked: text contradicting them tends to render as an overlay, not a replacement. When
|
||
writing or editing `user_prompt`:
|
||
|
||
- **Keep first, change later**: open with only "the camera keeps the direction and framing it
|
||
arrived with; the subject keeps the pose, motion and light it arrived with". Never describe
|
||
the previous segment's INTENDED ending as fact, and never open with a new composition,
|
||
character layout or scene.
|
||
- **Give the hold an action**: give the subject a breath, a weight shift or a gaze change so
|
||
the hold does not render as a freeze-frame.
|
||
- **Changes go later**: before about `00:02.000` no cut, no spoken line, no new key action;
|
||
write a required composition or scene change as an explicit cut.
|
||
- **Start actions mid-motion**: things already in progress at the seam start in motion —
|
||
never as if completing on frame one.
|
||
- **Self-contained**: the model never sees the previous prompt; fully define every `<Subject N>`,
|
||
state scene and light, and never say "the same one" or "as before" for identity or scene.
|
||
- **Images define identity only**: a reference image is not the first frame or a composition
|
||
anchor of `[Shot 1]`; context joins use neither image anchoring nor `[video continuation]`.
|
||
- **When more `context` segments follow, end in motion**: never write "ends", "arrives",
|
||
"settles", "centers", "slows down"; no cut and no new spoken line in the last second.
|
||
- **Need a clearly new composition**: use a `shot` segment with keyframes instead of `context`.
|
||
- **Timestamps**: timestamps in the prompt map onto the generated segment, whose start sits
|
||
about 0.92 s earlier than the saved segment; the node does not shift them automatically.
|
||
|
||
When `system_prompt` is empty, the task output node automatically appends the same rules to
|
||
the default MiniMax reference system prompt (entry rule: this segment is not the first and
|
||
uses `context`; closing rule: the next segment uses `context`). The rules reach the rewriting
|
||
LLM only through the `api` and `llm` prompt formats; `default` and `promptRelay` emit no
|
||
system prompt. A custom `system_prompt` disables the auto-append — write the rules yourself.
|
||
|
||
## Video and audio segment content
|
||
|
||
```json
|
||
{
|
||
"media_type": "video",
|
||
"source_type": "input",
|
||
"file_path": "codex/session-123/clip.mp4",
|
||
"file_name": "clip.mp4",
|
||
"duration": 5.0,
|
||
"muted": false,
|
||
"volume_db": 0,
|
||
"speed": 1
|
||
}
|
||
```
|
||
|
||
Audio sets `media_type` to `audio`. Get source durations with `ffprobe`; the timeline
|
||
duration follows the user's target, trimming and speed. If media must align with a task
|
||
segment, reuse the exact same `[start_frame, end_frame)` — never just similar seconds.
|
||
|
||
## Audio lock decisions
|
||
|
||
`audio_locked` is a track-level field on audio/video tracks. Locking an audio track keeps
|
||
its audio; locking a video track makes generation follow that video track's timeline and,
|
||
with no locked audio track, keeps the video's own sound. Both locks can coexist; the audio
|
||
track then wins as the `audio_lock` audio source while the video track still constrains the
|
||
picture timeline. It is different from the track's `locked` flag: `locked` only blocks
|
||
timeline editing and cannot substitute for `audio_locked`.
|
||
|
||
```json
|
||
{
|
||
"id": "music-track",
|
||
"name": "Master Song",
|
||
"type": "audio",
|
||
"audio_locked": true,
|
||
"muted": false,
|
||
"segments": [
|
||
{
|
||
"id": "song-segment",
|
||
"start_frame": 0,
|
||
"end_frame": 240,
|
||
"content": {
|
||
"media_type": "audio",
|
||
"source_type": "input",
|
||
"file_path": "codex/mv/song.wav",
|
||
"file_name": "song.wav"
|
||
}
|
||
}
|
||
]
|
||
}
|
||
```
|
||
|
||
A complete locked audio track can coexist with a partial locked video track. At runtime, when
|
||
the current task has no intersection with the video, the video is simply not passed in and
|
||
the master audio is still used — never split the project, extend the video, or stretch audio
|
||
just to satisfy validation. Validation allows a partial video lock only when the locked audio
|
||
continuously covers the selected tasks (multiple end-to-end audio segments are fine); without
|
||
a complete master audio it still checks the locked media's intersection with each task.
|
||
|
||
One TRACK_DATA may contain at most one `audio_locked: true` audio track and one
|
||
`audio_locked: true` video track. When switching the locked source of one type, explicitly set
|
||
the other tracks of that same type to `false`; never clear the other type's lock. Never write
|
||
`audio_locked` into a media segment's `content`; the legacy segment-level field only exists
|
||
for migration.
|
||
|
||
Decide in this order, from context:
|
||
|
||
1. The user supplies a video for subject/partial replacement and did not ask to remove or
|
||
replace its sound → lock that video track.
|
||
2. The user explicitly says "lock", "master audio", "keep the original sound", "don't
|
||
regenerate audio" → use the audio/video track they mean.
|
||
3. MV, lyric video, beat-sync video, dance video, or visuals generated to a full song's beat
|
||
and length → lock the song/score master track.
|
||
4. Narration, podcast or dialogue is the only basis for video length and shot rhythm → that
|
||
narration master track may be locked.
|
||
5. Ordinary background music, ambience, SFX, character reference audio or temporary scratch
|
||
audio is NOT auto-treated as locked, unless the user's wording makes it the generation
|
||
driver.
|
||
6. One locked media track already exists and the user did not ask to change it → prefer
|
||
keeping it; still check it intersects the task range being generated.
|
||
|
||
With multiple audio tracks, disambiguate first by name, file name, source duration, track
|
||
content and the user's description — e.g. `Master Song` or 完整歌曲 (full song) beats `SFX` or
|
||
ambience. If two or more plausible candidates remain, STOP and ask the user, listing each
|
||
candidate's track ID, name, file name and time range. Never decide by track order, first
|
||
audio track or longest duration.
|
||
|
||
Before locking also check:
|
||
|
||
- The locked track contains at least one media segment matching its type: audio track →
|
||
`media_type: "audio"`, video track → `media_type: "video"`; when locking a video's own
|
||
sound, confirm the source video really has an audio track.
|
||
- The locked media segments intersect every target task segment's `[start_frame, end_frame)`;
|
||
if not, adjust the timeline or ask first.
|
||
- As master audio it normally stays `muted: false`. Never touch unrelated solo, volume or
|
||
edit-lock state just "because locking".
|
||
- If the context clearly wants no original audio kept, ensure every audio/video track has
|
||
`audio_locked: false`.
|
||
|
||
## Shared media
|
||
|
||
`shared_reference` is a boolean on task image items and in audio/video segment `content`.
|
||
When several context task segments need the same image, audio clip or video as a reference,
|
||
mark that media shared; it does NOT need to overlap the task segments on the timeline.
|
||
|
||
```json
|
||
{
|
||
"id": "shared-audio-segment",
|
||
"start_frame": 0,
|
||
"end_frame": 120,
|
||
"content": {
|
||
"media_type": "audio",
|
||
"source_type": "input",
|
||
"file_path": "codex/voice/reference.wav",
|
||
"file_name": "reference.wav",
|
||
"shared_reference": true
|
||
}
|
||
}
|
||
```
|
||
|
||
Decision rules:
|
||
|
||
- Mark media shared only when the user says "make it shared/common media", "all context
|
||
segments reference this image/audio/video" or equivalent. Never auto-share just because
|
||
file names match.
|
||
- Task images are marked on their `content.images[]` items; several shared images are
|
||
allowed; they are ordered before each task's private images and common + private still
|
||
total at most 9 per task.
|
||
- Audio/video are marked on the media segment's `content`; at most one per audio/video
|
||
track; when switching, explicitly set the track's other segments to `false`.
|
||
- Shared audio is used from the start of the source file, at most 15 seconds; shared video
|
||
uses that source video as each task's reference video.
|
||
- Shared media is a cross-task reference, not timeline-audio reuse. Never set `audio_locked`
|
||
just because audio is shared; evaluate `audio_locked` separately, only when the user also
|
||
wants the audio kept as-is and driving the timeline.
|
||
- If the user wants one audio/video clip BOTH as a shared reference AND as normal timeline
|
||
output, express the two roles with a separate track or segment and say so in the delivery
|
||
summary.
|
||
- `speaker_reference` is the legacy audio-reference field, only for reading and migrating old
|
||
workflows; new or updated workflows always write `shared_reference`.
|
||
- With multiple unresolvable candidates, list the media/segment IDs, track names, file names,
|
||
durations or frame ranges and ask the user; never pick the first or the longest.
|
||
|
||
## Auto-orchestration decisions
|
||
|
||
- User-given shot/segment lengths win; otherwise derive from source durations or the
|
||
prompts' timing information, and state the derivation.
|
||
- Subject/partial video replacement mode with no user-specified lengths always splits task
|
||
segments into consecutive 10-second pieces, the last taking the remainder.
|
||
- "N seconds each" means consecutive `[0, N*fps)`, `[N*fps, 2N*fps)`, … — never add one extra
|
||
frame at the boundaries.
|
||
- Prompts go into their task segments; reference images into that task segment's `images`;
|
||
master video and audio into their own media tracks.
|
||
- For subject-replacement, MV or other master-audio-driven projects, after arranging the
|
||
timeline lock the single audio/video track and re-check it covers the target task range.
|
||
- Keep gaps between tracks by default; close gaps and shift later segments only when the
|
||
user asks for a tight arrangement.
|
||
- When adjusting existing segments keep their segment IDs; generate new UUIDs only for new
|
||
segments, tracks and image items.
|
||
|
||
## MultiTrack Project parameters
|
||
|
||
- `segment_start_number` is the 1-based first task segment index; execution converts with
|
||
`segment_start_number - 1`.
|
||
- `segment_count = -1` processes from the start index to the end; a non-negative value limits
|
||
how many task segments run this time.
|
||
- `project_save`: `new` or `override`. With `override` and count `-1`, execution deletes
|
||
saved segments after the start index before generating — say so clearly before changing it.
|
||
- `sampling_mode`: `single` or `dual`; `sampling_plan` must be a preset available on the
|
||
current node — never assume values from another workflow's environment.
|
||
- The editor resolution is the first-pass size; `dual` never scales the first pass down. The
|
||
second pass is `round(editor_size * upscale_by / 32) * 32` per dimension, Python-round to
|
||
nearest, ties to even — not a fixed ceiling. `upscale_by` is an independent Project
|
||
parameter, default `1.250`, three decimals, step `0.001`; old workflows keep their saved
|
||
value. Both upsample paths and the merged export use the upscaled, aligned size. Example:
|
||
`1344 x 768` at `1.250` gives a second pass of `1664 x 960`. If the user specifies a final
|
||
output size, check BOTH the editor size and the multiplier; the editor size alone is not
|
||
the dual-sampling final size.
|
||
- Single sampling / first-pass-only preview keeps the editor size; `upscale_by = 1.000`
|
||
upscales nothing but still allows a second pass; pure audio projects (`32 x 32`) do not
|
||
upscale. Old first-pass checkpoints and context latents that were shrunk by the old
|
||
back-calculation should be regenerated before continuing under the new rule.
|
||
- `1st_pass_only` only applies to explicit one-stage checkpoint flows; never enable it
|
||
automatically.
|
||
|
||
## The patch plan
|
||
|
||
The plan contains only actual changes. `node_id` may be omitted (topology locates the nodes);
|
||
`editor.resolution` is the complete DynamicCombo setting; `editor.track_data` is the complete
|
||
TRACK_DATA. Never put a whole node or workflow into the plan.
|
||
|
||
```json
|
||
{
|
||
"editor": {
|
||
"resolution": {
|
||
"resolution": "width x height (megapixels)",
|
||
"aspect_ratio": "16:9 (Widescreen)",
|
||
"megapixels": 0.9
|
||
},
|
||
"track_data": { "tracks": [], "total_length": 120, "frame_rate": 24 },
|
||
"format": "MiniMax"
|
||
},
|
||
"project": {
|
||
"project_save": "override",
|
||
"segment_start_number": 1,
|
||
"segment_count": -1,
|
||
"sampling_plan": "medium",
|
||
"sampling_mode": "single"
|
||
}
|
||
}
|
||
```
|
||
|
||
Applied with the node pack's own script:
|
||
|
||
```sh
|
||
python3 scripts/patch_workflow.py apply SOURCE.json --plan PLAN.json # dry-run
|
||
python3 scripts/patch_workflow.py apply SOURCE.json --plan PLAN.json --output RESULT.json --write
|
||
python3 scripts/patch_workflow.py inspect RESULT.json
|
||
```
|
||
|
||
---
|
||
|
||
## prompt_override — plain text form: `scene1 | scene2 | scene3` (the simple default)
|
||
|
||
The `prompt_override` input does not require any JSON. Paste plain text — one scene per
|
||
pipe-separated chunk:
|
||
|
||
```
|
||
<scene 1 text> | <scene 2 text> | <scene 3 text>
|
||
```
|
||
|
||
The editor's execute (`nodes/basic.py`): a non-JSON `prompt_override` with a value goes to
|
||
`build_multitrack_data_from_prompt_override` → `parse_override_segments`
|
||
(`utils/prompt_override.py`), which splits on `|`, strips each part and drops empties —
|
||
one task segment per part, a trailing `|` is a no-op. Proven by executing the pack's code
|
||
(sandbox `scripts/prove_plain_text_override.py`):
|
||
|
||
- Frames: the editor's existing `total_length` (its track_data / length input) is divided
|
||
EVENLY across the parts — there are no per-segment durations in this form. 720 total,
|
||
3 parts → [0-240] [240-480] [480-720].
|
||
- No pipe at all = ONE segment spanning the whole timeline (the "one giant clip" trap).
|
||
- Task type: parts default to `flf` → task_mode `default` → TaskOutput TYPE `t2v`.
|
||
- Optional per-part extras (inside the part text):
|
||
- `[s-e]` frame ranges, seconds form `[1.5s-3s]` (4-frame quanta), optional type suffix
|
||
`[0-240,r2v]` → that segment becomes task_mode `ref` / TYPE `r2v` (other types:
|
||
`flf`, `fmlf`, `ref`, `l2v`, `v2v`, `i2v`, `rv2v`, `vi2v`). With explicit ranges the
|
||
timeline length becomes the last end + 1. The marker sits at the END of the segment,
|
||
is stripped from the text before it reaches the task, and newlines INSIDE a segment
|
||
survive — only `|` splits. Worked example:
|
||
`examples/prompt_override_r2v_refmod.txt` — three six-section r2v scene prompts
|
||
(each carrying the Sam refmod verbatim in `subject_definitions`, no `<Picture N>`
|
||
because the story has zero images) with `[0-240,r2v]` / `[240-480,r2v]` /
|
||
`[480-720,r2v]` markers; proven through the pack's parser: 3 tasks TYPE `r2v`,
|
||
frames [0-240]/[240-480]/[480-720], task-duration sum 720, text byte-identical to
|
||
the plan shots.
|
||
- `@image1` / `@audio1` / `@video1` attach the editor's matching media input slots
|
||
positionally. The `<Picture N>` bracket form is NOT parsed in this path — it stays
|
||
literal text; only the JSON payload understands it.
|
||
- Continuity: parts become independent segments; there is no `continuity_mode` in the
|
||
text form (context joins are a JSON/TRACK_DATA feature).
|
||
|
||
This is the gate's DEFAULT paste form: the merge writes `prompt_override.txt` for every
|
||
zero-media story — scenes joined with `|`, and it appends the `[start-end]` /
|
||
`[start-end,type]` markers itself whenever the rows have unequal durations (the
|
||
markerless form splits the editor's `total_length` evenly) or a scene's task type differs
|
||
from the text default (flf → t2v, e.g. `[0-240,r2v]` for task_mode ref). Use the JSON
|
||
payload below only for what plain text cannot carry: per-segment continuity modes, track
|
||
locks or `<Picture N>` media tokens (stories with media — the text path attaches media
|
||
via `@imageN`/`@audioN`/`@videoN` only, and the gate validates both forms against the
|
||
approved guide).
|
||
|
||
---
|
||
|
||
## prompt_override — JSON payload form (optional: per-segment durations, modes, locks)
|
||
|
||
The `easy multiTrackEditor` node has an optional `prompt_override` input. When the editor's
|
||
`format` is `MiniMax` and the pasted value is a `minimax_prompt_override` JSON payload, the
|
||
editor rebuilds its whole task track from that payload: one task segment per `prompts`
|
||
entry, laid back-to-back on a cursor, each spanning `round(seconds × frame_rate)` frames
|
||
derived from its duration. **The payload has no frame fields and needs none** — never put
|
||
`start_frame`/`end_frame` in it. `total_length` becomes the sum of the durations, so a
|
||
3-prompt payload at the default 10s each renders as THREE chained ~10-second clips.
|
||
|
||
**The parser requires exactly two keys** (`parse_minimax_prompt_override`,
|
||
`utils/prompt_override.py`: it checks `type` and that `prompts` is a list — nothing else):
|
||
|
||
```json
|
||
{"type":"minimax_prompt_override","prompts":["prompt for segment one ...","prompt for segment two ..."]}
|
||
```
|
||
|
||
Every other field is OPTIONAL with a code default — write a key only when the story needs
|
||
a non-default value:
|
||
|
||
| key | default | meaning |
|
||
| --- | --- | --- |
|
||
| `duration` | `"10"` | SECONDS per clip (never frames), 2–15 enforced at parse time; single value or comma list with tail reuse |
|
||
| `generation_type` | `"r2v"` | per clip, same single/comma rule: `r2v` (task_mode ref) / `l2v` / `t2v`,`i2v`,`v2v` (default-mode tasks). TaskOutput's TYPE echoes this string verbatim |
|
||
| `continuity_mode` | `"shot"` | per clip: `shot` / `context` / `context_drift` (legacy `context_swap` is normalized to `context_drift`) |
|
||
| `system_prompt` | `""` | written into every clip's content when non-empty |
|
||
| `video_track_lock` | `0` | 1-based index of the video track to lock (0 = none) |
|
||
| `audio_track_lock` | `0` | 1-based index of the audio track to lock (0 = none) |
|
||
|
||
- `version` — the pack's own `easy minimaxPromptOverride` builder node writes
|
||
`"version": 1`, but **nothing in the pack reads it**; the parser never touches it.
|
||
Do not emit it.
|
||
- The code default for `generation_type` is **`r2v`** (reference mode) — a bare minimal
|
||
payload already renders as r2v/reference tasks with no key at all, which is why the
|
||
gate's refmod guidance needs no explicit generation_type.
|
||
|
||
Fallback warning (unchanged): **any other JSON pasted here is not rejected — it falls back
|
||
to the generic text parser that splits on `|`.** A JSON blob contains no `|`, so it becomes
|
||
ONE task segment spanning the whole timeline with the JSON dump as its prompt: the classic
|
||
"rendered the whole story as a single clip" failure.
|
||
|
||
- `prompts` — one entry per clip, in timeline order. The prompt text is the clip's full
|
||
generation prompt (same standalone composition rules as task segments: no cross-segment
|
||
references, context joins keep-then-change, only the last clip may come to rest).
|
||
- Media attaches ONLY through tokens in the prompt text: `<Picture N>`, `<Audio N>`,
|
||
`<Video N>` (the `@imageN`/`@audioN`/`@videoN` forms are accepted and rewritten, but this
|
||
gate standardizes on the bracket form). N is the 1-based position of the item in the
|
||
editor's matching `image` / `audio` / `video` input list — wire the media inventory into
|
||
those inputs in inventory order. A prompt with no token renders with no reference media.
|
||
The runtime renumbers tokens compactly for display, but slots are wired from the raw N.
|
||
|
||
The paste replaces the editor's existing task track (and derives its audio/video tracks
|
||
from the tokens), so the payload is self-contained: prompts, durations, modes and media
|
||
references all travel inside it. Edit task modes (`edit`) and full extra tracks are beyond
|
||
this payload — use the TRACK_DATA patch plan for those.
|