Compare commits
17 Commits
main
...
easy-media
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
55e4b27385 | ||
|
|
3587e676fa | ||
|
|
bbd46a4764 | ||
|
|
e3e66dd628 | ||
|
|
1473abcbd2 | ||
|
|
20143da60e | ||
|
|
9b197ae024 | ||
|
|
b45e6b72e7 | ||
|
|
7b6a2da472 | ||
|
|
8a9c79f06e | ||
|
|
eb090f765d | ||
|
|
5ab7096cfc | ||
|
|
fd874adbd3 | ||
|
|
8434bb4645 | ||
|
|
56b9d3b9e2 | ||
|
|
e02e338bef | ||
|
|
ea5c5a71b4 |
100
easy-media-track-gate/README.md
Normal file
100
easy-media-track-gate/README.md
Normal file
@ -0,0 +1,100 @@
|
||||
# Easy-Media Track Gate
|
||||
|
||||
A portable, self-contained agent skill that turns a video idea into a **validated
|
||||
`prompt_override` paste** for the ComfyUI-Easy-Media nodes (`easy multiTrackEditor`),
|
||||
built with the same outline → approval → one-segment-per-turn → merge → gate flow as
|
||||
`h3-plan-gate`, adapted to the Easy-Media multitrack schema.
|
||||
|
||||
The deliverable is a **plain-text `prompt_override` paste** (`prompt_override.txt`: the
|
||||
scene prompts joined with `|` on one line, `[start-end(,type)]` markers where scene
|
||||
lengths differ or a scene is not t2v) for the `easy multiTrackEditor` node's
|
||||
`prompt_override` input (format `MiniMax`) — the editor splits on pipes and renders one
|
||||
clip per scene, no JSON needed. The segment files the agent writes on the way there are
|
||||
plain text too (`segment_NN.txt`: the scene prompt verbatim + one `# handoff:` line; the
|
||||
approved outline row supplies id, frames, modes and images). JSON appears only where plain
|
||||
text cannot reach: `prompt_override.json` (a `minimax_prompt_override` payload — written
|
||||
automatically for stories with media, or with `--json` for continuity modes / track
|
||||
locks) and `plan.json` (`--plan`, the whole-workflow alternative applied by the node
|
||||
pack's own `skills/easy-media-multitrack-workflow/scripts/patch_workflow.py apply`).
|
||||
|
||||
## What the gate enforces
|
||||
|
||||
| Layer | Check |
|
||||
| --- | --- |
|
||||
| Schema | Mirrors every check of the node pack's `patch_workflow.py` at apply time: track/segment ids, sorting, overlap, frame ranges, media `source_type`↔field matching, ≤ 9 images per task, continuity enum, one locked audio + one locked video max, locked-media coverage of task ranges, shared-reference rules, DynamicCombo resolution tokens, project fields — so a plan that passes here cannot be rejected there for schema reasons |
|
||||
| Outline pins | Each segment's id, start/end frames (`round(seconds × fps)`), `task_mode`, `continuity_mode` and reference-image set come FROM the approved guide row when a plain-text fragment is loaded — nothing is retyped, so they cannot drift; JSON fragments are diffed against the row at the segment turn AND again at the merge |
|
||||
| Verbatim media contract | Media paths are typed only in the outline's `media_inventory`: plain-text fragments cannot carry paths at all, and locked tracks are built straight from the outline (a retyped JSON-fragment path fails with a closest-match hint) |
|
||||
| Context prompt rules | Self-contained prompts in every mode (no "same as before"); context/context_drift joins must open by keeping the delivered tail and must not open with a new composition; a join followed by another join must not end at rest |
|
||||
| Drift discipline | `context_drift` is all-or-nothing across the task range (first segment included), with the consecutive 10-second split unless the user pinned lengths |
|
||||
| Paste payload | `prompt_override` re-checked against the guide in BOTH forms: plain text (scene count/order, `[s-e,type]` markers vs the approved row frames and task types, `@imageN/@audioN/@videoN` refs vs the inventory, `<Picture N>` and stray `@` rejected) and JSON payload (durations, generation/continuity modes, exact `<Picture N>` sets, track locks) |
|
||||
| Fragment hygiene | Plain-text segment files reject `|` (would split the paste), hand-written `[start-end]` markers (the merge appends them), `@` (the editor strips it) and, on zero-media stories, `<Picture N>` tokens (JSON-only); the `# handoff:` line must describe the real final frame (15+ words) |
|
||||
|
||||
## Folder
|
||||
|
||||
```text
|
||||
easy-media-track-gate/
|
||||
├── SKILL.md agent procedure (the entry point)
|
||||
├── README.md this file
|
||||
├── verify_track_rules.py the gate: outline / segment / trackdata / plan / override / suite
|
||||
├── build_track_plan.py merge fragments → the paste (prompt_override.txt, or .json for
|
||||
│ media stories); --plan adds plan.json + track_data.json
|
||||
├── references/
|
||||
│ └── track-schema.md English translation of the node pack's Chinese
|
||||
│ multitrack-schema.md (all field names preserved)
|
||||
└── examples/ worked 3-segment r2v example: outline_sam_park.json, the
|
||||
plain-text segment_01–03.txt fragments, and the paste they
|
||||
merge into (prompt_override_r2v_refmod.txt); plus a media-story
|
||||
guide (outline.json), its JSON payload (prompt_override.json)
|
||||
and the patch-plan shape (plan.json)
|
||||
```
|
||||
|
||||
Standard library only; Windows-first (`py -3`), runs anywhere Python 3.8+ lives.
|
||||
|
||||
## Install (portable skills folder)
|
||||
|
||||
Copy the whole folder next to your other skills:
|
||||
|
||||
```text
|
||||
%USERPROFILE%\.agent\skills\easy-media-track-gate\
|
||||
```
|
||||
|
||||
`SKILL.md` must sit directly in that folder (not nested one level deeper). Start a fresh
|
||||
agent task after installing so the skill list refreshes. The node pack's own
|
||||
`easy-media-multitrack-workflow` skill stays installed independently — this gate produces
|
||||
its plan input and defers execution to it.
|
||||
|
||||
## Quick start
|
||||
|
||||
```sh
|
||||
:: smoke test
|
||||
py -3 "%USERPROFILE%\.agent\skills\easy-media-track-gate\verify_track_rules.py"
|
||||
|
||||
:: validate a guide / one plain-text segment
|
||||
py -3 "%USERPROFILE%\.agent\skills\easy-media-track-gate\verify_track_rules.py" outline outline.json
|
||||
py -3 "%USERPROFILE%\.agent\skills\easy-media-track-gate\verify_track_rules.py" segment segment_01.txt --index 1 --outline outline.json
|
||||
py -3 "%USERPROFILE%\.agent\skills\easy-media-track-gate\verify_track_rules.py" segment segment_02.txt --index 2 --prev segment_01.txt --outline outline.json
|
||||
|
||||
:: merge to the deliverable (writes ONLY the paste: prompt_override.txt for zero-media
|
||||
:: stories, prompt_override.json when the story has media; --json / --plan add the rest)
|
||||
py -3 "%USERPROFILE%\.agent\skills\easy-media-track-gate\build_track_plan.py" outline.json segment_01.txt segment_02.txt segment_03.txt
|
||||
|
||||
:: validate the paste
|
||||
py -3 "%USERPROFILE%\.agent\skills\easy-media-track-gate\verify_track_rules.py" override prompt_override.txt --outline outline.json
|
||||
|
||||
:: optional: patch plan for the node pack's own script (dry-run, then write)
|
||||
py -3 "%USERPROFILE%\.agent\skills\easy-media-track-gate\build_track_plan.py" outline.json segment_01.txt segment_02.txt segment_03.txt --plan plan.json
|
||||
py -3 <ComfyUI>\custom_nodes\ComfyUI-Easy-Media\skills\easy-media-multitrack-workflow\scripts\patch_workflow.py apply WORKFLOW.json --plan plan.json --output RESULT.json --write
|
||||
```
|
||||
|
||||
## Provenance notes
|
||||
|
||||
- The schema reference is a faithful English translation of
|
||||
`ComfyUI-Easy-Media/skills/easy-media-multitrack-workflow/references/multitrack-schema.md`
|
||||
(which is written in Chinese); field names, enums and limits were additionally
|
||||
cross-checked against the node pack's `patch_workflow.py` source, and the generated
|
||||
example plan was dry-run-applied to the bundled v1.3.0 blank template successfully.
|
||||
- Upstream leaves prompt quality, outline discipline and id/timing pins to discipline;
|
||||
this gate makes them machine-checked. Everything upstream checks mechanically is
|
||||
checked here with the same rules.
|
||||
- Upload (`/upload/image`) and queue (`submit_workflow.py`) remain the node pack's own
|
||||
flows and are only used on the user's explicit request.
|
||||
447
easy-media-track-gate/SKILL.md
Executable file
447
easy-media-track-gate/SKILL.md
Executable file
@ -0,0 +1,447 @@
|
||||
---
|
||||
name: easy-media-track-gate
|
||||
description: Turn a video idea into a validated MultiTrack plan for ComfyUI-Easy-Media
|
||||
(the `easy multiTrackEditor` node). Use when the user wants to plan task segments/shots
|
||||
on the multitrack timeline, write or extend a MultiTrack timeline, arrange prompts,
|
||||
reference images, master audio or reference-video locks, regenerate one task segment,
|
||||
or produce the `prompt_override` paste for the multitrack editor node — plain text
|
||||
`scene1|scene2|...` by default — plus the optional patch plan JSON that
|
||||
`patch_workflow.py apply` consumes. The scripts and
|
||||
references live next to this SKILL.md (the skill's install folder); run them from the
|
||||
user's working folder, where all generated files are written.
|
||||
---
|
||||
|
||||
# Easy-Media Track Gate — agent procedure
|
||||
|
||||
Goal: produce the `prompt_override` paste for the `easy multiTrackEditor` node in five
|
||||
moves: the user gives a video idea and a segment count (or an existing workflow to edit) →
|
||||
you build a timeline guide (`outline.json` — the ONLY JSON the default flow ever writes)
|
||||
and show it for **approval** → with the approved guide you compose the task segments one
|
||||
at a time (about 10 seconds each) as PLAIN TEXT files (`segment_01.txt`, …): each file is
|
||||
ONE scene's complete prompt plus a one-line `# handoff:` note — no ids, no frames, no
|
||||
media paths, no JSON, because the approved outline row supplies all of that → the scripts
|
||||
verify timing, continuity and prompt craft across the whole chain →
|
||||
`build_track_plan.py` merges the fragments and writes the deliverable. The deliverable is
|
||||
`prompt_override.txt` — PLAIN TEXT, the scene prompts joined with `|` on one line (the
|
||||
editor splits on pipes; no JSON), which the user pastes into the multitrack editor node's
|
||||
`prompt_override` input (format `MiniMax`) — the editor then renders one clip per scene,
|
||||
chained back-to-back. The merge adds `[start-end]` / `[start-end,type]` markers per scene
|
||||
when the story needs them (unequal lengths, or a scene that is not the plain-text default
|
||||
t2v — e.g. r2v). `prompt_override.json` (a `minimax_prompt_override` payload) exists only
|
||||
for what plain text cannot carry: `<Picture N>`/`<Audio N>`/`<Video N>` media tokens
|
||||
(stories with media — the merge writes it automatically then), per-segment continuity
|
||||
modes or track locks (`--json`). The patch plan (`plan.json` via `--plan` for the node
|
||||
pack's `patch_workflow.py`) is the optional whole-workflow alternative. Never deliver a
|
||||
paste that did not pass validation, and never compose a segment from a guide the user has
|
||||
not approved. Never hand the user any other JSON as the paste: a non-
|
||||
`minimax_prompt_override` value in `prompt_override` falls back to the editor's
|
||||
`|`-separated text parser and renders the whole story as ONE clip over the entire
|
||||
timeline — and never put `|` inside a scene prompt for the same reason.
|
||||
|
||||
## Paths and Python (read first — these exact forms prevent most errors)
|
||||
|
||||
- **SKILL_DIR** is the folder this SKILL.md was loaded from — the skill's install folder
|
||||
(default on Windows: `%userprofile%\.agent\skills\easy-media-track-gate`; macOS/Linux:
|
||||
`~/.agent/skills/easy-media-track-gate`). The scripts, the schema reference and the worked
|
||||
example all live in SKILL_DIR, not in the working folder. Every command below uses the
|
||||
Windows default install path; substitute your real SKILL_DIR if it differs.
|
||||
- Work in the user's **working folder** (the folder open in the editor). Write generated
|
||||
files there: `outline.json`, `segment_01.txt`, `segment_02.txt`, …, and the merge's
|
||||
`prompt_override.txt` — that is the whole default set. `prompt_override.json` appears
|
||||
for stories with media (or with `--json`), `plan.json`/`track_data.json` only with
|
||||
`--plan`. Never write generated files into SKILL_DIR.
|
||||
- Python command: Windows `py -3` (or `python`); macOS/Linux `python3`. If the working
|
||||
folder has a `.venv`, its interpreter works too (`.venv\Scripts\python.exe`). The scripts
|
||||
need only the Python standard library — nothing to install.
|
||||
- Start every task by running the smoke test under "Before you start". If it fails because
|
||||
SKILL_DIR is missing files, stop and tell the user the skill folder is incomplete.
|
||||
- **Never read `verify_track_rules.py`** — run it and work from its printed output. Cline's
|
||||
read_file is capped by hard-coded budgets that do NOT grow with the model's context
|
||||
window: about 48,000 characters per read, 2,000 lines per read. The ~110 KB validator
|
||||
arrives cut no matter how large the context window is. A truncation notice is Cline's
|
||||
read cap, not repo corruption — do not "fix" truncated files. `build_track_plan.py`
|
||||
(~15 KB) and `references/track-schema.md` (~26 KB) are safe to read whole.
|
||||
- **Never read back `prompt_override.txt`, `prompt_override.json`, `plan.json` or
|
||||
`track_data.json` once real segments exist** — a 10-segment plan passes the read caps
|
||||
today and exceeds them tomorrow. Validate with the `segment` / `override` commands (the
|
||||
`override` command takes BOTH the `.txt` paste and the `.json` payload) and trust the
|
||||
one-line OK summary; the shipped `examples/segment_01.txt`–`segment_03.txt` (the worked
|
||||
plain-text fragments), `examples/outline_sam_park.json` (their guide),
|
||||
`examples/prompt_override_r2v_refmod.txt` (the paste those fragments merge into),
|
||||
`examples/prompt_override.json` (2.5 KB) and `examples/plan.json` (6.8 KB) are safe to
|
||||
read if you need to see a shape.
|
||||
- Cline's editor rejects writes over 6,000 characters ("Editor input too large"). A segment
|
||||
fragment with a rich prompt often does. NEVER write part 2 with write_to_file — it
|
||||
replaces the whole file and part 1 is lost. Split instead: write the first part ending
|
||||
with the bare marker line `@@CONTINUE@@`, then replace_in_file with SEARCH `@@CONTINUE@@`
|
||||
and REPLACE the remaining content. The finished file must not contain the marker.
|
||||
- Cline also cuts command output at 8,000 characters. The embedded suite prints well under
|
||||
1 KB when it passes; a failing validation prints one actionable line. If output ever
|
||||
arrives truncated, re-run the single-file command (`segment`, `outline`, `plan`) — its
|
||||
output is tiny.
|
||||
|
||||
## Before you start (smoke test)
|
||||
|
||||
```sh
|
||||
py -3 "%USERPROFILE%\.agent\skills\easy-media-track-gate\verify_track_rules.py"
|
||||
py -3 "%USERPROFILE%\.agent\skills\easy-media-track-gate\build_track_plan.py" --check "%USERPROFILE%\.agent\skills\easy-media-track-gate\examples\plan.json"
|
||||
py -3 "%USERPROFILE%\.agent\skills\easy-media-track-gate\verify_track_rules.py" segment "%USERPROFILE%\.agent\skills\easy-media-track-gate\examples\segment_02.txt" --index 2 --outline "%USERPROFILE%\.agent\skills\easy-media-track-gate\examples\outline_sam_park.json" --prev "%USERPROFILE%\.agent\skills\easy-media-track-gate\examples\segment_01.txt"
|
||||
py -3 "%USERPROFILE%\.agent\skills\easy-media-track-gate\verify_track_rules.py" override "%USERPROFILE%\.agent\skills\easy-media-track-gate\examples\prompt_override_r2v_refmod.txt" --outline "%USERPROFILE%\.agent\skills\easy-media-track-gate\examples\outline_sam_park.json"
|
||||
py -3 "%USERPROFILE%\.agent\skills\easy-media-track-gate\verify_track_rules.py" override "%USERPROFILE%\.agent\skills\easy-media-track-gate\examples\prompt_override.json" --outline "%USERPROFILE%\.agent\skills\easy-media-track-gate\examples\outline.json"
|
||||
```
|
||||
|
||||
Expect the suite to print `SUITE PASS (57/57 cases)`, the `--check` to print `PLAN OK`
|
||||
with `total_length 720 frames = 30.00s`, the `segment` check to print `SEGMENT 2/3 OK
|
||||
(plain-text fragment): butterfly_chase_miss frames [240, 480)`, the first `override` to
|
||||
print `OVERRIDE OK (plain text): 3 scenes split on '|' with [start-end,type] markers,
|
||||
total 30.00s at 24 fps` and the second `OVERRIDE OK (JSON payload): 3 prompts, duration
|
||||
10 (default), total 30.00s`.
|
||||
|
||||
## Step 0 — Media paths come first (never invent one)
|
||||
|
||||
Every `file_path` in the plan must be a real path on the user's ComfyUI instance. Before
|
||||
writing the outline, establish the media inventory one of two ways:
|
||||
|
||||
- The user names the exact input paths (they know their `input/` subfolders). Use their
|
||||
strings verbatim — they become the pinned inventory.
|
||||
- The user explicitly asks you to upload → upload first and record the returned relative
|
||||
path (`subfolder/name`, `source_type: "input"`) for every file. Upload uses the node
|
||||
pack's own API (`POST /upload/image` with a `subfolder` like `codex/<project>`); never
|
||||
write a local absolute path into the plan.
|
||||
- If there is no real path and no upload request, stop and ask. A plan is still deliverable
|
||||
with obviously-fake placeholder paths only if you say plainly it cannot run until real
|
||||
media replaces them — never dress a placeholder up as a real path.
|
||||
|
||||
Upload and execution are external state changes: only upload/queue when the user explicitly
|
||||
asked (Step 5).
|
||||
|
||||
## Step 1 — Build the timeline guide (outline)
|
||||
|
||||
Read SKILL_DIR's `references/track-schema.md` once (safe whole-file read; it is the English
|
||||
translation of the node pack's Chinese schema doc). Then write `outline.json` — the guide
|
||||
for the whole timeline, and the only JSON the default flow ever writes — in exactly this
|
||||
shape (the shipped `examples/outline.json` is the worked model). Everything downstream is
|
||||
derived from it: frames, task types, continuity modes, images and ids all come from the
|
||||
approved rows, so the outline is also the single place media paths and resolution tokens
|
||||
are ever typed.
|
||||
|
||||
```json
|
||||
{
|
||||
"title": "short_snake_case_name",
|
||||
"lengths_user_pinned": false,
|
||||
"editor": {
|
||||
"format": "MiniMax",
|
||||
"resolution": { "resolution": "width x height (megapixels)",
|
||||
"aspect_ratio": "16:9 (Widescreen)", "megapixels": 0.8 }
|
||||
},
|
||||
"defaults": { "frame_rate": 24, "segment_seconds": 10 },
|
||||
"media_inventory": [
|
||||
{ "key": "SONG", "kind": "audio", "source_type": "input",
|
||||
"file_path": "codex/rooftop/song.wav", "file_name": "song.wav" },
|
||||
{ "key": "POSE_REF", "kind": "image", "source_type": "input",
|
||||
"file_path": "codex/rooftop/pose_ref.png", "file_name": "pose_ref.png" }
|
||||
],
|
||||
"tracks": {
|
||||
"master_audio": { "key": "SONG", "lock": true, "name": "Master Song", "role": "why it drives the timeline" },
|
||||
"reference_video": null
|
||||
},
|
||||
"project": { "project_save": "override", "segment_start_number": 1,
|
||||
"segment_count": -1, "sampling_plan": "medium", "sampling_mode": "single" },
|
||||
"segments": [
|
||||
{ "index": 1, "id": "seg_one_open", "start_seconds": 0, "duration_seconds": 10,
|
||||
"task_mode": "ref", "continuity_mode": "shot",
|
||||
"images": [ { "key": "POSE_REF", "shared_reference": false } ],
|
||||
"beat": "one sentence: what happens AND where the segment ends",
|
||||
"prompt_brief": "1-2 sentences: what the prompt keeps first, changes later, ends on" }
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
Field rules the validator enforces:
|
||||
|
||||
- `media_inventory` keys are UPPERCASE; every `file_path` is copied from Step 0 sources —
|
||||
this inventory is the **machine-checked verbatim source**: every segment's image items and
|
||||
every locked media track are diffed against it, so a retyped path fails on the spot. A
|
||||
paraphrase in the inventory cannot be repaired later. Order matters too: the inventory's
|
||||
image rows map to the editor's `image` input positions (`<Picture 1>`, `<Picture 2>`, …),
|
||||
audio rows to `audio` input positions, video rows to `video` input positions — the user
|
||||
wires the media into those node inputs in inventory order.
|
||||
- `tracks.master_audio` / `tracks.reference_video` name the ONE locked audio and ONE locked
|
||||
video track (the schema allows at most one of each). Lock the master song for MV/lyric/
|
||||
dance/beat projects, the narration track when narration drives the cut, and the reference
|
||||
video when the user supplies one for subject/partial replacement. Ordinary BGM/SFX/ambience
|
||||
is never locked. If two candidates could be the master, stop and ask the user — never pick
|
||||
by track order or duration.
|
||||
- Resolution: copy the user's intent into the exact DynamicCombo tokens — `16:9` means
|
||||
`16:9 (Widescreen)`, `0.9mp` means megapixels mode with the number taken verbatim; never
|
||||
convert MP into width/height. Keep the current workflow resolution when the user expressed
|
||||
no intent (then omit nothing — still write the block the user approved).
|
||||
- Segments: `start_seconds` are cumulative (previous start + duration + gap); ids are unique
|
||||
lowercase snake_case, never renamed later; `duration_seconds` about 10 unless the user
|
||||
says otherwise; only the last segment may come to rest. `continuity_mode` per segment:
|
||||
`shot` (standalone/new composition), `context` (continues the previous segment's real
|
||||
delivered tail), `context_drift` (context + subject/partial replacement from the locked
|
||||
reference video). **context_drift is all-or-nothing**: if any segment drifts, all do,
|
||||
first included; without user-pinned lengths the drift split is consecutive 10-second
|
||||
pieces (last takes the remainder) — set `lengths_user_pinned: true` only when the user
|
||||
actually specified the lengths.
|
||||
- `images` per segment: at most 9, each an inventory image key with its `shared_reference`
|
||||
flag. Reference images define IDENTITY, not composition. When several context segments
|
||||
reuse one image, prefer explicit private items (`shared_reference: false`, fixed ids,
|
||||
same file) over shared-image metadata — the current front-end regenerates UUIDs for
|
||||
sparse shared images, and unverified shared data risks unstable loads.
|
||||
- `project`: `project_save: "override"` with `segment_count: -1` deletes saved segments
|
||||
after the start index on execution — warn the user before choosing it. `upscale_by` is
|
||||
only written when the user asked (default 1.250, three decimals); the editor resolution
|
||||
is the first-pass size, and dual-sampling second pass = `round(editor_size * upscale_by /
|
||||
32) * 32`.
|
||||
|
||||
## Step 2 — Present the guide for approval (do not skip)
|
||||
|
||||
Show the user the finished guide in plain language — one line per segment: number, id,
|
||||
seconds, continuity mode, what happens and where it ends — plus the media inventory, the
|
||||
locked tracks, the resolution block and the project parameters. Then STOP and wait for
|
||||
their reply. The user approves it or asks for changes; apply every change to `outline.json`
|
||||
and, if anything changed, show the changed rows again. Start Step 3 only after an explicit
|
||||
approval. The approved guide is the contract: segments are composed from it, ids, frames
|
||||
and modes are never silently altered, and new story ideas from the user go through the
|
||||
guide first — never straight into a segment file.
|
||||
|
||||
## Step 3 — Compose the segments one at a time (plain text)
|
||||
|
||||
Work only from the approved guide. In story order: compose ONE segment file, validate it,
|
||||
fix it until it passes, and only then move to the next. Since you wrote the previous file
|
||||
yourself, open it and reuse its `# handoff:` line as the opening state of the next context
|
||||
join. Rewrite a segment as many times as it takes. If Cline rejects a write with
|
||||
'Editor input too large', split it with the marker pattern from the rules above.
|
||||
|
||||
**A segment file is plain text** (`segment_01.txt`, `segment_02.txt`, …): the scene's
|
||||
complete prompt, written exactly as it will sit inside the paste — multi-line section
|
||||
structure (`subject_definitions:`, `[Shot 1]`, sound cues) survives into the clip — and
|
||||
ONE trailing `# handoff:` line naming the real final frame. No id, no frame numbers, no
|
||||
task mode, no media paths, no JSON: the approved outline row supplies all of that when
|
||||
the scripts load the file, so none of it can be mistyped. The `# handoff:` line is the
|
||||
contract the next segment's join opens inside — pose, motion in progress, camera state
|
||||
(15+ words); it is stripped before anything reaches the model.
|
||||
|
||||
```text
|
||||
subject_definitions:
|
||||
<Subject 1> identity: Sam is a 9 year old boy with his blonde hair in pigtails …
|
||||
summary:
|
||||
[reference generation]
|
||||
detailed_description:
|
||||
[Shot 1] …the full scene prompt, multi-line, ending in motion…
|
||||
overall_soundscape:
|
||||
…
|
||||
# handoff: Sam stands tipped onto his toes at the path's edge, both hands still reaching,
|
||||
# fingers an inch short of the hovering butterfly's orange wings, the camera holding low…
|
||||
```
|
||||
|
||||
(The shipped `examples/segment_01.txt`–`segment_03.txt` are the worked model — the same
|
||||
story the example paste merges from, byte-identical.)
|
||||
|
||||
**What the plain-text form forbids** (the validator fails the file on the spot): no `|`
|
||||
(it would split the scene in the paste), no `[start-end]` markers (the merge appends
|
||||
them itself), no `@` (the editor strips it), and no `<Picture N>`/`<Audio N>`/`<Video N>`
|
||||
tokens on a zero-media story (JSON-only). On a story with media the tokens are allowed
|
||||
and diff-checked against the row's approved images. **Media copy rule, reduced to one
|
||||
sentence:** media paths are typed ONLY in `outline.json`'s `media_inventory` — segment
|
||||
files cannot carry them at all.
|
||||
|
||||
**JSON fragment escape hatch:** only when a segment needs `user_prompt_variant: "b"` or a
|
||||
custom `system_prompt` (both rare, both user-requested) write that ONE segment as JSON —
|
||||
`{"id": row id, "start_frame": row start, "end_frame": row end, "content": {"task_mode":
|
||||
…, "continuity_mode": …, "images": […], "user_prompt": …}, "_handoff": {"end_state":
|
||||
…}}` (full shape in `references/track-schema.md`). Frames and ids still come from the
|
||||
row, and the merge takes `.txt` and `.json` fragments in any mix.
|
||||
|
||||
```sh
|
||||
py -3 "%USERPROFILE%\.agent\skills\easy-media-track-gate\verify_track_rules.py" segment segment_01.txt --index 1 --outline outline.json
|
||||
py -3 "%USERPROFILE%\.agent\skills\easy-media-track-gate\verify_track_rules.py" segment segment_02.txt --index 2 --prev segment_01.txt --outline outline.json
|
||||
```
|
||||
|
||||
(`--total` is optional with `--outline`; `--prev` chains the previous file and enforces
|
||||
consecutive frames `[0,240), [240,480), …` — never an extra frame at a boundary. `.txt`
|
||||
and `.json` fragments both work in every position.)
|
||||
|
||||
### Prompt craft rules (violations fail the validator)
|
||||
|
||||
- Every task segment renders standalone — the model never sees any other segment's prompt.
|
||||
Fully restate scene, light and subject identity inside each prompt; never say "the same
|
||||
one", "as before", "the previous shot". This ban is checked in every mode.
|
||||
- Reference images define identity only: they are not the first frame or a composition
|
||||
anchor; context joins never use image anchoring or `[video continuation]`.
|
||||
- **context segments open by keeping the delivered tail** — the join starts from the
|
||||
previous segment's real last frames (about 0.92 s earlier than the saved boundary at
|
||||
24 fps), whose framing can differ from what the previous prompt imagined: open with the
|
||||
camera keeping its direction and the subject keeping pose, motion and light. Give the
|
||||
hold an action (a breath, a weight shift) so it does not render as a freeze-frame.
|
||||
- **Keep first, change later**: before about `00:02.000` no cut, no new key action; a
|
||||
required composition change is written as an explicit cut later in the prompt. For a
|
||||
genuinely new composition use a `shot` segment instead of `context`.
|
||||
- **End in motion**: a context segment followed by another context segment never ends at
|
||||
rest — no "comes to a stop", "settles", "holds still", "falls asleep" in the closing
|
||||
lines. Only the timeline's final segment may come to rest.
|
||||
- Compose, never summarize: subject anchor with its look, blocking, one camera idea,
|
||||
performance beats, physical texture (wardrobe behavior, weather on skin and fabric,
|
||||
light), and a concrete mid-action exit state. Decompose every action into ordered
|
||||
body mechanics ("bends his knees, grips the handle with his left hand, then stands"),
|
||||
naming the limb; camera, light and weather each get their own ordered visible stages.
|
||||
The validator trips a thin-prompt floor (60 words) — never pad to a band, and never
|
||||
hand in a one-liner.
|
||||
- `system_prompt` stays `""` unless the user asked for one: an empty system prompt lets the
|
||||
node auto-append the MiniMax reference rules for context joins; a custom one disables
|
||||
that (and needs the one-segment JSON escape hatch above).
|
||||
- The `# handoff:` line must be truthful — it is the contract the next join opens inside.
|
||||
|
||||
## Step 4 — Verify the chain, merge to the paste
|
||||
|
||||
Once every segment file passes, the merge is the continuity check: `build_track_plan.py`
|
||||
re-walks every fragment against the guide (ids, frames, modes, verbatim media paths, prompt
|
||||
rules, handoff chain — all pinned by the outline rows), builds the paste from the same
|
||||
validated fragments, and writes ONE file by default. Never write the paste by hand and
|
||||
never edit fragment prose during the merge.
|
||||
|
||||
```sh
|
||||
py -3 "%USERPROFILE%\.agent\skills\easy-media-track-gate\build_track_plan.py" outline.json segment_01.txt segment_02.txt segment_03.txt
|
||||
py -3 "%USERPROFILE%\.agent\skills\easy-media-track-gate\verify_track_rules.py" override prompt_override.txt --outline outline.json
|
||||
```
|
||||
|
||||
Zero-media stories get `prompt_override.txt` (the plain-text paste — THE deliverable);
|
||||
stories with media get `prompt_override.json` instead (the only form that carries
|
||||
`<Picture N>`/`<Audio N>`/`<Video N>` tokens). Two opt-ins: `--json` also writes the
|
||||
payload for a zero-media story that needs per-segment continuity modes or track locks,
|
||||
and `--plan plan.json` also writes the patch plan (plus `track_data.json` for inspection)
|
||||
for the whole-workflow patch path (Step 5B). Segment, track and image ids are stable
|
||||
UUIDv5 values derived from the project title, so a re-merge keeps ids consistent. The
|
||||
`override` command re-checks the paste against the guide: one scene per segment in order;
|
||||
plain text — `[start-end]`/`[start-end,type]` markers equal to the approved row frames and
|
||||
task types (or, without markers, equal row lengths), @imageN/@audioN/@videoN refs matching
|
||||
the inventory, no `<Picture N>` (JSON-only), no stray `@`, no `[number-number]` prose (the
|
||||
parser eats it as a frame range); JSON — durations inside the runtime's 2–15s window and
|
||||
equal to the approved rows, per-segment generation/continuity modes, the exact
|
||||
`<Picture N>` / `<Audio N>` / `<Video N>` reference set per prompt (a missing token
|
||||
renders with no reference media; a stray one attaches the wrong image), and the track
|
||||
locks.
|
||||
|
||||
## Step 5 — Deliver (plain-text paste first, patch script as the alternative)
|
||||
|
||||
### 5A — Default: the plain-text `prompt_override` paste
|
||||
|
||||
**The paste is plain text — `scene1|scene2|...` — unless the story needs the JSON option
|
||||
below.** Deliver `prompt_override.txt` (the merge's default output): the scene
|
||||
prompts joined with `|` on ONE line, one scene per pipe-chunk. The editor splits on pipes
|
||||
with no JSON at all: one clip per scene, back-to-back; a trailing `|` is a no-op. Newlines
|
||||
INSIDE a scene survive — multi-section prompt structure is fine — but `|` inside a scene
|
||||
would split it. For unequal scene lengths or a scene that is not the plain-text default
|
||||
t2v (e.g. r2v), the merge ends each scene with a `[start-end]` / `[start-end,type]`
|
||||
marker (`[0-240,r2v]|[240-480,r2v]|[480-720,r2v]` — the marker sits at the scene END,
|
||||
sets the frame range and task TYPE, and is stripped from the text before it reaches the
|
||||
model; the worked example `examples/prompt_override_r2v_refmod.txt` shows the full
|
||||
multi-section r2v form). A story with media needs the JSON option: `<Picture N>` tokens
|
||||
are NOT parsed in this path — they would stay literal text (the text path attaches media
|
||||
via `@imageN`/`@audioN`/`@videoN` only).
|
||||
|
||||
Deliver `prompt_override.txt` with these instructions:
|
||||
|
||||
1. In the editor node set `format` to `MiniMax`.
|
||||
2. Zero-media story: wire nothing — the editor demands no media inputs. (A story with
|
||||
media cannot go through this path — see the JSON option.)
|
||||
3. Select the `prompt_override` widget, clear it, paste the one-line plain text. The
|
||||
editor rebuilds the task track: one clip per pipe-chunk, back-to-back, frame ranges
|
||||
and task types as marked (or even splits of the editor's `total_length` when no
|
||||
markers are present — keep that length equal to the story total).
|
||||
4. **Rendering walks the timeline through `easy multiTrackTaskOutput` — and that node
|
||||
outputs ONE task segment per pass, selected by its `task_index` int input (default 0 =
|
||||
only the FIRST clip).** A workflow whose `task_index` stays 0 renders only the first
|
||||
scene no matter how many scenes the paste carries. Two ways to render all clips:
|
||||
- **Loop workflow (proper multi-shot):** use the node pack's
|
||||
`example_workflows/MinimaxH3_ForLoop_MultiShot.json`. Its `forLoopStart` takes
|
||||
`total` from `easy multiTrackInfoOutput.TASK_COUNT` (via a math node that also lets a
|
||||
PrimitiveInt pick one segment: -1 = all), and its index feeds
|
||||
`multiTrackTaskOutput.task_index`; each iteration generates one clip, batches it, and
|
||||
`mergeVideosFromPaths` concatenates everything after the loop. Paste the override
|
||||
into THAT editor and set the PrimitiveInt to -1.
|
||||
- **Manual fallback (no rewiring):** set `task_index` to 0, queue; then 1, queue; then
|
||||
2, queue. Three clip files, join them afterwards. (`task_index = -1` outputs the whole
|
||||
timeline's pre-rendered MEDIA — it is not a 3-clip generation.)
|
||||
5. **Optional LLM polish slot** (only if the user asks for it): inside the loop,
|
||||
`TaskOutput.SYSTEM_PROMPT/USER_PROMPT` may feed an LLM node whose output replaces the
|
||||
prompt at the render node, with an ifElse bypass back to raw `USER_PROMPT`. The skill's
|
||||
prompts are already final — leave the bypass OFF unless requested. Ways to fill the
|
||||
slot: any ComfyUI LLM node (system+user in, string out), or the pack's
|
||||
`easy multiTrackPromptEnhancer` with model `openai_compatible (local)` pointed at the
|
||||
user's own endpoint (`base_url` accepts any OpenAI-compatible server: Ollama, LM
|
||||
Studio, vLLM, llama.cpp server). Full wiring: `references/workflow_wiring.md`.
|
||||
|
||||
**JSON option (`prompt_override.json`) — only for what plain text cannot carry:**
|
||||
`<Picture N>`/`<Audio N>`/`<Video N>` media tokens (stories with images/song/reference
|
||||
video — the merge writes this payload automatically for them), per-segment continuity
|
||||
modes or track locks on a zero-media story (re-run the merge with `--json`). The payload
|
||||
is minimal by design — the parser requires only `type` and `prompts`; optional keys
|
||||
appear only when non-default (duration is SECONDS, `generation_type` defaults to `r2v` in
|
||||
the pack's code) — and is self-contained: one prompt per clip, media reference tokens
|
||||
inside the prompts, no frame fields anywhere. The paste steps are the same three as above
|
||||
(format MiniMax; wire the inventory's image rows into the editor's `image` input in
|
||||
inventory order, the master song into `audio`, the reference video into `video` — the
|
||||
`<Picture N>`/`<Audio N>`/`<Video N>` tokens resolve against those input lists
|
||||
positionally; clear the widget and paste the one-line JSON), and the same
|
||||
`task_index`/loop wiring applies afterwards.
|
||||
|
||||
**The paste target is the `prompt_override` input and only these two forms.** Never give
|
||||
the user a `plan.json`, an outline, an H3 plan or any other JSON to paste there: without
|
||||
`"type": "minimax_prompt_override"` the editor falls back to its `|`-separated text parser
|
||||
and renders the whole thing as ONE clip spanning the timeline (the single-clip failure).
|
||||
Never add `|` inside a scene for the same reason.
|
||||
|
||||
### 5B — Alternative: the node pack's patch script (whole-workflow patch)
|
||||
|
||||
When the user wants the timeline written into a workflow file directly (or needs `edit`
|
||||
task modes / extra tracks the paste payload cannot express), first produce the patch plan
|
||||
by re-running the merge with `--plan plan.json` (it also writes `track_data.json` for
|
||||
inspection), then apply it with the Easy-Media node pack's script — find it under the
|
||||
node pack folder (typically
|
||||
`<ComfyUI>\custom_nodes\ComfyUI-Easy-Media\skills\easy-media-multitrack-workflow\scripts\patch_workflow.py`;
|
||||
ask the user for the path and the workflow JSON if they are not at hand):
|
||||
|
||||
```sh
|
||||
py -3 <node-pack>\skills\easy-media-multitrack-workflow\scripts\patch_workflow.py apply WORKFLOW.json --plan plan.json
|
||||
py -3 <node-pack>\skills\easy-media-multitrack-workflow\scripts\patch_workflow.py apply WORKFLOW.json --plan plan.json --output RESULT.json --write
|
||||
py -3 <node-pack>\skills\easy-media-multitrack-workflow\scripts\patch_workflow.py inspect RESULT.json
|
||||
```
|
||||
|
||||
Always dry-run first; only write to a NEW output file (`--output RESULT.json --write`),
|
||||
never over the user's source workflow. `inspect` must show the editor's resolution, format,
|
||||
frame rate, `total_length`, the task track with all segments and the locked tracks with
|
||||
their media. Report: base workflow path, output file, editor/project node ids, resolution,
|
||||
timeline summary, media paths, continuity modes, locked tracks and whether the node graph
|
||||
stayed untouched (`graph_preserved: true`).
|
||||
|
||||
Actually opening the workflow in ComfyUI or queueing it (`submit_workflow.py`, UI-tab
|
||||
submission) is a separate authorization — only on the user's explicit request, and it is
|
||||
the node pack's own flow (see its `references/execution-api.md`). "Queued" never means
|
||||
"generated"; wait for the real completion status before reporting success.
|
||||
|
||||
## Revising after approval
|
||||
|
||||
- Guide-level change (timings, beats, modes, segment count, media): update `outline.json`
|
||||
first, show the changed rows, then rewrite only the affected segment files against it
|
||||
and re-run the Step 4 merge so the paste refreshes (plus the `--json`/`--plan` outputs
|
||||
when they were requested).
|
||||
- One segment reads badly: rewrite that `segment_NN.txt` (the outline row still pins its
|
||||
id and frames), re-validate it with the `segment` command, then re-merge. Never patch
|
||||
`prompt_override.txt`, `prompt_override.json` or `plan.json` directly — the merge
|
||||
rebuilds the paste from the fragments every time.
|
||||
|
||||
## If you cannot run Python
|
||||
|
||||
Say plainly that the result skipped script validation. Follow the rules manually: copy the
|
||||
`examples/` shapes exactly, compute frames as `round(seconds * frame_rate)`, copy every
|
||||
`file_path` and resolution token by selection (never retyping), and hand-assemble the
|
||||
paste the way Step 4 describes. Hand-write the paste as plain text — scene prompts
|
||||
joined with `|` on one line, one `[start-end(,type)]` marker at each scene end when
|
||||
lengths differ or the scene is not t2v, exactly like
|
||||
`examples/prompt_override_r2v_refmod.txt` — and use the JSON payload only when the story
|
||||
needs `<Picture N>` tokens, continuity modes or locks. Manual assembly is error-prone —
|
||||
prefer getting Python running (any of `py -3`, `python`, a `.venv`) over hand-building
|
||||
the paste.
|
||||
311
easy-media-track-gate/build_track_plan.py
Executable file
311
easy-media-track-gate/build_track_plan.py
Executable file
@ -0,0 +1,311 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Merge the approved outline + segment fragments into the prompt_override paste.
|
||||
|
||||
py -3 build_track_plan.py outline.json segment_01.txt segment_02.txt [segment_03.txt ...]
|
||||
py -3 build_track_plan.py outline.json segment_01.json ... --plan plan.json
|
||||
py -3 build_track_plan.py --check plan.json [--outline outline.json]
|
||||
|
||||
The deliverable is ONE paste file, plain text by default — everything else is opt-in:
|
||||
|
||||
1. `prompt_override.txt` (default output, zero-media stories) — plain text for the
|
||||
`easy multiTrackEditor` node's `prompt_override` input (format MiniMax): the scene
|
||||
prompts joined with '|' on one line, plus [start-end(,type)] markers when the rows
|
||||
have unequal durations or a scene is not the text default (flf -> t2v). The editor
|
||||
splits a non-JSON prompt_override on '|' — one clip per scene, no JSON needed.
|
||||
<Picture N>/<Audio N>/<Video N> tokens are NOT parsed in this path.
|
||||
2. `prompt_override.json` — written INSTEAD of the txt when the story has media (the
|
||||
minimal {"type": "minimax_prompt_override", "prompts": [...]} payload with media
|
||||
tokens; parser requires ONLY type+prompts, duration in SECONDS, other keys only when
|
||||
non-default), or alongside it with --json when a zero-media story needs per-segment
|
||||
continuity modes or track locks the text form cannot carry.
|
||||
3. `--plan plan.json` — the optional patch plan for the node pack's
|
||||
`patch_workflow.py apply` ({"editor": {..., "track_data": {...}}, "project": {...}}),
|
||||
plus `track_data.json` (pretty, for inspection) next to it. Written only when asked.
|
||||
|
||||
Segment fragments are plain text (segment_NN.txt: the scene prompt verbatim + a
|
||||
'# handoff:' line); the approved outline row supplies id, frames, modes and images.
|
||||
JSON fragments are still accepted for the rare variant-b / custom-system-prompt cases.
|
||||
The merge is the continuity check: it re-validates every fragment against the outline
|
||||
(ids, frames, modes, verbatim media paths, context-prompt rules, handoff chain) before
|
||||
anything is written. Never edit fragment prose here and never patch the paste by hand —
|
||||
fix the fragment, re-validate, re-merge. Segment/track/image ids are deterministic
|
||||
UUIDv5 values derived from the project title, so re-runs keep ids stable.
|
||||
|
||||
Standard library only.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import copy
|
||||
import json
|
||||
import sys
|
||||
import uuid
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parent))
|
||||
import verify_track_rules as vtr # noqa: E402
|
||||
|
||||
|
||||
def stable_id(*parts: str) -> str:
|
||||
seed = "easy-media-track-gate/" + "/".join(parts)
|
||||
return str(uuid.uuid5(uuid.NAMESPACE_URL, seed))
|
||||
|
||||
|
||||
def _finalize_image(img: dict[str, Any], inv_entry: dict[str, Any], seg_id: str, pos: int,
|
||||
shared_default: bool) -> dict[str, Any]:
|
||||
out = {
|
||||
"id": img.get("id") or stable_id(seg_id, f"image-{pos}"),
|
||||
"source_type": img.get("source_type", inv_entry["source_type"]),
|
||||
"file_path": inv_entry["path"],
|
||||
"file_name": img.get("file_name") or inv_entry.get("file_name")
|
||||
or inv_entry["path"].rsplit("/", 1)[-1],
|
||||
"shared_reference": img.get("shared_reference", shared_default),
|
||||
}
|
||||
if out["source_type"] == "url":
|
||||
out["url"] = img.get("url") or inv_entry["path"]
|
||||
out.pop("file_path", None)
|
||||
out.pop("file_name", None)
|
||||
return out
|
||||
|
||||
|
||||
def build(outline: dict[str, Any], fragments: list[dict[str, Any]]) -> dict[str, Any]:
|
||||
rows = outline["rows"]
|
||||
if len(fragments) != len(rows):
|
||||
raise vtr.Fail(f"expected {len(rows)} fragments (one per approved outline row), got {len(fragments)}")
|
||||
prev: dict[str, Any] | None = None
|
||||
for i, frag in enumerate(fragments):
|
||||
vtr.validate_fragment(frag, i + 1, len(rows), prev, outline)
|
||||
prev = frag
|
||||
|
||||
task_segments: list[dict[str, Any]] = []
|
||||
for i, (frag, row) in enumerate(zip(fragments, rows)):
|
||||
seg_id = stable_id(outline["title"], "segment", row["id"])
|
||||
content = copy.deepcopy(frag["content"])
|
||||
content.setdefault("media_type", "none")
|
||||
content["media_type"] = "none"
|
||||
content.setdefault("user_prompt_variant", "a")
|
||||
if content.get("user_prompt_variant") == "a":
|
||||
content.setdefault("user_prompt_b", "")
|
||||
content.setdefault("system_prompt", "")
|
||||
content.setdefault("muted", False)
|
||||
content.setdefault("volume_db", 0)
|
||||
if "ref_image_size" not in content:
|
||||
content["ref_image_size"] = "match"
|
||||
row_imgs = row.get("images", [])
|
||||
final_imgs = []
|
||||
for pos, (img, row_img) in enumerate(zip(content.get("images", []), row_imgs)):
|
||||
inv_entry = outline["inventory"][row_img["key"]]
|
||||
final_imgs.append(_finalize_image(img, inv_entry, seg_id, pos,
|
||||
bool(row_img.get("shared_reference", False))))
|
||||
content["images"] = final_imgs
|
||||
task_segments.append({
|
||||
"id": seg_id,
|
||||
"start_frame": frag["start_frame"],
|
||||
"end_frame": frag["end_frame"],
|
||||
"color": "var(--multitrack-task-bg)",
|
||||
"content": content,
|
||||
})
|
||||
|
||||
total = max((s["end_frame"] for s in task_segments), default=0)
|
||||
tracks: list[dict[str, Any]] = [{
|
||||
"id": stable_id(outline["title"], "track", "tasks"),
|
||||
"name": "Tasks",
|
||||
"type": "task",
|
||||
"color": "var(--multitrack-task-bg)",
|
||||
"muted": False,
|
||||
"solo": False,
|
||||
"volume_db": 0,
|
||||
"locked": False,
|
||||
"segments": task_segments,
|
||||
}]
|
||||
roles = outline["_roles"]
|
||||
for role, track_type in (("master_audio", "audio"), ("reference_video", "video")):
|
||||
spec = roles.get(role)
|
||||
if not spec:
|
||||
continue
|
||||
inv = outline["inventory"][spec["key"]]
|
||||
media_content: dict[str, Any] = {
|
||||
"media_type": track_type,
|
||||
"source_type": inv["source_type"],
|
||||
}
|
||||
if inv["source_type"] == "url":
|
||||
media_content["url"] = inv["path"]
|
||||
else:
|
||||
media_content["file_path"] = inv["path"]
|
||||
media_content["file_name"] = inv.get("file_name") or inv["path"].rsplit("/", 1)[-1]
|
||||
tracks.append({
|
||||
"id": stable_id(outline["title"], "track", role),
|
||||
"name": spec["name"],
|
||||
"type": track_type,
|
||||
"color": "var(--primary)",
|
||||
"muted": False,
|
||||
"solo": False,
|
||||
"volume_db": 0,
|
||||
"locked": False,
|
||||
"audio_locked": True,
|
||||
"segments": [{
|
||||
"id": stable_id(outline["title"], "media", role),
|
||||
"start_frame": 0,
|
||||
"end_frame": max(total, 1),
|
||||
"color": "var(--primary)",
|
||||
"content": media_content,
|
||||
}],
|
||||
})
|
||||
|
||||
track_data: dict[str, Any] = {
|
||||
"muted": False,
|
||||
"volume_db": 0,
|
||||
"task_markers": [],
|
||||
"task_overview": False,
|
||||
"tracks": tracks,
|
||||
"total_length": total,
|
||||
"frame_rate": outline["defaults"]["frame_rate"],
|
||||
}
|
||||
plan: dict[str, Any] = {
|
||||
"editor": {
|
||||
"resolution": copy.deepcopy(outline["editor"]["resolution"]),
|
||||
"track_data": track_data,
|
||||
"format": outline["editor"]["format"],
|
||||
},
|
||||
"project": copy.deepcopy(outline["project"]),
|
||||
}
|
||||
project = plan["project"]
|
||||
vtr.validate_plan(plan, outline=outline)
|
||||
return plan
|
||||
|
||||
|
||||
def _summary(plan: dict[str, Any]) -> str:
|
||||
data = plan["editor"]["track_data"]
|
||||
task_count = sum(len(t.get("segments", [])) for t in data["tracks"] if t.get("type") == "task")
|
||||
locked = [f"{t['type']}:{t.get('name', t['id'])}" for t in data["tracks"]
|
||||
if t.get("audio_locked") is True]
|
||||
fps = data["frame_rate"]
|
||||
return (f"{task_count} task segments, total_length {data['total_length']} frames "
|
||||
f"= {data['total_length'] / fps:.2f}s at {fps} fps; "
|
||||
f"tracks: {', '.join(t['type'] + ' ' + t.get('name', '') for t in data['tracks'])}; "
|
||||
f"audio_locked: {', '.join(locked) if locked else 'none'}")
|
||||
|
||||
|
||||
def _write_json(path: Path, value: dict[str, Any]) -> None:
|
||||
path.write_text(json.dumps(value, ensure_ascii=False, indent=2) + "\n", encoding="utf-8")
|
||||
|
||||
|
||||
def _main(argv: list[str]) -> int:
|
||||
parser = argparse.ArgumentParser(description=__doc__.splitlines()[0])
|
||||
parser.add_argument("inputs", nargs="*",
|
||||
help="outline.json then segment_01.(txt|json) ... (merge) or plan.json (--check)")
|
||||
parser.add_argument("-o", "--output", default=None,
|
||||
help="the paste file (default: prompt_override.txt for zero-media "
|
||||
"stories, prompt_override.json when the story has media)")
|
||||
parser.add_argument("--plan", default=None, metavar="PLAN_JSON",
|
||||
help="also write the patch plan (plan.json) for the node pack's "
|
||||
"patch_workflow.py apply, plus track_data.json next to it")
|
||||
parser.add_argument("--json", action="store_true",
|
||||
help="also write prompt_override.json (the payload for continuity "
|
||||
"modes / track locks the plain text cannot carry)")
|
||||
parser.add_argument("--check", action="store_true", help="re-validate an existing plan.json")
|
||||
parser.add_argument("--outline", help="outline to re-check the plan against")
|
||||
args = parser.parse_args(argv)
|
||||
|
||||
try:
|
||||
if args.check:
|
||||
if not args.inputs:
|
||||
raise vtr.Fail("--check needs the plan file path")
|
||||
plan = vtr.load_json(args.inputs[0], "patch plan")
|
||||
outline = None
|
||||
if args.outline:
|
||||
outline = vtr.load_json(args.outline, "outline")
|
||||
vtr.validate_outline(outline)
|
||||
vtr.validate_plan(plan, outline=outline)
|
||||
print(f"PLAN OK: {_summary(plan)}")
|
||||
return 0
|
||||
if len(args.inputs) < 2:
|
||||
raise vtr.Fail("usage: build_track_plan.py outline.json segment_01.txt "
|
||||
"[segment_02.txt ...] [--plan plan.json] [--json]")
|
||||
outline = vtr.load_json(args.inputs[0], "outline")
|
||||
vtr.validate_outline(outline)
|
||||
fragments = [vtr.load_fragment_any(p, outline, i)
|
||||
for i, p in enumerate(args.inputs[1:], 1)]
|
||||
# The merge is the continuity check: build() re-walks every fragment against
|
||||
# the outline (ids, frames, modes, verbatim media paths, prompt rules, handoff
|
||||
# chain) and validates the assembled TRACK_DATA — the plan object itself is
|
||||
# only written when --plan asks for it.
|
||||
plan = build(outline, fragments)
|
||||
uses_media = bool(
|
||||
outline["_roles"].get("master_audio")
|
||||
or outline["_roles"].get("reference_video")
|
||||
or any(row.get("images") for row in outline["rows"])
|
||||
)
|
||||
# Plain-text form — the DEFAULT paste: the prompt_override input splits on '|'
|
||||
# with no JSON at all (parse_override_segments -> one segment per part). Only
|
||||
# valid with zero media — <Picture N>/<Audio N>/<Video N> tokens are JSON-only.
|
||||
# Unequal row durations or non-default task types are carried by
|
||||
# [start-end(,type)] markers, which the builder emits per scene (proven:
|
||||
# [0-240,r2v] renders TYPE r2v, task-duration sum drives the MiniMax total).
|
||||
prompts_count = len(outline["rows"])
|
||||
total_s = sum(float(row["duration_seconds"]) for row in outline["rows"])
|
||||
paste_path = Path(args.output) if args.output else None
|
||||
json_path = None
|
||||
if uses_media:
|
||||
override = vtr.build_prompt_override(outline, fragments)
|
||||
vtr.validate_prompt_override(copy.deepcopy(override), outline)
|
||||
if paste_path is None:
|
||||
paste_path = Path("prompt_override.json")
|
||||
if paste_path.suffix.lower() != ".json":
|
||||
raise vtr.Fail("this story has media — the paste must be the JSON payload "
|
||||
"(<Picture N>/<Audio N>/<Video N> tokens are JSON-only); "
|
||||
"give -o a .json path")
|
||||
json_path = paste_path
|
||||
json_path.write_text(
|
||||
json.dumps(override, ensure_ascii=True, separators=(",", ":")) + "\n",
|
||||
encoding="utf-8")
|
||||
print(f"OVERRIDE OK (JSON payload): {prompts_count} prompts, "
|
||||
f"duration {override.get('duration', '10 (default)')}, "
|
||||
f"total {total_s:.2f}s; wrote {json_path} "
|
||||
f"({json_path.stat().st_size} chars, one line)")
|
||||
print("PASTE NOTE: this story has media — <Picture N>/<Audio N>/<Video N> tokens are "
|
||||
"JSON-only, so the paste is prompt_override.json (the text path would leave "
|
||||
"those tokens as literal prompt text)")
|
||||
else:
|
||||
paste_text = vtr.build_plain_text_override(outline, fragments)
|
||||
vtr.validate_prompt_override_text(paste_text, copy.deepcopy(outline))
|
||||
if paste_path is None:
|
||||
paste_path = Path("prompt_override.txt")
|
||||
paste_path.write_text(paste_text + "\n", encoding="utf-8")
|
||||
print(f"PLAIN TEXT OK (the paste): {prompts_count} scenes split on '|', "
|
||||
f"total {total_s:.2f}s; wrote {paste_path} "
|
||||
f"({paste_path.stat().st_size} chars) — [start-end,type] markers where "
|
||||
"lengths differ or a scene is not t2v")
|
||||
if args.json:
|
||||
override = vtr.build_prompt_override(outline, fragments)
|
||||
vtr.validate_prompt_override(copy.deepcopy(override), outline)
|
||||
json_path = paste_path.parent / "prompt_override.json"
|
||||
json_path.write_text(
|
||||
json.dumps(override, ensure_ascii=True, separators=(",", ":")) + "\n",
|
||||
encoding="utf-8")
|
||||
print(f"OVERRIDE OK (JSON payload): wrote {json_path} — for per-segment "
|
||||
"continuity modes / track locks the plain text cannot carry")
|
||||
if args.plan:
|
||||
plan_path = Path(args.plan)
|
||||
_write_json(plan_path, plan)
|
||||
_write_json(plan_path.parent / "track_data.json", plan["editor"]["track_data"])
|
||||
print(f"PLAN OK: {_summary(plan)}")
|
||||
print(f"wrote {plan_path} and {plan_path.parent / 'track_data.json'} — for the "
|
||||
"node pack's patch_workflow.py apply (dry-run first)")
|
||||
print(f"deliver: paste {paste_path} into the multitrack editor's prompt_override "
|
||||
f"input (format {outline['editor']['format']!r})")
|
||||
print("NOTE: easy multiTrackTaskOutput renders ONE task segment per pass "
|
||||
"(task_index, default 0 = first clip only).")
|
||||
print(" - all clips: use the pack's example_workflows/MinimaxH3_ForLoop_MultiShot.json "
|
||||
"(loop index -> task_index, PrimitiveInt -1 = all, auto-merge)")
|
||||
print(f" - or queue manually with task_index 0..{prompts_count - 1} and join the clips")
|
||||
return 0
|
||||
except vtr.Fail as exc:
|
||||
print(f"FAIL: {exc}")
|
||||
return 1
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(_main(sys.argv[1:]))
|
||||
89
easy-media-track-gate/examples/outline.json
Normal file
89
easy-media-track-gate/examples/outline.json
Normal file
@ -0,0 +1,89 @@
|
||||
{
|
||||
"title": "sunrise_rooftop_coffee",
|
||||
"lengths_user_pinned": true,
|
||||
"editor": {
|
||||
"format": "MiniMax",
|
||||
"resolution": {
|
||||
"resolution": "width x height (megapixels)",
|
||||
"aspect_ratio": "16:9 (Widescreen)",
|
||||
"megapixels": 0.8
|
||||
}
|
||||
},
|
||||
"defaults": {
|
||||
"frame_rate": 24,
|
||||
"segment_seconds": 10
|
||||
},
|
||||
"media_inventory": [
|
||||
{
|
||||
"key": "SONG",
|
||||
"kind": "audio",
|
||||
"source_type": "input",
|
||||
"file_path": "codex/rooftop/song.wav",
|
||||
"file_name": "song.wav"
|
||||
},
|
||||
{
|
||||
"key": "POSE_REF",
|
||||
"kind": "image",
|
||||
"source_type": "input",
|
||||
"file_path": "codex/rooftop/pose_ref.png",
|
||||
"file_name": "pose_ref.png"
|
||||
}
|
||||
],
|
||||
"tracks": {
|
||||
"master_audio": {
|
||||
"key": "SONG",
|
||||
"lock": true,
|
||||
"name": "Master Song",
|
||||
"role": "the user's song drives the whole timeline"
|
||||
},
|
||||
"reference_video": null
|
||||
},
|
||||
"project": {
|
||||
"project_save": "override",
|
||||
"segment_start_number": 1,
|
||||
"segment_count": -1,
|
||||
"sampling_plan": "medium",
|
||||
"sampling_mode": "single"
|
||||
},
|
||||
"segments": [
|
||||
{
|
||||
"index": 1,
|
||||
"id": "seg_one_open",
|
||||
"start_seconds": 0,
|
||||
"duration_seconds": 10,
|
||||
"task_mode": "ref",
|
||||
"continuity_mode": "shot",
|
||||
"images": [
|
||||
{ "key": "POSE_REF", "shared_reference": false }
|
||||
],
|
||||
"beat": "She pours coffee at the rooftop rail as the sun clears the skyline; ends mid-sip, city brightening.",
|
||||
"prompt_brief": "Establish rooftop, rail, skyline; pour coffee; end mid-sip with motion alive."
|
||||
},
|
||||
{
|
||||
"index": 2,
|
||||
"id": "seg_two_sip_wind",
|
||||
"start_seconds": 10,
|
||||
"duration_seconds": 10,
|
||||
"task_mode": "ref",
|
||||
"continuity_mode": "context",
|
||||
"images": [
|
||||
{ "key": "POSE_REF", "shared_reference": false }
|
||||
],
|
||||
"beat": "Wind tugs her cardigan mid-sip; she turns her face into the light; ends still turning.",
|
||||
"prompt_brief": "Keep framing and pose from the delivered tail; wind arrives at 00:02; end turning."
|
||||
},
|
||||
{
|
||||
"index": 3,
|
||||
"id": "seg_three_city_wakes",
|
||||
"start_seconds": 20,
|
||||
"duration_seconds": 10,
|
||||
"task_mode": "ref",
|
||||
"continuity_mode": "context",
|
||||
"images": [
|
||||
{ "key": "POSE_REF", "shared_reference": false }
|
||||
],
|
||||
"beat": "She lowers the cup as windows light up below; ends mid-exhale, steam curling.",
|
||||
"prompt_brief": "Keep the turn; lower the cup slowly; city windows light up; end mid-exhale."
|
||||
}
|
||||
]
|
||||
}
|
||||
63
easy-media-track-gate/examples/outline_sam_park.json
Normal file
63
easy-media-track-gate/examples/outline_sam_park.json
Normal file
@ -0,0 +1,63 @@
|
||||
{
|
||||
"title": "sam_park_butterfly",
|
||||
"lengths_user_pinned": true,
|
||||
"editor": {
|
||||
"format": "MiniMax",
|
||||
"resolution": {
|
||||
"resolution": "width x height (megapixels)",
|
||||
"aspect_ratio": "16:9 (Widescreen)",
|
||||
"megapixels": 0.8
|
||||
}
|
||||
},
|
||||
"defaults": {
|
||||
"frame_rate": 24,
|
||||
"segment_seconds": 10
|
||||
},
|
||||
"media_inventory": [],
|
||||
"tracks": {
|
||||
"master_audio": null,
|
||||
"reference_video": null
|
||||
},
|
||||
"project": {
|
||||
"project_save": "new",
|
||||
"segment_start_number": 1,
|
||||
"segment_count": -1,
|
||||
"sampling_plan": "medium",
|
||||
"sampling_mode": "single"
|
||||
},
|
||||
"segments": [
|
||||
{
|
||||
"index": 1,
|
||||
"id": "park_walk_spot",
|
||||
"start_seconds": 0,
|
||||
"duration_seconds": 10,
|
||||
"task_mode": "ref",
|
||||
"continuity_mode": "shot",
|
||||
"images": [],
|
||||
"beat": "Sam walks the sunny park path in his pink dress, the monarch butterfly flutters into view in front of him, and the scene ends mid-step as he reaches both hands out toward it.",
|
||||
"prompt_brief": "Self-contained r2v shot via the Sam refmod: walk, butterfly reveal, end mid-step reaching."
|
||||
},
|
||||
{
|
||||
"index": 2,
|
||||
"id": "butterfly_chase_miss",
|
||||
"start_seconds": 10,
|
||||
"duration_seconds": 10,
|
||||
"task_mode": "ref",
|
||||
"continuity_mode": "shot",
|
||||
"images": [],
|
||||
"beat": "Sam chases the butterfly across the grass swiping both hands at it, it dodges every grab and lifts out of reach, and the scene ends with him standing empty-handed, his face starting to crumple.",
|
||||
"prompt_brief": "Self-contained r2v shot: chase and misses, butterfly escapes, end standing, face crumpling."
|
||||
},
|
||||
{
|
||||
"index": 3,
|
||||
"id": "ground_cry",
|
||||
"start_seconds": 20,
|
||||
"duration_seconds": 10,
|
||||
"task_mode": "ref",
|
||||
"continuity_mode": "shot",
|
||||
"images": [],
|
||||
"beat": "Sam sinks down and sits on the ground, then cries like a baby, tears rolling down his cheeks as the butterfly flutters away above him.",
|
||||
"prompt_brief": "Self-contained r2v shot: sink and sit, crying with tears, butterfly departs; final segment may come to rest."
|
||||
}
|
||||
]
|
||||
}
|
||||
145
easy-media-track-gate/examples/plan.json
Normal file
145
easy-media-track-gate/examples/plan.json
Normal file
@ -0,0 +1,145 @@
|
||||
{
|
||||
"editor": {
|
||||
"resolution": {
|
||||
"resolution": "width x height (megapixels)",
|
||||
"aspect_ratio": "16:9 (Widescreen)",
|
||||
"megapixels": 0.8
|
||||
},
|
||||
"track_data": {
|
||||
"muted": false,
|
||||
"volume_db": 0,
|
||||
"task_markers": [],
|
||||
"task_overview": false,
|
||||
"tracks": [
|
||||
{
|
||||
"id": "3b53d88d-07b5-5f0e-9fc0-aa87503aae80",
|
||||
"name": "Tasks",
|
||||
"type": "task",
|
||||
"color": "var(--multitrack-task-bg)",
|
||||
"muted": false,
|
||||
"solo": false,
|
||||
"volume_db": 0,
|
||||
"locked": false,
|
||||
"segments": [
|
||||
{
|
||||
"id": "d2c2d695-f725-56ae-9024-bf9126443256",
|
||||
"start_frame": 0,
|
||||
"end_frame": 240,
|
||||
"color": "var(--multitrack-task-bg)",
|
||||
"content": {
|
||||
"media_type": "none",
|
||||
"task_mode": "ref",
|
||||
"continuity_mode": "shot",
|
||||
"ref_image_size": "match",
|
||||
"images": [
|
||||
{
|
||||
"id": "img-seg-one-open-1",
|
||||
"source_type": "input",
|
||||
"file_path": "codex/rooftop/pose_ref.png",
|
||||
"file_name": "pose_ref.png",
|
||||
"shared_reference": false
|
||||
}
|
||||
],
|
||||
"user_prompt": "Dawn light spills across an empty rooftop terrace above a hushed city. A young woman with a brown ponytail, cream cardigan over a white tee and blue jeans stands at the steel rail, matching the pose reference image for her identity only. She cradles a matte-black kettle in her right hand and tips it in one slow arc; a thread of steam rises as coffee fills the white cup in her left hand, her wrist rotating smoothly, elbow dropping as the kettle empties. She sets the kettle on the rail with a soft clink, lifts the cup, and brings it halfway to her lips as the sun's first rim crests the skyline and warms her cheek. The camera holds one slow push-in toward her profile, never cutting, while pigeon wings clap once somewhere below and the kettle's metal ticks as it cools.",
|
||||
"system_prompt": "",
|
||||
"user_prompt_variant": "a",
|
||||
"user_prompt_b": "",
|
||||
"muted": false,
|
||||
"volume_db": 0
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "69749e2e-3bd5-5e50-ae29-61bf607cba20",
|
||||
"start_frame": 240,
|
||||
"end_frame": 480,
|
||||
"color": "var(--multitrack-task-bg)",
|
||||
"content": {
|
||||
"media_type": "none",
|
||||
"task_mode": "ref",
|
||||
"continuity_mode": "context",
|
||||
"ref_image_size": "match",
|
||||
"images": [
|
||||
{
|
||||
"id": "img-seg-two-sip-wind-1",
|
||||
"source_type": "input",
|
||||
"file_path": "codex/rooftop/pose_ref.png",
|
||||
"file_name": "pose_ref.png",
|
||||
"shared_reference": false
|
||||
}
|
||||
],
|
||||
"user_prompt": "The camera holds the same framing it arrived with, and she keeps the pose, motion and light it arrived with: the cup still traveling toward her lips, steam bending in the same breeze. Her chest rises with one slow breath, weight easing onto her left hip as the cardigan's hem starts to lift. Around the two-second mark the wind strengthens into a gust that ruffles her ponytail and snaps the cardigan open a finger's width; her fingers reclose it while she turns her face slowly into the light, eyelids narrowing. Steam tears off the cup in ribbons. The camera keeps its single slow drift leftward with no cut, the city below sharpening frame by frame, traffic swelling from hum to roar, a rooftop antenna creaking softly, her cardigan snapping in steady wind as the turn continues.",
|
||||
"system_prompt": "",
|
||||
"user_prompt_variant": "a",
|
||||
"user_prompt_b": "",
|
||||
"muted": false,
|
||||
"volume_db": 0
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "6d0f444e-c6b1-5d14-a3c6-a29cd594beb7",
|
||||
"start_frame": 480,
|
||||
"end_frame": 720,
|
||||
"color": "var(--multitrack-task-bg)",
|
||||
"content": {
|
||||
"media_type": "none",
|
||||
"task_mode": "ref",
|
||||
"continuity_mode": "context",
|
||||
"ref_image_size": "match",
|
||||
"images": [
|
||||
{
|
||||
"id": "img-seg-three-city-wakes-1",
|
||||
"source_type": "input",
|
||||
"file_path": "codex/rooftop/pose_ref.png",
|
||||
"file_name": "pose_ref.png",
|
||||
"shared_reference": false
|
||||
}
|
||||
],
|
||||
"user_prompt": "The join keeps everything it is handed: same framing, same breeze, the tail of her turn still in motion as her face comes level with the camera. She continues the turn until her eyes find the skyline, then lowers the cup in one unhurried arc to the rail, her shoulder rolling with the descent. Below, windows begin to light up floor by floor, a slow tide of warm rectangles climbing the tower across the street while a bus door hisses open. She exhales a long breath, steam curling off the cup in the same wind, ponytail still lifting, cardigan snapping softly. The camera keeps its slow lateral drift and never cuts; the city's morning roar builds under the song as her chest keeps rising and falling mid-breath.",
|
||||
"system_prompt": "",
|
||||
"user_prompt_variant": "a",
|
||||
"user_prompt_b": "",
|
||||
"muted": false,
|
||||
"volume_db": 0
|
||||
}
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "51945496-eaa4-5ba0-bb78-201bbfee5819",
|
||||
"name": "Master Song",
|
||||
"type": "audio",
|
||||
"color": "var(--primary)",
|
||||
"muted": false,
|
||||
"solo": false,
|
||||
"volume_db": 0,
|
||||
"locked": false,
|
||||
"audio_locked": true,
|
||||
"segments": [
|
||||
{
|
||||
"id": "ee335ed8-75cd-5278-969f-144cba015861",
|
||||
"start_frame": 0,
|
||||
"end_frame": 720,
|
||||
"color": "var(--primary)",
|
||||
"content": {
|
||||
"media_type": "audio",
|
||||
"source_type": "input",
|
||||
"file_path": "codex/rooftop/song.wav",
|
||||
"file_name": "song.wav"
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
],
|
||||
"total_length": 720,
|
||||
"frame_rate": 24
|
||||
},
|
||||
"format": "MiniMax"
|
||||
},
|
||||
"project": {
|
||||
"project_save": "override",
|
||||
"segment_start_number": 1,
|
||||
"segment_count": -1,
|
||||
"sampling_plan": "medium",
|
||||
"sampling_mode": "single"
|
||||
}
|
||||
}
|
||||
1
easy-media-track-gate/examples/prompt_override.json
Normal file
1
easy-media-track-gate/examples/prompt_override.json
Normal file
@ -0,0 +1 @@
|
||||
{"type":"minimax_prompt_override","prompts":["<Picture 1> <Audio 1> Dawn light spills across an empty rooftop terrace above a hushed city. A young woman with a brown ponytail, cream cardigan over a white tee and blue jeans stands at the steel rail, matching the pose reference image for her identity only. She cradles a matte-black kettle in her right hand and tips it in one slow arc; a thread of steam rises as coffee fills the white cup in her left hand, her wrist rotating smoothly, elbow dropping as the kettle empties. She sets the kettle on the rail with a soft clink, lifts the cup, and brings it halfway to her lips as the sun's first rim crests the skyline and warms her cheek. The camera holds one slow push-in toward her profile, never cutting, while pigeon wings clap once somewhere below and the kettle's metal ticks as it cools.","<Picture 1> <Audio 1> The camera holds the same framing it arrived with, and she keeps the pose, motion and light it arrived with: the cup still traveling toward her lips, steam bending in the same breeze. Her chest rises with one slow breath, weight easing onto her left hip as the cardigan's hem starts to lift. Around the two-second mark the wind strengthens into a gust that ruffles her ponytail and snaps the cardigan open a finger's width; her fingers reclose it while she turns her face slowly into the light, eyelids narrowing. Steam tears off the cup in ribbons. The camera keeps its single slow drift leftward with no cut, the city below sharpening frame by frame, traffic swelling from hum to roar, a rooftop antenna creaking softly, her cardigan snapping in steady wind as the turn continues.","<Picture 1> <Audio 1> The join keeps everything it is handed: same framing, same breeze, the tail of her turn still in motion as her face comes level with the camera. She continues the turn until her eyes find the skyline, then lowers the cup in one unhurried arc to the rail, her shoulder rolling with the descent. Below, windows begin to light up floor by floor, a slow tide of warm rectangles climbing the tower across the street while a bus door hisses open. She exhales a long breath, steam curling off the cup in the same wind, ponytail still lifting, cardigan snapping softly. The camera keeps its slow lateral drift and never cuts; the city's morning roar builds under the song as her chest keeps rising and falling mid-breath."],"continuity_mode":"shot,context,context","audio_track_lock":1}
|
||||
@ -0,0 +1,46 @@
|
||||
subject_definitions:
|
||||
<Subject 1> identity: Sam is a 9 year old boy with his blonde hair in pigtails that dresses like a baby girl; voice: Sam is a 9 year old boy with his blonde hair in pigtails that dresses like a baby girl
|
||||
<Subject 2> is a small monarch butterfly with bright orange wings, black veins, black-spotted wing edges and a thin dark body
|
||||
summary:
|
||||
[reference generation]
|
||||
retention_analysis:
|
||||
<Subject 1>: fully_preserved
|
||||
<Subject 2>: fully_preserved
|
||||
detailed_description:
|
||||
Bright photoreal park realism on a warm late morning, sunlit green lawn and pale gravel path under fresh leaf shadows, the butterfly's orange the one saturated accent in a soft natural palette.
|
||||
Gentle camera kept low at the child's eye height, gliding smoothly alongside the walk.
|
||||
[Shot 1] <Subject 1> strolls along the pale gravel path through the sunlit park, his arms swinging loosely at his sides, sun warm on his face, his weight rolling heel to toe over the crunching gravel with every step. The camera glides low at his eye height, one slow step behind his shoulder, then another, holding him centered as sunlit lawn and fresh leaf shadows slide past behind him, then settles as he stops. At the path's edge a small monarch butterfly with bright orange wings, black veins and black-spotted wing edges darts up out of the flower bed, crosses in front of him in two quick darts, and settles into a hover at his chest height. <Subject 1>'s right foot plants mid-stride, his arms slow and rise, and his eyes lock onto the hovering butterfly as his mouth falls open; he rises onto his toes and both small hands reach toward it, fingers spreading wide, his whole body tipping forward with the reach. The butterfly hangs in the air just out of reach, its wings opening and closing slowly while his fingertips strain closer. End mid-action: both hands still reaching, fingers an inch short of the orange wings, his weight tipped onto his toes, the camera holding the butterfly and his hands in the same frame.
|
||||
overall_soundscape:
|
||||
songbirds calling in the trees, a light breeze moving through the leaves, distant playground chatter, small footsteps crunching on the gravel path.
|
||||
non_diegetic_music:
|
||||
N/A [0-240,r2v]|subject_definitions:
|
||||
<Subject 1> identity: Sam is a 9 year old boy with his blonde hair in pigtails that dresses like a baby girl; voice: Sam is a 9 year old boy with his blonde hair in pigtails that dresses like a baby girl
|
||||
<Subject 2> is a small monarch butterfly with bright orange wings, black veins, black-spotted wing edges and a thin dark body
|
||||
summary:
|
||||
[reference generation]
|
||||
retention_analysis:
|
||||
<Subject 1>: fully_preserved
|
||||
<Subject 2>: fully_preserved
|
||||
detailed_description:
|
||||
Bright photoreal park realism on a warm late morning, sunlit green lawn and pale gravel path under fresh leaf shadows, the butterfly's orange the one saturated accent in a soft natural palette.
|
||||
Gentle camera kept low at the child's eye height, gliding smoothly alongside the walk.
|
||||
[Shot 1] Continue the same reach without resetting the motion: <Subject 1>'s fingertips close on empty air as the small monarch butterfly with bright orange wings, black veins and black-spotted wing edges snaps its wings shut, drops a hand-width, and darts away over the lawn. His planted right foot pushes off the gravel, his left foot lands on the short springy grass beyond the path's edge, grass bending flat under each landing step, and he is running, three quick strides, then four, his arms pumping at his sides. The camera picks up its low glide again and tracks after him one step, then another, keeping his back and the fluttering butterfly in the same frame. He swings his right hand in a high arc at the butterfly and misses by a hand-width; it lifts, he lunges forward, and his left hand crosses over his right to clap shut on empty air. The butterfly banks around him in a tight orange loop and rises to a hover just above his reach. He staggers to a stop, both hands still fisted around nothing, his chest heaving with quick breaths, sun warm on his upturned face; his eyebrows knot, his chin dips, and his mouth begins to pull down at the corners as his hands lower slowly to his sides. End mid-action: the butterfly hanging in a steady hover above his head, his lower lip starting to tremble, his shoulders beginning to sink.
|
||||
overall_soundscape:
|
||||
the same songbirds and light leaf-breeze continue as the gravel crunch gives way to a soft grass swish under running steps, quick panting breaths rise, each swipe of his hands whiffs through the air, distant playground chatter still drifting.
|
||||
non_diegetic_music:
|
||||
N/A [240-480,r2v]|subject_definitions:
|
||||
<Subject 1> identity: Sam is a 9 year old boy with his blonde hair in pigtails that dresses like a baby girl; voice: Sam is a 9 year old boy with his blonde hair in pigtails that dresses like a baby girl
|
||||
<Subject 2> is a small monarch butterfly with bright orange wings, black veins, black-spotted wing edges and a thin dark body
|
||||
summary:
|
||||
[reference generation]
|
||||
retention_analysis:
|
||||
<Subject 1>: fully_preserved
|
||||
<Subject 2>: fully_preserved
|
||||
detailed_description:
|
||||
Bright photoreal park realism on a warm late morning, sunlit green lawn and pale gravel path under fresh leaf shadows, the butterfly's orange the one saturated accent in a soft natural palette.
|
||||
Gentle camera kept low at the child's eye height, gliding smoothly alongside the walk.
|
||||
[Shot 1] Continue the same sinking without resetting the motion: <Subject 1>'s fists come undone at his sides, his shoulders drop the last of their fight, and his knees fold under him; he drops his weight straight down, his bottom landing on the short grass with a soft thump, both hands slapping the lawn beside him to steady himself. The camera glides one slow arc from behind his shoulder around to his front, settles at his seated eye height, and holds still. In front of him the small monarch butterfly with bright orange wings, black veins and black-spotted wing edges rises in two slow wingbeats, drifts back over the flower bed, and shrinks to a fleck of orange among the blossoms. <Subject 1> watches it go with his mouth crumpling; then his eyes squeeze shut, his mouth opens wide in the first wail, and he cries like an infant, tears spilling over his lower lashes and rolling down both cheeks, sun warm on his tear-streaked face. His right fist rubs hard across his left eye, then his left fist across his right eye, his shoulders hitching with each sob, his head tipping back on the big wails and dropping forward between them, his heels digging into the grass and pulling close to his bottom. He stays seated in the grass, crying, tears streaking to his chin, the flower bed across the lawn holding only the spot where the butterfly vanished.
|
||||
overall_soundscape:
|
||||
the same songbirds and light leaf-breeze continue, the grass swishing once as he sinks down, then the first wail rising over the ambience with hitching sobs carrying on, distant playground chatter still drifting.
|
||||
non_diegetic_music:
|
||||
N/A [480-720,r2v]
|
||||
17
easy-media-track-gate/examples/segment_01.txt
Normal file
17
easy-media-track-gate/examples/segment_01.txt
Normal file
@ -0,0 +1,17 @@
|
||||
subject_definitions:
|
||||
<Subject 1> identity: Sam is a 9 year old boy with his blonde hair in pigtails that dresses like a baby girl; voice: Sam is a 9 year old boy with his blonde hair in pigtails that dresses like a baby girl
|
||||
<Subject 2> is a small monarch butterfly with bright orange wings, black veins, black-spotted wing edges and a thin dark body
|
||||
summary:
|
||||
[reference generation]
|
||||
retention_analysis:
|
||||
<Subject 1>: fully_preserved
|
||||
<Subject 2>: fully_preserved
|
||||
detailed_description:
|
||||
Bright photoreal park realism on a warm late morning, sunlit green lawn and pale gravel path under fresh leaf shadows, the butterfly's orange the one saturated accent in a soft natural palette.
|
||||
Gentle camera kept low at the child's eye height, gliding smoothly alongside the walk.
|
||||
[Shot 1] <Subject 1> strolls along the pale gravel path through the sunlit park, his arms swinging loosely at his sides, sun warm on his face, his weight rolling heel to toe over the crunching gravel with every step. The camera glides low at his eye height, one slow step behind his shoulder, then another, holding him centered as sunlit lawn and fresh leaf shadows slide past behind him, then settles as he stops. At the path's edge a small monarch butterfly with bright orange wings, black veins and black-spotted wing edges darts up out of the flower bed, crosses in front of him in two quick darts, and settles into a hover at his chest height. <Subject 1>'s right foot plants mid-stride, his arms slow and rise, and his eyes lock onto the hovering butterfly as his mouth falls open; he rises onto his toes and both small hands reach toward it, fingers spreading wide, his whole body tipping forward with the reach. The butterfly hangs in the air just out of reach, its wings opening and closing slowly while his fingertips strain closer. End mid-action: both hands still reaching, fingers an inch short of the orange wings, his weight tipped onto his toes, the camera holding the butterfly and his hands in the same frame.
|
||||
overall_soundscape:
|
||||
songbirds calling in the trees, a light breeze moving through the leaves, distant playground chatter, small footsteps crunching on the gravel path.
|
||||
non_diegetic_music:
|
||||
N/A
|
||||
# handoff: Sam stands tipped onto his toes at the path's edge, both hands still reaching, fingers an inch short of the hovering butterfly's orange wings, the camera holding low at his eye height with the butterfly and his hands in the same frame.
|
||||
17
easy-media-track-gate/examples/segment_02.txt
Normal file
17
easy-media-track-gate/examples/segment_02.txt
Normal file
@ -0,0 +1,17 @@
|
||||
subject_definitions:
|
||||
<Subject 1> identity: Sam is a 9 year old boy with his blonde hair in pigtails that dresses like a baby girl; voice: Sam is a 9 year old boy with his blonde hair in pigtails that dresses like a baby girl
|
||||
<Subject 2> is a small monarch butterfly with bright orange wings, black veins, black-spotted wing edges and a thin dark body
|
||||
summary:
|
||||
[reference generation]
|
||||
retention_analysis:
|
||||
<Subject 1>: fully_preserved
|
||||
<Subject 2>: fully_preserved
|
||||
detailed_description:
|
||||
Bright photoreal park realism on a warm late morning, sunlit green lawn and pale gravel path under fresh leaf shadows, the butterfly's orange the one saturated accent in a soft natural palette.
|
||||
Gentle camera kept low at the child's eye height, gliding smoothly alongside the walk.
|
||||
[Shot 1] Continue the same reach without resetting the motion: <Subject 1>'s fingertips close on empty air as the small monarch butterfly with bright orange wings, black veins and black-spotted wing edges snaps its wings shut, drops a hand-width, and darts away over the lawn. His planted right foot pushes off the gravel, his left foot lands on the short springy grass beyond the path's edge, grass bending flat under each landing step, and he is running, three quick strides, then four, his arms pumping at his sides. The camera picks up its low glide again and tracks after him one step, then another, keeping his back and the fluttering butterfly in the same frame. He swings his right hand in a high arc at the butterfly and misses by a hand-width; it lifts, he lunges forward, and his left hand crosses over his right to clap shut on empty air. The butterfly banks around him in a tight orange loop and rises to a hover just above his reach. He staggers to a stop, both hands still fisted around nothing, his chest heaving with quick breaths, sun warm on his upturned face; his eyebrows knot, his chin dips, and his mouth begins to pull down at the corners as his hands lower slowly to his sides. End mid-action: the butterfly hanging in a steady hover above his head, his lower lip starting to tremble, his shoulders beginning to sink.
|
||||
overall_soundscape:
|
||||
the same songbirds and light leaf-breeze continue as the gravel crunch gives way to a soft grass swish under running steps, quick panting breaths rise, each swipe of his hands whiffs through the air, distant playground chatter still drifting.
|
||||
non_diegetic_music:
|
||||
N/A
|
||||
# handoff: Sam stands empty-handed on the lawn with both fists undone at his sides, the butterfly hanging in a steady hover just above his head, his lower lip starting to tremble and his shoulders beginning to sink while the camera holds at his eye height.
|
||||
17
easy-media-track-gate/examples/segment_03.txt
Normal file
17
easy-media-track-gate/examples/segment_03.txt
Normal file
@ -0,0 +1,17 @@
|
||||
subject_definitions:
|
||||
<Subject 1> identity: Sam is a 9 year old boy with his blonde hair in pigtails that dresses like a baby girl; voice: Sam is a 9 year old boy with his blonde hair in pigtails that dresses like a baby girl
|
||||
<Subject 2> is a small monarch butterfly with bright orange wings, black veins, black-spotted wing edges and a thin dark body
|
||||
summary:
|
||||
[reference generation]
|
||||
retention_analysis:
|
||||
<Subject 1>: fully_preserved
|
||||
<Subject 2>: fully_preserved
|
||||
detailed_description:
|
||||
Bright photoreal park realism on a warm late morning, sunlit green lawn and pale gravel path under fresh leaf shadows, the butterfly's orange the one saturated accent in a soft natural palette.
|
||||
Gentle camera kept low at the child's eye height, gliding smoothly alongside the walk.
|
||||
[Shot 1] Continue the same sinking without resetting the motion: <Subject 1>'s fists come undone at his sides, his shoulders drop the last of their fight, and his knees fold under him; he drops his weight straight down, his bottom landing on the short grass with a soft thump, both hands slapping the lawn beside him to steady himself. The camera glides one slow arc from behind his shoulder around to his front, settles at his seated eye height, and holds still. In front of him the small monarch butterfly with bright orange wings, black veins and black-spotted wing edges rises in two slow wingbeats, drifts back over the flower bed, and shrinks to a fleck of orange among the blossoms. <Subject 1> watches it go with his mouth crumpling; then his eyes squeeze shut, his mouth opens wide in the first wail, and he cries like an infant, tears spilling over his lower lashes and rolling down both cheeks, sun warm on his tear-streaked face. His right fist rubs hard across his left eye, then his left fist across his right eye, his shoulders hitching with each sob, his head tipping back on the big wails and dropping forward between them, his heels digging into the grass and pulling close to his bottom. He stays seated in the grass, crying, tears streaking to his chin, the flower bed across the lawn holding only the spot where the butterfly vanished.
|
||||
overall_soundscape:
|
||||
the same songbirds and light leaf-breeze continue, the grass swishing once as he sinks down, then the first wail rising over the ambience with hitching sobs carrying on, distant playground chatter still drifting.
|
||||
non_diegetic_music:
|
||||
N/A
|
||||
# handoff: Final frame: Sam stays seated in the grass crying, tears streaking to his chin, heels pulled close to his bottom, the flower bed across the lawn holding only the spot where the butterfly vanished.
|
||||
582
easy-media-track-gate/references/track-schema.md
Executable file
582
easy-media-track-gate/references/track-schema.md
Executable file
@ -0,0 +1,582 @@
|
||||
# MultiTrack Data and Orchestration Rules (English reference)
|
||||
|
||||
English reference for the `easy multiTrackEditor` TRACK_DATA and the patch plan, translated
|
||||
from the node pack's own skill doc (`skills/easy-media-multitrack-workflow/references/multitrack-schema.md`
|
||||
in ComfyUI-Easy-Media, which is written in Chinese). Front-end types live in
|
||||
`frontend/src/types/multitrack.ts`; execution logic lives in `nodes/basic.py` of the node pack.
|
||||
Field names are preserved exactly as the schema spells them.
|
||||
|
||||
## Resolution (DynamicCombo)
|
||||
|
||||
`resolution` is a DynamicCombo widget of `easy multiTrackEditor`. It is NOT part of
|
||||
TRACK_DATA. Changing it requires rebuilding the sub-widgets for the chosen mode and syncing
|
||||
the workflow's existing `widgets_values_named`. Write it through `patch_workflow.py`'s
|
||||
`editor.resolution` — never by guessing array positions.
|
||||
|
||||
| Need | `resolution` label | Required sub-fields |
|
||||
| --- | --- | --- |
|
||||
| Fixed preset output size | e.g. `1344 x 768 (16:9)` | optional `resize_method`, default `stretch` |
|
||||
| Explicit custom size | `width x height (custom)` | `width`, `height`, optional `resize_method` |
|
||||
| Follow first video's ratio | `width x height (auto)` | optional `resize_method` |
|
||||
| Fixed short/long edge | `width x height (shortest)` / `width x height (longest)` | `resize_to_pixel`, optional `resize_method` |
|
||||
| Ratio + total pixel budget | `width x height (megapixels)` | `aspect_ratio`, `megapixels` |
|
||||
|
||||
Selection order: exact width/height from the user → an existing fixed preset, else custom;
|
||||
only orientation or aspect + quality/pixel budget given → megapixels; keep source video
|
||||
ratio → auto; "short/long edge = N" → shortest/longest. If the user expresses no resolution
|
||||
intent, keep the current setting. The model format's default size is only a reference and
|
||||
never overrides an explicit user requirement.
|
||||
|
||||
### MP and ratio in natural language
|
||||
|
||||
Case-insensitive `mp`, `Mpx`, `megapixel(s)` (and Chinese "百万像素") mean the megapixels
|
||||
unit. `0.9mp`, `1 MP`, `0.8 megapixels` select megapixels mode and the number is taken
|
||||
verbatim. A ratio in the same sentence maps to the full DynamicCombo `aspect_ratio` value:
|
||||
|
||||
| User ratio | `aspect_ratio` |
|
||||
| --- | --- |
|
||||
| `1:1` | `1:1 (Square)` |
|
||||
| `2:3` | `2:3 (Portrait Photo)` |
|
||||
| `3:2` | `3:2 (Photo)` |
|
||||
| `3:4` | `3:4 (Portrait Standard)` |
|
||||
| `4:3` | `4:3 (Standard)` |
|
||||
| `9:16` | `9:16 (Portrait Widescreen)` |
|
||||
| `16:9` | `16:9 (Widescreen)` |
|
||||
| `21:9` | `21:9 (Ultrawide)` |
|
||||
|
||||
Example: "generate a 16:9 0.9mp video" normalizes with no follow-up question to:
|
||||
|
||||
```json
|
||||
{
|
||||
"resolution": "width x height (megapixels)",
|
||||
"aspect_ratio": "16:9 (Widescreen)",
|
||||
"megapixels": 0.9
|
||||
}
|
||||
```
|
||||
|
||||
Never convert the MP number into width/height and never round it. MP only: if already in
|
||||
megapixels mode keep its `aspect_ratio`; else pick `16:9`/`9:16` from explicit
|
||||
landscape/portrait wording; still unclear → ask. Contradictory exact size, ratio and MP →
|
||||
ask. Custom example:
|
||||
|
||||
```json
|
||||
{
|
||||
"resolution": "width x height (custom)",
|
||||
"width": 1024,
|
||||
"height": 576,
|
||||
"resize_method": "crop"
|
||||
}
|
||||
```
|
||||
|
||||
Valid `resize_method`: `stretch`, `resize`, `pad`, `pad (white)`, `pad_edge`,
|
||||
`pad_edge_pixel`, `crop`, `pillarbox_blur`. Prefer pad/resize to keep the whole picture,
|
||||
crop to fill the canvas; never change an existing resize method the user did not mention.
|
||||
|
||||
## Time semantics
|
||||
|
||||
- `start_frame` is inclusive, `end_frame` exclusive; segment frames = `end_frame - start_frame`.
|
||||
- Seconds → frames: `round(seconds * frame_rate)`; every segment has at least 1 frame.
|
||||
- Segments in one track are sorted by `start_frame` and never overlap; gaps are allowed.
|
||||
Different tracks of the same kind schedule independently.
|
||||
- With segments present, `total_length` should be the maximum `end_frame` over all tracks.
|
||||
With no segments the editor's current default length may stay.
|
||||
- Changing FPS must re-map every segment boundary, `origin_start_frame` and task marker
|
||||
proportionally — not just `frame_rate`.
|
||||
|
||||
## Top-level TRACK_DATA object
|
||||
|
||||
```json
|
||||
{
|
||||
"muted": false,
|
||||
"volume_db": 0,
|
||||
"task_markers": [],
|
||||
"task_overview": false,
|
||||
"tracks": [],
|
||||
"total_length": 240,
|
||||
"frame_rate": 24
|
||||
}
|
||||
```
|
||||
|
||||
Track `type` is one of `task`, `video`, `audio`, `subtitle`. Keep unknown fields for
|
||||
forward compatibility.
|
||||
|
||||
## Common track and segment shape
|
||||
|
||||
```json
|
||||
{
|
||||
"id": "uuid",
|
||||
"name": "Video 0",
|
||||
"type": "video",
|
||||
"color": "var(--primary)",
|
||||
"muted": false,
|
||||
"solo": false,
|
||||
"volume_db": 0,
|
||||
"locked": false,
|
||||
"segments": [
|
||||
{
|
||||
"id": "uuid",
|
||||
"start_frame": 0,
|
||||
"end_frame": 120,
|
||||
"color": "var(--primary)",
|
||||
"content": {}
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
New tracks use the project's existing color tokens: task tracks `var(--multitrack-task-bg)`,
|
||||
video tracks `var(--primary)`. Keep an existing track's own color.
|
||||
|
||||
## Task segment content
|
||||
|
||||
```json
|
||||
{
|
||||
"media_type": "none",
|
||||
"task_mode": "ref",
|
||||
"continuity_mode": "shot",
|
||||
"ref_image_size": "match",
|
||||
"images": [],
|
||||
"user_prompt": "the user prompt",
|
||||
"system_prompt": "",
|
||||
"user_prompt_variant": "a",
|
||||
"user_prompt_b": "",
|
||||
"muted": false,
|
||||
"volume_db": 0
|
||||
}
|
||||
```
|
||||
|
||||
- `task_mode`: `default`, `l2v`, `ref`, `edit`.
|
||||
- RefMod workflows (identity driven by `MiniMaxH3RefModsLoader` + `MiniMaxH3RefModApply`
|
||||
outside the timeline): author segments as `task_mode: "ref"` even with no per-segment
|
||||
images — the payload then carries `generation_type: "r2v"`, TaskOutput TYPE becomes
|
||||
`r2v`, and the matchLine/promptLine pair switches `minimaxH3ToVideo` to `reference`
|
||||
mode, which pairs with the RefMod conditioning. `ref` rows with `images: []` are
|
||||
legal (refmod pictures ride the conditioning, not the timeline); prompts stay
|
||||
picture-token-free. Verified live by the user: the relay one-pass render works with
|
||||
r2v and refmods "work better with ref2va (r2v)".
|
||||
- `continuity_mode`: `shot` (standalone shot), `context` (normal context join) or
|
||||
`context_drift` (context drift control); the legacy value `context_swap` is still read.
|
||||
- `ref_image_size`: `match` or `max`.
|
||||
- `images` holds at most 9 items; their order defines the picture-reference order.
|
||||
- When editing an existing task keep its `system_prompt`, prompt variant, continuity and
|
||||
reference-image policy unless the user asked to change them.
|
||||
- If the current variant is `b`, edit `user_prompt_b`; otherwise edit `user_prompt`. Never
|
||||
switch the variant unintentionally.
|
||||
|
||||
Task reference image item:
|
||||
|
||||
```json
|
||||
{
|
||||
"id": "uuid",
|
||||
"source_type": "input",
|
||||
"file_path": "codex/session-123/reference.png",
|
||||
"file_name": "reference.png"
|
||||
}
|
||||
```
|
||||
|
||||
`source_type` is one of `input`, `output`, `local`, `url`, `slot`. The path field must
|
||||
match the source: `input`/`output` use `file_path`, `local` uses `local_path`, `url` uses
|
||||
`url`, `slot` uses `slot_name`. Never label a local absolute path as `input`.
|
||||
|
||||
When several context segments reuse the same reference image, an image item may set
|
||||
`"shared_reference": true`; it becomes a common image that is automatically ordered before
|
||||
each task's private images; common + private images still total at most 9 per task. The
|
||||
current front-end `synchronizeSharedTaskImages` generates new UUIDs for missing common-image
|
||||
metadata on later tasks, so repeatedly normalizing the same sparse raw data yields different
|
||||
IDs; from this code alone the full React maximum-update-depth / widget-unmount trigger chain
|
||||
cannot be confirmed. Until the target front-end is verified to load such data stably, store
|
||||
an explicit private image item per task (`shared_reference: false`) with a fixed ID pointing
|
||||
at the same uploaded file — no duplication or re-upload needed. Switching continuity mode
|
||||
does not fix this.
|
||||
|
||||
## Context drift control mode
|
||||
|
||||
`context_drift` is the "context drift control" join: it carries the previous segment's scene
|
||||
and motion context while letting the model replace subjects, people, objects or局部 regions
|
||||
from a reference video. When the user supplies a video as a subject/person/object/clothing/face
|
||||
replacement reference, orchestrate as follows:
|
||||
|
||||
- Set `content.continuity_mode` to `context_drift` on ALL task segments in scope — not only
|
||||
the second and later ones. The first segment keeps the same mode too (it has no previous
|
||||
generated context, but consistency matters for project metadata and re-generation).
|
||||
- Put the reference video on a video track covering the target task segments that need motion
|
||||
reference. If the project also has passages that do not depend on the dance/reference video
|
||||
(e.g. a verse), keep one complete project — the full locked audio covers all tasks and the
|
||||
video only covers its part. If the video is also needed across tasks as a model reference,
|
||||
set `shared_reference: true` on the video segment's `content`.
|
||||
- Set track-level `audio_locked: true` on that video track so the source video's timeline
|
||||
drives generation, and explicitly set `audio_locked: false` on every OTHER video track. To
|
||||
replace the original sound, lock a separate audio track, which then wins as the audio
|
||||
source. When only reusing the video's own sound, confirm before locking that the video
|
||||
actually contains a usable audio track and covers all target task segments.
|
||||
- If the user pinned segment lengths or boundaries, the user wins. Otherwise split task
|
||||
segments into consecutive 10-second pieces: each piece is `round(10 * frame_rate)` frames
|
||||
and the last piece takes the reference video's remaining frames — do not pad, do not drop
|
||||
the remainder, do not create an empty tail segment. The reference video itself can stay a
|
||||
single video segment covering the whole range; it does not need to be duplicated per task
|
||||
segment.
|
||||
- If the user explicitly asked to drop/replace/not keep the reference video's original sound,
|
||||
do not auto-lock that video track and follow the user's audio plan instead.
|
||||
|
||||
## Context segment prompts
|
||||
|
||||
Applies to tasks with `task_mode: "ref"` and `continuity_mode: "context"`;
|
||||
`context_drift` (and legacy `context_swap`) do not use this section. Such a segment starts at
|
||||
the previous segment's actually delivered tail (22 frames ≈ 0.92 s at 24 fps) whose framing
|
||||
and camera motion may differ from what the previous prompt imagined — and those frames are
|
||||
locked: text contradicting them tends to render as an overlay, not a replacement. When
|
||||
writing or editing `user_prompt`:
|
||||
|
||||
- **Keep first, change later**: open with only "the camera keeps the direction and framing it
|
||||
arrived with; the subject keeps the pose, motion and light it arrived with". Never describe
|
||||
the previous segment's INTENDED ending as fact, and never open with a new composition,
|
||||
character layout or scene.
|
||||
- **Give the hold an action**: give the subject a breath, a weight shift or a gaze change so
|
||||
the hold does not render as a freeze-frame.
|
||||
- **Changes go later**: before about `00:02.000` no cut, no spoken line, no new key action;
|
||||
write a required composition or scene change as an explicit cut.
|
||||
- **Start actions mid-motion**: things already in progress at the seam start in motion —
|
||||
never as if completing on frame one.
|
||||
- **Self-contained**: the model never sees the previous prompt; fully define every `<Subject N>`,
|
||||
state scene and light, and never say "the same one" or "as before" for identity or scene.
|
||||
- **Images define identity only**: a reference image is not the first frame or a composition
|
||||
anchor of `[Shot 1]`; context joins use neither image anchoring nor `[video continuation]`.
|
||||
- **When more `context` segments follow, end in motion**: never write "ends", "arrives",
|
||||
"settles", "centers", "slows down"; no cut and no new spoken line in the last second.
|
||||
- **Need a clearly new composition**: use a `shot` segment with keyframes instead of `context`.
|
||||
- **Timestamps**: timestamps in the prompt map onto the generated segment, whose start sits
|
||||
about 0.92 s earlier than the saved segment; the node does not shift them automatically.
|
||||
|
||||
When `system_prompt` is empty, the task output node automatically appends the same rules to
|
||||
the default MiniMax reference system prompt (entry rule: this segment is not the first and
|
||||
uses `context`; closing rule: the next segment uses `context`). The rules reach the rewriting
|
||||
LLM only through the `api` and `llm` prompt formats; `default` and `promptRelay` emit no
|
||||
system prompt. A custom `system_prompt` disables the auto-append — write the rules yourself.
|
||||
|
||||
## Video and audio segment content
|
||||
|
||||
```json
|
||||
{
|
||||
"media_type": "video",
|
||||
"source_type": "input",
|
||||
"file_path": "codex/session-123/clip.mp4",
|
||||
"file_name": "clip.mp4",
|
||||
"duration": 5.0,
|
||||
"muted": false,
|
||||
"volume_db": 0,
|
||||
"speed": 1
|
||||
}
|
||||
```
|
||||
|
||||
Audio sets `media_type` to `audio`. Get source durations with `ffprobe`; the timeline
|
||||
duration follows the user's target, trimming and speed. If media must align with a task
|
||||
segment, reuse the exact same `[start_frame, end_frame)` — never just similar seconds.
|
||||
|
||||
## Audio lock decisions
|
||||
|
||||
`audio_locked` is a track-level field on audio/video tracks. Locking an audio track keeps
|
||||
its audio; locking a video track makes generation follow that video track's timeline and,
|
||||
with no locked audio track, keeps the video's own sound. Both locks can coexist; the audio
|
||||
track then wins as the `audio_lock` audio source while the video track still constrains the
|
||||
picture timeline. It is different from the track's `locked` flag: `locked` only blocks
|
||||
timeline editing and cannot substitute for `audio_locked`.
|
||||
|
||||
```json
|
||||
{
|
||||
"id": "music-track",
|
||||
"name": "Master Song",
|
||||
"type": "audio",
|
||||
"audio_locked": true,
|
||||
"muted": false,
|
||||
"segments": [
|
||||
{
|
||||
"id": "song-segment",
|
||||
"start_frame": 0,
|
||||
"end_frame": 240,
|
||||
"content": {
|
||||
"media_type": "audio",
|
||||
"source_type": "input",
|
||||
"file_path": "codex/mv/song.wav",
|
||||
"file_name": "song.wav"
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
A complete locked audio track can coexist with a partial locked video track. At runtime, when
|
||||
the current task has no intersection with the video, the video is simply not passed in and
|
||||
the master audio is still used — never split the project, extend the video, or stretch audio
|
||||
just to satisfy validation. Validation allows a partial video lock only when the locked audio
|
||||
continuously covers the selected tasks (multiple end-to-end audio segments are fine); without
|
||||
a complete master audio it still checks the locked media's intersection with each task.
|
||||
|
||||
One TRACK_DATA may contain at most one `audio_locked: true` audio track and one
|
||||
`audio_locked: true` video track. When switching the locked source of one type, explicitly set
|
||||
the other tracks of that same type to `false`; never clear the other type's lock. Never write
|
||||
`audio_locked` into a media segment's `content`; the legacy segment-level field only exists
|
||||
for migration.
|
||||
|
||||
Decide in this order, from context:
|
||||
|
||||
1. The user supplies a video for subject/partial replacement and did not ask to remove or
|
||||
replace its sound → lock that video track.
|
||||
2. The user explicitly says "lock", "master audio", "keep the original sound", "don't
|
||||
regenerate audio" → use the audio/video track they mean.
|
||||
3. MV, lyric video, beat-sync video, dance video, or visuals generated to a full song's beat
|
||||
and length → lock the song/score master track.
|
||||
4. Narration, podcast or dialogue is the only basis for video length and shot rhythm → that
|
||||
narration master track may be locked.
|
||||
5. Ordinary background music, ambience, SFX, character reference audio or temporary scratch
|
||||
audio is NOT auto-treated as locked, unless the user's wording makes it the generation
|
||||
driver.
|
||||
6. One locked media track already exists and the user did not ask to change it → prefer
|
||||
keeping it; still check it intersects the task range being generated.
|
||||
|
||||
With multiple audio tracks, disambiguate first by name, file name, source duration, track
|
||||
content and the user's description — e.g. `Master Song` or 完整歌曲 (full song) beats `SFX` or
|
||||
ambience. If two or more plausible candidates remain, STOP and ask the user, listing each
|
||||
candidate's track ID, name, file name and time range. Never decide by track order, first
|
||||
audio track or longest duration.
|
||||
|
||||
Before locking also check:
|
||||
|
||||
- The locked track contains at least one media segment matching its type: audio track →
|
||||
`media_type: "audio"`, video track → `media_type: "video"`; when locking a video's own
|
||||
sound, confirm the source video really has an audio track.
|
||||
- The locked media segments intersect every target task segment's `[start_frame, end_frame)`;
|
||||
if not, adjust the timeline or ask first.
|
||||
- As master audio it normally stays `muted: false`. Never touch unrelated solo, volume or
|
||||
edit-lock state just "because locking".
|
||||
- If the context clearly wants no original audio kept, ensure every audio/video track has
|
||||
`audio_locked: false`.
|
||||
|
||||
## Shared media
|
||||
|
||||
`shared_reference` is a boolean on task image items and in audio/video segment `content`.
|
||||
When several context task segments need the same image, audio clip or video as a reference,
|
||||
mark that media shared; it does NOT need to overlap the task segments on the timeline.
|
||||
|
||||
```json
|
||||
{
|
||||
"id": "shared-audio-segment",
|
||||
"start_frame": 0,
|
||||
"end_frame": 120,
|
||||
"content": {
|
||||
"media_type": "audio",
|
||||
"source_type": "input",
|
||||
"file_path": "codex/voice/reference.wav",
|
||||
"file_name": "reference.wav",
|
||||
"shared_reference": true
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Decision rules:
|
||||
|
||||
- Mark media shared only when the user says "make it shared/common media", "all context
|
||||
segments reference this image/audio/video" or equivalent. Never auto-share just because
|
||||
file names match.
|
||||
- Task images are marked on their `content.images[]` items; several shared images are
|
||||
allowed; they are ordered before each task's private images and common + private still
|
||||
total at most 9 per task.
|
||||
- Audio/video are marked on the media segment's `content`; at most one per audio/video
|
||||
track; when switching, explicitly set the track's other segments to `false`.
|
||||
- Shared audio is used from the start of the source file, at most 15 seconds; shared video
|
||||
uses that source video as each task's reference video.
|
||||
- Shared media is a cross-task reference, not timeline-audio reuse. Never set `audio_locked`
|
||||
just because audio is shared; evaluate `audio_locked` separately, only when the user also
|
||||
wants the audio kept as-is and driving the timeline.
|
||||
- If the user wants one audio/video clip BOTH as a shared reference AND as normal timeline
|
||||
output, express the two roles with a separate track or segment and say so in the delivery
|
||||
summary.
|
||||
- `speaker_reference` is the legacy audio-reference field, only for reading and migrating old
|
||||
workflows; new or updated workflows always write `shared_reference`.
|
||||
- With multiple unresolvable candidates, list the media/segment IDs, track names, file names,
|
||||
durations or frame ranges and ask the user; never pick the first or the longest.
|
||||
|
||||
## Auto-orchestration decisions
|
||||
|
||||
- User-given shot/segment lengths win; otherwise derive from source durations or the
|
||||
prompts' timing information, and state the derivation.
|
||||
- Subject/partial video replacement mode with no user-specified lengths always splits task
|
||||
segments into consecutive 10-second pieces, the last taking the remainder.
|
||||
- "N seconds each" means consecutive `[0, N*fps)`, `[N*fps, 2N*fps)`, … — never add one extra
|
||||
frame at the boundaries.
|
||||
- Prompts go into their task segments; reference images into that task segment's `images`;
|
||||
master video and audio into their own media tracks.
|
||||
- For subject-replacement, MV or other master-audio-driven projects, after arranging the
|
||||
timeline lock the single audio/video track and re-check it covers the target task range.
|
||||
- Keep gaps between tracks by default; close gaps and shift later segments only when the
|
||||
user asks for a tight arrangement.
|
||||
- When adjusting existing segments keep their segment IDs; generate new UUIDs only for new
|
||||
segments, tracks and image items.
|
||||
|
||||
## MultiTrack Project parameters
|
||||
|
||||
- `segment_start_number` is the 1-based first task segment index; execution converts with
|
||||
`segment_start_number - 1`.
|
||||
- `segment_count = -1` processes from the start index to the end; a non-negative value limits
|
||||
how many task segments run this time.
|
||||
- `project_save`: `new` or `override`. With `override` and count `-1`, execution deletes
|
||||
saved segments after the start index before generating — say so clearly before changing it.
|
||||
- `sampling_mode`: `single` or `dual`; `sampling_plan` must be a preset available on the
|
||||
current node — never assume values from another workflow's environment.
|
||||
- The editor resolution is the first-pass size; `dual` never scales the first pass down. The
|
||||
second pass is `round(editor_size * upscale_by / 32) * 32` per dimension, Python-round to
|
||||
nearest, ties to even — not a fixed ceiling. `upscale_by` is an independent Project
|
||||
parameter, default `1.250`, three decimals, step `0.001`; old workflows keep their saved
|
||||
value. Both upsample paths and the merged export use the upscaled, aligned size. Example:
|
||||
`1344 x 768` at `1.250` gives a second pass of `1664 x 960`. If the user specifies a final
|
||||
output size, check BOTH the editor size and the multiplier; the editor size alone is not
|
||||
the dual-sampling final size.
|
||||
- Single sampling / first-pass-only preview keeps the editor size; `upscale_by = 1.000`
|
||||
upscales nothing but still allows a second pass; pure audio projects (`32 x 32`) do not
|
||||
upscale. Old first-pass checkpoints and context latents that were shrunk by the old
|
||||
back-calculation should be regenerated before continuing under the new rule.
|
||||
- `1st_pass_only` only applies to explicit one-stage checkpoint flows; never enable it
|
||||
automatically.
|
||||
|
||||
## The patch plan
|
||||
|
||||
The plan contains only actual changes. `node_id` may be omitted (topology locates the nodes);
|
||||
`editor.resolution` is the complete DynamicCombo setting; `editor.track_data` is the complete
|
||||
TRACK_DATA. Never put a whole node or workflow into the plan.
|
||||
|
||||
```json
|
||||
{
|
||||
"editor": {
|
||||
"resolution": {
|
||||
"resolution": "width x height (megapixels)",
|
||||
"aspect_ratio": "16:9 (Widescreen)",
|
||||
"megapixels": 0.9
|
||||
},
|
||||
"track_data": { "tracks": [], "total_length": 120, "frame_rate": 24 },
|
||||
"format": "MiniMax"
|
||||
},
|
||||
"project": {
|
||||
"project_save": "override",
|
||||
"segment_start_number": 1,
|
||||
"segment_count": -1,
|
||||
"sampling_plan": "medium",
|
||||
"sampling_mode": "single"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Applied with the node pack's own script:
|
||||
|
||||
```sh
|
||||
python3 scripts/patch_workflow.py apply SOURCE.json --plan PLAN.json # dry-run
|
||||
python3 scripts/patch_workflow.py apply SOURCE.json --plan PLAN.json --output RESULT.json --write
|
||||
python3 scripts/patch_workflow.py inspect RESULT.json
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## prompt_override — plain text form: `scene1 | scene2 | scene3` (the simple default)
|
||||
|
||||
The `prompt_override` input does not require any JSON. Paste plain text — one scene per
|
||||
pipe-separated chunk:
|
||||
|
||||
```
|
||||
<scene 1 text> | <scene 2 text> | <scene 3 text>
|
||||
```
|
||||
|
||||
The editor's execute (`nodes/basic.py`): a non-JSON `prompt_override` with a value goes to
|
||||
`build_multitrack_data_from_prompt_override` → `parse_override_segments`
|
||||
(`utils/prompt_override.py`), which splits on `|`, strips each part and drops empties —
|
||||
one task segment per part, a trailing `|` is a no-op. Proven by executing the pack's code
|
||||
(sandbox `scripts/prove_plain_text_override.py`):
|
||||
|
||||
- Frames: the editor's existing `total_length` (its track_data / length input) is divided
|
||||
EVENLY across the parts — there are no per-segment durations in this form. 720 total,
|
||||
3 parts → [0-240] [240-480] [480-720].
|
||||
- No pipe at all = ONE segment spanning the whole timeline (the "one giant clip" trap).
|
||||
- Task type: parts default to `flf` → task_mode `default` → TaskOutput TYPE `t2v`.
|
||||
- Optional per-part extras (inside the part text):
|
||||
- `[s-e]` frame ranges, seconds form `[1.5s-3s]` (4-frame quanta), optional type suffix
|
||||
`[0-240,r2v]` → that segment becomes task_mode `ref` / TYPE `r2v` (other types:
|
||||
`flf`, `fmlf`, `ref`, `l2v`, `v2v`, `i2v`, `rv2v`, `vi2v`). With explicit ranges the
|
||||
timeline length becomes the last end + 1. The marker sits at the END of the segment,
|
||||
is stripped from the text before it reaches the task, and newlines INSIDE a segment
|
||||
survive — only `|` splits. Worked example:
|
||||
`examples/prompt_override_r2v_refmod.txt` — three six-section r2v scene prompts
|
||||
(each carrying the Sam refmod verbatim in `subject_definitions`, no `<Picture N>`
|
||||
because the story has zero images) with `[0-240,r2v]` / `[240-480,r2v]` /
|
||||
`[480-720,r2v]` markers; proven through the pack's parser: 3 tasks TYPE `r2v`,
|
||||
frames [0-240]/[240-480]/[480-720], task-duration sum 720, text byte-identical to
|
||||
the plan shots.
|
||||
- `@image1` / `@audio1` / `@video1` attach the editor's matching media input slots
|
||||
positionally. The `<Picture N>` bracket form is NOT parsed in this path — it stays
|
||||
literal text; only the JSON payload understands it.
|
||||
- Continuity: parts become independent segments; there is no `continuity_mode` in the
|
||||
text form (context joins are a JSON/TRACK_DATA feature).
|
||||
|
||||
This is the gate's DEFAULT paste form: the merge writes `prompt_override.txt` for every
|
||||
zero-media story — scenes joined with `|`, and it appends the `[start-end]` /
|
||||
`[start-end,type]` markers itself whenever the rows have unequal durations (the
|
||||
markerless form splits the editor's `total_length` evenly) or a scene's task type differs
|
||||
from the text default (flf → t2v, e.g. `[0-240,r2v]` for task_mode ref). Use the JSON
|
||||
payload below only for what plain text cannot carry: per-segment continuity modes, track
|
||||
locks or `<Picture N>` media tokens (stories with media — the text path attaches media
|
||||
via `@imageN`/`@audioN`/`@videoN` only, and the gate validates both forms against the
|
||||
approved guide).
|
||||
|
||||
---
|
||||
|
||||
## prompt_override — JSON payload form (optional: per-segment durations, modes, locks)
|
||||
|
||||
The `easy multiTrackEditor` node has an optional `prompt_override` input. When the editor's
|
||||
`format` is `MiniMax` and the pasted value is a `minimax_prompt_override` JSON payload, the
|
||||
editor rebuilds its whole task track from that payload: one task segment per `prompts`
|
||||
entry, laid back-to-back on a cursor, each spanning `round(seconds × frame_rate)` frames
|
||||
derived from its duration. **The payload has no frame fields and needs none** — never put
|
||||
`start_frame`/`end_frame` in it. `total_length` becomes the sum of the durations, so a
|
||||
3-prompt payload at the default 10s each renders as THREE chained ~10-second clips.
|
||||
|
||||
**The parser requires exactly two keys** (`parse_minimax_prompt_override`,
|
||||
`utils/prompt_override.py`: it checks `type` and that `prompts` is a list — nothing else):
|
||||
|
||||
```json
|
||||
{"type":"minimax_prompt_override","prompts":["prompt for segment one ...","prompt for segment two ..."]}
|
||||
```
|
||||
|
||||
Every other field is OPTIONAL with a code default — write a key only when the story needs
|
||||
a non-default value:
|
||||
|
||||
| key | default | meaning |
|
||||
| --- | --- | --- |
|
||||
| `duration` | `"10"` | SECONDS per clip (never frames), 2–15 enforced at parse time; single value or comma list with tail reuse |
|
||||
| `generation_type` | `"r2v"` | per clip, same single/comma rule: `r2v` (task_mode ref) / `l2v` / `t2v`,`i2v`,`v2v` (default-mode tasks). TaskOutput's TYPE echoes this string verbatim |
|
||||
| `continuity_mode` | `"shot"` | per clip: `shot` / `context` / `context_drift` (legacy `context_swap` is normalized to `context_drift`) |
|
||||
| `system_prompt` | `""` | written into every clip's content when non-empty |
|
||||
| `video_track_lock` | `0` | 1-based index of the video track to lock (0 = none) |
|
||||
| `audio_track_lock` | `0` | 1-based index of the audio track to lock (0 = none) |
|
||||
|
||||
- `version` — the pack's own `easy minimaxPromptOverride` builder node writes
|
||||
`"version": 1`, but **nothing in the pack reads it**; the parser never touches it.
|
||||
Do not emit it.
|
||||
- The code default for `generation_type` is **`r2v`** (reference mode) — a bare minimal
|
||||
payload already renders as r2v/reference tasks with no key at all, which is why the
|
||||
gate's refmod guidance needs no explicit generation_type.
|
||||
|
||||
Fallback warning (unchanged): **any other JSON pasted here is not rejected — it falls back
|
||||
to the generic text parser that splits on `|`.** A JSON blob contains no `|`, so it becomes
|
||||
ONE task segment spanning the whole timeline with the JSON dump as its prompt: the classic
|
||||
"rendered the whole story as a single clip" failure.
|
||||
|
||||
- `prompts` — one entry per clip, in timeline order. The prompt text is the clip's full
|
||||
generation prompt (same standalone composition rules as task segments: no cross-segment
|
||||
references, context joins keep-then-change, only the last clip may come to rest).
|
||||
- Media attaches ONLY through tokens in the prompt text: `<Picture N>`, `<Audio N>`,
|
||||
`<Video N>` (the `@imageN`/`@audioN`/`@videoN` forms are accepted and rewritten, but this
|
||||
gate standardizes on the bracket form). N is the 1-based position of the item in the
|
||||
editor's matching `image` / `audio` / `video` input list — wire the media inventory into
|
||||
those inputs in inventory order. A prompt with no token renders with no reference media.
|
||||
The runtime renumbers tokens compactly for display, but slots are wired from the raw N.
|
||||
|
||||
The paste replaces the editor's existing task track (and derives its audio/video tracks
|
||||
from the tokens), so the payload is self-contained: prompts, durations, modes and media
|
||||
references all travel inside it. Edit task modes (`edit`) and full extra tracks are beyond
|
||||
this payload — use the TRACK_DATA patch plan for those.
|
||||
202
easy-media-track-gate/references/workflow_wiring.md
Normal file
202
easy-media-track-gate/references/workflow_wiring.md
Normal file
@ -0,0 +1,202 @@
|
||||
# Workflow wiring: render ALL scenes in one run + the LLM slot
|
||||
|
||||
Verified against ComfyUI-Easy-Media source (`nodes/basic.py`, `nodes/minimax.py`,
|
||||
`nodes/project.py`, `utils/llm_api.py`) and `example_workflows/MinimaxH3_ForLoop_MultiShot.json`.
|
||||
|
||||
## Why only scene 1 renders
|
||||
|
||||
The render chain reads the timeline through `easy multiTrackTaskOutput`, which outputs
|
||||
**one task segment per pass**, picked by its `task_index` int input. Default `0` →
|
||||
the first clip only. The number of scenes inside the pasted `prompt_override` (plain
|
||||
pipe text or the JSON payload) does not change this: the paste is complete (all scenes
|
||||
are in it); the loop lives in the workflow, not in the paste. Nothing needs to be added
|
||||
to it for multi-shot.
|
||||
|
||||
Side facts (same node, verified in source):
|
||||
|
||||
- `task_index = -1` switches the node to **full-timeline mode**: window 0..720,
|
||||
LENGTH = all frames snapped to the H3 17k+5 grid (720 → 719), media from all
|
||||
segments. What the prompt becomes depends on `prompt_format` — see the
|
||||
single-pass section below (`promptRelay` makes it render all scenes in one pass —
|
||||
which also puts every frame into ONE sampler call; OOM ceiling below).
|
||||
- `easy multiTrackInfoOutput` (tracks_info in) exposes `TASK_COUNT` = number of task
|
||||
segments — 3 for the Sam payload. Also WIDTH/HEIGHT/TOTAL_FRAMES/FPS.
|
||||
|
||||
## The prompt_override input takes TWO forms (proven)
|
||||
|
||||
`MultiTrackEditor.execute` (`nodes/basic.py` L1954-1957): with `format = "MiniMax"` it
|
||||
calls `is_minimax_prompt_override(prompt_override)` FIRST. Only a payload with
|
||||
`"type": "minimax_prompt_override"` and a `prompts` list takes the JSON path
|
||||
(`utils/prompt_override.py` `parse_minimax_prompt_override`). Anything else with a value
|
||||
falls to the generic `|`-text parser — which is a legitimate form in its own right, not
|
||||
just a failure mode (`track-schema.md` "plain text form"):
|
||||
|
||||
- Plain pipe text `scene1|scene2|scene3` → N task segments, evenly sliced over the
|
||||
editor's `total_length` (proven by executing the pack's code,
|
||||
`scripts/prove_plain_text_override.py`). This is the DEFAULT form — this gate's
|
||||
`prompt_override.txt` — with optional `[start-end(,type)]` markers per scene for
|
||||
explicit frame ranges / non-t2v types (`examples/prompt_override_r2v_refmod.txt`).
|
||||
- A full TRACKS_INFO JSON contains no `|` → ONE segment whose prompt is the literal
|
||||
JSON dump (STEP A of `prove_relay_chain.py`): the garbage single-clip failure. So:
|
||||
paste plain text (default) or the minimal `minimax_prompt_override` JSON payload
|
||||
(media tokens / continuity modes / locks) into `prompt_override`; full TRACKS_INFO
|
||||
belongs only in `track_data`.
|
||||
|
||||
## Single-pass timeline render: task_index=-1 + prompt_format=promptRelay (short timelines only)
|
||||
|
||||
This is the pattern the user's own graph (`my timeline ref.json`) is wired for —
|
||||
no ForLoop, one sampler pass, all scenes in one generation. It renders, but only
|
||||
when the WHOLE timeline fits in one pass — the OOM ceiling at the end of this
|
||||
section is a hard limit:
|
||||
|
||||
1. Node 186 (String → `prompt_override`): the minimal payload — `type` + `prompts` are
|
||||
the only required keys (`track-schema.md`); optional keys only when non-default.
|
||||
2. `easy multiTrackTaskOutput`: `task_index = -1` AND `prompt_format = "promptRelay"`.
|
||||
3. Everything else stays as-is: TYPE output is the payload's `generation_type` string
|
||||
verbatim (`task_type` is set from it in the override path, and the parser's code
|
||||
default is `r2v` — so even the minimal 2-key payload renders TYPE `r2v` → matchLine →
|
||||
`multi_frames`), LENGTH output becomes 719, one SamplerCustomAdvanced pass renders the
|
||||
whole timeline.
|
||||
|
||||
With `promptRelay` + full-timeline mode, TaskOutput builds the prompt from ALL
|
||||
task segments (`_format_marker_task_prompt_relay`), each scene suffixed with its
|
||||
native H3 frame range, joined by ` | `:
|
||||
|
||||
```
|
||||
<scene 1 text> [0-240] | <scene 2 text> [240-480] | <scene 3 text> [480-720]
|
||||
```
|
||||
|
||||
SYSTEM_PROMPT output is empty in this mode; the ranges are the H3 timeline
|
||||
syntax (`_FRAME_RANGE_RE` parses the same `[s-e]` / `[s-e,s-type]` tokens in the
|
||||
override parser). Proven by executing the repo's own functions on the Sam
|
||||
payload (STEP C: 3 ranges, length 719, type t2v).
|
||||
|
||||
Both settings are required — `task_index = -1` with `prompt_format = "default"`
|
||||
keeps only the FIRST task's prompt (STEP D): a 719-frame latent conditioned on
|
||||
scene 1 alone.
|
||||
|
||||
**OOM ceiling (user-verified 2026-11):** one pass = the ENTIRE timeline's frames
|
||||
in a single sampler call. At the Sam story's 3 × 240 = 720 total frames the
|
||||
render attempted one ~30 s clip and died with an out-of-memory error — and 720
|
||||
is also ~2× the trained range (~124–362 frames, node tooltip: "longer is
|
||||
untested"), so quality was never validated that far either. The pack's own
|
||||
long-sequence plumbing (`multi_frames`, BlockSparseAttention `min_tokens 12288`)
|
||||
does not remove the VRAM cost of one 720-frame pass on a consumer card.
|
||||
|
||||
Rule: use `task_index = -1` + `promptRelay` only when the whole timeline stays
|
||||
inside the trained range — total ≤ ~360 frames, i.e. two short scenes. Anything
|
||||
longer: render per clip (next section) and join the files.
|
||||
|
||||
## The single-run loop (from the pack's own ForLoop example)
|
||||
|
||||
One queue run renders every clip and concatenates them:
|
||||
|
||||
```
|
||||
editor.TRACKS_INFO ─→ multiTrackInfoOutput ─→ TASK_COUNT ─→ forLoopStart.total
|
||||
editor.TRACKS_INFO ─→ multiTrackTaskOutput (images/audio/video too)
|
||||
forLoopStart.index ─→ multiTrackTaskOutput.task_index
|
||||
multiTrackTaskOutput.SYSTEM_PROMPT/USER_PROMPT/TYPE/LENGTH/IMAGES/AUDIO ─→ render chain
|
||||
render chain → saveVideo.file_path ─→ batchAnything ─→ forLoopEnd.initial_value1
|
||||
forLoopEnd.value1 ─→ mergeVideosFromPaths ─→ SaveVideo
|
||||
```
|
||||
|
||||
The example adds two math nodes so a single PrimitiveInt `k` (default `-1`) chooses:
|
||||
|
||||
- `total = (k<=-1)*TASK_COUNT + (k>-1)*1`
|
||||
- `task_index = (k<=-1)*loop_index + (k>-1)*k`
|
||||
|
||||
`k = -1` → loop all segments (all scenes, one run). `k = 2` → render only segment 2.
|
||||
|
||||
Manual fallback without any rewiring (verified in source, `nodes/basic.py`
|
||||
`MultiTrackTaskOutput`): keep `prompt_format = "promptRelay"` and just set
|
||||
`task_index` to 0, 1, 2 — one queue run each. The all-tasks relay fires only when
|
||||
`output_full_timeline` is set (-1) or the segment's own prompt contains `[s-e]`
|
||||
markers (`marker_mode`); for a marker-free payload, promptRelay at
|
||||
`task_index = k` formats the ONE selected clip as `<scene k text> [start-end]`
|
||||
with LENGTH = that clip's frame count (240 — inside the trained range). `default`
|
||||
gives the bare scene text if the range suffix is not wanted. Join the files.
|
||||
|
||||
Per-iteration the task output gives LENGTH ≈ frames for that clip only (240 frames =
|
||||
10 s — inside the model's trained range), so each loop pass is a normal single-clip
|
||||
generation.
|
||||
|
||||
## The LLM slot (where your own LLM connects)
|
||||
|
||||
In the ForLoop example the per-scene prompt may pass through an LLM before rendering:
|
||||
|
||||
```
|
||||
TaskOutput.SYSTEM_PROMPT ─→ [LLM node].system_prompt
|
||||
TaskOutput.USER_PROMPT ─→ [LLM node].custom_prompt
|
||||
TaskOutput.IMAGES ─→ [LLM node].images (optional, vision only)
|
||||
[LLM node].output ─→ ifElse("Using Prompt Enhance") ─→ cleanGpuUsed ─→ render.prompt
|
||||
ifElse.on_false = raw TaskOutput.USER_PROMPT (bypass switch)
|
||||
```
|
||||
|
||||
The example's LLM is `llama_cpp_instruct_adv` + `llama_cpp_model_loader` (a local GGUF).
|
||||
That node is replaceable — it is just "system+user (+images) in, one string out".
|
||||
Three ways to fill the slot:
|
||||
|
||||
### Option A — no LLM node at all (default for this skill)
|
||||
|
||||
The skill already wrote every scene prompt (identity text, continuity rules, LoRA
|
||||
trigger) into the paste (`prompt_override.txt`, or `prompt_override.json` for the JSON
|
||||
form). Leave the enhance toggle OFF: raw
|
||||
`USER_PROMPT` flows to the render node each pass. The "LLM that made all the scenes"
|
||||
is this skill, outside ComfyUI.
|
||||
|
||||
### Option B — the pack's built-in enhancer with any OpenAI-compatible server
|
||||
|
||||
Add `easy multiTrackPromptEnhancer` inside the loop:
|
||||
|
||||
- `system_prompt` ← TaskOutput.SYSTEM_PROMPT, `user_prompt` ← TaskOutput.USER_PROMPT,
|
||||
`type` ← TaskOutput.TYPE, `length` ← TaskOutput.LENGTH, images/audio/video optional.
|
||||
- `model` = `openai_compatible (local)`; set `base_url` to ANY OpenAI-compatible
|
||||
Chat Completions server — Ollama `http://127.0.0.1:11434/v1`, LM Studio
|
||||
`http://127.0.0.1:1234/v1`, vLLM, llama.cpp server, or a hosted endpoint
|
||||
(empty base_url reads `OPENAI_LOCAL_BASE_URL` from the pack's config.yaml, else
|
||||
`http://127.0.0.1:8080/v1`). `apikey`/`model_name` optional depending on server.
|
||||
Built-in presets also include MiniMax H3-Context-IR (`v2/h3_context_ir`),
|
||||
bigmodel.cn (GLM) and RunningHub LLM keys from config.yaml.
|
||||
- Output `PROMPT` → render node's prompt.
|
||||
|
||||
### Option C — any ComfyUI LLM node in the loop
|
||||
|
||||
Keep the example's wiring shape and swap node 86 for the preferred LLM node
|
||||
(system_prompt + user_prompt inputs, STRING output). `prompt_format` on TaskOutput
|
||||
can stay `api`; setting it to `llm` instead makes USER_PROMPT a single ready-made
|
||||
chat transcript for LLM nodes that have only one prompt input.
|
||||
|
||||
## Local project path (alternative, no loop nodes)
|
||||
|
||||
`easy multitrackProject` (editor tracks_info in, `segment_count = -1`) builds and runs
|
||||
every segment itself in one queue — see `MiniMaxH3_v1.3.1_Easy_Project_AllInOne.json`.
|
||||
It contains no LLM node; prompts come from the timeline JSON. This is the pack's own
|
||||
agent-driven setup: an external LLM (agent) writes the whole `track_data`, then one
|
||||
run renders it. The pack even ships `skills/easy-media-multitrack-workflow` +
|
||||
`patch_workflow.py` for exactly that flow (our SKILL.md step 5B).
|
||||
|
||||
## The built-in LLM system prompts (published, not locked)
|
||||
|
||||
The rewrite-LLM contract ships in the open in the pack:
|
||||
`utils/prompt_builder.py` — `MINIMAX_BASE_PROMPT` (~15.7 KB, "Video Prompt Writing
|
||||
Guide (T2VA / I2VA / FL2VA / L2VA)") and `MINIMAX_REF_PROMPT` (~23.5 KB, "Full-
|
||||
Reference Mode Rewrite Output Format Guide" — six-section rewrite with
|
||||
`<Subject N>` / `<Picture N>` / `<Video N>` / `<Audio N>` labels).
|
||||
|
||||
Selection (`build_prompt_request`, MiniMax branch):
|
||||
|
||||
- `task_mode` `default` / `l2v` (our t2v payloads) → `MINIMAX_BASE_PROMPT`
|
||||
- `task_mode` `ref` / `edit` → `MINIMAX_REF_PROMPT`
|
||||
- a non-empty per-segment `system_prompt` overrides both; with an empty one the
|
||||
context-segment continuity rules ride the MiniMax rewrite API server-side
|
||||
(entry rule: not-first segment using `context`; tail rule: next segment uses
|
||||
`context`) — per the pack's own `multitrack-schema.md`.
|
||||
|
||||
Delivery: the system prompt only leaves TaskOutput via `prompt_format`
|
||||
`api` or `llm` (`default` / `promptRelay` output an empty SYSTEM_PROMPT).
|
||||
`build_llm_prompt` folds system+user into one transcript for single-input LLM
|
||||
nodes. To use the user's own LLM with the same behavior: paste
|
||||
`minimax_base_prompt.txt` (or the ref guide for ref-mode work) into the LLM
|
||||
node's system prompt field and wire TaskOutput SYSTEM_PROMPT/USER_PROMPT into
|
||||
it — or leave the LLM's system prompt empty and pass TaskOutput.SYSTEM_PROMPT
|
||||
through, which already carries the selected guide verbatim.
|
||||
2095
easy-media-track-gate/verify_track_rules.py
Normal file
2095
easy-media-track-gate/verify_track_rules.py
Normal file
File diff suppressed because it is too large
Load Diff
@ -224,6 +224,15 @@ system prompt emits. Report the delivered frames/seconds it prints.
|
||||
|
||||
## Step 5 — Deliver (the node makes the multi-clip video)
|
||||
|
||||
**This plan targets the standalone MiniMax H3 Context Loop pack only.** ComfyUI-Easy-Media
|
||||
(yolain) ships its own H3 node family built on the MultiTrack system and has NO
|
||||
`plan_json`/`shots` consumer — a chain plan pasted into an Easy-Media workflow is read by
|
||||
no node, and the workflow's default single task segment (0-240 frames = exactly 10 s at
|
||||
24 fps) renders the whole story as one 10-second clip. For that pack use the
|
||||
`easy-media-track-gate` skill instead: one task segment per shot in the multiTrackEditor
|
||||
TRACK_DATA, a for-loop rendering each segment as its own H3 generation, and
|
||||
mergeVideosFromPaths stitching the clips.
|
||||
|
||||
Verified against the node pack source (`chain_nodes.py`, `MiniMaxH3ChainPlan` /
|
||||
`MiniMaxH3ChainPlanModern`): the parser is a plain `json.loads` over the effective
|
||||
plan string — a non-empty STRING connected to the forced `plan_json_input` socket
|
||||
|
||||
@ -133,7 +133,7 @@ a an the and or but nor so yet if then else as is are was were be been being am
|
||||
do does did done have has had having will would shall should can could may might
|
||||
must of to in on at by for from into onto over under with without within near
|
||||
past through during before after above below between against along across behind
|
||||
beyond than too very just even also still already always never only own same all
|
||||
beyond than too very just even also still already always never only own same all like
|
||||
any both each every few more most other some such no not nor while when where how
|
||||
why who whom whose which what that this these those there here now then once
|
||||
twice s t ok up down out off again further
|
||||
|
||||
Loading…
Reference in New Issue
Block a user