Portable self-contained skill (%USERPROFILE%\.agent\skills\easy-media-track-gate\): - references/track-schema.md: full English translation of the node pack's Chinese multitrack-schema.md (resolution DynamicCombo rules, time semantics, task segment content, context/context_drift joins, audio locks, shared media, project params, patch plan shape); field names cross-checked against patch_workflow.py source. - verify_track_rules.py (66 KB, stdlib only): mirrors every check the node pack's patch_workflow.py performs at apply time (ids, sorting, overlap, frames, source_type field matching, <=9 images, continuity enums, one locked audio + one locked video, locked-media task coverage, shared_reference rules, resolution tokens, project fields) and adds the gates upstream leaves to discipline: outline pins (id/frames/ modes/image set diffed per fragment AND at the merge), verbatim media contract (file_path character-for-character vs media_inventory, closest-match hint on rewrite), context prompt rules (self-contained in all modes, keep-then-change opener for joins, end-in-motion tail when another join follows), drift all-or- nothing + consecutive 10s split unless lengths_user_pinned. Subcommands: outline/ segment/trackdata/plan/suite; no-outline runs stay schema-only. Embedded suite 25 cases, prints ~0.7 KB (far under Cline's 8k cap). - build_track_plan.py: merge re-walks every fragment against the guide, assembles TRACK_DATA (task track from fragments; locked master-audio/reference-video tracks built from the outline, never retyped), writes plan.json + track_data.json with stable UUIDv5 ids; --check mode re-validates. - examples/: worked 3-segment rooftop example (outline, 3 fragments, plan.json, track_data.json) - merge is byte-deterministic and the generated plan dry-run- applies to the bundled v1.3.0 blank template with graph_preserved: true. - SKILL.md: five-move procedure with Cline cap workarounds (never read the 66 KB validator, @@CONTINUE@@ split, never read back generated plans), Step 0 media paths (upload only on explicit request; never invent a path), approval pause, prompt craft rules, delivery through the node pack's own patch_workflow.py (dry-run -> --write to a NEW file -> inspect); queueing stays the node pack's own flow on explicit user request.
321 lines
19 KiB
Markdown
321 lines
19 KiB
Markdown
---
|
|
name: easy-media-track-gate
|
|
description: Turn a video idea into a validated MultiTrack patch plan for ComfyUI-Easy-Media
|
|
(the `easy multiTrackEditor` / `easy multitrackProject` nodes). Use when the user wants to
|
|
plan task segments/shots on the multitrack timeline, write or extend a MultiTrack timeline,
|
|
arrange prompts, reference images, master audio or reference-video locks, regenerate one
|
|
task segment, or produce the patch plan JSON that `patch_workflow.py apply` consumes. The
|
|
scripts and references live next to this SKILL.md (the skill's install folder); run them
|
|
from the user's working folder, where all generated files are written.
|
|
---
|
|
|
|
# Easy-Media Track Gate — agent procedure
|
|
|
|
Goal: produce `plan.json`, the patch plan applied to a ComfyUI workflow with the Easy-Media
|
|
node pack's own `patch_workflow.py`, in five moves: the user gives a video idea and a
|
|
segment count (or an existing workflow to edit) → you build a timeline guide (`outline.json`)
|
|
and show it for **approval** → with the approved guide you compose the task segments one at
|
|
a time (about 10 seconds each), each turn existing so that ONE segment gets a complete
|
|
prompt, the right references and the right continuity mode → the scripts verify schema,
|
|
timing and continuity across the whole chain → `build_track_plan.py` assembles the TRACK_DATA
|
|
and the patch plan at the end. Never deliver a plan that did not pass validation, and never
|
|
compose a segment from a guide the user has not approved.
|
|
|
|
## Paths and Python (read first — these exact forms prevent most errors)
|
|
|
|
- **SKILL_DIR** is the folder this SKILL.md was loaded from — the skill's install folder
|
|
(default on Windows: `%userprofile%\.agent\skills\easy-media-track-gate`; macOS/Linux:
|
|
`~/.agent/skills/easy-media-track-gate`). The scripts, the schema reference and the worked
|
|
example all live in SKILL_DIR, not in the working folder. Every command below uses the
|
|
Windows default install path; substitute your real SKILL_DIR if it differs.
|
|
- Work in the user's **working folder** (the folder open in the editor). Write generated
|
|
files there: `outline.json`, `segment_01.json`, `segment_02.json`, …, `plan.json`,
|
|
`track_data.json`. Never write generated files into SKILL_DIR.
|
|
- Python command: Windows `py -3` (or `python`); macOS/Linux `python3`. If the working
|
|
folder has a `.venv`, its interpreter works too (`.venv\Scripts\python.exe`). The scripts
|
|
need only the Python standard library — nothing to install.
|
|
- Start every task by running the smoke test under "Before you start". If it fails because
|
|
SKILL_DIR is missing files, stop and tell the user the skill folder is incomplete.
|
|
- **Never read `verify_track_rules.py`** — run it and work from its printed output. Cline's
|
|
read_file is capped by hard-coded budgets that do NOT grow with the model's context
|
|
window: about 48,000 characters per read, 2,000 lines per read. The 66 KB validator
|
|
arrives cut no matter how large the context window is. A truncation notice is Cline's
|
|
read cap, not repo corruption — do not "fix" truncated files. `build_track_plan.py`
|
|
(~9 KB) and `references/track-schema.md` (~22 KB, 465 lines) are safe to read whole.
|
|
- **Never read back `plan.json` or `track_data.json` once real segments exist** — a
|
|
10-segment plan passes the read caps today and exceeds them tomorrow. Validate with
|
|
`--check` and trust the one-line OK summary; the shipped `examples/plan.json` (6.8 KB)
|
|
is safe to read if you need to see the shape.
|
|
- Cline's editor rejects writes over 6,000 characters ("Editor input too large"). A segment
|
|
fragment with a rich prompt often does. NEVER write part 2 with write_to_file — it
|
|
replaces the whole file and part 1 is lost. Split instead: write the first part ending
|
|
with the bare marker line `@@CONTINUE@@`, then replace_in_file with SEARCH `@@CONTINUE@@`
|
|
and REPLACE the remaining content. The finished file must not contain the marker.
|
|
- Cline also cuts command output at 8,000 characters. The embedded suite prints well under
|
|
1 KB when it passes; a failing validation prints one actionable line. If output ever
|
|
arrives truncated, re-run the single-file command (`segment`, `outline`, `plan`) — its
|
|
output is tiny.
|
|
|
|
## Before you start (smoke test)
|
|
|
|
```sh
|
|
py -3 "%USERPROFILE%\.agent\skills\easy-media-track-gate\verify_track_rules.py"
|
|
py -3 "%USERPROFILE%\.agent\skills\easy-media-track-gate\build_track_plan.py" --check "%USERPROFILE%\.agent\skills\easy-media-track-gate\examples\plan.json"
|
|
```
|
|
|
|
Expect the suite to print `SUITE PASS (25/25 cases)` and the second command to print
|
|
`PLAN OK` with `total_length 720 frames = 30.00s`.
|
|
|
|
## Step 0 — Media paths come first (never invent one)
|
|
|
|
Every `file_path` in the plan must be a real path on the user's ComfyUI instance. Before
|
|
writing the outline, establish the media inventory one of two ways:
|
|
|
|
- The user names the exact input paths (they know their `input/` subfolders). Use their
|
|
strings verbatim — they become the pinned inventory.
|
|
- The user explicitly asks you to upload → upload first and record the returned relative
|
|
path (`subfolder/name`, `source_type: "input"`) for every file. Upload uses the node
|
|
pack's own API (`POST /upload/image` with a `subfolder` like `codex/<project>`); never
|
|
write a local absolute path into the plan.
|
|
- If there is no real path and no upload request, stop and ask. A plan is still deliverable
|
|
with obviously-fake placeholder paths only if you say plainly it cannot run until real
|
|
media replaces them — never dress a placeholder up as a real path.
|
|
|
|
Upload and execution are external state changes: only upload/queue when the user explicitly
|
|
asked (Step 5).
|
|
|
|
## Step 1 — Build the timeline guide (outline)
|
|
|
|
Read SKILL_DIR's `references/track-schema.md` once (safe whole-file read; it is the English
|
|
translation of the node pack's Chinese schema doc). Then write `outline.json` — the guide
|
|
for the whole timeline — in exactly this shape (the shipped `examples/outline.json` is the
|
|
worked model):
|
|
|
|
```json
|
|
{
|
|
"title": "short_snake_case_name",
|
|
"lengths_user_pinned": false,
|
|
"editor": {
|
|
"format": "MiniMax",
|
|
"resolution": { "resolution": "width x height (megapixels)",
|
|
"aspect_ratio": "16:9 (Widescreen)", "megapixels": 0.8 }
|
|
},
|
|
"defaults": { "frame_rate": 24, "segment_seconds": 10 },
|
|
"media_inventory": [
|
|
{ "key": "SONG", "kind": "audio", "source_type": "input",
|
|
"file_path": "codex/rooftop/song.wav", "file_name": "song.wav" },
|
|
{ "key": "POSE_REF", "kind": "image", "source_type": "input",
|
|
"file_path": "codex/rooftop/pose_ref.png", "file_name": "pose_ref.png" }
|
|
],
|
|
"tracks": {
|
|
"master_audio": { "key": "SONG", "lock": true, "name": "Master Song", "role": "why it drives the timeline" },
|
|
"reference_video": null
|
|
},
|
|
"project": { "project_save": "override", "segment_start_number": 1,
|
|
"segment_count": -1, "sampling_plan": "medium", "sampling_mode": "single" },
|
|
"segments": [
|
|
{ "index": 1, "id": "seg_one_open", "start_seconds": 0, "duration_seconds": 10,
|
|
"task_mode": "ref", "continuity_mode": "shot",
|
|
"images": [ { "key": "POSE_REF", "shared_reference": false } ],
|
|
"beat": "one sentence: what happens AND where the segment ends",
|
|
"prompt_brief": "1-2 sentences: what the prompt keeps first, changes later, ends on" }
|
|
]
|
|
}
|
|
```
|
|
|
|
Field rules the validator enforces:
|
|
|
|
- `media_inventory` keys are UPPERCASE; every `file_path` is copied from Step 0 sources —
|
|
this inventory is the **machine-checked verbatim source**: every segment's image items and
|
|
every locked media track are diffed against it, so a retyped path fails on the spot. A
|
|
paraphrase in the inventory cannot be repaired later.
|
|
- `tracks.master_audio` / `tracks.reference_video` name the ONE locked audio and ONE locked
|
|
video track (the schema allows at most one of each). Lock the master song for MV/lyric/
|
|
dance/beat projects, the narration track when narration drives the cut, and the reference
|
|
video when the user supplies one for subject/partial replacement. Ordinary BGM/SFX/ambience
|
|
is never locked. If two candidates could be the master, stop and ask the user — never pick
|
|
by track order or duration.
|
|
- Resolution: copy the user's intent into the exact DynamicCombo tokens — `16:9` means
|
|
`16:9 (Widescreen)`, `0.9mp` means megapixels mode with the number taken verbatim; never
|
|
convert MP into width/height. Keep the current workflow resolution when the user expressed
|
|
no intent (then omit nothing — still write the block the user approved).
|
|
- Segments: `start_seconds` are cumulative (previous start + duration + gap); ids are unique
|
|
lowercase snake_case, never renamed later; `duration_seconds` about 10 unless the user
|
|
says otherwise; only the last segment may come to rest. `continuity_mode` per segment:
|
|
`shot` (standalone/new composition), `context` (continues the previous segment's real
|
|
delivered tail), `context_drift` (context + subject/partial replacement from the locked
|
|
reference video). **context_drift is all-or-nothing**: if any segment drifts, all do,
|
|
first included; without user-pinned lengths the drift split is consecutive 10-second
|
|
pieces (last takes the remainder) — set `lengths_user_pinned: true` only when the user
|
|
actually specified the lengths.
|
|
- `images` per segment: at most 9, each an inventory image key with its `shared_reference`
|
|
flag. Reference images define IDENTITY, not composition. When several context segments
|
|
reuse one image, prefer explicit private items (`shared_reference: false`, fixed ids,
|
|
same file) over shared-image metadata — the current front-end regenerates UUIDs for
|
|
sparse shared images, and unverified shared data risks unstable loads.
|
|
- `project`: `project_save: "override"` with `segment_count: -1` deletes saved segments
|
|
after the start index on execution — warn the user before choosing it. `upscale_by` is
|
|
only written when the user asked (default 1.250, three decimals); the editor resolution
|
|
is the first-pass size, and dual-sampling second pass = `round(editor_size * upscale_by /
|
|
32) * 32`.
|
|
|
|
## Step 2 — Present the guide for approval (do not skip)
|
|
|
|
Show the user the finished guide in plain language — one line per segment: number, id,
|
|
seconds, continuity mode, what happens and where it ends — plus the media inventory, the
|
|
locked tracks, the resolution block and the project parameters. Then STOP and wait for
|
|
their reply. The user approves it or asks for changes; apply every change to `outline.json`
|
|
and, if anything changed, show the changed rows again. Start Step 3 only after an explicit
|
|
approval. The approved guide is the contract: segments are composed from it, ids, frames
|
|
and modes are never silently altered, and new story ideas from the user go through the
|
|
guide first — never straight into a segment file.
|
|
|
|
## Step 3 — Compose the segments one at a time
|
|
|
|
Work only from the approved guide. In story order: compose ONE segment file, validate it,
|
|
fix it until it passes, and only then move to the next. Since you wrote the previous file
|
|
yourself, open it and reuse its `_handoff.end_state` as the opening state of the next
|
|
context join. Rewrite a segment as many times as it takes. If Cline rejects a write with
|
|
'Editor input too large', split it with the marker pattern from the rules above.
|
|
|
|
**Media copy rule (the one writers keep breaking):** every `file_path` in a segment's
|
|
`images` is a character-for-character COPY of the outline's `media_inventory` path for that
|
|
key. Select the text in `outline.json`, copy, paste — never retype it from memory, never
|
|
fix a typo in it, never reword a folder name. The commands below pass `--outline`, so the
|
|
validator diffs the paths against the guide and a rewrite fails the segment on the spot.
|
|
The same discipline covers the resolution tokens and file names: the merge copies
|
|
`editor.resolution` and the locked media tracks straight from the outline, so the outline
|
|
is the only place they are ever typed.
|
|
|
|
Each fragment is exactly this shape:
|
|
|
|
```json
|
|
{
|
|
"id": "from the outline row",
|
|
"start_frame": 0,
|
|
"end_frame": 240,
|
|
"content": {
|
|
"media_type": "none",
|
|
"task_mode": "ref",
|
|
"continuity_mode": "shot",
|
|
"ref_image_size": "match",
|
|
"images": [
|
|
{ "id": "img-seg-one-open-1", "source_type": "input",
|
|
"file_path": "copied verbatim from the inventory", "file_name": "pose_ref.png",
|
|
"shared_reference": false }
|
|
],
|
|
"user_prompt": "the full segment prompt",
|
|
"system_prompt": "",
|
|
"user_prompt_variant": "a",
|
|
"user_prompt_b": "",
|
|
"muted": false,
|
|
"volume_db": 0
|
|
},
|
|
"_handoff": { "end_state": "the real final frame in 2-3 sentences: pose, motion in progress, camera state" }
|
|
}
|
|
```
|
|
|
|
`start_frame`/`end_frame` are computed, not guessed: `round(seconds * frame_rate)`,
|
|
start inclusive, end exclusive, and they must match the approved row exactly. If the
|
|
variant is `b`, write `user_prompt_b` and leave `user_prompt` alone — never switch the
|
|
variant on your own.
|
|
|
|
```sh
|
|
py -3 "%USERPROFILE%\.agent\skills\easy-media-track-gate\verify_track_rules.py" segment segment_01.json --index 1 --outline outline.json
|
|
py -3 "%USERPROFILE%\.agent\skills\easy-media-track-gate\verify_track_rules.py" segment segment_02.json --index 2 --prev segment_01.json --outline outline.json
|
|
```
|
|
|
|
(`--total` is optional with `--outline`; `--prev` chains the previous file and enforces
|
|
consecutive frames `[0,240), [240,480), …` — never an extra frame at a boundary.)
|
|
|
|
### Prompt craft rules (violations fail the validator)
|
|
|
|
- Every task segment renders standalone — the model never sees any other segment's prompt.
|
|
Fully restate scene, light and subject identity inside each prompt; never say "the same
|
|
one", "as before", "the previous shot". This ban is checked in every mode.
|
|
- Reference images define identity only: they are not the first frame or a composition
|
|
anchor; context joins never use image anchoring or `[video continuation]`.
|
|
- **context segments open by keeping the delivered tail** — the join starts from the
|
|
previous segment's real last frames (about 0.92 s earlier than the saved boundary at
|
|
24 fps), whose framing can differ from what the previous prompt imagined: open with the
|
|
camera keeping its direction and the subject keeping pose, motion and light. Give the
|
|
hold an action (a breath, a weight shift) so it does not render as a freeze-frame.
|
|
- **Keep first, change later**: before about `00:02.000` no cut, no new key action; a
|
|
required composition change is written as an explicit cut later in the prompt. For a
|
|
genuinely new composition use a `shot` segment instead of `context`.
|
|
- **End in motion**: a context segment followed by another context segment never ends at
|
|
rest — no "comes to a stop", "settles", "holds still", "falls asleep" in the closing
|
|
lines. Only the timeline's final segment may come to rest.
|
|
- Compose, never summarize: subject anchor with its look, blocking, one camera idea,
|
|
performance beats, physical texture (wardrobe behavior, weather on skin and fabric,
|
|
light), and a concrete mid-action exit state. Decompose every action into ordered
|
|
body mechanics ("bends his knees, grips the handle with his left hand, then stands"),
|
|
naming the limb; camera, light and weather each get their own ordered visible stages.
|
|
The validator trips a thin-prompt floor (60 words) — never pad to a band, and never
|
|
hand in a one-liner.
|
|
- `system_prompt` stays `""` unless the user asked for one: an empty system prompt lets the
|
|
node auto-append the MiniMax reference rules for context joins; a custom one disables
|
|
that.
|
|
- `_handoff.end_state` must be truthful — it is the contract the next join opens inside.
|
|
|
|
## Step 4 — Verify the chain, put it all together at the end
|
|
|
|
Once every segment file passes, the merge is the continuity check: `build_track_plan.py`
|
|
re-walks every fragment against the guide (ids, frames, modes, verbatim media paths, prompt
|
|
rules, handoff chain), assembles the TRACK_DATA (task track from the fragments; the locked
|
|
master-audio / reference-video tracks built straight from the outline), computes
|
|
`total_length`, and only then writes the deliverable. Never write the plan by hand and
|
|
never edit fragment prose during the merge.
|
|
|
|
```sh
|
|
py -3 "%USERPROFILE%\.agent\skills\easy-media-track-gate\build_track_plan.py" outline.json segment_01.json segment_02.json segment_03.json -o plan.json
|
|
py -3 "%USERPROFILE%\.agent\skills\easy-media-track-gate\build_track_plan.py" --check plan.json --outline outline.json
|
|
```
|
|
|
|
The merge writes `plan.json` (the patch plan) and `track_data.json` (pretty, for
|
|
inspection) and prints the delivered frames/seconds, the track list and the locked tracks.
|
|
Segment, track and image ids are stable UUIDv5 values derived from the project title, so a
|
|
re-merge keeps ids consistent.
|
|
|
|
## Step 5 — Deliver (the node pack's own script applies it)
|
|
|
|
The deliverable is applied to the user's workflow by the Easy-Media node pack's script —
|
|
find it under the node pack folder (typically
|
|
`<ComfyUI>\custom_nodes\ComfyUI-Easy-Media\skills\easy-media-multitrack-workflow\scripts\patch_workflow.py`;
|
|
ask the user for the path and the workflow JSON if they are not at hand):
|
|
|
|
```sh
|
|
py -3 <node-pack>\skills\easy-media-multitrack-workflow\scripts\patch_workflow.py apply WORKFLOW.json --plan plan.json
|
|
py -3 <node-pack>\skills\easy-media-multitrack-workflow\scripts\patch_workflow.py apply WORKFLOW.json --plan plan.json --output RESULT.json --write
|
|
py -3 <node-pack>\skills\easy-media-multitrack-workflow\scripts\patch_workflow.py inspect RESULT.json
|
|
```
|
|
|
|
Always dry-run first; only write to a NEW output file (`--output RESULT.json --write`),
|
|
never over the user's source workflow. `inspect` must show the editor's resolution, format,
|
|
frame rate, `total_length`, the task track with all segments and the locked tracks with
|
|
their media. Report: base workflow path, output file, editor/project node ids, resolution,
|
|
timeline summary, media paths, continuity modes, locked tracks and whether the node graph
|
|
stayed untouched (`graph_preserved: true`).
|
|
|
|
Actually opening the workflow in ComfyUI or queueing it (`submit_workflow.py`, UI-tab
|
|
submission) is a separate authorization — only on the user's explicit request, and it is
|
|
the node pack's own flow (see its `references/execution-api.md`). "Queued" never means
|
|
"generated"; wait for the real completion status before reporting success.
|
|
|
|
## Revising after approval
|
|
|
|
- Guide-level change (timings, beats, modes, segment count, media): update `outline.json`
|
|
first, show the changed rows, then rewrite only the affected segments against it and
|
|
re-run the Step 4 merge so `plan.json` refreshes.
|
|
- One segment reads badly: rewrite that `segment_NN.json` (keep its id and frames from the
|
|
guide), re-validate it with the `segment` command, then re-merge. Never patch `plan.json`
|
|
directly — the merge rebuilds it from the fragments every time.
|
|
|
|
## If you cannot run Python
|
|
|
|
Say plainly that the result skipped script validation. Follow the rules manually: copy the
|
|
`examples/` shapes exactly, compute frames as `round(seconds * frame_rate)`, copy every
|
|
`file_path` and resolution token by selection (never retyping), and hand-assemble the
|
|
TRACK_DATA the way Step 4 describes. Manual assembly is error-prone — prefer getting
|
|
Python running (any of `py -3`, `python`, a `.venv`) over hand-building the plan.
|