mmprompt/easy-media-track-gate/SKILL.md

24 KiB
Executable File
Raw Blame History

name description
easy-media-track-gate Turn a video idea into a validated MultiTrack plan for ComfyUI-Easy-Media (the `easy multiTrackEditor` node). Use when the user wants to plan task segments/shots on the multitrack timeline, write or extend a MultiTrack timeline, arrange prompts, reference images, master audio or reference-video locks, regenerate one task segment, or produce the `prompt_override` paste payload for the multitrack editor node (plus the optional patch plan JSON that `patch_workflow.py apply` consumes). The scripts and references live next to this SKILL.md (the skill's install folder); run them from the user's working folder, where all generated files are written.

Easy-Media Track Gate — agent procedure

Goal: produce a validated plan for the easy multiTrackEditor node in five moves: the user gives a video idea and a segment count (or an existing workflow to edit) → you build a timeline guide (outline.json) and show it for approval → with the approved guide you compose the task segments one at a time (about 10 seconds each), each turn existing so that ONE segment gets a complete prompt, the right references and the right continuity mode → the scripts verify schema, timing and continuity across the whole chain → build_track_plan.py assembles the deliverables at the end. The PRIMARY deliverable is prompt_override.json, a minimax_prompt_override payload the user pastes into the multitrack editor node's prompt_override input (format MiniMax) — the editor then renders one clip per prompt, chained back-to-back. The patch plan (plan.json for the node pack's patch_workflow.py) is the optional whole-workflow alternative. Never deliver a plan that did not pass validation, and never compose a segment from a guide the user has not approved. Never hand the user any other JSON as the paste: a non-minimax_prompt_override value in prompt_override falls back to the editor's |-separated text parser and renders the whole story as ONE clip over the entire timeline.

Paths and Python (read first — these exact forms prevent most errors)

  • SKILL_DIR is the folder this SKILL.md was loaded from — the skill's install folder (default on Windows: %userprofile%\.agent\skills\easy-media-track-gate; macOS/Linux: ~/.agent/skills/easy-media-track-gate). The scripts, the schema reference and the worked example all live in SKILL_DIR, not in the working folder. Every command below uses the Windows default install path; substitute your real SKILL_DIR if it differs.
  • Work in the user's working folder (the folder open in the editor). Write generated files there: outline.json, segment_01.json, segment_02.json, …, plan.json, track_data.json. Never write generated files into SKILL_DIR.
  • Python command: Windows py -3 (or python); macOS/Linux python3. If the working folder has a .venv, its interpreter works too (.venv\Scripts\python.exe). The scripts need only the Python standard library — nothing to install.
  • Start every task by running the smoke test under "Before you start". If it fails because SKILL_DIR is missing files, stop and tell the user the skill folder is incomplete.
  • Never read verify_track_rules.py — run it and work from its printed output. Cline's read_file is capped by hard-coded budgets that do NOT grow with the model's context window: about 48,000 characters per read, 2,000 lines per read. The 81 KB validator arrives cut no matter how large the context window is. A truncation notice is Cline's read cap, not repo corruption — do not "fix" truncated files. build_track_plan.py (~10 KB) and references/track-schema.md (~26 KB) are safe to read whole.
  • Never read back plan.json, track_data.json or prompt_override.json once real segments exist — a 10-segment plan passes the read caps today and exceeds them tomorrow. Validate with --check / the override command and trust the one-line OK summary; the shipped examples/prompt_override.json (2.5 KB) and examples/plan.json (6.8 KB) are safe to read if you need to see the shape.
  • Cline's editor rejects writes over 6,000 characters ("Editor input too large"). A segment fragment with a rich prompt often does. NEVER write part 2 with write_to_file — it replaces the whole file and part 1 is lost. Split instead: write the first part ending with the bare marker line @@CONTINUE@@, then replace_in_file with SEARCH @@CONTINUE@@ and REPLACE the remaining content. The finished file must not contain the marker.
  • Cline also cuts command output at 8,000 characters. The embedded suite prints well under 1 KB when it passes; a failing validation prints one actionable line. If output ever arrives truncated, re-run the single-file command (segment, outline, plan) — its output is tiny.

Before you start (smoke test)

py -3 "%USERPROFILE%\.agent\skills\easy-media-track-gate\verify_track_rules.py"
py -3 "%USERPROFILE%\.agent\skills\easy-media-track-gate\build_track_plan.py" --check "%USERPROFILE%\.agent\skills\easy-media-track-gate\examples\plan.json"
py -3 "%USERPROFILE%\.agent\skills\easy-media-track-gate\verify_track_rules.py" override "%USERPROFILE%\.agent\skills\easy-media-track-gate\examples\prompt_override.json" --outline "%USERPROFILE%\.agent\skills\easy-media-track-gate\examples\outline.json"

Expect the suite to print SUITE PASS (34/34 cases) and the other two commands to print PLAN OK with total_length 720 frames = 30.00s and OVERRIDE OK: 3 prompts, duration 10, total 30.00s.

Step 0 — Media paths come first (never invent one)

Every file_path in the plan must be a real path on the user's ComfyUI instance. Before writing the outline, establish the media inventory one of two ways:

  • The user names the exact input paths (they know their input/ subfolders). Use their strings verbatim — they become the pinned inventory.
  • The user explicitly asks you to upload → upload first and record the returned relative path (subfolder/name, source_type: "input") for every file. Upload uses the node pack's own API (POST /upload/image with a subfolder like codex/<project>); never write a local absolute path into the plan.
  • If there is no real path and no upload request, stop and ask. A plan is still deliverable with obviously-fake placeholder paths only if you say plainly it cannot run until real media replaces them — never dress a placeholder up as a real path.

Upload and execution are external state changes: only upload/queue when the user explicitly asked (Step 5).

Step 1 — Build the timeline guide (outline)

Read SKILL_DIR's references/track-schema.md once (safe whole-file read; it is the English translation of the node pack's Chinese schema doc). Then write outline.json — the guide for the whole timeline — in exactly this shape (the shipped examples/outline.json is the worked model):

{
  "title": "short_snake_case_name",
  "lengths_user_pinned": false,
  "editor": {
    "format": "MiniMax",
    "resolution": { "resolution": "width x height (megapixels)",
                    "aspect_ratio": "16:9 (Widescreen)", "megapixels": 0.8 }
  },
  "defaults": { "frame_rate": 24, "segment_seconds": 10 },
  "media_inventory": [
    { "key": "SONG", "kind": "audio", "source_type": "input",
      "file_path": "codex/rooftop/song.wav", "file_name": "song.wav" },
    { "key": "POSE_REF", "kind": "image", "source_type": "input",
      "file_path": "codex/rooftop/pose_ref.png", "file_name": "pose_ref.png" }
  ],
  "tracks": {
    "master_audio": { "key": "SONG", "lock": true, "name": "Master Song", "role": "why it drives the timeline" },
    "reference_video": null
  },
  "project": { "project_save": "override", "segment_start_number": 1,
               "segment_count": -1, "sampling_plan": "medium", "sampling_mode": "single" },
  "segments": [
    { "index": 1, "id": "seg_one_open", "start_seconds": 0, "duration_seconds": 10,
      "task_mode": "ref", "continuity_mode": "shot",
      "images": [ { "key": "POSE_REF", "shared_reference": false } ],
      "beat": "one sentence: what happens AND where the segment ends",
      "prompt_brief": "1-2 sentences: what the prompt keeps first, changes later, ends on" }
  ]
}

Field rules the validator enforces:

  • media_inventory keys are UPPERCASE; every file_path is copied from Step 0 sources — this inventory is the machine-checked verbatim source: every segment's image items and every locked media track are diffed against it, so a retyped path fails on the spot. A paraphrase in the inventory cannot be repaired later. Order matters too: the inventory's image rows map to the editor's image input positions (<Picture 1>, <Picture 2>, …), audio rows to audio input positions, video rows to video input positions — the user wires the media into those node inputs in inventory order.
  • tracks.master_audio / tracks.reference_video name the ONE locked audio and ONE locked video track (the schema allows at most one of each). Lock the master song for MV/lyric/ dance/beat projects, the narration track when narration drives the cut, and the reference video when the user supplies one for subject/partial replacement. Ordinary BGM/SFX/ambience is never locked. If two candidates could be the master, stop and ask the user — never pick by track order or duration.
  • Resolution: copy the user's intent into the exact DynamicCombo tokens — 16:9 means 16:9 (Widescreen), 0.9mp means megapixels mode with the number taken verbatim; never convert MP into width/height. Keep the current workflow resolution when the user expressed no intent (then omit nothing — still write the block the user approved).
  • Segments: start_seconds are cumulative (previous start + duration + gap); ids are unique lowercase snake_case, never renamed later; duration_seconds about 10 unless the user says otherwise; only the last segment may come to rest. continuity_mode per segment: shot (standalone/new composition), context (continues the previous segment's real delivered tail), context_drift (context + subject/partial replacement from the locked reference video). context_drift is all-or-nothing: if any segment drifts, all do, first included; without user-pinned lengths the drift split is consecutive 10-second pieces (last takes the remainder) — set lengths_user_pinned: true only when the user actually specified the lengths.
  • images per segment: at most 9, each an inventory image key with its shared_reference flag. Reference images define IDENTITY, not composition. When several context segments reuse one image, prefer explicit private items (shared_reference: false, fixed ids, same file) over shared-image metadata — the current front-end regenerates UUIDs for sparse shared images, and unverified shared data risks unstable loads.
  • project: project_save: "override" with segment_count: -1 deletes saved segments after the start index on execution — warn the user before choosing it. upscale_by is only written when the user asked (default 1.250, three decimals); the editor resolution is the first-pass size, and dual-sampling second pass = round(editor_size * upscale_by / 32) * 32.

Step 2 — Present the guide for approval (do not skip)

Show the user the finished guide in plain language — one line per segment: number, id, seconds, continuity mode, what happens and where it ends — plus the media inventory, the locked tracks, the resolution block and the project parameters. Then STOP and wait for their reply. The user approves it or asks for changes; apply every change to outline.json and, if anything changed, show the changed rows again. Start Step 3 only after an explicit approval. The approved guide is the contract: segments are composed from it, ids, frames and modes are never silently altered, and new story ideas from the user go through the guide first — never straight into a segment file.

Step 3 — Compose the segments one at a time

Work only from the approved guide. In story order: compose ONE segment file, validate it, fix it until it passes, and only then move to the next. Since you wrote the previous file yourself, open it and reuse its _handoff.end_state as the opening state of the next context join. Rewrite a segment as many times as it takes. If Cline rejects a write with 'Editor input too large', split it with the marker pattern from the rules above.

Media copy rule (the one writers keep breaking): every file_path in a segment's images is a character-for-character COPY of the outline's media_inventory path for that key. Select the text in outline.json, copy, paste — never retype it from memory, never fix a typo in it, never reword a folder name. The commands below pass --outline, so the validator diffs the paths against the guide and a rewrite fails the segment on the spot. The same discipline covers the resolution tokens and file names: the merge copies editor.resolution and the locked media tracks straight from the outline, so the outline is the only place they are ever typed.

Each fragment is exactly this shape:

{
  "id": "from the outline row",
  "start_frame": 0,
  "end_frame": 240,
  "content": {
    "media_type": "none",
    "task_mode": "ref",
    "continuity_mode": "shot",
    "ref_image_size": "match",
    "images": [
      { "id": "img-seg-one-open-1", "source_type": "input",
        "file_path": "copied verbatim from the inventory", "file_name": "pose_ref.png",
        "shared_reference": false }
    ],
    "user_prompt": "the full segment prompt",
    "system_prompt": "",
    "user_prompt_variant": "a",
    "user_prompt_b": "",
    "muted": false,
    "volume_db": 0
  },
  "_handoff": { "end_state": "the real final frame in 2-3 sentences: pose, motion in progress, camera state" }
}

start_frame/end_frame are computed, not guessed: round(seconds * frame_rate), start inclusive, end exclusive, and they must match the approved row exactly. If the variant is b, write user_prompt_b and leave user_prompt alone — never switch the variant on your own.

py -3 "%USERPROFILE%\.agent\skills\easy-media-track-gate\verify_track_rules.py" segment segment_01.json --index 1 --outline outline.json
py -3 "%USERPROFILE%\.agent\skills\easy-media-track-gate\verify_track_rules.py" segment segment_02.json --index 2 --prev segment_01.json --outline outline.json

(--total is optional with --outline; --prev chains the previous file and enforces consecutive frames [0,240), [240,480), … — never an extra frame at a boundary.)

Prompt craft rules (violations fail the validator)

  • Every task segment renders standalone — the model never sees any other segment's prompt. Fully restate scene, light and subject identity inside each prompt; never say "the same one", "as before", "the previous shot". This ban is checked in every mode.
  • Reference images define identity only: they are not the first frame or a composition anchor; context joins never use image anchoring or [video continuation].
  • context segments open by keeping the delivered tail — the join starts from the previous segment's real last frames (about 0.92 s earlier than the saved boundary at 24 fps), whose framing can differ from what the previous prompt imagined: open with the camera keeping its direction and the subject keeping pose, motion and light. Give the hold an action (a breath, a weight shift) so it does not render as a freeze-frame.
  • Keep first, change later: before about 00:02.000 no cut, no new key action; a required composition change is written as an explicit cut later in the prompt. For a genuinely new composition use a shot segment instead of context.
  • End in motion: a context segment followed by another context segment never ends at rest — no "comes to a stop", "settles", "holds still", "falls asleep" in the closing lines. Only the timeline's final segment may come to rest.
  • Compose, never summarize: subject anchor with its look, blocking, one camera idea, performance beats, physical texture (wardrobe behavior, weather on skin and fabric, light), and a concrete mid-action exit state. Decompose every action into ordered body mechanics ("bends his knees, grips the handle with his left hand, then stands"), naming the limb; camera, light and weather each get their own ordered visible stages. The validator trips a thin-prompt floor (60 words) — never pad to a band, and never hand in a one-liner.
  • system_prompt stays "" unless the user asked for one: an empty system prompt lets the node auto-append the MiniMax reference rules for context joins; a custom one disables that.
  • _handoff.end_state must be truthful — it is the contract the next join opens inside.

Step 4 — Verify the chain, put it all together at the end

Once every segment file passes, the merge is the continuity check: build_track_plan.py re-walks every fragment against the guide (ids, frames, modes, verbatim media paths, prompt rules, handoff chain), assembles the TRACK_DATA (task track from the fragments; the locked master-audio / reference-video tracks built straight from the outline), computes total_length, builds the minimax_prompt_override paste payload from the same validated fragments, and only then writes the deliverables. Never write the plan by hand and never edit fragment prose during the merge.

py -3 "%USERPROFILE%\.agent\skills\easy-media-track-gate\build_track_plan.py" outline.json segment_01.json segment_02.json segment_03.json -o plan.json
py -3 "%USERPROFILE%\.agent\skills\easy-media-track-gate\build_track_plan.py" --check plan.json --outline outline.json
py -3 "%USERPROFILE%\.agent\skills\easy-media-track-gate\verify_track_rules.py" override prompt_override.json --outline outline.json

The merge writes prompt_override.json (the paste payload, one line), plan.json (the patch plan) and track_data.json (pretty, for inspection), and prints the delivered durations/seconds, the track list and the locked tracks. Segment, track and image ids are stable UUIDv5 values derived from the project title, so a re-merge keeps ids consistent. The override command re-checks the paste payload against the guide: one prompt per segment, durations inside the runtime's 2–15s window and equal to the approved rows, per-segment generation/continuity modes, the exact <Picture N> / <Audio N> / <Video N> reference set per prompt (a missing token renders with no reference media; a stray one attaches the wrong image), and the track locks.

Step 5 — Deliver (paste payload first, patch script as the alternative)

5A — Default: the prompt_override paste

prompt_override.json is what the user pastes into the multitrack editor node's prompt_override input (easy multiTrackEditor). It is self-contained: one prompt per clip, per-clip durations and continuity modes, media reference tokens inside the prompts. Deliver it with these instructions:

  1. In the editor node set format to MiniMax (the payload is parsed as MiniMax only in that format).
  2. Wire the media IF the outline has any: the inventory's image rows (in inventory order) into the editor's image input, the master song into audio, the reference video into video. The <Picture N> / <Audio N> / <Video N> tokens resolve against those input lists positionally. With an empty inventory ([], pure t2v) nothing gets wired — the editor demands no media inputs at all.
  3. Select the prompt_override widget, clear it, paste the whole one-line JSON. The editor rebuilds the task track: one clip per prompt, back-to-back, durations as specified.
  4. Rendering walks the timeline through easy multiTrackTaskOutput — and that node outputs ONE task segment per pass, selected by its task_index int input (default 0 = only the FIRST clip). A workflow whose task_index stays 0 renders only the first scene no matter how many prompts the override carries. Two ways to render all clips:
    • Loop workflow (proper multi-shot): use the node pack's example_workflows/MinimaxH3_ForLoop_MultiShot.json. Its forLoopStart takes total from easy multiTrackInfoOutput.TASK_COUNT (via a math node that also lets a PrimitiveInt pick one segment: -1 = all), and its index feeds multiTrackTaskOutput.task_index; each iteration generates one clip, batches it, and mergeVideosFromPaths concatenates everything after the loop. Paste the override into THAT editor and set the PrimitiveInt to -1.
    • Manual fallback (no rewiring): set task_index to 0, queue; then 1, queue; then 2, queue. Three clip files, join them afterwards. (task_index = -1 outputs the whole timeline's pre-rendered MEDIA — it is not a 3-clip generation.)
  5. Optional LLM polish slot (only if the user asks for it): inside the loop, TaskOutput.SYSTEM_PROMPT/USER_PROMPT may feed an LLM node whose output replaces the prompt at the render node, with an ifElse bypass back to raw USER_PROMPT. The skill's prompts are already final — leave the bypass OFF unless requested. Ways to fill the slot: any ComfyUI LLM node (system+user in, string out), or the pack's easy multiTrackPromptEnhancer with model openai_compatible (local) pointed at the user's own endpoint (base_url accepts any OpenAI-compatible server: Ollama, LM Studio, vLLM, llama.cpp server). Full wiring: references/workflow_wiring.md.

The paste target is the prompt_override input and only that payload type. Never give the user a plan.json, an outline, an H3 plan or any other JSON to paste there: without "type": "minimax_prompt_override" the editor falls back to its |-separated text parser and renders the whole thing as ONE clip spanning the timeline (the single-clip failure). Never add | inside prompts for the same reason.

5B — Alternative: the node pack's patch script (whole-workflow patch)

When the user wants the timeline written into a workflow file directly (or needs edit task modes / extra tracks the paste payload cannot express), the deliverable is applied by the Easy-Media node pack's script — find it under the node pack folder (typically <ComfyUI>\custom_nodes\ComfyUI-Easy-Media\skills\easy-media-multitrack-workflow\scripts\patch_workflow.py; ask the user for the path and the workflow JSON if they are not at hand):

py -3 <node-pack>\skills\easy-media-multitrack-workflow\scripts\patch_workflow.py apply WORKFLOW.json --plan plan.json
py -3 <node-pack>\skills\easy-media-multitrack-workflow\scripts\patch_workflow.py apply WORKFLOW.json --plan plan.json --output RESULT.json --write
py -3 <node-pack>\skills\easy-media-multitrack-workflow\scripts\patch_workflow.py inspect RESULT.json

Always dry-run first; only write to a NEW output file (--output RESULT.json --write), never over the user's source workflow. inspect must show the editor's resolution, format, frame rate, total_length, the task track with all segments and the locked tracks with their media. Report: base workflow path, output file, editor/project node ids, resolution, timeline summary, media paths, continuity modes, locked tracks and whether the node graph stayed untouched (graph_preserved: true).

Actually opening the workflow in ComfyUI or queueing it (submit_workflow.py, UI-tab submission) is a separate authorization — only on the user's explicit request, and it is the node pack's own flow (see its references/execution-api.md). "Queued" never means "generated"; wait for the real completion status before reporting success.

Revising after approval

  • Guide-level change (timings, beats, modes, segment count, media): update outline.json first, show the changed rows, then rewrite only the affected segments against it and re-run the Step 4 merge so plan.json and prompt_override.json refresh.
  • One segment reads badly: rewrite that segment_NN.json (keep its id and frames from the guide), re-validate it with the segment command, then re-merge. Never patch plan.json or prompt_override.json directly — the merge rebuilds both from the fragments every time.

If you cannot run Python

Say plainly that the result skipped script validation. Follow the rules manually: copy the examples/ shapes exactly, compute frames as round(seconds * frame_rate), copy every file_path and resolution token by selection (never retyping), and hand-assemble the TRACK_DATA the way Step 4 describes. Manual assembly is error-prone — prefer getting Python running (any of py -3, python, a .venv) over hand-building the plan.