MM-prompter/SKILL.md

12 KiB
Raw Blame History

name description
mm-prompter Writes multi-segment ref2va video prompts plus T2I character-sheet prompts and per-segment first-frame image prompts from a vague scene idea and a total video length in seconds. Use when the user gives a rough video/scene concept (optionally with song lyrics or an audio file name) and wants numbered video-generation segments joined by "|", character sheets as Image-N.json for characters that lack an image reference, Segment-img-N.json frame prompts, and an image-map.txt manifest. Optional opt-in FL2VA mode adds last-frame prompts and first-last video prompts.

MM Prompt Creator

Turn one vague prompt and one number (total seconds) into a complete, ready-to-run generation package:

  1. segments.txt — N segment prompts, joined by |. The downstream video creator node splits the total length evenly across segments and generates them one by one. Default format is full-reference (ref2va); FL2VA (first–last frame) is an opt-in alternative (see Mode selection).
  2. Image-N.json — one 16:9 character-sheet text-to-image prompt per character that has no user-provided image reference. Consumed one at a time by a ComfyUI workflow.
  3. Segment-img-N.json — one 16:9 first-frame image prompt per segment (both modes); the image engine renders it using the character sheets as reference images. In FL2VA mode also Segment-img-N-last.json per segment for the last frame.
  4. image-map.txt — manifest that says which <Subject N> maps to which image file.

Read references/ref2va-segment-format.md before drafting the first segment and references/t2i-character-sheet.md before writing the first JSON. When the run uses FL2VA mode, also read references/fl2va-segment-format.md; always read references/frame-image-prompts.md before writing frame files. A complete end-to-end example (including the audio and FL2VA variants) is in references/worked-example.md.

Mode selection

  • ref2va (default). Every run uses the six-section ref2va segment format unless the user explicitly opts into first–last frames.
  • FL2VA (opt-in). Switch when the user asks for "FL2V", "FL2VA", "first-last", "first and last frame", or "start and end frame" per segment. In FL2VA mode the video side receives ONLY the segment's first-frame image, last-frame image, and the FL2VA-format prompt defined in references/fl2va-segment-format.md — never the character sheets. The sheets are still generated: they feed the image side, which renders the first/last frames with consistent characters. If the user's wording is ambiguous, stay in ref2va mode; never mix the two formats in one run.

Inputs

Input Required Notes
Vague prompt yes Rough scene/idea text from the user
Total video length in seconds yes Drives segment count
Song lyrics no Guide pacing only; NEVER appear in any output
Audio file mention no e.g. "use midnight.mp3" → enables <Audio 1>
Per-character image references no User-supplied files (e.g. mary.png)
Style hints no Honor if given; otherwise invent and lock
Frame mode no Default ref2va; FL2VA (first–last) only when explicitly requested

Make reasonable creative choices and proceed. Do not block on questions; this skill exists to expand vagueness into concrete, consistent detail.

Scope rules

  • Subjects are recurring on-screen characters only: people, animals, creatures, robots that act as characters. Settings, props, and crowds stay in prose; they never get <Subject N> numbers or sheets.
  • Numbering is global across the whole video, assigned in order of first on-screen appearance: <Subject 1>, <Subject 2>, ... For every subject, the image number follows the subject number: Subject 2's reference is cited as <Picture 2> and, if generated, saved as Image-2.json.
  • A subject that does not appear in a segment must not be listed in that segment's subject_definitions.
  • No <Video N> labels: this skill never takes video input.
  • Output language: English.

Pipeline (follow in order)

Step 1 — Intake. Parse the vague prompt. List every character that will be on screen. For each, decide: user-provided image reference (record filename) or needs a generated sheet. Detect audio file mentions and lyrics. Detect style hints.

Step 2 — Segment count. n = max(1, round(total_seconds / 10)) with round-half-up: 8s→1, 15s→2, 21s→2, 34s→3, 55s→6. Plan each segment as one beat of roughly total_seconds / n. Assume segments flow into each other as one longer scene unless the user says otherwise, so segment k's end state matches segment k+1's start state.

Step 3 — Lock the story bible (before drafting). Decide once, in a short working note, and never change it afterward: each subject's hair, eyes, build, wardrobe, props, and palette; the setting; time of day; lighting; color palette; camera character; style. Where the vague prompt is silent, invent specifics and lock them. Every segment and every sheet must use the identical locked values.

Step 4 — Assign numbering. Subject 1..M by first on-screen appearance across the whole video. Map references: user file (e.g. mary.png) or Image-N.json.

Step 5 — Draft each segment in the selected mode. ref2va: exactly per references/ref2va-segment-format.md — six sections in order, global labels, default one continuous [Shot 1] per segment, extra shots only for time-synced changes, timestamps relative to the segment start; when a subject sings to the provided audio, use the exact fixed audio template and the exact lip-sync phrase defined in that reference. FL2VA: exactly per references/fl2va-segment-format.md — instruction line

  • three fields, single-shot interpolation arc from first-frame state to last-frame state, no reference labels.

Step 6 — Write the frame image prompts per references/frame-image-prompts.md: one Segment-img-N.json per segment (N = 1..n) describing that segment's opening state, and — FL2VA mode only — one Segment-img-N-last.json per segment describing its final state. Captions reuse the locked story-bible values verbatim and anchor each character to the supplied reference images.

Step 7 — Write the character sheets per references/t2i-character-sheet.md, one Image-N.json per subject lacking a user reference. Every caption describes the fixed four-panel photorealistic turnaround (face close-up, front, side facing 90° right, back) on a white background with no text, no labels, and no borders — studio photography of a real person, never a cartoon or designed card — and must reuse the locked story-bible values verbatim.

Step 8 — Write the manifest image-map.txt, one line per subject:

Subject 1 - mary.png
Subject 2 - Image-2.json

User-supplied files keep their filenames; generated sheets are always Image-N.json. The manifest never lists frame files — Segment-img-N numbering follows segments, not subjects.

Step 9 — Assemble segments.txt. Strip each segment text, join with a single | (no spaces added around it). Exactly n segments, exactly n−1 | characters, no trailing separator, no extra headers or numbering. The | character must never appear inside a segment's own text.

Step 10 — Validate against the checklist below. When the environment can run Python, also run python scripts/validate_outputs.py --segments segments.txt --expected-n <n> (add --mode fl2va --total-seconds <total> in FL2VA mode).

Output location

Write all outputs (segments.txt, Image-N.json, Segment-img-N.json frames, Segment-img-N-last.json in FL2VA mode, image-map.txt) into the current project folder of the project the skill is running on. If no project folder is obvious, use the current working directory. Never write outputs into the skill's own folder.

Hard rules

  1. Lyrics NEVER appear in any output text — not in detailed_description, not in <d> tags, not in the audio sections. They only shape pacing and per-segment content. Singing is expressed solely by the lip-sync phrase and the fixed audio template.
  2. Song vocals come from the reused audio: do not invent (Sx) speaker IDs or <d> blocks for song lyrics. Ordinary spoken dialogue (non-song) may still use (Sx) + <d>[Language] ...</d> per the format reference.
  3. Global numbering is immutable across segments, sheets, and manifest.
  4. The audio template and lip-sync phrase are used verbatim with only N substituted.
  5. [Shot 1] never carries a timestamp; later shots use At MM:SS.mmm relative to the segment start.
  6. Every subject appearing anywhere in the video must have a manifest line.
  7. Frame captions describe single full-bleed 16:9 film stills in the locked style — photorealistic photographic language, no sheet/panel vocabulary, no text or borders, no lyrics, no timestamps or labels, and every on-screen character anchored to the supplied reference images with verbatim story-bible traits.
  8. FL2VA mode: the instruction line of every segment is the exact fixed template with only the final shot index and the two-decimal duration substituted; the prompt contains no <Subject N> / <Picture N> / <Video N> / <Audio N> labels.

Validation checklist (all must pass before delivery)

  1. segments.txt has exactly n segments and exactly n−1 | separators.
  2. (ref2va mode) Each segment contains all six sections in order: subject_definitions, summary, retention_analysis, detailed_description, overall_soundscape, non_diegetic_music.
  3. (ref2va mode) Subject numbering is global and gapless (1..M); <Picture N> cited inside <Subject N> always uses the same N. In FL2VA mode numbering lives on the image side (manifest + sheets) only.
  4. Every subject in the prompts has a manifest line; every Image-N.json in the manifest exists, parses, has exactly the two keys expanded_caption and aspect_ratio, and aspect_ratio is "16:9". No orphan Image-*.json files.
  5. No <Video anywhere; no lyric text anywhere.
  6. (ref2va mode) Singing segments: summary prefix [reference generation + audio reuse], the exact fixed audio template in the sound sections, and the exact lip-sync phrase inside detailed_description. Non-singing segments use [reference generation].
  7. [Shot 1] has no timestamp; any At MM:SS.mmm appears only on [Shot N], N≥2.
  8. Sheet captions and segment descriptions agree with the locked story bible.
  9. Each sheet caption describes the four-panel photorealistic turnaround (face close-up panel, full-body front, side facing 90° right, back) on a clean white background with equal spacing and no text, no labels, no borders; it uses photorealistic/photographic language and contains no cartoon, illustration, or design-board vocabulary (cartoon, anime, mascot, name tag, swatch, palette strip, design board, CGI, 3D render...).
  10. One Segment-img-N.json exists per segment (plus Segment-img-N-last.json per segment in FL2VA mode, and none in ref2va mode); each is two-key JSON with "aspect_ratio": "16:9", photorealistic language, no banned vocabulary.
  11. FL2VA mode: each segment's first line is the exact FL2VA instruction with the correct final shot index and two-decimal duration; then integrated_multimodal_description:, overall_soundscape:, non_diegetic_music: in order, and none of the six ref2va section labels appear.

References

  • references/ref2va-segment-format.md — authoritative per-segment format for the default mode (sections, labels, shots, audio template, retention markers).
  • references/fl2va-segment-format.md — the opt-in FL2VA first–last frame prompt format (instruction line, interpolation arc, dialogue and audio rules).
  • references/t2i-character-sheet.md — the four-panel photorealistic turnaround sheet rules and the two-key JSON output contract.
  • references/frame-image-prompts.md — per-segment first/last frame image prompt rules (Segment-img-N.json, Segment-img-N-last.json).
  • references/worked-example.md — full end-to-end example, including audio and FL2VA variants.