MM-prompter/SKILL.md

226 lines
14 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

---
name: mm-prompter
description: Writes multi-segment ref2va video prompts plus T2I character-sheet prompts and per-segment first-frame image prompts from a vague scene idea and a total video length in seconds. Use when the user gives a rough video/scene concept (optionally with song lyrics or an audio file name) and wants numbered video-generation segments joined by "|", character sheets as Image-N.json for characters that lack an image reference, Segment-img-N.json frame prompts, and an image-map.txt manifest. Optional opt-in FL2VA mode adds last-frame prompts and first-last video prompts. Optional opt-in segmented mode writes one segment-N/segment.txt file per segment using @PictureN/@Audio1-style @-references instead of the joined file.
---
# MM Prompt Creator
Turn one vague prompt and one number (total seconds) into a complete, ready-to-run
generation package:
1. `segments.txt` — N segment prompts, joined by `|`. The downstream video creator
node splits the total length evenly across segments and generates them one by one.
Default format is full-reference (ref2va); FL2VA (first–last frame) and segmented
(one `segment-N/segment.txt` file per segment with `@PictureN`/`@AudioN` tokens)
are opt-in alternatives (see Mode selection).
2. `Image-N.json` — one 16:9 character-sheet text-to-image prompt per character that has
no user-provided image reference. Consumed one at a time by a ComfyUI workflow.
3. `Segment-img-N.json` — one 16:9 first-frame image prompt per segment (both modes);
the image engine renders it using the character sheets as reference images. In
FL2VA mode also `Segment-img-N-last.json` per segment for the last frame.
4. `image-map.txt` — manifest that says which `<Subject N>` maps to which image file.
Read `references/ref2va-segment-format.md` before drafting the first segment and
`references/t2i-character-sheet.md` before writing the first JSON. When the run uses
FL2VA mode, also read `references/fl2va-segment-format.md`; always read
`references/frame-image-prompts.md` before writing frame files. A complete end-to-end
example (including the audio and FL2VA variants) is in `references/worked-example.md`.
## Mode selection
- **ref2va (default).** Every run uses the six-section ref2va segment format unless
the user explicitly opts into first–last frames.
- **FL2VA (opt-in).** Switch when the user asks for "FL2V", "FL2VA", "first-last",
"first and last frame", or "start and end frame" per segment. In FL2VA mode the
video side receives ONLY the segment's first-frame image, last-frame image, and the
FL2VA-format prompt defined in `references/fl2va-segment-format.md` — never the
character sheets. The sheets are still generated: they feed the image side, which
renders the first/last frames with consistent characters. If the user's wording is
ambiguous, stay in ref2va mode; never mix the two formats in one run.
- **Segmented (opt-in).** Switch when the user asks for each segment as its own
`segment.txt` file or for `@Picture1` / `@Audio1`-style references. This is an
adaptation of ref2va defined in `references/segmented-output-format.md`: the
segment content follows `references/ref2va-segment-format.md` unchanged, but each
segment is written to its own `segment-N/segment.txt` file (no `segments.txt`, no
`|` separators) and reference tokens are written as `@PictureN` / `@AudioN`;
`<Subject N>` stays unchanged. Never mix formats in one run.
## Inputs
| Input | Required | Notes |
| --- | --- | --- |
| Vague prompt | yes | Rough scene/idea text from the user |
| Total video length in seconds | yes | Drives segment count |
| Song lyrics | no | Guide pacing only; NEVER appear in any output |
| Audio file mention | no | e.g. "use midnight.mp3" → enables `<Audio 1>` |
| Per-character image references | no | User-supplied files (e.g. `mary.png`) |
| Style hints | no | Honor if given; otherwise invent and lock |
| Frame mode | no | Default ref2va; FL2VA (first–last) only when explicitly requested |
Make reasonable creative choices and proceed. Do not block on questions; this skill
exists to expand vagueness into concrete, consistent detail.
## Scope rules
- Subjects are **recurring on-screen characters only**: people, animals, creatures,
robots that act as characters. Settings, props, and crowds stay in prose; they never
get `<Subject N>` numbers or sheets.
- Numbering is **global across the whole video**, assigned in order of first on-screen
appearance: `<Subject 1>`, `<Subject 2>`, ... For every subject, the image number
follows the subject number: Subject 2's reference is cited as `<Picture 2>` and, if
generated, saved as `Image-2.json`.
- A subject that does not appear in a segment must not be listed in that segment's
`subject_definitions`.
- No `<Video N>` labels: this skill never takes video input.
- Output language: English.
## Pipeline (follow in order)
**Step 1 — Intake.** Parse the vague prompt. List every character that will be on
screen. For each, decide: user-provided image reference (record filename) or needs a
generated sheet. Detect audio file mentions and lyrics. Detect style hints.
**Step 2 — Segment count.** `n = max(1, round(total_seconds / 10))` with round-half-up:
8s→1, 15s→2, 21s→2, 34s→3, 55s→6. Plan each segment as one beat of roughly
`total_seconds / n`. Assume segments flow into each other as one longer scene unless
the user says otherwise, so segment k's end state matches segment k+1's start state.
**Step 3 — Lock the story bible (before drafting).** Decide once, in a short working
note, and never change it afterward: each subject's hair, eyes, build, wardrobe, props,
and palette; the setting; time of day; lighting; color palette; camera character; style.
Where the vague prompt is silent, invent specifics and lock them. Every segment and
every sheet must use the identical locked values.
**Step 4 — Assign numbering.** Subject 1..M by first on-screen appearance across the
whole video. Map references: user file (e.g. `mary.png`) or `Image-N.json`.
**Step 5 — Draft each segment** in the selected mode. ref2va: exactly per
`references/ref2va-segment-format.md` — six sections in order, global labels, default
one continuous `[Shot 1]` per segment, extra shots only for time-synced changes,
timestamps relative to the segment start; when a subject sings to the provided audio,
use the exact fixed audio template and the exact lip-sync phrase defined in that
reference. FL2VA: exactly per `references/fl2va-segment-format.md` — instruction line
+ three fields, single-shot interpolation arc from first-frame state to last-frame
state, no reference labels.
**Step 6 — Write the frame image prompts** per `references/frame-image-prompts.md`:
one `Segment-img-N.json` per segment (N = 1..n) describing that segment's opening
state, and — FL2VA mode only — one `Segment-img-N-last.json` per segment describing
its final state. Captions reuse the locked story-bible values verbatim and anchor
each character to the supplied reference images.
**Step 7 — Write the character sheets** per `references/t2i-character-sheet.md`, one
`Image-N.json` per subject lacking a user reference. Every caption describes the
fixed four-panel photorealistic turnaround (face close-up, front, side facing 90°
right, back) on a white background with no text, no labels, and no borders — studio
photography of a real person, never a cartoon or designed card — and must reuse the
locked story-bible values verbatim.
**Step 8 — Write the manifest** `image-map.txt`, one line per subject:
```
Subject 1 - mary.png
Subject 2 - Image-2.json
```
User-supplied files keep their filenames; generated sheets are always `Image-N.json`.
The manifest never lists frame files — `Segment-img-N` numbering follows segments,
not subjects.
**Step 9 — Assemble the segment outputs.** ref2va/FL2VA: strip each segment text and
join with a single `|` (no spaces added around it). Exactly n segments, exactly n−1
`|` characters, no trailing separator, no extra headers or numbering; the `|`
character must never appear inside a segment's own text. Segmented: instead write
each segment to its own `segment-N/segment.txt` file per
`references/segmented-output-format.md` — no joined file, no `|` anywhere, and
`<Picture N>` / `<Audio N>` written as `@PictureN` / `@AudioN` (`<Subject N>`
unchanged).
**Step 10 — Validate** against the checklist below. When the environment can run Python,
also run `python scripts/validate_outputs.py --segments segments.txt --expected-n <n>`
(add `--mode fl2va --total-seconds <total>` in FL2VA mode; in segmented mode run
`python scripts/validate_outputs.py --mode segmented --segments-dir <folder> --expected-n <n>`).
## Output location
Write all outputs (`segments.txt`, `Image-N.json`, `Segment-img-N.json` frames,
`Segment-img-N-last.json` in FL2VA mode, `image-map.txt`) into the current project
folder of the project the skill is running on. If no project folder is
obvious, use the current working directory. Never write outputs into the skill's own
folder.
## Hard rules
1. Lyrics NEVER appear in any output text — not in `detailed_description`, not in
`<d>` tags, not in the audio sections. They only shape pacing and per-segment
content. Singing is expressed solely by the lip-sync phrase and the fixed audio
template.
2. Song vocals come from the reused audio: do not invent `(Sx)` speaker IDs or `<d>`
blocks for song lyrics. Ordinary spoken dialogue (non-song) may still use `(Sx)` +
`<d>[Language] ...</d>` per the format reference.
3. Global numbering is immutable across segments, sheets, and manifest.
4. The audio template and lip-sync phrase are used verbatim with only `N` substituted.
5. `[Shot 1]` never carries a timestamp; later shots use `At MM:SS.mmm` relative to the
segment start.
6. Every subject appearing anywhere in the video must have a manifest line.
7. Frame captions describe single full-bleed 16:9 film stills in the locked style —
photorealistic photographic language, no sheet/panel vocabulary, no text or
borders, no lyrics, no timestamps or labels, and every on-screen character anchored
to the supplied reference images with verbatim story-bible traits.
8. FL2VA mode: the instruction line of every segment is the exact fixed template with
only the final shot index and the two-decimal duration substituted; the prompt
contains no `<Subject N>` / `<Picture N>` / `<Video N>` / `<Audio N>` labels.
## Validation checklist (all must pass before delivery)
1. `segments.txt` has exactly n segments and exactly n−1 `|` separators.
2. (ref2va mode) Each segment contains all six sections in order: `subject_definitions`,
`summary`, `retention_analysis`, `detailed_description`, `overall_soundscape`,
`non_diegetic_music`.
3. (ref2va mode) Subject numbering is global and gapless (1..M); `<Picture N>` cited
inside `<Subject N>` always uses the same N. In FL2VA mode numbering lives on the
image side (manifest + sheets) only.
4. Every subject in the prompts has a manifest line; every `Image-N.json` in the
manifest exists, parses, has exactly the two keys `expanded_caption` and
`aspect_ratio`, and `aspect_ratio` is `"16:9"`. No orphan `Image-*.json` files.
5. No `<Video` anywhere; no lyric text anywhere.
6. (ref2va mode) Singing segments: summary prefix `[reference generation + audio
reuse]`, the exact fixed audio template in the sound sections, and the exact
lip-sync phrase inside `detailed_description`. Non-singing segments use
`[reference generation]`.
7. `[Shot 1]` has no timestamp; any `At MM:SS.mmm` appears only on `[Shot N]`, N≥2.
8. Sheet captions and segment descriptions agree with the locked story bible.
9. Each sheet caption describes the four-panel photorealistic turnaround (face
close-up panel, full-body front, side facing 90° right, back) on a clean white
background with equal spacing and no text, no labels, no borders; it uses
photorealistic/photographic language and contains no cartoon, illustration, or
design-board vocabulary (cartoon, anime, mascot, name tag, swatch, palette strip,
design board, CGI, 3D render...).
10. One `Segment-img-N.json` exists per segment (plus `Segment-img-N-last.json` per
segment in FL2VA mode, and none in ref2va mode); each is two-key JSON with
`"aspect_ratio": "16:9"`, photorealistic language, no banned vocabulary.
11. FL2VA mode: each segment's first line is the exact FL2VA instruction with the
correct final shot index and two-decimal duration; then `integrated_multimodal_description:`,
`overall_soundscape:`, `non_diegetic_music:` in order, and none of the six ref2va
section labels appear.
12. (Segmented mode) Exactly n `segment-N/segment.txt` files, folders numbered 1..n
with no gaps; each file is the ref2va single-segment format with `<Picture N>` /
`<Audio N>` written as `@PictureN` / `@AudioN`, `<Subject N>` unchanged, and no `|`
character anywhere.
## References
- `references/ref2va-segment-format.md` — authoritative per-segment format for the
default mode (sections, labels, shots, audio template, retention markers).
- `references/fl2va-segment-format.md` — the opt-in FL2VA first–last frame prompt
format (instruction line, interpolation arc, dialogue and audio rules).
- `references/segmented-output-format.md` — the opt-in segmented layout: one
`segment-N/segment.txt` file per segment with `@PictureN` / `@AudioN` reference
tokens; otherwise identical to the ref2va format.
- `references/t2i-character-sheet.md` — the four-panel photorealistic turnaround
sheet rules and the two-key JSON output contract.
- `references/frame-image-prompts.md` — per-segment first/last frame image prompt
rules (`Segment-img-N.json`, `Segment-img-N-last.json`).
- `references/worked-example.md` — full end-to-end example, including audio and
FL2VA variants.