226 lines
14 KiB
Markdown
226 lines
14 KiB
Markdown
---
|
||
name: mm-prompter
|
||
description: Writes multi-segment ref2va video prompts plus T2I character-sheet prompts and per-segment first-frame image prompts from a vague scene idea and a total video length in seconds. Use when the user gives a rough video/scene concept (optionally with song lyrics or an audio file name) and wants numbered video-generation segments joined by "|", character sheets as Image-N.json for characters that lack an image reference, Segment-img-N.json frame prompts, and an image-map.txt manifest. Optional opt-in FL2VA mode adds last-frame prompts and first-last video prompts. Optional opt-in segmented mode writes one segment-N/segment.txt file per segment using @PictureN/@Audio1-style @-references instead of the joined file.
|
||
---
|
||
|
||
# MM Prompt Creator
|
||
|
||
Turn one vague prompt and one number (total seconds) into a complete, ready-to-run
|
||
generation package:
|
||
|
||
1. `segments.txt` — N segment prompts, joined by `|`. The downstream video creator
|
||
node splits the total length evenly across segments and generates them one by one.
|
||
Default format is full-reference (ref2va); FL2VA (first–last frame) and segmented
|
||
(one `segment-N/segment.txt` file per segment with `@PictureN`/`@AudioN` tokens)
|
||
are opt-in alternatives (see Mode selection).
|
||
2. `Image-N.json` — one 16:9 character-sheet text-to-image prompt per character that has
|
||
no user-provided image reference. Consumed one at a time by a ComfyUI workflow.
|
||
3. `Segment-img-N.json` — one 16:9 first-frame image prompt per segment (both modes);
|
||
the image engine renders it using the character sheets as reference images. In
|
||
FL2VA mode also `Segment-img-N-last.json` per segment for the last frame.
|
||
4. `image-map.txt` — manifest that says which `<Subject N>` maps to which image file.
|
||
|
||
Read `references/ref2va-segment-format.md` before drafting the first segment and
|
||
`references/t2i-character-sheet.md` before writing the first JSON. When the run uses
|
||
FL2VA mode, also read `references/fl2va-segment-format.md`; always read
|
||
`references/frame-image-prompts.md` before writing frame files. A complete end-to-end
|
||
example (including the audio and FL2VA variants) is in `references/worked-example.md`.
|
||
|
||
## Mode selection
|
||
|
||
- **ref2va (default).** Every run uses the six-section ref2va segment format unless
|
||
the user explicitly opts into first–last frames.
|
||
- **FL2VA (opt-in).** Switch when the user asks for "FL2V", "FL2VA", "first-last",
|
||
"first and last frame", or "start and end frame" per segment. In FL2VA mode the
|
||
video side receives ONLY the segment's first-frame image, last-frame image, and the
|
||
FL2VA-format prompt defined in `references/fl2va-segment-format.md` — never the
|
||
character sheets. The sheets are still generated: they feed the image side, which
|
||
renders the first/last frames with consistent characters. If the user's wording is
|
||
ambiguous, stay in ref2va mode; never mix the two formats in one run.
|
||
- **Segmented (opt-in).** Switch when the user asks for each segment as its own
|
||
`segment.txt` file or for `@Picture1` / `@Audio1`-style references. This is an
|
||
adaptation of ref2va defined in `references/segmented-output-format.md`: the
|
||
segment content follows `references/ref2va-segment-format.md` unchanged, but each
|
||
segment is written to its own `segment-N/segment.txt` file (no `segments.txt`, no
|
||
`|` separators) and reference tokens are written as `@PictureN` / `@AudioN`;
|
||
`<Subject N>` stays unchanged. Never mix formats in one run.
|
||
|
||
## Inputs
|
||
|
||
| Input | Required | Notes |
|
||
| --- | --- | --- |
|
||
| Vague prompt | yes | Rough scene/idea text from the user |
|
||
| Total video length in seconds | yes | Drives segment count |
|
||
| Song lyrics | no | Guide pacing only; NEVER appear in any output |
|
||
| Audio file mention | no | e.g. "use midnight.mp3" → enables `<Audio 1>` |
|
||
| Per-character image references | no | User-supplied files (e.g. `mary.png`) |
|
||
| Style hints | no | Honor if given; otherwise invent and lock |
|
||
| Frame mode | no | Default ref2va; FL2VA (first–last) only when explicitly requested |
|
||
|
||
Make reasonable creative choices and proceed. Do not block on questions; this skill
|
||
exists to expand vagueness into concrete, consistent detail.
|
||
|
||
## Scope rules
|
||
|
||
- Subjects are **recurring on-screen characters only**: people, animals, creatures,
|
||
robots that act as characters. Settings, props, and crowds stay in prose; they never
|
||
get `<Subject N>` numbers or sheets.
|
||
- Numbering is **global across the whole video**, assigned in order of first on-screen
|
||
appearance: `<Subject 1>`, `<Subject 2>`, ... For every subject, the image number
|
||
follows the subject number: Subject 2's reference is cited as `<Picture 2>` and, if
|
||
generated, saved as `Image-2.json`.
|
||
- A subject that does not appear in a segment must not be listed in that segment's
|
||
`subject_definitions`.
|
||
- No `<Video N>` labels: this skill never takes video input.
|
||
- Output language: English.
|
||
|
||
## Pipeline (follow in order)
|
||
|
||
**Step 1 — Intake.** Parse the vague prompt. List every character that will be on
|
||
screen. For each, decide: user-provided image reference (record filename) or needs a
|
||
generated sheet. Detect audio file mentions and lyrics. Detect style hints.
|
||
|
||
**Step 2 — Segment count.** `n = max(1, round(total_seconds / 10))` with round-half-up:
|
||
8s→1, 15s→2, 21s→2, 34s→3, 55s→6. Plan each segment as one beat of roughly
|
||
`total_seconds / n`. Assume segments flow into each other as one longer scene unless
|
||
the user says otherwise, so segment k's end state matches segment k+1's start state.
|
||
|
||
**Step 3 — Lock the story bible (before drafting).** Decide once, in a short working
|
||
note, and never change it afterward: each subject's hair, eyes, build, wardrobe, props,
|
||
and palette; the setting; time of day; lighting; color palette; camera character; style.
|
||
Where the vague prompt is silent, invent specifics and lock them. Every segment and
|
||
every sheet must use the identical locked values.
|
||
|
||
**Step 4 — Assign numbering.** Subject 1..M by first on-screen appearance across the
|
||
whole video. Map references: user file (e.g. `mary.png`) or `Image-N.json`.
|
||
|
||
**Step 5 — Draft each segment** in the selected mode. ref2va: exactly per
|
||
`references/ref2va-segment-format.md` — six sections in order, global labels, default
|
||
one continuous `[Shot 1]` per segment, extra shots only for time-synced changes,
|
||
timestamps relative to the segment start; when a subject sings to the provided audio,
|
||
use the exact fixed audio template and the exact lip-sync phrase defined in that
|
||
reference. FL2VA: exactly per `references/fl2va-segment-format.md` — instruction line
|
||
+ three fields, single-shot interpolation arc from first-frame state to last-frame
|
||
state, no reference labels.
|
||
|
||
**Step 6 — Write the frame image prompts** per `references/frame-image-prompts.md`:
|
||
one `Segment-img-N.json` per segment (N = 1..n) describing that segment's opening
|
||
state, and — FL2VA mode only — one `Segment-img-N-last.json` per segment describing
|
||
its final state. Captions reuse the locked story-bible values verbatim and anchor
|
||
each character to the supplied reference images.
|
||
|
||
**Step 7 — Write the character sheets** per `references/t2i-character-sheet.md`, one
|
||
`Image-N.json` per subject lacking a user reference. Every caption describes the
|
||
fixed four-panel photorealistic turnaround (face close-up, front, side facing 90°
|
||
right, back) on a white background with no text, no labels, and no borders — studio
|
||
photography of a real person, never a cartoon or designed card — and must reuse the
|
||
locked story-bible values verbatim.
|
||
|
||
**Step 8 — Write the manifest** `image-map.txt`, one line per subject:
|
||
|
||
```
|
||
Subject 1 - mary.png
|
||
Subject 2 - Image-2.json
|
||
```
|
||
|
||
User-supplied files keep their filenames; generated sheets are always `Image-N.json`.
|
||
The manifest never lists frame files — `Segment-img-N` numbering follows segments,
|
||
not subjects.
|
||
|
||
**Step 9 — Assemble the segment outputs.** ref2va/FL2VA: strip each segment text and
|
||
join with a single `|` (no spaces added around it). Exactly n segments, exactly n−1
|
||
`|` characters, no trailing separator, no extra headers or numbering; the `|`
|
||
character must never appear inside a segment's own text. Segmented: instead write
|
||
each segment to its own `segment-N/segment.txt` file per
|
||
`references/segmented-output-format.md` — no joined file, no `|` anywhere, and
|
||
`<Picture N>` / `<Audio N>` written as `@PictureN` / `@AudioN` (`<Subject N>`
|
||
unchanged).
|
||
|
||
**Step 10 — Validate** against the checklist below. When the environment can run Python,
|
||
also run `python scripts/validate_outputs.py --segments segments.txt --expected-n <n>`
|
||
(add `--mode fl2va --total-seconds <total>` in FL2VA mode; in segmented mode run
|
||
`python scripts/validate_outputs.py --mode segmented --segments-dir <folder> --expected-n <n>`).
|
||
|
||
## Output location
|
||
|
||
Write all outputs (`segments.txt`, `Image-N.json`, `Segment-img-N.json` frames,
|
||
`Segment-img-N-last.json` in FL2VA mode, `image-map.txt`) into the current project
|
||
folder of the project the skill is running on. If no project folder is
|
||
obvious, use the current working directory. Never write outputs into the skill's own
|
||
folder.
|
||
|
||
## Hard rules
|
||
|
||
1. Lyrics NEVER appear in any output text — not in `detailed_description`, not in
|
||
`<d>` tags, not in the audio sections. They only shape pacing and per-segment
|
||
content. Singing is expressed solely by the lip-sync phrase and the fixed audio
|
||
template.
|
||
2. Song vocals come from the reused audio: do not invent `(Sx)` speaker IDs or `<d>`
|
||
blocks for song lyrics. Ordinary spoken dialogue (non-song) may still use `(Sx)` +
|
||
`<d>[Language] ...</d>` per the format reference.
|
||
3. Global numbering is immutable across segments, sheets, and manifest.
|
||
4. The audio template and lip-sync phrase are used verbatim with only `N` substituted.
|
||
5. `[Shot 1]` never carries a timestamp; later shots use `At MM:SS.mmm` relative to the
|
||
segment start.
|
||
6. Every subject appearing anywhere in the video must have a manifest line.
|
||
7. Frame captions describe single full-bleed 16:9 film stills in the locked style —
|
||
photorealistic photographic language, no sheet/panel vocabulary, no text or
|
||
borders, no lyrics, no timestamps or labels, and every on-screen character anchored
|
||
to the supplied reference images with verbatim story-bible traits.
|
||
8. FL2VA mode: the instruction line of every segment is the exact fixed template with
|
||
only the final shot index and the two-decimal duration substituted; the prompt
|
||
contains no `<Subject N>` / `<Picture N>` / `<Video N>` / `<Audio N>` labels.
|
||
|
||
## Validation checklist (all must pass before delivery)
|
||
|
||
1. `segments.txt` has exactly n segments and exactly n−1 `|` separators.
|
||
2. (ref2va mode) Each segment contains all six sections in order: `subject_definitions`,
|
||
`summary`, `retention_analysis`, `detailed_description`, `overall_soundscape`,
|
||
`non_diegetic_music`.
|
||
3. (ref2va mode) Subject numbering is global and gapless (1..M); `<Picture N>` cited
|
||
inside `<Subject N>` always uses the same N. In FL2VA mode numbering lives on the
|
||
image side (manifest + sheets) only.
|
||
4. Every subject in the prompts has a manifest line; every `Image-N.json` in the
|
||
manifest exists, parses, has exactly the two keys `expanded_caption` and
|
||
`aspect_ratio`, and `aspect_ratio` is `"16:9"`. No orphan `Image-*.json` files.
|
||
5. No `<Video` anywhere; no lyric text anywhere.
|
||
6. (ref2va mode) Singing segments: summary prefix `[reference generation + audio
|
||
reuse]`, the exact fixed audio template in the sound sections, and the exact
|
||
lip-sync phrase inside `detailed_description`. Non-singing segments use
|
||
`[reference generation]`.
|
||
7. `[Shot 1]` has no timestamp; any `At MM:SS.mmm` appears only on `[Shot N]`, N≥2.
|
||
8. Sheet captions and segment descriptions agree with the locked story bible.
|
||
9. Each sheet caption describes the four-panel photorealistic turnaround (face
|
||
close-up panel, full-body front, side facing 90° right, back) on a clean white
|
||
background with equal spacing and no text, no labels, no borders; it uses
|
||
photorealistic/photographic language and contains no cartoon, illustration, or
|
||
design-board vocabulary (cartoon, anime, mascot, name tag, swatch, palette strip,
|
||
design board, CGI, 3D render...).
|
||
10. One `Segment-img-N.json` exists per segment (plus `Segment-img-N-last.json` per
|
||
segment in FL2VA mode, and none in ref2va mode); each is two-key JSON with
|
||
`"aspect_ratio": "16:9"`, photorealistic language, no banned vocabulary.
|
||
11. FL2VA mode: each segment's first line is the exact FL2VA instruction with the
|
||
correct final shot index and two-decimal duration; then `integrated_multimodal_description:`,
|
||
`overall_soundscape:`, `non_diegetic_music:` in order, and none of the six ref2va
|
||
section labels appear.
|
||
12. (Segmented mode) Exactly n `segment-N/segment.txt` files, folders numbered 1..n
|
||
with no gaps; each file is the ref2va single-segment format with `<Picture N>` /
|
||
`<Audio N>` written as `@PictureN` / `@AudioN`, `<Subject N>` unchanged, and no `|`
|
||
character anywhere.
|
||
|
||
## References
|
||
|
||
- `references/ref2va-segment-format.md` — authoritative per-segment format for the
|
||
default mode (sections, labels, shots, audio template, retention markers).
|
||
- `references/fl2va-segment-format.md` — the opt-in FL2VA first–last frame prompt
|
||
format (instruction line, interpolation arc, dialogue and audio rules).
|
||
- `references/segmented-output-format.md` — the opt-in segmented layout: one
|
||
`segment-N/segment.txt` file per segment with `@PictureN` / `@AudioN` reference
|
||
tokens; otherwise identical to the ref2va format.
|
||
- `references/t2i-character-sheet.md` — the four-panel photorealistic turnaround
|
||
sheet rules and the two-key JSON output contract.
|
||
- `references/frame-image-prompts.md` — per-segment first/last frame image prompt
|
||
rules (`Segment-img-N.json`, `Segment-img-N-last.json`).
|
||
- `references/worked-example.md` — full end-to-end example, including audio and
|
||
FL2VA variants.
|