From 45b4c1c21142681662d0e29c3f85a06dc9e7a786 Mon Sep 17 00:00:00 2001 From: mm-prompter-dev Date: Thu, 12 Nov 2026 19:57:08 +0000 Subject: [PATCH] Add Segment-img-N frame prompts and opt-in FL2VA first-last mode (ref2va stays default) --- README.md | 29 ++- SKILL.md | 113 +++++++--- references/fl2va-segment-format.md | 141 +++++++++++++ references/frame-image-prompts.md | 89 ++++++++ references/ref2va-segment-format.md | 12 +- references/worked-example.md | 105 ++++++++- scripts/validate_outputs.py | 317 ++++++++++++++++++++++------ 7 files changed, 700 insertions(+), 106 deletions(-) create mode 100644 references/fl2va-segment-format.md create mode 100644 references/frame-image-prompts.md diff --git a/README.md b/README.md index 490cf95..0d8a6da 100644 --- a/README.md +++ b/README.md @@ -3,11 +3,15 @@ An agent skill that turns a vague prompt plus a total video length (seconds) into a ready-to-run generation package: -- `segments.txt` — N segment prompts in full-reference (ref2va) format, joined by `|`. - The video creator node splits the total length evenly across segments - (`N = max(1, round(total_seconds / 10))`) and generates them one by one. +- `segments.txt` — N segment prompts, joined by `|`. The video creator node splits the + total length evenly across segments (`N = max(1, round(total_seconds / 10))`) and + generates them one by one. Default format is full-reference (ref2va); FL2VA + (first–last frame) is an opt-in alternative. - `Image-N.json` — one 16:9 character-sheet text-to-image prompt per character that has no user-provided image reference, consumed one at a time by a ComfyUI workflow. +- `Segment-img-N.json` — one 16:9 first-frame image prompt per segment, rendered by + the image engine with the character sheets as reference inputs; FL2VA mode adds + `Segment-img-N-last.json` per segment for the last frame. - `image-map.txt` — manifest mapping each `` to its image file (user-provided filename or generated `Image-N.json`). @@ -15,10 +19,12 @@ ready-to-run generation package: | Path | Purpose | | --- | --- | -| `SKILL.md` | Skill entry point: triggers, pipeline, output contract, validation checklist | -| `references/ref2va-segment-format.md` | Authoritative per-segment format (sections, labels, shots, audio template) | -| `references/t2i-character-sheet.md` | Character-sheet caption rules and the two-key JSON contract | -| `references/worked-example.md` | Complete end-to-end example, including the audio (singing) variant | +| `SKILL.md` | Skill entry point: triggers, mode selection, pipeline, output contract, validation checklist | +| `references/ref2va-segment-format.md` | Authoritative per-segment format for the default mode (sections, labels, shots, audio template) | +| `references/fl2va-segment-format.md` | The opt-in FL2VA first–last frame video prompt format | +| `references/t2i-character-sheet.md` | Character-sheet caption rules (four-panel photorealistic turnaround) and the two-key JSON contract | +| `references/frame-image-prompts.md` | Per-segment first/last frame image prompt rules (`Segment-img-N.json`) | +| `references/worked-example.md` | Complete end-to-end example, including the audio (singing) and FL2VA variants | | `scripts/validate_outputs.py` | Optional validator the agent can run on its outputs | | `segment-guide.md`, `t2i-prompt-rules/` | Source rule documents the skill was adapted from | @@ -27,8 +33,11 @@ ready-to-run generation package: 1. Pull this repo into your agent's skills folder so that `SKILL.md` is discoverable. 2. Give the agent a vague scene prompt and the total video length in seconds, plus optionally: song lyrics, an audio file mention (e.g. "use midnight.mp3"), - per-character image references, and style hints. + per-character image references, style hints, and — only if wanted — an explicit + request for FL2VA / first-last frame mode. 3. The agent writes `segments.txt`, `Image-N.json` for characters lacking references, - and `image-map.txt` into the project folder. + `Segment-img-N.json` (and `Segment-img-N-last.json` in FL2VA mode), and + `image-map.txt` into the project folder. 4. Verify before use: `python3 scripts/validate_outputs.py --segments segments.txt - --expected-n [--dir .] [--lyrics-file lyrics.txt]` + --expected-n [--dir .] [--lyrics-file lyrics.txt]` — add + `--mode fl2va --total-seconds ` for FL2VA runs. diff --git a/SKILL.md b/SKILL.md index 3590bde..4431a6b 100644 --- a/SKILL.md +++ b/SKILL.md @@ -1,6 +1,6 @@ --- name: mm-prompter -description: Writes multi-segment ref2va video prompts plus T2I character-sheet prompts from a vague scene idea and a total video length in seconds. Use when the user gives a rough video/scene concept (optionally with song lyrics or an audio file name) and wants numbered video-generation segments joined by "|", character sheets as Image-N.json for characters that lack an image reference, and an image-map.txt manifest. +description: Writes multi-segment ref2va video prompts plus T2I character-sheet prompts and per-segment first-frame image prompts from a vague scene idea and a total video length in seconds. Use when the user gives a rough video/scene concept (optionally with song lyrics or an audio file name) and wants numbered video-generation segments joined by "|", character sheets as Image-N.json for characters that lack an image reference, Segment-img-N.json frame prompts, and an image-map.txt manifest. Optional opt-in FL2VA mode adds last-frame prompts and first-last video prompts. --- # MM Prompt Creator @@ -8,16 +8,34 @@ description: Writes multi-segment ref2va video prompts plus T2I character-sheet Turn one vague prompt and one number (total seconds) into a complete, ready-to-run generation package: -1. `segments.txt` — N segment prompts in full-reference (ref2va) format, joined by `|`. - The downstream video creator node splits the total length evenly across segments and - generates them one by one. +1. `segments.txt` — N segment prompts, joined by `|`. The downstream video creator + node splits the total length evenly across segments and generates them one by one. + Default format is full-reference (ref2va); FL2VA (first–last frame) is an opt-in + alternative (see Mode selection). 2. `Image-N.json` — one 16:9 character-sheet text-to-image prompt per character that has no user-provided image reference. Consumed one at a time by a ComfyUI workflow. -3. `image-map.txt` — manifest that says which `` maps to which image file. +3. `Segment-img-N.json` — one 16:9 first-frame image prompt per segment (both modes); + the image engine renders it using the character sheets as reference images. In + FL2VA mode also `Segment-img-N-last.json` per segment for the last frame. +4. `image-map.txt` — manifest that says which `` maps to which image file. -Read `references/ref2va-segment-format.md` before drafting the first segment, and -`references/t2i-character-sheet.md` before writing the first JSON. A complete end-to-end -example (including the audio variant) is in `references/worked-example.md`. +Read `references/ref2va-segment-format.md` before drafting the first segment and +`references/t2i-character-sheet.md` before writing the first JSON. When the run uses +FL2VA mode, also read `references/fl2va-segment-format.md`; always read +`references/frame-image-prompts.md` before writing frame files. A complete end-to-end +example (including the audio and FL2VA variants) is in `references/worked-example.md`. + +## Mode selection + +- **ref2va (default).** Every run uses the six-section ref2va segment format unless + the user explicitly opts into first–last frames. +- **FL2VA (opt-in).** Switch when the user asks for "FL2V", "FL2VA", "first-last", + "first and last frame", or "start and end frame" per segment. In FL2VA mode the + video side receives ONLY the segment's first-frame image, last-frame image, and the + FL2VA-format prompt defined in `references/fl2va-segment-format.md` — never the + character sheets. The sheets are still generated: they feed the image side, which + renders the first/last frames with consistent characters. If the user's wording is + ambiguous, stay in ref2va mode; never mix the two formats in one run. ## Inputs @@ -29,6 +47,7 @@ example (including the audio variant) is in `references/worked-example.md`. | Audio file mention | no | e.g. "use midnight.mp3" → enables `