MM-prompter/references/fl2va-segment-format.md

142 lines
8.9 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

# FL2VA Segment Format (first–last frame mode, optional)
This reference defines the per-segment video prompt when the run uses FL2VA mode —
the user-opt-in alternative to the default ref2va format. Select FL2VA mode only when
the user asks for it (phrases like "FL2V", "FL2VA", "first-last", "first and last
frame", "start and end frame"). The default is always ref2va
(`references/ref2va-segment-format.md`).
## 1. What changes in FL2VA mode
| | ref2va (default) | FL2VA (opt-in) |
| --- | --- | --- |
| Images the video side receives | the character-sheet images cited as `<Picture N>` | ONLY the segment's first-frame image and last-frame image |
| Video prompt format | six sections, full-reference labels | instruction line + three fields, no reference labels |
| Frame image prompts | `Segment-img-N.json` (first frames) | `Segment-img-N.json` AND `Segment-img-N-last.json` |
| Character sheets + manifest | yes | YES — still generated; they feed the image side so it can render the frames with consistent characters, but they are NOT supplied to the video side |
| `segments.txt` | n segments joined by bare `|` | identical (n segments, n−1 separators) |
## 2. Prompt anatomy (exact)
Each segment is one plain-text document. Line 1 is the instruction line, followed by
ONE blank line, then three fields, each inline (`label: content`) with a blank line
between fields:
```text
How the reference pictures align with the target video — Picture 1 (from Shot 1) aligns with the 0.00-second mark of the target video; Picture 2 (from Shot N) aligns with the S.SS-second mark of the target video.
integrated_multimodal_description: [Shot 1] <the full description>
overall_soundscape: <1–4 sentences of ambience>
non_diegetic_music: <1–3 sentences of score, or N/A>
```
Instruction-line substitutions, and nothing else may change:
- `Picture 1` is the segment's first-frame image; `Picture 2` is the segment's
last-frame image.
- The first `Shot N` is literally `Shot 1` (the video starts in the first shot).
- The second `Shot N` is the index of the segment's FINAL shot. With the preferred
single-shot layout this is `Shot 1`.
- `S.SS` is the segment's duration — total_seconds / n — formatted with EXACTLY two
decimals: 21s/2 segments → `10.50`; 34s/3 → `11.33`; 10s/1 → `10.00`.
- The em dash and the semicolon are part of the fixed template.
- No full-reference labels anywhere in an FL2VA prompt: never `<Subject N>`,
`<Picture N>`, `<Video N>`, `<Audio N>`.
## 3. `integrated_multimodal_description`
One flowing description starting directly with `[Shot 1]` (the style is woven into
the opening sentences — there is no separate style paragraph). Characters are named
by natural identity phrases that carry the locked story-bible traits at first mention
(the trait wording must match the sheets and the story bible verbatim). Speaker IDs
`(S1)`, `(S2)` identify voices; see section 5.
**Structure — the interpolation arc.** FL2VA generates one segment between two still
frames, so the description is a continuous path from the first frame to the last
frame:
1. Open exactly on the FIRST-FRAME state (what `Segment-img-N.json` depicts).
2. Observable intermediate changes — actions, light shifts, approach, reactions.
3. Progressively narrowing differences — the motion and framing converge on the
target state.
4. Land exactly on the LAST-FRAME state (what `Segment-img-N-last.json` depicts) in
the final sentence; the last frame is reached at the segment's end.
Target roughly 350–500 words distributed across the whole segment.
**Shots.** Prefer ONE continuous `[Shot 1]` per segment. Split into
`[Shot N] At MM:SS.mmm, ...` (N ≥ 2, timestamps strictly increasing, relative to the
segment start) only when the segment genuinely needs a cut. `[Shot 1]` never carries
a timestamp. When Picture 2 is cited in the instruction with `from Shot N`, the
last-frame state must be reached inside that final shot.
**Cuts.** Use only these phrases for a cut: `the camera cuts to`, `the shot cuts
to`, `the shot transitions to`, `the shot changes to`, `the shot switches to`.
Cross-dissolves, fades, wipes appear only when the user explicitly requests them.
Small distance or micro-angle changes are camera moves, never cuts.
**Camera movement vocabulary** (write as natural English sentences, never stacked
tags; add amplitude and speed where relevant — `with small amplitude`,
`at slow speed`): Zoom In, Zoom Out, Push In, Pull Out, Pan Left, Pan Right, Truck
Left, Truck Right, Tilt Up, Tilt Down, Pedestal Up, Pedestal Down, Arc Shot,
Tracking Shot, Static Shot, Shake Slightly, Shake Strongly, POV, Roll Clockwise,
Roll Counterclockwise.
**Screen text** visible in the scene is preserved exactly, inside English double
quotes.
## 4. `overall_soundscape` and `non_diegetic_music`
- `overall_soundscape` — 1–4 sentences of diegetic ambience and physical sounds.
Write `N/A` only when the user explicitly demands total silence.
- `non_diegetic_music` — 1–3 sentences describing audience-only score:
instrumentation, tempo, dynamics. No emotion words. `N/A` when there is no score.
Never place a reused song here.
**Song reuse in FL2VA mode** (the user supplies a song file): the audio file is
supplied to the video engine alongside the two frames and is the segment's complete
final audio track. The prompt expresses it with these fixed pieces, substituting
only the speaker number:
```text
overall_soundscape: The supplied song plays as the segment's complete final audio track, with (S1) performing its vocals live.
non_diegetic_music: N/A
```
Inside `integrated_multimodal_description`, at each visible singing point, include
the exact phrase `(S1) lips stay synced to the song's vocals` (substitute the real
speaker number; no possessive apostrophe). Never quote lyrics — lyric words shape
pacing only and never appear in any output.
## 5. Dialogue, voices, and speakers
- Give every vocal source a stable ID in order of first speech: `(S1)`, `(S2)`, ...
Two speakers at once: `(S1,S2)`. Reuse the same ID for the same voice everywhere.
- Identity phrases, speaker IDs, and actions stay OUTSIDE the `<d>` tags; inside
`<d>` there is ONLY the language tag and the user's exact words:
`<d>[English] Last summer, I went to my grandfather's house.</d>`
- Off-screen speech uses the exact phrase `says in an off-screen voiceover`,
followed by a clause stating the speaker's lips stay closed.
- Dialogue that continues across a cut is wrapped in `<scenetrans>` ... `</scenetrans>`
around the whole exchange; speech truncated by the video ending is wrapped in
`<cutoff>` ... `</cutoff>`.
- Ordinary spoken dialogue only; song lyrics never use `<d>` (see section 4).
## 6. Worked micro-example
One 10.50-second segment (segment 1 of the 21s desert project; full example in
`references/worked-example.md`):
```text
How the reference pictures align with the target video — Picture 1 (from Shot 1) aligns with the 0.00-second mark of the target video; Picture 2 (from Shot 1) aligns with the 10.50-second mark of the target video.
integrated_multimodal_description: [Shot 1] In a cinematic sci-fi realism style with low dusk light, long violet shadows, and a muted red-and-violet palette, the video opens exactly on the first frame: a wide establishing view of a red desert at dusk, the sky a deep gradient of burnt orange fading into indigo, where the astronaut — the woman in her thirties with short black hair under a clear dome helmet, wearing the matte-white EVA suit with orange stripe accents, grey reinforced gloves, and the wrist-mounted scanner on her left forearm — crests the rippled dune ridge on the left of frame mid-step, and the scout robot — the boxy machine with the pale-grey shell, single round blue lens eye, and short antenna — wheels up the slope to her right, treads leaving a neat triple track, while the three-story monolith of black glass stands small ahead on the basin floor, its surface drinking the last light. The camera tracks laterally at walking pace as the pair descends the dune's near face, boots sinking into cooling sand and wheels humming softly, the monolith growing taller in frame with each stride until a faint sub-bass hum pushes fine grit outward from its base in slow pulses. Reaching the flat basin, the camera eases into a slow push-in with small amplitude: the astronaut stops two meters short of the glass, raises the wrist scanner, and its amber readout reflects in her dome while the robot's blue lens dilates with a soft mechanical whir and its head tilts toward the monolith. The differences to the final frame narrow steadily — the walk ends, the scanner rises, the lens settles — and the shot lands exactly on the last frame: the two figures framed side by side before the monolith's mirrored face at dusk, scanner up, lens dilated, still and close, as the hum deepens.
overall_soundscape: Dry desert wind with fine grit against the helmet shell, the crunch of boots in cooling sand, the soft whir of the robot's wheels, and the monolith's low pulsing hum.
non_diegetic_music: N/A
```