MM-prompter/references/fl2va-segment-format.md

8.9 KiB
Raw Permalink Blame History

FL2VA Segment Format (first–last frame mode, optional)

This reference defines the per-segment video prompt when the run uses FL2VA mode — the user-opt-in alternative to the default ref2va format. Select FL2VA mode only when the user asks for it (phrases like "FL2V", "FL2VA", "first-last", "first and last frame", "start and end frame"). The default is always ref2va (references/ref2va-segment-format.md).

1. What changes in FL2VA mode

ref2va (default) FL2VA (opt-in)
Images the video side receives the character-sheet images cited as <Picture N> ONLY the segment's first-frame image and last-frame image
Video prompt format six sections, full-reference labels instruction line + three fields, no reference labels
Frame image prompts Segment-img-N.json (first frames) Segment-img-N.json AND Segment-img-N-last.json
Character sheets + manifest yes YES — still generated; they feed the image side so it can render the frames with consistent characters, but they are NOT supplied to the video side
segments.txt n segments joined by bare ` `

2. Prompt anatomy (exact)

Each segment is one plain-text document. Line 1 is the instruction line, followed by ONE blank line, then three fields, each inline (label: content) with a blank line between fields:

How the reference pictures align with the target video — Picture 1 (from Shot 1) aligns with the 0.00-second mark of the target video; Picture 2 (from Shot N) aligns with the S.SS-second mark of the target video.

integrated_multimodal_description: [Shot 1] <the full description>

overall_soundscape: <1–4 sentences of ambience>

non_diegetic_music: <1–3 sentences of score, or N/A>

Instruction-line substitutions, and nothing else may change:

  • Picture 1 is the segment's first-frame image; Picture 2 is the segment's last-frame image.
  • The first Shot N is literally Shot 1 (the video starts in the first shot).
  • The second Shot N is the index of the segment's FINAL shot. With the preferred single-shot layout this is Shot 1.
  • S.SS is the segment's duration — total_seconds / n — formatted with EXACTLY two decimals: 21s/2 segments → 10.50; 34s/3 → 11.33; 10s/1 → 10.00.
  • The em dash and the semicolon are part of the fixed template.
  • No full-reference labels anywhere in an FL2VA prompt: never <Subject N>, <Picture N>, <Video N>, <Audio N>.

3. integrated_multimodal_description

One flowing description starting directly with [Shot 1] (the style is woven into the opening sentences — there is no separate style paragraph). Characters are named by natural identity phrases that carry the locked story-bible traits at first mention (the trait wording must match the sheets and the story bible verbatim). Speaker IDs (S1), (S2) identify voices; see section 5.

Structure — the interpolation arc. FL2VA generates one segment between two still frames, so the description is a continuous path from the first frame to the last frame:

  1. Open exactly on the FIRST-FRAME state (what Segment-img-N.json depicts).
  2. Observable intermediate changes — actions, light shifts, approach, reactions.
  3. Progressively narrowing differences — the motion and framing converge on the target state.
  4. Land exactly on the LAST-FRAME state (what Segment-img-N-last.json depicts) in the final sentence; the last frame is reached at the segment's end.

Target roughly 350–500 words distributed across the whole segment.

Shots. Prefer ONE continuous [Shot 1] per segment. Split into [Shot N] At MM:SS.mmm, ... (N ≥ 2, timestamps strictly increasing, relative to the segment start) only when the segment genuinely needs a cut. [Shot 1] never carries a timestamp. When Picture 2 is cited in the instruction with from Shot N, the last-frame state must be reached inside that final shot.

Cuts. Use only these phrases for a cut: the camera cuts to, the shot cuts to, the shot transitions to, the shot changes to, the shot switches to. Cross-dissolves, fades, wipes appear only when the user explicitly requests them. Small distance or micro-angle changes are camera moves, never cuts.

Camera movement vocabulary (write as natural English sentences, never stacked tags; add amplitude and speed where relevant — with small amplitude, at slow speed): Zoom In, Zoom Out, Push In, Pull Out, Pan Left, Pan Right, Truck Left, Truck Right, Tilt Up, Tilt Down, Pedestal Up, Pedestal Down, Arc Shot, Tracking Shot, Static Shot, Shake Slightly, Shake Strongly, POV, Roll Clockwise, Roll Counterclockwise.

Screen text visible in the scene is preserved exactly, inside English double quotes.

4. overall_soundscape and non_diegetic_music

  • overall_soundscape — 1–4 sentences of diegetic ambience and physical sounds. Write N/A only when the user explicitly demands total silence.
  • non_diegetic_music — 1–3 sentences describing audience-only score: instrumentation, tempo, dynamics. No emotion words. N/A when there is no score. Never place a reused song here.

Song reuse in FL2VA mode (the user supplies a song file): the audio file is supplied to the video engine alongside the two frames and is the segment's complete final audio track. The prompt expresses it with these fixed pieces, substituting only the speaker number:

overall_soundscape: The supplied song plays as the segment's complete final audio track, with (S1) performing its vocals live.

non_diegetic_music: N/A

Inside integrated_multimodal_description, at each visible singing point, include the exact phrase (S1) lips stay synced to the song's vocals (substitute the real speaker number; no possessive apostrophe). Never quote lyrics — lyric words shape pacing only and never appear in any output.

5. Dialogue, voices, and speakers

  • Give every vocal source a stable ID in order of first speech: (S1), (S2), ... Two speakers at once: (S1,S2). Reuse the same ID for the same voice everywhere.
  • Identity phrases, speaker IDs, and actions stay OUTSIDE the <d> tags; inside <d> there is ONLY the language tag and the user's exact words: <d>[English] Last summer, I went to my grandfather's house.</d>
  • Off-screen speech uses the exact phrase says in an off-screen voiceover, followed by a clause stating the speaker's lips stay closed.
  • Dialogue that continues across a cut is wrapped in <scenetrans> ... </scenetrans> around the whole exchange; speech truncated by the video ending is wrapped in <cutoff> ... </cutoff>.
  • Ordinary spoken dialogue only; song lyrics never use <d> (see section 4).

6. Worked micro-example

One 10.50-second segment (segment 1 of the 21s desert project; full example in references/worked-example.md):

How the reference pictures align with the target video — Picture 1 (from Shot 1) aligns with the 0.00-second mark of the target video; Picture 2 (from Shot 1) aligns with the 10.50-second mark of the target video.

integrated_multimodal_description: [Shot 1] In a cinematic sci-fi realism style with low dusk light, long violet shadows, and a muted red-and-violet palette, the video opens exactly on the first frame: a wide establishing view of a red desert at dusk, the sky a deep gradient of burnt orange fading into indigo, where the astronaut — the woman in her thirties with short black hair under a clear dome helmet, wearing the matte-white EVA suit with orange stripe accents, grey reinforced gloves, and the wrist-mounted scanner on her left forearm — crests the rippled dune ridge on the left of frame mid-step, and the scout robot — the boxy machine with the pale-grey shell, single round blue lens eye, and short antenna — wheels up the slope to her right, treads leaving a neat triple track, while the three-story monolith of black glass stands small ahead on the basin floor, its surface drinking the last light. The camera tracks laterally at walking pace as the pair descends the dune's near face, boots sinking into cooling sand and wheels humming softly, the monolith growing taller in frame with each stride until a faint sub-bass hum pushes fine grit outward from its base in slow pulses. Reaching the flat basin, the camera eases into a slow push-in with small amplitude: the astronaut stops two meters short of the glass, raises the wrist scanner, and its amber readout reflects in her dome while the robot's blue lens dilates with a soft mechanical whir and its head tilts toward the monolith. The differences to the final frame narrow steadily — the walk ends, the scanner rises, the lens settles — and the shot lands exactly on the last frame: the two figures framed side by side before the monolith's mirrored face at dusk, scanner up, lens dilated, still and close, as the hum deepens.

overall_soundscape: Dry desert wind with fine grit against the helmet shell, the crunch of boots in cooling sand, the soft whir of the robot's wheels, and the monolith's low pulsing hum.

non_diegetic_music: N/A