mmprompt/README.md

309 lines
18 KiB
Markdown
Executable File
Raw Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

# mmprompt
## h3-plan-gate — MiniMax H3 plan-JSON agent workflow + validator
Prompt set and tooling that turn a story idea into the `plan_json_input` JSON for the
**Production Plan** node (`MiniMax H3 Context Loop Plan` / `MiniMax H3 Plan (Modern)`) in
[ComfyUI-MiniMaxH3-Context-Loop](https://github.com/ethanfel/ComfyUI-MiniMaxH3-Context-Loop),
verified against commit `43afeff`. The deliverable is always **one plan JSON with one shot
per scene** — the node renders every shot as its own clip and chains the clips through the
H3 context loop into one continuous multi-clip video, so pasting one `plan.json` and
queuing once is all the rendering takes.
Instead of asking the LLM to emit that whole JSON in one turn, the gate is agent-shaped:
one outline turn, then **the LLM writes each scene one at a time** — a single turn that
exists purely to compose that scene from every element a good scene needs (subject,
style, continuity, blocking, camera, performance, texture, sound, exit state) — and the
scripts **put it all together at the end** into the same plan JSON the single-pass
prompt would have produced, validated before it ever reaches the node. The point is
compositional attention, not word counts: the validator checks structure and trips only
a thin-scene floor, so any scene can be rewritten until its elements are right, then
re-merged without touching the rest.
### Files
| File | Role |
|---|---|
| `SKILL.md` | Agent Skills entry point — this whole folder is a skill. Cline: copy it to `.cline/skills/h3-plan-gate/` (workspace) or `~/.cline/skills/` (global); any Agent-Skills-compatible agent (Claude Code, Cursor, OpenCode, Codex) can load it too |
| `h3_scene_agent_system_prompt.md` | Stage 1 + 3 system prompts: **PROMPT A** (scene-guide planner) and **PROMPT B** (one-scene-per-turn writer with a `_handoff` chain) — Stage 2 is the human approval between them |
| `h3_plan_assembler_system_prompt.md` | Stage 4 LLM merge prompt — only when you cannot run scripts |
| `h3_plan_json_system_prompt.md` | Single-pass variant (whole plan in one turn) — fastest for 2–4 scene stories |
| `build_plan.py` | Deterministic Stage 4 merger — verifies the continuity across the chain and puts all the scenes together at the end: strips `_handoff`, injects `"prompt_prefix": [""]` + defaults, validates everything before writing `plan.json` |
| `verify_plan_rules.py` | Node-replica validator (final plans *and* per-turn fragments): six-section Ref2VA prompt contract, the three subject-definition forms, refmod isolation, structure, handoff truth, continuity, thin-scene floor — no word band — with an embedded test suite |
| `count_prompt_tokens.py` | o200k token counter for the prompt fences (needs `pip install tiktoken`) |
| `h3_agent_workflow_examples/` | Worked 3-scene candy-store run in the six-section Ref2VA setup (refmod-pinned `<Subject 1>` + no-image shopkeeper): `outline.json` → `scene_01..03.json` → `plan.json` |
### Agent workflow (story in, plan.json out)
The loop the user runs: **story + scene count → scene guide → approval → one scene at a
time → continuity verified → plan.json**.
**Stage 1 — Scene guide.** Put PROMPT A (from `h3_scene_agent_system_prompt.md`) in the
system field, the story idea (and how many scenes) in the user field. Save the reply as
`outline.json`. It freezes scene count, ids, durations, beats, verbatim dialogue
distribution, and the `style_and_cast` block so later turns cannot drift.
**Stage 2 — Approval (human in the loop, never skipped).** Show the user the guide in
plain language — one line per scene plus cast and style — and wait. They approve it or
ask for changes; apply every change to `outline.json` and show the changed rows again.
Stage 3 starts only after an explicit approval — the guide is the contract the scenes
are composed against.
**Stage 3 — One scene per turn (~10 s scenes from the approved guide).** Put PROMPT B in
the system field and paste this turn template as the user message:
```text
STORY: <the full story idea>
STYLE AND CAST: <paste the "style_and_cast" object from the outline>
OUTLINE: <paste the "scenes" array from the outline>
WRITE SCENE: 2
PREVIOUS HANDOFF: <paste the previous scene's "_handoff" object — or: This is scene 1.>
```
Save each reply as `scene_NN.json` (zero-padded). Each fragment carries the native H3
six-section Ref2VA prompt (`subject_definitions:` → `summary: [reference generation]` →
`retention_analysis:` → `detailed_description:` → `overall_soundscape:` →
`non_diegetic_music: N/A`), the full scene composition, and a truthful `_handoff`
object for the next turn. Optional gate after each turn:
```sh
.venv\Scripts\python.exe h3-plan-gate/verify_plan_rules.py fragment scene_02.json --index 2 --prev scene_01.json
```
(All command blocks below use the Windows venv form; on macOS/Linux replace
`.venv\Scripts\python.exe` with `python3`.)
**Stage 4 — Continuity verified, assembly (deterministic, preferred):**
```sh
.venv\Scripts\python.exe h3-plan-gate/build_plan.py outline.json scene_01.json scene_02.json scene_03.json -o plan.json
```
The merge IS the continuity check: it validates every fragment in the `_handoff` chain
(each scene opens inside the previous scene's real end_state, soundscapes continue, ids,
durations, and dialogue match the approved guide), then validates the merged plan — only
clean JSON is written. Do **not** edit fragment prose by hand during assembly; if you
want a scene changed, re-run its Stage 3
turn and re-merge. Use the LLM assembler prompt only in a pure-chat setup with no script
execution, and diff its output against the fragments if you do (small models transcribe
long strings unreliably; `build_plan.py` is byte-exact by construction).
## Example: story idea → final output
The complete run of the shipped example (`h3_agent_workflow_examples/`) — the candy-store
reference run, with a refmod-pinned child subject and a no-image shopkeeper in the
six-section Ref2VA setup. The same files your agent should produce for any story.
**1. Give Cline the story idea.** With the skill installed, the whole request is:
```text
Use the h3-plan-gate skill. Story: a 9 year old boy called Sam — blonde hair in
pigtails, dresses like a baby girl — visits the candy store with his small red coin
purse: he browses the shelves, picks a red-and-white swirl lollipop, and pays the
shopkeeper. Three scenes. Use this character block for Sam exactly as given:
&identity: Sam is a 9 year old boy with his blonde hair in pigtails that dresses like
a baby girl; voice: Sam is a 9 year old boy with his blonde hair in pigtails that
dresses like a baby girl&
```
**2. The agent runs the smoke test** — proving the toolchain before writing anything:
```sh
.venv\Scripts\python.exe h3-plan-gate/verify_plan_rules.py
.venv\Scripts\python.exe h3-plan-gate/build_plan.py --check h3-plan-gate/h3_agent_workflow_examples/plan.json
```
Expect the suite to pass and, from the second command, `PLAN OK` with
`total delivered: 685 frames = 28.54s`.
**3. The agent writes `outline.json`** — the skeleton that freezes cast, style, ids,
durations, and dialogue so later scenes cannot drift. Sam is a user-refmod subject
(strip the `&` wrapper at the scene stage); the shopkeeper has no reference image, so
his definition carries physical traits only and the wardrobe moves to the scene bodies:
```json
{
"title": "candy_store",
"defaults": { "duration_seconds": 10, "steps": 5 },
"style_and_cast": {
"subjects": [
"<Subject 1> — Sam (user refmod, carry verbatim into subject_definitions): identity: Sam is a 9 year old boy with his blonde hair in pigtails that dresses like a baby girl; voice: Sam is a 9 year old boy with his blonde hair in pigtails that dresses like a baby girl",
"<Subject 2> — the shopkeeper: 55 year old man, short grey hair, brown eyes, heavy build; green apron over a rolled-sleeve white shirt, dark trousers (look woven into each [Shot 1] body)"
],
"style": "Warm photoreal candy-store realism, soft late-afternoon light pooling through the front windows, saturated jar-glass golds and wrapper reds under a gentle film grain, static eye-level camera held at the child's height.",
"soundscape_seed": "the low hum of a candy-store fridge case, cellophane wrapper crinkle, muffled street traffic behind the front glass"
},
"scenes": [
{ "index": 1, "id": "candy_browse", "duration_seconds": 10,
"beat": "Sam browses the candy shelves, taps the glass in front of a swirl lollipop, and turns toward the counter mid-step.",
"dialogue": [] },
{ "index": 2, "id": "lollipop_pickup", "duration_seconds": 10,
"beat": "Sam picks up the red-and-white swirl lollipop from the counter and his left hand finds the small red coin purse at his hip.",
"dialogue": [] },
{ "index": 3, "id": "pay_shopkeeper", "duration_seconds": 10,
"beat": "Sam pays the shopkeeper, coins into the springing register drawer, lollipop raised high.",
"dialogue": [] }
]
}
```
**4. The agent shows the guide — the user approves or changes it.** Before any scene is
written, the agent presents the guide in plain language and waits:
```text
The guide for "candy_store" — 3 scenes, warm photoreal candy-store realism:
1. candy_browse 10s — Sam browses the candy shelves, taps the glass in front of
a swirl lollipop, turns toward the counter mid-step. No dialogue.
2. lollipop_pickup 10s — Sam picks up the swirl lollipop and his left hand finds the
red coin purse at his hip. No dialogue.
3. pay_shopkeeper 10s — Sam pays the shopkeeper, coins into the springing register
drawer, lollipop raised high. No dialogue.
Cast: <Subject 1> Sam (user refmod, verbatim). <Subject 2> the shopkeeper (no image —
traits only, wardrobe woven into the scene bodies).
Approve, or tell me what to change.
```
```text
Approved — go.
```
Any change request ("make scene 2 slower", "he counts the coins first") is
applied to `outline.json` first and the changed rows shown again — scenes are composed
only from the approved guide.
**5. The agent composes each scene one at a time** — one file per turn, each turn
given over to making that one scene a complete composition, validating it before
moving to the next:
```sh
.venv\Scripts\python.exe h3-plan-gate/verify_plan_rules.py fragment scene_01.json --index 1 --total 3
.venv\Scripts\python.exe h3-plan-gate/verify_plan_rules.py fragment scene_02.json --index 2 --total 3 --prev scene_01.json
.venv\Scripts\python.exe h3-plan-gate/verify_plan_rules.py fragment scene_03.json --index 3 --total 3 --prev scene_02.json
```
Each fragment carries the six-section Ref2VA prompt and a truthful `_handoff`. Note the
refmod isolation in the body: `<Subject 1>` is referenced by the bare token — the refmod
already pins the look, and the workflow injects the reference blocks itself. Scene 2
(body elided — the real file carries the full blocking, camera work, texture, and
lighting detail):
```json
{
"id": "lollipop_pickup",
"duration_seconds": 10,
"prompt": [
"subject_definitions:",
"<Subject 1> identity: Sam is a 9 year old boy with his blonde hair in pigtails that dresses like a baby girl; voice: Sam is a 9 year old boy with his blonde hair in pigtails that dresses like a baby girl",
"<Subject 2> is a 55 year old man with short grey hair, brown eyes and a heavy build",
"summary:",
"[reference generation]",
"retention_analysis:",
"<Subject 1>: fully_preserved",
"<Subject 2>: fully_preserved",
"detailed_description:",
"Warm photoreal candy-store realism, soft late-afternoon light pooling through the front windows, saturated jar-glass golds and wrapper reds under a gentle film grain.",
"Static eye-level camera held at <Subject 1>'s eye height, shallow depth of field on the candy jars.",
"[Shot 1] Continuing the same shot without resetting the motion: <Subject 1> closes the last step to the counter and leans both hands on the polished wood. … End mid-motion: the lollipop held high in his right fist, his left hand finding the small red coin purse on its strap at his hip.",
"overall_soundscape:",
"the same low fridge-case hum and wrapper crinkle continue, the lollipop's cellophane crackling under small fingers, the till drawer rattling softly below the counter, street traffic still muffled behind the front glass.",
"non_diegetic_music:",
"N/A"
],
"_handoff": { "end_state": "…", "soundscape": "…", "dialogue_used": [] }
}
```
**6. The agent verifies the continuity and puts it all together at the end** — merging
every scene into one plan JSON, never writing plan JSON by hand:
```sh
.venv\Scripts\python.exe h3-plan-gate/build_plan.py outline.json scene_01.json scene_02.json scene_03.json -o plan.json
.venv\Scripts\python.exe h3-plan-gate/build_plan.py --check plan.json
```
Output:
```text
outline: outline.json (3 scenes)
fragment 1/3 … fragment 2/3 … fragment 3/3 …
Merged plan:
shot 1 id='candy_browse' raw=243 delivered=243 steps=5
shot 2 id='lollipop_pickup' raw=243 delivered=221 steps=5
shot 3 id='pay_shopkeeper' raw=243 delivered=221 steps=5
total delivered: 685 frames = 28.54s
shot 1 id='candy_browse' six-section ref2va OK (2 subject(s): no-image, refmod; body 138 words)
shot 2 id='lollipop_pickup' six-section ref2va OK (2 subject(s): no-image, refmod; body 156 words)
shot 3 id='pay_shopkeeper' six-section ref2va OK (2 subject(s): no-image, refmod; body 156 words)
H3 section structure: all shots carry the six Ref2VA sections in canonical order
wrote plan.json - paste its full contents into plan_json_input
```
The deliverable is the single `plan.json` — `"prompt_prefix": [""]`, `defaults`, and
three six-section shots: exactly the shape the single-pass system prompt emits (full
file: `h3_agent_workflow_examples/plan.json`).
**7. Render the multi-clip video.** Paste the full contents of `plan.json` into the
Production Plan node's **internal `plan_json` widget** and queue **once** — that widget
is the one paste surface the pack declares `"dynamicPrompts": False`, so the frontend
dynamic-prompt serializer cannot strip the JSON's braces. Disconnect `plan_json_input`
first (a non-empty connected socket silently overrides the widget), select all, delete,
paste, queue. The node renders each shot as its own clip and chains the clips through
the H3 context loop — scene 2 continues from scene 1's final frames, scene 3 from scene
2's — producing ONE continuous multi-clip video: 685 delivered frames ≈ 28.5 s at 24 fps
(every scene after the first loses ~1 second to the fixed overlap trim that stitches the
clips together). The widget must hold exactly this one JSON object — first character
`{`, last character `}`, nothing before or after. If ComfyUI reports
`H3 Chain Plan JSON is invalid: Extra data: line 1 column 16 (char 15)`, the paste went
through a `dynamicPrompts: true` surface (prompt-typed widgets, third-party string nodes
with dynamic prompts) and the frontend ate the braces — the parsed head is just the
plan's quoted first key (`"prompt_prefix"` = 15 chars). Never paste a plan into a
prompt-typed node.
Want scene 2 different? Just tell the agent "rewrite scene 2, keep everything else": it
rewrites only `scene_02.json`, re-validates it, and re-merges — refreshing `plan.json`
in one command.
### Validating
```sh
.venv\Scripts\python.exe h3-plan-gate/verify_plan_rules.py # embedded pass/fail suite
.venv\Scripts\python.exe h3-plan-gate/verify_plan_rules.py plan plan.json # node-rule replica on a final plan
.venv\Scripts\python.exe h3-plan-gate/build_plan.py --check plan.json # same checks + frame/duration report
```
Smoke test with the shipped examples:
```sh
.venv\Scripts\python.exe h3-plan-gate/build_plan.py --check h3-plan-gate/h3_agent_workflow_examples/plan.json
# expect: PLAN OK with "total delivered: 685 frames = 28.54s"
```
### Wiring the result
Paste the full contents of `plan.json` into the Plan node's internal **`plan_json`**
widget — declared `"dynamicPrompts": False` in the pack source, the one surface where
ComfyUI's frontend dynamic-prompt serializer (which resolves `{a|b}` groups and strips
braces in `dynamicPrompts: true` widgets at queue time) cannot touch the JSON.
Disconnect `plan_json_input` first (a non-empty connected value silently overrides the
widget for that run), select all, delete, paste, and queue once — the node's context
loop turns the multi-shot plan into one multi-clip video by itself; no per-scene
renders, no editor stitching. The `plan_json_input` socket stays for programmatic
delivery (an LLM story director node producing the plan at execution time); a hand
paste through a string node is safe only if that node provably leaves braces alone —
probe it with `{"shots":[{"id":"probe","prompt":["test card"]}]}`:
`Extra data: line 1 column 8 (char 7)` means the node eats braces. Resolution,
context length, audio mode, and `base_seed` stay on the node's own widgets (the JSON
cannot set them; default canvas is 960×544 — set width/height widgets and wire the
width/height outputs into the conditioning node and sampler latent to avoid square
output).
### Requirements
- `build_plan.py` and `verify_plan_rules.py`: Python 3 stdlib only — no venv packages
needed; the `.venv\Scripts\python.exe` form just matches a typical Windows project
layout, and plain `python` / `python3` works anywhere.
- `count_prompt_tokens.py`: needs `tiktoken`.
- LLM settings: temperature 0.3–0.7; enable `response_format: {"type": "json_object"}` or
grammar-constrained JSON if the server supports it. Each scene turn is ≈5k tokens, so
ctx 16384 is comfortable; 128k runs the whole workflow many times over.