295 lines
16 KiB
Markdown
Executable File
295 lines
16 KiB
Markdown
Executable File
# mmprompt
|
||
|
||
## h3-plan-gate — MiniMax H3 plan-JSON agent workflow + validator
|
||
|
||
Prompt set and tooling that turn a story idea into the `plan_json_input` JSON for the
|
||
**Production Plan** node (`MiniMax H3 Context Loop Plan` / `MiniMax H3 Plan (Modern)`) in
|
||
[ComfyUI-MiniMaxH3-Context-Loop](https://github.com/ethanfel/ComfyUI-MiniMaxH3-Context-Loop),
|
||
verified against commit `43afeff`. The deliverable is always **one plan JSON with one shot
|
||
per scene** — the node renders every shot as its own clip and chains the clips through the
|
||
H3 context loop into one continuous multi-clip video, so pasting one `plan.json` and
|
||
queuing once is all the rendering takes.
|
||
|
||
Instead of asking the LLM to emit that whole JSON in one turn, the gate is agent-shaped:
|
||
one outline turn, then **the LLM writes each scene one at a time** — a single turn that
|
||
exists purely to compose that scene from every element a good scene needs (subject,
|
||
style, continuity, blocking, camera, performance, texture, sound, exit state) — and the
|
||
scripts **put it all together at the end** into the same plan JSON the single-pass
|
||
prompt would have produced, validated before it ever reaches the node. The point is
|
||
compositional attention, not word counts: the validator checks structure and trips only
|
||
a thin-scene floor, so any scene can be rewritten until its elements are right, then
|
||
re-merged without touching the rest.
|
||
|
||
### Files
|
||
|
||
| File | Role |
|
||
|---|---|
|
||
| `SKILL.md` | Agent Skills entry point — this whole folder is a skill. Cline: copy it to `.cline/skills/h3-plan-gate/` (workspace) or `~/.cline/skills/` (global); any Agent-Skills-compatible agent (Claude Code, Cursor, OpenCode, Codex) can load it too |
|
||
| `h3_scene_agent_system_prompt.md` | Stage 1 + 3 system prompts: **PROMPT A** (scene-guide planner) and **PROMPT B** (one-scene-per-turn writer with a `_handoff` chain) — Stage 2 is the human approval between them |
|
||
| `h3_plan_assembler_system_prompt.md` | Stage 4 LLM merge prompt — only when you cannot run scripts |
|
||
| `h3_plan_json_system_prompt.md` | Single-pass variant (whole plan in one turn) — fastest for 2–4 scene stories |
|
||
| `build_plan.py` | Deterministic Stage 4 merger — verifies the continuity across the chain and puts all the scenes together at the end: strips `_handoff`, injects `"prompt_prefix": [""]` + defaults, validates everything before writing `plan.json` |
|
||
| `verify_plan_rules.py` | Node-replica validator (final plans *and* per-turn fragments): six-section Ref2VA prompt contract, the three subject-definition forms, refmod isolation, structure, handoff truth, continuity, thin-scene floor — no word band — with an embedded test suite |
|
||
| `count_prompt_tokens.py` | o200k token counter for the prompt fences (needs `pip install tiktoken`) |
|
||
| `h3_agent_workflow_examples/` | Worked 3-scene candy-store run in the six-section Ref2VA setup (refmod-pinned `<Subject 1>` + no-image shopkeeper): `outline.json` → `scene_01..03.json` → `plan.json` |
|
||
|
||
### Agent workflow (story in, plan.json out)
|
||
|
||
The loop the user runs: **story + scene count → scene guide → approval → one scene at a
|
||
time → continuity verified → plan.json**.
|
||
|
||
**Stage 1 — Scene guide.** Put PROMPT A (from `h3_scene_agent_system_prompt.md`) in the
|
||
system field, the story idea (and how many scenes) in the user field. Save the reply as
|
||
`outline.json`. It freezes scene count, ids, durations, beats, verbatim dialogue
|
||
distribution, and the `style_and_cast` block so later turns cannot drift.
|
||
|
||
**Stage 2 — Approval (human in the loop, never skipped).** Show the user the guide in
|
||
plain language — one line per scene plus cast and style — and wait. They approve it or
|
||
ask for changes; apply every change to `outline.json` and show the changed rows again.
|
||
Stage 3 starts only after an explicit approval — the guide is the contract the scenes
|
||
are composed against.
|
||
|
||
**Stage 3 — One scene per turn (~10 s scenes from the approved guide).** Put PROMPT B in
|
||
the system field and paste this turn template as the user message:
|
||
|
||
```text
|
||
STORY: <the full story idea>
|
||
STYLE AND CAST: <paste the "style_and_cast" object from the outline>
|
||
OUTLINE: <paste the "scenes" array from the outline>
|
||
WRITE SCENE: 2
|
||
PREVIOUS HANDOFF: <paste the previous scene's "_handoff" object — or: This is scene 1.>
|
||
```
|
||
|
||
Save each reply as `scene_NN.json` (zero-padded). Each fragment carries the native H3
|
||
six-section Ref2VA prompt (`subject_definitions:` → `summary: [reference generation]` →
|
||
`retention_analysis:` → `detailed_description:` → `overall_soundscape:` →
|
||
`non_diegetic_music: N/A`), the full scene composition, and a truthful `_handoff`
|
||
object for the next turn. Optional gate after each turn:
|
||
|
||
```sh
|
||
.venv\Scripts\python.exe h3-plan-gate/verify_plan_rules.py fragment scene_02.json --index 2 --prev scene_01.json
|
||
```
|
||
|
||
(All command blocks below use the Windows venv form; on macOS/Linux replace
|
||
`.venv\Scripts\python.exe` with `python3`.)
|
||
|
||
**Stage 4 — Continuity verified, assembly (deterministic, preferred):**
|
||
|
||
```sh
|
||
.venv\Scripts\python.exe h3-plan-gate/build_plan.py outline.json scene_01.json scene_02.json scene_03.json -o plan.json
|
||
```
|
||
|
||
The merge IS the continuity check: it validates every fragment in the `_handoff` chain
|
||
(each scene opens inside the previous scene's real end_state, soundscapes continue, ids,
|
||
durations, and dialogue match the approved guide), then validates the merged plan — only
|
||
clean JSON is written. Do **not** edit fragment prose by hand during assembly; if you
|
||
want a scene changed, re-run its Stage 3
|
||
turn and re-merge. Use the LLM assembler prompt only in a pure-chat setup with no script
|
||
execution, and diff its output against the fragments if you do (small models transcribe
|
||
long strings unreliably; `build_plan.py` is byte-exact by construction).
|
||
|
||
## Example: story idea → final output
|
||
|
||
The complete run of the shipped example (`h3_agent_workflow_examples/`) — the candy-store
|
||
reference run, with a refmod-pinned child subject and a no-image shopkeeper in the
|
||
six-section Ref2VA setup. The same files your agent should produce for any story.
|
||
|
||
**1. Give Cline the story idea.** With the skill installed, the whole request is:
|
||
|
||
```text
|
||
Use the h3-plan-gate skill. Story: a 9 year old boy called Sam — blonde hair in
|
||
pigtails, dresses like a baby girl — visits the candy store with his small red coin
|
||
purse: he browses the shelves, picks a red-and-white swirl lollipop, and pays the
|
||
shopkeeper. Three scenes. Use this character block for Sam exactly as given:
|
||
&identity: Sam is a 9 year old boy with his blonde hair in pigtails that dresses like
|
||
a baby girl; voice: Sam is a 9 year old boy with his blonde hair in pigtails that
|
||
dresses like a baby girl&
|
||
```
|
||
|
||
**2. The agent runs the smoke test** — proving the toolchain before writing anything:
|
||
|
||
```sh
|
||
.venv\Scripts\python.exe h3-plan-gate/verify_plan_rules.py
|
||
.venv\Scripts\python.exe h3-plan-gate/build_plan.py --check h3-plan-gate/h3_agent_workflow_examples/plan.json
|
||
```
|
||
|
||
Expect the suite to pass and, from the second command, `PLAN OK` with
|
||
`total delivered: 685 frames = 28.54s`.
|
||
|
||
**3. The agent writes `outline.json`** — the skeleton that freezes cast, style, ids,
|
||
durations, and dialogue so later scenes cannot drift. Sam is a user-refmod subject
|
||
(strip the `&` wrapper at the scene stage); the shopkeeper has no reference image, so
|
||
his definition carries physical traits only and the wardrobe moves to the scene bodies:
|
||
|
||
```json
|
||
{
|
||
"title": "candy_store",
|
||
"defaults": { "duration_seconds": 10, "steps": 5 },
|
||
"style_and_cast": {
|
||
"subjects": [
|
||
"<Subject 1> — Sam (user refmod, carry verbatim into subject_definitions): identity: Sam is a 9 year old boy with his blonde hair in pigtails that dresses like a baby girl; voice: Sam is a 9 year old boy with his blonde hair in pigtails that dresses like a baby girl",
|
||
"<Subject 2> — the shopkeeper: 55 year old man, short grey hair, brown eyes, heavy build; green apron over a rolled-sleeve white shirt, dark trousers (look woven into each [Shot 1] body)"
|
||
],
|
||
"style": "Warm photoreal candy-store realism, soft late-afternoon light pooling through the front windows, saturated jar-glass golds and wrapper reds under a gentle film grain, static eye-level camera held at the child's height.",
|
||
"soundscape_seed": "the low hum of a candy-store fridge case, cellophane wrapper crinkle, muffled street traffic behind the front glass"
|
||
},
|
||
"scenes": [
|
||
{ "index": 1, "id": "candy_browse", "duration_seconds": 10,
|
||
"beat": "Sam browses the candy shelves, taps the glass in front of a swirl lollipop, and turns toward the counter mid-step.",
|
||
"dialogue": [] },
|
||
{ "index": 2, "id": "lollipop_pickup", "duration_seconds": 10,
|
||
"beat": "Sam picks up the red-and-white swirl lollipop from the counter and his left hand finds the small red coin purse at his hip.",
|
||
"dialogue": [] },
|
||
{ "index": 3, "id": "pay_shopkeeper", "duration_seconds": 10,
|
||
"beat": "Sam pays the shopkeeper, coins into the springing register drawer, lollipop raised high.",
|
||
"dialogue": [] }
|
||
]
|
||
}
|
||
```
|
||
|
||
**4. The agent shows the guide — the user approves or changes it.** Before any scene is
|
||
written, the agent presents the guide in plain language and waits:
|
||
|
||
```text
|
||
The guide for "candy_store" — 3 scenes, warm photoreal candy-store realism:
|
||
1. candy_browse 10s — Sam browses the candy shelves, taps the glass in front of
|
||
a swirl lollipop, turns toward the counter mid-step. No dialogue.
|
||
2. lollipop_pickup 10s — Sam picks up the swirl lollipop and his left hand finds the
|
||
red coin purse at his hip. No dialogue.
|
||
3. pay_shopkeeper 10s — Sam pays the shopkeeper, coins into the springing register
|
||
drawer, lollipop raised high. No dialogue.
|
||
Cast: <Subject 1> Sam (user refmod, verbatim). <Subject 2> the shopkeeper (no image —
|
||
traits only, wardrobe woven into the scene bodies).
|
||
Approve, or tell me what to change.
|
||
```
|
||
|
||
```text
|
||
Approved — go.
|
||
```
|
||
|
||
Any change request ("make scene 2 slower", "he counts the coins first") is
|
||
applied to `outline.json` first and the changed rows shown again — scenes are composed
|
||
only from the approved guide.
|
||
|
||
**5. The agent composes each scene one at a time** — one file per turn, each turn
|
||
given over to making that one scene a complete composition, validating it before
|
||
moving to the next:
|
||
|
||
```sh
|
||
.venv\Scripts\python.exe h3-plan-gate/verify_plan_rules.py fragment scene_01.json --index 1 --total 3
|
||
.venv\Scripts\python.exe h3-plan-gate/verify_plan_rules.py fragment scene_02.json --index 2 --total 3 --prev scene_01.json
|
||
.venv\Scripts\python.exe h3-plan-gate/verify_plan_rules.py fragment scene_03.json --index 3 --total 3 --prev scene_02.json
|
||
```
|
||
|
||
Each fragment carries the six-section Ref2VA prompt and a truthful `_handoff`. Note the
|
||
refmod isolation in the body: `<Subject 1>` is referenced by the bare token — the refmod
|
||
already pins the look, and the workflow injects the reference blocks itself. Scene 2
|
||
(body elided — the real file carries the full blocking, camera work, texture, and
|
||
lighting detail):
|
||
|
||
```json
|
||
{
|
||
"id": "lollipop_pickup",
|
||
"duration_seconds": 10,
|
||
"prompt": [
|
||
"subject_definitions:",
|
||
"<Subject 1> identity: Sam is a 9 year old boy with his blonde hair in pigtails that dresses like a baby girl; voice: Sam is a 9 year old boy with his blonde hair in pigtails that dresses like a baby girl",
|
||
"<Subject 2> is a 55 year old man with short grey hair, brown eyes and a heavy build",
|
||
"summary:",
|
||
"[reference generation]",
|
||
"retention_analysis:",
|
||
"<Subject 1>: fully_preserved",
|
||
"<Subject 2>: fully_preserved",
|
||
"detailed_description:",
|
||
"Warm photoreal candy-store realism, soft late-afternoon light pooling through the front windows, saturated jar-glass golds and wrapper reds under a gentle film grain.",
|
||
"Static eye-level camera held at <Subject 1>'s eye height, shallow depth of field on the candy jars.",
|
||
"[Shot 1] Continuing the same shot without resetting the motion: <Subject 1> closes the last step to the counter and leans both hands on the polished wood. … End mid-motion: the lollipop held high in his right fist, his left hand finding the small red coin purse on its strap at his hip.",
|
||
"overall_soundscape:",
|
||
"the same low fridge-case hum and wrapper crinkle continue, the lollipop's cellophane crackling under small fingers, the till drawer rattling softly below the counter, street traffic still muffled behind the front glass.",
|
||
"non_diegetic_music:",
|
||
"N/A"
|
||
],
|
||
"_handoff": { "end_state": "…", "soundscape": "…", "dialogue_used": [] }
|
||
}
|
||
```
|
||
|
||
**6. The agent verifies the continuity and puts it all together at the end** — merging
|
||
every scene into one plan JSON, never writing plan JSON by hand:
|
||
|
||
```sh
|
||
.venv\Scripts\python.exe h3-plan-gate/build_plan.py outline.json scene_01.json scene_02.json scene_03.json -o plan.json
|
||
.venv\Scripts\python.exe h3-plan-gate/build_plan.py --check plan.json
|
||
```
|
||
|
||
Output:
|
||
|
||
```text
|
||
outline: outline.json (3 scenes)
|
||
fragment 1/3 … fragment 2/3 … fragment 3/3 …
|
||
Merged plan:
|
||
shot 1 id='candy_browse' raw=243 delivered=243 steps=5
|
||
shot 2 id='lollipop_pickup' raw=243 delivered=221 steps=5
|
||
shot 3 id='pay_shopkeeper' raw=243 delivered=221 steps=5
|
||
total delivered: 685 frames = 28.54s
|
||
shot 1 id='candy_browse' six-section ref2va OK (2 subject(s): no-image, refmod; body 138 words)
|
||
shot 2 id='lollipop_pickup' six-section ref2va OK (2 subject(s): no-image, refmod; body 156 words)
|
||
shot 3 id='pay_shopkeeper' six-section ref2va OK (2 subject(s): no-image, refmod; body 156 words)
|
||
H3 section structure: all shots carry the six Ref2VA sections in canonical order
|
||
wrote plan.json - paste its full contents into plan_json_input
|
||
```
|
||
|
||
The deliverable is the single `plan.json` — `"prompt_prefix": [""]`, `defaults`, and
|
||
three six-section shots: exactly the shape the single-pass system prompt emits (full
|
||
file: `h3_agent_workflow_examples/plan.json`).
|
||
|
||
**7. Render the multi-clip video.** Paste the full contents of `plan.json` into the
|
||
Production Plan node's forced STRING input `plan_json_input` and queue **once**. The node
|
||
renders each shot as its own clip and chains the clips through the H3 context loop —
|
||
scene 2 continues from scene 1's final frames, scene 3 from scene 2's — producing ONE
|
||
continuous multi-clip video: 685 delivered frames ≈ 28.5 s at 24 fps (every scene after
|
||
the first loses ~1 second to the fixed overlap trim that stitches the clips together).
|
||
The `plan_json_input` field must hold exactly this one JSON object — first character `{`,
|
||
last character `}`, nothing before or after. If ComfyUI reports
|
||
`H3 Chain Plan JSON is invalid: Extra data: …`, the field picked up extra text: clear it
|
||
completely and paste once.
|
||
|
||
Want scene 2 different? Just tell the agent "rewrite scene 2, keep everything else": it
|
||
rewrites only `scene_02.json`, re-validates it, and re-merges — refreshing `plan.json`
|
||
in one command.
|
||
|
||
### Validating
|
||
|
||
```sh
|
||
.venv\Scripts\python.exe h3-plan-gate/verify_plan_rules.py # embedded pass/fail suite
|
||
.venv\Scripts\python.exe h3-plan-gate/verify_plan_rules.py plan plan.json # node-rule replica on a final plan
|
||
.venv\Scripts\python.exe h3-plan-gate/build_plan.py --check plan.json # same checks + frame/duration report
|
||
```
|
||
|
||
Smoke test with the shipped examples:
|
||
|
||
```sh
|
||
.venv\Scripts\python.exe h3-plan-gate/build_plan.py --check h3-plan-gate/h3_agent_workflow_examples/plan.json
|
||
# expect: PLAN OK with "total delivered: 685 frames = 28.54s"
|
||
```
|
||
|
||
### Wiring the result
|
||
|
||
Paste the full contents of `plan.json` into the Plan node's forced STRING input
|
||
**`plan_json_input`** (a non-empty connected value overrides the visual editor's plan for
|
||
that run) and queue once — the node's context loop turns the multi-shot plan into one
|
||
multi-clip video by itself; no per-scene renders, no editor stitching. Resolution,
|
||
context length, audio mode, and `base_seed` stay on the node's own widgets (the JSON
|
||
cannot set them; default canvas is 960×544 — set width/height widgets and wire the
|
||
width/height outputs into the conditioning node and sampler latent to avoid square
|
||
output).
|
||
|
||
### Requirements
|
||
|
||
- `build_plan.py` and `verify_plan_rules.py`: Python 3 stdlib only — no venv packages
|
||
needed; the `.venv\Scripts\python.exe` form just matches a typical Windows project
|
||
layout, and plain `python` / `python3` works anywhere.
|
||
- `count_prompt_tokens.py`: needs `tiktoken`.
|
||
- LLM settings: temperature 0.3–0.7; enable `response_format: {"type": "json_object"}` or
|
||
grammar-constrained JSON if the server supports it. Each scene turn is ≈5k tokens, so
|
||
ctx 16384 is comfortable; 128k runs the whole workflow many times over.
|