337 lines
20 KiB
Markdown
Executable File
337 lines
20 KiB
Markdown
Executable File
# mmprompt
|
||
|
||
## h3-plan-gate — MiniMax H3 plan-JSON agent workflow + validator
|
||
|
||
Prompt set and tooling that turn a story idea into the `plan_json_input` JSON for the
|
||
**Production Plan** node (`MiniMax H3 Context Loop Plan` / `MiniMax H3 Plan (Modern)`) in
|
||
[ComfyUI-MiniMaxH3-Context-Loop](https://github.com/ethanfel/ComfyUI-MiniMaxH3-Context-Loop),
|
||
verified against commit `43afeff`. The deliverable is always **one plan JSON with one shot
|
||
per scene** — the node renders every shot as its own clip and chains the clips through the
|
||
H3 context loop into one continuous multi-clip video, so pasting one `plan.json` and
|
||
queuing once is all the rendering takes.
|
||
|
||
Instead of asking the LLM to emit that whole JSON in one turn, the gate is agent-shaped:
|
||
one outline turn, then **the LLM writes each scene one at a time** — a single turn that
|
||
exists purely to compose that scene from every element a good scene needs (subject,
|
||
style, continuity, blocking, camera, performance, texture, sound, exit state) — and the
|
||
scripts **put it all together at the end** into the same plan JSON the single-pass
|
||
prompt would have produced, validated before it ever reaches the node. The point is
|
||
compositional attention, not word counts: the validator checks structure and trips only
|
||
a thin-scene floor, so any scene can be rewritten until its elements are right, then
|
||
re-merged without touching the rest.
|
||
|
||
### Files
|
||
|
||
| File | Role |
|
||
|---|---|
|
||
| `SKILL.md` | Agent Skills entry point — this whole folder is a skill. Cline: copy it to `.cline/skills/h3-plan-gate/` (workspace) or `~/.cline/skills/` (global); any Agent-Skills-compatible agent (Claude Code, Cursor, OpenCode, Codex) can load it too |
|
||
| `h3_scene_agent_system_prompt.md` | Stage 1 + 3 system prompts: **PROMPT A** (scene-guide planner) and **PROMPT B** (one-scene-per-turn writer with a `_handoff` chain) — Stage 2 is the human approval between them |
|
||
| `h3_plan_assembler_system_prompt.md` | Stage 4 LLM merge prompt — only when you cannot run scripts |
|
||
| `h3_plan_json_system_prompt.md` | Single-pass variant (whole plan in one turn) — fastest for 2–4 scene stories |
|
||
| `build_plan.py` | Deterministic Stage 4 merger — verifies the continuity across the chain and puts all the scenes together at the end: strips `_handoff`, injects `"prompt_prefix": [""]` + defaults, validates everything before writing `plan.json` |
|
||
| `verify_plan_rules.py` | Node-replica validator (final plans *and* per-turn fragments): six-section Ref2VA prompt contract, the three subject-definition forms, refmod isolation, structure, handoff truth, continuity, thin-scene floor — no word band — with an embedded test suite |
|
||
| `count_prompt_tokens.py` | o200k token counter for the prompt fences (needs `pip install tiktoken`) |
|
||
| `h3_agent_workflow_examples/` | Worked 3-scene candy-store run in the six-section Ref2VA setup (refmod-pinned `<Subject 1>` + no-image shopkeeper): `outline.json` → `scene_01..03.json` → `plan.json` |
|
||
|
||
### Agent workflow (story in, plan.json out)
|
||
|
||
The loop the user runs: **story + scene count → scene guide → approval → one scene at a
|
||
time → continuity verified → plan.json**.
|
||
|
||
**Stage 1 — Scene guide.** Put PROMPT A (from `h3_scene_agent_system_prompt.md`) in the
|
||
system field, the story idea (and how many scenes) in the user field. Save the reply as
|
||
`outline.json`. It freezes scene count, ids, durations, beats, verbatim dialogue
|
||
distribution, and the `style_and_cast` block so later turns cannot drift.
|
||
|
||
**Stage 2 — Approval (human in the loop, never skipped).** Show the user the guide in
|
||
plain language — one line per scene plus cast and style — and wait. They approve it or
|
||
ask for changes; apply every change to `outline.json` and show the changed rows again.
|
||
Stage 3 starts only after an explicit approval — the guide is the contract the scenes
|
||
are composed against.
|
||
|
||
**Stage 3 — One scene per turn (~10 s scenes from the approved guide).** Put PROMPT B in
|
||
the system field and paste this turn template as the user message:
|
||
|
||
```text
|
||
STORY: <the full story idea>
|
||
STYLE AND CAST: <paste the "style_and_cast" object from the outline>
|
||
OUTLINE: <paste the "scenes" array from the outline>
|
||
WRITE SCENE: 2
|
||
PREVIOUS HANDOFF: <paste the previous scene's "_handoff" object — or: This is scene 1.>
|
||
```
|
||
|
||
Save each reply as `scene_NN.json` (zero-padded). Each fragment carries the native H3
|
||
six-section Ref2VA prompt (`subject_definitions:` → `summary: [reference generation]` →
|
||
`retention_analysis:` → `detailed_description:` → `overall_soundscape:` →
|
||
`non_diegetic_music: N/A`), the full scene composition, and a truthful `_handoff`
|
||
object for the next turn. Optional gate after each turn:
|
||
|
||
```sh
|
||
.venv\Scripts\python.exe h3-plan-gate/verify_plan_rules.py fragment scene_02.json --index 2 --prev scene_01.json
|
||
```
|
||
|
||
(All command blocks below use the Windows venv form; on macOS/Linux replace
|
||
`.venv\Scripts\python.exe` with `python3`.)
|
||
|
||
**Stage 4 — Continuity verified, assembly (deterministic, preferred):**
|
||
|
||
```sh
|
||
.venv\Scripts\python.exe h3-plan-gate/build_plan.py outline.json scene_01.json scene_02.json scene_03.json -o plan.json
|
||
```
|
||
|
||
The merge IS the continuity check: it validates every fragment in the `_handoff` chain
|
||
(each scene opens inside the previous scene's real end_state, soundscapes continue, ids,
|
||
durations, and dialogue match the approved guide), then validates the merged plan — only
|
||
clean JSON is written. Do **not** edit fragment prose by hand during assembly; if you
|
||
want a scene changed, re-run its Stage 3
|
||
turn and re-merge. Use the LLM assembler prompt only in a pure-chat setup with no script
|
||
execution, and diff its output against the fragments if you do (small models transcribe
|
||
long strings unreliably; `build_plan.py` is byte-exact by construction).
|
||
|
||
## Example: story idea → final output
|
||
|
||
The complete run of the shipped example (`h3_agent_workflow_examples/`) — the candy-store
|
||
reference run, with a refmod-pinned child subject and a no-image shopkeeper in the
|
||
six-section Ref2VA setup. The same files your agent should produce for any story.
|
||
|
||
**1. Give Cline the story idea.** With the skill installed, the whole request is:
|
||
|
||
```text
|
||
Use the h3-plan-gate skill. Story: a 9 year old boy called Sam — blonde hair in
|
||
pigtails, dresses like a baby girl — visits the candy store with his small red coin
|
||
purse: he browses the shelves, picks a red-and-white swirl lollipop, and pays the
|
||
shopkeeper. Three scenes. Use this character block for Sam exactly as given:
|
||
&identity: Sam is a 9 year old boy with his blonde hair in pigtails that dresses like
|
||
a baby girl; voice: Sam is a 9 year old boy with his blonde hair in pigtails that
|
||
dresses like a baby girl&
|
||
```
|
||
|
||
**2. The agent runs the smoke test** — proving the toolchain before writing anything:
|
||
|
||
```sh
|
||
.venv\Scripts\python.exe h3-plan-gate/verify_plan_rules.py
|
||
.venv\Scripts\python.exe h3-plan-gate/build_plan.py --check h3-plan-gate/h3_agent_workflow_examples/plan.json
|
||
```
|
||
|
||
Expect the suite to pass and, from the second command, `PLAN OK` with
|
||
`total delivered: 685 frames = 28.54s`.
|
||
|
||
**3. The agent writes `outline.json`** — the skeleton that freezes cast, style, ids,
|
||
durations, and dialogue so later scenes cannot drift. Sam is a user-refmod subject
|
||
(strip the `&` wrapper at the scene stage); the shopkeeper has no reference image, so
|
||
his definition carries physical traits only and the wardrobe moves to the scene bodies:
|
||
|
||
```json
|
||
{
|
||
"title": "candy_store",
|
||
"defaults": { "duration_seconds": 10, "steps": 5 },
|
||
"style_and_cast": {
|
||
"subjects": [
|
||
"<Subject 1> — Sam (user refmod, carry verbatim into subject_definitions): identity: Sam is a 9 year old boy with his blonde hair in pigtails that dresses like a baby girl; voice: Sam is a 9 year old boy with his blonde hair in pigtails that dresses like a baby girl",
|
||
"<Subject 2> — the shopkeeper: 55 year old man, short grey hair, brown eyes, heavy build; green apron over a rolled-sleeve white shirt, dark trousers (look woven into each [Shot 1] body)"
|
||
],
|
||
"style": "Warm photoreal candy-store realism, soft late-afternoon light pooling through the front windows, saturated jar-glass golds and wrapper reds under a gentle film grain, static eye-level camera held at the child's height.",
|
||
"soundscape_seed": "the low hum of a candy-store fridge case, cellophane wrapper crinkle, muffled street traffic behind the front glass"
|
||
},
|
||
"scenes": [
|
||
{ "index": 1, "id": "candy_browse", "duration_seconds": 10,
|
||
"beat": "Sam browses the candy shelves, taps the glass in front of a swirl lollipop, and turns toward the counter mid-step.",
|
||
"dialogue": [] },
|
||
{ "index": 2, "id": "lollipop_pickup", "duration_seconds": 10,
|
||
"beat": "Sam picks up the red-and-white swirl lollipop from the counter and his left hand finds the small red coin purse at his hip.",
|
||
"dialogue": [] },
|
||
{ "index": 3, "id": "pay_shopkeeper", "duration_seconds": 10,
|
||
"beat": "Sam pays the shopkeeper, coins into the springing register drawer, lollipop raised high.",
|
||
"dialogue": [] }
|
||
]
|
||
}
|
||
```
|
||
|
||
**4. The agent shows the guide — the user approves or changes it.** Before any scene is
|
||
written, the agent presents the guide in plain language and waits:
|
||
|
||
```text
|
||
The guide for "candy_store" — 3 scenes, warm photoreal candy-store realism:
|
||
1. candy_browse 10s — Sam browses the candy shelves, taps the glass in front of
|
||
a swirl lollipop, turns toward the counter mid-step. No dialogue.
|
||
2. lollipop_pickup 10s — Sam picks up the swirl lollipop and his left hand finds the
|
||
red coin purse at his hip. No dialogue.
|
||
3. pay_shopkeeper 10s — Sam pays the shopkeeper, coins into the springing register
|
||
drawer, lollipop raised high. No dialogue.
|
||
Cast: <Subject 1> Sam (user refmod, verbatim). <Subject 2> the shopkeeper (no image —
|
||
traits only, wardrobe woven into the scene bodies).
|
||
Approve, or tell me what to change.
|
||
```
|
||
|
||
```text
|
||
Approved — go.
|
||
```
|
||
|
||
Any change request ("make scene 2 slower", "he counts the coins first") is
|
||
applied to `outline.json` first and the changed rows shown again — scenes are composed
|
||
only from the approved guide.
|
||
|
||
**5. The agent composes each scene one at a time** — one file per turn, each turn
|
||
given over to making that one scene a complete composition, validating it before
|
||
moving to the next:
|
||
|
||
```sh
|
||
.venv\Scripts\python.exe h3-plan-gate/verify_plan_rules.py fragment scene_01.json --index 1 --total 3
|
||
.venv\Scripts\python.exe h3-plan-gate/verify_plan_rules.py fragment scene_02.json --index 2 --total 3 --prev scene_01.json
|
||
.venv\Scripts\python.exe h3-plan-gate/verify_plan_rules.py fragment scene_03.json --index 3 --total 3 --prev scene_02.json
|
||
```
|
||
|
||
Each fragment carries the six-section Ref2VA prompt and a truthful `_handoff`. Note the
|
||
refmod isolation in the body: `<Subject 1>` is referenced by the bare token — the refmod
|
||
already pins the look, and the workflow injects the reference blocks itself. Scene 2
|
||
(body elided — the real file carries the full blocking, camera work, texture, and
|
||
lighting detail):
|
||
|
||
```json
|
||
{
|
||
"id": "lollipop_pickup",
|
||
"duration_seconds": 10,
|
||
"prompt": [
|
||
"subject_definitions:",
|
||
"<Subject 1> identity: Sam is a 9 year old boy with his blonde hair in pigtails that dresses like a baby girl; voice: Sam is a 9 year old boy with his blonde hair in pigtails that dresses like a baby girl",
|
||
"<Subject 2> is a 55 year old man with short grey hair, brown eyes and a heavy build",
|
||
"summary:",
|
||
"[reference generation]",
|
||
"retention_analysis:",
|
||
"<Subject 1>: fully_preserved",
|
||
"<Subject 2>: fully_preserved",
|
||
"detailed_description:",
|
||
"Warm photoreal candy-store realism, soft late-afternoon light pooling through the front windows, saturated jar-glass golds and wrapper reds under a gentle film grain.",
|
||
"Static eye-level camera held at <Subject 1>'s eye height, shallow depth of field on the candy jars.",
|
||
"[Shot 1] Continuing the same shot without resetting the motion: <Subject 1> closes the last step to the counter and leans both hands on the polished wood. … End mid-motion: the lollipop held high in his right fist, his left hand finding the small red coin purse on its strap at his hip.",
|
||
"overall_soundscape:",
|
||
"the same low fridge-case hum and wrapper crinkle continue, the lollipop's cellophane crackling under small fingers, the till drawer rattling softly below the counter, street traffic still muffled behind the front glass.",
|
||
"non_diegetic_music:",
|
||
"N/A"
|
||
],
|
||
"_handoff": { "end_state": "…", "soundscape": "…", "dialogue_used": [] }
|
||
}
|
||
```
|
||
|
||
**6. The agent verifies the continuity and puts it all together at the end** — merging
|
||
every scene into one plan JSON, never writing plan JSON by hand:
|
||
|
||
```sh
|
||
.venv\Scripts\python.exe h3-plan-gate/build_plan.py outline.json scene_01.json scene_02.json scene_03.json -o plan.json
|
||
.venv\Scripts\python.exe h3-plan-gate/build_plan.py --check plan.json
|
||
```
|
||
|
||
Output:
|
||
|
||
```text
|
||
outline: outline.json (3 scenes)
|
||
fragment 1/3 … fragment 2/3 … fragment 3/3 …
|
||
Merged plan:
|
||
shot 1 id='candy_browse' raw=243 delivered=243 steps=5
|
||
shot 2 id='lollipop_pickup' raw=243 delivered=221 steps=5
|
||
shot 3 id='pay_shopkeeper' raw=243 delivered=221 steps=5
|
||
total delivered: 685 frames = 28.54s
|
||
shot 1 id='candy_browse' six-section ref2va OK (2 subject(s): no-image, refmod; body 138 words)
|
||
shot 2 id='lollipop_pickup' six-section ref2va OK (2 subject(s): no-image, refmod; body 156 words)
|
||
shot 3 id='pay_shopkeeper' six-section ref2va OK (2 subject(s): no-image, refmod; body 156 words)
|
||
H3 section structure: all shots carry the six Ref2VA sections in canonical order
|
||
wrote plan.json - paste its full contents into the Plan node's internal plan_json widget and queue once (never through a multiline or prompt-typed text input; see SKILL.md Step 5)
|
||
```
|
||
|
||
The deliverable is the single `plan.json` — `"prompt_prefix": [""]`, `defaults`, and
|
||
three six-section shots: exactly the shape the single-pass system prompt emits (full
|
||
file: `h3_agent_workflow_examples/plan.json`).
|
||
|
||
**7. Render the multi-clip video.** Paste the full contents of `plan.json` into the
|
||
Production Plan node's **internal `plan_json` widget** and queue **once** — that widget
|
||
is the one paste surface the pack declares `"dynamicPrompts": False`, so the frontend
|
||
dynamic-prompt serializer cannot strip the JSON's braces. Disconnect `plan_json_input`
|
||
first (a non-empty connected socket silently overrides the widget), select all, delete,
|
||
paste, queue. The node renders each shot as its own clip and chains the clips through
|
||
the H3 context loop — scene 2 continues from scene 1's final frames, scene 3 from scene
|
||
2's — producing ONE continuous multi-clip video: 685 delivered frames ≈ 28.5 s at 24 fps
|
||
(every scene after the first loses ~1 second to the fixed overlap trim that stitches the
|
||
clips together). The widget must hold exactly this one JSON object — first character
|
||
`{`, last character `}`, nothing before or after. If ComfyUI reports
|
||
`H3 Chain Plan JSON is invalid: Extra data: line 1 column 16 (char 15)`, the paste went
|
||
through a `dynamicPrompts: true` surface (prompt-typed widgets, third-party string nodes
|
||
with dynamic prompts) and the frontend ate the braces — the parsed head is just the
|
||
plan's quoted first key (`"prompt_prefix"` = 15 chars). Never paste a plan into a
|
||
prompt-typed node.
|
||
|
||
Want scene 2 different? Just tell the agent "rewrite scene 2, keep everything else": it
|
||
rewrites only `scene_02.json`, re-validates it, and re-merges — refreshing `plan.json`
|
||
in one command.
|
||
|
||
### Validating
|
||
|
||
```sh
|
||
.venv\Scripts\python.exe h3-plan-gate/verify_plan_rules.py # embedded pass/fail suite
|
||
.venv\Scripts\python.exe h3-plan-gate/verify_plan_rules.py plan plan.json # node-rule replica on a final plan
|
||
.venv\Scripts\python.exe h3-plan-gate/build_plan.py --check plan.json # same checks + frame/duration report
|
||
```
|
||
|
||
Smoke test with the shipped examples:
|
||
|
||
```sh
|
||
.venv\Scripts\python.exe h3-plan-gate/build_plan.py --check h3-plan-gate/h3_agent_workflow_examples/plan.json
|
||
# expect: PLAN OK with "total delivered: 685 frames = 28.54s"
|
||
```
|
||
|
||
### Wiring the result
|
||
|
||
Paste the full contents of `plan.json` into the Plan node's internal **`plan_json`**
|
||
widget — declared `"dynamicPrompts": False` in the pack source, the one surface where
|
||
ComfyUI's frontend dynamic-prompt serializer (which resolves `{a|b}` groups and strips
|
||
braces in `dynamicPrompts: true` widgets at queue time) cannot touch the JSON.
|
||
Disconnect `plan_json_input` first (a non-empty connected value silently overrides the
|
||
widget for that run), select all, delete, paste, and queue once — the node's context
|
||
loop turns the multi-shot plan into one multi-clip video by itself; no per-scene
|
||
renders, no editor stitching. The `plan_json_input` socket stays for programmatic
|
||
delivery (an LLM story director node producing the plan at execution time); a hand
|
||
paste through a string node is safe only if that node provably leaves braces alone —
|
||
probe it with `{"shots":[{"id":"probe","prompt":["test card"]}]}`:
|
||
`Extra data: line 1 column 8 (char 7)` means the node eats braces. Resolution,
|
||
context length, audio mode, and `base_seed` stay on the node's own widgets (the JSON
|
||
cannot set them; default canvas is 960×544 — set width/height widgets and wire the
|
||
width/height outputs into the conditioning node and sampler latent to avoid square
|
||
output).
|
||
|
||
### Requirements
|
||
|
||
- **Local models through Cline (Windows 11 + llama.cpp):** Cline's file reads are capped by
|
||
hard-coded budgets that do not scale with the model's context window — about 48,000
|
||
characters and 2,000 lines per read, and any single line over 2,000 characters is cut
|
||
with a `[line truncated]` marker (command output and editor writes have similar caps).
|
||
Raising `llama-server -c` does not lift them: a 131072-token context runs this workflow
|
||
comfortably and still sees Cline's truncation notices. `-c` only needs to fit a whole
|
||
task (32768+ is plenty) and prevent the separate "request exceeds context" error.
|
||
SKILL.md works within the caps: the scripts are run, never read (`verify_plan_rules.py`
|
||
is ~62 KB and exceeds the read cap by design), the big prompt files are read in bounded
|
||
line ranges cited in Steps 1, 3, and the short-stories section, and one-line plan files
|
||
are copied or submitted with commands, never read back.
|
||
|
||
- `build_plan.py` and `verify_plan_rules.py`: Python 3 stdlib only — no venv packages
|
||
needed; the `.venv\Scripts\python.exe` form just matches a typical Windows project
|
||
layout, and plain `python` / `python3` works anywhere.
|
||
- `count_prompt_tokens.py`: needs `tiktoken`.
|
||
- LLM settings: temperature 0.3–0.7; enable `response_format: {"type": "json_object"}` or
|
||
grammar-constrained JSON if the server supports it. Each scene turn is ≈5k tokens, so
|
||
ctx 16384 is comfortable; 128k runs the whole workflow many times over.
|
||
|
||
### Running the skill inside Cline (VS Code + llama.cpp)
|
||
|
||
1. Clone this repo and open the **repo root** as the workspace — Cline loads `.clinerules/`
|
||
and `.cline/skills/` from the workspace root only.
|
||
2. Provider: **OpenAI Compatible**, Base URL `http://localhost:8080/v1` (your
|
||
`llama-server`), any Model ID, context 32768 or more (`-c 131072` is comfortable).
|
||
3. Ask for a plan normally (`.clinerules` routes the agent to `h3-plan-gate/SKILL.md`), or
|
||
type `/h3-plan` with the story and scene count — the workflow in
|
||
`.clinerules/workflows/h3-plan.md` runs the skill end to end, including the
|
||
outline-approval pause.
|
||
4. `.cline/skills/h3-plan-gate/` is a stub that registers the skill with Cline's skill
|
||
system; the real procedure stays in `h3-plan-gate/SKILL.md` (single source of truth).
|
||
5. Approvals: enable auto-approve for reading files and executing commands (local model,
|
||
nothing to spend), or approve each call — the skill pauses by itself at the
|
||
outline-approval step either way.
|