mmprompt/h3-plan-gate/count_prompt_tokens.py
Z User 83cc5145bf Realign: one plan JSON, written one scene at a time
The deliverable is the single merged plan.json — the same shape the
single-pass system prompt emits — and the multi-clip video is what the
node itself produces from it: each shot renders as its own clip, chained
through the H3 context loop into one continuous video. Removed the
--clips per-scene export (external stitching is NOT the design).

Workflow restated everywhere: outline turn -> the LLM writes each scene
one at a time (one focused turn per scene) -> build_plan.py puts it all
together at the end into plan.json -> paste once, queue once.

build_plan.py: --clips option and export block removed; docstring now
describes the node-side chaining. README: intro/example/validating/
wiring rewritten around the single deliverable. SKILL.md: goal, Step 3
(Put it all together at the end), Step 4 (the node makes the multi-clip
video), and the revising loop updated. Suite + --check green.
2026-10-29 19:00:11 +00:00

29 lines
891 B
Python
Executable File

"""Count o200k tokens (proxy for Llama/Qwen tokenizers) of every ```text fence
in the given markdown files; .json args are counted as-is.
Usage: python count_prompt_tokens.py FILE.md [FILE.md ...] [FILE.json ...]
"""
import re
import sys
import tiktoken
enc = tiktoken.get_encoding("o200k_base")
for path in sys.argv[1:]:
with open(path, encoding="utf-8") as fh:
text = fh.read()
print(path)
if path.endswith(".json"):
n = len(enc.encode(text))
print(f" whole file: {n:5d} tok ({len(text)} chars)")
continue
fences = re.findall(r"```text\n(.*?)\n```", text, re.S)
total = 0
for i, fence in enumerate(fences, 1):
t = len(enc.encode(fence))
total += t
first = fence.lstrip().splitlines()[0][:60]
print(f" fence {i}: {t:5d} tok ({len(fence):6d} chars) | {first}")
print(f" total: {total} tok")