inpasu (インパス) — how a video becomes a script

The pipeline at a glance

video file
  └─ 1. sample      FFmpeg, 1–2 fps, normalize/trim letterbox
  └─ 2. detect      find dialogue box + name tag (OpenCV heuristics)
  └─ 3. stabilize   typewriter effect: emit only when box content is stable
  └─ 4. dedupe      perceptual hash of box region across frames
  └─ 5. ocr         manga-ocr per line; plausibility-score the output;
                    escalate low-confidence crops to Claude Vision (Batch API)
  └─ 6. assemble    merge multi-box speeches, attach speakers → script
  └─ 7. export      JSONL script + vocab extraction for ruura import

Stages 1–4 are fully local and free. Stage 5 is local in the common case (manga-ocr on the GPU); only a few percent of crops escalate to the cloud. On a 3.5-hour stream rip, 25,558 sampled frames collapse to 535 boxes → 271 dialogue speeches — the funnel below is doing real work at every step.

1 · Sample — video → frames

FFmpeg decodes the video and emits frames at 1–2 fps straight into NumPy over a pipe (nothing hits disk until a frame is interesting). Decode dominates wall time and scales with resolution, not sampling rate, so 2 fps is free relative to 1 fps — it is the recommended setting for real runs. Each sampled frame gets a timestamp derived from its output index.

2 · Detect — find the dialogue box

The counter-intuitive core insight: detect the white border, not the dark interior. Dark-pixel thresholding fails because the translucent box merges with dark scene content. Instead we threshold for white pixels (HSV V>190, S<70) and look for a connected component that is hollow: wide, in the lower half of the frame, but with a fill ratio below 0.15 — a rectangle outline, not a filled blob.

Three more tests turn candidates into a confirmed dialogue box:

• Dark interior — median V < 80 inside the rect. A bright interior is a floating speech bubble (idle chatter), a different frame type skipped in v1.
• Centering — dialogue boxes are horizontally centered exactly (measured offset 0.000 at both 1080p and 4K). Battle command menus anchor left (~0.23 off) and save/system prompts drift off-center via letterbox trim; the centering test (≤0.01 of width) rejects all of them. The blue line below marks frame center.
• Name tag split — the speaker plate's border touches the box border, so they are one connected component. We split it by per-row horizontal extent: a contiguous run of narrow rows at the top is the tag. No narrow rows ⇒ unnamed NPC, which is valid, not a failure.

Geometry constants — border colour thresholds, fill ceiling, tag width, centering tolerance — all live in profiles/dqxi.toml, never in code, so a new game is a new profile rather than a patch.

3 · Stabilize — wait for the typewriter

DQ reveals dialogue with a typewriter effect, so the same box appears across many sampled frames in different states of completion. The stabilizer hashes the box interior's ink mask (not its pixels — that makes the signature immune to the scene moving behind the translucent box) and emits content only when the same hash repeats on consecutive samples: the typewriter has settled and the player is reading.

Long speeches that overflow the box auto-scroll, so a naive "drop unconfirmed content on change" rule loses lines that scroll past between samples. The fix compares ink masks on every supersede: growth (old ink ⊆ dilated new ink) is a typewriter reveal and the partial is dropped; replacement (ink moved or vanished) was a complete page and is emitted, flagged stable:false. Nothing is lost; the cost is some overlap, deliberately absorbed downstream by the line cache (stage 5) and the scroll-fold (stage 7).

4 · Dedupe — perceptual hashing

Every emitted box's 63-bit DCT pHash (of the ink mask) goes into a per-job set; settled content matching an already-seen hash is dropped — repeated NPC speeches, detector flicker across camera cuts. The margins are wide open: the same content hashes 0–2 bits apart across samples (codec noise, background swirl), different content 16–32 bits apart, so the thresholds (stable_dist=4, dedupe_dist=6) sit comfortably in the empty middle. Lookups are scoped to a recent time window — matching across a whole 50-hour job once let unrelated lines collide by pHash coincidence (devlog 0014).

5 · OCR — the segment-stacking trick

This is the stage with the most hard-won detail. First, the box interior is binarized at V>170 to get an ink mask, with the border frame masked off.

Line banding + furigana stripping. A horizontal projection of the ink mask splits it into line bands. Bands shorter than 0.55× the tallest band are furigana (ruby readings) and are dropped — readings come from JMdict later, so furigana in the OCR output is just contamination. (This degrades below ~1080p where bands merge, so low-res input warns.)

Why per-line OCR was garbage, and the fix. manga-ocr's vision transformer resizes its input to a fixed square. A raw 1640×70 dialogue line is therefore squashed ~23:1 horizontally — glyphs collapse to ~6px wide and the model reads salad (兄貴と → 「実と). The fix exploits the full-width spaces (全角スペース) DQ puts at phrase boundaries: split the line at those wide gaps, stack the segments vertically into a roughly square block, invert, 2× upscale. That block matches the manga text-block distribution the model was trained on, and the same line reads near-perfectly.

Seen the way the model sees them — both forced to a 224×224 square — the problem is obvious:

The full set of stacked blocks actually fed to manga-ocr for this box — one per kept line:

manga-ocr returns, line for line:

「それはそうとダメよマノロ。
こんな危ないことしたら。
さっルキと一緒に村に戻ってなさい。

The first two lines are exact. The third reads さっルキ where the screen shows さっ␣ルキ — the full-width space between them is gone. That is by design: stacking concatenates the segments and drops the 全角スペース (devlog 0004), which is harmless when it falls at a real phrase boundary but here fuses さっ(さと) and ルキ into the non-word さっルキ. manga-ocr's glyphs are all correct; it's the segmentation that slipped — and that is exactly the kind of divergence the plausibility scorer in the next stage is built to flag (this line scores 0.93 against 1.0 for its neighbours). For contrast, the failure mode stacking fixes: on a similar box, naive per-line OCR returned 「「実と一緒にいたい．．． for a line whose truth was 「兄貴と一緒にいたい…… (devlog 0004).

The speaker tag takes the un-stacked path. Tags are short and read perfectly as a single inverted crop; stacking them turns plate-border remnants into bogus segments (マヤ → ーマヤー), so extract_lines(stack=False) for the tag.

… reads エマ.

Crops are cached by line-crop pHash within a time window, so the overlapping scroll pages from stage 3 cost one model call per physical line. On the 3.5-hour stream, 37% of crops never touched the model.

6 · Score — escalate the junk

Every OCR'd line is rated 0–1 by score.py: tokenize with fugashi (UniDic), then blend four signals — JMdict hit-rate over content words (35%), MeCab unknown-token rate (25%), grammar anomalies (25%), character-set sanity (15%). The per-box minimum line score is compared to the profile's confidence_floor=0.7; boxes below it are the Claude Vision escalation set.

The scorer is deliberately not a catch-all. It catches unreadable garble. It does not try to catch system messages (those are valid Japanese and score high — stage 7 drops them structurally) or clipped mid-scroll lines (real-morpheme salad that scores fine — stage 7's stable-twin fold removes them with no model call). Calibrated on a 1,397-line corpus, 75% of lines score exactly 1.0 and the escalation set is ~1.1% — all genuine junk.

Our worked example's box scores 0.93 (the さっルキ space-loss line): flagged as imperfect, but comfortably above the 0.7 floor, so it doesn't escalate — not every blemish is worth a cloud call, and a dropped space is for stage 7 to tidy, not Opus.

Escalation (stage 5b) re-reads just those crops with Claude (claude-opus-4-8) via the Batch API — async fits the pipeline, costs half the live price (~$1 for 295 boxes across the full 64-video corpus). Vision, not text-repair, because the failure mode is the OCR hallucinating plausible wrong kanji — only the pixels arbitrate. Corrections are rescored with the same scorer and logged with a full before/after audit trail; nothing is silently overwritten.

7 · Assemble — speeches & speakers

Stage 7 turns per-box OCR records into a script. It normalizes text (．．． → ……) and speaker tags (OCR plate slips like グレイグ・ → グレイグ), folds scroll pages (consecutive same-speaker boxes whose head/tail lines fuzzy-match merge, keeping the better-scored variant per physical line — this is where clipped mid-scroll garbage loses to the clean stable read), splits speeches at the opening 「 every DQ utterance carries, drops typewriter-fragment prefixes, and routes untagged-and-quoteless boxes to system.jsonl as system messages.

The example box becomes one script record:

{"t": 574.0, "speaker": "エマ", "text": "「それはそうとダメよマノロ。こんな危ないことしたら。さっルキと一緒に村に戻ってなさい。", "frame": "frames/t0000574.00.png", "ocr": "manga-ocr", "score": 0.93}

At corpus scale: the 44 raw cutscene detections become 8 boxes become 6 speeches that match the scene line for line; the full 64-video DQXI corpus yields 5,700 attributed speeches (~198k characters), player-name-replaced so it matches the voiced script, ready for the ruura SRS handoff.