How the extraction layer decides which model runs, what it checks before anything reaches the graph, and what that costs per document. Half of what follows is a reversal — the architecture here is what survived being measured.
The API is the daily path. The manual path is the insurance.
Mode A: extraction over the LLM API, unattended, on a timer.
Mode B: a docpack is built, pasted into a chat model by hand, and the result imported.
Mode B is slower and it is not a fallback that was ever removed. It means the pipeline is not hostage to one vendor, one key, or one month's budget.
Three stages, three models, resolved in a fixed order.
Extraction runs on Sonnet 5. Disambiguation runs on a nano-class model. Narratives run on Haiku 4.5.
Resolution order: the domain's own profile, then the pipeline-wide default, then the framework floor. A domain that names no model inherits one, so naming a model for one domain cannot move another.
No live domain pins a model today. The mechanism exists so that when one needs to, it does not become a global change.
The Sonnet 5 move was made for intro pricing that was expected to expire. The scheduled rise was cancelled instead, so the upgrade is permanently cheaper than the model it replaced rather than merely no more expensive.
Cheap-first, escalate on failure. It cost more than not doing it.
The design: run a cheap model first, escalate to the specialist only when a quality gate fails. It pays off if a meaningful share of documents stay cheap. Break-even needed escalation under 50%.
First measurement: 80% escalated. Five weeks of prompt tuning and about 250 extractions later: 69%. The gap to break-even was never closed.
The analysis said it plainly — running the specialist on everything would actually be cheaper.
Replaced by a single model submitted through the batch API, which is roughly 29% cheaper than the mix it replaced. The downstream stages stay synchronous; batching about sixty short calls would have saved pennies and added a second wait cycle.
The mechanism was wrong for the corpus, not undertrained. Remaining failures were empty responses and truncated JSON, and no prompt fixes those.
A cheap model's failure modes do not stay in their lane. Nano returned a blank string on some documents; the fallback on escalation failure was the cheap result, which was that same blank string, so the document was skipped in silence rather than flagged. Fifteen of twenty escalation failures were that. Four more were JSON truncated mid-output on long documents.
The gate was generating false positives. It was also doing real work.
The high-confidence gate flags strong claims carrying weak evidence. On the film corpus it fired often, because trade-press paraphrase reads like a confident assertion, so it was switched off for that domain.
With it off, a relation entered the graph asserting that the film 2001: A Space Odyssey wrote Stanley Kubrick. Confidence 0.92.
Inverted subject and object, high confidence, no gate. The gate was not miscalibrated so much as pointed at output the cheap tier was producing; disabling it treated the alarm instead of the fire.
All four are CPU checks. None of them asks a model whether a model was right.
Evidence fidelity: the quoted snippet must actually appear in the source text. This is the anti-hallucination gate, and it is the one that matters most — fabricated evidence poisons the graph silently, which is worse than a parse error because nothing reports it.
Orphan endpoints: both ends of a relation must resolve to entities the extraction actually declared.
Zero value: a document of real length must yield at least one entity, or something went wrong upstream.
High confidence, bad evidence: the stronger the claim, the stronger its evidence has to be.
A cheaper model re-runs the same jobs. Its output is scored and thrown away.
Shadow mode runs a candidate model silently alongside the primary. Nothing it produces is served. The point is a record of how it would have done on real documents rather than on a benchmark.
Promotion is gated on measured thresholds, not judgement: schema compliance at 95%, entity agreement at 85%, relation agreement at 80%, and a floor of 100 documents before the numbers count at all.
The document floor is the part that matters. It is what stops a promotion decision being made on a good week.
This is the mechanism that would have caught the cheap tier early, had it existed then.
Five weeks, three domains, about $234 against a plan of $35 to $45.
Film $153.25, semiconductors $79.42, weapons detection $1.89. Film is 65% of extraction spend.
The plan's per-document basis was an order of magnitude low: it assumed about $0.012, and film measures $0.1063, semiconductors $0.0849.
79% of film's cost is output tokens. Film's documents are the smaller ones — about 7,000 input tokens per call against 12,600 — so the premium is the size of the extraction payload, not the size of the article.
Every figure here is reconstructed from recorded token usage rather than estimated. The gap between plan and reality is the reason the reconstruction exists.
The obvious lever is cutting film's documents. It is the wrong lever.
Film costs the most and is the domain under evaluation. Cutting its document budget is the first thing anyone would reach for.
The lens film exists to prove is built from mention counts over a rolling seven-day window, capped. Halving the documents roughly halves the mention counts, pushes entities under the cap, and moves the rankings.
So the volume is the measurement. Cutting it would evaluate a degraded corpus and then generalise the answer to the full one — which is the exact error the evaluation was set up to avoid.
Holding costs roughly 1.4 times the baseline until a separate change lands. That is accepted on purpose, and written down as accepted, which is the difference between a cost and a leak.
The proof-point that budget protects had itself stopped updating. The scoring stage looped one query per entity, nine of them, across 34,043 film entities — roughly 300,000 round trips — and crossed a 600-second timeout as the graph grew. The job that reads the results re-served the last good snapshot and exited zero, so every check stayed green while the output aged 38 days. Semiconductors hit the same wall a month later. The fix was set-based scoring: eight queries total, whatever the entity count. The old per-entity version is deliberately kept as the reference a test checks the new one against.