OCR does not fail loudly, and that is the problem
A recogniser that returns garbage is easy to handle; a recogniser that returns a plausible wrong digit at 94% confidence is how a scanned invoice quietly becomes a different invoice, and how the difference reaches an approval queue without anybody noticing.
11 August 2026
Roughly 11% of the documents our users index have no text layer at all, and another 6% carry one that is worse than nothing, which means that for close to a fifth of a real corpus recognition is not a preprocessing step but the entire pipeline: whatever OCR produces is the document, and every later stage inherits its mistakes without any way of seeing that they are mistakes.
Modern recognisers are good, with character error rates on clean office scans sitting under half a percent, and that number sounds like a solved problem for exactly as long as it takes to notice two things about it: the errors are not uniformly distributed across the fields you care about, and the confidence scores attached to them are not calibrated against anything.
Prose is protected, numbers are not
An OCR error inside a sentence is usually survivable, because context does the repair for you: a reader, a retriever and a language model all reconstruct "recieve" or "1nvoice" without effort, since the surrounding words constrain what the token could plausibly have been.
Digits have no such context, and in an amount field any digit is exactly as plausible as any other, so nothing in the string suggests that a 7 is unlikely in that position. The same recogniser, working at the same quality on the same page, therefore carries a benign error rate through the prose and a dangerous one through every number on it.
In prose an OCR error is a typo. In an amount field it is a different invoice, and nothing downstream will notice.
Our measurements on the scanned subset of the benchmark corpus, per field type:
| Field type | Character error rate | Field-level error rate | Recoverable by context |
|---|---|---|---|
| Body prose | 0.4% | n/a | Almost always |
| Vendor name | 0.6% | 2.1% | Usually, against a known vendor list |
| Dates | 0.9% | 3.4% | Partly, format constraints help |
| Invoice number | 1.3% | 6.8% | Rarely, they are arbitrary strings |
| Amounts | 1.1% | 5.2% | Never, without arithmetic |
| Amounts, low-quality scan | 3.7% | 16.9% | Never, without arithmetic |
A 5.2% field-level error rate on amounts means that one invoice in twenty carries a wrong number, presented with no visible sign that anything went wrong anywhere in the chain, and that is the figure which ought to decide your architecture rather than the character error rate everybody quotes.
Confidence scores are a genre of fiction
Every recogniser will tell you how sure it is. We plotted reported confidence against measured accuracy across 300,000 recognised numeric fields, and while the relationship is real it is weak enough to be useless exactly where it matters: inside the 90-to-99 band, where most output lands, reported confidence carries almost no information about whether the field is actually right.
The systematic confusions are the familiar ones, 0 and 8, 1 and 7, 3 and 8, 5 and 6, along with the European handwritten 1 whose serif runs long enough to pass for a 7, and in every one of those cases the recogniser is confident for a defensible reason, since the glyph genuinely does look like the thing it decided on.
What we do instead
- Two engines must agree on any digit run. Character-level agreement between independent recognisers is a far better calibrated signal than either one's confidence. Disagreement on a numeric span marks it unreadable.
- Arithmetic is the third vote. On invoices the reconciliation check catches most single-digit errors, because a wrong digit almost never leaves the totals consistent. This is covered in the line-item post, and it is the strongest signal we have on any scanned financial document.
- Unreadable means unreadable. A span that fails both checks is indexed as an image region with no text value. A question that depends on it gets "the scan is not legible here", plus the cropped region so a person can read it themselves in two seconds.
- Preprocessing is per-page, not per-document. Deskew, denoise and binarisation thresholds get chosen per page from the page itself. Documents are frequently a mix of a clean print and a fax of a fax.
- Never a language model as a repairer. Asking a model to clean up OCR output produces text that reads beautifully and has quietly invented plausible numbers. We tried it. It improved every readability metric and made the system materially less trustworthy.
The cost, stated honestly
Running two recognisers roughly doubles ingestion CPU time on the scanned subset, which on a 40,000-document corpus with 11% scans works out to about 25 minutes of extra work on a laptop-class machine, once, at index time. Nobody who has watched a wrong amount get quoted back to them with full confidence has ever argued with us about the 25 minutes.
Refusing on illegible spans also costs coverage, in that about 1.4% of questions against heavily scanned corpora come back "not legible" instead of an answer, and we consider the correct value for that number to be greater than zero. A system that answers every question about a smudged fax is not better at reading than one that declines; it is better at guessing.
Engine selection is configurable in quire.toml; the default pair is chosen for licence compatibility ahead of benchmark scores, and both run fully offline.
Parsing PDFs is still the hardest part
Why reading order is a guess, and the failure taxonomy we keep to stop arguments.
The Quire Note
One email a month, engineering-first.