Parsing PDFs is still the hardest part

People assume the hard part of an answer engine is the retrieval, or the model, or the ranking. In our issue tracker, 46% of open bugs labelled wrong-answer resolve to a parsing defect. The retrieval worked. The model was fine. The text it was handed was already wrong two stages earlier.

The reason is structural. A PDF does not contain paragraphs, tables, headings or reading order. It contains instructions for placing glyphs at coordinates on a page, plus optional hints nobody is obliged to emit correctly. Everything above that (this run of glyphs is a word, these words are a sentence, this sentence belongs to the left column and not the right one, these numbers are a table) is inference we perform, and every inference has a failure rate.

What actually goes wrong

We keep a taxonomy because it stops arguments. Every parse bug gets a class, and every class gets a regression case. The percentages are the share of documents in our three donated corpora that trip each class at least once.

Note which one is biggest. Header and footer bleed is not glamorous and it is not hard, and it caused more bad retrieval than table loss and reading order combined, because it poisons every chunk, where table loss hits a few. Boring bugs win on volume.

Reading order is a guess

A minority of PDFs carry structure tags describing logical order. Most carry nothing, and some carry tags worse than nothing, because a generator emitted them once and a later editing tool did not maintain them. So we reconstruct order geometrically: glyphs into lines, lines into blocks, blocks into columns, columns into a page sequence.

That holds until a document does something entirely reasonable that breaks the assumption: a pull quote spanning two columns, a sidebar that is visually adjacent but logically an aside, a table continuing across a page break with its header repeated. So we do not treat order as certain. Each block carries a confidence, and blocks under threshold are indexed but flagged, which means an answer depending on them arrives with a parse warning attached to the citation instead of silently.

A parser that never says "I am unsure about this page" is not more accurate, only less honest about the same accuracy.

Three extractors and a vote

We run more than one extraction path over every document and compare. Character-level agreement between paths is a cheap and surprisingly well calibrated proxy for extraction quality, far better calibrated than any single extractor's own confidence, usually a fiction.

• Text layer extraction where the PDF has a usable one. Fast, exact when it works.
• Layout-aware extraction that rebuilds blocks and tables geometrically. Slower, better on anything with structure.
• OCR for image-only pages, and for pages where the first two disagree badly enough to suggest the text layer is lying. That happens, especially with documents scanned and then given a bad text layer by an office scanner.

When paths agree, we index and move on. When they disagree on prose, we take the layout path. When they disagree on a number, we pick no winner: the span is marked unreliable, and a question whose answer depends on it gets a refusal in place of a coin flip. Why that trade is worth making is the whole of the refusal post.

The heuristics we are not proud of

Some of this is unprincipled and we will not pretend otherwise. It is in the repository, it is commented, and it beats the principled version we tried first.

• Repeated-line suppression. A line appearing in the same vertical band on more than 60% of pages is a running header or footer and gets stripped. It occasionally eats a legitimate repeated heading, so it is disabled under four pages.
• Hyphen rejoining with a dictionary check. Join a trailing hyphen to the next line only if the joined form is a known word, or if neither half is. Wrong for compounds in some languages, hence per-language configuration.
• Digit-run protection. No normalisation step may touch a run of digits, separators and a currency symbol. Smart quotes, dash normalisation, whitespace collapsing: all of them skip numeric spans, because every one has silently altered an amount at least once.
• Page-level bail-out. A page producing almost no characters but containing a large image is a scan, not an empty page. Obvious. Was missing for a year, and every answer it lost looked like a retrieval bug.

Provenance is the part that pays off

Every extracted span keeps the page and bounding box it came from. It costs storage, it complicates every intermediate structure, and it is the single change that did most for user trust, because a citation can highlight the exact region of the exact page.

It improved debugging more than we expected too. A user reporting "this number is wrong" now sends a citation that renders the source region. About a third of those turn out to be the parser reading the right region of the wrong table. Without coordinates that report is unactionable, and the bug lives forever.

What we would do first

If you are building this and want the most quality per hour spent: kill header and footer contamination, keep coordinates, stop normalising numbers. None of it is research. All three moved our attribution score more than any model change we made that year.

The extraction pipeline is in github.com/getquire/quire under quire/ingest/pdf. The regression generators live in refusal-bench; the documents themselves are not ours to publish.

---