Chunking is a layout problem, not a token problem

The standard recipe is 512 tokens with 50 tokens of overlap. It is easy to implement, it is easy to reason about, and on continuous prose it is fine. Run it over a policy document with numbered clauses, or a contract, or anything containing a table, and it does two damaging things: it cuts structures in half, and it strips the context that made a fragment interpretable.

A window boundary landing in the middle of a table produces two chunks, neither of which contains both the header row and the number. The chunk with the number is retrievable and meaningless. The chunk with the header is meaningful and has no number. Overlap does not fix it, it just moves the boundary.

Cut where the document already cuts

Documents come with boundaries. Headings, clause numbers, list items, table row groups, page breaks in some formats, sections in others. Those boundaries were placed by the author to mark where one idea ends and another begins. That is exactly the judgement a chunker is trying to make and consistently worse at.

So the rule is: cut at the strongest structural boundary within a size band, never in the middle of a table or a numbered clause, and treat the token count as a constraint on the cut, never the thing being optimised. Chunks come out uneven, some 90 tokens, some 800, and that unevenness is the point. A three-line clause is a complete unit of meaning and padding it to 512 tokens with its neighbours makes it less retrievable, not more.

Carry the path down

Every chunk gets a prefix built from its position in the document: the title, the heading chain above it, the clause number, and for tables the column header path described in the tables post. The prefix is indexed and shown, not hidden metadata.

This is unglamorous and it is the single highest-value thing in this post. A fragment reading "must be submitted within 30 days" matches almost nothing useful. The same fragment prefixed with "Supplier Agreement 2026 > 7. Invoicing > 7.3 Late submission" matches the way people ask, and it makes the retrieved snippet interpretable on its own, which matters more, because the person reading the answer has to judge whether the citation supports the claim.

A chunk that a person cannot evaluate without opening the document is not a citation, it is a promise.

Numbers

Same corpora, same embeddings, same retriever, same evaluation set. Only the chunker changed.

The table column is where the difference is dramatic, which makes sense. That is the failure the fixed window causes most directly. The heading path is worth seven points of overall accuracy for what is, in implementation terms, string concatenation.

What it costs

Honest list, because layout chunking is not free:

• It depends on parse quality. If the structure extraction is wrong, the chunker inherits the error. On a badly parsed document, fixed windows degrade more gracefully. This is the real trade, and it is why parse confidence gates which chunker runs.
• Chunk sizes are uneven, which complicates batching and makes throughput less predictable.
• Very long sections still need splitting and the fallback inside them is, unavoidably, a windowed split. There is no boundary to use, so we use the boring thing.
• Prefixes cost tokens in both indexing and context. About 8% overhead on our corpora, and the cheapest eight percent we spend.

The thing we removed

Overlap. Once chunks align with structure, overlap buys nothing measurable and costs index size and duplicate retrieval hits: the same passage arriving twice in the context under two chunk identities, which makes an answer look better supported than it is. Removing it shrank the index by 11% and improved deduplication of citations. It was in there because everyone else has it. Bad reason. It survives a long time.

Chunking is in quire/index/chunk and the strategy is selectable per source. If you index code or logs, the layout chunker is the wrong tool and there is a separate one.

---