Your RAG benchmark is measuring the wrong thing

Public retrieval benchmarks are built on clean corpora: Wikipedia, news archives, curated document sets. Every document is well-formed, none of them contradict each other, nothing is superseded, and every user can see everything. Real corpora have none of these properties, and the differences are not cosmetic. They're where systems actually fail.

We scored 0.81 on a well-known public benchmark and then watched our system confidently return a 2021 travel policy to someone asking about the current one. The benchmark had no notion of a superseded document, so it could not have caught that. It wasn't wrong; it was measuring a different thing than the one we cared about.

Four properties public benchmarks don't have

Documents contradict each other

A real corpus contains four versions of the same policy, two of which are drafts, one of which was never approved but got shared widely enough to look canonical. Ranking by relevance retrieves all four. The question of which one is true is not a retrieval question, and no amount of embedding quality answers it.

Recency is a first-class signal, and it is not monotone

Usually the newest document wins. But "what was our position in 2023" wants the old one, and a superseded policy is still the right answer to a question about history. Any system that hard-codes recency weighting gets the second class wrong; any system that ignores it gets the first class wrong most of the time.

Not everyone can see everything

This is the one that turns a bug into an incident. A benchmark with no access control cannot measure permission leakage, and permission leakage is the failure that ends a deployment. We wrote about the specific shape of this in an earlier post. The short version: filtering after retrieval leaks through ranking, latency and phrasing long before it leaks through the answer text.

Half the corpus is barely parseable

Scanned contracts, screenshots pasted into tickets, tables built out of merged cells and manual line breaks, a spreadsheet where the real data starts at row 14. Public benchmarks give you clean text. Real corpora give you a PDF someone printed and re-scanned at an angle.

What we measure now

Note what isn't on that list: answer quality as judged by a model. We measured it for a year. It correlates with how much people like reading the output and not at all with whether they can rely on it, and optimising for it quietly makes the system more confident and less correct.

If your evaluation can't tell a confident wrong answer from a correct one, it will reward the confident wrong answer, because it reads better.

How to build the corpus you need

You cannot buy this benchmark. It has to be built from a corpus that looks like your customers' corpora, which means it has to come from users willing to let you evaluate against their real content. Three groups donated corpora under a narrow agreement, each with a different flavour of mess, and every release runs against all three.

• Take a snapshot and freeze it. A corpus that changes under you makes every comparison meaningless.
• Label at the span level. "This answer is good" is unactionable. "This claim is not in this span" is a bug report.
• Include the questions people asked and gave up on. Your logs contain them: reformulated three times, then abandoned. That's your hardest set and it's free.
• Red-team permissions on every release. Automated, adversarial, treated as a release blocker; nobody files it as a report.

The uncomfortable part

Doing this properly means your headline numbers get worse, and they get worse right before you need to show them to someone. Our attribution score dropped 14 points the week we added the scanned-document set, and it dropped another 6 when we started counting table-cell errors as failures instead of parse warnings.

Both drops were the benchmark starting to tell the truth. The alternative is a number that goes up while the tool gets less trustworthy, and you find out from an issue titled "it made this up".

The harness is open source at github.com/getquire/refusal-bench. The corpora aren't ours to share, but the generators and the scoring run against anything you point them at.

---