Teaching a model to say "I don't know" is harder than teaching it to answer

Every retrieval system has a failure mode that doesn't show up in the demo: the question whose answer isn't in the corpus. The user asks it anyway, because they have no way of knowing what's in there. A well-behaved system says so. Most systems produce a fluent paragraph assembled from the nearest three documents, and the user has no way to tell the difference.

We shipped that failure for eight months. Our internal question set was built from questions our early users actually asked, which means it was built almost entirely from questions that had answers , because people ask about things they believe exist. Our attribution numbers looked great. Our refusal rate was 12%, and we told ourselves that was because the corpus was good.

A benchmark made only of answerable questions measures fluency, not honesty.

Building an unanswerable set

The fix started with generating questions we knew had no support. Four categories, 1,000 questions each:

• Absent facts. Plausible questions about topics genuinely not in the corpus. "What was the Q3 2019 headcount?" against a corpus starting in 2021.
• Superseded facts. Questions whose only support is a document that has been explicitly replaced. The trap here is that the old document is still highly relevant by any embedding measure.
• Adjacent facts. Questions where the corpus contains something close enough to be dangerous: the EU number when you asked about EMEA.
• Composed facts. Questions requiring a join the corpus doesn't support. Two documents, each with half an answer, and no basis for connecting them.

Against that set, our 12% refusal rate turned into a 12% correct-refusal rate. The system was inventing an answer roughly seven times out of eight when it had no business producing one. Adjacent facts were the worst category. Nobody was surprised once we looked: high retrieval scores, plausible text, wrong scope.

Three things that didn't work

1. Asking the model nicely

The obvious first attempt: instruct the model to answer only from the provided context and to say "I don't know" otherwise. This moved correct refusal from 12% to about 34%, and it cost us 9 points of recall on answerable questions because the model started declining things it could in fact support. Prompting trades one error for the other along a single axis. You can slide the point; you can't move the curve.

2. Retrieval score thresholds

If the top-ranked span scores below x, refuse. This is appealing because it's cheap and it's a single number. It also doesn't work, because retrieval scores are not calibrated across queries. A well-phrased question about an absent topic pulls high-scoring irrelevant spans; a badly-phrased question about a well-covered topic pulls low-scoring correct ones. We spent three weeks on per-query normalisation and got the correct-refusal rate to 51% at the cost of 6 points of recall. Better than prompting, still not a product.

3. Model self-reported confidence

We asked the model to score its own confidence and refused below a threshold. The scores were fluent, well-formatted, and almost uncorrelated with correctness, a Spearman coefficient of 0.31 against human labels. Self-reported confidence measures how confident the text sounds. Our conflict-detection feature was built on the wreckage of this attempt, so it wasn't wasted, but as a refusal signal it was noise with good manners.

What worked, and it's boring

An entailment check. After generation, every sentence in the draft answer is decomposed into atomic claims, and each claim is checked against its cited span by a small model doing one job: does this span support this claim, yes or no. Unsupported claims are cut. If cutting leaves nothing, the answer becomes a refusal that shows the nearest documents.

for claim in decompose(draft):
    span = citation_for(claim)
    if not entails(span, claim):     # small, cheap, single-purpose
        drop(claim)

if not remaining_claims:
    return Refusal(nearest=top_documents)

Correct refusal went to 94%. Recall on answerable questions dropped 1.2 points. Median latency rose 180ms, which we bought back elsewhere by running the check concurrently with streaming and holding unverified sentences until they clear.

Three things made it work that we didn't expect going in:

• Claim decomposition matters more than the entailment model. Swapping the checker for a frontier model gained 1.4 points. Improving how we split sentences into atomic claims gained 11.
• The checker must not see the question. Given the question, it starts reasoning about whether the answer is plausible rather than whether the span says it. We pass the span and the claim, nothing else.
• Numbers need special handling. "Roughly 340" entails "340" in one direction and not the other, and a checker without an explicit numeric rule gets this wrong about a third of the time.

The part we still don't handle well

Composed facts. If the corpus has EU revenue in one document and UK revenue in another, and you ask for EMEA, we now correctly decline to invent a total. But the genuinely useful behaviour is to say "I can give you EU and UK, and here's what's missing", and our decomposition currently isn't good enough to produce that reliably. We ship the refusal with both documents attached and let the human do the join. It's honest and it's unsatisfying.

If you're building this

• Build the unanswerable set first. Your current numbers are wrong and you can't act on that until you can see it.
• Measure correct-refusal rate and recall as a pair, always. A single number lets you fool yourself in either direction.
• Post-hoc verification beats better prompting. Generation and verification want different objectives, so give them different models.
• Don't let the verifier see the question.

Our benchmark harness is open source at github.com/getquire/refusal-bench, along with the four question generators. The corpora are ours and stay ours, but the generators run against anything.

---