Every duplicate invoice is a near-duplicate

Our first duplicate detector hashed file bytes. It found 340 duplicates in a 40,000-document corpus. Everyone was pleased for about a day. Then someone pointed out that byte-identical files are the case nobody worries about. They are the same attachment, saved twice. They cost storage. They do not cost money.

Here is the expensive case. An invoice arrives as a PDF in January. It gets paid. In March it arrives again, this time a scan of a printout. Different filename. A reminder stamp across the top. A slightly different total, because someone added a late fee. Two documents. Zero shared bytes. One obligation.

What "same invoice" means

There is no single definition. So this cannot be a boolean. Think in relations instead. A person handles each one differently:

That last row breaks a naive rule immediately. A monthly retainer produces twelve invoices a year. Same vendor. Same amount. All twelve legitimate. Key on vendor and amount without periodicity awareness and you get a flood of false positives. The flood gets ignored. Then the matcher is worse than nothing, because people have stopped reading it.

Blocking, then scoring

Every pair in a 40,000-invoice corpus is 800 million comparisons. So: the standard two-stage shape. Cheap blocking keys generate candidate pairs. Expensive scoring runs only inside a block.

Our blocking keys, all of which run and are unioned:

• Normalised vendor plus rounded total. Catches most resends and format variants.
• Invoice number stem. The number with separators stripped and suffixes removed, which links amendments to originals.
• Total plus date window. Amount equal within a cent, issue dates within 45 days, no vendor requirement. Catches the case where the vendor name was recognised differently on a scan.
• Purchase order reference. Where present, the strongest key we have, and the only one that catches split deliveries.

Scoring inside a block stays deliberately simple. Field-level similarity on vendor, number, dates, total and line-item structure. Fixed weights, in a config file, where someone can read them. We tried a learned scorer. It scored about the same on our corpora. Nobody could explain a single one of its decisions to a finance team. That decided it.

A duplicate flag someone cannot justify to their auditor is not a feature, it is an argument waiting to happen.

The costs are not symmetric

A false negative is a payment made twice. A false positive is thirty seconds of attention. Those are not the same size, and the threshold should say so. Only up to a point, though. Push it too far and the reviewer's attention runs out. A queue nobody works detects nothing.

We default to 0.70. Thirteen pairs per thousand invoices is a few minutes of review, and 96% recall buys that easily. Go below and precision collapses faster than recall improves. Familiar shape. It is why the last percent of recall rarely pays for itself.

Present pairs, not verdicts

The output is a list of pairs. Each one renders side by side. Differing fields highlighted, both source regions visible, and a one-line statement of why they matched: same vendor and total, invoice numbers differ by a suffix, 61 days apart.

Reviewers clear those in seconds. The resolution is recorded, so the pair never resurfaces. That record is also the only labelled data anyone has for this problem. It is what lets you tune a threshold against a corpus instead of a hunch.

Matching lives in quire/analysis/dedupe. The weights are in quire.toml and are meant to be edited; the defaults come from three corpora and will not be right for yours.

---