The inbox is the ingestion pipeline

Ask where a small team's invoices live and the honest answer is a mailbox. There may be a folder on a drive as well, and it will be incomplete, because the folder is a copy someone maintains by hand and the mailbox is where the documents actually arrive.

So mailbox ingestion turns out to be the main path for a lot of people, and it brings a specific set of problems that do not exist when you point a tool at a directory.

Multiplicity is the default

One invoice generates, in a typical thread: the original message with the attachment, a forward to accounts with the same attachment re-encoded, a reply with the attachment quoted back, a reminder with a fresh copy, and someone's copy saved to the drive and re-attached later. Five arrivals. One document.

Content hashing catches the identical ones and misses the rest, because re-encoding changes bytes without changing content. So attachments are deduplicated after extraction, on normalised text plus structure, and the surviving record keeps every arrival as provenance: which message, which sender, which date, which thread. That provenance is what answers "when did we first receive this", a question people ask constantly and no folder can answer.

The body is a document too

A lot of information exists only in the message body: the approval, the disputed line, the promise to pay next week, the correction that means the attached invoice is not the one to use. Ingesting only attachments loses all of it, and the loss is invisible.

Bodies are indexed as documents in their own right, linked to their thread and to any attachments in the same message. Quoted history is stripped, badly, because quoting has no standard and every client marks it differently, and the failure mode is either duplicating a whole thread into every reply or dropping something someone wrote inline in the quote.

Every mail parser is a pile of heuristics standing in a trench coat. Ours is too. The difference is whether the heuristics are visible.

What the pipeline actually does

• Walk containers to a bounded depth. Zips inside zips, messages attached to messages. Depth-limited, size-limited, count-limited, because unbounded recursion over an untrusted attachment is a resource exhaustion bug waiting to be reported.
• Type by content, never by extension. A file named invoice.pdf that is actually a scanned image, or an .xls that is really HTML, are both common enough to be routine.
• Drop signature and tracking images. Logos, tracking pixels and signature images make up a large share of attachments by count and none by value. Filtered on size, dimensions and repetition across messages.
• Never execute anything. No macros, no external entity resolution, no network fetch for remote content, no following links found in message bodies. Parsers run with no network access, and that is not configurable.
• Keep the thread. Message identifiers give a real reply graph. Subject-line grouping is the fallback and it merges unrelated conversations that share a generic subject, which happens more than you would like.

Numbers from a real mailbox

One contributor's accounts mailbox, five years, donated under a narrow agreement and used with permission:

The line worth staring at is 38,900 to 7,100. Over 80% of what arrives is noise or repetition, and a system that indexes it all wastes more than storage. It fills every retrieval result with five copies of the same thing and pushes the useful document off the page.

Privacy, since this is a mailbox

Everything here runs locally. Nothing leaves the machine, there is no telemetry, and the ingest process has no network access at all. That property is easy to state, easy to verify with a packet capture, and worth verifying instead of believing. A mailbox is the most sensitive corpus most people own, and it is the one where "trust us" is least appropriate.

Deletion is honoured properly, too. Removing a source removes derived text, embeddings, thumbnails and cached regions, not just the row that points at them. Deletion that leaves derivatives behind is not deletion, and it is a surprisingly common bug in systems that grew an index before they grew a delete path.

Mailbox connectors are in quire/ingest/mail. IMAP, Maildir, mbox and PST all work offline; there is no hosted component and there is not going to be one.

---