We deleted our chat interface

For eighteen months, indexing a corpus dropped you into a message box. Obvious design. Demoed beautifully. The numbers underneath were bad, and it took us far too long to read them right.

Sessions were short. One question, one answer, no follow-up. We read that as success. Wrong. The follow-ups were happening. They were just happening in a fresh session, question retyped from scratch, because nobody could find the old one.

What a conversation is bad at

• Re-running. The most common real workflow is asking the same question every month against a changed corpus. In a chat log, that means scrolling to find what you asked in April and copying it. There is no other affordance.
• Reviewing. A colleague needs to check whether the answer is supported. A transcript containing eleven exchanges, four of which were dead ends, is a bad artefact to hand someone.
• Diffing. "Has this changed since last quarter" is the question underneath half of all the others, and two chat sessions cannot be compared.
• Scripting. Nobody can put a chat box in a Makefile.
• Being wrong usefully. When an answer is wrong, the useful next step is inspecting the retrieval, not rephrasing the question. A message box only offers rephrasing, so that is what people did, three times on average, and then they gave up.

The interface taught people that a bad answer was their fault for asking badly. It almost never was.

What replaced it

A command. One question in, one answer out. Citations attached. Exit code zero, or non-zero if the system wouldn't answer. Queries live in files. Files live in the repo with everything else. They get reviewed in pull requests. They run on a schedule. Their output diffs against the last run.

The terminal was never the point. A durable object was. Once a question is a file, the awkward things people kept attempting in chat go ordinary. Version it. Share it. Parameterise it. Run it over another corpus. Put it in CI and fail the build when the answer moves.

Numbers

Watch the re-run figure. Two thirds of all usage is a question somebody already asked. A conversation obstructs that workload. A file makes it trivial.

What we lost

Real costs. Exploration got worse. Chat was better for poking at a strange corpus with vague questions, and we have not replaced that. Onboarding is harder too. A message box needs no explanation. A command has flags. And some users left. That is what removing a thing people used costs you.

One piece of the old design survived. An interactive mode, explicitly a scratchpad. One key writes any question in it out to a file. Explore there. Anything worth keeping graduates to the repo.

The general version

Chat is a default. It is what an interface becomes when nobody has decided what the work actually is. For work that gets reviewed, repeated and cited, the durable artefact is the point. The conversation was wrapping paper. We kept mistaking it for the gift.

The interactive scratchpad is quire ask -i. Everything it does, the non-interactive form does too, which is deliberate: nothing is reachable only by typing at a prompt.

---