How we handle hallucinations in production AI assistants

One of the first questions clients ask when we start a project is some version of: "What happens when it makes something up?"

It's the right question. AI assistants can produce confident-sounding answers that are wrong — a phenomenon called hallucination. And in a production environment, where the assistant's outputs have real consequences, this has to be taken seriously.

Here's how we think about it and what we actually build.

Hallucination can't be eliminated entirely. The goal isn't zero hallucinations — it's designing a system where hallucinations are rare, caught before they cause harm, and handled gracefully when they occur.

This is a different framing than "how do we prevent wrong answers?" It's "how do we build a system that's safe to use even knowing that wrong answers will sometimes happen?"

The most effective way to reduce hallucination is to limit what the assistant uses as a source. An assistant instructed to answer only from a specific, curated knowledge base — and to say "I don't know" when the answer isn't there — produces far fewer wrong answers than one with access to its full training data.

This is why knowledge base design is one of the most important parts of any assistant build. The quality, accuracy, and completeness of the source material directly determines the quality of the outputs.

We design every assistant with an explicit "I don't know" path. When the assistant can't find a reliable answer in its knowledge base, it doesn't guess — it acknowledges the limit and offers an alternative: escalate to a human, check a specific document, or come back with more information.

Left to their own devices, AI models tend toward helpfulness — which means they'll try to answer even when they shouldn't. The system prompt has to explicitly override this tendency.

We tell clients: assume the assistant will occasionally be wrong, and build a feedback mechanism from day one. When a user catches an error, there should be an easy way to flag it. That feedback — what went wrong, in what scenario — is the input that lets us refine the knowledge base and prompts over time.

An assistant that gets better over time is one that has a reliable way of learning from its mistakes. That loop doesn't happen automatically. It has to be designed.