Your RAG System Is Not Hallucinating. It Is Retrieving Badly.

A retrieval system gives a confident, fluent, wrong answer. The reaction is almost always the same: blame the model, or rewrite the prompt. Add a line telling it not to make things up. Try a larger model.

That reaction assumes the model invented something. Far more often it did not. It was handed the wrong passages and answered them faithfully. The output is wrong; the generation step is innocent. If you only ever inspect the answer, every failure looks like a model failure, because the answer is the only part you are looking at.

The habit worth building is to log what was retrieved alongside what was said. The first time you read the passages that produced a bad answer, the diagnosis usually becomes obvious — and it is rarely the model.

Chunking sets the ceiling, and nothing downstream raises it

How a document is split decides the best answer the system can ever give. Split a table across two chunks and neither half means anything. Split a procedure at step four and the retrieved half is not merely incomplete, it is misleading, because it reads as though it were whole.

No amount of retrieval quality recovers this. A perfect search over badly formed chunks returns exactly the wrong thing, with high confidence, every time. The rule we use is that a chunk has to be a complete thought — something a person could read on its own and act on.

On one question bank we had to rebuild ingestion so that a question and its answer key were guaranteed to stay a single atomic chunk. Nothing about the search changed. The wrong answers stopped, because the material finally arrived intact.

One kind of search is not enough

Dense embeddings match meaning, which is what makes them worth having, and they are unreliable on exact identifiers. Form numbers, SKUs, error codes and product names are precisely the terms a user is most certain about and least willing to see ignored. Semantically, one reference number looks much like another.

Keyword search has the mirror-image failure: it finds the literal string and misses every paraphrase. Neither is a general solution, and the argument about which is better is the wrong argument.

Run both, in parallel, and merge the two ranked lists with reciprocal rank fusion. RRF needs no score calibration between the two systems — it combines positions, not scores, which is what makes it robust when one retriever is confident and the other is not.

Reranking is cheap where it counts

First-stage retrieval is tuned for speed across millions of vectors, and that tuning costs accuracy. A cross-encoder reads the query and a passage together rather than comparing two independent embeddings, so it judges relevance far better — and it is far too slow to run over the whole index.

That constraint resolves itself once you stop thinking of it as a search method. Retrieve broadly, then rerank only the top handful. Expensive per item, trivial at that volume, and it reorders results more than most teams expect.

A floor — and the floor is also a bug waiting to happen

Some questions have no answer in the material. The system needs a score threshold below which it declines: "I do not have that." Users forgive a system that admits a gap. A confident wrong answer costs you the user, and it costs you the next ten answers too, because they now check everything.

This is where we made our own mistake, and it is worth describing because it is silent. We set the threshold too high. Valid content began scoring below it and was dropped — answers that should have been returned simply were not. There was no error, no warning, nothing in a log saying a result had been suppressed. The system looked like it was working and was quietly refusing to answer.

The error was treating the threshold as a tidy round number rather than a measurement. Cross-encoder scores are not cosine similarities and do not live on a 0-to-1 scale; legitimate passages routinely sit well into negative values. The only way to set that number is to run real queries, read the scores real content receives, and put the floor underneath them.

A threshold is a destructive decision applied automatically, thousands of times, where nobody sees the thing it removed. That deserves the same scrutiny as a rule that unpublishes records, for exactly the same reason: the failures are invisible by construction.

What this changes about how you build

The pipeline that results is unglamorous and mostly has nothing to do with the model: normalise the query, search two ways at once, fuse the rankings, rerank the top results properly, apply a measured floor, and only then generate an answer from what survived.

Prompt engineering is the last part, and the smallest. It is worth doing once the retrieval underneath it is sound, and it cannot compensate when it is not. If your evaluation only records whether answers were right, add the retrieved context to the record. Most teams discover their model was never the problem.

• Log retrieved passages with every answer. You cannot debug generation without seeing its input.
• Read chunks, do not just count them. Splitting defects are obvious to a reader and invisible to a metric.
• Measure the threshold against real scores. A number chosen because it looks neat will silently suppress valid answers.
• Test with exact identifiers. Reference numbers and codes are where a dense-only system fails most visibly to the people who matter.