The complaint usually arrives in the same shape: "it worked fine at launch, and now it gives wrong answers." What follows is almost always the same response — someone rewrites the system prompt, the answers change slightly, the complaint returns two weeks later.
Prompt-first debugging fails here because in a retrieval-augmented system, generation is the last stage and the least likely culprit. If the wrong chunks reach the model, no prompt saves you. The model is being asked to answer from material that does not contain the answer, and it will do exactly that.
This is the procedure we use to isolate where a RAG pipeline is actually failing. Work it in order. Do not change anything until you reach the end.
Step 0: Build the failure corpus first
Before diagnosing, collect twenty to fifty real queries that produced bad answers, with the answer each should have produced and the source document that contains it.
This is tedious and everyone wants to skip it. Do not skip it. Without it you have no way to tell whether a change helped, and you will spend weeks making adjustments that trade one failure class for another. This set is also the regression suite you keep afterwards.
Sort the failures into rough buckets as you collect them: wrong answer, incomplete answer, refused when it should have answered, answered when it should have refused, right answer wrong source. These buckets point at different stages, and the distribution alone often tells you where to look.
Step 1: Can the answer be retrieved at all?
For each failure, run the query through retrieval only and inspect the top ten chunks. You are asking one question: is the correct chunk anywhere in the results?
If the correct chunk is present but not ranked first, retrieval works and ranking does not. Skip to step 3.
If the correct chunk is absent from the top ten entirely, the problem is upstream of ranking — in the corpus, the chunking, or the embedding. Continue to step 2.
If the correct chunk does not exist in the index at all, you have a corpus problem, and the fix is ingestion, not retrieval. Common causes: the document was never ingested, it is a scanned PDF with no text layer and OCR was never run, it lives in a permission scope the ingestion service cannot read, or it was ingested once and the source has been updated eleven times since.
That last one deserves emphasis. A large share of "the AI is wrong" reports are actually "the AI is quoting a document that was accurate when it was indexed." Check the ingestion timestamp against the source's modified date before you touch anything else. Re-indexing on a schedule fixes more perceived quality problems than any model change.
Step 2: Is the chunking destroying the answer?
If the right content is in the corpus but retrieval cannot find it, look at how the document was split. Print the actual chunk containing the answer and read it as if you knew nothing else.
Typical defects:
- The answer is split across a boundary. The condition is at the end of chunk 7, the consequence at the start of chunk 8. Neither chunk answers the question and both score poorly against it. Fixed-size splitting with no overlap causes this constantly.
- Tables have been flattened into unreadable rows. A pricing or eligibility table reduced to a run of numbers with no headers is semantically meaningless to both the embedding model and the generation model.
- The chunk has no context. "It must be submitted within 30 days" — of what, under which policy, for which product? The heading hierarchy was discarded during ingestion. Prepending the document title and section path to each chunk is a small change with an outsized effect.
- Chunks are too large. A two-thousand-token chunk covering four topics has a diluted embedding. It matches everything weakly and nothing strongly.
Chunking is where the largest quality gains usually sit, and it is the least glamorous part of the system, which is why it is rarely the part anyone revisits.
Step 3: Is ranking putting the wrong thing first?
If the right chunk is retrieved but ranked fifth, and you pass the top three to the model, the answer was never available to it.
Check three things:
Semantic-only retrieval. Pure vector search is weak on exact identifiers — product codes, policy numbers, error codes, names. "Error E4021" embeds to something near every other error discussion. Hybrid retrieval, combining BM25 keyword scoring with vector similarity, fixes an entire class of failures that no amount of embedding tuning will.
No re-ranking stage. Retrieve broadly, then re-rank precisely. Pull the top thirty candidates by hybrid search and re-score them with a cross-encoder against the query. This is one of the highest-return additions to a naive pipeline, at the cost of some latency.
Near-duplicate documents competing. Three versions of the same policy all score highly and consume your entire context window with the same information, one version of which is current. Deduplicate at ingestion and keep a canonical flag. When the top three chunks are three drafts of one document, the system had one document's worth of information and no way to know which draft counts.
Step 4: Is the context window being spent badly?
Assemble the exact context passed to the model for a failing query and read it in full.
Look for: the correct chunk present but buried in the middle of a long context, where models attend to it least; context filled with boilerplate headers and footers that survived ingestion; the same content repeated across chunks; and total context so long that the instruction to cite sources is competing with twelve thousand tokens of material.
More context is not better. A tightly assembled context of three relevant chunks outperforms twenty marginal ones, and costs less.
Step 5: Is generation the actual problem?
Only now is it reasonable to look at the prompt. Take a failing query where the correct chunk was in the assembled context and the answer was still wrong. That is a genuine generation failure, and it usually has one of three causes:
- No instruction to abstain. If the prompt does not explicitly permit "the provided context does not contain this information," the model will synthesise something. It has been trained to be useful.
- Citation is requested but not enforced. If the answer format does not require a resolvable source identifier per claim, citations become decorative and stop tracking the actual source.
- Conflicting instructions. A prompt that says "answer only from the context" and also "be helpful and comprehensive" is asking for two different behaviours, and you get the second one.
Step 6: Fix in order of expected return, and measure each change
Work in this sequence, re-running the failure corpus after each change so you know what actually moved:
- Corpus hygiene — re-index stale documents, remove superseded versions, fix ingestion gaps. Cheapest, and frequently the largest single improvement.
- Chunking — semantic or heading-aware splitting with overlap, preserved heading context, structured handling for tables.
- Hybrid retrieval — add keyword scoring alongside vector similarity.
- Re-ranking — a cross-encoder pass over a wider candidate set.
- Context assembly — fewer, better chunks; deduplicated; relevant material positioned early.
- Prompt and output contract — explicit abstention, enforced citation, one clear instruction.
The order matters. Teams that start at step six and work upward spend months on prompt engineering to compensate for a chunking decision made in a week-one sprint.
What "fixed" should mean
Set the target before you start. A retrieval system is healthy when: the correct source appears in the top three for a defined majority of your failure corpus, every factual claim carries a resolvable citation, the abstention rate on out-of-corpus questions is high rather than zero, and the whole thing is re-measured automatically whenever the corpus, the chunking, or the model changes.
That last point is what keeps it fixed. A RAG system is not a build, it is a corpus that changes weekly, and quality decays without a standing evaluation loop.
If you would rather not run this procedure yourself, it is roughly the first week of an AI audit and rescue engagement, and it is also how we build knowledge systems and RAG pipelines in the first place. The related failure modes across the rest of an LLM application — cost drift, latency creep, silent tool failures — are covered in why custom LLM apps fail silently in production.
Bring us the queries that are failing and we will tell you which stage is at fault before anyone quotes a rebuild.