The part of RAG that gets tested the least

Retrieval-augmented generation has two real stages: find the relevant document, then generate an answer from it. Most evaluation attention goes to the second stage, does the generated text read well, is it fluent, does it sound like a good answer. The first stage is easier to skip checking, and it's also where things go wrong more often than the framing suggests.

A small, real knowledge base

Five short documents, a fictional product's support docs:

docs = [
    "The free plan includes up to 3 projects and 500MB of storage.",
    "Upgrading to Pro removes the 3-project limit and adds 50GB of storage.",
    "Passwords must be at least 12 characters and include one number.",
    "API rate limits are 100 requests per minute on the free plan, 1000 on Pro.",
    "Support tickets are answered within 24 hours on Pro, 72 hours on the free plan.",
]

A real retrieval failure, caught by checking the similarity score

question = "What's the password requirement?"
# TF-IDF + cosine similarity retrieval
best_idx = sims.argmax()
print(f"retrieved (similarity={sims[best_idx]:.2f}): {docs[best_idx]}")

retrieved (similarity=0.05): The free plan includes up to 3 projects and 500MB of storage.

Wrong document. The actual password-requirements document exists in the knowledge base, word for word, and the retriever didn't return it. The similarity score, 0.05, is itself a real signal something's off, a genuinely relevant match would score meaningfully higher, but that score is easy to never look at if the pipeline only surfaces the final generated answer.

What the generator does with the wrong document

prompt = f"Answer using only the context.\nContext: {retrieved_doc}\nQuestion: {question}\nAnswer:"
model.generate(**inputs, max_new_tokens=40)

generated answer: '3 projects'

A confident, fluent, completely wrong answer. This isn't the model hallucinating, inventing information from nowhere, it followed the instructions exactly: answer the question using the given context. The context was simply about the wrong topic entirely, and nothing in the generated text signals that, no hedge, no "I'm not sure," no visible trace of the mismatch. Read the generated answer alone, with no visibility into what document produced it, and it looks like a normal, plausible response.

The same question, with the right document

correct_doc = "Passwords must be at least 12 characters and include one number."

generated answer: 'at least 12 characters and include one number'

Correct, and generated by the exact same model, same prompt template, same generation settings. The only variable that changed was which document retrieval handed it. The generator's own quality was never the problem in either case.

Why this matters for how RAG systems actually get evaluated

Judging a RAG system by reading its final answers and asking "does this sound right" measures something real, but it systematically misses this exact failure mode, a fluent, well-formed answer built on the wrong source. The retrieval step needs its own, separate evaluation: given a question, did the system actually retrieve a document that contains the answer, checked directly, independent of whatever the generator does with it afterward. A system that retrieves correctly 95% of the time and generates well from what it's given will outperform one that retrieves correctly 70% of the time and phrases wrong answers beautifully, and reading generated text alone can't reliably tell those two systems apart.

The takeaway

A RAG pipeline's generated output being fluent and confident says nothing about whether it's grounded in the right source, the failure mode demonstrated directly above produces text that's fluent, confident, and about the wrong thing. Evaluating retrieval separately, checking whether the right document actually got retrieved for a given question, not just whether the final answer reads well, is what catches this before a user does.