Does RAG Answer From the Passage — or From Memory?
In my previous article(https://huggingface.co/blog/nazeerbashashaik/rag-retrieval-evaluation), I tested whether a RAG system retrieves the right passage for a question. That is only half the job. Retrieval finding the correct passage means nothing if the model then ignores it — so this time I tested the second half: given the passage, does the model actually answer from it?
The questions I wanted answered were specific. Would the model stay faithful to the passage, or would it hallucinate? And the sharper test: if the answer is not in the passage at all, would the model admit that — or would it quietly answer from its own memory as if the passage weren't there?
That last question is the one that matters most, because the whole promise of RAG is that answers come from your documents. A model that answers from memory instead has quietly broken that promise, while looking exactly as confident as one that didn't. The setup reused everything from part one — the same man-page corpus (dig, tcpdump, tshark) and the same 8 questions — with one addition: a local open language model (Mistral-7B, 4-bit) to generate the answers, prompted explicitly to use only the provided context, and to say so if the answer wasn't there.
The test
To probe faithfulness, I did something deliberately adversarial: I fed the model a passage that did not contain the answer to the question, and watched what it did — refuse honestly, or answer anyway from its own memory.
The surprise
The result was mixed in an interesting way. For some questions the model correctly said the answer wasn't in the passage. For others, it confidently answered anyway — reciting the correct definition from its training, even though the passage in front of it was about something else entirely. I ran each case multiple times to be sure this wasn't random, and it wasn't: the same inputs failed the same way, consistently — some 10 times out of 10. So the failure was real and stable. The question was why — why did the model override the passage for some inputs but not others?
Two theories, both wrong
My first instinct was that the question was to blame. Maybe the model overrode the passage for flags it knew especially well — the famous options it had seen thousands of times in training. I tested that: I took well-known flags and obscure ones and fed them all wrong passages. They failed at about the same rate. Theory one was wrong.
My second theory was that simple flags — the ones with a crisp, one-line meaning — were easier for the model to answer reflexively from memory, while complex flags forced it to actually read the passage. I tested that too, sorting flags into simple and complex groups. The split was identical across both groups. Theory two was wrong.
Refuting my own guesses was the useful part. It forced me to stop looking at the question and look at what I'd been varying without noticing.
The real cause: it was the passage, not the question
When I lined up which inputs failed against which passages they'd been paired with, the pattern was obvious. The failures weren't tied to particular questions at all — they were tied to particular passages.
When the wrong passage still looked like the right kind of content — documentation about command options — the model treated it as license to answer, and pulled the answer from memory. When the wrong passage was clearly a different kind of text — prose describing the tool rather than its options — the model correctly refused.
The proof was direct: I could take the same question, pair it with one wrong passage and get a confident hallucinated answer, then pair it with a different wrong passage and get an honest refusal. Same question, different passage, opposite behavior. The model's faithfulness was a property of the retrieved passage, not of the question being asked.
The takeaway
The validity of a RAG answer depends on the quality of what was retrieved — and not in the way I first assumed. It isn't just that bad retrieval means the model lacks the answer. It's that a plausible-but-wrong passage is more dangerous than an obviously irrelevant one, because the plausible one gives the model permission to answer confidently from memory, ignoring the passage entirely. A retriever that returns "close but wrong" can be worse than one that returns nothing.
And that reframes the risk. A model that says "the answer isn't in the provided context" is doing the right thing — a no-answer is honest and safe. A model that produces a confident answer from memory, while looking exactly as authoritative, is the real hazard: a wrong answer is far worse than no answer, because nothing on the surface tells you it's wrong.
This connects the two halves of the RAG story. Part one showed that retrieval can silently return the wrong passage. Part two shows that when it does, the model may not save you — it may confidently paper over the mistake. Retrieval quality and generation faithfulness are not separate problems. They are the same problem, and you have to test both.
Note on tools: built in Python on a free Google Colab GPU, using a local 4-bit Mistral-7B for generation. AI assistance was used for the code; the test design, the adversarial setup, and the analysis are my own.