RAG fails in the plumbing, not in the model
When the assistant gets it wrong, the problem is usually in the path the data takes to reach it. Two real cases from one of my projects.
How can a chat answer accurately about a document the model never saw in training?
That is the question I solved in practice while building the assistant for my online résumé: a chat that answers, in real time, questions about my career.
The technique behind it has a name: RAG, retrieval augmented generation. Instead of relying only on what it learned in training, the model looks up the right information before answering.
Simple to explain. Sophisticated to build.
How the flow works
- Your question becomes a numeric vector, the embedding: a point on a map of meaning.
- That point is compared with thousands of others in a vector database, looking for the passages closest in meaning.
- Only after this search does the model receive the question together with those passages and write an answer tied to what is in the documents, not to imagination.
Three steps in theory. In practice, this is where it gets interesting.
Problem 1: broken routing
Questions about the most recent experience broke the routing by résumé section. The reason: keywords were matched against fragments of text, not whole words. Without finding the right section, the system fell back to pure similarity search and could answer with an old job instead of the current one.
The root cause was not even the model. It was the routing logic and a cache of the embedding index that never refreshed when the résumé changed.
Problem 2: conversation memory
Someone asked a question and followed up with another that depended on the first ("and where is that company headquartered?"). The system did not understand that "that company" referred to what had been said before.
The fix was to rewrite the question with the model itself before the search, bringing in the reference from the previous message, without changing the final answer.
The takeaway
This is the part of AI Engineering that strikes me the most: RAG rarely fails in the model. It fails in the plumbing around it.
RAG is not magic. It is well-thought-out architecture.
Originally published in Portuguese on LinkedIn, where the conversation continues in the comments.
