The hard part of RAG is knowing when you found nothing

The first RAG that seems to work almost always deceives you. You ask about a document that is in the base, search brings back a similar chunk, the model answers, and it looks like memory has been solved.

The real test comes when the answer is not there.

Vector search always returns a first result. Cosine similarity orders vectors; it does not judge whether the subject exists. If the base has nothing about the question, the top of the ranking still arrives. And it often arrives with a number high enough to look like evidence.

After that, the model does what models do: writes on top of the context it received. The error does not show up as an error. It shows up as a confident answer, with an irrelevant chunk used as the foundation.

Similar is not relevant

"The most similar" is a relationship inside the base, not a decision about the world. If I search for a subject that does not exist, the system still has to choose something as the most similar. The best result among bad options is still bad.

That difference changes the design of RAG. It is not enough to ask "which chunk reached the top?". The right question is "did the top result pass a threshold that means relevance for this base?". Without that threshold, RAG never says "I do not know". It only changes guesses.

The problem is that a threshold does not come from intuition. Every base has a noise floor.

The noise floor

The noise floor is the score that a random chunk tends to receive in your own base. Not in a tutorial sample, not in a provider benchmark, in yours. Long documents, repeated terms, similar names, and standardized language push similarity upward even when the chunk answers nothing.

Without measuring this, any cutoff is a guess. A threshold that is too low accepts bad context. A threshold that is too high turns the system into someone who says "I do not know" to a question that did have an answer.

The practical way is to test questions you know do not exist and see which scores they get. Then test questions that do exist and see where the truly good scores appear. The zone between the two is where calibration lives.

Rerank and the right to deny

Embedding approximates by neighborhood of subject. Rerank reads the question and the chunk together. That difference is small in the name and large in the result.

The reranker can demote a chunk that uses similar words but does not answer the question. It also helps separate the chunk that only mentions the topic from the one that contains the needed information. In many systems, this second step changes perceived quality more than switching the model that writes the final answer.

The main point is not only better ordering. It is allowing the system to say: nothing here helps. Without that right to deny, every RAG becomes a machine for justifying the least wrong chunk.

Portuguese and chunk size

Content in Portuguese also charges its price. An embedding model trained only for English may poorly approximate terms, inflections, and writing style. The symptom is not an exception in the log. It is a mediocre answer that looks acceptable until someone who knows the subject reads it.

Chunk size decides as much as the model. A large chunk dilutes information: the question hits one part, but the generator receives several ideas together. A small chunk loses the antecedent: the pronoun, the definition, or the condition was in the previous paragraph.

That is why I treat chunking as part of the product, not as an indexing detail. If the chunk does not carry enough context to be cited alone, it should not be delivered alone. If it carries too much, it becomes noise packaged as a source.

Where it breaks

An indexed stale document is worse than a missing document. The system finds it, gives it a high score, and the model reads it as truth. Absence can at least become "I do not know"; staleness becomes a wrong answer with a source.

There is also too much fear. A threshold tuned to never fail starts denying an answer when it was in the base. Calibration is a tradeoff, not a final victory: accept some false positive risk or frustrate the user with false negative. Pretending you can eliminate both only pushes the decision into an invisible place.