Eighteen RAG pipelines, same model, same embeddings, same documents: the FRAMES benchmark leaves little room for interpretation. The best configuration reaches 78.9% end-to-end accuracy, while an agent loop that can read the results and search again hits 92.7%. The gap is not just numerical: it is roughly equivalent to handing the model the right articles from the start.

The test isolated the components that are usually taken for granted in a RAG pipeline: hybrid search, reranking, query decomposition, and query expansion. The goal was to measure how much each one actually contributed. The 824 multi-hop questions from FRAMES served as the testbed, with an LLM judge and a second independent evaluator checking the answers. The agreement between the two was almost total, with a Cohenโ€™s ฮบ between 0.93 and 0.98. But the interesting result is not the winner.

The reranking paradox

Adding a small reranker removed 9 percentage points from the best pipeline. A larger one barely helped. This kind of result forces a rethink: reranking has long been treated as a near-mandatory step for improving the quality of documents passed to the model. Here, on multi-hop questions, that intermediate filter can discard exactly the passages needed to connect the facts. If the reranker is not up to the task, the damage is not neutral: it is worse than not using it.

The agent loop, by contrast, does not try to get everything right on the first pass. It reads, evaluates the results, and can repeat the search. This explains why 92.7% approaches the score you would get by providing the correct sources directly. The difference between 78.9% and 92.7% is the cost of static retrieval that gets document selection wrong: the limit is not the modelโ€™s ability to answer, but the quality of the retrieved context.

Citations that prove nothing

There is a second signal, perhaps more insidious. Models sometimes fill gaps by drawing on training memory, even when explicitly told to stick to the retrieved documents. The answers remain full of citations, but those citations do not prove that the content was actually read. The team had to verify every correct answer against the text actually shown to the system. In a corporate or regulated context, this is critical: citing a document is not the same as proving retrieval.

For anyone designing self-hosted RAG stacks, the benchmark carries a structural lesson. Additional components are not free: they consume resources, lengthen the pipeline, and can degrade results if they are not measured in their own domain. The agent loop delivers more accuracy but multiplies model calls and searches. In terms of TCO and inference requirements, that means planning capacity for multiple passes, not for a single answer. That is no small detail in on-premise environments where VRAM and latency are concrete constraints.

The FRAMES result does not say which component to adopt in absolute terms. It says the order of priorities needs to be reconsidered: iterative retrieval and source verification first, then ranking accessories. And a benchmark based only on retrieval scores can tell a different story from the real answer accuracy.