AI & Agents
Your RAG Pipeline Retrieves the Wrong Chunks. Here's the Fix.
Thousands of pages in, a confident wrong answer out. Where chunking and similarity search break, what a page-index approach does differently, and when long context beats RAG entirely.
RAG retrieves wrong chunksfix RAG pipelineRAG chunking problems
Picture a document Q&A system fed roughly 3,000 pages. It returns a fluent, confident answer, and the answer is wrong. The bill for that afternoon of testing is $40 in API calls. Nothing crashed, no error in the logs, and the model did exactly what it was told.
The model got the wrong context. That is the failure I see most often in RAG systems, and it is easy to blame the wrong layer, so most of the effort goes into prompts when the retrieval step is the thing that broke.
Why does my RAG pipeline return the wrong chunks?
The standard recipe is familiar. Take a big PDF, cut it into 500-word chunks (usually because 500 is the default, not because anyone decided it), turn each chunk into a vector embedding, store them in something like Pinecone or Chroma, and at query time pull the top-K most similar chunks and hand them to the LLM.
It works, up to a point. It also rests on three assumptions that break in predictable ways.
| Assumption | What actually happens |
|---|---|
| A chunk holds a complete idea | A fixed window splits a concept at word 501. Retrieval returns half a thought and the model invents the other half. |
| The user phrases the question like the document | A user asks “how does this work?” and the document says “mechanism of operation”. Vectors that should line up sit further apart than you would like. |
| Cross-references don’t matter | A clause reads “as per Rule 12.4.3 of Appendix B” and the rule lives on page 578. The search lands on page 4, finds nothing useful on 578, and the model answers anyway. |
The last one is the worst, because the answer sounds right. The text the model needed existed in your data. It never saw it.
What can I fix before changing architecture?
Quite a lot, and I would try these first.
- Change how you split. If you chunk at a fixed size, ideas get cut. Splitting at meaning boundaries (a new section, a new clause, a new topic) keeps each chunk readable by itself.
- Test with questions that use the wrong vocabulary. Take ten questions phrased the way a user would phrase them, not the way the document does, and see what comes back. This is the cheapest way I know to find the vocabulary gap.
- Check for references. Search your documents for phrases like “see section” or “as per”. If they are everywhere, similarity search on isolated chunks will keep failing on them.
If you have done all that and the system still answers confidently and wrongly on dense, cross-referenced material, the limit is probably the approach itself rather than your implementation.
What is vectorless RAG (page index)?
There is a newer pattern, sometimes called Page Index or vectorless RAG, that drops embeddings, chunking and similarity search entirely. An LLM reads the document first and builds a hierarchical table of contents. At query time the LLM reasons its way down that tree.
Think about how you use a thick book. You do not scan page 1 onward. You open the index, find the chapter, go to the section. Page Index asks the model to behave the same way.
Each node in the tree stores a title, a node ID pointing back to the original pages, a summary, and its child nodes. The source example I like is a film script, Sholay:
- Root: the full summary
- Act 1: life in Ramgarh, with scenes for Gabbar’s introduction and the recruitment of Jai and Veeru
- Act 2: the conflict escalates, including the scene where Thakur loses his arms
- Act 3: the reckoning
Ask “why did Thakur lose his arms?” and the system does not search 3,000 pages. It reads node summaries, picks the branch that fits, and fetches the original text only from that node. The full script never enters the context window.
The model decides where the segments break, at the points where a character enters, a plot turns, or a legal clause ends. Those are boundaries in meaning, and that is the part I find most interesting.
On the three failures above: the context stays small because you send three sections instead of three thousand pages, the vocabulary mismatch turns into a reasoning problem rather than a matching problem, and a clause that says “as per Rule 12.4.3” can point straight at the node holding that rule.
What does it cost?
I would not sell this as free. Reasoning takes time, so it is slower than a vector lookup. Building the tree and traversing it both cost tokens, so at high query volume with tight latency targets, vector RAG probably still wins on unit economics. And the engineering is heavier: instead of chunk, embed, retrieve, you now think about tree structure, node design, summary quality and traversal strategy.
My rough rule: reach for it when the documents are long and structured (contracts, research papers, technical docs), when query phrasing varies a lot, and when a wrong answer actually hurts. If your content is uniform and your queries are predictable, plain vector RAG is fine.
Is RAG dead? When is long context the better tool?
A separate challenge to RAG comes from the other direction. Modern models accept 1M+ tokens, which is roughly the whole Lord of the Rings trilogy in one prompt. So you can skip the vector database, the embedding model, the reranker and the sync jobs, and just send the data to the model.
Long context also does something RAG cannot: it can reason about gaps. Ask “which security requirements were omitted from the final release?” and RAG pulls snippets about requirements and snippets about release notes. It cannot retrieve the space between them. A model that sees both documents in full can.
RAG still earns its place for three reasons:
- Compute cost. Stuffing 250K tokens into every query gets expensive fast.
- Needle in a haystack. Attention dilutes at extreme lengths, and retrieval removes noise.
- Scale. Your data lake is petabytes, and no context window covers that.
So the split I would use: long context for bounded, reasoning-heavy work such as a single contract or a technical comparison, and RAG for searching enterprise-scale knowledge. I do not think there is a universal architecture, and anyone selling one is guessing.
A short checklist before you rebuild anything
- Log the retrieved chunks next to every wrong answer. Fix retrieval before blaming generation.
- Test with questions phrased in the user’s words, not the document’s.
- Count the cross-references in your corpus.
- Ask whether the whole document fits in a context window. If it does, try that first.
- Move to a page-index approach only if accuracy matters more than latency and cost.
I wrote about a related trap, where a demo works and production does not, in the proof-of-concept to production gap. RAG pipelines are a common place for it to show up.