告别2022年的落后分块：基于多尺度索引与倒数排名融合的高效RAG检索策略

Hi everybody. Thank you for coming today. Welcome to a talk about retrieval. My name is Yuval. I work at AI21, which is essentially an AI research lab. And today I want to talk to you about something that most people don't want to talk about, which is chunking. And I hope to convince you by the end that chunking isn't dead and there is something to do with that.

And really, if you are at X, LinkedIn, wherever, you've probably seen that RAG is dead, right? I think people also killed MCP lately, and RAG is dead again. Long live agentic retrieval, agentic search. And there comes a time where you have to ask yourself: how many times can RAG die? And even when someone says, "Well, RAG isn't dead," like Jerry, the CEO of LlamaIndex, they still have to kill something, and apparently this something is chunking—like "don't invest in it, don't do it."

And the reason that people said that chunking is dead is because everybody's using agentic search now, right? You have gs, you have ls, you have finds. All of these are great, but these are still not enough if you have a lot of data and you have various amounts of queries.

And I think that the main reason that a lot of people don't like to talk about chunking is because it's not the fun part, right? In every RAG system, we have two stages. The first stage is the boring one that you do in the beginning: you have a lot of data, you have to preprocess it, you have to decide on the chunk size, and then you have to store everything in a vector DB. The other part is the retrieval part—essentially the one that happens per query. This is much easier to do and much easier to optimize; you can use all your queries and then play with top-k, hybrid search, and those kinds of things. It's much more fun to do retrieval tuning.

So I will claim that if something has to be dead, then it's probably retrieval tuning. Agentic search probably killed that. But still, agentic search—even if we accept the fact that it killed retrieval tuning—is still not good enough when you have scale and a lot of data. It costs a lot of money and token-maxing. And underneath, if the data itself is not ordered in the right way in your folders and directories, you still get something inefficient.

Let's think of a timely example: the FIFA World Cup. Imagine we have a dataset containing all FIFA World Cups, where every directory is, say, the '98 one, the 2002 one, and so on. If your query asks which team won the most World Cups, you can't just go to a folder and ask that. You have to go to every folder, see who won, and aggregate this together, which is very inefficient. (The answer, by the way, is Brazil.)

So retrieval didn't actually die; it got demoted into plumbing. Everybody who worked on any RAG system knows the feeling: day one, week one, or month one, you pick some sort of chunk size (let's say 512), put some overlap (10–20%), index everything, and forget all about it. We talk a lot about fixed chunking strategies where if your chunk is too big, you get the whole picture (which is nice) but lose a lot of nuance and meaningful embeddings; whereas if your chunks are too small, the big picture is lost. Chunking is essentially a lossy compression. No matter what we do, we're losing something.

There is no single right chunk size. Many who work on data say, "No, we optimized our system on our corpus and it works well." We thought so too. But unlike model benchmarks where you can easily overfit, in RAG you cannot optimize a single chunk size per dataset—it is query-dependent.

How can I be so sure and claim such a thing? Because we ran experiments. Instead of asking what the best chunk size per dataset is, we took datasets and duplicated them several times—in this case, six times—where each instance had a different chunk size (e.g., 200, 1000, etc.).

We did this across several datasets: QMSum (meeting transcripts), NarrativeQA (question answering on novels), and the Seinfeld dataset (trivia questions on Seinfeld transcripts, an in-house trolling dataset we published).

First, we looked at how queries different in nature perform across chunk sizes. For example, in Seinfeld:

1. "What is the name of Jerry's favorite cereal?" — A very focused, specific question where the answer is contained. A smaller chunk size (100 tokens) achieves Rank 1 versus Rank >50 on larger chunks.
2. "Who does Jerry describe as his nemesis and pure evil?" — (The answer is Newman.) If you look at the transcript, it's not found easily; if you use a small chunk size, you will not get the answer.

We then asked: what if we had an Oracle (or a genie) that could tell us the best chunk size for every query? This is the Oracle experiment—to see the potential. In the graph (Recall@K vs K), the blue lines represent individual fixed chunk sizes, while the orange line represents the Oracle picking the best chunk size per query.

Across several datasets, the blue lines intersect, meaning no single chunk size dominates. More importantly, the gap between the Oracle line and all fixed blue lines is huge—around 20% to 40% improvement just from chunking strategy. That gap is the cost of picking an arbitrary number like 512 or 1000.

The core challenge is an information asymmetry problem: at indexing time, when you control the chunk size, you don't know what the queries will be; at retrieval time, when you have the query, you cannot adjust the chunk size because it is already fixed.

Prior works like Anthropic's Contextual Retrieval enrich chunks to improve the latent space, but they still stay within the fixed chunk size paradigm. We took a different approach: why commit to one chunk size when you can commit to several?

We call this Multiscale Indexing. At indexing time, we duplicate the database and chunk it with $N$ different window sizes. At retrieval time, we run $N$ parallel retrieval calls per query.

How do we combine them? We cannot use the Oracle in production. Since chunks of different sizes are not directly comparable by similarity scores, we adopt a document-level retrieval approach: when a chunk matches, we retrieve the entire parent document. Now we have $N$ different document rankings, and retrieval becomes a voting process. We aggregate these rankings using Reciprocal Rank Fusion (RRF)—a simple, lightweight formula that requires no dedicated model training.

The results across QMSum, NarrativeQA, Seinfeld, and FinanceBench demonstrate that this method consistently matches or beats the best fixed chunk size, improving recall across Recall@1 to Recall@10 by 20% to 40% (and 10% to 40% on MTEB benchmarks).

There are trade-offs: it is not a free lunch. It incurs an additional storage cost of 2x to 5x ($O(1)$ constant overhead). However, latency is virtually unaffected because retrieval calls run in parallel and RRF computation is instantaneous.

Future work includes determining the optimal number and specific values of chunk sizes dynamically rather than arbitrarily choosing (50, 100, 200...), as well as exploring ranking fusion methods beyond RRF. Agents didn't kill retrieval; the issue is that systems are still stuck with 2022 infrastructure. Simple multiscale strategies can instantly yield substantial gains.