# I Turned On Ten RAG Retrieval Layers. Half of Them Did Not Earn Their Place. *The R in RAG, built end to end for Turkish tax law. Every layer explained and measured against real rulings, including the ones that lost.* --- I built the retrieval half of a RAG system for tax law. There are more decisions in that half than most write-ups admit: how you cut the documents up before you index them, how you search them once you have, and what you do with the results before the model ever sees them. Each of those is a layer you can switch on, and I switched them on one at a time to find out which ones were earning their place. Ten layers: - **Five earned it.** Clause-level chunking, a bigger embedding model, a bigger candidate pool, the law's own history in the corpus, and a date filter. - **Two only worked sometimes.** Hybrid search and cross-encoder reranking both won on one half of the questions and lost on the other. - **Two could not be told apart** from what they replaced: a more varied candidate pool, and rewriting the question first. - **One made things worse**, and it was the first thing everyone reaches for: an embedding model, losing to plain word counting. The two layers I was most confident about are in the bottom half of that list. The best embedding model I tested has 305 million parameters, and it beat one with 8 billion until the questions with a subject in them took most of that back. Every number here is measured, which is only possible because of where I ran it. The Turkish tax authority publishes its rulings, and every ruling names the articles of law it rests on, so the labels were written by the institution that applies the law rather than by me. They needed cleaning before they were usable, and what was wrong with them turned into a finding of its own. ## The job Here is one of those rulings, translated from Turkish. > **Question.** Our association is renovating a mosque. Are the goods and services we buy for it exempt from VAT? > > **Articles the ruling is based on.** Article 17 (exemptions). Most questions do not map to a single article. This one draws three: ```mermaid %%{init: {'themeVariables': {'fontSize': '11px'}}}%% flowchart LR Q["`**Question** We operate in a technology park under an exemption. Do maturity and exchange rate differences fall under the VAT exemption too?`"] --> R(["`**Retrieval**`"]) R --> A1["`**Article 20**, the tax base *The tax base is the consideration received in return for the supply.*`"] R --> A2["`**Article 24**, items included in the base *Maturity differences, price differences and exchange rate differences are included in the tax base.*`"] R --> A3["`**Provisional Article 2** *Work that spans more than one calendar year.*`"] ``` There are 882 of these rulings in the benchmark. The job being measured is the same in all of them: a question goes in, and the articles it turns on should come out. Sometimes that is one article and sometimes it is three, and nothing tells the system in advance which case it is looking at. When the wrong articles come back, none of the work downstream can recover, because the model will go on to write a confident answer on top of the wrong law. ## What is the R in RAG? Everything so far happens before the model writes a word. That part has a name, and it is the one letter of RAG that nobody breaks down. RAG stands for retrieval-augmented generation. The three words describe three very different amounts of work. - **R**, retrieval. Everything that decides which passages exist to be assembled in the first place. - **A**, augmented. More a promise than a stage. It is the assembly step: what goes into the context window, in what order, and under what instruction. - **G**, generation. A language model writes the answer. The part everyone can picture, and the part you control least. R sets the ceiling. A passage that never enters the context window cannot be recovered by anything downstream, and the model falls back on memory instead, which is what you built a RAG system to avoid. Ask one about a 2019 transaction and it will quote today's rule, fluently, with no idea the law moved. So A and G are named here and then left alone. Everything below is R. ## The rules I set first An ablation is only worth reading if the rules were fixed before the results came in. Mine: - One layer at a time. No bundled changes, because a bundle only ever tells you that the bundle worked. - Every layer is judged against whatever was winning when it arrived, not against the baseline. That bar keeps rising, and several layers that would have looked impressive against a keyword search died on this rule instead. - A larger number does not count on its own. Every comparison is bootstrapped over 10,000 resamples, and where a gain does not survive that I say so, including in the two cases where it cost me a result I wanted. - Nothing was fine-tuned. Every model is used as published. - Everything ran locally on one desktop GPU with 32 GB. No hosted APIs, so the costs in this post are seconds per question, not dollars. - Not every ruling can be scored. Sixty-nine of the 882 rest entirely on other tax laws that I never collected, so the articles they cite are not in the corpus and no method could ever return them. Those are dropped and 813 remain. - The metric is recall@10: of the articles a ruling cites, how many came back in the top ten. Three cited and two found scores two thirds. Order inside those ten does not count, and neither do the wrong answers sitting next to the right ones. ## Start with no model at all Before adding anything, I needed a number to add it to. The first setup was the smallest one that could answer the question at all. The VAT Law has 110 articles. Index them as they are, feed in the taxpayer's question exactly as it was written, and see how many of the articles the ruling cites come back in the top ten. ```mermaid flowchart LR C["`**Corpus** 110 articles, as written`"] --> S(["`**BM25** word overlap`"]) Q["`**Question** as the taxpayer wrote it`"] --> S S --> T["`**Top 10 articles**`"] ``` > **BM25, in one paragraph.** It scores documents by word overlap, weighted so that rare words count for more than common ones. No model, no embeddings, no GPU, no inference. It has been the default in information retrieval since the 1990s, which is why I started there. **recall@10 = 0.349** ## Let a model read the question The next thing to add is the layer most people have in mind when they say RAG. Three steps: - Embed all 110 articles once, with a pretrained model. - Embed the taxpayer's question, with the same model. - Rank the articles by how close they sit to the question. Same corpus as before, same questions, same top ten. ```mermaid flowchart LR C["`**Corpus** 110 articles`"] --> ME(["`**Embedding model**`"]) --> CV["`**Article vectors**`"] Q["`**Question** as the taxpayer wrote it`"] --> MQ(["`**Embedding model**`"]) --> QV["`**Question vector**`"] CV --> S(["`**Cosine similarity**`"]) QV --> S S --> T["`**Top 10 articles**`"] ``` > **Embedding search, in one paragraph.** The model turns a passage into a list of numbers, a point in space. Passages about the same thing land near each other. The question becomes a point too, so the search is simply this: which articles sit closest to it? Words no longer have to match. *Teslim* and *tesliminde* are two entries in a keyword index and one idea to a model. I took the two models a lot of people reach for first on multilingual retrieval, multilingual-e5-large and BGE-M3, both around 560 million parameters, both used exactly as published. This was supposed to be the upgrade. Beating a method that counts words and has no idea what any of them mean should not have been the hard part. Both lost, by roughly a quarter, to a keyword search from the 1990s. The number below is BGE-M3, the better of the two. e5-large came in a little lower. **recall@10 = 0.260** ## Cut the long articles at their own seams Before blaming the models, look at what they were given. - The exemptions article is 35,099 characters. Forty times the median. It covers charitable donations, cultural events, banking, education for disabled people, scrap metal. - Whole, it filled 32 gigabytes of VRAM. It never finished a batch. The larger models have no score above. - Window size is not the limit. e5-large reads 512 tokens. BGE-M3 reads sixteen times more. They landed within a point of each other. - One article gets one vector. It has to stand for scrap metal and for disabled education at once. That is what chunking is for. ```mermaid flowchart LR A["`**One article** 35,099 characters one vector`"] --> C(["`**Split at the numbering the article already has**`"]) C --> P1["`**4/g** scrap metal, plastic, rubber, glass, paper`"] C --> P2["`**4/s** equipment made for disabled people`"] C --> P3["`*and eighteen more*`"] ``` > **Chunking, in one paragraph.** Index pieces of a document rather than the whole thing. Each piece gets its own vector, so one subject is not averaged in with everything else that happens to sit under the same heading. Long articles already come divided into numbered clauses. Split them there, nowhere else. The corpus goes from 110 articles to 174 pieces. I ran all three methods again on the new corpus. All three got better, but not by the same amount. The keyword search moved a little. The two embedding models moved a lot. BGE-M3 had been a quarter behind the keyword search, and now it was even with it. There is a cost on the other side, at least for the keyword search. One procedural article is cited by 235 of the rulings. Before the split it came back 40 times. After, 21. The pieces of the longer articles had taken its place in the top ten. The best score after the split still belongs to the keyword search. **recall@10 = 0.377** ## Reach for a bigger embedding model A keyword search from the 1990s was still ahead. The obvious next move is to leave the two models everyone starts with and go through the shelf. Five more went in. - Qwen3-Embedding at 8 billion parameters. - The same family at 4 billion, and again at 0.6 billion. - A model fine tuned on Turkish search data. Home advantage, in theory. - Nomic v2, a mixture of experts, 305 million active parameters. | Model | Size | recall@10 | |---|---|---| | Keyword search | none | 0.377 | | Qwen3-Embedding | 4B | 0.333 | | e5-large | 560M | 0.337 | | Qwen3-Embedding | 0.6B | 0.344 | | Turkish fine tune | 560M | 0.366 | | BGE-M3 | 560M | 0.376 | | Qwen3-Embedding | 8B | 0.517 | | Nomic v2 | 305M active | 0.593 | Size predicted nothing. The 4 billion model sits at the bottom, under the 0.6 billion one, and the fine tune trained on Turkish did not win on Turkish. The smallest thing on the shelf won, and it won by a wide margin. First real jump in the project. **recall@10 = 0.593** ## 67% of the rulings have no subject Before adding another layer I stopped and read the data. A few hundred rulings, question next to cited article. What I found had nothing to do with models. Here is a real one. > **Question.** Which VAT rate applies to sales of caviar? > > **Article the ruling cites.** Article 28, titled Rate. ```mermaid flowchart LR Q["`**Question** Which VAT rate applies to sales of caviar?`"] --> S(["`**The subject** caviar`"]) Q --> W(["`**One generic word** rate`"]) S --> X["`Nothing. No article in the law names a single food.`"] W --> A["`**Article 28**, titled Rate *The rate is 10 percent for every taxable transaction. The President may raise it fourfold, or lower it to 1 percent.*`"] A --> G["`**The article the ruling cites**`"] ``` What the data actually looks like: - 54% of all citations in the benchmark go to three articles. - 67% of the rulings cite nothing but those three. - They are the rate article, the article that says what VAT applies to, and the one on withholding. - The rate for caviar is in none of them. It is in a government decree, in a list at the end, under food, row twelve. - The article the ruling cites only says who has the power to set rates. Legal hook, not answer. So finding those three is word spotting. Understanding the case does not help. - Almost every ruling that cites the rate article uses the word rate, *oran*. That is the article's title. - With the word, the winning model finds the article four times in five. - Without it, fewer than half. - On the 67%, the winning model scores 0.582 and word counting 0.275. - On the 33% with a real subject, 0.646 against 0.624. Draw the questions again and word counting wins a quarter of the time. On the questions that are really about something, the best embedding model I tested and a keyword search from the 1990s are tied. The sweep ranked them on the three procedural articles. So from here on every layer gets two numbers. - All 813 rulings, the way the benchmark comes. - The 270 that name something specific, with those three articles taken out of the answer key. I did not throw the other 543 away. They are real questions and a real system has to answer them. But one number covering both hides which layer did what. **recall@10 on all 813 = 0.593** **recall@10 on the 270 = 0.646** ## Search words and meaning at the same time The two methods fail in different ways. - Word counting finds the exact term and misses the paraphrase. - The embedding model finds the paraphrase and drifts off the exact term. There is no reason to pick one. ```mermaid flowchart LR Q["`**Question**`"] --> B(["`**Word search**`"]) --> BL["`**Ranking A**`"] Q --> D(["`**Embedding search**`"]) --> DL["`**Ranking B**`"] BL --> M(["`**Merge**`"]) DL --> M M --> T["`**Top 10**`"] ``` > **Merging two rankings, in one paragraph.** The two scores cannot be compared. One counts words, the other measures distance. Their positions can. So each article gets points from where it sits in each list, first place worth the most, and the points are added up. On the 270 questions with a real subject this is the best result so far. It beats word counting, and it beats the embedding model, and both gaps hold in every draw. Two methods that were tied with each other are not tied with their own combination. The raw number went the other way. It dropped, though not far enough for me to prove the drop is real. The reason is the last section. The merge gives both lists equal weight, so it pays off when the two are close and costs when one is far behind. - On the 270 the two are close, as the last section showed. Both lists are mostly right, and right about different articles. - On the 543 they are nowhere near each other: word counting scores 0.277 and the embedding model 0.581. Half the evidence now comes from a list that is wrong most of the time, and it pushes correct articles out of the top ten. Those 543 are 67% of the raw set, so they decide the raw number. Nothing about the method changed between the two columns. The questions did. **recall@10 on all 813 = 0.570** **recall@10 on the 270 = 0.705** ## Add a reranker Every method so far has looked at the question and the article separately. The question becomes words or a vector, the article became words or a vector months earlier, and the search compares the two. Nothing ever reads them side by side. A reranker does. It is too slow to run over a whole corpus, so it runs in second place: a cheap search picks 20 candidates, and the reranker reorders them. ```mermaid flowchart LR Q["`**Question**`"] --> S(["`**Search**`"]) --> C["`**20 candidates**`"] C --> R(["`**Reranker** reads question and article together`"]) --> T["`**Top 10**`"] ``` > **Cross-encoder, in one paragraph.** An embedding model has to describe an article before it knows what will be asked. A cross-encoder gets both at once and answers a narrower question: does this passage answer this question? It is slower by orders of magnitude, which is why it never sees more than a handful of candidates. I tried two: the multilingual reranker most people reach for, at 568 million parameters, and an 8 billion one from the Qwen family. - On the raw set both of them work, and the big one sets a new record. - On the 270 real questions both of them lose to the merge from the last section, and both losses hold in every draw. So this is the first layer in the post that is provably worse than the thing it replaced, and it is the one I would have shipped without checking. > **What it costs.** On the real questions the two rerankers score the same, so the bigger model bought nothing. It also takes about two seconds to read one question, and getting it down to two took a detour: the card is new enough that the fast attention kernels everyone builds against are not published for it yet, and the reranker crawled until I ran it inside a server image that ships its own. The merge that beat it takes no time at all, because both of its rankings were already in a cache. The models are not where this went wrong. A reranker can only choose from what it is handed, and it was handed 20 candidates from a single ranking. If the article was not in those 20 the score for that question is zero, however well the model reads. The merge had no such cutoff. Both of its lists covered the whole corpus, so an article the embedding model buried could still be pulled up by the word search. **recall@10 on all 813 = 0.642** **recall@10 on the 270 = 0.663** ## Make the candidate pool bigger If the reranker is limited by what it is handed, hand it more. Same pipeline, same model, 50 candidates instead of 20. - Both columns improve, and both gains hold in every draw. - The regression from the last section is gone. - On the real questions it lands at 0.707. The merge, two sections ago, was at 0.705. Redraw the questions ten thousand times and neither one pulls ahead. So the diagnosis was right and the fix works, and what it buys is a tie with something that runs off a cache in no time at all. > **What it costs.** The reranker scores every candidate on its own, so the bill grows with the pool. At 50 candidates that is about 2.3 seconds a question, roughly double the 20 candidate run, and half an hour to get through the benchmark once. **recall@10 on all 813 = 0.675** **recall@10 on the 270 = 0.707** ## Make the candidate pool more varied More of the same list is not the only way to hand the reranker more. The word search has its own opinion about every question, and nothing in the pipeline was using it any more. So: take the top 20 from each, put them together, rerank whatever comes out. About 32 candidates, from two sources instead of one. ```mermaid flowchart LR Q["`**Question**`"] --> B(["`**Word search**`"]) --> P["`**One pool** ~32 candidates`"] Q --> D(["`**Embedding search**`"]) --> P P --> R(["`**Reranker**`"]) --> T["`**Top 10**`"] ``` On the 270 real questions this is the highest number anywhere in the post: 0.724. On the raw set it comes in a little under the last section. Neither difference survives the draws. Against 50 candidates from one source, mixing two sources is worth nothing I can prove, in either column. Two sections ago it would have been a clear win. The pool of 20 was the thing to beat then, and this beats it comfortably. By the time I got around to testing it the bar had moved, and my own rule says the bar is what counts. > **What it costs.** Two searches instead of one, and a pool that changes size from question to question depending on how much the two lists overlap. About 3.7 seconds a question. **recall@10 on all 813 = 0.665** **recall@10 on the 270 = 0.724** ## Rewrite the question first Every question in the benchmark is a letter to the tax office, and it reads like one. > **As filed.** In your request form it is stated that you work in accounting and bookkeeping, that you purchased a four wheeled electric motor bicycle, brand volta ev1, customs code 87.03, for use in your business, and your opinion is requested on whether the tax paid on this vehicle may be deducted. > > **After the rewrite.** Whether VAT on the purchase of an electric motor bicycle can be deducted, and whether the purchase falls under an exemption. I had Qwen3-8B, running on the same machine, rewrite all 877 questions. The plan was to move them closer to the language the law is written in. That is not really what it did. It mostly deleted things. - The median question goes from 61 words to 31. - 836 of the 877 open with the same sentence of filing formula. It survives in two rewrites, along with the brand names and the customs codes. ```mermaid flowchart LR Q["`**Question**`"] --> RW(["`**LLM rewrite**`"]) --> R["`**Rewritten question**`"] Q --> S1(["`**Word and embedding search**`"]) --> P["`**One pool** ~47 candidates`"] R --> S2(["`**Word and embedding search**`"]) --> P P --> CE(["`**Reranker**`"]) --> T["`**Top 10**`"] Q -.->|only the original| CE ``` > **Where the rewrite is allowed to act.** It is a fourth search, not a replacement, and everything the four searches return lands in one pool. The reranker only ever reads the original question, so a model that invents something can widen the pool but cannot decide the answer. Search with the rewrite instead of the original and both methods improve, in every draw. Word counting goes from 0.378 to 0.463, the largest single gain in the project. The embedding model goes from 0.645 to 0.689 on the real questions. In the pipeline it is worth less. The numbers below are the highest pair in the post, and neither one holds against the leader it has to beat. By now there are two searches, a shared pool and a reranker in front of it, and whatever the rewrite fixes, something in there was already fixing most of it. > **What it costs.** Rewriting all 877 questions took about four minutes, once. The pool then grows from 32 candidates to 47, and the reranker takes about 5.1 seconds a question. **recall@10 on all 813 = 0.690** **recall@10 on the 270 = 0.737** ## Put the law's history in the corpus Every layer so far searched the same corpus: 174 pieces of text, the law as it reads today. That corpus carries an assumption I never wrote down. It assumes the right answer is in there somewhere. For a ruling from 2019 it often does not. The article was amended in 2021, and the amendment is the only version there. No reranker fixes that. The text is not there to be ranked. > **What changes with the numbers.** The corpus is 456 pieces instead of 174. The score now asks for the right version of the article, not just the right article. Only the 153 questions that touch an article which actually changed can be scored. These three sections are not on the same scale as the ten above them. Where the history comes from: - Nobody publishes it as data. It is 178 footnotes, scattered across two official copies of the law. - A local model turned them into structured edits and produced 165 of them. - 54 survived. Every edit had to be found in the text it claimed to change, and the whole chain had to replay forward onto the text I actually have. - Nothing was accepted because the model said so. The rest went to a rejects file. That leaves 88 versions of 8 articles. The exemptions article alone has 13. 153 questions cite an article with more than one version. In 129 of them the law moved after the ruling was written, so for 84% of them the current text is the wrong answer. **version-level recall@10, only the current text in the corpus = 0.089** Arithmetic, not a finding. The right version is not in the corpus, so the ceiling is 0.157. Now put every past version into the corpus and change nothing else. Same searches, same reranker, still no idea what date the question is about. **version-level recall@10, every past version in the corpus, no date filter = 0.535** Six times better, from data alone. My guess is that old versions read like their own decade, and a question from 2019 shares words with the text in force in 2019. A guess, not a measurement. ## Filter by the date the question is about So far the search has been picking between versions on wording alone, with no idea which one was law when the question was asked. It has that date. Every ruling carries the day it was written, and every version in the corpus carries the days it was in force. The rule is the obvious one: a question from 2019 may only see text that was law in 2019. ```mermaid flowchart LR Q["`**Question** asked in 2019`"] --> F(["`**Date filter**`"]) C["`**456 versions**`"] --> F F --> V["`**Only what was law in 2019**`"] --> S(["`**Word and embedding search**`"]) --> P["`**Pool**`"] --> CE(["`**Reranker**`"]) --> T["`**Top 10**`"] ``` I built it the other way round first. Search everything, then throw out the candidates whose dates do not fit. That scores 0.738 and looks fine until you ask what the pool was holding. - The exemptions article has 13 versions and they read almost the same. - A search that ranks by content puts several of them in its top 20, competing with each other for the same slots. - Throw out the wrong dates afterwards and most of the pool is gone. The right version may never have been in it. - Mask them first and every slot goes to a version that could actually apply. The pool grows from 24 candidates to 48 and costs nothing extra. On these 153 questions the move is worth 0.738 to 0.768, and the bootstrap will not call it. Score the same change on all 809 questions at article level and it is 0.609 to 0.670, certain. Small samples do not settle arguments. > **What this number does not prove.** The date filter and the answer key are built from the same version index. If the reconstruction dated an amendment wrong, both would be wrong together and the score would never notice. **version-level recall@10, every past version in the corpus, date filter on = 0.768** ## What survived Ten layers, one picture. This is what a question goes through now. ```mermaid sequenceDiagram participant Q as Question participant RW as LLM rewriter participant DF as Date filter participant IX as Word and embedding search participant CE as Reranker Q->>RW: everyday Turkish RW-->>Q: legal wording Q->>DF: the date the ruling was written DF-->>IX: only the versions in force on that date par original question Q->>IX: two searches and rewritten question Q->>IX: two searches end IX-->>CE: one pool, about 48 candidates CE-->>Q: top 10 article versions ``` Every layer, in the order I added it. A verdict of tied means the bootstrap could not separate it from the layer before it. | Layer | all 813 | the 270 | | |---|---|---|---| | Word counting, no model | 0.349 | — | baseline | | An embedding model | 0.260 | — | lost | | Clauses instead of whole articles | 0.377 | — | won | | A bigger embedding model | 0.593 | 0.646 | won | | Both searches merged | 0.570 | 0.705 | mixed | | A reranker | 0.642 | 0.663 | mixed | | A pool of 50 instead of 20 | 0.675 | 0.707 | won | | A pool built from two searches | 0.665 | 0.724 | tied | | A rewritten question as a third source | 0.690 | 0.737 | tied | The last three rows of the project are scored differently: the right version of the article, on the 153 questions where the law moved. | Setup | version-level recall@10 | | |---|---|---| | Only the current text in the corpus | 0.089 | ceiling 0.157 | | Every past version in the corpus | 0.535 | won | | Date filter applied before the ranking | 0.768 | won | What the ledger says, and I did not expect most of it: - The two largest gains in the post are not models. One is cutting long articles into clauses. The other is putting the law's own history into the corpus. - The layers I was most confident about, a bigger embedding model aside, either lost outright or could not be separated from what they replaced. - 5 of the 10 never proved themselves on the questions that have a subject. I kept them. A layer that cannot be told apart from the one before it is not the same thing as a layer that lost. - Every number above 0.7 was won on 270 questions. That is a small set, and the honest reading of the top of this table is that four methods are tied for first. ## What I did not try Ten layers is not the list of everything that could go here. It is the list of what I could measure one at a time against a fixed answer key. The rest are noted, not planned. Each of them would change the shape of the system rather than add a step to it, which is exactly why they did not belong in an ablation. - Fine-tuning the retriever. The training split has 728 rulings in it and no model in this post has ever seen them. - GraphRAG. Articles cite articles, rulings cite both, and none of that structure is in the index. Retrieval here treats every piece of text as an island. - Agentic retrieval. No index at all. A coding agent reads a repository by grepping for a word, opening whatever looks promising, following the references it finds in there, and stopping when it has enough. Point that at a corpus of law and every ranking decision in this post disappears. - Long context instead of retrieval. The article corpus is small enough to hand to a model whole, and I never measured what that scores. - Query routing. A question about a rate and a question about an exemption probably want different searches. Here they get the same one. - A third independent source in the pool. Variety paid twice. Nothing says two searches and a rewrite is where it stops. - Whatever else is out there. This list is what I happen to know about, and retrieval moves fast enough that it will be short of something by the time anyone reads it. Data, code and every result file are public. - The benchmark: [er3nhf/kdv-rag-benchmark](https://huggingface.co/datasets/er3nhf/kdv-rag-benchmark). The rulings, the corpus, the splits. - The code: [er3n/dokumevzuat](https://github.com/er3n/dokumevzuat). The scrapers, the parsers, the eval scripts, and the raw output of every run in this post.