RAG: How LLMs Answer From Your Docs
Have you ever asked an LLM about your docs and got an answer it made up? RAG (Retrieval-Augmented Generation) fixes that: you cut your docs into chunks, paste the closest ones above your question, and the model answers from them. By the end you build one on your own files and know where it breaks.
Get new posts in your inbox. No spam, unsubscribe any time.
Have you ever asked an LLM about your docs and got an answer it made up? RAG (Retrieval-Augmented Generation) fixes that: you cut your docs into chunks, paste the closest ones above your question, and the model answers from them. By the end you build one on your own files and know where it breaks.
Isn't a smart model supposed to know my docs already?
Let's build "ask the docs", an assistant that answers from a team's engineering docs. Our test pile is 12 markdown files from the Vibki repo, 21,805 words. A model knows only its frozen training text, and these files were never in it, so it rarely says "I don't know". As our post on LLM tokens showed, it writes the likeliest next words, so you get the right style and the wrong facts, like a closed-book exam you never studied for.
Can you paste the whole folder into every prompt? Not for long: the folder only grows, a prompt has a size limit (the context window), and the model re-reads every word for every question.
What does RAG actually do between my question and the answer?
The smallest thing to hand the model is the right few pages. That is RAG, which turns the exam open-book: before "ask the docs" replies, we paste the pages that matter above the question. Lewis et al. named it in their 2020 paper "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks". Five steps:
- Cut your docs into chunks, one idea each, like index cards.
- Turn each chunk into an embedding, a list of numbers for what it is about, so similar meaning sits close, like cafés on a map.
- Turn the question into numbers the same way.
- Pick the k chunks closest to the question (top-k), like a librarian fetching the right ones.
- Paste them above the question with one rule: answer only from this text, else say "not found".
How does a folder of markdown turn into numbers you can compare?
Steps 1 to 3 are ordinary code. Save this snippet and the four after it, in order, as one file, rag_demo.py. Setup first:
💻Bash123pip install onnxruntime==1.23.2 tokenizers==0.22.2 numpy==2.4.2 google-genai==1.61.0 huggingface_hub==1.4.1 hf download sentence-transformers/all-MiniLM-L6-v2 onnx/model.onnx tokenizer.json --local-dir minilm mkdir docs # put your markdown files here
🐍Python12345678910111213141516import os, re, glob, math, collections import numpy as np import onnxruntime as ort from tokenizers import Tokenizer from google import genai HERE = os.path.dirname(os.path.abspath(__file__)) def chunks_of(path): text = open(path, encoding="utf-8").read() out = [] for sec in re.split(r"\n(?=#{1,3} )", text): # one piece per markdown section words = sec.split() for i in range(0, len(words), 120): # long sections: cut at 120 words piece = " ".join(words[i:i + 120]) ...
chunks_of is step 1: one chunk per markdown section. embed is steps 2 and 3: the model turns each token, a small piece of a word, into 384 numbers, and we average them into one point per text. The all-MiniLM-L6-v2 model card on Hugging Face says text longer than 256 word pieces is cut off. That is why chunks stay short.
📄Text112 files -> 302 chunks -> vectors (302, 384)
Now finding the nearest of 302 points is arithmetic.
Which pages does the model get to read, and does the answer change?
Which three of the 302 chunks does a question get to read? We embed the question, take the three nearest by multiplying the chunk numbers with the question's, and paste them under this rule:
📄Text12345678910111213Answer only from the text below. If the answer is not in the text, reply exactly: not found. TEXT: [CLAUDE.md] ## Deploy - Pushes to `main` run `.github/workflows/deploy-cloudflare.yml`: two independent jobs. ... [notes_supabase-local-vs-production-strategy.md] ### FORBIDDEN Operations (WRITE) **NEVER use these tools directly - use ... [notes_supabase-local-vs-production-strategy.md] ## Prerequisites - Supabase MCP server connected to production - ... QUESTION: When does the deploy refuse to run db push, and what do I do about it?
🐍Python12345678910111213141516client = genai.Client(api_key=os.environ["GEMINI_API_KEY"]) def ask(prompt): return client.models.generate_content(model="gemini-3-flash-preview", contents=prompt).text.strip() def top_k(q, k=3): return [chunks[i] for i in np.argsort(-(vecs @ embed([q])[0]))[:k]] def rag_prompt(q, k=3, pick=top_k): ctx = "\n\n".join(f"[{f}]\n{t}" for f, t in pick(q, k)) return ("Answer only from the text below. If the answer is not in the text, reply exactly: not found.\n\n" f"TEXT:\n{ctx}\n\nQUESTION: {q}") Q = "When does the deploy refuse to run db push, and what do I do about it?" print("\nQ:", Q) ...
📄Text123456789Q: When does the deploy refuse to run db push, and what do I do about it? --- no docs --- In the context of Prisma (the most common tool using the `db push` command), the command is designed to synchronize your database schema with your `sc ... --- retrieved --- [CLAUDE.md] ## Deploy - Pushes to `main` run `.github/workflows/deploy-cloudflare.... [notes_supabase-local-vs-production-strategy.md] ### FORBIDDEN Operations (WRITE) **NEVER use these tools directly - us... [notes_supabase-local-vs-production-strategy.md] ## Prerequisites - Supabase MCP server connected to production - Local... --- with docs --- `db push` is refused when the remote has migration versions missing from `supabase/migrations/` (diverged history). You must reconcile with `supabase migration repair` first.
Without docs, the model answers about Prisma, a different tool. With the three chunks on its exam desk it answers from CLAUDE.md and copies its line "db push is refused when the remote has migration versions missing", because those words sit right above the question.
When does RAG still get it wrong?
It worked for that question, but the librarian can still miss: RAG ranks chunks by closeness of meaning, so the right chunk can land low. Thakur et al. say in their BEIR benchmark paper that BM25, a classic keyword ranking, is a robust baseline, while dense (embedding) retrieval models like ours often underperform other approaches.
Ask which secrets the deploy-supabase job needs. The chunk listing them ranks 7th of 302, so the model never sees it and says not found., where without docs it invented SUPABASE_PROJECT_ID.
🐍Python123456789def rank_of(q, needle): # where does the chunk holding the answer land, by meaning alone? order = np.argsort(-(vecs @ embed([q])[0])) return 1 + next(r for r, i in enumerate(order) if needle in chunks[i][1]) Q1 = "Which secrets does the deploy-supabase job need, and which one is only for migrations?" print("\nQ1:", Q1) print("no docs named:", sorted(set(re.findall(r"SUPABASE_[A-Z_]+", ask(Q1))))) print("answer chunk rank by meaning alone:", rank_of(Q1, "SUPABASE_DB_PASSWORD"), "of", len(chunks)) print("k=3:", ask(rag_prompt(Q1, 3)))
📄Text1234Q1: Which secrets does the deploy-supabase job need, and which one is only for migrations? no docs named: ['SUPABASE_ACCESS_TOKEN', 'SUPABASE_DB_PASSWORD', 'SUPABASE_PROJECT_ID'] answer chunk rank by meaning alone: 7 of 302 k=3: not found.
Fix 1 reads more pages: k goes from 3 to 8. k=5 still misses, k=8 answers, and every question pastes eight chunks.
🐍Python12print("k=5:", ask(rag_prompt(Q1, 5))) print("k=8:", ask(rag_prompt(Q1, 8)))
📄Text12k=5: not found. k=8: The `deploy-supabase` job needs `SUPABASE_ACCESS_TOKEN`, `SUPABASE_PROJECT_REF`, and `SUPABASE_DB_PASSWORD`. The `SUPABASE_DB_PASSWORD` secret is only for migrations.
Fix 2, hybrid_top_k, ranks twice, by meaning and by shared rare words (a word found in few chunks counts more), then merges the two ranks and keeps three. It answers at k=3 and rescues a question asked in different words.
🐍Python12345678910111213141516def words(t): return re.findall(r"[a-z0-9_\-]+", t.lower()) chunk_words = [set(words(t)) for _, t in chunks] df = collections.Counter(w for d in chunk_words for w in d) # in how many chunks does each word occur? def hybrid_top_k(q, k=3): rare = [sum(math.log(len(chunks) / df[w]) for w in set(words(q)) if w in d) for d in chunk_words] by_meaning = np.argsort(np.argsort(-(vecs @ embed([q])[0]))) # each chunk's rank by meaning by_words = np.argsort(np.argsort(-np.array(rare))) # each chunk's rank by shared rare words merged = 1 / (60 + by_meaning) + 1 / (60 + by_words) # reciprocal rank fusion return [chunks[i] for i in np.argsort(-merged)[:k]] print("hybrid k=3:", ask(rag_prompt(Q1, 3, hybrid_top_k))) Q2 = "The pipeline stopped pushing database changes, the server has versions my folder lacks. What now?" print("\nQ2 rank by meaning alone:", rank_of(Q2, "supabase migration repair")) print("Q2 k=3:", ask(rag_prompt(Q2, 3))) ...
📄Text12345hybrid k=3: The `deploy-supabase` job needs the following secrets: `SUPABASE_ACCESS_TOKEN`, `SUPABASE_PROJECT_REF`, and `SUPABASE_DB_PASSWORD`. The `SUPABASE_DB_PASSWORD` secret is only for migrations. Q2 rank by meaning alone: 5 Q2 k=3: not found. Q2 hybrid k=3: reconcile with `supabase migration repair` first.
Conclusion
So, does a smart model already know your docs? No: our "ask the docs" assistant talked about Prisma until we chunked the files, embedded them and pasted the nearest chunks above the question, and then it answered from CLAUDE.md. The pick of pages fails quietly, like a librarian fetching the wrong ones. Tomorrow, run 10 of your own files on three questions you can answer and check the pasted chunks first. Next time: how a chat model knows a sentence is an instruction.
Sources
- Lewis et al., Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks, 2020
- Thakur et al., BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models, 2021
- Hugging Face, sentence-transformers/all-MiniLM-L6-v2 model card
Enjoyed this post?
Get the next one in your inbox. No spam, unsubscribe any time.
Comments
0 comments
