RAG over a book where every answer sentence is checked against its evidence.
What I didBuilt end to end: retrieval, answers, the answer checker and the evaluations.
Every number on this page links to the file it came from, pinned to bookmind @ bb8f37f.
Headline results
The short version
The problem
Asking a chatbot about a book gives you fluent answers you cannot trust. I wanted answers grounded in cited passages, and a way to measure how often the system makes things up.
What I built
Retrieval first, from scratch. An EPUB becomes 326 citeable chunks, BM25 is written by hand, embeddings come from Ollama, and the two rankings are merged by a fusion step that fits in ten lines. On top of that sit answers that cite chapters, a checker that matches each answer sentence to its evidence and labels it supported, weak or unsupported, and a quiz mode that writes study questions from a chapter’s own sentences. It runs on a free local model or on Claude.
How I tested it
A test set of 42 questions: 38 that the book answers, each labelled with the chapter that answers it, and 4 off-topic ones the system must refuse. I wrote 26 of the questions after the first version and committed them before measuring the new retrievers, so they could not be tuned to the new code.
What I would do next
Grow the set past 100 questions with passage-level labels, so the hybrid gain can be confirmed or ruled out. Then run the answer-quality metrics (citation accuracy, refusal correctness, hallucination against a closed-book baseline) that the README still marks as needing a model run.
How it works
Pick a request and watch it go through the system. The red ones are the requests I use to attack it. They end where the system is supposed to stop them.
Passages are retrieved, the model answers with chapter citations, and every answer sentence is labelled against its evidence.
- Question
- Hybrid search
- Grounded answer
- Sentence check
- Cited answer
Two flows share one index. A question is retrieved, answered and checked. Two of the three traces end in a refusal or a flagged sentence on purpose.
Test results
9 claims, each with the bar I held it to and what I measured. 1 failed and 3 are unclear. They stay in, because a list where everything passes tells you nothing.
| Result | Claim | Expected | What I measured | Source |
|---|---|---|---|---|
| PASS | Hybrid retrieval finds the right chapter | Recall@5 above either retriever alone | Recall@5 95%, against 92% for BM25 and 89% for embeddings (38 questions) | README.md ↗ @ bb8f37f |
| PASS | Hybrid never loses a question BM25 found | n/a | it found one more, ranked the right chapter higher on 13 questions and lower on 7 | README.md ↗ @ bb8f37f |
| UNCLEAR | Hybrid ranks better than BM25 by more than noiseHybrid is the default because it was best or joint best on every slice and never worse on recall, not because the gain is proven. 38 questions cannot rule out luck. | an MRR gain whose interval excludes zero | MRR +0.093, 95% interval -0.03 to +0.22, which includes zero | README.md ↗ @ bb8f37f |
| PASS | The sentence checker never gives a green check to a bad sentenceSimilarity is not meaning, so a sentence that reverses a passage can still look close, and the calibration set is small and from one model. | n/a | 0 of 29 wrong-evidence or invented sentences were marked supported | README.md ↗ @ bb8f37f |
| PASS | Every citation points at a real chapterIt checks that a cited chapter exists. Whether the passage supports the sentence is the sentence checker's job. | n/a | checked deterministically, even without a model or an API key | README.md ↗ @ bb8f37f |
| PASS | It runs without a paid APIIt works, slowly. The 0.5b model answered in 17 s but copied the passage instead of answering. | n/a | qwen2.5:7b on a laptop CPU takes 27 s to 2 min per cited answer | README.md ↗ @ bb8f37f |
| FAIL | Retrieval is judged at passage level | n/a | chapter level only; a hit means a passage from the right chapter | README.md ↗ @ bb8f37f |
| UNCLEAR | Off-topic questions are refused | all 4 off-topic questions refused | one off-topic question was refused in a manual check; the full measurement still needs a model run | README.md ↗ @ bb8f37f |
| UNCLEAR | RAG hallucinates less than closed-book | n/a | not yet measured at scale | README.md ↗ @ bb8f37f |
The numbers
Each headline is the conclusion. Open “View the data” under a chart for the exact values.
Hybrid finds the right chapter more often than either retriever alone.
Recall@5 by retriever
38 answerable questions. A hit is a passage from the right chapter, not the exact passage.
Hybrid is best or joint best on every slice, but the gain over BM25 is not proven.
MRR by question slice
The MRR difference over BM25 is +0.093 with a 95% interval of -0.03 to +0.22, which includes zero.
No unsupported sentence got a green check.
What the sentence checker said
Thresholds were set on 8 real qwen2.5:7b answers labelled by hand. The calibration set is small and from one model.
What went wrong
Concept maps were built, then deleted
- How it surfaced
- Word co-occurrence graphs, and graphs from LLM-extracted triples over a truncated prompt, gave readers nothing they could act on.
- What I did
- I removed the feature. A shipped graph nobody can use costs more than the feature is worth.
The first test set could not tell two retrievers apart
- How it surfaced
- With only 12 questions, a single question moved Recall@5 by 8 points, so no comparison between retrievers meant anything.
- What I did
- I grew the set to 38 answerable and 4 off-topic questions, written and committed before dense or hybrid retrieval was measured so they could not be tuned to the new code.
Each retriever failed in a different way
- How it surfaced
- BM25 matches words, so it misses a passage that says the same thing differently. Embeddings match meaning but drift toward passages that are merely on the same topic, and they are worse at exact terms.
- What I did
- Run both and merge the rankings with Reciprocal Rank Fusion, which uses only ranks, so the two score scales never need to be reconciled.
The smallest local model copied instead of answering
- How it surfaced
- qwen2.5:0.5b answered in 17 seconds, but it pasted the retrieved passage back instead of answering the question.
- What I did
- The default stays on the 7b model. If no model is reachable, an extractive fallback returns the top passage with its citation and says so.
Choices I would defend
Write the retriever by hand
I wanted every part understood, not magic. BM25 is mine, the fusion fits in ten lines, and one file is the only code that talks to a model.
Fuse ranks, not scores
Reciprocal Rank Fusion gives each passage 1/(60 + rank) per list. BM25's unbounded scores and cosine similarity never share a scale. The constant 60 is the standard value, not tuned here.
Check answers with embeddings, not another LLM
Matching each answer sentence to its evidence is cheap, deterministic and cannot hallucinate. The thresholds were set on 8 real answers (21 sentences) labelled by hand.
Print the limits next to the headline
The README shows the interval that includes zero and the chapter-level labels right beside the 95%, so the number cannot be read without its caveat.