Real retrieval-augmented generation: can an AI find the right plan document in a library of six, and stick strictly to what it says?
88 questions answered against a 6-plan library of real Summary of Benefits and Coverage (SBC)
documents (PPO, HDHP, HMO x2, EPO, and a second PPO — from CMS, Auburn University, the State of
Illinois, Northwestern University, Cochise County AZ, and CalPERS). This is v3 of the project, and
the first version that's actually RAG: each document is chunked and embedded with Voyage AI, and
every question is answered by retrieving the top-5 most relevant chunks — not by pasting every
document into the prompt — then generating an answer strictly from those chunks. Every test case
is graded twice: did retrieval pull back the right document, and was the final answer correct?
Includes 14 adversarial/hallucination-check questions and 14 cross-plan comparisons. Model tested:
claude-sonnet-5. Full write-up on
GitHub.
TL;DR: 85.2% answer accuracy (75/88), 90.9% retrieval accuracy (80/88), and — notably — the
model never once stated a wrong fact with confidence: every failure was an accurate "not stated in
the document" triggered by incomplete retrieval, not a hallucination. The clearest weak spot is
cross-plan comparisons: retrieval found both plans' chunks 100% of the time, but the model only
answered correctly 60% of the time (6/10) — a case where good retrieval didn't guarantee good
reasoning. Zero hallucinations on all 8 adversarial/absent-fact questions.
Grading note: the raw eval run scored 76/88 (86.4%) on the automated substring grader. One
of those, question #44, is a false positive: the model answered "Not stated in the document" to a
yes/no question, and the grader's substring check matched because the word "not" contains "no."
The stats above are manually corrected for that one case (75/88, 85.2%) — the table below still
shows the raw per-question grading exactly as promptfoo scored it, including that one mismatch,
so nothing here is hidden.
Ask the benefits library
Ask a question about any of the 6 plans in this library (Plan A through Plan F — see the README
for what each one is). This is the real RAG pipeline running live: your question gets embedded,
compared against ~260 precomputed chunks from the 6 source documents, and the top 5 matching
chunks are retrieved and shown below the answer — so you can see exactly what the model was (and
wasn't) given to work with. No API key needed on your end, and nothing is stored.
Limited to a handful of requests per visitor per hour.