Evaluating RAG with context precision, recall and an LLM judge — Sasitha

Why retrieval needs its own score

A RAG answer can go wrong in two places: the retriever brings back the wrong passages, or the generator misuses the right ones. Scoring only the final answer blurs the two together, so a regression in your chunking strategy looks exactly like a regression in your prompt.

We wanted a signal that isolates the retriever, runs in CI and is cheap enough to run on every change.

The two metrics

Context precision asks: of the chunks we retrieved, how many were actually relevant — and were the relevant ones ranked near the top? Context recall asks the opposite: of the facts needed for the reference answer, how many appear somewhere in the retrieved context?

Low precisionNoise in the context window. Tune top-k, re-ranking and chunk size.

Low recallMissing evidence. Look at embeddings, query rewriting and coverage of the corpus.

Prompting the judge

Both metrics need a relevance decision per chunk or per claim, which is where the LLM judge comes in. We keep each judgement small and binary, so it is easy to audit:

for chunk in retrieved:
    verdict = judge(
        question=q,
        reference=ground_truth,
        context=chunk,
        ask="Is this context useful for the reference answer? yes/no",
    )
precision = ranked_precision(verdicts)

[Describe your judge model, temperature and how the golden dataset was built.]

What we learned

[Two or three concrete findings — where the judge disagreed with humans, which change moved the metrics most, and what you would do differently.]