When a RAG answer disappoints, teams often change chunk size, swap embeddings, or add prompt instructions. Without a baseline, those changes produce a few better-looking demos rather than evidence. A minimal evaluation separates observable stages: did retrieval find the supporting material, did the answer use only that material, does each citation support its claim, and does the system abstain when evidence is insufficient? This is a product-specific deterministic set, not a claim to a universal benchmark.

Start with forty valuable questions

Select representative tasks from real documents: direct facts, facts composed across passages, similar terminology, superseded versions, permission boundaries, and unanswerable questions. For every case, record the question, allowed document collection, one or more evidence spans, acceptable answer points, and whether abstention is required. Identify evidence by document id, version, and stable paragraph or character location rather than only a mutable URL.

Have domain experts review the set. Split development and holdout cases. Tune only on development and run holdout at decision points. If every failure immediately changes its gold answer, the evaluation gradually conforms to the current system. Record the source and reason for a new case and keep old runs replayable.

from dataclasses import dataclass

@dataclass(frozen=True)
class RagCase:
    case_id: str
    question: str
    evidence_ids: frozenset[str]
    required_points: tuple[str, ...]
    should_abstain: bool

Keep the schema small. Do not copy private source documents into ordinary CI logs. CI can exercise sanitized or synthetic passages, while a controlled environment runs the protected set.

Evaluate retrieval independently

For each question, save the top-k document and chunk ids, scores, filters, and retrieval configuration. Compute evidence recall at k: did a chunk containing a gold span appear? Also inspect irrelevant-chunk rate and the rank of the first supporting result. Testing only the final answer confuses “the generator ignored evidence” with “retrieval never found it.”

When comparing chunking, keep embedding model, index snapshot, and queries fixed. Raising k can improve recall while increasing distraction, context cost, and latency. Add hybrid retrieval, reranking, and metadata filters in distinct experiments, changing one variable at a time.

Authorization filters must run before retrieval results reach generation. A test that expects the model to ignore a forbidden document is testing an unsafe architecture. Include two users with different access to the same query and prove that retrieved ids differ correctly.

Verify citations precisely

Every factual answer claim should reference an evidence id. Deterministic checks establish that the id exists, was retrieved for this request, and is visible to the user. Domain review decides whether the cited span actually supports the claim. Referencing a generally relevant document is not the same as citing a supporting sentence; broad citations can conceal hallucination.

Citation precision and coverage are useful summaries, but retain the error list. One unsupported compliance number may be more damaging than several correctly cited minor details. For versioned sources, prefer the newest allowed version unless the question explicitly asks about history.

Keep quoted evidence within a controlled length and preserve its exact location. Rendering a citation should never expose adjacent private text that was not part of the answer.

Make abstention a first-class capability

Include cases where the corpus lacks an answer, the question has a false premise, sources conflict, or permission prevents access. The system should explain what evidence is missing rather than fill the gap from general knowledge. Evaluate correct abstention, unnecessary abstention, and answers that should have abstained. Choose thresholds according to product risk, not maximum answer rate.

The generation prompt can require evidence-only answers and citations, but a prompt is not enforcement. Add deterministic validation for money, compliance, identifiers, and operational instructions. Route uncertain high-risk cases to a person. If the product promises enterprise-grounded answers, the model must respect the corpus boundary even when it may know a plausible answer.

Use an LLM judge as supporting evidence

An LLM judge can apply a rubric for faithfulness, relevance, and completeness and help scale review. It remains biased and nondeterministic. Pin the judge model, version, prompt, sampling, and input format. Measure agreement on a human-labeled subset. Save a rationale for diagnosis, but do not treat eloquent reasoning as ground truth. Recalibrate after a model change.

Prefer deterministic signals where possible: presence of gold evidence, valid citation ids, schema validity, and exact values. Use the judge for semantic aspects that rules cannot capture. Report both rather than collapsing the pipeline into one average judge score.

Version the entire chain

Every run records corpus snapshot, chunking code, embedding, index, reranker, generator, prompt, judge, and evaluation commit. Save results by case so a failure can replay from document through answer. Segment metrics by source type, language, and question class; an average can hide a severe weakness in tables or non-English text.

RAG tuning begins with a diagnosable baseline: trustworthy gold spans, independent retrieval recall, claim-level citations, a formal abstention set, and a human-calibrated judge. Freeze versions, change one variable, and keep failures. “The new model is better” then becomes an answerable statement about which stage, which tasks, and what cost improved.