Was the mistake in retrieval or generation?

A wrong final answer may result from missing evidence, poor ranking, or a model misusing correct evidence. End to end accuracy alone does not identify the cause. Inspect the retrieved set and citations for each failed question.

How is a trustworthy test set built?

Collect representative user questions, including paraphrases, rare terms, multi document questions, no answer cases, and permission boundaries. Mark the documents or passages that truly support each answer. Review ambiguous labels and update the set as queries and sources change. Multimodal retrieval and analytics shows why retrieval quality affects later analysis.

Which metrics matter?

Recall at a chosen candidate count checks whether required evidence was found. Ranking measures whether it appears near the top. Evidence coverage and citation accuracy show whether all important claims can be supported. Compare these with latency and cost, and manually inspect high impact or borderline examples.

FAQ

Is final answer accuracy enough?

No. It can hide retrieval errors or accidental correct guesses.

How large must the evaluation set be?

There is no universal size; cover key scenarios and expand with real traffic.

Can automatic evaluation replace human review?

It helps regression checks, while important edge cases still need human evidence review.