Eval Readiness Checklist | AI Evals Handbook

📊 Data & Test Sets

Golden set defined50-200 curated examples covering critical user journeys and known edge cases.

Edge cases catalogedAdversarial inputs, ambiguous queries, multi-language, and out-of-scope requests.

Data freshness policySchedule defined for refreshing test sets (e.g., monthly review, quarterly expansion).

Synthetic vs. real data ratio documentedBalance between synthetic test cases and real production samples is explicit.

Label quality reviewedInter-annotator agreement measured (≥0.8 Cohen's kappa) or single-expert reviewed.

Primary KPI definedSingle north-star metric aligned with business impact (e.g., consequence-weighted accuracy).

Support metrics identifiedLatency, cost-per-eval, retrieval precision, faithfulness, or domain-specific metrics.

Rubric documentedClear grading criteria for qualitative assessments. Each dimension has 3-5 score levels.

Baselines establishedCurrent performance benchmarked. "Good enough" thresholds agreed with stakeholders.

Failure taxonomy createdError categories defined (e.g., hallucination, refusal, latency spike, wrong persona).

Eval pipeline automatedTests run on every PR or nightly, not just ad-hoc manual checks.

LLM-as-Judge configuredJudge model selected, prompt tuned, calibrated against human ratings (≥85% agreement).

Regression tests in CIKnown failure cases locked as regression tests. New failures added within 48 hours.

Results stored & versionedEval results tied to model version, prompt version, and retrieval config.

Canary deployment plannedNew model versions tested on 5-10% traffic before full rollout.

Query distribution trackedMonitoring for shifts in topic, intent, or complexity of incoming queries.

Embedding drift alerts setCosine similarity or distribution distance triggers configured.

Human escalation rate monitoredRising escalation rates trigger investigation (>15% threshold recommended).

Confidence calibration reviewedModel confidence aligns with actual accuracy. Overconfident failures flagged.

Refresh cadence definedMonthly golden set review + quarterly deep audit scheduled.

Eval owner assignedOne person (PM or Eng Lead) owns the eval suite, rubric, and release gate.

Release gate definedClear pass/fail criteria: "Don't ship if consequence-weighted accuracy < X%."

Stakeholder reporting cadence setWeekly or sprint-level eval reports going to engineering and product leadership.

Incident response planWhat happens when evals reveal a critical failure in production? On-call defined.

Compliance alignment checkedEval requirements mapped to regulatory frameworks (SOC2, HIPAA, EU AI Act).

Confidence threshold for escalation setBelow what score does the system escalate to a human reviewer?

Human review sample rate definedPercentage of "passed" outputs still randomly reviewed (5-10% recommended).

Feedback loop implementedHuman corrections feed back into golden set and rubric updates.

Annotator guidelines documentedClear instructions for human reviewers, including examples for each score level.

All critical test cases passZero failures in high-consequence scenarios.

Performance regression checkedNew version ≥ previous version on all primary metrics.

Sign-off obtainedEval owner and product lead have reviewed and approved the eval report.