eval-harness

Formal evaluation framework for Claude Code sessions implementing eval-driven development principles. A formal evaluation framework for Claude Code sessions, implementing eval-driven development (EDD) principles. Eval-Driven Development treats evals as the "unit tests of AI development": Test if Claude can do something it couldn't before: Ensure changes don't break existing functionality: Deterministic checks using code: Use Claude to evaluate open-ended outputs: Flag for manual review: "At least one success in k attempts" "All k trials succeed" Write code to pass the defined evals. Creates eval definition file at .claude/evals/feature-name.md Runs current evals and reports status Generates full eval report Store evals in project: Use product evals when behavior quality cannot be captured by unit tests alone. Recommended thresholds: - Defines capability and regression evals with pass/fail criteria before implementation, treating evals as unit tests for AI-assisted workflows - Supports three grader types: code-based (deterministic checks via bash/grep), model-based (Claude-as-judge), and human review for manual adjudication - Tracks reliability with pass@k metrics (success within k attempts) and pass^k (all k trials succeed), with recommended thresholds of pass@3 ≥ 90% for capabilities and pass^3 = 100% for regressions - Integrates into Claude Code workflow with commands to...