eval-harness

A formal evaluation framework for Claude Code sessions, implementing eval-driven development (EDD) principles. Eval-Driven Development treats evals as the "unit tests of AI development": Test if Claude can do something it couldn't before: Ensure changes don't break existing functionality: Deterministic checks using code: Use Claude to evaluate open-ended outputs: Flag for manual review: "At least one success in k attempts" "All k trials succeed" Write code to pass the defined evals. Creates eval definition file at .claude/evals/feature-name.md Runs current evals and reports status Generates full eval report Store evals in project: Use product evals when behavior quality cannot be captured by unit tests alone. Recommended thresholds: - Setting up eval-driven development (EDD) for AI-assisted workflows - Defining pass/fail criteria for Claude Code task completion - Measuring agent reliability with pass@k metrics - Creating regression test suites for prompt or agent changes - Benchmarking agent performance across model versions - Define expected behavior BEFORE implementation - Run evals continuously during development - Track regressions with each change - Use pass@k metrics for reliability measurement - pass@1: First attempt success rate - pass@3: Success within 3 attempts - Typical target: pass@3 > 90% - Higher bar for reliability - pass^3: 3 consecutive successes - Use for...