agent-evaluation

Comprehensive evaluation framework for designing, building, and monitoring AI agent performance across coding, conversational, research, and computer-use agents. Based on Anthropic's "Demystifying evals for AI agents" Benchmarks : Grading Strategy : Key Metrics : Benchmarks : Grading Strategy (Multi-dimensional): Key Metrics : Grading Dimensions : Benchmarks : Grading Strategy : Solution : Add harder tasks, check for eval saturation (Step 7) Solution : Add more examples to rubric, use structured output, ensemble graders Solution : Use toon mode, parallelize, sample subset for PR checks Solution : Add production failure cases to eval suite, increase diversity - Covers three grader types (code-based, model-based, human) with trade-offs and best practices for each agent category - Provides an 8-step roadmap from initial task creation through production monitoring, including environment isolation, outcome-focused grading, and saturation detection - Includes benchmarks for major agent types: SWE-bench for coding, WebArena for computer use, τ2-Bench for conversational agents - Offers CI/CD integration patterns, A/B testing templates, and production sampling strategies for real-time quality monitoring - Designing evaluation systems for AI agents - Building benchmarks for coding, conversational, or research agents - Creating graders (code-based, model-based, human) - Implementing...