agent-eval

A lightweight CLI tool for comparing coding agents head-to-head on reproducible tasks. Every "which coding agent is best?" comparison runs on vibes — this tool systematizes it. Note: Install agent-eval from its repository after reviewing the source. Define tasks declaratively. Each task specifies what to do, which files to touch, and how to judge success: Each agent run gets its own git worktree — no Docker required. This provides reproducibility isolation so agents cannot interfere with each other or corrupt the base repo. Create a tasks/ directory with YAML files, one per task: Execute agents against your tasks: Each run: Generate a comparison report: - Comparing coding agents (Claude Code, Aider, Codex, etc.) on your own codebase - Measuring agent performance before adopting a new tool or model - Running regression checks when an agent updates its model or tooling - Producing data-backed agent selection decisions for a team - Creates a fresh git worktree from the specified commit - Hands the prompt to the agent - Runs the judge criteria - Records pass/fail, cost, and time - Start with 3-5 tasks that represent your real workload, not toy examples - Run at least 3 trials per agent to capture variance — agents are non-deterministic - Pin the commit in your task YAML so results are reproducible across days/weeks - Include at least one deterministic judge (tests, build) per task...