langsmith-evaluator

Build evaluation pipelines for LangSmith with LLM-as-Judge and custom code evaluators. Authentication is REQUIRED: either set the LANGSMITH_API_KEY environment variable, or pass the --api-key flag to CLI commands (preferred): IMPORTANT: Always check the environment variables or .env file for LANGSMITH_PROJECT before querying or interacting with LangSmith. This tells you which project contains the relevant traces and data. If the LangSmith project is not available, use your best judgement to identify the right one. Python Dependencies CLI Tool (for uploading evaluators) JavaScript Dependencies CRITICAL: Before writing ANY evaluator or extraction logic, you MUST: Output structures vary significantly by framework, agent type, and configuration. Never assume the shape - always verify first. Query LangSmith traces to when outputs don't contain needed data to understand how to extract from execution. Offline Evaluators (attached to datasets): Online Evaluators (attached to projects): CRITICAL - Return Format: CRITICAL - Local vs Uploaded Differences: NOTE: LLM-as-Judge upload is currently not supported by the CLI — only code evaluators are supported.