llm-evaluation

Systematic evaluation of LLM applications using automated metrics, human feedback, and statistical testing. Master comprehensive evaluation strategies for LLM applications, from automated metrics to human evaluation and A/B testing. Fast, repeatable, scalable evaluation using computed scores. Text Generation: Classification: Retrieval (RAG): Manual assessment for quality aspects difficult to automate. Dimensions: Use stronger LLMs to evaluate weaker model outputs. Approaches: - Covers three evaluation approaches: automated metrics (BLEU, ROUGE, BERTScore, accuracy, precision/recall), human evaluation across dimensions like accuracy and coherence, and LLM-as-Judge for pointwise, pairwise, and reference-based scoring - Includes implementations for text generation, classification, and retrieval (RAG) evaluation with ready-to-use metric functions and custom metric support - Provides A/B testing framework with statistical significance testing, effect size calculation, and regression detection to catch performance drops before deployment - Integrates with LangSmith for dataset management and experiment tracking, plus benchmarking utilities for tracking progress over time - Measuring LLM application performance systematically - Comparing different models or prompts - Detecting performance regressions before deployment - Validating improvements from prompt changes - Building...