llm-evaluation

Master comprehensive evaluation strategies for LLM applications, from automated metrics to human evaluation and A/B testing. Fast, repeatable, scalable evaluation using computed scores. Text Generation: Classification: Retrieval (RAG): Manual assessment for quality aspects difficult to automate. Dimensions: Use stronger LLMs to evaluate weaker model outputs. Approaches: - Measuring LLM application performance systematically - Comparing different models or prompts - Detecting performance regressions before deployment - Validating improvements from prompt changes - Building confidence in production systems - Establishing baselines and tracking progress over time - Debugging unexpected model behavior - BLEU : N-gram overlap (translation) - ROUGE : Recall-oriented (summarization) - METEOR : Semantic similarity - BERTScore : Embedding-based similarity - Perplexity : Language model confidence - Accuracy : Percentage correct - Precision/Recall/F1 : Class-specific performance - Confusion Matrix : Error patterns - AUC-ROC : Ranking quality - MRR : Mean Reciprocal Rank - NDCG : Normalized Discounted Cumulative Gain - Precision@K : Relevant in top K - Recall@K : Coverage in top K - Accuracy : Factual correctness - Coherence : Logical flow - Relevance : Answers the question - Fluency : Natural language quality - Safety : No harmful content - Helpfulness : Useful to the user - Pointwise :...