long-context

Use Long Context techniques when you need to: Key Techniques : RoPE (Rotary Position Embeddings), YaRN, ALiBi (Attention with Linear Biases), Position Interpolation Papers : RoFormer (arXiv 2104.09864), YaRN (arXiv 2309.00071), ALiBi (arXiv 2108.12409), Position Interpolation (arXiv 2306.15595) How it works: Mathematical formulation: Advantages: Key innovation: Parameters: Performance: Core idea: Formula: Advantages: Technique: Formula: Results: - Process long documents (32k, 64k, 128k+ tokens) with transformer models - Extend context windows of pre-trained models (LLaMA, Mistral, etc.) - Implement efficient positional encodings (RoPE, ALiBi) - Train models with length extrapolation capabilities - Deploy models that handle variable-length inputs efficiently - Fine-tune existing models for longer contexts with minimal compute - Encodes absolute position via rotation matrix - Provides relative position dependency in attention - Enables length extrapolation - Decaying inter-token dependency with distance - Compatible with linear attention - Better extrapolation than absolute position encodings - NTK-aware interpolation (Neural Tangent Kernel) - Attention temperature scaling - Efficient context extension (10× less tokens vs baselines) - Extends LLaMA to 128k tokens - 2.5× less training steps than baselines - State-of-the-art context window extension - No positional embeddings...