moe-training

Use MoE Training when you need to: Notable MoE Models : Mixtral 8x7B (Mistral AI), DeepSeek-V3, Switch Transformers (Google), GLaM (Google), NLLB-MoE (Meta) Key Components: Top-1 Routing (Switch Transformer): Top-2 Routing (Mixtral): Expert Choice Routing: Auxiliary Loss: Router Z-Loss (Stability): - Train larger models with limited compute (5× cost reduction vs dense models) - Scale model capacity without proportional compute increase - Achieve better performance per compute budget than dense models - Specialize experts for different domains/tasks/languages - Reduce inference latency with sparse activation (only 13B/47B params active in Mixtral) - Implement SOTA models like Mixtral 8x7B, DeepSeek-V3, Switch Transformers - Experts : Multiple specialized FFN networks (typically 8-128) - Router/Gate : Learned network that selects which experts to use - Top-k Routing : Activate only k experts per token (k=1 or k=2) - Load Balancing : Ensure even expert utilization - DeepSpeed MoE Tutorial : - Mixtral Paper : - Switch Transformers : - HuggingFace MoE Guide : - NVIDIA MoE Blog : - references/architectures.md - MoE model architectures (Mixtral, Switch, DeepSeek-V3) - references/training.md - Advanced training techniques and optimization - references/inference.md - Production deployment and serving patterns