flash-moe-inference

Skill by ara.so — Daily 2026 Skills collection. Flash-MoE is a pure C/Objective-C/Metal inference engine that runs Qwen3.5-397B-A17B (397B parameter Mixture-of-Experts) on a MacBook Pro with 48GB RAM at 4.4+ tokens/second. It streams 209GB of expert weights from NVMe SSD on demand — no Python, no ML frameworks, just C, Objective-C, and hand-tuned Metal shaders. The Makefile compiles infer.m , chat.m , main.m with Metal shader compilation for shaders.metal . The model has 60 transformer layers : The shaders.metal file contains hand-written kernels. Key kernels: The core innovation — loading only K=4 active experts per layer from SSD: Why pread() not mmap() : mmap incurs per-page fault overhead on cold data (~5x slower). Direct pread() with OS page cache achieves ~71% hit rate naturally. The recurrence update uses Accelerate BLAS — 64% faster than scalar: Key principle : On Apple Silicon, GPU DMA and SSD DMA share the same memory controller.