grepai-chunking

This skill covers how GrepAI splits code files into chunks for embedding, and how to optimize chunking for your codebase. Chunking is the process of splitting source files into smaller segments for embedding: Embedding models have optimal input sizes: The target number of tokens per chunk. Tokens shared between adjacent chunks. Preserves context at boundaries. With size=512 and overlap=50: GrepAI uses approximate token counting: Example: ≈ 45 tokens Larger overlap = more chunks = larger index: Before changing settings, save a search result: After changing settings and re-indexing: GrepAI tries to split at logical boundaries: This means actual chunk sizes may vary slightly from the target. ❌ Problem: Search results are too fragmented ✅ Solution: Increase chunk size: ❌ Problem: Search results have too much irrelevant context ✅ Solution: Decrease chunk size: ❌ Problem: Results miss related code at function boundaries ✅ Solution: Increase overlap: ❌ Problem: Index is too large ✅ Solutions: Chunking status: - Optimizing search accuracy - Adjusting for code style (verbose vs. concise) - Troubleshooting search results - Understanding how indexing works - Too large chunks: Less precise search results - Too small chunks: Lost context, fragmented results - Just right: Good balance of precision and context - ~4 characters = 1 token (for English text) - Code varies based on identifiers...