tavily-crawl

Multi-page website crawler with semantic filtering and markdown export. Crawl a website and extract content from multiple pages. Supports saving each page as a local markdown file. If tvly is not found on PATH, install it first: Do not skip this step or fall back to other tools. See tavily-cli for alternative install methods and auth options. For agentic use (feeding results to an LLM): Always use --instructions + --chunks-per-source . Returns only relevant chunks instead of full pages — prevents context explosion. For data collection (saving to files): Use --output-dir without --chunks-per-source to get full pages as markdown files.