crawl4ai

Crawl4AI provides comprehensive web crawling and data extraction capabilities. This skill supports both CLI (recommended for quick tasks) and Python SDK (for programmatic control). Choose your interface: For SDK configuration details: SDK Guide - Configuration (lines 61-150) Both CLI and SDK use the same underlying configuration: Browser Configuration: Crawler Configuration: For complete parameters: CLI Config | SDK Config Every crawl returns: Crawl4AI excels at generating clean, well-formatted markdown: Filter configuration: For content filters: Content Processing (lines 2481-3101) No LLM required - fast, deterministic, cost-free. CLI: Schema format: For complex or irregular content: CLI: For extraction details: Extraction Strategies (lines 4522-5429) CLI: Crawler config: CLI (sequential): Python SDK (concurrent): For batch processing: arun_many() Reference (lines 1057-1224) CLI: For session management: Advanced Features (lines 5429-5940) CLI: For comprehensive API documentation, see Complete SDK Reference . - CLI ( crwl ) - Quick, scriptable commands: CLI Guide - Python SDK - Full programmatic control: SDK Guide - headless : Run with/without GUI - viewport_width/height : Browser dimensions - user_agent : Custom user agent - proxy_config : Proxy settings - page_timeout : Max page load time (ms) - wait_for : CSS selector or JS condition to wait for - cache_mode : bypass,...