web-scraping

Reliable web scraping with cascading fallbacks, anti-bot bypass, and poison pill detection. Patterns for reliable, ethical web scraping with fallback strategies and anti-bot handling. Implement multiple extraction strategies with automatic fallback: Use browser developer tools to discover APIs: When you copy a cURL from dev tools, it includes many parameters. Strip it down by: Source: Leon Yin, "Finding Undocumented APIs," Inspect Element , 2023 Detect paywalls, anti-bot pages, and other failures: - Implements a scraping cascade architecture with four strategies: trafilatura for fast article extraction, requests with rotating user agents, Playwright with stealth mode for JavaScript-heavy sites, and async Playwright for Jupyter notebooks - Includes poison pill detection to identify paywalls, CAPTCHAs, rate limits, Cloudflare blocks, and login walls using pattern matching and status code analysis - Covers reverse-engineering undocumented APIs through browser dev tools, with examples for autocomplete endpoints and parameter stripping - Provides ready-to-use patterns for YouTube (yt-dlp), Instagram (instaloader), and TikTok scraping, including metadata extraction, video downloads, and transcript retrieval - Emphasizes ethical scraping: robots.txt compliance, rate limiting, request delays per domain, and respectful User-Agent headers - Open developer tools (right-click → Inspect,...