web-scraping

Patterns for reliable, ethical web scraping with fallback strategies and anti-bot handling. Implement multiple extraction strategies with automatic fallback: Use browser developer tools to discover APIs: When you copy a cURL from dev tools, it includes many parameters. Strip it down by: Source: Leon Yin, "Finding Undocumented APIs," Inspect Element , 2023 Detect paywalls, anti-bot pages, and other failures: - Open developer tools (right-click → Inspect, or F12) - Go to the Network tab to monitor all requests - Filter by Fetch/XHR to show only API calls - Trigger the action you want to capture (search, scroll, click) - Analyze the response — usually JSON with key-value pairs - Copy as cURL (right-click the request) - Convert to code using curlconverter.com - Remove unnecessary cookies — test without them first - Keep authentication tokens if required - Identify the input parameters you can modify (like prefix for search terms) - Test parameter values — some expire, so periodically verify - Always check robots.txt before scraping - Respect rate limits and add delays between requests - Don't scrape personal data without consent - Cache responses to avoid redundant requests - Identify yourself with a descriptive User-Agent when appropriate - Stop if you receive explicit blocking signals