data-scraper-agent

Build a production-ready, AI-powered data collection agent for any public data source. Runs on a schedule, enriches results with a free LLM, stores to a database, and improves over time. Stack: Python · Gemini Flash (free) · GitHub Actions (free) · Notion / Sheets / Supabase Every data scraper agent has three layers: Build agents to auto-fallback across Gemini models on quota exhaustion: Never call the LLM once per item. Always batch: Ask the user: Common examples to prompt: Generate this directory structure for the user: Template for any data source: HTML scraping pattern: RSS feed pattern: Integration with your storage layer: after each run, query your DB for items with positive/negative status and call save_feedback() with the extracted patterns. Before marking the agent complete: A complete working agent built with this exact architecture would scrape 4+ sources, batch Gemini calls, learn from Applied/Rejected decisions stored in Notion, and run 100% free on GitHub Actions. Follow Steps 1–9 above to build your own. - User wants to scrape or monitor any public website or API - User says "build a bot that checks...", "monitor X for me", "collect data from..." - User wants to track jobs, prices, news, repos, sports scores, events, listings - User asks how to automate data collection without paying for hosting - User wants an agent that gets smarter over time based on their...