Skip to main content
Back to registry

web-scraping-automation

aaaaqwq/claude-code-skills

Installs194
Install command
npx skills add https://github.com/aaaaqwq/claude-code-skills --skill web-scraping-automation
Security audits
Gen Agent Trust HubPASS
SocketFAIL
SnykWARN
About this skill
此技能专门用于自动化网站数据爬取和 API 接口调用,包括: 目标分析 : 方案设计 : 脚本开发 : 测试优化 : - 分析和爬取网站结构 - 调用和测试 REST/GraphQL API - 创建自动化爬虫脚本 - 数据解析和清洗 - 处理反爬虫机制 - 定时任务和数据存储 - "爬取这个网站的产品信息" - "帮我调用这个 API 并解析返回数据" - "创建一个脚本定时抓取新闻" - "分析这个网站的 API 接口文档" - "绕过这个网站的反爬虫限制" - requests :HTTP 请求库 - BeautifulSoup4 :HTML 解析 - Scrapy :专业爬虫框架 - Selenium :浏览器自动化 - Playwright :现代浏览器自动化 - axios :HTTP 客户端 - cheerio :服务端 jQuery - puppeteer :Chrome 自动化 - node-fetch :Fetch API - 目标分析 : - 检查网站结构和数据位置 - 分析 API 接口和认证方式 - 评估反爬虫机制 - 方案设计 : - 选择合适的技术栈 - 设计数据提取策略 - 规划错误处理和重试机制 - 脚本开发 : - 编写爬虫代码 - 实现数据解析逻辑 - 添加日志和监控 - 测试优化 : - 验证数据准确性 - 优化性能和稳定性 - 处理边界情况 - 遵守 robots.txt 规则 - 设置合理的请求间隔 - 使用 User-Agent 和请求头 - 实现错误重试机制 - 数据去重和验证 - 使用代理池(如需要) - 保存原始数据和日志 - 请求头伪装 :模拟真实浏览器 - 代理轮换 :使用代理池 - 验证码处理 :OCR 或第三方服务 - Cookie 管理 :维护会话状态 - 请求频率控制 :避免触发限制 - JavaScript 渲染 :使用 Selenium/Playwright - CSV/Excel :简单数据导出 - JSON :结构化数据存储 - 数据库 :MySQL、PostgreSQL、MongoDB - 云存储 :S3、OSS - 数据仓库 :用于大规模数据分析

Source description provided by the upstream skill listing. Community reviews and install context appear in the sections below.

Community Reviews

Latest reviews

Sign in to review

No community reviews yet. Be the first to review.

Browse this skill in context
FAQ
What does web-scraping-automation do?

web-scraping-automation is listed in SkillJury, but the source summary is still sparse.

Is web-scraping-automation good?

web-scraping-automation does not have approved reviews yet, so SkillJury cannot publish a community verdict.

What agent does web-scraping-automation work with?

web-scraping-automation currently lists compatibility with codex, gemini-cli, opencode, cursor, kimi-cli, github-copilot.

What are alternatives to web-scraping-automation?

Skills in the same category include telegram-bot-builder, flutter-app-size, sharp-edges, iterative-retrieval.

How do I install web-scraping-automation?

npx skills add https://github.com/aaaaqwq/claude-code-skills --skill web-scraping-automation

Related skills

More from aaaaqwq/claude-code-skills

Related skills

Alternatives in Software Engineering