seo-technical

As of 2025-2026, AI companies actively crawl the web to train models and power AI search. Managing these crawlers via robots.txt is a critical technical SEO consideration. Known AI crawlers: Key distinctions: Example, selective AI crawler blocking: Recommendation: Consider your AI visibility strategy before blocking. Being cited by AI systems drives brand awareness and referral traffic. Cross-reference the seo-geo skill for full AI visibility optimization. Google updated its JavaScript SEO documentation in December 2025 with critical clarifications: Best practice: Serve critical SEO elements (canonical, meta robots, structured data, title, meta description) in the initial server-rendered HTML rather than relying on JavaScript injection. If DataForSEO MCP tools are available, use on_page_instant_pages for real page analysis (status codes, page timing, broken links, on-page checks), on_page_lighthouse for Lighthouse audits (performance, accessibility, SEO scores), and domain_analytics_technologies_domain_technologies for technology stack detection. If Google API credentials are configured, use python scripts/pagespeed_check.py --json for real PSI + CrUX field data (replaces lab-only CWV estimates), python scripts/crux_history.py --json for 25-week CWV trends, and python scripts/gsc_inspect.py --json for real indexation status per URL.