Web Crawling
Crawling whole sites and sitemaps. As of 2026-11-21, 231 indexed products offer it (56 with full profiles), in 9 forms: 218 MCP servers, 3 Agent skills, 2 Infrastructure, 2 Plugins. 160 were published in the last 90 days, against 37 in the 90 days before. Firecrawl has the most GitHub stars, 42% of the total across 140 products with their own repository.
CapabilitySegment label: Crowded, outpacing the catalogue DER
Available as
All in the directoryProfiled products grouped by product form. Within each form: own-repository products by GitHub stars, then products in a subfolder of a larger repository (they show that repository's stars), up to 10 each.
MCP servers 218 indexed · 45 profiled
Protocol-level reach: 36 of the 45 active platforms AgentSidekick tracks document support for MCP. DER Which platforms
| # | Product | Publisher | Capabilities | GitHub stars | Last push | Save and compare |
|---|---|---|---|---|---|---|
| 1 | Scrapling MCP ServerWeb scraping with stealth HTTP, real browsers, and Cloudflare bypass. CSS selectors supported. | D4Vinci | 105k | 4d ago | ||
| 2 | Douyin_TikTok_Download_API🚀 自托管的抖音、TikTok、B 站数据采集与无水印视频下载 API,支持 AI agent 通过 MCP 调用与离线语音转文字,Docker 一键部署。| Self-hosted TikTok, Douyin & Bilibili scraper and no-watermark video downloader — async REST API, MCP server for AI agents, CLI and web console, plus offline speech-to-text for downloaded videos. Self-healing identity pool, PostgreSQL archive, one docker compose up. | Evil0ctal | 22k | today | ||
| 3 | Firecrawl MCP ServerFirecrawl, the context API for AI agents: search, scrape, interact with the web, plus paper search. | firecrawl | 7.9k | today | ||
| 4 | wigoloLocal-first web intelligence MCP server for AI coding agents | KnockOutEZ | 5.6k | 1d ago | ||
| 5 | brightdata-mcpBright Data's Web MCP server enabling AI agents to search, extract & navigate the web | brightdata | 2.7k | 3d ago | ||
| 6 | webclawWeb scraping, search, crawling, structured extraction and research for AI agents. | 0xMassi | 2.4k | today | ||
| 7 | CRW Web ScraperOpen-source web scraper for AI agents with scrape, crawl, and map tools | fastcrw | 1.3k | today | ||
| 8 | CRW Web ScraperOpen-source web scraper for AI agents with scrape, crawl, and map tools | us | 1.3k | today | ||
| 9 | fastCRWScrape, crawl, map & search the web. Open-source, self-hostable Rust crawler & search for AI agents. | us | 1.3k | today | ||
| 10 | fortressStealth browser for AI agents: fetch pages behind Cloudflare/DataDome/CAPTCHA, extract clean data. | tiliondev | 817 | 18d ago |
Agent skills 3 indexed · 3 profiled
Protocol-level reach: 26 of the 45 active platforms AgentSidekick tracks document support for the Agent Skills format. DER Which platforms
| # | Product | Publisher | Capabilities | GitHub stars | Last push | Save and compare |
|---|---|---|---|---|---|---|
| 1 | boss-zhipin-scraperBoss直聘爬虫 / BOSS直聘职位数据抓取工具,基于 Chrome CDP 协议复用真实登录态,绕过字体反爬,输出明文薪资 JSON/CSV + 薪资技能分析。A Chrome-CDP-based BOSS Zhipin job scraper/crawler. | eatmoreduck | 1.8k | 18d ago | ||
| 2 | seo-opsSEO foundation checks as an Agent Skill: give it a URL, get a crawler's-eye pass/fail report with evidence. 26 structural checks, zero LLM, deterministic. | tigerless-labs | 992 | 15d ago | ||
| 3 | chatgpt-seoTurn a page, draft, or SEO export into a sourced fix plan inside ChatGPT. | AgriciDaniel | 938 | 1mo ago |
Infrastructure 2 indexed · 2 profiled
No compatibility evidence is collected for this form.
| # | Product | Publisher | Capabilities | GitHub stars | Last push | Save and compare |
|---|---|---|---|---|---|---|
| 1 | chatWebChatWeb can crawl web pages, read PDF, DOCX, TXT, and extract the main content, then answer your questions based on the content, or summarize the key points. | SkywalkerDarren | 916 | 6mo ago | ||
| 2 | markdown-crawlerA multithreaded 🕸️ web crawler that recursively crawls a website and creates a 🔽 markdown file for each page, designed for LLM RAG | paulpierre | 480 | 4mo ago |
Plugins 2 indexed · 2 profiled
2 of these are Claude plugins, which Claude Code documents (protocol-level). DER
| # | Product | Publisher | Capabilities | GitHub stars | Last push | Save and compare |
|---|---|---|---|---|---|---|
| 1 | awesome-claude-pluginsAutomated index of Claude Code plugins and adoption metrics across GitHub repositories, powered by a Node.js/TypeScript crawler | quemsah | 1.7k | today | ||
| 2 | nimbleNimble web data toolkit — search, extract, map, crawl the web and work with structured data agents | Nimbleway | 75 | 3d ago |
Rules packs 2 indexed · 1 profiled
No compatibility evidence is collected for this form.
| # | Product | Publisher | Capabilities | GitHub stars | Last push | Save and compare |
|---|---|---|---|---|---|---|
| 1 | AI-Cursor-Scraping-AssistantA powerful tool that leverages Cursor AI and MCP (Model Context Protocol) to easily generate web scrapers for various types of websites. | TheWebScrapingClub | 56 | 1y ago |
Agent projects 1 indexed · 1 profiled
No compatibility evidence is collected for this form.
AI APIs 1 indexed · 1 profiled
No compatibility evidence is collected for this form.
Libraries 1 indexed · 1 profiled
No compatibility evidence is collected for this form.
| # | Product | Publisher | Capabilities | GitHub stars | Last push | Save and compare |
|---|---|---|---|---|---|---|
| 1 | Crawl4AIOpen-source async web crawler that outputs LLM-ready Markdown and structured extraction, running headless browsers locally. | unclecode | 88k | 1d ago |
Frameworks 1 indexed · 0 profiled
No compatibility evidence is collected for this form.
1 indexed, none with a full profile yet.
Which platforms these forms can reach
All platformsProtocol-level: the platform documents support for the standard. No specific pairing has been tested.
MCP servers 218 indexed here
36 platforms document MCP support CUR.
ContinueOpenCodeClaude CodeCodexGemini CLIZedOpenHandsClineWarpgooseQwen CodeBolt (bolt.new)KiroAmazon Bedrock AgentCoreAmpApps in ChatGPT / Apps SDK (now ChatGPT Plugins)Augment CodeChatGPTClaude (claude.ai apps)Claude DesktopCursorDevinDevin Desktop (formerly Windsurf)Factory (Droid)Gemini (app)Gemini Enterprise Agent Platform (formerly Vertex AI / Vertex AI Agent Builder)GitHub CopilotGitHub Copilot cloud agent (formerly Copilot coding agent)Google AntigravityJetBrains AI AssistantJunieLovableMicrosoft Copilot StudioReplit AgentTabninev0
Agent skills 3 indexed here
26 platforms document Agent Skills support CUR.
OpenClawOpenCodeClaude CodeCodexGemini CLIOpenHandsWarpgooseBolt (bolt.new)KiroAmpApps in ChatGPT / Apps SDK (now ChatGPT Plugins)ChatGPTClaude (claude.ai apps)Claude DesktopCursorDevinDevin Desktop (formerly Windsurf)Factory (Droid)Gemini (app)GitHub CopilotGitHub Copilot cloud agent (formerly Copilot coding agent)Google AntigravityJunieReplit AgentTabnine
Platforms publishers name
| Platform | Profiled products naming it |
|---|---|
| Claude Code | 1 |
| ChatGPT | 1 |
| Cursor | 1 |
Integrations most often paired
Third-party services named by products with this capability, whole catalogue.
| Integration | Products | Share |
|---|---|---|
| GitHub | 2 | 0.9% |
| Google Analytics | 2 | 0.9% |
| 2 | 0.9% | |
| YouTube | 2 | 0.9% |
| Docker | 1 | 0.4% |
| Google Ads | 1 | 0.4% |
| Google Maps | 1 | 0.4% |
| PostgreSQL | 1 | 0.4% |
| Shopify | 1 | 0.4% |
| Solana | 1 | 0.4% |
Keywords that trigger this capability
Matched as whole words, ignoring case, in a product's name, description and repository topics.
| Keyword | Products matched |
|---|---|
| web crawling | 4 |
| crawler | 154 |
| crawl websites | 2 |
| sitemap | 58 |
| crawl4ai | 7 |
| spider | 6 |
Segment statistics
Whole catalogue (231 indexed products). Calculated 2026-11-21.
Crowded, outpacing the catalogue. Among the largest fifth of capabilities, and its launches grew faster than the catalogue’s over the last two 90-day windows. Size is ranked against all capabilities; momentum compares launches in the last 90 days with the 90 before, relative to the catalogue’s own ratio (62 or more = faster, 38 or less = slower; under 5 dated launches it is not measured). The HHI of 0.263 means stars are spread as if across about 4 equally popular projects — highly concentrated by the common 0.15 / 0.25 thresholds. Stars measure attention, not usage or quality.
Limits
- Classification is by the keywords above. A product that offers this capability without using one of them is not counted; one that uses a keyword in another sense is counted wrongly. Methodology
- Platform reach is protocol-level: it says a platform documents the standard, not that any product here was tested on it. Platforms named by publishers are their own claims.
- Stars belong to repositories. Products in a subfolder of a large repository show that whole repository’s stars and are excluded from the concentration figures.
- Launch counts use each source’s publication date; for auto-discovered repositories that is the repository’s creation date.
Page data last updated .