AgentSidekick

Web Crawling

Crawling whole sites and sitemaps. As of 2026-11-21, 231 indexed products offer it (56 with full profiles), in 9 forms: 218 MCP servers, 3 Agent skills, 2 Infrastructure, 2 Plugins. 160 were published in the last 90 days, against 37 in the 90 days before. Firecrawl has the most GitHub stars, 42% of the total across 140 products with their own repository.

CapabilitySegment label: Crowded, outpacing the catalogue DER

Available as

All in the directory

Profiled products grouped by product form. Within each form: own-repository products by GitHub stars, then products in a subfolder of a larger repository (they show that repository's stars), up to 10 each.

MCP servers 218 indexed · 45 profiled

Protocol-level reach: 36 of the 45 active platforms AgentSidekick tracks document support for MCP. DER Which platforms

Web Crawling: MCP servers by GitHub stars
#ProductPublisherCapabilities GitHub stars Last push Save and compare
1Scrapling MCP ServerWeb scraping with stealth HTTP, real browsers, and Cloudflare bypass. CSS selectors supported.D4Vinci105k4d ago
2Douyin_TikTok_Download_API🚀 自托管的抖音、TikTok、B 站数据采集与无水印视频下载 API,支持 AI agent 通过 MCP 调用与离线语音转文字,Docker 一键部署。| Self-hosted TikTok, Douyin & Bilibili scraper and no-watermark video downloader — async REST API, MCP server for AI agents, CLI and web console, plus offline speech-to-text for downloaded videos. Self-healing identity pool, PostgreSQL archive, one docker compose up.Evil0ctal22ktoday
3Firecrawl MCP ServerFirecrawl, the context API for AI agents: search, scrape, interact with the web, plus paper search.firecrawl7.9ktoday
4wigoloLocal-first web intelligence MCP server for AI coding agentsKnockOutEZ5.6k1d ago
5brightdata-mcpBright Data's Web MCP server enabling AI agents to search, extract & navigate the webbrightdata2.7k3d ago
6webclawWeb scraping, search, crawling, structured extraction and research for AI agents.0xMassi2.4ktoday
7CRW Web ScraperOpen-source web scraper for AI agents with scrape, crawl, and map toolsfastcrw1.3ktoday
8CRW Web ScraperOpen-source web scraper for AI agents with scrape, crawl, and map toolsus1.3ktoday
9fastCRWScrape, crawl, map & search the web. Open-source, self-hostable Rust crawler & search for AI agents.us1.3ktoday
10fortressStealth browser for AI agents: fetch pages behind Cloudflare/DataDome/CAPTCHA, extract clean data.tiliondev81718d ago

All 45 profiled MCP servers with this capability

Agent skills 3 indexed · 3 profiled

Protocol-level reach: 26 of the 45 active platforms AgentSidekick tracks document support for the Agent Skills format. DER Which platforms

Web Crawling: Agent skills by GitHub stars
#ProductPublisherCapabilities GitHub stars Last push Save and compare
1boss-zhipin-scraperBoss直聘爬虫 / BOSS直聘职位数据抓取工具,基于 Chrome CDP 协议复用真实登录态,绕过字体反爬,输出明文薪资 JSON/CSV + 薪资技能分析。A Chrome-CDP-based BOSS Zhipin job scraper/crawler.eatmoreduck1.8k18d ago
2seo-opsSEO foundation checks as an Agent Skill: give it a URL, get a crawler's-eye pass/fail report with evidence. 26 structural checks, zero LLM, deterministic.tigerless-labs99215d ago
3chatgpt-seoTurn a page, draft, or SEO export into a sourced fix plan inside ChatGPT.AgriciDaniel9381mo ago

Infrastructure 2 indexed · 2 profiled

No compatibility evidence is collected for this form.

Web Crawling: Infrastructure by GitHub stars
#ProductPublisherCapabilities GitHub stars Last push Save and compare
1chatWebChatWeb can crawl web pages, read PDF, DOCX, TXT, and extract the main content, then answer your questions based on the content, or summarize the key points.SkywalkerDarren9166mo ago
2markdown-crawlerA multithreaded 🕸️ web crawler that recursively crawls a website and creates a 🔽 markdown file for each page, designed for LLM RAGpaulpierre4804mo ago

Plugins 2 indexed · 2 profiled

2 of these are Claude plugins, which Claude Code documents (protocol-level). DER

Web Crawling: Plugins by GitHub stars
#ProductPublisherCapabilities GitHub stars Last push Save and compare
1awesome-claude-pluginsAutomated index of Claude Code plugins and adoption metrics across GitHub repositories, powered by a Node.js/TypeScript crawlerquemsah1.7ktoday
2nimbleNimble web data toolkit — search, extract, map, crawl the web and work with structured data agentsNimbleway753d ago

Rules packs 2 indexed · 1 profiled

No compatibility evidence is collected for this form.

Web Crawling: Rules packs by GitHub stars
#ProductPublisherCapabilities GitHub stars Last push Save and compare
1AI-Cursor-Scraping-AssistantA powerful tool that leverages Cursor AI and MCP (Model Context Protocol) to easily generate web scrapers for various types of websites.TheWebScrapingClub561y ago

Agent projects 1 indexed · 1 profiled

No compatibility evidence is collected for this form.

Web Crawling: Agent projects by GitHub stars
#ProductPublisherCapabilities GitHub stars Last push Save and compare
1moliBest headless browser for AI agents. Lite, Fast, High-Compatibility. Built in Rustlexmount24ktoday

AI APIs 1 indexed · 1 profiled

No compatibility evidence is collected for this form.

Web Crawling: AI APIs by GitHub stars
#ProductPublisherCapabilities GitHub stars Last push Save and compare
1FirecrawlAPI that crawls and scrapes websites into clean Markdown or structured JSON, with search, map and agentic extract endpoints.firecrawl206ktoday

Libraries 1 indexed · 1 profiled

No compatibility evidence is collected for this form.

Web Crawling: Libraries by GitHub stars
#ProductPublisherCapabilities GitHub stars Last push Save and compare
1Crawl4AIOpen-source async web crawler that outputs LLM-ready Markdown and structured extraction, running headless browsers locally.unclecode88k1d ago

Frameworks 1 indexed · 0 profiled

No compatibility evidence is collected for this form.

1 indexed, none with a full profile yet.

Which platforms these forms can reach

All platforms

Protocol-level: the platform documents support for the standard. No specific pairing has been tested.

Platforms publishers name

PlatformProfiled products naming it
Claude Code1
ChatGPT1
Cursor1

Integrations most often paired

Third-party services named by products with this capability, whole catalogue.

IntegrationProducts Share
GitHub20.9%
Google Analytics20.9%
Reddit20.9%
YouTube20.9%
Docker10.4%
Google Ads10.4%
Google Maps10.4%
PostgreSQL10.4%
Shopify10.4%
Solana10.4%

Keywords that trigger this capability

Matched as whole words, ignoring case, in a product's name, description and repository topics.

KeywordProducts matched
web crawling4
crawler154
crawl websites2
sitemap58
crawl4ai7
spider6

Segment statistics

Whole catalogue (231 indexed products). Calculated 2026-11-21.

231DERIndexed products56 with full profiles
160DERLaunched, last 90 days37 in the 90 days before
64DERMomentum vs catalogue50 = same pace as the catalogue
42%DERLargest project's share of starsAcross 140 products with their own repository
0.263DERConcentration (HHI)About 4 equal-sized projects’ worth; highly concentrated
2SRCMedian starsPer product with its own repository
57%DEROpen sourceMarked open source; unknown counts as not

Crowded, outpacing the catalogue. Among the largest fifth of capabilities, and its launches grew faster than the catalogue’s over the last two 90-day windows. Size is ranked against all capabilities; momentum compares launches in the last 90 days with the 90 before, relative to the catalogue’s own ratio (62 or more = faster, 38 or less = slower; under 5 dated launches it is not measured). The HHI of 0.263 means stars are spread as if across about 4 equally popular projects — highly concentrated by the common 0.15 / 0.25 thresholds. Stars measure attention, not usage or quality.

Limits

  • Classification is by the keywords above. A product that offers this capability without using one of them is not counted; one that uses a keyword in another sense is counted wrongly. Methodology
  • Platform reach is protocol-level: it says a platform documents the standard, not that any product here was tested on it. Platforms named by publishers are their own claims.
  • Stars belong to repositories. Products in a subfolder of a large repository show that whole repository’s stars and are excluded from the concentration figures.
  • Launch counts use each source’s publication date; for auto-discovered repositories that is the repository’s creation date.

Page data last updated .