AgentSidekick

Web Scraping

Crawling sites and extracting content. As of 2026-11-21, 922 indexed products match this subcategory (190 with full profiles); 633 were published in the last 90 days, against 147 in the 90 days before, faster than the catalogue as a whole. By GitHub stars, Firecrawl leads, holding 24% of the stars of the 744 products here with their own repository. The most common form is MCP servers (96% of indexed products).

SubcategorySegment label: Crowded, outpacing the catalogue DER

Segment statistics

Whole catalogue (all 922 indexed products, not only profiled ones). Calculated 2026-11-21.

922DERIndexed products190 with full profiles
633DERLaunched, last 90 days147 in the 90 days before
65DERMomentum vs catalogue50 = same pace as the catalogue
24%DERLargest project's share of starsAcross 744 products with their own repository
0.112DERConcentration (HHI)About 9 equal-sized projects’ worth; unconcentrated
0SRCMedian starsPer product with its own repository
67%DEROpen sourceMarked open source; unknown counts as not

Crowded, outpacing the catalogue. Among the largest fifth of nodes at its level, and its launches grew faster than the catalogue’s over the last two 90-day windows. Size is ranked against the other subcategory nodes; momentum compares this node’s launches in the last 90 days with the 90 before, relative to the whole catalogue’s ratio (score 62 or more = faster, 38 or less = slower; below 5 dated launches it is not measured and the label uses size alone).

Concentration. The Herfindahl–Hirschman index (HHI) adds up each project’s squared share of stars. At 0.112, attention is spread as if across about 9 equally popular projects. Using the common competition-policy thresholds (0.15 and 0.25), that is unconcentrated. Stars measure attention, not usage or quality.

Which product forms serve it

The same need is met by MCP servers, agent skills, plugins, extensions and frameworks. This counts each form across the whole catalogue.

Products in Web Scraping by product type, as of 2026-11-21
Product type IndexedShareWith a profile
MCP servers88696.1%156
Agent skills131.4%13
Plugins80.87%8
Agent projects50.54%5
Rules packs20.22%1
Libraries20.22%2
Infrastructure20.22%2
AI APIs20.22%2
Agent SDKs10.11%1
Frameworks10.11%0

4 capabilities

Indexed = whole catalogue; profiled = products with their own page. A product can sit in several, so the rows do not add up to the total.

CapabilityIndexed ProfiledNew, 90 daysLabel
Web CrawlingCrawling whole sites and sitemaps.23156160Crowded, outpacing the catalogue
Structured ScrapingPulling structured records from pages.15918139Outpacing the catalogue
Content ExtractionTurning a URL into clean text or Markdown.852737Established
Proxies & Anti-botProxies and stealth for scraping at scale.602125Lagging the catalogue

Top 40 products by GitHub stars

Search the directory

Profiled products only (190 of 922), ranked by the stars of their own repository. Products that live in a subfolder of a larger repository (common for skills) show that whole repository's stars and are ranked after them; products without a repository come last. Stars are popularity, not quality or installs.

Web Scraping: top profiled products by GitHub stars, 2026-11-21
#ProductTypePublisherCapabilities GitHub stars Last push LicenceSave and compare
1FirecrawlAPI that crawls and scrapes websites into clean Markdown or structured JSON, with search, map and agentic extract endpoints.APIfirecrawl206ktodayAGPL-3.0
2Browser UsePython library that lets LLM agents control a Playwright/CDP browser by reading page state and issuing actions, with a hosted cloud option.Librarybrowser-use122ktodayMIT
3Scrapling MCP ServerWeb scraping with stealth HTTP, real browsers, and Cloudflare bypass. CSS selectors supported.MCP serverD4Vinci105k4d agoBSD-3-Clause
4Crawl4AIOpen-source async web crawler that outputs LLM-ready Markdown and structured extraction, running headless browsers locally.Libraryunclecode88k1d agoApache-2.0
5ai-website-cloner-templateClone any website with one command using AI coding agentsAgent projectJCodesMore40k11d agoMIT
6changedetection.io (official)Official changedetection.io MCP: web page change tracking, back-in-stock and price alerts.MCP serverchangedetection.io36k1d agoApache-2.0
7CloakBrowserStealth Chromium that passes every bot detection test. Drop-in Playwright replacement with source-level fingerprint patches. 50+ tests passed.Agent projectCloakHQ33k1d agoMIT
8StagehandBrowser automation SDK that mixes Playwright-style code with natural-language act, extract and observe primitives and an agent mode.SDKbrowserbase26ktodayMIT
9moliBest headless browser for AI agents. Lite, Fast, High-Compatibility. Built in RustAgent projectlexmount24ktodayApache-2.0
10Douyin_TikTok_Download_API🚀 自托管的抖音、TikTok、B 站数据采集与无水印视频下载 API,支持 AI agent 通过 MCP 调用与离线语音转文字,Docker 一键部署。| Self-hosted TikTok, Douyin & Bilibili scraper and no-watermark video downloader — async REST API, MCP server for AI agents, CLI and web console, plus offline speech-to-text for downloaded videos. Self-healing identity pool, PostgreSQL archive, one docker compose up.MCP serverEvil0ctal22ktodayApache-2.0
11Skill_SeekersConvert documentation websites, GitHub repositories, and PDFs into Claude AI skills with automatic conflict detectionMCP serveryusufkaraaslan15k5d agoMIT
12nanobrowserOpen-source AI agent for web automation. Use your real, logged-in browser to automate daily tasks.Agent projectnanobrowser14k4d agoApache-2.0
13SeleniumBase MCPStealthy browser automation, testing, and web-scraping via CDP Mode.MCP serverseleniumbase13ktodayMIT
14Jina ReaderAPI that converts any URL into LLM-friendly Markdown (r.jina.ai) and runs web search returning cleaned page content (s.jina.ai).APIjina-ai12k6mo agoApache-2.0
15nopecha-extensionAutomated CAPTCHA solver for your browser. Works with Selenium, Puppeteer, Playwright, and more.Agent projectNopeCHALLC11k1mo agoMIT
16google-maps-scraperscrape data from Google Maps. Extracts data such as the name, address, phone number, website URL, rating, reviews number, latitude and longitude, reviews,email and more for each placeAgent skillgosom8.3ktodayMIT
17Firecrawl MCP ServerFirecrawl, the context API for AI agents: search, scrape, interact with the web, plus paper search.MCP serverfirecrawl7.9k1d agoMIT
18skillsBrowser automation CLI built for AI agents. Break through anti-bot walls, hand off to humans across platforms when stuck. Parallel multi-task execution, independent multi-session operation, isolated multi-account browsing.Agent skillbrowser-act6.4k17d agoMIT
19wigoloLocal-first web intelligence MCP server for AI coding agentsMCP serverKnockOutEZ5.6k1d ago—
20exaExa AI web search, deep research, and content extraction. Provides MCP tools and research skills for comprehensive web search, people discovery, company research, academic papers, and more.Pluginexa-labs5.3ktodayMIT
21designlangMeasure any site's design system, score how generic a page looks, check UI against your brand.MCP serverManavarya095.2k3d agoMIT
22stealth-browser-mcpStealth browser automation for AI agents over MCP and the Chrome DevTools Protocol: navigation, network hooks, DOM extraction, and pixel-accurate UI cloning.MCP servervibheksoni3.4k1d agoMIT
23brightdata-mcpBright Data's Web MCP server enabling AI agents to search, extract & navigate the webMCP serverbrightdata2.7k3d agoMIT
24google-maps-scraper-kitFree, open-source Google Maps scraper for lead generation (Python, no API key, no Docker). Exports name, phone, email, website, rating and Instagram/Facebook/LinkedIn/TikTok/YouTube/X links to CSV. 100 places with emails in 44 s; --grid beats Google's 120 cap. MX email check, ad-pixel detection. Skill for Claude Code, Codex and Cursor.Agent skillMahanaicoach2.6k2d agoAGPL-3.0
25webclawWeb scraping, search, crawling, structured extraction and research for AI agents.MCP server0xMassi2.4ktodayAGPL-3.0
26anysearch-mcp-serverUnified real-time search MCP server supporting general web search, vertical domain search, parallel batch search, and full-page URL content extraction.MCP serveranysearch-ai2.1k5d agoApache-2.0
27boss-zhipin-scraperBoss直聘爬虫 / BOSS直聘职位数据抓取工具,基于 Chrome CDP 协议复用真实登录态,绕过字体反爬,输出明文薪资 JSON/CSV + 薪资技能分析。A Chrome-CDP-based BOSS Zhipin job scraper/crawler.Agent skilleatmoreduck1.8k18d agoMIT
28awesome-claude-pluginsAutomated index of Claude Code plugins and adoption metrics across GitHub repositories, powered by a Node.js/TypeScript crawlerPluginquemsah1.7ktoday—
29openserpSelf-hosted web search for AI agents and SEO. Google, Bing, Yandex, Baidu, DuckDuckGo & Ecosia, rendered in a real browser: ranked results and clean page Markdown. API, CLI, MCP 🎉MCP serverkarust1.6k1d agoMIT
30CRW Web ScraperOpen-source web scraper for AI agents with scrape, crawl, and map toolsMCP serverfastcrw1.3ktodayAGPL-3.0
31CRW Web ScraperOpen-source web scraper for AI agents with scrape, crawl, and map toolsMCP serverus1.3ktodayAGPL-3.0
32fastCRWScrape, crawl, map & search the web. Open-source, self-hostable Rust crawler & search for AI agents.MCP serverus1.3ktodayAGPL-3.0
33anymdAny file → clean Markdown for AI agents: PDF, Office, EPUB, HTML, images, audio/video. Local MCP.MCP serverSylphxAI1.1ktodayMIT
34CitraCitra — PDF answers with page-level proof: structure, tables, OCR, and page citations.MCP serverSylphxAI1.1ktodayMIT
35PDF Reader MCPEvidence-first PDF MCP. Agent Document Twin with citeable page+bbox evidence.MCP serverSylphxAI1.1ktodayMIT
36seo-opsSEO foundation checks as an Agent Skill: give it a URL, get a crawler's-eye pass/fail report with evidence. 26 structural checks, zero LLM, deterministic.Agent skilltigerless-labs99215d ago—
37x-readerUniversal content reader MCP Server for 10+ platformsMCP serverrunesleo99210d agoMIT
38huashu-md-htmlmd/html 双向流水线 · 三个能力一站式:万物→md / md→精美html / html→md。封装 markitdown + Pandoc + html-to-markdown + trafilatura。4 套反 AI slop 主题。Claude Code skill.Agent skillalchaincyf9486d agoMIT
39chatgpt-seoTurn a page, draft, or SEO export into a sourced fix plan inside ChatGPT.Agent skillAgriciDaniel9381mo agoMIT
40chatWebChatWeb can crawl web pages, read PDF, DOCX, TXT, and extract the main content, then answer your questions based on the content, or summarize the key points.InfrastructureSkywalkerDarren9166mo agoMIT

Most common integrations

Third-party services named by products in this node, whole catalogue.

IntegrationProducts Share
YouTube313.4%
LinkedIn141.5%
Playwright141.5%
Reddit111.2%
Google Maps101.1%
GitHub91.0%
Google Ads60.7%
Shopify60.7%
Workday50.5%
Docker40.4%
Solana40.4%
Google Cloud30.3%
X (Twitter)30.3%
Google Analytics20.2%
Kubernetes20.2%

Launches per month

New Web Scraping products per monthSRC

Whole catalogue, by publication month, 2024-12 to 2026-11. The last month is partial; undated products are excluded.

New Web Scraping products per monthWhole catalogue, by publication month, 2024-12 to 2026-11. The last month is partial; undated products are excluded.. Products published: 19 points, latest 192 products in 2026-11.
Show the data as a table
New Web Scraping products per month. Whole catalogue, by publication month, 2024-12 to 2026-11. The last month is partial; undated products are excluded.
Products published (products)
2024-122
2025-033
2025-042
2025-081
2025-096
2025-108
2025-113
2025-128
2026-013
2026-0217
2026-0325
2026-0428
2026-0519
2026-0644
2026-0766
2026-0863
2026-09142
2026-10270
2026-11192

Related opportunities

Opportunity radar

Scored gaps for this node or anything beneath it. Scores rank hypotheses; they do not predict demand.

OpportunityCompetitionConfidenceScore
Coverage gap: Proxies & Anti-botProxies & Anti-botHighLow56.1
Coverage gap: Content ExtractionContent ExtractionHighLow53.9

Limits of these numbers

  • Products are placed here by whole-word keyword rules on their name, description and repository topics, and inherit every ancestor of a matched node. Products that do this without naming it are missed; ambiguous words cause false matches. Methodology
  • About half of the catalogue matches no node at all, so absolute counts understate every category.
  • Launch counts use the publication date each source reports; for auto-discovered repositories that is the repository’s creation date.
  • Stars and concentration cover only products with their own repository (not a subfolder of a larger one). They measure attention, not adoption.

Page data last updated .