AgentSidekick

Web Data

Searching and extracting information from the web. As of 2026-11-21, 1,470 indexed products match this category (287 with full profiles); 967 were published in the last 90 days, against 258 in the 90 days before, roughly in step with the catalogue as a whole. By GitHub stars, Firecrawl leads, holding 20% of the stars of the 1,064 products here with their own repository. The most common form is MCP servers (96% of indexed products).

CategorySegment label: Established DER

Segment statistics

Whole catalogue (all 1,470 indexed products, not only profiled ones). Calculated 2026-11-21.

1,470DERIndexed products287 with full profiles
967DERLaunched, last 90 days258 in the 90 days before
58DERMomentum vs catalogue50 = same pace as the catalogue
20%DERLargest project's share of starsAcross 1,064 products with their own repository
0.080DERConcentration (HHI)About 12 equal-sized projects’ worth; unconcentrated
0SRCMedian starsPer product with its own repository
61%DEROpen sourceMarked open source; unknown counts as not

Established. Mid-sized for its level, with launches roughly in step with the catalogue. Size is ranked against the other category nodes; momentum compares this node’s launches in the last 90 days with the 90 before, relative to the whole catalogue’s ratio (score 62 or more = faster, 38 or less = slower; below 5 dated launches it is not measured and the label uses size alone).

Concentration. The Herfindahl–Hirschman index (HHI) adds up each project’s squared share of stars. At 0.080, attention is spread as if across about 12 equally popular projects. Using the common competition-policy thresholds (0.15 and 0.25), that is unconcentrated. Stars measure attention, not usage or quality.

Which product forms serve it

The same need is met by MCP servers, agent skills, plugins, extensions and frameworks. This counts each form across the whole catalogue.

Products in Web Data by product type, as of 2026-11-21
Product type IndexedShareWith a profile
MCP servers1,41095.9%230
Agent skills241.6%24
Plugins100.68%10
AI APIs80.54%8
Agent projects60.41%6
Frameworks40.27%3
Infrastructure30.20%2
Rules packs20.14%1
Libraries20.14%2
Agent SDKs10.07%1

2 subcategories

Indexed = whole catalogue; profiled = products with their own page. A product can sit in several, so the rows do not add up to the total.

SubcategoryIndexed ProfiledNew, 90 daysLabel
Web ScrapingCrawling sites and extracting content.922190633Crowded, outpacing the catalogue
Web SearchSearch engines and search APIs for agents.654145387Established

Top 40 products by GitHub stars

Search the directory

Profiled products only (287 of 1,470), ranked by the stars of their own repository. Products that live in a subfolder of a larger repository (common for skills) show that whole repository's stars and are ranked after them; products without a repository come last. Stars are popularity, not quality or installs.

Web Data: top profiled products by GitHub stars, 2026-11-21
#ProductTypePublisherCapabilities GitHub stars Last push LicenceSave and compare
1FirecrawlAPI that crawls and scrapes websites into clean Markdown or structured JSON, with search, map and agentic extract endpoints.APIfirecrawl206ktodayAGPL-3.0
2Browser UsePython library that lets LLM agents control a Playwright/CDP browser by reading page state and issuing actions, with a hosted cloud option.Librarybrowser-use122ktodayMIT
3Scrapling MCP ServerWeb scraping with stealth HTTP, real browsers, and Cloudflare bypass. CSS selectors supported.MCP serverD4Vinci105k4d agoBSD-3-Clause
4Crawl4AIOpen-source async web crawler that outputs LLM-ready Markdown and structured extraction, running headless browsers locally.Libraryunclecode88k1d agoApache-2.0
5TrendRadar⭐AI-driven public opinion & trend monitor with multi-platform aggregation, RSS, web search, MCP & smart alerts.🎯 告别信息过载,你的 AI 舆情监控助手与热点筛选工具!聚合多平台热点 + RSS 订阅 + 定时联网搜索,支持关键词与 AI 智能筛选。AI 翻译 + AI 分析简报直推手机,支持接入 MCP 架构,赋能 AI 自然语言对话分析、实时检索、情感洞察与趋势预测。支持 Docker,数据本地/云端自持。集成微信/飞书/钉钉/Telegram/邮件/ntfy/Bark/Slack 等渠道智能推送。MCP serversansan064ktodayGPL-3.0
6ai-website-cloner-templateClone any website with one command using AI coding agentsAgent projectJCodesMore40k11d agoMIT
7changedetection.io (official)Official changedetection.io MCP: web page change tracking, back-in-stock and price alerts.MCP serverchangedetection.io36k1d agoApache-2.0
8CloakBrowserStealth Chromium that passes every bot detection test. Drop-in Playwright replacement with source-level fingerprint patches. 50+ tests passed.Agent projectCloakHQ33k1d agoMIT
9OpenAI APIAPI for GPT and reasoning models via the Responses and Chat Completions endpoints, with built-in tools such as web search, file search, code interpreter and computer use.APIopenai32k1d agoApache-2.0
10OpenSEOKeyword research, site audits, rank tracking and AI visibility for your AI agent.MCP servereveryapp.com28k1d agoMIT
11StagehandBrowser automation SDK that mixes Playwright-style code with natural-language act, extract and observe primitives and an agent mode.SDKbrowserbase26ktodayMIT
12moliBest headless browser for AI agents. Lite, Fast, High-Compatibility. Built in RustAgent projectlexmount24ktodayApache-2.0
13Douyin_TikTok_Download_API🚀 自托管的抖音、TikTok、B 站数据采集与无水印视频下载 API,支持 AI agent 通过 MCP 调用与离线语音转文字,Docker 一键部署。| Self-hosted TikTok, Douyin & Bilibili scraper and no-watermark video downloader — async REST API, MCP server for AI agents, CLI and web console, plus offline speech-to-text for downloaded videos. Self-healing identity pool, PostgreSQL archive, one docker compose up.MCP serverEvil0ctal22ktodayApache-2.0
14Skill_SeekersConvert documentation websites, GitHub repositories, and PDFs into Claude AI skills with automatic conflict detectionMCP serveryusufkaraaslan15k5d agoMIT
15nanobrowserOpen-source AI agent for web automation. Use your real, logged-in browser to automate daily tasks.Agent projectnanobrowser14k4d agoApache-2.0
16SeleniumBase MCPStealthy browser automation, testing, and web-scraping via CDP Mode.MCP serverseleniumbase13ktodayMIT
17Jina ReaderAPI that converts any URL into LLM-friendly Markdown (r.jina.ai) and runs web search returning cleaned page content (s.jina.ai).APIjina-ai12k6mo agoApache-2.0
18AIHOTChinese AI news: search topics, explore hot events, and read daily, weekly and monthly briefs.MCP serverKKKKhazix11ktodayMIT
19nopecha-extensionAutomated CAPTCHA solver for your browser. Works with Selenium, Puppeteer, Playwright, and more.Agent projectNopeCHALLC11k1mo agoMIT
20google-maps-scraperscrape data from Google Maps. Extracts data such as the name, address, phone number, website URL, rating, reviews number, latitude and longitude, reviews,email and more for each placeAgent skillgosom8.3ktodayMIT
21Firecrawl MCP ServerFirecrawl, the context API for AI agents: search, scrape, interact with the web, plus paper search.MCP serverfirecrawl7.9ktodayMIT
22big-AGIThe AI workspace for experts. Switch across 800+ models mid-chat, tune 40+ parameters, Beam models side by side. BYO keys, self-hostable. Built on our native-protocol engine: reliable, hyper-configurable, and fast.Agent projectenricoros7.2ktodayMIT
23MindSearch🔍 An LLM-based Multi-agent Framework of Web Search Engine (like Perplexity.ai Pro and SearchGPT)FrameworkInternLM6.9k1y agoApache-2.0
24Airweave SearchMCP server for searching Airweave collections with natural language queries.MCP serverairweave-ai6.5k5mo agoMIT
25skillsBrowser automation CLI built for AI agents. Break through anti-bot walls, hand off to humans across platforms when stuck. Parallel multi-task execution, independent multi-session operation, isolated multi-account browsing.Agent skillbrowser-act6.4k17d agoMIT
26wigoloLocal-first web intelligence MCP server for AI coding agentsMCP serverKnockOutEZ5.6k1d ago—
27ExaOfficial Exa MCP for web search, content fetching, and multi-step research.MCP serverexa.ai5.3ktodayMIT
28exaExa AI web search, deep research, and content extraction. Provides MCP tools and research skills for comprehensive web search, people discovery, company research, academic papers, and more.Pluginexa-labs5.3ktodayMIT
29designlangMeasure any site's design system, score how generic a page looks, check UI against your brand.MCP serverManavarya095.2k3d agoMIT
30Claude APIAPI for Claude models with tool use, extended thinking, prompt caching, a remote MCP connector and server-side tools such as web search and code execution.APIAnthropic4.1k1d agoApache-2.0
31Gemini APIDeveloper API for Gemini models with long context, multimodal input, function calling and grounding with Google Search.APIgoogle4.1ktodayApache-2.0
32stealth-browser-mcpStealth browser automation for AI agents over MCP and the Chrome DevTools Protocol: navigation, network hooks, DOM extraction, and pixel-accurate UI cloning.MCP servervibheksoni3.4k1d agoMIT
33ddgsA metasearch library that aggregates results from diverse web search servicesMCP serverdeedy53.1k28d agoMIT
34brightdata-mcpBright Data's Web MCP server enabling AI agents to search, extract & navigate the webMCP serverbrightdata2.7k3d agoMIT
35google-maps-scraper-kitFree, open-source Google Maps scraper for lead generation (Python, no API key, no Docker). Exports name, phone, email, website, rating and Instagram/Facebook/LinkedIn/TikTok/YouTube/X links to CSV. 100 places with emails in 44 s; --grid beats Google's 120 cap. MX email check, ad-pixel detection. Skill for Claude Code, Codex and Cursor.Agent skillMahanaicoach2.6k2d agoAGPL-3.0
36tavily-mcpMCP server for advanced web search using TavilyMCP servertavily-ai2.5k3d agoMIT
37webclawWeb scraping, search, crawling, structured extraction and research for AI agents.MCP server0xMassi2.4ktodayAGPL-3.0
38company-research-agentAn agentic company research tool powered by LangGraph and Tavily that conducts deep diligence on companies using a multi-agent framework. It leverages OpenRouter on the backend for rapid inference with any leading model.Frameworkguy-hartstein2.3k18d agoApache-2.0
39mcp-gscGoogle Search Console Insights with Claude AI for SEOsMCP serverAminForou2.3k1d agoMIT
40last30days-skill-cn给 AI Agent 的中文互联网研究 Skill:检索 8 大平台最近 30 天的讨论,追踪跨平台热点,生成附原始来源的报告,并提供每日全网热榜。支持 Claude Code、Codex、OpenClaw。Agent skillJesseovo2.2k1d ago—

Most common integrations

Third-party services named by products in this node, whole catalogue.

IntegrationProducts Share
YouTube412.8%
Google Maps261.8%
Google Analytics221.5%
Reddit191.3%
LinkedIn171.2%
GitHub151.0%
Playwright151.0%
Google Ads80.5%
Shopify80.5%
Google Cloud60.4%
Workday60.4%
X (Twitter)60.4%
Solana50.3%
Telegram50.3%
Docker40.3%

Launches per month

New Web Data products per monthSRC

Whole catalogue, by publication month, 2024-12 to 2026-11. The last month is partial; undated products are excluded.

New Web Data products per monthWhole catalogue, by publication month, 2024-12 to 2026-11. The last month is partial; undated products are excluded.. Products published: 21 points, latest 303 products in 2026-11.
Show the data as a table
New Web Data products per month. Whole catalogue, by publication month, 2024-12 to 2026-11. The last month is partial; undated products are excluded.
Products published (products)
2024-122
2025-021
2025-034
2025-043
2025-061
2025-082
2025-0913
2025-1012
2025-116
2025-129
2026-019
2026-0226
2026-0345
2026-0448
2026-0534
2026-0671
2026-07107
2026-08121
2026-09208
2026-10409
2026-11303

Related opportunities

Opportunity radar

Scored gaps for this node or anything beneath it. Scores rank hypotheses; they do not predict demand.

OpportunityCompetitionConfidenceScore
Coverage gap: Proxies & Anti-botProxies & Anti-botHighLow56.1
Coverage gap: Content ExtractionContent ExtractionHighLow53.9

Limits of these numbers

  • Products are placed here by whole-word keyword rules on their name, description and repository topics, and inherit every ancestor of a matched node. Products that do this without naming it are missed; ambiguous words cause false matches. Methodology
  • About half of the catalogue matches no node at all, so absolute counts understate every category.
  • Launch counts use the publication date each source reports; for auto-discovered repositories that is the repository’s creation date.
  • Stars and concentration cover only products with their own repository (not a subfolder of a larger one). They measure attention, not adoption.

Page data last updated .