Benchmarking
Comparing models and agents on standard benchmarks. As of 2026-11-21, 297 indexed products offer it (101 with full profiles), in 8 forms: 216 MCP servers, 39 Infrastructure, 19 Agent skills, 8 Plugins. 139 were published in the last 90 days, against 76 in the 90 days before. artemis has the most GitHub stars, 22% of the total across 198 products with their own repository.
CapabilitySegment label: Crowded, slowing DER
Available as
All in the directoryProfiled products grouped by product form. Within each form: own-repository products by GitHub stars, then products in a subfolder of a larger repository (they show that repository's stars), up to 10 each.
MCP servers 216 indexed · 25 profiled
Protocol-level reach: 36 of the 45 active platforms AgentSidekick tracks document support for MCP. DER Which platforms
| # | Product | Publisher | Capabilities | GitHub stars | Last push | Save and compare |
|---|---|---|---|---|---|---|
| 1 | AssetOpsBenchAssetOpsBench - Industry 4.0: A unified benchmark and framework for building, orchestrating, and evaluating domain-specific AI agents for Industry 4.0 asset operations and maintenance, with 460+ scenarios, 5 specialist agents (IoT, FMSR, TSFM, Work Order,...), and multi-agent orchestration blueprints (MetaAgent, AgentHive) over MCP. | IBM | 2.5k | today | ||
| 2 | ADRADR secures enterprise AI agents through observability, security benchmarking, and threat detection. Deployed at Uber. | uber | 2.1k | 1d ago | ||
| 3 | hotpath-cloudRead-only access to Rust profiling reports from CI: benchmark history, diffs and policies | hotpath.rs | 1.9k | today | ||
| 4 | Nautilus Compass MemoryLocal-first long-term memory for AI agents, with byte-recomputable signed verification receipts. | chunxiaoxx | 1.3k | 1d ago | ||
| 5 | aseprite-ai-artistYour coding agent paints pixel art in the Aseprite window you already have open. MCP server + skills for Claude Code, omp, Codex, Gemini CLI and Cursor — draws, looks at its work, animates and exports. Gallery and benchmark: pixeli.pebbly.space | with-pebbly | 718 | 12d ago | ||
| 6 | Polymarket Paper TraderPolymarket paper trading for AI agents: $10k paper money, real order books, official fees. | agent-next | 509 | 20d ago | ||
| 7 | mcpmarkMCPMark is a comprehensive, stress-testing MCP benchmark designed to evaluate model and agent capabilities in real-world MCP use. | eval-sys | 475 | 5mo ago | ||
| 8 | classifier.dev docsclassifier.dev documentation as tools: list, read and search the reference, benchmark and guides. | classifier.dev | 441 | 3d ago | ||
| 9 | Risk-Aware Trading AgentsAI trading agents for US stocks: point-in-time news backtests, a frozen 2026 market, paper trading | jackson-cmd | 356 | today | ||
| 10 | MCPBenchThe evaluation benchmark on MCP servers | modelscope | 252 | 1y ago |
Infrastructure 39 indexed · 39 profiled
No compatibility evidence is collected for this form.
| # | Product | Publisher | Capabilities | GitHub stars | Last push | Save and compare |
|---|---|---|---|---|---|---|
| 1 | chinese-llm-benchmark非线智能 NoneLinear - ReLE评测:中文AI大模型能力评测(持续更新):目前已囊括374个大模型,覆盖chatgpt、gpt-5.4、谷歌gemini-3.1-pro、Claude-4.6、文心ERNIE-X1.1、ERNIE-5.0、qwen3.6-max、qwen3.6-plus、百川、讯飞星火、商汤senseChat等商用模型, 以及step3.5-flash、kimi-k2.6、ernie4.5、MiniMax-M2.7、deepseek-v4、Qwen3.6、llama4、智谱GLM-5.1、MiMo-V2、LongCat、gemma4、mistral等开源大模型。不仅提供排行榜,也提供规模超200万的大模型缺陷库!方便广大社区研究分析、改进大模型。 | jeinlee1991 | 6.5k | 1d ago | ||
| 2 | lmms-evalOne-for-All Multimodal Evaluation Toolkit Across Text, Image, Video, and Audio Tasks | EvolvingLMMs-Lab | 4.5k | 1d ago | ||
| 3 | ClawBenchOpen-source benchmark for browser AI agents on daily tasks. | TIGER-AI-Lab | 1.3k | 16d ago | ||
| 4 | VectorDBBenchBenchmark for vector databases. | zilliztech | 1.2k | 3d ago | ||
| 5 | keystone-benchA clinical decision-evidence benchmark: pull the one fact an answer rests on, and see whether the assistant's action moves with the evidence. Built on HealthBench. | xinxuxin | 1.1k | 1mo ago | ||
| 6 | SkillCorpusOpen-source infrastructure that turns scattered SKILL.md files into curated, retrieval-ready agent-skill corpora—with retrieval and evaluation tooling included. | EverMind-AI | 706 | 2d ago | ||
| 7 | Awesome-LLM-EvalAwesome-LLM-Eval: a curated list of tools, datasets/benchmark, demos, leaderboard, papers, docs and models, mainly for Evaluation on LLMs. 一个由工具、基准/数据、演示、排行榜和大模型等组成的精选列表,主要面向基础大模型评测,旨在探求生成式AI的技术边界. | onejune2018 | 661 | 12mo ago | ||
| 8 | EnterpriseRAG-BenchDataset and benchmark for RAG on company internal documents. | onyx-dot-app | 639 | 2mo ago | ||
| 9 | TrustLLM[ICML 2024] TrustLLM: LLM trustworthiness evaluation across truthfulness, safety, fairness, robustness, privacy and ethics. Python/CLI toolkit for local models, compatible APIs and AI agent orchestration. | HowieHwong | 635 | 1mo ago | ||
| 10 | rsi_evalA lightweight Python SDK and CLI for evaluating recursive self-improvement (RSI) methods across agent benchmarks. | Danielxu0208 | 471 | 22d ago |
Agent skills 19 indexed · 19 profiled
Protocol-level reach: 26 of the 45 active platforms AgentSidekick tracks document support for the Agent Skills format. DER Which platforms
| # | Product | Publisher | Capabilities | GitHub stars | Last push | Save and compare |
|---|---|---|---|---|---|---|
| 1 | SkillEvaluatorMulti-tier framework for evaluating AI agent skills and plugins with quality gates, security scanning, harness compatibility checks, semantic overlap detection, synthetic evaluation dataset generation, and live agent evaluation that measures how skills affect agent behavior. Ships reusable CI templates that bring every check to any repository. | NVIDIA | 4.3k | today | ||
| 2 | evoturns your codebase into an autoresearch loop — discovers what to measure, instruments the benchmark, then runs tree search with parallel subagents. | evo-hq | 1.5k | 3d ago | ||
| 3 | Claude-Skills-Governance-Risk-and-ComplianceClaude Skills for Governance, Risk, & Compliance (GRC): Expert-level compliance guidance for ISO 27001, SOC 2, FedRAMP, GDPR, HIPAA, NIST CSF, PCI DSS, EU AI Act, ISO 42001, ISO 27701, DORA, CSRD, India's DPDPA, CMMC 2.0, NIST AI Risk, SWIFT, CCPA/CPRA, and others. Benchmark 92% (with skills) vs 53% (without skills). Updated Monthly. | Sushegaad | 1.1k | 5d ago | ||
| 4 | GLM-5.3-Flash-J-Space-Capability-Realization-ReportGLM-5.3-Flash × J-Space capability realization — benchmark presentation of the J-Space Cognition Suite | Tiger3807861189 | 1.0k | 2mo ago | ||
| 5 | PaperOrchestraAn automated AI research-paper writer based off Google's PaperOrchestra paper's implementation through a skills - benchmark + autoraters using any coding agent (Claude Code, Cursor, Antigravity, Cline, Aider). No API keys, no LLM SDKs. | Ar9av | 767 | 17d ago | ||
| 6 | SatoriClaude Code skill & open-source wisdom companion (v6.6.0, Apache 2.0) for CLI, IDE, and web chat. One lens per turn, local SQLite vault, 350-case benchmark (98.0% code non-activation). Install: npx skills add MetcalfSolutions/Satori. Built to be outgrown. | MetcalfSolutions | 124 | today | ||
| 7 | skill-creatorCreate new skills, modify and improve existing skills, and measure skill performance. Use when users want to create a skill from scratch, edit, or optimize an existing skill, run evals to test a skill, benchmark skill performance with variance analysis, or optimize a skill's description for better triggering accuracy. | anthropics | — | 5d ago | ||
| 8 | arborApplies Arbor Hypothesis Tree Refinement to research artifacts with repeatable evaluators, including model training, agent harnesses, data synthesis and benchmark optimization. Uses persistent hypotheses, isolated experiments, evidence propagation and held-out candidate comparison for multi-experiment research runs. Includes a standard-library state manager and guidance for the RUC-NLPIR Arbor CLI. | K-Dense-AI | — | 5d ago | ||
| 9 | skill-creatorCreate, edit, evaluate, and optimize skills, including benchmarking and description tuning for better triggering. | CherryHQ | — | today | ||
| 10 | arize-experimentCreates, runs, and analyzes Arize experiments for evaluating and comparing model performance. Covers experiment CRUD, exporting runs, comparing results, and evaluation workflows using the ax CLI. Use when the user mentions create experiment, run experiment, compare models, model performance, evaluate AI, experiment results, benchmark, A/B test models, or measure accuracy. | github | — | 1d ago |
Plugins 8 indexed · 6 profiled
6 of these are Claude plugins, which Claude Code documents (protocol-level). DER
| # | Product | Publisher | Capabilities | GitHub stars | Last push | Save and compare |
|---|---|---|---|---|---|---|
| 1 | agy-staffHire Google's Antigravity CLI (agy) as a fast Gemini staffer for Claude Code, OpenAI Codex, and DeepSeek Harness. | keli-wen | 905 | 2d ago | ||
| 2 | codspeedCodSpeed is the all-in-one performance testing toolkit. Dive into benchmarking results, flamegraphs, and performance comparisons — give Claude granular profiling context to pinpoint bottlenecks and autonomously iterate on performance via the CodSpeed MCP server. | codspeedhq | 395 | today | ||
| 3 | Hill-ClimbMake your coding agent prove "done". Each change is kept only if a locked test or benchmark scores it better. Claude Code and Codex skill. | Bil0000 | 144 | 8d ago | ||
| 4 | intuit-quickbooksQuickBooks business insights and actions: financial health briefings across P&L, cash flow, balance sheet and A/R, industry benchmarking, payroll cost analysis, overdue-invoice reminders, estimate/invoice drafting from email, payroll onboarding and base-pay changes (with confirmation), and QuickBooks Capital lending guidance. Requires the Intuit QuickBooks connector. | intuit | 1 | 2mo ago | ||
| 5 | skill-creatorCreate new skills, improve existing skills, and measure skill performance. Use when users want to create a skill from scratch, update or optimize an existing skill, run evals to test a skill, or benchmark skill performance with variance analysis. | anthropics | — | today | ||
| 6 | arckit-uk-gcloudUK G-Cloud Supplier Bid-Authoring Overlay — 15 supplier-side commands driving a G-Cloud 15 (RM1557.15) Digital Marketplace submission end to end: supplier profile, social value, lot questions, Procurement Act 2023 declaration, service design, Service Definition Documents (Lots 1a/1b/2a/2b/3), pricing, security evidence, competitor benchmark, submission review, and GCA submission packaging. Recipe: uk-gcloud-submission. Requires arckit core. Proprietary, experimental. | tractorjuice | — | 1d ago |
Rules packs 5 indexed · 2 profiled
No compatibility evidence is collected for this form.
| # | Product | Publisher | Capabilities | GitHub stars | Last push | Save and compare |
|---|---|---|---|---|---|---|
| 1 | AIGXOpen, MIT context format for AI coding agents: centralized .aigx/ rules + a per-file boundary index that targets the right rules, forbidden imports, and gotchas to each file. The only context format validated in a controlled benchmark. Tool-agnostic; nothing injected into your source. | Lolner95 | 114 | 2mo ago | ||
| 2 | skillcheckBlind A/B testing for AI agent skills, .cursorrules, CLAUDE.md, and system prompts. Detects placebo and harmful instructions. | sx4im | 31 | 4d ago |
Agent projects 4 indexed · 4 profiled
No compatibility evidence is collected for this form.
| # | Product | Publisher | Capabilities | GitHub stars | Last push | Save and compare |
|---|---|---|---|---|---|---|
| 1 | artemisARTEMIS turns natural-language instructions into reliable Android automation. It automates end-to-end workflows, captures logs, and integrates seamlessly with AI coding assistants such as Antigravity, Codex, and Claude Code. It also achieves 99%+ success rate on AndroidWorld Benchmark. | 18k | 1mo ago | |||
| 2 | tokscale🛰️ Track token usage across AI coding agents from your terminal. 🏅 Global leaderboard with trillions of tokens tracked. | junhoyeo | 5.9k | 3d ago | ||
| 3 | all-agentic-architectures35 production-grade agentic AI architectures (Reflexion, LATS, GraphRAG, MemGPT, Voyager, BrowserAgent, ...) — a Python library and runnable textbook with multi-provider LLM support and a 17-task benchmark leaderboard. | FareedKhan-dev | 4.8k | 5d ago | ||
| 4 | little-coderA harness optimized to smaller LLMs | itayinbarr | 2.7k | 2mo ago |
Frameworks 4 indexed · 4 profiled
No compatibility evidence is collected for this form.
| # | Product | Publisher | Capabilities | GitHub stars | Last push | Save and compare |
|---|---|---|---|---|---|---|
| 1 | enterprise-deep-researchSalesforce Enterprise Deep Research | SalesforceAIResearch | 1.2k | 25d ago | ||
| 2 | VectorizedMultiAgentSimulatorVMAS is a vectorized differentiable simulator designed for efficient Multi-Agent Reinforcement Learning benchmarking. It is comprised of a vectorized 2D physics engine written in PyTorch and a set of challenging multi-robot scenarios. Additional scenarios can be implemented through a simple and modular interface. | proroklab | 631 | 6mo ago | ||
| 3 | meta-agents-research-environmentsMeta Agents Research Environments is a comprehensive platform designed to evaluate AI agents in dynamic, realistic scenarios. Unlike static benchmarks, this platform introduces evolving environments where agents must adapt their strategies as new information becomes available, mirroring real-world challenges. | facebookresearch | 581 | 1mo ago | ||
| 4 | crab🦀️ CRAB: Cross-environment Agent Benchmark for Multimodal Language Model Agents. https://crab.camel-ai.org/ | camel-ai | 435 | 5d ago |
Libraries 2 indexed · 2 profiled
No compatibility evidence is collected for this form.
| # | Product | Publisher | Capabilities | GitHub stars | Last push | Save and compare |
|---|---|---|---|---|---|---|
| 1 | agentweavePre-inference tool and agent routing for MCP, A2A and tool-rich LLM systems. Reduce candidate tools before model inference with policy-aware routing and recovery. | sauravsingla | 42 | 1mo ago | ||
| 2 | red-team-blue-team-agent-fabricAI agent security harness, adversarial testing: 661 executable tests in the v4.31.1 release, across MCP, A2A, x402/L402, decision governance, benchmark integrity, human-in-the-loop, skill supply chain. Five target shapes: a verdict must be able to be wrong AND to be right. OWASP Agentic v1.1 T1-T17, AIUC-1 crosswalk (2026-10-15), NIST AI 800-2. | msaleme | 34 | today |
Which platforms these forms can reach
All platformsProtocol-level: the platform documents support for the standard. No specific pairing has been tested.
MCP servers 216 indexed here
36 platforms document MCP support CUR.
ContinueOpenCodeClaude CodeCodexGemini CLIZedOpenHandsClineWarpgooseQwen CodeBolt (bolt.new)KiroAmazon Bedrock AgentCoreAmpApps in ChatGPT / Apps SDK (now ChatGPT Plugins)Augment CodeChatGPTClaude (claude.ai apps)Claude DesktopCursorDevinDevin Desktop (formerly Windsurf)Factory (Droid)Gemini (app)Gemini Enterprise Agent Platform (formerly Vertex AI / Vertex AI Agent Builder)GitHub CopilotGitHub Copilot cloud agent (formerly Copilot coding agent)Google AntigravityJetBrains AI AssistantJunieLovableMicrosoft Copilot StudioReplit AgentTabninev0
Agent skills 19 indexed here
26 platforms document Agent Skills support CUR.
OpenClawOpenCodeClaude CodeCodexGemini CLIOpenHandsWarpgooseBolt (bolt.new)KiroAmpApps in ChatGPT / Apps SDK (now ChatGPT Plugins)ChatGPTClaude (claude.ai apps)Claude DesktopCursorDevinDevin Desktop (formerly Windsurf)Factory (Droid)Gemini (app)GitHub CopilotGitHub Copilot cloud agent (formerly Copilot coding agent)Google AntigravityJunieReplit AgentTabnine
Platforms publishers name
| Platform | Profiled products naming it |
|---|---|
| Claude Code | 7 |
| Codex | 5 |
| Google Antigravity | 3 |
| Cursor | 2 |
| ChatGPT | 1 |
| Gemini CLI | 1 |
| Aider | 1 |
| Cline | 1 |
| OpenClaw | 1 |
| OpenCode | 1 |
Integrations most often paired
Third-party services named by products with this capability, whole catalogue.
| Integration | Products | Share |
|---|---|---|
| GitHub | 4 | 1.3% |
| AWS | 1 | 0.3% |
| Android | 1 | 0.3% |
| Google Cloud | 1 | 0.3% |
| Kubernetes | 1 | 0.3% |
| 1 | 0.3% | |
| MySQL | 1 | 0.3% |
| PostgreSQL | 1 | 0.3% |
| QuickBooks | 1 | 0.3% |
| Salesforce | 1 | 0.3% |
| Solana | 1 | 0.3% |
| Stripe | 1 | 0.3% |
Keywords that trigger this capability
Matched as whole words, ignoring case, in a product's name, description and repository topics.
| Keyword | Products matched |
|---|---|
| benchmark | 193 |
| leaderboard | 53 |
| swe-bench | 0 |
| model comparison | 7 |
| benchmarking | 38 |
| humaneval | 6 |
Segment statistics
Whole catalogue (297 indexed products). Calculated 2026-11-21.
Crowded, slowing. Among the largest fifth of capabilities, but its launches grew more slowly than the catalogue’s. Size is ranked against all capabilities; momentum compares launches in the last 90 days with the 90 before, relative to the catalogue’s own ratio (62 or more = faster, 38 or less = slower; under 5 dated launches it is not measured). The HHI of 0.077 means stars are spread as if across about 13 equally popular projects — unconcentrated by the common 0.15 / 0.25 thresholds. Stars measure attention, not usage or quality.
Limits
- Classification is by the keywords above. A product that offers this capability without using one of them is not counted; one that uses a keyword in another sense is counted wrongly. Methodology
- Platform reach is protocol-level: it says a platform documents the standard, not that any product here was tested on it. Platforms named by publishers are their own claims.
- Stars belong to repositories. Products in a subfolder of a large repository show that whole repository’s stars and are excluded from the concentration figures.
- Launch counts use each source’s publication date; for auto-discovered repositories that is the repository’s creation date.
Page data last updated .