AgentSidekick

Benchmarking

Comparing models and agents on standard benchmarks. As of 2026-11-21, 297 indexed products offer it (101 with full profiles), in 8 forms: 216 MCP servers, 39 Infrastructure, 19 Agent skills, 8 Plugins. 139 were published in the last 90 days, against 76 in the 90 days before. artemis has the most GitHub stars, 22% of the total across 198 products with their own repository.

CapabilitySegment label: Crowded, slowing DER

Available as

All in the directory

Profiled products grouped by product form. Within each form: own-repository products by GitHub stars, then products in a subfolder of a larger repository (they show that repository's stars), up to 10 each.

MCP servers 216 indexed · 25 profiled

Protocol-level reach: 36 of the 45 active platforms AgentSidekick tracks document support for MCP. DER Which platforms

Benchmarking: MCP servers by GitHub stars
#ProductPublisherCapabilities GitHub stars Last push Save and compare
1AssetOpsBenchAssetOpsBench - Industry 4.0: A unified benchmark and framework for building, orchestrating, and evaluating domain-specific AI agents for Industry 4.0 asset operations and maintenance, with 460+ scenarios, 5 specialist agents (IoT, FMSR, TSFM, Work Order,...), and multi-agent orchestration blueprints (MetaAgent, AgentHive) over MCP.IBM2.5ktoday
2ADRADR secures enterprise AI agents through observability, security benchmarking, and threat detection. Deployed at Uber.uber2.1k1d ago
3hotpath-cloudRead-only access to Rust profiling reports from CI: benchmark history, diffs and policieshotpath.rs1.9ktoday
4Nautilus Compass MemoryLocal-first long-term memory for AI agents, with byte-recomputable signed verification receipts.chunxiaoxx1.3k1d ago
5aseprite-ai-artistYour coding agent paints pixel art in the Aseprite window you already have open. MCP server + skills for Claude Code, omp, Codex, Gemini CLI and Cursor — draws, looks at its work, animates and exports. Gallery and benchmark: pixeli.pebbly.spacewith-pebbly71812d ago
6Polymarket Paper TraderPolymarket paper trading for AI agents: $10k paper money, real order books, official fees.agent-next50920d ago
7mcpmarkMCPMark is a comprehensive, stress-testing MCP benchmark designed to evaluate model and agent capabilities in real-world MCP use.eval-sys4755mo ago
8classifier.dev docsclassifier.dev documentation as tools: list, read and search the reference, benchmark and guides.classifier.dev4413d ago
9Risk-Aware Trading AgentsAI trading agents for US stocks: point-in-time news backtests, a frozen 2026 market, paper tradingjackson-cmd356today
10MCPBenchThe evaluation benchmark on MCP serversmodelscope2521y ago

All 25 profiled MCP servers with this capability

Infrastructure 39 indexed · 39 profiled

No compatibility evidence is collected for this form.

Benchmarking: Infrastructure by GitHub stars
#ProductPublisherCapabilities GitHub stars Last push Save and compare
1chinese-llm-benchmark非线智能 NoneLinear - ReLE评测:中文AI大模型能力评测(持续更新):目前已囊括374个大模型,覆盖chatgpt、gpt-5.4、谷歌gemini-3.1-pro、Claude-4.6、文心ERNIE-X1.1、ERNIE-5.0、qwen3.6-max、qwen3.6-plus、百川、讯飞星火、商汤senseChat等商用模型, 以及step3.5-flash、kimi-k2.6、ernie4.5、MiniMax-M2.7、deepseek-v4、Qwen3.6、llama4、智谱GLM-5.1、MiMo-V2、LongCat、gemma4、mistral等开源大模型。不仅提供排行榜,也提供规模超200万的大模型缺陷库!方便广大社区研究分析、改进大模型。jeinlee19916.5k1d ago
2lmms-evalOne-for-All Multimodal Evaluation Toolkit Across Text, Image, Video, and Audio TasksEvolvingLMMs-Lab4.5k1d ago
3ClawBenchOpen-source benchmark for browser AI agents on daily tasks.TIGER-AI-Lab1.3k16d ago
4VectorDBBenchBenchmark for vector databases.zilliztech1.2k3d ago
5keystone-benchA clinical decision-evidence benchmark: pull the one fact an answer rests on, and see whether the assistant's action moves with the evidence. Built on HealthBench.xinxuxin1.1k1mo ago
6SkillCorpusOpen-source infrastructure that turns scattered SKILL.md files into curated, retrieval-ready agent-skill corpora—with retrieval and evaluation tooling included.EverMind-AI7062d ago
7Awesome-LLM-EvalAwesome-LLM-Eval: a curated list of tools, datasets/benchmark, demos, leaderboard, papers, docs and models, mainly for Evaluation on LLMs. 一个由工具、基准/数据、演示、排行榜和大模型等组成的精选列表,主要面向基础大模型评测,旨在探求生成式AI的技术边界.onejune201866112mo ago
8EnterpriseRAG-BenchDataset and benchmark for RAG on company internal documents.onyx-dot-app6392mo ago
9TrustLLM[ICML 2024] TrustLLM: LLM trustworthiness evaluation across truthfulness, safety, fairness, robustness, privacy and ethics. Python/CLI toolkit for local models, compatible APIs and AI agent orchestration.HowieHwong6351mo ago
10rsi_evalA lightweight Python SDK and CLI for evaluating recursive self-improvement (RSI) methods across agent benchmarks.Danielxu020847122d ago

All 39 profiled Infrastructure with this capability

Agent skills 19 indexed · 19 profiled

Protocol-level reach: 26 of the 45 active platforms AgentSidekick tracks document support for the Agent Skills format. DER Which platforms

Benchmarking: Agent skills by GitHub stars
#ProductPublisherCapabilities GitHub stars Last push Save and compare
1SkillEvaluatorMulti-tier framework for evaluating AI agent skills and plugins with quality gates, security scanning, harness compatibility checks, semantic overlap detection, synthetic evaluation dataset generation, and live agent evaluation that measures how skills affect agent behavior. Ships reusable CI templates that bring every check to any repository.NVIDIA4.3ktoday
2evoturns your codebase into an autoresearch loop — discovers what to measure, instruments the benchmark, then runs tree search with parallel subagents.evo-hq1.5k3d ago
3Claude-Skills-Governance-Risk-and-ComplianceClaude Skills for Governance, Risk, & Compliance (GRC): Expert-level compliance guidance for ISO 27001, SOC 2, FedRAMP, GDPR, HIPAA, NIST CSF, PCI DSS, EU AI Act, ISO 42001, ISO 27701, DORA, CSRD, India's DPDPA, CMMC 2.0, NIST AI Risk, SWIFT, CCPA/CPRA, and others. Benchmark 92% (with skills) vs 53% (without skills). Updated Monthly.Sushegaad1.1k5d ago
4GLM-5.3-Flash-J-Space-Capability-Realization-ReportGLM-5.3-Flash × J-Space capability realization — benchmark presentation of the J-Space Cognition SuiteTiger38078611891.0k2mo ago
5PaperOrchestraAn automated AI research-paper writer based off Google's PaperOrchestra paper's implementation through a skills - benchmark + autoraters using any coding agent (Claude Code, Cursor, Antigravity, Cline, Aider). No API keys, no LLM SDKs.Ar9av76717d ago
6SatoriClaude Code skill & open-source wisdom companion (v6.6.0, Apache 2.0) for CLI, IDE, and web chat. One lens per turn, local SQLite vault, 350-case benchmark (98.0% code non-activation). Install: npx skills add MetcalfSolutions/Satori. Built to be outgrown.MetcalfSolutions124today
7skill-creatorCreate new skills, modify and improve existing skills, and measure skill performance. Use when users want to create a skill from scratch, edit, or optimize an existing skill, run evals to test a skill, benchmark skill performance with variance analysis, or optimize a skill's description for better triggering accuracy.anthropics—5d ago
8arborApplies Arbor Hypothesis Tree Refinement to research artifacts with repeatable evaluators, including model training, agent harnesses, data synthesis and benchmark optimization. Uses persistent hypotheses, isolated experiments, evidence propagation and held-out candidate comparison for multi-experiment research runs. Includes a standard-library state manager and guidance for the RUC-NLPIR Arbor CLI.K-Dense-AI—5d ago
9skill-creatorCreate, edit, evaluate, and optimize skills, including benchmarking and description tuning for better triggering.CherryHQ—today
10arize-experimentCreates, runs, and analyzes Arize experiments for evaluating and comparing model performance. Covers experiment CRUD, exporting runs, comparing results, and evaluation workflows using the ax CLI. Use when the user mentions create experiment, run experiment, compare models, model performance, evaluate AI, experiment results, benchmark, A/B test models, or measure accuracy.github—1d ago

All 19 profiled Agent skills with this capability

Plugins 8 indexed · 6 profiled

6 of these are Claude plugins, which Claude Code documents (protocol-level). DER

Benchmarking: Plugins by GitHub stars
#ProductPublisherCapabilities GitHub stars Last push Save and compare
1agy-staffHire Google's Antigravity CLI (agy) as a fast Gemini staffer for Claude Code, OpenAI Codex, and DeepSeek Harness.keli-wen9052d ago
2codspeedCodSpeed is the all-in-one performance testing toolkit. Dive into benchmarking results, flamegraphs, and performance comparisons — give Claude granular profiling context to pinpoint bottlenecks and autonomously iterate on performance via the CodSpeed MCP server.codspeedhq395today
3Hill-ClimbMake your coding agent prove "done". Each change is kept only if a locked test or benchmark scores it better. Claude Code and Codex skill.Bil00001448d ago
4intuit-quickbooksQuickBooks business insights and actions: financial health briefings across P&L, cash flow, balance sheet and A/R, industry benchmarking, payroll cost analysis, overdue-invoice reminders, estimate/invoice drafting from email, payroll onboarding and base-pay changes (with confirmation), and QuickBooks Capital lending guidance. Requires the Intuit QuickBooks connector.intuit12mo ago
5skill-creatorCreate new skills, improve existing skills, and measure skill performance. Use when users want to create a skill from scratch, update or optimize an existing skill, run evals to test a skill, or benchmark skill performance with variance analysis.anthropics—today
6arckit-uk-gcloudUK G-Cloud Supplier Bid-Authoring Overlay — 15 supplier-side commands driving a G-Cloud 15 (RM1557.15) Digital Marketplace submission end to end: supplier profile, social value, lot questions, Procurement Act 2023 declaration, service design, Service Definition Documents (Lots 1a/1b/2a/2b/3), pricing, security evidence, competitor benchmark, submission review, and GCA submission packaging. Recipe: uk-gcloud-submission. Requires arckit core. Proprietary, experimental.tractorjuice—1d ago

Rules packs 5 indexed · 2 profiled

No compatibility evidence is collected for this form.

Benchmarking: Rules packs by GitHub stars
#ProductPublisherCapabilities GitHub stars Last push Save and compare
1AIGXOpen, MIT context format for AI coding agents: centralized .aigx/ rules + a per-file boundary index that targets the right rules, forbidden imports, and gotchas to each file. The only context format validated in a controlled benchmark. Tool-agnostic; nothing injected into your source.Lolner951142mo ago
2skillcheckBlind A/B testing for AI agent skills, .cursorrules, CLAUDE.md, and system prompts. Detects placebo and harmful instructions.sx4im314d ago

Agent projects 4 indexed · 4 profiled

No compatibility evidence is collected for this form.

Benchmarking: Agent projects by GitHub stars
#ProductPublisherCapabilities GitHub stars Last push Save and compare
1artemisARTEMIS turns natural-language instructions into reliable Android automation. It automates end-to-end workflows, captures logs, and integrates seamlessly with AI coding assistants such as Antigravity, Codex, and Claude Code. It also achieves 99%+ success rate on AndroidWorld Benchmark.google18k1mo ago
2tokscale🛰️ Track token usage across AI coding agents from your terminal. 🏅 Global leaderboard with trillions of tokens tracked.junhoyeo5.9k3d ago
3all-agentic-architectures35 production-grade agentic AI architectures (Reflexion, LATS, GraphRAG, MemGPT, Voyager, BrowserAgent, ...) — a Python library and runnable textbook with multi-provider LLM support and a 17-task benchmark leaderboard.FareedKhan-dev4.8k5d ago
4little-coderA harness optimized to smaller LLMsitayinbarr2.7k2mo ago

Frameworks 4 indexed · 4 profiled

No compatibility evidence is collected for this form.

Benchmarking: Frameworks by GitHub stars
#ProductPublisherCapabilities GitHub stars Last push Save and compare
1enterprise-deep-researchSalesforce Enterprise Deep ResearchSalesforceAIResearch1.2k25d ago
2VectorizedMultiAgentSimulatorVMAS is a vectorized differentiable simulator designed for efficient Multi-Agent Reinforcement Learning benchmarking. It is comprised of a vectorized 2D physics engine written in PyTorch and a set of challenging multi-robot scenarios. Additional scenarios can be implemented through a simple and modular interface.proroklab6316mo ago
3meta-agents-research-environmentsMeta Agents Research Environments is a comprehensive platform designed to evaluate AI agents in dynamic, realistic scenarios. Unlike static benchmarks, this platform introduces evolving environments where agents must adapt their strategies as new information becomes available, mirroring real-world challenges.facebookresearch5811mo ago
4crab🦀️ CRAB: Cross-environment Agent Benchmark for Multimodal Language Model Agents. https://crab.camel-ai.org/camel-ai4355d ago

Libraries 2 indexed · 2 profiled

No compatibility evidence is collected for this form.

Benchmarking: Libraries by GitHub stars
#ProductPublisherCapabilities GitHub stars Last push Save and compare
1agentweavePre-inference tool and agent routing for MCP, A2A and tool-rich LLM systems. Reduce candidate tools before model inference with policy-aware routing and recovery.sauravsingla421mo ago
2red-team-blue-team-agent-fabricAI agent security harness, adversarial testing: 661 executable tests in the v4.31.1 release, across MCP, A2A, x402/L402, decision governance, benchmark integrity, human-in-the-loop, skill supply chain. Five target shapes: a verdict must be able to be wrong AND to be right. OWASP Agentic v1.1 T1-T17, AIUC-1 crosswalk (2026-10-15), NIST AI 800-2.msaleme34today

Which platforms these forms can reach

All platforms

Protocol-level: the platform documents support for the standard. No specific pairing has been tested.

Platforms publishers name

Integrations most often paired

Third-party services named by products with this capability, whole catalogue.

IntegrationProducts Share
GitHub41.3%
AWS10.3%
Android10.3%
Google Cloud10.3%
Kubernetes10.3%
LinkedIn10.3%
MySQL10.3%
PostgreSQL10.3%
QuickBooks10.3%
Salesforce10.3%
Solana10.3%
Stripe10.3%

Keywords that trigger this capability

Matched as whole words, ignoring case, in a product's name, description and repository topics.

KeywordProducts matched
benchmark193
leaderboard53
swe-bench0
model comparison7
benchmarking38
humaneval6

Segment statistics

Whole catalogue (297 indexed products). Calculated 2026-11-21.

297DERIndexed products101 with full profiles
139DERLaunched, last 90 days76 in the 90 days before
25DERMomentum vs catalogue50 = same pace as the catalogue
22%DERLargest project's share of starsAcross 198 products with their own repository
0.077DERConcentration (HHI)About 13 equal-sized projects’ worth; unconcentrated
3SRCMedian starsPer product with its own repository
58%DEROpen sourceMarked open source; unknown counts as not

Crowded, slowing. Among the largest fifth of capabilities, but its launches grew more slowly than the catalogue’s. Size is ranked against all capabilities; momentum compares launches in the last 90 days with the 90 before, relative to the catalogue’s own ratio (62 or more = faster, 38 or less = slower; under 5 dated launches it is not measured). The HHI of 0.077 means stars are spread as if across about 13 equally popular projects — unconcentrated by the common 0.15 / 0.25 thresholds. Stars measure attention, not usage or quality.

Limits

  • Classification is by the keywords above. A product that offers this capability without using one of them is not counted; one that uses a keyword in another sense is counted wrongly. Methodology
  • Platform reach is protocol-level: it says a platform documents the standard, not that any product here was tested on it. Platforms named by publishers are their own claims.
  • Stars belong to repositories. Products in a subfolder of a large repository show that whole repository’s stars and are excluded from the concentration figures.
  • Launch counts use each source’s publication date; for auto-discovered repositories that is the repository’s creation date.

Page data last updated .