AgentSidekick

Agent Evaluation

Scoring, benchmarking and red-teaming models and agents. As of 2026-11-21, 507 indexed products match this subcategory (224 with full profiles); 215 were published in the last 90 days, against 120 in the 90 days before, slower than the catalogue as a whole. By GitHub stars, iFixAi leads, holding 7% of the stars of the 390 products here with their own repository. The most common form is MCP servers (64% of indexed products).

SubcategorySegment label: Lagging the catalogue DER

Segment statistics

Whole catalogue (all 507 indexed products, not only profiled ones). Calculated 2026-11-21.

507DERIndexed products224 with full profiles
215DERLaunched, last 90 days120 in the 90 days before
24DERMomentum vs catalogue50 = same pace as the catalogue
7%DERLargest project's share of starsAcross 390 products with their own repository
0.034DERConcentration (HHI)About 30 equal-sized projects’ worth; unconcentrated
11SRCMedian starsPer product with its own repository
67%DEROpen sourceMarked open source; unknown counts as not

Lagging the catalogue. Mid-sized for its level, with launches growing more slowly than the catalogue’s. Size is ranked against the other subcategory nodes; momentum compares this node’s launches in the last 90 days with the 90 before, relative to the whole catalogue’s ratio (score 62 or more = faster, 38 or less = slower; below 5 dated launches it is not measured and the label uses size alone).

Concentration. The Herfindahl–Hirschman index (HHI) adds up each project’s squared share of stars. At 0.034, attention is spread as if across about 30 equally popular projects. Using the common competition-policy thresholds (0.15 and 0.25), that is unconcentrated. Stars measure attention, not usage or quality.

Which product forms serve it

The same need is met by MCP servers, agent skills, plugins, extensions and frameworks. This counts each form across the whole catalogue.

Products in Agent Evaluation by product type, as of 2026-11-21
Product type IndexedShareWith a profile
MCP servers32564.1%48
Infrastructure9819.3%98
Agent skills316.1%31
Developer tools112.2%11
Frameworks102.0%10
Plugins91.8%7
Libraries91.8%9
Rules packs61.2%2
Agent projects61.2%6
SaaS products20.39%2

3 capabilities

Indexed = whole catalogue; profiled = products with their own page. A product can sit in several, so the rows do not add up to the total.

CapabilityIndexed ProfiledNew, 90 daysLabel
BenchmarkingComparing models and agents on standard benchmarks.297101139Crowded, slowing
EvaluationRunning eval suites to grade outputs.985230Lagging the catalogue
Red TeamingAdversarial testing for safety and robustness.28218Thin

Top 40 products by GitHub stars

Search the directory

Profiled products only (224 of 507), ranked by the stars of their own repository. Products that live in a subfolder of a larger repository (common for skills) show that whole repository's stars and are ranked after them; products without a repository come last. Stars are popularity, not quality or installs.

Agent Evaluation: top profiled products by GitHub stars, 2026-11-21
#ProductTypePublisherCapabilities GitHub stars Last push LicenceSave and compare
1iFixAiIndependent Auditing of AI Agents. Run by human or the agent itself, to answer the most crucial question in the AI Agent Economy. Is the agent doing what is supposed to do? With iFixAi you can have this answer in less than 120 seconds.Infrastructureifixai-ai47k1d agoApache-2.0
2agentsPlugins, agents, skills, and commands for Claude Code, Codex CLI, Cursor, OpenCode, Antigravity CLI, GitHub Copilot, and Pi, written once in Markdown and adapted to each tool by generators with validation and evals.Agent skillwshobson43k3d agoMIT
3LightRAGGraph-enhanced RAG system that indexes entities and relations alongside vectors for dual-level retrieval, with a server and web UI.LibraryHKU Data Intelligence Lab41ktodayMIT
4LangfuseOpen-source LLM engineering platform for tracing, prompt management, evaluations and datasets, self-hosted or Langfuse Cloud.Developer toolClickHouse (Langfuse)38ktodayMIT
5MastraTypeScript framework for agents, workflows, RAG, memory and evals, with a local dev playground and deploy targets for common hosts.FrameworkMastra30ktodayApache-2.0
6MLflowOpen-source ML/GenAI lifecycle platform whose GenAI features include OpenTelemetry-compatible tracing, evaluation and prompt registry for agents.Developer toolDatabricks (Linux Foundation)29ktodayApache-2.0
7promptfooCLI and library for declarative prompt/model test suites, side-by-side comparisons and automated red-teaming of LLM apps and agents.Developer toolOpenAI (Promptfoo)28ktodayMIT
8OpikOpen-source platform for tracing, evaluating and monitoring LLM apps and agents, with LLM-as-judge metrics and prompt optimisation.Developer toolComet23ktodayApache-2.0
9OpenAI EvalsFramework and registry of benchmarks for evaluating LLMs; OpenAI also offers a hosted Evals API in its platform.Frameworkopenai20k7mo agoMIT
10DeepEvalPytest-style framework for evaluating LLM outputs, RAG pipelines and agents with research-backed metrics and synthetic test generation.Libraryconfident-ai20k1d agoApache-2.0
11deepevalSkills for adding DeepEval evaluations, tracing, datasets, Confident AI reports, and iterative improvement loops to AI applications.Pluginconfident-ai20k1d agoApache-2.0
12artemisARTEMIS turns natural-language instructions into reliable Android automation. It automates end-to-end workflows, captures logs, and integrates seamlessly with AI coding assistants such as Antigravity, Codex, and Claude Code. It also achieves 99%+ success rate on AndroidWorld Benchmark.Agent projectgoogle18k1mo agoApache-2.0
13RagasLibrary of metrics and test-set generation for evaluating RAG and agent applications, including faithfulness and context precision.LibraryVibrant Labs16k7d agoApache-2.0
14Arize PhoenixSelf-hostable tracing and evaluation tool built on OpenTelemetry/OpenInference for inspecting LLM and agent runs, datasets and experiments.Developer toolArize-ai12ktodayElastic-2.0
15garakLLM vulnerability scanner that probes models and apps for jailbreaks, prompt injection, data leakage and toxicity.Developer toolNVIDIA10k3d agoApache-2.0
16failproofaiObservability and enforcement for AI agent harnesses. Capture every run and runtime reliability with policy enforcement. InfrastructureFailproofAI9.9ktoday—
17oumiEasily fine-tune (SFT/RL), evaluate, and deploy Qwen, Gemma, or any open agentic LLM/VLM!Infrastructureoumi-ai9.4ktodayApache-2.0
18AI-Infra-GuardA full-stack AI Red Teaming platform securing AI ecosystems via Agent Scan, Skills Scan, MCP scan, AI Infra scan and LLM jailbreak evaluation.InfrastructureTencent7.6k1d agoApache-2.0
19gepaOptimize prompts, code, and more with AI-powered Reflective OptimizationAgent skillgepa-ai7.5ktodayMIT
20chinese-llm-benchmark非线智能 NoneLinear - ReLE评测:中文AI大模型能力评测(持续更新):目前已囊括374个大模型,覆盖chatgpt、gpt-5.4、谷歌gemini-3.1-pro、Claude-4.6、文心ERNIE-X1.1、ERNIE-5.0、qwen3.6-max、qwen3.6-plus、百川、讯飞星火、商汤senseChat等商用模型, 以及step3.5-flash、kimi-k2.6、ernie4.5、MiniMax-M2.7、deepseek-v4、Qwen3.6、llama4、智谱GLM-5.1、MiMo-V2、LongCat、gemma4、mistral等开源大模型。不仅提供排行榜,也提供规模超200万的大模型缺陷库!方便广大社区研究分析、改进大模型。Infrastructurejeinlee19916.5k1d ago—
21OuroborosPins an acceptance spec; the verify command and expected output never enter the success contract.MCP serverQ006.4ktodayMIT
22HeliconeProxy-based LLM observability that logs requests, costs and latency with one-line integration, plus an AI gateway.Developer toolMintlify (Helicone)6.3k4d agoApache-2.0
23AgentOpsSDK and dashboard for monitoring agent sessions, tool calls, costs and failures, with integrations for CrewAI, OpenAI Agents SDK, AG2 and others.Developer toolAgentOps6.0k4mo agoMIT
24GiskardTesting library that scans LLM agents and RAG apps for hallucination, prompt injection and harmful-content vulnerabilities and generates test suites.LibraryGiskard6.0k1d agoApache-2.0
25tokscale🛰️ Track token usage across AI coding agents from your terminal. 🏅 Global leaderboard with trillions of tokens tracked.Agent projectjunhoyeo5.9k3d agoMIT
26coze-loopNext-generation AI Agent Optimization Platform: Cozeloop addresses challenges in AI agent development by providing full-lifecycle management capabilities from development, debugging, and evaluation to monitoring. Infrastructurecoze-dev5.8k4d agoApache-2.0
27LLM-Engineers-HandbookThe LLM's practical guide: From the fundamentals to deploying advanced LLM and RAG apps to AWS using LLMOps best practicesInfrastructurePacktPublishing5.4k7mo agoMIT
28awesome-harness-engineeringAwesome list for AI agent harness engineering: tools, patterns, evals, memory, MCP, permissions, observability, and orchestration.Agent projectai-boost5.2ktoday—
29PyRITPython Risk Identification Tool for automated red-teaming of generative AI systems with attack orchestrators and scorers.Developer toolmicrosoft5.0ktodayMIT
30AgentaOpen-source LLMOps platform for prompt management, evaluation and observability with a playground for comparing variants.Developer toolAgenta4.9ktodayMIT
31all-agentic-architectures35 production-grade agentic AI architectures (Reflexion, LATS, GraphRAG, MemGPT, Voyager, BrowserAgent, ...) — a Python library and runnable textbook with multi-provider LLM support and a 17-task benchmark leaderboard.Agent projectFareedKhan-dev4.8k5d agoMIT
32Pydantic LogfireOpenTelemetry-based observability service with first-class tracing for LLM calls, Pydantic AI agents and general Python apps, queryable with SQL.SaaS productpydantic4.6ktodayMIT
33lmms-evalOne-for-All Multimodal Evaluation Toolkit Across Text, Image, Video, and Audio TasksInfrastructureEvolvingLMMs-Lab4.5k1d ago—
34SkillEvaluatorMulti-tier framework for evaluating AI agent skills and plugins with quality gates, security scanning, harness compatibility checks, semantic overlap detection, synthetic evaluation dataset generation, and live agent evaluation that measures how skills affect agent behavior. Ships reusable CI templates that bring every check to any repository.Agent skillNVIDIA4.3ktodayApache-2.0
35AI-Engineer-HeadquartersA collection of scientific methods, processes, algorithms, and systems to build stories & models.Infrastructurehemansnation3.8k24d ago—
36TruLensLibrary for instrumenting and evaluating LLM apps and agents with feedback functions such as groundedness and relevance.LibrarySnowflake3.7k1d agoMIT
37LaminarOpen-source observability platform for agents with tracing, browser-session recordings, evaluations and SQL over trace data.Developer toolLaminar3.5k1d agoApache-2.0
38InspectPython framework for LLM and agent evaluations with datasets, solvers, scorers, sandboxed tool execution and a log viewer.FrameworkUK AI Security Institute3.3k1d agoMIT
39little-coderA harness optimized to smaller LLMsAgent projectitayinbarr2.7k2mo agoApache-2.0
40generative-aiComprehensive resources on Generative AI, including a detailed roadmap, projects, use cases, interview preparation, and coding preparation.MCP servergenieincodebottle2.7k12d agoMIT

Most common integrations

Third-party services named by products in this node, whole catalogue.

IntegrationProducts Share
GitHub51.0%
PostgreSQL30.6%
AWS20.4%
Android10.2%
Discord10.2%
Google Cloud10.2%
Kubernetes10.2%
LinkedIn10.2%
MySQL10.2%
Playwright10.2%
QuickBooks10.2%
Salesforce10.2%
Solana10.2%
Stripe10.2%
n8n10.2%

Launches per month

New Agent Evaluation products per monthSRC

Whole catalogue, by publication month, 2024-11 to 2026-11. The last month is partial; undated products are excluded.

New Agent Evaluation products per monthWhole catalogue, by publication month, 2024-11 to 2026-11. The last month is partial; undated products are excluded.. Products published: 24 points, latest 72 products in 2026-11.
Show the data as a table
New Agent Evaluation products per month. Whole catalogue, by publication month, 2024-11 to 2026-11. The last month is partial; undated products are excluded.
Products published (products)
2024-112
2024-122
2025-012
2025-022
2025-043
2025-053
2025-063
2025-072
2025-081
2025-094
2025-102
2025-112
2025-123
2026-013
2026-029
2026-0317
2026-0417
2026-0523
2026-0626
2026-0740
2026-0867
2026-0941
2026-1086
2026-1172

Related opportunities

Opportunity radar

Scored gaps for this node or anything beneath it. Scores rank hypotheses; they do not predict demand.

OpportunityCompetitionConfidenceScore
Coverage gap: Red TeamingRed TeamingHighLow58.1

Limits of these numbers

  • Products are placed here by whole-word keyword rules on their name, description and repository topics, and inherit every ancestor of a matched node. Products that do this without naming it are missed; ambiguous words cause false matches. Methodology
  • About half of the catalogue matches no node at all, so absolute counts understate every category.
  • Launch counts use the publication date each source reports; for auto-discovered repositories that is the repository’s creation date.
  • Stars and concentration cover only products with their own repository (not a subfolder of a larger one). They measure attention, not adoption.

Page data last updated .