Agent Evaluation
Scoring, benchmarking and red-teaming models and agents. As of 2026-11-21, 507 indexed products match this subcategory (224 with full profiles); 215 were published in the last 90 days, against 120 in the 90 days before, slower than the catalogue as a whole. By GitHub stars, iFixAi leads, holding 7% of the stars of the 390 products here with their own repository. The most common form is MCP servers (64% of indexed products).
SubcategorySegment label: Lagging the catalogue DER
Segment statistics
Whole catalogue (all 507 indexed products, not only profiled ones). Calculated 2026-11-21.
Lagging the catalogue. Mid-sized for its level, with launches growing more slowly than the catalogue’s. Size is ranked against the other subcategory nodes; momentum compares this node’s launches in the last 90 days with the 90 before, relative to the whole catalogue’s ratio (score 62 or more = faster, 38 or less = slower; below 5 dated launches it is not measured and the label uses size alone).
Concentration. The Herfindahl–Hirschman index (HHI) adds up each project’s squared share of stars. At 0.034, attention is spread as if across about 30 equally popular projects. Using the common competition-policy thresholds (0.15 and 0.25), that is unconcentrated. Stars measure attention, not usage or quality.
Which product forms serve it
The same need is met by MCP servers, agent skills, plugins, extensions and frameworks. This counts each form across the whole catalogue.
| Product type | Indexed | Share | With a profile |
|---|---|---|---|
| MCP servers | 325 | 64.1% | 48 |
| Infrastructure | 98 | 19.3% | 98 |
| Agent skills | 31 | 6.1% | 31 |
| Developer tools | 11 | 2.2% | 11 |
| Frameworks | 10 | 2.0% | 10 |
| Plugins | 9 | 1.8% | 7 |
| Libraries | 9 | 1.8% | 9 |
| Rules packs | 6 | 1.2% | 2 |
| Agent projects | 6 | 1.2% | 6 |
| SaaS products | 2 | 0.39% | 2 |
3 capabilities
Indexed = whole catalogue; profiled = products with their own page. A product can sit in several, so the rows do not add up to the total.
| Capability | Indexed | Profiled | New, 90 days | Label |
|---|---|---|---|---|
| BenchmarkingComparing models and agents on standard benchmarks. | 297 | 101 | 139 | Crowded, slowing |
| EvaluationRunning eval suites to grade outputs. | 98 | 52 | 30 | Lagging the catalogue |
| Red TeamingAdversarial testing for safety and robustness. | 28 | 21 | 8 | Thin |
Top 40 products by GitHub stars
Search the directoryProfiled products only (224 of 507), ranked by the stars of their own repository. Products that live in a subfolder of a larger repository (common for skills) show that whole repository's stars and are ranked after them; products without a repository come last. Stars are popularity, not quality or installs.
| # | Product | Type | Publisher | Capabilities | GitHub stars | Last push | Licence | Save and compare |
|---|---|---|---|---|---|---|---|---|
| 1 | iFixAiIndependent Auditing of AI Agents. Run by human or the agent itself, to answer the most crucial question in the AI Agent Economy. Is the agent doing what is supposed to do? With iFixAi you can have this answer in less than 120 seconds. | Infrastructure | ifixai-ai | 47k | 1d ago | Apache-2.0 | ||
| 2 | agentsPlugins, agents, skills, and commands for Claude Code, Codex CLI, Cursor, OpenCode, Antigravity CLI, GitHub Copilot, and Pi, written once in Markdown and adapted to each tool by generators with validation and evals. | Agent skill | wshobson | 43k | 3d ago | MIT | ||
| 3 | LightRAGGraph-enhanced RAG system that indexes entities and relations alongside vectors for dual-level retrieval, with a server and web UI. | Library | HKU Data Intelligence Lab | 41k | today | MIT | ||
| 4 | LangfuseOpen-source LLM engineering platform for tracing, prompt management, evaluations and datasets, self-hosted or Langfuse Cloud. | Developer tool | ClickHouse (Langfuse) | 38k | today | MIT | ||
| 5 | MastraTypeScript framework for agents, workflows, RAG, memory and evals, with a local dev playground and deploy targets for common hosts. | Framework | Mastra | 30k | today | Apache-2.0 | ||
| 6 | MLflowOpen-source ML/GenAI lifecycle platform whose GenAI features include OpenTelemetry-compatible tracing, evaluation and prompt registry for agents. | Developer tool | Databricks (Linux Foundation) | 29k | today | Apache-2.0 | ||
| 7 | promptfooCLI and library for declarative prompt/model test suites, side-by-side comparisons and automated red-teaming of LLM apps and agents. | Developer tool | OpenAI (Promptfoo) | 28k | today | MIT | ||
| 8 | OpikOpen-source platform for tracing, evaluating and monitoring LLM apps and agents, with LLM-as-judge metrics and prompt optimisation. | Developer tool | Comet | 23k | today | Apache-2.0 | ||
| 9 | OpenAI EvalsFramework and registry of benchmarks for evaluating LLMs; OpenAI also offers a hosted Evals API in its platform. | Framework | openai | 20k | 7mo ago | MIT | ||
| 10 | DeepEvalPytest-style framework for evaluating LLM outputs, RAG pipelines and agents with research-backed metrics and synthetic test generation. | Library | confident-ai | 20k | 1d ago | Apache-2.0 | ||
| 11 | deepevalSkills for adding DeepEval evaluations, tracing, datasets, Confident AI reports, and iterative improvement loops to AI applications. | Plugin | confident-ai | 20k | 1d ago | Apache-2.0 | ||
| 12 | artemisARTEMIS turns natural-language instructions into reliable Android automation. It automates end-to-end workflows, captures logs, and integrates seamlessly with AI coding assistants such as Antigravity, Codex, and Claude Code. It also achieves 99%+ success rate on AndroidWorld Benchmark. | Agent project | 18k | 1mo ago | Apache-2.0 | |||
| 13 | RagasLibrary of metrics and test-set generation for evaluating RAG and agent applications, including faithfulness and context precision. | Library | Vibrant Labs | 16k | 7d ago | Apache-2.0 | ||
| 14 | Arize PhoenixSelf-hostable tracing and evaluation tool built on OpenTelemetry/OpenInference for inspecting LLM and agent runs, datasets and experiments. | Developer tool | Arize-ai | 12k | today | Elastic-2.0 | ||
| 15 | garakLLM vulnerability scanner that probes models and apps for jailbreaks, prompt injection, data leakage and toxicity. | Developer tool | NVIDIA | 10k | 3d ago | Apache-2.0 | ||
| 16 | failproofaiObservability and enforcement for AI agent harnesses. Capture every run and runtime reliability with policy enforcement. | Infrastructure | FailproofAI | 9.9k | today | — | ||
| 17 | oumiEasily fine-tune (SFT/RL), evaluate, and deploy Qwen, Gemma, or any open agentic LLM/VLM! | Infrastructure | oumi-ai | 9.4k | today | Apache-2.0 | ||
| 18 | AI-Infra-GuardA full-stack AI Red Teaming platform securing AI ecosystems via Agent Scan, Skills Scan, MCP scan, AI Infra scan and LLM jailbreak evaluation. | Infrastructure | Tencent | 7.6k | 1d ago | Apache-2.0 | ||
| 19 | gepaOptimize prompts, code, and more with AI-powered Reflective Optimization | Agent skill | gepa-ai | 7.5k | today | MIT | ||
| 20 | chinese-llm-benchmark非线智能 NoneLinear - ReLE评测:中文AI大模型能力评测(持续更新):目前已囊括374个大模型,覆盖chatgpt、gpt-5.4、谷歌gemini-3.1-pro、Claude-4.6、文心ERNIE-X1.1、ERNIE-5.0、qwen3.6-max、qwen3.6-plus、百川、讯飞星火、商汤senseChat等商用模型, 以及step3.5-flash、kimi-k2.6、ernie4.5、MiniMax-M2.7、deepseek-v4、Qwen3.6、llama4、智谱GLM-5.1、MiMo-V2、LongCat、gemma4、mistral等开源大模型。不仅提供排行榜,也提供规模超200万的大模型缺陷库!方便广大社区研究分析、改进大模型。 | Infrastructure | jeinlee1991 | 6.5k | 1d ago | — | ||
| 21 | OuroborosPins an acceptance spec; the verify command and expected output never enter the success contract. | MCP server | Q00 | 6.4k | today | MIT | ||
| 22 | HeliconeProxy-based LLM observability that logs requests, costs and latency with one-line integration, plus an AI gateway. | Developer tool | Mintlify (Helicone) | 6.3k | 4d ago | Apache-2.0 | ||
| 23 | AgentOpsSDK and dashboard for monitoring agent sessions, tool calls, costs and failures, with integrations for CrewAI, OpenAI Agents SDK, AG2 and others. | Developer tool | AgentOps | 6.0k | 4mo ago | MIT | ||
| 24 | GiskardTesting library that scans LLM agents and RAG apps for hallucination, prompt injection and harmful-content vulnerabilities and generates test suites. | Library | Giskard | 6.0k | 1d ago | Apache-2.0 | ||
| 25 | tokscale🛰️ Track token usage across AI coding agents from your terminal. 🏅 Global leaderboard with trillions of tokens tracked. | Agent project | junhoyeo | 5.9k | 3d ago | MIT | ||
| 26 | coze-loopNext-generation AI Agent Optimization Platform: Cozeloop addresses challenges in AI agent development by providing full-lifecycle management capabilities from development, debugging, and evaluation to monitoring. | Infrastructure | coze-dev | 5.8k | 4d ago | Apache-2.0 | ||
| 27 | LLM-Engineers-HandbookThe LLM's practical guide: From the fundamentals to deploying advanced LLM and RAG apps to AWS using LLMOps best practices | Infrastructure | PacktPublishing | 5.4k | 7mo ago | MIT | ||
| 28 | awesome-harness-engineeringAwesome list for AI agent harness engineering: tools, patterns, evals, memory, MCP, permissions, observability, and orchestration. | Agent project | ai-boost | 5.2k | today | — | ||
| 29 | PyRITPython Risk Identification Tool for automated red-teaming of generative AI systems with attack orchestrators and scorers. | Developer tool | microsoft | 5.0k | today | MIT | ||
| 30 | AgentaOpen-source LLMOps platform for prompt management, evaluation and observability with a playground for comparing variants. | Developer tool | Agenta | 4.9k | today | MIT | ||
| 31 | all-agentic-architectures35 production-grade agentic AI architectures (Reflexion, LATS, GraphRAG, MemGPT, Voyager, BrowserAgent, ...) — a Python library and runnable textbook with multi-provider LLM support and a 17-task benchmark leaderboard. | Agent project | FareedKhan-dev | 4.8k | 5d ago | MIT | ||
| 32 | Pydantic LogfireOpenTelemetry-based observability service with first-class tracing for LLM calls, Pydantic AI agents and general Python apps, queryable with SQL. | SaaS product | pydantic | 4.6k | today | MIT | ||
| 33 | lmms-evalOne-for-All Multimodal Evaluation Toolkit Across Text, Image, Video, and Audio Tasks | Infrastructure | EvolvingLMMs-Lab | 4.5k | 1d ago | — | ||
| 34 | SkillEvaluatorMulti-tier framework for evaluating AI agent skills and plugins with quality gates, security scanning, harness compatibility checks, semantic overlap detection, synthetic evaluation dataset generation, and live agent evaluation that measures how skills affect agent behavior. Ships reusable CI templates that bring every check to any repository. | Agent skill | NVIDIA | 4.3k | today | Apache-2.0 | ||
| 35 | AI-Engineer-HeadquartersA collection of scientific methods, processes, algorithms, and systems to build stories & models. | Infrastructure | hemansnation | 3.8k | 24d ago | — | ||
| 36 | TruLensLibrary for instrumenting and evaluating LLM apps and agents with feedback functions such as groundedness and relevance. | Library | Snowflake | 3.7k | 1d ago | MIT | ||
| 37 | LaminarOpen-source observability platform for agents with tracing, browser-session recordings, evaluations and SQL over trace data. | Developer tool | Laminar | 3.5k | 1d ago | Apache-2.0 | ||
| 38 | InspectPython framework for LLM and agent evaluations with datasets, solvers, scorers, sandboxed tool execution and a log viewer. | Framework | UK AI Security Institute | 3.3k | 1d ago | MIT | ||
| 39 | little-coderA harness optimized to smaller LLMs | Agent project | itayinbarr | 2.7k | 2mo ago | Apache-2.0 | ||
| 40 | generative-aiComprehensive resources on Generative AI, including a detailed roadmap, projects, use cases, interview preparation, and coding preparation. | MCP server | genieincodebottle | 2.7k | 12d ago | MIT |
Most common integrations
Third-party services named by products in this node, whole catalogue.
| Integration | Products | Share |
|---|---|---|
| GitHub | 5 | 1.0% |
| PostgreSQL | 3 | 0.6% |
| AWS | 2 | 0.4% |
| Android | 1 | 0.2% |
| Discord | 1 | 0.2% |
| Google Cloud | 1 | 0.2% |
| Kubernetes | 1 | 0.2% |
| 1 | 0.2% | |
| MySQL | 1 | 0.2% |
| Playwright | 1 | 0.2% |
| QuickBooks | 1 | 0.2% |
| Salesforce | 1 | 0.2% |
| Solana | 1 | 0.2% |
| Stripe | 1 | 0.2% |
| n8n | 1 | 0.2% |
Launches per month
New Agent Evaluation products per monthSRC
Whole catalogue, by publication month, 2024-11 to 2026-11. The last month is partial; undated products are excluded.
Show the data as a table
| Products published (products) | |
|---|---|
| 2024-11 | 2 |
| 2024-12 | 2 |
| 2025-01 | 2 |
| 2025-02 | 2 |
| 2025-04 | 3 |
| 2025-05 | 3 |
| 2025-06 | 3 |
| 2025-07 | 2 |
| 2025-08 | 1 |
| 2025-09 | 4 |
| 2025-10 | 2 |
| 2025-11 | 2 |
| 2025-12 | 3 |
| 2026-01 | 3 |
| 2026-02 | 9 |
| 2026-03 | 17 |
| 2026-04 | 17 |
| 2026-05 | 23 |
| 2026-06 | 26 |
| 2026-07 | 40 |
| 2026-08 | 67 |
| 2026-09 | 41 |
| 2026-10 | 86 |
| 2026-11 | 72 |
Related opportunities
Opportunity radarScored gaps for this node or anything beneath it. Scores rank hypotheses; they do not predict demand.
| Opportunity | Competition | Confidence | Score |
|---|---|---|---|
| Coverage gap: Red TeamingRed Teaming | High | Low | 58.1 |
Limits of these numbers
- Products are placed here by whole-word keyword rules on their name, description and repository topics, and inherit every ancestor of a matched node. Products that do this without naming it are missed; ambiguous words cause false matches. Methodology
- About half of the catalogue matches no node at all, so absolute counts understate every category.
- Launch counts use the publication date each source reports; for auto-discovered repositories that is the repository’s creation date.
- Stars and concentration cover only products with their own repository (not a subfolder of a larger one). They measure attention, not adoption.
Page data last updated .