Evaluation
Running eval suites to grade outputs. As of 2026-11-21, 98 indexed products offer it (52 with full profiles), in 10 forms: 53 MCP servers, 20 Infrastructure, 8 Agent skills, 5 Developer tools. 30 were published in the last 90 days, against 27 in the 90 days before. agents has the most GitHub stars, 21% of the total across 81 products with their own repository.
CapabilitySegment label: Lagging the catalogue DER
Available as
All in the directoryProfiled products grouped by product form. Within each form: own-repository products by GitHub stars, then products in a subfolder of a larger repository (they show that repository's stars), up to 10 each.
MCP servers 53 indexed · 8 profiled
Protocol-level reach: 36 of the 45 active platforms AgentSidekick tracks document support for MCP. DER Which platforms
| # | Product | Publisher | Capabilities | GitHub stars | Last push | Save and compare |
|---|---|---|---|---|---|---|
| 1 | inspectorTesting and evaluation platform to chat, inspect, and debug MCP servers, MCP apps, and ChatGPT apps. | MCPJam | 2.3k | today | ||
| 2 | AISafetyHotAI Safety HOT: daily AI safety news, papers and incidents in one place — jailbreaks, prompt injection, alignment, evals, governance. Website, daily digest, RSS and MCP. | wuyoscar | 2.1k | today | ||
| 3 | heymBuild agentic systems. Run them with confidence. Orchestrate agents, automate business processes, inspect every execution, and keep humans in control. Deploy Heym on your own infrastructure. | heymrun | 1.8k | today | ||
| 4 | tuuiA desktop MCP client designed as a tool unitary utility integration, accelerating AI adoption through the Model Context Protocol (MCP) and enabling cross-vendor LLM API orchestration. | AI-QL | 1.2k | 6mo ago | ||
| 5 | ratelThe self-improvement layer for AI agents. Ratel reads your agent's traces, finds what fails, recommends fixes and runs evals on real sessions before you ship. Self-hosted: one app, one Postgres. Includes the TypeScript and Python SDK | ratel-ai | 566 | today | ||
| 6 | remnicOpen-source memory and context for user-aware agents: scoped memory, provenance, retrieval quality, correction, boundaries, evals, and MCP/HTTP access. | joshuaswarren | 235 | today | ||
| 7 | Model FinderFind the cheapest AI model that passes your tests: audit repo models, build evals, routing, sizing | indigloo.co | 7 | today | ||
| 8 | Sikaru DocsRead-only search over the public Sikaru documentation: managed agents, inference, evals and traces. | sikaru.ai | 0 | 17d ago |
Infrastructure 20 indexed · 20 profiled
No compatibility evidence is collected for this form.
| # | Product | Publisher | Capabilities | GitHub stars | Last push | Save and compare |
|---|---|---|---|---|---|---|
| 1 | failproofaiObservability and enforcement for AI agent harnesses. Capture every run and runtime reliability with policy enforcement. | FailproofAI | 9.9k | today | ||
| 2 | uptrainUpTrain is an open-source unified platform to evaluate and improve Generative AI applications. We provide grades for 20+ preconfigured checks (covering language, code, embedding use-cases), perform root cause analysis on failure cases and give insights on how to resolve them. | uptrain-ai | 2.4k | 2y ago | ||
| 3 | future-agiOpen-source, end-to-end platform for evaluating, observing, and improving LLM and AI agent applications. Tracing · Evals · Simulations · Datasets · Gateway · Guardrails. Self-hostable. Apache 2.0. | future-agi | 2.3k | today | ||
| 4 | Tracely-aiTrace-native CI/CD for AI agents — production failures become regression tests that block the PR. Auto-detect, cluster, freeze into hermetic cases, replay in CI for $0. | Jwuthri | 1.1k | today | ||
| 5 | judgevalThe Continuous-Improvement Stack for Agents. Our environment data and evals power agent improvement and monitoring. | JudgmentLabs | 1.1k | 1d ago | ||
| 6 | awesome-evalsA curated, non-BS library of the best resources for building and evaluating AI agents — papers, blogs, talks, tools, benchmarks. Maintained by BenchFlow. | benchflow-ai | 1.0k | 16d ago | ||
| 7 | overmindThe platform for continuously improving AI agents. | overmind-core | 772 | today | ||
| 8 | ai-engineer-notebooksHands-on, framework-free Colab notebooks for the AI Engineer / Forward Deployed Engineer (FDE) skill set — model APIs, structured output, tool calling, RAG, evals-as-the-spine, agents (loop from scratch, tool design, guardrails, MCP, Skills), fine-tuning vs LoRA, prompt-injection/security, LLMOps, and customer craft. Runs on the free Groq API. | calmrocks | 756 | 14d ago | ||
| 9 | verifywiseComplete AI governance and LLM Evals platform with support for EU AI Act, ISO 42001, NIST AI RMF and 20+ more AI frameworks and regulations. Join our Discord channel: https://discord.com/invite/d3k3E4uEpR | verifywise-ai | 376 | today | ||
| 10 | athina-evalsPython SDK for running evaluations on LLM generated responses | athina-ai | 302 | 1y ago |
Agent skills 8 indexed · 8 profiled
Protocol-level reach: 26 of the 45 active platforms AgentSidekick tracks document support for the Agent Skills format. DER Which platforms
| # | Product | Publisher | Capabilities | GitHub stars | Last push | Save and compare |
|---|---|---|---|---|---|---|
| 1 | agentsPlugins, agents, skills, and commands for Claude Code, Codex CLI, Cursor, OpenCode, Antigravity CLI, GitHub Copilot, and Pi, written once in Markdown and adapted to each tool by generators with validation and evals. | wshobson | 43k | 3d ago | ||
| 2 | agent-skills-platformBuild tested agent skills and govern their lifecycle through a user-defined marketplace: evidence, discovery, updates, rollback, quarantine, and 17-platform distribution. | FrancyJGLisboa | 2.4k | 2mo ago | ||
| 3 | paperthinThe skillbook for recursive self-improvement (RSI). NVIDIA's latest proof, unpacked with timeless engineering. Distilled from the OpenAI harness engineering hackathon winner's zero-touch method. | LilMGenius | 1.5k | 6d ago | ||
| 4 | SkillForgeA skill creator that proves its skills work. Evidence-driven skill creation for Claude Code and Codex: baseline-tested generation, per-skill regression evals, ecosystem doctor, cross-runtime compile, and an opt-in proactive advisor. | tripleyak | 918 | 3mo ago | ||
| 5 | agent-skills-evalA test runner for agentskills.io-style AI agent skills | darkrishabh | 819 | 5d ago | ||
| 6 | skill-creatorCreate new skills, modify and improve existing skills, and measure skill performance. Use when users want to create a skill from scratch, edit, or optimize an existing skill, run evals to test a skill, benchmark skill performance with variance analysis, or optimize a skill's description for better triggering accuracy. | anthropics | — | 5d ago | ||
| 7 | arize-datasetCreates, manages, and queries Arize datasets and examples. Covers dataset CRUD, appending examples, exporting data, and file-based dataset creation using the ax CLI. Use when the user needs test data, evaluation examples, or mentions create dataset, list datasets, export dataset, append examples, dataset version, golden dataset, or test set. | github | — | 1d ago | ||
| 8 | advanced-evaluationDesign LLM-as-judge rubrics, direct scores and pairwise comparisons. Use for judge models, evals, position bias and calibration against human labels. | sickn33 | — | 1d ago |
Developer tools 5 indexed · 5 profiled
No compatibility evidence is collected for this form.
| # | Product | Publisher | Capabilities | GitHub stars | Last push | Save and compare |
|---|---|---|---|---|---|---|
| 1 | promptfooCLI and library for declarative prompt/model test suites, side-by-side comparisons and automated red-teaming of LLM apps and agents. | OpenAI (Promptfoo) | 28k | today | ||
| 2 | Arize PhoenixSelf-hostable tracing and evaluation tool built on OpenTelemetry/OpenInference for inspecting LLM and agent runs, datasets and experiments. | Arize-ai | 12k | today | ||
| 3 | AgentOpsSDK and dashboard for monitoring agent sessions, tool calls, costs and failures, with integrations for CrewAI, OpenAI Agents SDK, AG2 and others. | AgentOps | 6.0k | 4mo ago | ||
| 4 | PyRITPython Risk Identification Tool for automated red-teaming of generative AI systems with attack orchestrators and scorers. | microsoft | 5.0k | today | ||
| 5 | LaminarOpen-source observability platform for agents with tracing, browser-session recordings, evaluations and SQL over trace data. | Laminar | 3.5k | today |
Frameworks 3 indexed · 3 profiled
No compatibility evidence is collected for this form.
| # | Product | Publisher | Capabilities | GitHub stars | Last push | Save and compare |
|---|---|---|---|---|---|---|
| 1 | MastraTypeScript framework for agents, workflows, RAG, memory and evals, with a local dev playground and deploy targets for common hosts. | Mastra | 30k | today | ||
| 2 | OpenAI EvalsFramework and registry of benchmarks for evaluating LLMs; OpenAI also offers a hosted Evals API in its platform. | openai | 20k | 7mo ago | ||
| 3 | waku-agentWaku Agent is an open-source AI agent harness you own: loop, memory, evals, and an always-on layer (event inbox, timers, approvals, outbox), in code that stays readable. | ShenSeanChen | 2.2k | today |
Agent projects 2 indexed · 2 profiled
No compatibility evidence is collected for this form.
| # | Product | Publisher | Capabilities | GitHub stars | Last push | Save and compare |
|---|---|---|---|---|---|---|
| 1 | awesome-harness-engineeringAwesome list for AI agent harness engineering: tools, patterns, evals, memory, MCP, permissions, observability, and orchestration. | ai-boost | 5.2k | today | ||
| 2 | building-a-coding-agent-from-scratch-courseLearn harness engineering by building Claude Code from scratch. Free, open-source course: 8 articles, 6 videos, 1 codebase. | itsdecodingai | 1.2k | 28d ago |
Libraries 2 indexed · 2 profiled
No compatibility evidence is collected for this form.
| # | Product | Publisher | Capabilities | GitHub stars | Last push | Save and compare |
|---|---|---|---|---|---|---|
| 1 | GiskardTesting library that scans LLM agents and RAG apps for hallucination, prompt injection and harmful-content vulnerabilities and generates test suites. | Giskard | 6.0k | 1d ago | ||
| 2 | TruLensLibrary for instrumenting and evaluating LLM apps and agents with feedback functions such as groundedness and relevance. | Snowflake | 3.7k | today |
SaaS products 2 indexed · 2 profiled
No compatibility evidence is collected for this form.
| # | Product | Publisher | Capabilities | GitHub stars | Last push | Save and compare |
|---|---|---|---|---|---|---|
| 1 | Pydantic LogfireOpenTelemetry-based observability service with first-class tracing for LLM calls, Pydantic AI agents and general Python apps, queryable with SQL. | pydantic | 4.6k | today | ||
| 2 | BraintrustPlatform for running evals, comparing experiments, managing prompts and logging production traces of LLM apps and agents. | Braintrust | — | — |
Rules packs 2 indexed · 1 profiled
No compatibility evidence is collected for this form.
| # | Product | Publisher | Capabilities | GitHub stars | Last push | Save and compare |
|---|---|---|---|---|---|---|
| 1 | skillcheckBlind A/B testing for AI agent skills, .cursorrules, CLAUDE.md, and system prompts. Detects placebo and harmful instructions. | sx4im | 31 | 4d ago |
Plugins 1 indexed · 1 profiled
1 of these are Claude plugins, which Claude Code documents (protocol-level). DER
| # | Product | Publisher | Capabilities | GitHub stars | Last push | Save and compare |
|---|---|---|---|---|---|---|
| 1 | skill-creatorCreate new skills, improve existing skills, and measure skill performance. Use when users want to create a skill from scratch, update or optimize an existing skill, run evals to test a skill, or benchmark skill performance with variance analysis. | anthropics | — | today |
Which platforms these forms can reach
All platformsProtocol-level: the platform documents support for the standard. No specific pairing has been tested.
MCP servers 53 indexed here
36 platforms document MCP support CUR.
ContinueOpenCodeClaude CodeCodexGemini CLIZedOpenHandsClineWarpgooseQwen CodeBolt (bolt.new)KiroAmazon Bedrock AgentCoreAmpApps in ChatGPT / Apps SDK (now ChatGPT Plugins)Augment CodeChatGPTClaude (claude.ai apps)Claude DesktopCursorDevinDevin Desktop (formerly Windsurf)Factory (Droid)Gemini (app)Gemini Enterprise Agent Platform (formerly Vertex AI / Vertex AI Agent Builder)GitHub CopilotGitHub Copilot cloud agent (formerly Copilot coding agent)Google AntigravityJetBrains AI AssistantJunieLovableMicrosoft Copilot StudioReplit AgentTabninev0
Agent skills 8 indexed here
26 platforms document Agent Skills support CUR.
OpenClawOpenCodeClaude CodeCodexGemini CLIOpenHandsWarpgooseBolt (bolt.new)KiroAmpApps in ChatGPT / Apps SDK (now ChatGPT Plugins)ChatGPTClaude (claude.ai apps)Claude DesktopCursorDevinDevin Desktop (formerly Windsurf)Factory (Droid)Gemini (app)GitHub CopilotGitHub Copilot cloud agent (formerly Copilot coding agent)Google AntigravityJunieReplit AgentTabnine
Platforms publishers name
| Platform | Profiled products naming it |
|---|---|
| Claude Code | 4 |
| Codex | 3 |
| Cursor | 2 |
| GitHub Copilot | 1 |
| Google Antigravity | 1 |
| OpenCode | 1 |
| ChatGPT | 1 |
| Apps in ChatGPT / Apps SDK (now ChatGPT Plugins) | 1 |
Integrations most often paired
Third-party services named by products with this capability, whole catalogue.
| Integration | Products | Share |
|---|---|---|
| Discord | 1 | 1.0% |
| GitHub | 1 | 1.0% |
| PostgreSQL | 1 | 1.0% |
Keywords that trigger this capability
Matched as whole words, ignoring case, in a product's name, description and repository topics.
| Keyword | Products matched |
|---|---|
| llm eval | 11 |
| eval suite | 0 |
| llm-as-a-judge | 0 |
| evals | 86 |
| golden dataset | 1 |
| eval framework | 0 |
Segment statistics
Whole catalogue (98 indexed products). Calculated 2026-11-21.
Lagging the catalogue. Mid-sized among capabilities, with launches growing more slowly than the catalogue’s. Size is ranked against all capabilities; momentum compares launches in the last 90 days with the 90 before, relative to the catalogue’s own ratio (62 or more = faster, 38 or less = slower; under 5 dated launches it is not measured). The HHI of 0.103 means stars are spread as if across about 10 equally popular projects — unconcentrated by the common 0.15 / 0.25 thresholds. Stars measure attention, not usage or quality.
Limits
- Classification is by the keywords above. A product that offers this capability without using one of them is not counted; one that uses a keyword in another sense is counted wrongly. Methodology
- Platform reach is protocol-level: it says a platform documents the standard, not that any product here was tested on it. Platforms named by publishers are their own claims.
- Stars belong to repositories. Products in a subfolder of a large repository show that whole repository’s stars and are excluded from the concentration figures.
- Launch counts use each source’s publication date; for auto-discovered repositories that is the repository’s creation date.
Page data last updated .