AgentSidekick

Evaluation

Running eval suites to grade outputs. As of 2026-11-21, 98 indexed products offer it (52 with full profiles), in 10 forms: 53 MCP servers, 20 Infrastructure, 8 Agent skills, 5 Developer tools. 30 were published in the last 90 days, against 27 in the 90 days before. agents has the most GitHub stars, 21% of the total across 81 products with their own repository.

CapabilitySegment label: Lagging the catalogue DER

Available as

All in the directory

Profiled products grouped by product form. Within each form: own-repository products by GitHub stars, then products in a subfolder of a larger repository (they show that repository's stars), up to 10 each.

MCP servers 53 indexed · 8 profiled

Protocol-level reach: 36 of the 45 active platforms AgentSidekick tracks document support for MCP. DER Which platforms

Evaluation: MCP servers by GitHub stars
#ProductPublisherCapabilities GitHub stars Last push Save and compare
1inspectorTesting and evaluation platform to chat, inspect, and debug MCP servers, MCP apps, and ChatGPT apps.MCPJam2.3ktoday
2AISafetyHotAI Safety HOT: daily AI safety news, papers and incidents in one place — jailbreaks, prompt injection, alignment, evals, governance. Website, daily digest, RSS and MCP.wuyoscar2.1ktoday
3heymBuild agentic systems. Run them with confidence. Orchestrate agents, automate business processes, inspect every execution, and keep humans in control. Deploy Heym on your own infrastructure.heymrun1.8ktoday
4tuuiA desktop MCP client designed as a tool unitary utility integration, accelerating AI adoption through the Model Context Protocol (MCP) and enabling cross-vendor LLM API orchestration.AI-QL1.2k6mo ago
5ratelThe self-improvement layer for AI agents. Ratel reads your agent's traces, finds what fails, recommends fixes and runs evals on real sessions before you ship. Self-hosted: one app, one Postgres. Includes the TypeScript and Python SDKratel-ai566today
6remnicOpen-source memory and context for user-aware agents: scoped memory, provenance, retrieval quality, correction, boundaries, evals, and MCP/HTTP access.joshuaswarren235today
7Model FinderFind the cheapest AI model that passes your tests: audit repo models, build evals, routing, sizingindigloo.co7today
8Sikaru DocsRead-only search over the public Sikaru documentation: managed agents, inference, evals and traces.sikaru.ai017d ago

Infrastructure 20 indexed · 20 profiled

No compatibility evidence is collected for this form.

Evaluation: Infrastructure by GitHub stars
#ProductPublisherCapabilities GitHub stars Last push Save and compare
1failproofaiObservability and enforcement for AI agent harnesses. Capture every run and runtime reliability with policy enforcement. FailproofAI9.9ktoday
2uptrainUpTrain is an open-source unified platform to evaluate and improve Generative AI applications. We provide grades for 20+ preconfigured checks (covering language, code, embedding use-cases), perform root cause analysis on failure cases and give insights on how to resolve them.uptrain-ai2.4k2y ago
3future-agiOpen-source, end-to-end platform for evaluating, observing, and improving LLM and AI agent applications. Tracing · Evals · Simulations · Datasets · Gateway · Guardrails. Self-hostable. Apache 2.0.future-agi2.3ktoday
4Tracely-aiTrace-native CI/CD for AI agents — production failures become regression tests that block the PR. Auto-detect, cluster, freeze into hermetic cases, replay in CI for $0.Jwuthri1.1ktoday
5judgevalThe Continuous-Improvement Stack for Agents. Our environment data and evals power agent improvement and monitoring.JudgmentLabs1.1k1d ago
6awesome-evalsA curated, non-BS library of the best resources for building and evaluating AI agents — papers, blogs, talks, tools, benchmarks. Maintained by BenchFlow.benchflow-ai1.0k16d ago
7overmindThe platform for continuously improving AI agents.overmind-core772today
8ai-engineer-notebooksHands-on, framework-free Colab notebooks for the AI Engineer / Forward Deployed Engineer (FDE) skill set — model APIs, structured output, tool calling, RAG, evals-as-the-spine, agents (loop from scratch, tool design, guardrails, MCP, Skills), fine-tuning vs LoRA, prompt-injection/security, LLMOps, and customer craft. Runs on the free Groq API.calmrocks75614d ago
9verifywiseComplete AI governance and LLM Evals platform with support for EU AI Act, ISO 42001, NIST AI RMF and 20+ more AI frameworks and regulations. Join our Discord channel: https://discord.com/invite/d3k3E4uEpRverifywise-ai376today
10athina-evalsPython SDK for running evaluations on LLM generated responsesathina-ai3021y ago

All 20 profiled Infrastructure with this capability

Agent skills 8 indexed · 8 profiled

Protocol-level reach: 26 of the 45 active platforms AgentSidekick tracks document support for the Agent Skills format. DER Which platforms

Evaluation: Agent skills by GitHub stars
#ProductPublisherCapabilities GitHub stars Last push Save and compare
1agentsPlugins, agents, skills, and commands for Claude Code, Codex CLI, Cursor, OpenCode, Antigravity CLI, GitHub Copilot, and Pi, written once in Markdown and adapted to each tool by generators with validation and evals.wshobson43k3d ago
2agent-skills-platformBuild tested agent skills and govern their lifecycle through a user-defined marketplace: evidence, discovery, updates, rollback, quarantine, and 17-platform distribution.FrancyJGLisboa2.4k2mo ago
3paperthinThe skillbook for recursive self-improvement (RSI). NVIDIA's latest proof, unpacked with timeless engineering. Distilled from the OpenAI harness engineering hackathon winner's zero-touch method.LilMGenius1.5k6d ago
4SkillForgeA skill creator that proves its skills work. Evidence-driven skill creation for Claude Code and Codex: baseline-tested generation, per-skill regression evals, ecosystem doctor, cross-runtime compile, and an opt-in proactive advisor.tripleyak9183mo ago
5agent-skills-evalA test runner for agentskills.io-style AI agent skillsdarkrishabh8195d ago
6skill-creatorCreate new skills, modify and improve existing skills, and measure skill performance. Use when users want to create a skill from scratch, edit, or optimize an existing skill, run evals to test a skill, benchmark skill performance with variance analysis, or optimize a skill's description for better triggering accuracy.anthropics—5d ago
7arize-datasetCreates, manages, and queries Arize datasets and examples. Covers dataset CRUD, appending examples, exporting data, and file-based dataset creation using the ax CLI. Use when the user needs test data, evaluation examples, or mentions create dataset, list datasets, export dataset, append examples, dataset version, golden dataset, or test set.github—1d ago
8advanced-evaluationDesign LLM-as-judge rubrics, direct scores and pairwise comparisons. Use for judge models, evals, position bias and calibration against human labels.sickn33—1d ago

Developer tools 5 indexed · 5 profiled

No compatibility evidence is collected for this form.

Evaluation: Developer tools by GitHub stars
#ProductPublisherCapabilities GitHub stars Last push Save and compare
1promptfooCLI and library for declarative prompt/model test suites, side-by-side comparisons and automated red-teaming of LLM apps and agents.OpenAI (Promptfoo)28ktoday
2Arize PhoenixSelf-hostable tracing and evaluation tool built on OpenTelemetry/OpenInference for inspecting LLM and agent runs, datasets and experiments.Arize-ai12ktoday
3AgentOpsSDK and dashboard for monitoring agent sessions, tool calls, costs and failures, with integrations for CrewAI, OpenAI Agents SDK, AG2 and others.AgentOps6.0k4mo ago
4PyRITPython Risk Identification Tool for automated red-teaming of generative AI systems with attack orchestrators and scorers.microsoft5.0ktoday
5LaminarOpen-source observability platform for agents with tracing, browser-session recordings, evaluations and SQL over trace data.Laminar3.5ktoday

Frameworks 3 indexed · 3 profiled

No compatibility evidence is collected for this form.

Evaluation: Frameworks by GitHub stars
#ProductPublisherCapabilities GitHub stars Last push Save and compare
1MastraTypeScript framework for agents, workflows, RAG, memory and evals, with a local dev playground and deploy targets for common hosts.Mastra30ktoday
2OpenAI EvalsFramework and registry of benchmarks for evaluating LLMs; OpenAI also offers a hosted Evals API in its platform.openai20k7mo ago
3waku-agentWaku Agent is an open-source AI agent harness you own: loop, memory, evals, and an always-on layer (event inbox, timers, approvals, outbox), in code that stays readable.ShenSeanChen2.2ktoday

Agent projects 2 indexed · 2 profiled

No compatibility evidence is collected for this form.

Evaluation: Agent projects by GitHub stars
#ProductPublisherCapabilities GitHub stars Last push Save and compare
1awesome-harness-engineeringAwesome list for AI agent harness engineering: tools, patterns, evals, memory, MCP, permissions, observability, and orchestration.ai-boost5.2ktoday
2building-a-coding-agent-from-scratch-courseLearn harness engineering by building Claude Code from scratch. Free, open-source course: 8 articles, 6 videos, 1 codebase.itsdecodingai1.2k28d ago

Libraries 2 indexed · 2 profiled

No compatibility evidence is collected for this form.

Evaluation: Libraries by GitHub stars
#ProductPublisherCapabilities GitHub stars Last push Save and compare
1GiskardTesting library that scans LLM agents and RAG apps for hallucination, prompt injection and harmful-content vulnerabilities and generates test suites.Giskard6.0k1d ago
2TruLensLibrary for instrumenting and evaluating LLM apps and agents with feedback functions such as groundedness and relevance.Snowflake3.7ktoday

SaaS products 2 indexed · 2 profiled

No compatibility evidence is collected for this form.

Evaluation: SaaS products by GitHub stars
#ProductPublisherCapabilities GitHub stars Last push Save and compare
1Pydantic LogfireOpenTelemetry-based observability service with first-class tracing for LLM calls, Pydantic AI agents and general Python apps, queryable with SQL.pydantic4.6ktoday
2BraintrustPlatform for running evals, comparing experiments, managing prompts and logging production traces of LLM apps and agents.Braintrust——

Rules packs 2 indexed · 1 profiled

No compatibility evidence is collected for this form.

Evaluation: Rules packs by GitHub stars
#ProductPublisherCapabilities GitHub stars Last push Save and compare
1skillcheckBlind A/B testing for AI agent skills, .cursorrules, CLAUDE.md, and system prompts. Detects placebo and harmful instructions.sx4im314d ago

Plugins 1 indexed · 1 profiled

1 of these are Claude plugins, which Claude Code documents (protocol-level). DER

Evaluation: Plugins by GitHub stars
#ProductPublisherCapabilities GitHub stars Last push Save and compare
1skill-creatorCreate new skills, improve existing skills, and measure skill performance. Use when users want to create a skill from scratch, update or optimize an existing skill, run evals to test a skill, or benchmark skill performance with variance analysis.anthropics—today

Which platforms these forms can reach

All platforms

Protocol-level: the platform documents support for the standard. No specific pairing has been tested.

Platforms publishers name

Integrations most often paired

Third-party services named by products with this capability, whole catalogue.

IntegrationProducts Share
Discord11.0%
GitHub11.0%
PostgreSQL11.0%

Keywords that trigger this capability

Matched as whole words, ignoring case, in a product's name, description and repository topics.

KeywordProducts matched
llm eval11
eval suite0
llm-as-a-judge0
evals86
golden dataset1
eval framework0

Segment statistics

Whole catalogue (98 indexed products). Calculated 2026-11-21.

98DERIndexed products52 with full profiles
30DERLaunched, last 90 days27 in the 90 days before
11DERMomentum vs catalogue50 = same pace as the catalogue
21%DERLargest project's share of starsAcross 81 products with their own repository
0.103DERConcentration (HHI)About 10 equal-sized projects’ worth; unconcentrated
65SRCMedian starsPer product with its own repository
67%DEROpen sourceMarked open source; unknown counts as not

Lagging the catalogue. Mid-sized among capabilities, with launches growing more slowly than the catalogue’s. Size is ranked against all capabilities; momentum compares launches in the last 90 days with the 90 before, relative to the catalogue’s own ratio (62 or more = faster, 38 or less = slower; under 5 dated launches it is not measured). The HHI of 0.103 means stars are spread as if across about 10 equally popular projects — unconcentrated by the common 0.15 / 0.25 thresholds. Stars measure attention, not usage or quality.

Limits

  • Classification is by the keywords above. A product that offers this capability without using one of them is not counted; one that uses a keyword in another sense is counted wrongly. Methodology
  • Platform reach is protocol-level: it says a platform documents the standard, not that any product here was tested on it. Platforms named by publishers are their own claims.
  • Stars belong to repositories. Products in a subfolder of a large repository show that whole repository’s stars and are excluded from the concentration figures.
  • Launch counts use each source’s publication date; for auto-discovered repositories that is the repository’s creation date.

Page data last updated .