AgentSidekick

How do AI agent evaluation and testing tools compare?

Short answer DER

As of 2026-11-21, the most-starred evaluation and testing tools in AgentSidekick's curated records are promptfoo (27,513 GitHub stars), OpenAI Evals (19,776) and DeepEval (19,582), ranked by GitHub stars of each project's own repository. We track 9 curated evaluation and testing tools; 4 have MCP support recorded, and 1 had no star data at build time, so they are listed after.

AgentSidekick, as of ยท based on 9 products

The evidence

Agent evaluation tools compared: 9 curated recordsCUR

Records researched by AgentSidekick against official sites, docs and repositories; stars and push dates are read from GitHub. "Not recorded" means we found no MCP support statement, not that it is absent.

Agent evaluation tools compared: 9 curated records
#ProductTypePublisherLicenceOpen sourcePricing modelGitHub starsLast pushMCP support
1promptfooDeveloper toolOpenAI (Promptfoo)MITYesopen-source27,5132026-11-21Yes (curated)
2OpenAI EvalsFrameworkopenaiMITYesopen-source19,7762026-04-14Not recorded
3DeepEvalLibraryconfident-aiApache-2.0Yesopen-source19,5822026-11-19Yes (curated)
4RagasLibraryVibrant LabsApache-2.0Yesopen-source16,3942026-11-14Not recorded
5GiskardLibraryGiskardApache-2.0Yesopen-source5,9652026-11-20Not recorded
6AgentaDeveloper toolAgentaMITYesfreemium4,9252026-11-21Not recorded
7TruLensLibrarySnowflakeMITYesopen-source3,6642026-11-20Not recorded
8InspectFrameworkUK AI Security InstituteMITYesopen-source3,3022026-11-20Yes (curated)
9BraintrustSaaS productBraintrustUnknownNofreemiumUnknownUnknownYes (curated)

How the ranking was made. The list is ordered by GitHub stars of the product's own repository, highest first (a product inside a shared monorepo has no stars of its own and is listed after); products without star data follow, alphabetically. Only curated records in the "evaluation-testing" segment are included. GitHub stars measure popularity, not quality, security or fit for your use.

Limit. Pricing model is a category (open-source, freemium, paid, usage-based), not a price. AgentSidekick has not benchmarked or security-reviewed any of these products.

Which licences do they use? MIT 5, Apache-2.0 3 and Unknown 1.

Which are not open source? Braintrust.

Which was updated most recently? promptfoo, last pushed 2026-11-21.

How big is the wider category? The Agent Evaluation category holds 507 catalogue products, including uncurated ones (derived by keyword rules).

Sources

Pages the records behind this answer were taken from. Counts and rankings are computed by the site from these records.

How claims are labelled and ranked: claim kinds, ranking rules, compatibility statuses. Spotted an error? See corrections.

Page data last updated .