How do AI agent evaluation and testing tools compare?
Short answer DER
As of 2026-11-21, the most-starred evaluation and testing tools in AgentSidekick's curated records are promptfoo (27,513 GitHub stars), OpenAI Evals (19,776) and DeepEval (19,582), ranked by GitHub stars of each project's own repository. We track 9 curated evaluation and testing tools; 4 have MCP support recorded, and 1 had no star data at build time, so they are listed after.
The evidence
Agent evaluation tools compared: 9 curated recordsCUR
Records researched by AgentSidekick against official sites, docs and repositories; stars and push dates are read from GitHub. "Not recorded" means we found no MCP support statement, not that it is absent.
| # | Product | Type | Publisher | Licence | Open source | Pricing model | GitHub stars | Last push | MCP support |
|---|---|---|---|---|---|---|---|---|---|
| 1 | promptfoo | Developer tool | OpenAI (Promptfoo) | MIT | Yes | open-source | 27,513 | 2026-11-21 | Yes (curated) |
| 2 | OpenAI Evals | Framework | openai | MIT | Yes | open-source | 19,776 | 2026-04-14 | Not recorded |
| 3 | DeepEval | Library | confident-ai | Apache-2.0 | Yes | open-source | 19,582 | 2026-11-19 | Yes (curated) |
| 4 | Ragas | Library | Vibrant Labs | Apache-2.0 | Yes | open-source | 16,394 | 2026-11-14 | Not recorded |
| 5 | Giskard | Library | Giskard | Apache-2.0 | Yes | open-source | 5,965 | 2026-11-20 | Not recorded |
| 6 | Agenta | Developer tool | Agenta | MIT | Yes | freemium | 4,925 | 2026-11-21 | Not recorded |
| 7 | TruLens | Library | Snowflake | MIT | Yes | open-source | 3,664 | 2026-11-20 | Not recorded |
| 8 | Inspect | Framework | UK AI Security Institute | MIT | Yes | open-source | 3,302 | 2026-11-20 | Yes (curated) |
| 9 | Braintrust | SaaS product | Braintrust | Unknown | No | freemium | Unknown | Unknown | Yes (curated) |
How the ranking was made. The list is ordered by GitHub stars of the product's own repository, highest first (a product inside a shared monorepo has no stars of its own and is listed after); products without star data follow, alphabetically. Only curated records in the "evaluation-testing" segment are included. GitHub stars measure popularity, not quality, security or fit for your use.
Limit. Pricing model is a category (open-source, freemium, paid, usage-based), not a price. AgentSidekick has not benchmarked or security-reviewed any of these products.
Which licences do they use? MIT 5, Apache-2.0 3 and Unknown 1.
Which are not open source? Braintrust.
Which was updated most recently? promptfoo, last pushed 2026-11-21.
How big is the wider category? The Agent Evaluation category holds 507 catalogue products, including uncurated ones (derived by keyword rules).
Related
Sources
Pages the records behind this answer were taken from. Counts and rankings are computed by the site from these records.
- promptfoo: promptfoo.dev (external)checked
- OpenAI Evals: developers.openai.com (external)checked
- DeepEval: deepeval.com (external)checked
- Ragas: docs.ragas.io (external)checked
- Giskard: giskard.ai (external)checked
- Agenta: agenta.ai (external)checked
- TruLens: trulens.org (external)checked
- Inspect: inspect.aisi.org.uk (external)checked
How claims are labelled and ranked: claim kinds, ranking rules, compatibility statuses. Spotted an error? See corrections.