ai-evaluation-tools
Infrastructureby aglio-labHEU
The comprehensive list of AI evaluation tools: 300+ open-source and commercial LLM evaluation frameworks, platforms, benchmarks, observability, red teaming, and guardrails — for LLMs, RAG, and AI agents.
SRC Publisher-stated: these are the publisher's own words, not an assessment.
Key facts
- Type
- InfrastructureSRC
- Publisher
- aglio-lab · individualSRC
- Licence
- CC0-1.0SRC
- Open source
- YesDER
- Hosting
- Unknown
- Latest version
- Unknown
- Languages
- pythonSRC
- First published
- SRC
- In this catalogue since
- DER
- Last verified
- Not verified by a researcher. Last seen in a source on 2026-11-21.
Capabilities and categories
Assigned by keyword rules from the description unless marked otherwise. The matched word is shown so you can judge the match.
| Capability | Evidence | Kind |
|---|---|---|
| Agent Evaluation | matched llm evaluation | DER |
| Agent Governance | matched guardrails | DER |
| Agent Monitoring | matched llm observability | DER |
| Application Monitoring | matched observability | DER |
| Retrieval-Augmented Generation | matched rag | DER |
| Evaluation | matched evals | DER |
| Policy Enforcement | matched guardrails | DER |
| Red Teaming | matched red teaming | DER |
Filed underAgent InfrastructureData & KnowledgeDeveloper ToolsAgent ObservabilityAgent SecurityDevOps & SRERetrieval & RAG
Integrations DER
No third-party service is named in the description or package metadata.
Platform compatibility
No pairing has been tested by AgentSidekick. Each status says what kind of evidence exists, not whether it works for you.
Unknown47
No compatibility evidence collected for this product type.
Install
Read from registry metadata. Follow the publisher's instructions; we have not run this.
No install information
No package, remote endpoint, install link or documentation link appears in any source for this product.
The repository may carry instructions in its README.
Permissions and signals
Signals, not a safety verdict. They describe what the product's metadata references. Security review: not reviewed.
No signals collected
Automated inspection found nothing to report, or there was no metadata to inspect. That is not evidence of safety.
Known advisories for the published package SRC
An advisory existing does not mean the listed version is affected: compare the vulnerable range with the version you install. None found is not evidence of safety.
No advisories found
No published advisory matches a package this product declares. Products without a declared package cannot be matched at all. This is not evidence of safety.
Evaluation
Separate dimensions, each with its evidence. There is no overall score: the dimensions do not add up to one number honestly.
- Source authority
Auto-discovered
Found through repository metadata; type and categories are inferred, not confirmed.
- Publisher verification
Not verified
No publisher identity check is available for this source.
- Maintenance activity
Slowing
Last push 2026-08-05.
- Licence clarity
CC0-1.0
Licence identifier as reported by GitHub repository metadata.
- Installation clarity
Unknown
No install information in any source.
- Documentation
Repository / website only
No dedicated documentation URL on record. AgentSidekick has not assessed documentation quality.
- Compatibility evidence
Unknown
No compatibility evidence collected.
- Security review
Not reviewed
AgentSidekick has not audited this product. No automated signals were collected. That is not evidence of safety.
Pricing
Not published in any source we track. The code is open source, which says nothing about the cost of any hosted service.
Repository activity
aglio-lab/ai-evaluation-tools. Read from the GitHub API on 2026-11-21.
Version history SRC
No version list is published in any source we track.
Change history
What changed between two ingests of the same source, and when the product was first seen. Registry releases are under Version history.
- First seen
Competitors and alternativesDER
Paired by shared capabilities and integrations. Similar on paper; not tested side by side.
Direct
Same type, overlapping capabilities.
Infrastructureby FailproofAIHEU
SharesApplication MonitoringAgent GovernancePolicy EnforcementAgent Monitoring
Infrastructureby anhermonHEU
SharesApplication MonitoringAgent GovernancePolicy EnforcementAgent Monitoring
Infrastructureby future-agiHEU
SharesApplication MonitoringAgent GovernancePolicy EnforcementAgent Monitoring
Infrastructureby talhaanwarchHEU
SharesApplication MonitoringAgent GovernancePolicy EnforcementAgent Monitoring
Infrastructureby cristianodabcHEU
SharesApplication MonitoringAgent MonitoringAgent EvaluationEvaluation
Infrastructureby JudgmentLabsHEU
SharesApplication MonitoringAgent MonitoringAgent EvaluationEvaluation
Adjacent
A different form or a partial overlap.
Developer toolby OpenAI (Promptfoo)CUR
SharesAgent EvaluationEvaluationRed TeamingRetrieval-Augmented Generation
Libraryby GiskardCUR
SharesAgent GovernanceAgent EvaluationEvaluationRetrieval-Augmented Generation
Provenance
Which source set each field, what kind of claim it is, and when.
| Field | Source | Kind | Set on |
|---|---|---|---|
| Canonical URL | GitHub topic search | SRC | 2026-11-21 |
| Publisher | GitHub topic search | SRC | 2026-11-21 |
| Description | GitHub topic search | SRC | 2026-11-21 |
| Languages | GitHub repository metadata | SRC | 2026-11-21 |
| Licence | GitHub repository metadata | SRC | 2026-11-21 |
| Name | GitHub topic search | SRC | 2026-11-21 |
| Open source | GitHub repository metadata | DER | 2026-11-21 |
| Type | GitHub topic search | SRC | 2026-11-21 |
| Published | GitHub topic search | SRC | 2026-11-21 |
| Repository | GitHub topic search | SRC | 2026-11-21 |
| Source URL | GitHub topic search | SRC | 2026-11-21 |
Sources
- GitHub topic search: github.com (external)checked
Not verified by a researcher. Last seen in a source on . Report incorrect information