AgentSidekick

ai-evaluation-tools

Infrastructureby aglio-labHEU

The comprehensive list of AI evaluation tools: 300+ open-source and commercial LLM evaluation frameworks, platforms, benchmarks, observability, red teaming, and guardrails — for LLMs, RAG, and AI agents.

SRC Publisher-stated: these are the publisher's own words, not an assessment.

RepositorySource listing

Key facts

Type
InfrastructureSRC
Publisher
aglio-lab · individualSRC
Licence
CC0-1.0SRC
Open source
YesDER
Hosting
Unknown
Latest version
Unknown
Languages
pythonSRC
First published
SRC
In this catalogue since
DER
Last verified
Not verified by a researcher. Last seen in a source on 2026-11-21.

Capabilities and categories

Assigned by keyword rules from the description unless marked otherwise. The matched word is shown so you can judge the match.

CapabilityEvidenceKind
Agent Evaluationmatched llm evaluationDER
Agent Governancematched guardrailsDER
Agent Monitoringmatched llm observabilityDER
Application Monitoringmatched observabilityDER
Retrieval-Augmented Generationmatched ragDER
Evaluationmatched evalsDER
Policy Enforcementmatched guardrailsDER
Red Teamingmatched red teamingDER

Filed underAgent InfrastructureData & KnowledgeDeveloper ToolsAgent ObservabilityAgent SecurityDevOps & SRERetrieval & RAG

Integrations DER

No third-party service is named in the description or package metadata.

Platform compatibility

No pairing has been tested by AgentSidekick. Each status says what kind of evidence exists, not whether it works for you.

Unknown47

No compatibility evidence collected for this product type.

Install

Read from registry metadata. Follow the publisher's instructions; we have not run this.

No install information

No package, remote endpoint, install link or documentation link appears in any source for this product.

The repository may carry instructions in its README.

Permissions and signals

Signals, not a safety verdict. They describe what the product's metadata references. Security review: not reviewed.

No signals collected

Automated inspection found nothing to report, or there was no metadata to inspect. That is not evidence of safety.

Known advisories for the published package SRC

An advisory existing does not mean the listed version is affected: compare the vulnerable range with the version you install. None found is not evidence of safety.

No advisories found

No published advisory matches a package this product declares. Products without a declared package cannot be matched at all. This is not evidence of safety.

Evaluation

Separate dimensions, each with its evidence. There is no overall score: the dimensions do not add up to one number honestly.

Source authority

Auto-discovered

Found through repository metadata; type and categories are inferred, not confirmed.

Publisher verification

Not verified

No publisher identity check is available for this source.

Maintenance activity

Slowing

Last push 2026-08-05.

Licence clarity

CC0-1.0

Licence identifier as reported by GitHub repository metadata.

Installation clarity

Unknown

No install information in any source.

Documentation

Repository / website only

No dedicated documentation URL on record. AgentSidekick has not assessed documentation quality.

Compatibility evidence

Unknown

No compatibility evidence collected.

Security review

Not reviewed

AgentSidekick has not audited this product. No automated signals were collected. That is not evidence of safety.

Pricing

Not published in any source we track. The code is open source, which says nothing about the cost of any hosted service.

Repository activity

aglio-lab/ai-evaluation-tools. Read from the GitHub API on 2026-11-21.

16SRCGitHub stars
16SRCForks
4SRCOpen issues
3mo agoSRCLast push2026-08-05
2026.07SRCLatest release2026-07-16

Version history SRC

No version list is published in any source we track.

Change history

What changed between two ingests of the same source, and when the product was first seen. Registry releases are under Version history.

  1. First seen

Competitors and alternativesDER

Paired by shared capabilities and integrations. Similar on paper; not tested side by side.

Direct

Same type, overlapping capabilities.

failproofai

Infrastructureby FailproofAIHEU

9.9k GitHub starspushed todaylicence unknown

SharesApplication MonitoringAgent GovernancePolicy EnforcementAgent Monitoring

SharesApplication MonitoringAgent GovernancePolicy EnforcementAgent Monitoring

future-agi

Infrastructureby future-agiHEU

2.3k GitHub starspushed todayApache-2.0

SharesApplication MonitoringAgent GovernancePolicy EnforcementAgent Monitoring

awesome-llmops-platforms

Infrastructureby talhaanwarchHEU

40 GitHub starspushed 5d agoCC0-1.0

SharesApplication MonitoringAgent GovernancePolicy EnforcementAgent Monitoring

aludel

Infrastructureby cristianodabcHEU

41 GitHub starspushed 4d agoApache-2.0

SharesApplication MonitoringAgent MonitoringAgent EvaluationEvaluation

judgeval

Infrastructureby JudgmentLabsHEU

1.1k GitHub starspushed 1d agoApache-2.0

SharesApplication MonitoringAgent MonitoringAgent EvaluationEvaluation

Adjacent

A different form or a partial overlap.

promptfoo

Developer toolby OpenAI (Promptfoo)CUR

28k GitHub starspushed todayMIT

SharesAgent EvaluationEvaluationRed TeamingRetrieval-Augmented Generation

PyRIT

Developer toolby microsoftCUR

5.0k GitHub starspushed todayMIT

SharesAgent GovernanceAgent EvaluationEvaluationRed Teaming

Giskard

Libraryby GiskardCUR

6.0k GitHub starspushed 1d agoApache-2.0

SharesAgent GovernanceAgent EvaluationEvaluationRetrieval-Augmented Generation

Provenance

Which source set each field, what kind of claim it is, and when.

FieldSourceKindSet on
Canonical URLGitHub topic searchSRC2026-11-21
PublisherGitHub topic searchSRC2026-11-21
DescriptionGitHub topic searchSRC2026-11-21
LanguagesGitHub repository metadataSRC2026-11-21
LicenceGitHub repository metadataSRC2026-11-21
NameGitHub topic searchSRC2026-11-21
Open sourceGitHub repository metadataDER2026-11-21
TypeGitHub topic searchSRC2026-11-21
PublishedGitHub topic searchSRC2026-11-21
RepositoryGitHub topic searchSRC2026-11-21
Source URLGitHub topic searchSRC2026-11-21

Sources

Not verified by a researcher. Last seen in a source on . Report incorrect information

Page data last updated .