AgentSidekick

promptfoo

Developer toolby OpenAI (Promptfoo)CUR

CLI and library for declarative prompt/model test suites, side-by-side comparisons and automated red-teaming of LLM apps and agents.

CUR Description written by a researcher from official sources.

WebsiteRepositoryDocumentationSource listing

Key facts

Type
Developer tool · hostedCUR
Publisher
OpenAI (Promptfoo) · organizationCUR
Licence
MITCUR
Open source
YesCUR
Hosting
Unknown
Latest version
Unknown
Languages
typescriptCUR
First published
SRC
In this catalogue since
DER
Last verified
CUR

Capabilities and categories

Assigned by keyword rules from the description unless marked otherwise. The matched word is shown so you can judge the match.

CapabilityEvidenceKind
Agent Evaluationmatched segment: evaluation-testingCUR
Retrieval-Augmented Generationmatched ragDER
Evaluationmatched llm evalDER
Prompt Optimizationmatched prompt engineeringDER
Red Teamingmatched promptfooDER

Filed underAgent InfrastructureData & KnowledgeAgent ObservabilityModels & GatewaysRetrieval & RAGLLM Delegation

Integrations DER

No third-party service is named in the description or package metadata.

Platform compatibility

No pairing has been tested by AgentSidekick. Each status says what kind of evidence exists, not whether it works for you.

Unknown47

No compatibility evidence collected for this product type.

Install

Read from registry metadata. Follow the publisher's instructions; we have not run this.

No machine-readable install metadata is on record. The publisher's instructions are at the documentation.

Permissions and signals

Signals, not a safety verdict. They describe what the product's metadata references. Security review: not reviewed.

No signals collected

Automated inspection found nothing to report, or there was no metadata to inspect. That is not evidence of safety.

Known advisories for the published package SRC

An advisory existing does not mean the listed version is affected: compare the vulnerable range with the version you install. None found is not evidence of safety.

No advisories found

No published advisory matches a package this product declares. Products without a declared package cannot be matched at all. This is not evidence of safety.

Evaluation

Separate dimensions, each with its evidence. There is no overall score: the dimensions do not add up to one number honestly.

Source authority

Checked against official sources

Record researched by AgentSidekick on 2026-11-21; sources listed below.

Publisher verification

Organisation account

Published under a GitHub organisation or named vendor. Identity not independently confirmed.

Maintenance activity

Active

Last push 2026-11-21 (0 days before this page was built).

Licence clarity

MIT

Licence identifier as reported by Curated research.

Installation clarity

Linked

An install or documentation link is available; steps are not machine-readable.

Documentation

Docs linked

A documentation URL is on record.

Compatibility evidence

Unknown

No compatibility evidence collected.

Security review

Not reviewed

AgentSidekick has not audited this product. No automated signals were collected. That is not evidence of safety.

Pricing

Pricing model: open-source CUR No plan prices are on record.

Repository activity

promptfoo/promptfoo. Read from the GitHub API on 2026-11-21.

28kSRCGitHub stars
2.8kSRCForks
170SRCOpen issues
todaySRCLast push2026-11-21
code-scan-action-0.2.3SRCLatest release2026-11-18

Version history SRC

No version list is published in any source we track.

Change history

What changed between two ingests of the same source, and when the product was first seen. Registry releases are under Version history.

  1. First seen

Competitors and alternativesDER

Full alternatives page

Paired by shared capabilities and integrations. Similar on paper; not tested side by side.

Direct

Same type, overlapping capabilities.

Agenta

Developer toolby AgentaCUR

4.9k GitHub starspushed todayMIT

SharesAgent Evaluation

Ragas

Libraryby Vibrant LabsCUR

16k GitHub starspushed 7d agoApache-2.0

SharesRetrieval-Augmented GenerationAgent Evaluation

Giskard

Libraryby GiskardCUR

6.0k GitHub starspushed 1d agoApache-2.0

SharesAgent Evaluation

TruLens

Libraryby SnowflakeCUR

3.7k GitHub starspushed todayMIT

SharesAgent Evaluation

SharesAgent Evaluation

Inspect

Frameworkby UK AI Security InstituteCUR

3.3k GitHub starspushed todayMIT

SharesAgent Evaluation

Adjacent

A different form or a partial overlap.

aludel

Infrastructureby cristianodabcHEU

41 GitHub starspushed 4d agoApache-2.0

SharesPrompt OptimizationEvaluationRed TeamingAgent Evaluation

ai-evaluation-tools

Infrastructureby aglio-labHEU

16 GitHub starspushed 3mo agoCC0-1.0

SharesEvaluationRed TeamingRetrieval-Augmented GenerationAgent Evaluation

skillcheck

Rules packby sx4imHEU

31 GitHub starspushed 4d agoMIT

SharesPrompt OptimizationEvaluationAgent Evaluation

judgeval

Infrastructureby JudgmentLabsHEU

1.1k GitHub starspushed 1d agoApache-2.0

SharesPrompt OptimizationEvaluationAgent Evaluation

gepa

Agent skillby gepa-aiHEU

7.5k GitHub starspushed todayMIT

SharesPrompt OptimizationAgent Evaluation

awesome-evals

Infrastructureby benchflow-aiHEU

1.0k GitHub starspushed 16d agolicence unknown

SharesEvaluationAgent Evaluation

Head-to-head pagesvs OpenAI Evalsvs DeepEvalvs Ragasvs Giskardvs Agentavs TruLensvs Inspectvs Braintrust

Provenance

Which source set each field, what kind of claim it is, and when.

FieldSourceKindSet on
Canonical URLCurated researchCUR2026-11-21
PublisherCurated researchCUR2026-11-21
DescriptionCurated researchCUR2026-11-21
DocumentationCurated researchCUR2026-11-21
Integration typeCurated researchCUR2026-11-21
LanguagesCurated researchCUR2026-11-21
LicenceCurated researchCUR2026-11-21
NameGitHub topic searchSRC2026-11-21
Open sourceCurated researchCUR2026-11-21
Pricing modelCurated researchCUR2026-11-21
TypeCurated researchCUR2026-11-21
PublishedGitHub topic searchSRC2026-11-21
RepositoryGitHub topic searchSRC2026-11-21
Source URLCurated researchCUR2026-11-21
WebsiteCurated researchCUR2026-11-21

Sources

Last verified by a researcher on . Report incorrect information

Page data last updated .