AgentSidekick

Agent Observability

Seeing, measuring and evaluating what agents do. As of 2026-11-21, 951 indexed products match this category (440 with full profiles); 367 were published in the last 90 days, against 210 in the 90 days before, slower than the catalogue as a whole. By GitHub stars, iFixAi leads, holding 6% of the stars of the 736 products here with their own repository. The most common form is MCP servers (64% of indexed products).

CategorySegment label: Lagging the catalogue DER

Segment statistics

Whole catalogue (all 951 indexed products, not only profiled ones). Calculated 2026-11-21.

951DERIndexed products440 with full profiles
367DERLaunched, last 90 days210 in the 90 days before
23DERMomentum vs catalogue50 = same pace as the catalogue
6%DERLargest project's share of starsAcross 736 products with their own repository
0.023DERConcentration (HHI)About 44 equal-sized projects’ worth; unconcentrated
11SRCMedian starsPer product with its own repository
71%DEROpen sourceMarked open source; unknown counts as not

Lagging the catalogue. Mid-sized for its level, with launches growing more slowly than the catalogue’s. Size is ranked against the other category nodes; momentum compares this node’s launches in the last 90 days with the 90 before, relative to the whole catalogue’s ratio (score 62 or more = faster, 38 or less = slower; below 5 dated launches it is not measured and the label uses size alone).

Concentration. The Herfindahl–Hirschman index (HHI) adds up each project’s squared share of stars. At 0.023, attention is spread as if across about 44 equally popular projects. Using the common competition-policy thresholds (0.15 and 0.25), that is unconcentrated. Stars measure attention, not usage or quality.

Which product forms serve it

The same need is met by MCP servers, agent skills, plugins, extensions and frameworks. This counts each form across the whole catalogue.

Products in Agent Observability by product type, as of 2026-11-21
Product type IndexedShareWith a profile
MCP servers61364.5%122
Infrastructure17418.3%168
Agent skills626.5%62
Plugins323.4%25
Libraries151.6%14
Frameworks141.5%14
Developer tools131.4%13
Agent projects90.95%9
Rules packs80.84%2
SaaS products50.53%5
Agent SDKs20.21%2
IDE extensions20.21%2
Protocols & standards10.11%1
AI APIs10.11%1

2 subcategories

Indexed = whole catalogue; profiled = products with their own page. A product can sit in several, so the rows do not add up to the total.

SubcategoryIndexed ProfiledNew, 90 daysLabel
Agent MonitoringTracing, logging and tracking cost and health of LLM apps.515262166Lagging the catalogue
Agent EvaluationScoring, benchmarking and red-teaming models and agents.507224215Lagging the catalogue

Top 40 products by GitHub stars

Search the directory

Profiled products only (440 of 951), ranked by the stars of their own repository. Products that live in a subfolder of a larger repository (common for skills) show that whole repository's stars and are ranked after them; products without a repository come last. Stars are popularity, not quality or installs.

Agent Observability: top profiled products by GitHub stars, 2026-11-21
#ProductTypePublisherCapabilities GitHub stars Last push LicenceSave and compare
1iFixAiIndependent Auditing of AI Agents. Run by human or the agent itself, to answer the most crucial question in the AI Agent Economy. Is the agent doing what is supposed to do? With iFixAi you can have this answer in less than 120 seconds.Infrastructureifixai-ai47k1d agoApache-2.0
2agentsPlugins, agents, skills, and commands for Claude Code, Codex CLI, Cursor, OpenCode, Antigravity CLI, GitHub Copilot, and Pi, written once in Markdown and adapted to each tool by generators with validation and evals.Agent skillwshobson43k3d agoMIT
3LightRAGGraph-enhanced RAG system that indexes entities and relations alongside vectors for dual-level retrieval, with a server and web UI.LibraryHKU Data Intelligence Lab41ktodayMIT
4LangfuseOpen-source LLM engineering platform for tracing, prompt management, evaluations and datasets, self-hosted or Langfuse Cloud.Developer toolClickHouse (Langfuse)38ktodayMIT
5OpenAI Agents SDKLightweight SDK for multi-agent workflows with agents, handoffs, guardrails, sessions and built-in tracing; works with OpenAI and other model providers.SDKopenai31ktodayMIT
6MastraTypeScript framework for agents, workflows, RAG, memory and evals, with a local dev playground and deploy targets for common hosts.FrameworkMastra30ktodayApache-2.0
7MLflowOpen-source ML/GenAI lifecycle platform whose GenAI features include OpenTelemetry-compatible tracing, evaluation and prompt registry for agents.Developer toolDatabricks (Linux Foundation)29ktodayApache-2.0
8promptfooCLI and library for declarative prompt/model test suites, side-by-side comparisons and automated red-teaming of LLM apps and agents.Developer toolOpenAI (Promptfoo)28ktodayMIT
9OpikOpen-source platform for tracing, evaluating and monitoring LLM apps and agents, with LLM-as-judge metrics and prompt optimisation.Developer toolComet23ktodayApache-2.0
10serve☁️ Build multimodal AI applications with cloud-native stackInfrastructurejina-ai22k1y agoApache-2.0
11OpenAI EvalsFramework and registry of benchmarks for evaluating LLMs; OpenAI also offers a hosted Evals API in its platform.Frameworkopenai20k7mo agoMIT
12DeepEvalPytest-style framework for evaluating LLM outputs, RAG pipelines and agents with research-backed metrics and synthetic test generation.Libraryconfident-ai20k1d agoApache-2.0
13deepevalSkills for adding DeepEval evaluations, tracing, datasets, Confident AI reports, and iterative improvement loops to AI applications.Pluginconfident-ai20k1d agoApache-2.0
14artemisARTEMIS turns natural-language instructions into reliable Android automation. It automates end-to-end workflows, captures logs, and integrates seamlessly with AI coding assistants such as Antigravity, Codex, and Claude Code. It also achieves 99%+ success rate on AndroidWorld Benchmark.Agent projectgoogle18k1mo agoApache-2.0
15RagasLibrary of metrics and test-set generation for evaluating RAG and agent applications, including faithfulness and context precision.LibraryVibrant Labs16k7d agoApache-2.0
16RagaAI-CatalystPython SDK for Agent AI Observability, Monitoring and Evaluation Framework. Includes features like agent, llm and tools tracing, debugging multi-agentic system, self-hosted dashboard and advanced analytics with timeline and execution graph view Infrastructureraga-ai-hub16k9mo agoApache-2.0
17Arize PhoenixSelf-hostable tracing and evaluation tool built on OpenTelemetry/OpenInference for inspecting LLM and agent runs, datasets and experiments.Developer toolArize-ai12ktodayElastic-2.0
18CodeBurnAI coding spend, tokens and plan quotas across Claude Code, Codex, Cursor and 40+ more toolsMCP servergetagentseal12k1d agoMIT
19VoltAgentTypeScript agent framework with memory, workflows, supervisor/sub-agent patterns and a companion observability console.FrameworkVoltAgent11k15d agoMIT
20garakLLM vulnerability scanner that probes models and apps for jailbreaks, prompt injection, data leakage and toxicity.Developer toolNVIDIA10k3d agoApache-2.0
21failproofaiObservability and enforcement for AI agent harnesses. Capture every run and runtime reliability with policy enforcement. InfrastructureFailproofAI9.9ktoday—
22BifrostGo-based LLM gateway exposing many providers through one OpenAI-compatible API with failover, load balancing, governance and an MCP gateway.InfrastructureMaxim AI9.9ktodayApache-2.0
23oumiEasily fine-tune (SFT/RL), evaluate, and deploy Qwen, Gemma, or any open agentic LLM/VLM!Infrastructureoumi-ai9.4ktodayApache-2.0
24evidentlyEvidently is ​​an open-source ML and LLM observability framework. Evaluate, test, and monitor any AI-powered system or data pipeline. From tabular data to Gen AI. 100+ metrics.Infrastructureevidentlyai8.1k1mo agoApache-2.0
25llm-gatewayOpen Source LLM Gateway 🦚Infrastructuremnfst7.7k4d agoMIT
26AI-Infra-GuardA full-stack AI Red Teaming platform securing AI ecosystems via Agent Scan, Skills Scan, MCP scan, AI Infra scan and LLM jailbreak evaluation.InfrastructureTencent7.6k1d agoApache-2.0
27OpenLLMetryOpenTelemetry-based instrumentation for LLM providers, vector DBs and frameworks that exports traces to any OTel backend.Librarytraceloop7.5k2d agoApache-2.0
28gepaOptimize prompts, code, and more with AI-powered Reflective OptimizationAgent skillgepa-ai7.5ktodayMIT
29greptimedbThe open-source observability database. One columnar engine for metrics, logs, and traces, on object storage.Agent projectGreptimeTeam6.8ktodayApache-2.0
30chinese-llm-benchmark非线智能 NoneLinear - ReLE评测:中文AI大模型能力评测(持续更新):目前已囊括374个大模型,覆盖chatgpt、gpt-5.4、谷歌gemini-3.1-pro、Claude-4.6、文心ERNIE-X1.1、ERNIE-5.0、qwen3.6-max、qwen3.6-plus、百川、讯飞星火、商汤senseChat等商用模型, 以及step3.5-flash、kimi-k2.6、ernie4.5、MiniMax-M2.7、deepseek-v4、Qwen3.6、llama4、智谱GLM-5.1、MiMo-V2、LongCat、gemma4、mistral等开源大模型。不仅提供排行榜,也提供规模超200万的大模型缺陷库!方便广大社区研究分析、改进大模型。Infrastructurejeinlee19916.5k1d ago—
31mission-controlSelf-hosted control plane for AI agents: dispatch tasks, review runs, track spend, and operate OpenClaw, Claude Code, Codex, and other runtimes.Agent projectbuilderz-labs6.5k4d agoMIT
32OuroborosPins an acceptance spec; the verify command and expected output never enter the success contract.MCP serverQ006.4ktodayMIT
33HeliconeProxy-based LLM observability that logs requests, costs and latency with one-line integration, plus an AI gateway.Developer toolMintlify (Helicone)6.3k4d agoApache-2.0
34AgentOpsSDK and dashboard for monitoring agent sessions, tool calls, costs and failures, with integrations for CrewAI, OpenAI Agents SDK, AG2 and others.Developer toolAgentOps6.0k4mo agoMIT
35GiskardTesting library that scans LLM agents and RAG apps for hallucination, prompt injection and harmful-content vulnerabilities and generates test suites.LibraryGiskard6.0k1d agoApache-2.0
36tokscale🛰️ Track token usage across AI coding agents from your terminal. 🏅 Global leaderboard with trillions of tokens tracked.Agent projectjunhoyeo5.9k3d agoMIT
37coze-loopNext-generation AI Agent Optimization Platform: Cozeloop addresses challenges in AI agent development by providing full-lifecycle management capabilities from development, debugging, and evaluation to monitoring. Infrastructurecoze-dev5.8k4d agoApache-2.0
38LLM-Engineers-HandbookThe LLM's practical guide: From the fundamentals to deploying advanced LLM and RAG apps to AWS using LLMOps best practicesInfrastructurePacktPublishing5.4k7mo agoMIT
39awesome-harness-engineeringAwesome list for AI agent harness engineering: tools, patterns, evals, memory, MCP, permissions, observability, and orchestration.Agent projectai-boost5.2ktoday—
40LangWatchPlatform for LLM tracing, evaluations and agent simulation testing, self-hosted or cloud.Developer toollangwatch5.0ktodayApache-2.0

Most common integrations

Third-party services named by products in this node, whole catalogue.

IntegrationProducts Share
Google Cloud90.9%
GitHub80.8%
AWS70.7%
PostgreSQL70.7%
Azure60.6%
Kubernetes40.4%
Datadog30.3%
Grafana30.3%
Android20.2%
Discord20.2%
Solana20.2%
Stripe20.2%
Terraform20.2%
Blender10.1%
Google Ads10.1%

Launches per month

New Agent Observability products per monthSRC

Whole catalogue, by publication month, 2024-11 to 2026-11. The last month is partial; undated products are excluded.

New Agent Observability products per monthWhole catalogue, by publication month, 2024-11 to 2026-11. The last month is partial; undated products are excluded.. Products published: 25 points, latest 114 products in 2026-11.
Show the data as a table
New Agent Observability products per month. Whole catalogue, by publication month, 2024-11 to 2026-11. The last month is partial; undated products are excluded.
Products published (products)
2024-112
2024-124
2025-014
2025-022
2025-033
2025-044
2025-056
2025-063
2025-074
2025-081
2025-098
2025-106
2025-113
2025-128
2026-015
2026-0227
2026-0345
2026-0450
2026-0555
2026-0653
2026-0770
2026-08106
2026-0981
2026-10145
2026-11114

Related opportunities

Opportunity radar

Scored gaps for this node or anything beneath it. Scores rank hypotheses; they do not predict demand.

OpportunityCompetitionConfidenceScore
Coverage gap: LoggingLoggingMediumLow59.1
Coverage gap: Red TeamingRed TeamingHighLow58.1
Coverage gap: Performance MonitoringPerformance MonitoringHighLow55.5

Limits of these numbers

  • Products are placed here by whole-word keyword rules on their name, description and repository topics, and inherit every ancestor of a matched node. Products that do this without naming it are missed; ambiguous words cause false matches. Methodology
  • About half of the catalogue matches no node at all, so absolute counts understate every category.
  • Launch counts use the publication date each source reports; for auto-discovered repositories that is the repository’s creation date.
  • Stars and concentration cover only products with their own repository (not a subfolder of a larger one). They measure attention, not adoption.

Page data last updated .