Agent Observability
Seeing, measuring and evaluating what agents do. As of 2026-11-21, 951 indexed products match this category (440 with full profiles); 367 were published in the last 90 days, against 210 in the 90 days before, slower than the catalogue as a whole. By GitHub stars, iFixAi leads, holding 6% of the stars of the 736 products here with their own repository. The most common form is MCP servers (64% of indexed products).
CategorySegment label: Lagging the catalogue DER
Segment statistics
Whole catalogue (all 951 indexed products, not only profiled ones). Calculated 2026-11-21.
Lagging the catalogue. Mid-sized for its level, with launches growing more slowly than the catalogue’s. Size is ranked against the other category nodes; momentum compares this node’s launches in the last 90 days with the 90 before, relative to the whole catalogue’s ratio (score 62 or more = faster, 38 or less = slower; below 5 dated launches it is not measured and the label uses size alone).
Concentration. The Herfindahl–Hirschman index (HHI) adds up each project’s squared share of stars. At 0.023, attention is spread as if across about 44 equally popular projects. Using the common competition-policy thresholds (0.15 and 0.25), that is unconcentrated. Stars measure attention, not usage or quality.
Which product forms serve it
The same need is met by MCP servers, agent skills, plugins, extensions and frameworks. This counts each form across the whole catalogue.
| Product type | Indexed | Share | With a profile |
|---|---|---|---|
| MCP servers | 613 | 64.5% | 122 |
| Infrastructure | 174 | 18.3% | 168 |
| Agent skills | 62 | 6.5% | 62 |
| Plugins | 32 | 3.4% | 25 |
| Libraries | 15 | 1.6% | 14 |
| Frameworks | 14 | 1.5% | 14 |
| Developer tools | 13 | 1.4% | 13 |
| Agent projects | 9 | 0.95% | 9 |
| Rules packs | 8 | 0.84% | 2 |
| SaaS products | 5 | 0.53% | 5 |
| Agent SDKs | 2 | 0.21% | 2 |
| IDE extensions | 2 | 0.21% | 2 |
| Protocols & standards | 1 | 0.11% | 1 |
| AI APIs | 1 | 0.11% | 1 |
2 subcategories
Indexed = whole catalogue; profiled = products with their own page. A product can sit in several, so the rows do not add up to the total.
| Subcategory | Indexed | Profiled | New, 90 days | Label |
|---|---|---|---|---|
| Agent MonitoringTracing, logging and tracking cost and health of LLM apps. | 515 | 262 | 166 | Lagging the catalogue |
| Agent EvaluationScoring, benchmarking and red-teaming models and agents. | 507 | 224 | 215 | Lagging the catalogue |
Top 40 products by GitHub stars
Search the directoryProfiled products only (440 of 951), ranked by the stars of their own repository. Products that live in a subfolder of a larger repository (common for skills) show that whole repository's stars and are ranked after them; products without a repository come last. Stars are popularity, not quality or installs.
| # | Product | Type | Publisher | Capabilities | GitHub stars | Last push | Licence | Save and compare |
|---|---|---|---|---|---|---|---|---|
| 1 | iFixAiIndependent Auditing of AI Agents. Run by human or the agent itself, to answer the most crucial question in the AI Agent Economy. Is the agent doing what is supposed to do? With iFixAi you can have this answer in less than 120 seconds. | Infrastructure | ifixai-ai | 47k | 1d ago | Apache-2.0 | ||
| 2 | agentsPlugins, agents, skills, and commands for Claude Code, Codex CLI, Cursor, OpenCode, Antigravity CLI, GitHub Copilot, and Pi, written once in Markdown and adapted to each tool by generators with validation and evals. | Agent skill | wshobson | 43k | 3d ago | MIT | ||
| 3 | LightRAGGraph-enhanced RAG system that indexes entities and relations alongside vectors for dual-level retrieval, with a server and web UI. | Library | HKU Data Intelligence Lab | 41k | today | MIT | ||
| 4 | LangfuseOpen-source LLM engineering platform for tracing, prompt management, evaluations and datasets, self-hosted or Langfuse Cloud. | Developer tool | ClickHouse (Langfuse) | 38k | today | MIT | ||
| 5 | OpenAI Agents SDKLightweight SDK for multi-agent workflows with agents, handoffs, guardrails, sessions and built-in tracing; works with OpenAI and other model providers. | SDK | openai | 31k | today | MIT | ||
| 6 | MastraTypeScript framework for agents, workflows, RAG, memory and evals, with a local dev playground and deploy targets for common hosts. | Framework | Mastra | 30k | today | Apache-2.0 | ||
| 7 | MLflowOpen-source ML/GenAI lifecycle platform whose GenAI features include OpenTelemetry-compatible tracing, evaluation and prompt registry for agents. | Developer tool | Databricks (Linux Foundation) | 29k | today | Apache-2.0 | ||
| 8 | promptfooCLI and library for declarative prompt/model test suites, side-by-side comparisons and automated red-teaming of LLM apps and agents. | Developer tool | OpenAI (Promptfoo) | 28k | today | MIT | ||
| 9 | OpikOpen-source platform for tracing, evaluating and monitoring LLM apps and agents, with LLM-as-judge metrics and prompt optimisation. | Developer tool | Comet | 23k | today | Apache-2.0 | ||
| 10 | serve☁️ Build multimodal AI applications with cloud-native stack | Infrastructure | jina-ai | 22k | 1y ago | Apache-2.0 | ||
| 11 | OpenAI EvalsFramework and registry of benchmarks for evaluating LLMs; OpenAI also offers a hosted Evals API in its platform. | Framework | openai | 20k | 7mo ago | MIT | ||
| 12 | DeepEvalPytest-style framework for evaluating LLM outputs, RAG pipelines and agents with research-backed metrics and synthetic test generation. | Library | confident-ai | 20k | 1d ago | Apache-2.0 | ||
| 13 | deepevalSkills for adding DeepEval evaluations, tracing, datasets, Confident AI reports, and iterative improvement loops to AI applications. | Plugin | confident-ai | 20k | 1d ago | Apache-2.0 | ||
| 14 | artemisARTEMIS turns natural-language instructions into reliable Android automation. It automates end-to-end workflows, captures logs, and integrates seamlessly with AI coding assistants such as Antigravity, Codex, and Claude Code. It also achieves 99%+ success rate on AndroidWorld Benchmark. | Agent project | 18k | 1mo ago | Apache-2.0 | |||
| 15 | RagasLibrary of metrics and test-set generation for evaluating RAG and agent applications, including faithfulness and context precision. | Library | Vibrant Labs | 16k | 7d ago | Apache-2.0 | ||
| 16 | RagaAI-CatalystPython SDK for Agent AI Observability, Monitoring and Evaluation Framework. Includes features like agent, llm and tools tracing, debugging multi-agentic system, self-hosted dashboard and advanced analytics with timeline and execution graph view | Infrastructure | raga-ai-hub | 16k | 9mo ago | Apache-2.0 | ||
| 17 | Arize PhoenixSelf-hostable tracing and evaluation tool built on OpenTelemetry/OpenInference for inspecting LLM and agent runs, datasets and experiments. | Developer tool | Arize-ai | 12k | today | Elastic-2.0 | ||
| 18 | CodeBurnAI coding spend, tokens and plan quotas across Claude Code, Codex, Cursor and 40+ more tools | MCP server | getagentseal | 12k | 1d ago | MIT | ||
| 19 | VoltAgentTypeScript agent framework with memory, workflows, supervisor/sub-agent patterns and a companion observability console. | Framework | VoltAgent | 11k | 15d ago | MIT | ||
| 20 | garakLLM vulnerability scanner that probes models and apps for jailbreaks, prompt injection, data leakage and toxicity. | Developer tool | NVIDIA | 10k | 3d ago | Apache-2.0 | ||
| 21 | failproofaiObservability and enforcement for AI agent harnesses. Capture every run and runtime reliability with policy enforcement. | Infrastructure | FailproofAI | 9.9k | today | — | ||
| 22 | BifrostGo-based LLM gateway exposing many providers through one OpenAI-compatible API with failover, load balancing, governance and an MCP gateway. | Infrastructure | Maxim AI | 9.9k | today | Apache-2.0 | ||
| 23 | oumiEasily fine-tune (SFT/RL), evaluate, and deploy Qwen, Gemma, or any open agentic LLM/VLM! | Infrastructure | oumi-ai | 9.4k | today | Apache-2.0 | ||
| 24 | evidentlyEvidently is an open-source ML and LLM observability framework. Evaluate, test, and monitor any AI-powered system or data pipeline. From tabular data to Gen AI. 100+ metrics. | Infrastructure | evidentlyai | 8.1k | 1mo ago | Apache-2.0 | ||
| 25 | llm-gatewayOpen Source LLM Gateway 🦚 | Infrastructure | mnfst | 7.7k | 4d ago | MIT | ||
| 26 | AI-Infra-GuardA full-stack AI Red Teaming platform securing AI ecosystems via Agent Scan, Skills Scan, MCP scan, AI Infra scan and LLM jailbreak evaluation. | Infrastructure | Tencent | 7.6k | 1d ago | Apache-2.0 | ||
| 27 | OpenLLMetryOpenTelemetry-based instrumentation for LLM providers, vector DBs and frameworks that exports traces to any OTel backend. | Library | traceloop | 7.5k | 2d ago | Apache-2.0 | ||
| 28 | gepaOptimize prompts, code, and more with AI-powered Reflective Optimization | Agent skill | gepa-ai | 7.5k | today | MIT | ||
| 29 | greptimedbThe open-source observability database. One columnar engine for metrics, logs, and traces, on object storage. | Agent project | GreptimeTeam | 6.8k | today | Apache-2.0 | ||
| 30 | chinese-llm-benchmark非线智能 NoneLinear - ReLE评测:中文AI大模型能力评测(持续更新):目前已囊括374个大模型,覆盖chatgpt、gpt-5.4、谷歌gemini-3.1-pro、Claude-4.6、文心ERNIE-X1.1、ERNIE-5.0、qwen3.6-max、qwen3.6-plus、百川、讯飞星火、商汤senseChat等商用模型, 以及step3.5-flash、kimi-k2.6、ernie4.5、MiniMax-M2.7、deepseek-v4、Qwen3.6、llama4、智谱GLM-5.1、MiMo-V2、LongCat、gemma4、mistral等开源大模型。不仅提供排行榜,也提供规模超200万的大模型缺陷库!方便广大社区研究分析、改进大模型。 | Infrastructure | jeinlee1991 | 6.5k | 1d ago | — | ||
| 31 | mission-controlSelf-hosted control plane for AI agents: dispatch tasks, review runs, track spend, and operate OpenClaw, Claude Code, Codex, and other runtimes. | Agent project | builderz-labs | 6.5k | 4d ago | MIT | ||
| 32 | OuroborosPins an acceptance spec; the verify command and expected output never enter the success contract. | MCP server | Q00 | 6.4k | today | MIT | ||
| 33 | HeliconeProxy-based LLM observability that logs requests, costs and latency with one-line integration, plus an AI gateway. | Developer tool | Mintlify (Helicone) | 6.3k | 4d ago | Apache-2.0 | ||
| 34 | AgentOpsSDK and dashboard for monitoring agent sessions, tool calls, costs and failures, with integrations for CrewAI, OpenAI Agents SDK, AG2 and others. | Developer tool | AgentOps | 6.0k | 4mo ago | MIT | ||
| 35 | GiskardTesting library that scans LLM agents and RAG apps for hallucination, prompt injection and harmful-content vulnerabilities and generates test suites. | Library | Giskard | 6.0k | 1d ago | Apache-2.0 | ||
| 36 | tokscale🛰️ Track token usage across AI coding agents from your terminal. 🏅 Global leaderboard with trillions of tokens tracked. | Agent project | junhoyeo | 5.9k | 3d ago | MIT | ||
| 37 | coze-loopNext-generation AI Agent Optimization Platform: Cozeloop addresses challenges in AI agent development by providing full-lifecycle management capabilities from development, debugging, and evaluation to monitoring. | Infrastructure | coze-dev | 5.8k | 4d ago | Apache-2.0 | ||
| 38 | LLM-Engineers-HandbookThe LLM's practical guide: From the fundamentals to deploying advanced LLM and RAG apps to AWS using LLMOps best practices | Infrastructure | PacktPublishing | 5.4k | 7mo ago | MIT | ||
| 39 | awesome-harness-engineeringAwesome list for AI agent harness engineering: tools, patterns, evals, memory, MCP, permissions, observability, and orchestration. | Agent project | ai-boost | 5.2k | today | — | ||
| 40 | LangWatchPlatform for LLM tracing, evaluations and agent simulation testing, self-hosted or cloud. | Developer tool | langwatch | 5.0k | today | Apache-2.0 |
Most common integrations
Third-party services named by products in this node, whole catalogue.
| Integration | Products | Share |
|---|---|---|
| Google Cloud | 9 | 0.9% |
| GitHub | 8 | 0.8% |
| AWS | 7 | 0.7% |
| PostgreSQL | 7 | 0.7% |
| Azure | 6 | 0.6% |
| Kubernetes | 4 | 0.4% |
| Datadog | 3 | 0.3% |
| Grafana | 3 | 0.3% |
| Android | 2 | 0.2% |
| Discord | 2 | 0.2% |
| Solana | 2 | 0.2% |
| Stripe | 2 | 0.2% |
| Terraform | 2 | 0.2% |
| Blender | 1 | 0.1% |
| Google Ads | 1 | 0.1% |
Launches per month
New Agent Observability products per monthSRC
Whole catalogue, by publication month, 2024-11 to 2026-11. The last month is partial; undated products are excluded.
Show the data as a table
| Products published (products) | |
|---|---|
| 2024-11 | 2 |
| 2024-12 | 4 |
| 2025-01 | 4 |
| 2025-02 | 2 |
| 2025-03 | 3 |
| 2025-04 | 4 |
| 2025-05 | 6 |
| 2025-06 | 3 |
| 2025-07 | 4 |
| 2025-08 | 1 |
| 2025-09 | 8 |
| 2025-10 | 6 |
| 2025-11 | 3 |
| 2025-12 | 8 |
| 2026-01 | 5 |
| 2026-02 | 27 |
| 2026-03 | 45 |
| 2026-04 | 50 |
| 2026-05 | 55 |
| 2026-06 | 53 |
| 2026-07 | 70 |
| 2026-08 | 106 |
| 2026-09 | 81 |
| 2026-10 | 145 |
| 2026-11 | 114 |
Related opportunities
Opportunity radarScored gaps for this node or anything beneath it. Scores rank hypotheses; they do not predict demand.
| Opportunity | Competition | Confidence | Score |
|---|---|---|---|
| Coverage gap: LoggingLogging | Medium | Low | 59.1 |
| Coverage gap: Red TeamingRed Teaming | High | Low | 58.1 |
| Coverage gap: Performance MonitoringPerformance Monitoring | High | Low | 55.5 |
Limits of these numbers
- Products are placed here by whole-word keyword rules on their name, description and repository topics, and inherit every ancestor of a matched node. Products that do this without naming it are missed; ambiguous words cause false matches. Methodology
- About half of the catalogue matches no node at all, so absolute counts understate every category.
- Launch counts use the publication date each source reports; for auto-discovered repositories that is the repository’s creation date.
- Stars and concentration cover only products with their own repository (not a subfolder of a larger one). They measure attention, not adoption.
Page data last updated .