Data Pipelines
Moving and transforming data between systems. As of 2026-11-21, 172 indexed products match this subcategory (96 with full profiles); 67 were published in the last 90 days, against 33 in the 90 days before, slower than the catalogue as a whole. By GitHub stars, kestra leads, holding 22% of the stars of the 127 products here with their own repository. The most common form is MCP servers (77% of indexed products).
SubcategorySegment label: Thin DER
Segment statistics
Whole catalogue (all 172 indexed products, not only profiled ones). Calculated 2026-11-21.
Thin. Among the smallest 30% of nodes at its level, and launches are not outpacing the catalogue. Size is ranked against the other subcategory nodes; momentum compares this node’s launches in the last 90 days with the 90 before, relative to the whole catalogue’s ratio (score 62 or more = faster, 38 or less = slower; below 5 dated launches it is not measured and the label uses size alone).
Concentration. The Herfindahl–Hirschman index (HHI) adds up each project’s squared share of stars. At 0.095, attention is spread as if across about 10 equally popular projects. Using the common competition-policy thresholds (0.15 and 0.25), that is unconcentrated. Stars measure attention, not usage or quality.
Which product forms serve it
The same need is met by MCP servers, agent skills, plugins, extensions and frameworks. This counts each form across the whole catalogue.
| Product type | Indexed | Share | With a profile |
|---|---|---|---|
| MCP servers | 133 | 77.3% | 57 |
| Agent skills | 13 | 7.6% | 13 |
| Plugins | 12 | 7.0% | 12 |
| Infrastructure | 7 | 4.1% | 7 |
| Agent projects | 5 | 2.9% | 5 |
| IDE extensions | 1 | 0.58% | 1 |
| Automation platforms | 1 | 0.58% | 1 |
3 capabilities
Indexed = whole catalogue; profiled = products with their own page. A product can sit in several, so the rows do not add up to the total.
| Capability | Indexed | Profiled | New, 90 days | Label |
|---|---|---|---|---|
| Data TransformationCleaning and reshaping datasets. | 67 | 32 | 25 | Lagging the catalogue |
| ETL OrchestrationScheduling batch data jobs. | 50 | 25 | 14 | Lagging the catalogue |
| Streaming DataEvent streams and message queues. | 50 | 39 | 25 | Established |
Top 40 products by GitHub stars
Search the directoryProfiled products only (96 of 172), ranked by the stars of their own repository. Products that live in a subfolder of a larger repository (common for skills) show that whole repository's stars and are ranked after them; products without a repository come last. Stars are popularity, not quality or installs.
| # | Product | Type | Publisher | Capabilities | GitHub stars | Last push | Licence | Save and compare |
|---|---|---|---|---|---|---|---|---|
| 1 | kestraOrchestrate everything, everywhere. The open-source control plane for workflows across data, infrastructure, business processes and AI agents. Declarative YAML, event-driven, any language, Git-versioned, scalable to millions of executions. | Agent project | kestra-io | 30k | 1d ago | Apache-2.0 | ||
| 2 | WindmillDeveloper platform that turns scripts in many languages into workflows, UIs and scheduled jobs, with AI steps and agents. | Automation platform | windmill-labs | 19k | today | AGPL-3.0 | ||
| 3 | mcpReal-time Kubernetes network traffic visibility and API analysis for HTTP, gRPC, Redis, Kafka, DNS. | MCP server | kubeshark | 12k | 11d ago | Apache-2.0 | ||
| 4 | cocoindexIncremental engine for long horizon agents 🌟 Star if you like it! | Agent project | cocoindex-io | 12k | today | Apache-2.0 | ||
| 5 | evidentlyEvidently is an open-source ML and LLM observability framework. Evaluate, test, and monitor any AI-powered system or data pipeline. From tabular data to Gen AI. 100+ metrics. | Infrastructure | evidentlyai | 8.1k | 1mo ago | Apache-2.0 | ||
| 6 | unstractLLM-Driven Extraction of Unstructured Data — Built for API Deployments & ETL Pipeline Workflows | MCP server | Zipstack | 7.5k | 1d ago | AGPL-3.0 | ||
| 7 | pinotApache Pinot: a real-time analytics database for low-latency SQL and vector search, serving user-facing and agent-facing applications | Agent project | apache | 6.2k | today | Apache-2.0 | ||
| 8 | JitsuManage Jitsu data pipelines: destinations, streams, connections, functions, live events. | MCP server | jitsu.com | 5.1k | 1d ago | MIT | ||
| 9 | whodbWhere data access meets operational intelligence | MCP server | clidey | 5.1k | today | Apache-2.0 | ||
| 10 | memgraphHigh-performance open-source in-memory graph database for GraphRAG, AI memory, agentic AI, and real-time graph analytics. Cypher-compatible, built in C++. | Agent project | memgraph | 4.7k | today | — | ||
| 11 | daguSelf-hostable workflow orchestrator for teams whose main work isn't orchestration. Declarative YAML over your scripts, SSH commands, containers, etc; keep workflows separate from business logic. One binary, no database, runs on limited H/W resources. Alternative to Airflow / Cron / Job Scheduler. | MCP server | dagucloud | 4.6k | today | GPL-3.0 | ||
| 12 | pythonOpen-source Python challenge courses — your AI coding agent teaches, verify.py decides when you're done. | Agent project | flypythoncom | 4.2k | 11d ago | — | ||
| 13 | aws-data-analyticsData lake, analytics, and ETL workflows with S3 Tables, AWS Glue, and Athena. | Plugin | aws | 3.5k | today | Apache-2.0 | ||
| 14 | hamiltonApache Hamilton helps data scientists and engineers define testable, modular, self-documenting dataflows, that encode lineage/tracing and metadata. Runs and scales everywhere python does. | Infrastructure | apache | 2.6k | 5d ago | Apache-2.0 | ||
| 15 | OpenMeterReal-time usage metering and billing for AI and API products, aggregating events into meters, entitlements and invoices. | Infrastructure | OpenMeter | 2.5k | today | Apache-2.0 | ||
| 16 | GoNaviHigh-performance multi-data-source database client — ~30MB, AI & MCP ready, zero Electron bloat. | 高性能多数据源数据库客户端:约 30MB,AI 与 MCP 就绪,告别 Electron 膨胀。 | MCP server | Syngnat | 2.1k | 1d ago | Apache-2.0 | ||
| 17 | Seal Open DataFrench public open data (data.gouv.fr) cross-referenced by commune, EPCI and department. | MCP server | sealopendata.fr | 1.6k | 1d ago | MIT | ||
| 18 | OpenContractsThe open document intelligence platform for builders and hackers - DMS for the agentic world | Infrastructure | Open-Source-Legal | 1.6k | today | MIT | ||
| 19 | NeumAINeum AI is a best-in-class framework to manage the creation and synchronization of vector embeddings at large scale. | Infrastructure | NeumTry | 868 | 2y ago | Apache-2.0 | ||
| 20 | dbt-mcpA MCP (Model Context Protocol) server for interacting with dbt. | MCP server | dbt-labs | 624 | 3d ago | Apache-2.0 | ||
| 21 | components-contribCommunity driven, reusable components for distributed, durable, and agentic AI applications across all cloud service providers | Infrastructure | dapr | 605 | 1d ago | Apache-2.0 | ||
| 22 | Power User for dbtMakes VS Code work seamlessly with dbt core and dbt cloud: auto-complete, preview, column lineage, AI docs generation, health checks, cost estimation etc | IDE extension | Altimate Inc. | 586 | today | MIT | ||
| 23 | FinSight-AIAI equity research agent with resilient workflows, evidence-grounded RAG, versioned reports, and automated quality evaluation. | Infrastructure | juanjuandog | 562 | 2mo ago | MIT | ||
| 24 | csv-data-summarizer-claude-skillA Claude Skill that automatically analyzes uploaded CSV files — generating summary statistics, detecting missing data, and creating quick visualizations using Python and pandas. | Agent skill | coffeefuelbump | 487 | 1y ago | — | ||
| 25 | huashu-excel数据分析与 Excel 全流程 skill:体检脏表、清洗、对齐需求、分析、对账、交付。让 AI 算出来的数字经得起追问。跨 agent 通用,依赖仅 openpyxl。 | Agent skill | alchaincyf | 477 | 6d ago | MIT | ||
| 26 | astronomer-data-agentsData engineering for Apache Airflow and Astronomer. Author DAGs with best practices, debug pipeline failures, trace data lineage, profile tables, migrate Airflow 2 to 3, and manage local and cloud deployments. | Plugin | astronomer | 462 | 1d ago | Apache-2.0 | ||
| 27 | dataData engineering for Apache Airflow and Astronomer. Author DAGs with best practices, debug pipeline failures, trace data lineage, profile tables, migrate Airflow 2 to 3, and manage local and cloud deployments. | Plugin | astronomer | 462 | 1d ago | Apache-2.0 | ||
| 28 | data-engineeringData engineering plugin - warehouse exploration, pipeline authoring, Airflow integration | Plugin | astronomer | 462 | 1d ago | Apache-2.0 | ||
| 29 | data-agent-kitThis plugin provides a specialized suite of skills for data engineers and database practitioners working on Google Cloud. It acts as an expert assistant, allowing you to use natural language prompts in your preferred coding agent to architect complex data pipelines, transform data with dbt, write Spark and BigQuery SQL notebooks, and orchestrate end-to-end workflows across GCP's data ecosystem. | Plugin | GoogleCloudPlatform | 252 | 1d ago | Apache-2.0 | ||
| 30 | dexDex lets your agent build your analytics. | Plugin | exmergo | 141 | today | Apache-2.0 | ||
| 31 | GoldenAnalysisRead-only cross-cutting analysis, metrics, and reporting across the Golden Suite. | MCP server | benseverndev-oss | 139 | today | MIT | ||
| 32 | GoldenMatchFind duplicate records in 30 seconds. Zero-config entity resolution, 97.2% F1 out of the box. | MCP server | benseverndev-oss | 139 | today | MIT | ||
| 33 | mcp-cn-commerce8 read-only MCP platform entries; support is operation-specific; merchant live acceptance pending | MCP server | TonyWang-hub | 102 | 5d ago | MIT | ||
| 34 | strimzi-kafka-cliManage Apache Kafka on Kubernetes via Strimzi. Clusters, topics, users, connectors, MM2. | MCP server | SystemCraftsman | 91 | 3mo ago | Apache-2.0 | ||
| 35 | mcpConnect your AI assistants to Keboola and expose your data, transformations, SQL queries, ... | MCP server | keboola.com | 84 | today | MIT | ||
| 36 | Misata: synthetic data libraryDeclarative synthetic data: multi-table data that holds the facts you declare, checked by DuckDB. | MCP server | rasinmuhammed | 79 | 20d ago | MIT | ||
| 37 | dbtrailMySQL change tracking with instant row-level recovery and forensic attribution for compliance. | MCP server | dbtrail.com | 73 | today | Apache-2.0 | ||
| 38 | ClariLayerTurns selected past conversations into durable context your AI can use across projects and sessions. | MCP server | clarilayer | 68 | today | — | ||
| 39 | streaming-skills-pluginSkills for streaming application developers, covering Kafka and Flink client libraries and Schema Registry | Plugin | confluentinc | 66 | 9d ago | Apache-2.0 | ||
| 40 | oracle-ai-data-platform-workbench-databricks-migratorDrive the Oracle AI Data Platform (AIDP) Databricks Migration Toolkit in natural language. Plans and executes automated Databricks → AIDP migrations of notebooks, jobs, schedules, and catalog DDL — Pass-1 dependency rewrite + Pass-2 cell-by-cell execute/verify/fix on a live AIDP cluster with Claude. | Plugin | oracle-samples | 62 | 1d ago | UPL-1.0 |
Most common integrations
Third-party services named by products in this node, whole catalogue.
| Integration | Products | Share |
|---|---|---|
| Apache Kafka | 27 | 15.7% |
| PostgreSQL | 13 | 7.6% |
| MySQL | 6 | 3.5% |
| Redis | 6 | 3.5% |
| Elasticsearch | 5 | 2.9% |
| AWS | 3 | 1.7% |
| BigQuery | 3 | 1.7% |
| MongoDB | 3 | 1.7% |
| Snowflake | 3 | 1.7% |
| Databricks | 2 | 1.2% |
| Kubernetes | 2 | 1.2% |
| Azure | 1 | 0.6% |
| GitHub | 1 | 0.6% |
| Google Cloud | 1 | 0.6% |
| Google Sheets | 1 | 0.6% |
Launches per month
New Data Pipelines products per monthSRC
Whole catalogue, by publication month, 2025-03 to 2026-11. The last month is partial; undated products are excluded.
Show the data as a table
| Products published (products) | |
|---|---|
| 2025-03 | 2 |
| 2025-09 | 1 |
| 2025-10 | 4 |
| 2026-01 | 4 |
| 2026-02 | 8 |
| 2026-03 | 5 |
| 2026-04 | 6 |
| 2026-05 | 10 |
| 2026-06 | 10 |
| 2026-07 | 10 |
| 2026-08 | 17 |
| 2026-09 | 14 |
| 2026-10 | 18 |
| 2026-11 | 27 |
Related opportunities
Opportunity radarScored gaps for this node or anything beneath it. Scores rank hypotheses; they do not predict demand.
No scored opportunity here
Capability gaps are only scored inside subcategories with at least 20 indexed products, and only the top 80 are kept. None falls under this node.
Limits of these numbers
- Products are placed here by whole-word keyword rules on their name, description and repository topics, and inherit every ancestor of a matched node. Products that do this without naming it are missed; ambiguous words cause false matches. Methodology
- About half of the catalogue matches no node at all, so absolute counts understate every category.
- Launch counts use the publication date each source reports; for auto-discovered repositories that is the repository’s creation date.
- Stars and concentration cover only products with their own repository (not a subfolder of a larger one). They measure attention, not adoption.
Page data last updated .