AgentSidekick

Data Pipelines

Moving and transforming data between systems. As of 2026-11-21, 172 indexed products match this subcategory (96 with full profiles); 67 were published in the last 90 days, against 33 in the 90 days before, slower than the catalogue as a whole. By GitHub stars, kestra leads, holding 22% of the stars of the 127 products here with their own repository. The most common form is MCP servers (77% of indexed products).

SubcategorySegment label: Thin DER

Segment statistics

Whole catalogue (all 172 indexed products, not only profiled ones). Calculated 2026-11-21.

172DERIndexed products96 with full profiles
67DERLaunched, last 90 days33 in the 90 days before
29DERMomentum vs catalogue50 = same pace as the catalogue
22%DERLargest project's share of starsAcross 127 products with their own repository
0.095DERConcentration (HHI)About 10 equal-sized projects’ worth; unconcentrated
4SRCMedian starsPer product with its own repository
75%DEROpen sourceMarked open source; unknown counts as not

Thin. Among the smallest 30% of nodes at its level, and launches are not outpacing the catalogue. Size is ranked against the other subcategory nodes; momentum compares this node’s launches in the last 90 days with the 90 before, relative to the whole catalogue’s ratio (score 62 or more = faster, 38 or less = slower; below 5 dated launches it is not measured and the label uses size alone).

Concentration. The Herfindahl–Hirschman index (HHI) adds up each project’s squared share of stars. At 0.095, attention is spread as if across about 10 equally popular projects. Using the common competition-policy thresholds (0.15 and 0.25), that is unconcentrated. Stars measure attention, not usage or quality.

Which product forms serve it

The same need is met by MCP servers, agent skills, plugins, extensions and frameworks. This counts each form across the whole catalogue.

Products in Data Pipelines by product type, as of 2026-11-21
Product type IndexedShareWith a profile
MCP servers13377.3%57
Agent skills137.6%13
Plugins127.0%12
Infrastructure74.1%7
Agent projects52.9%5
IDE extensions10.58%1
Automation platforms10.58%1

3 capabilities

Indexed = whole catalogue; profiled = products with their own page. A product can sit in several, so the rows do not add up to the total.

CapabilityIndexed ProfiledNew, 90 daysLabel
Data TransformationCleaning and reshaping datasets.673225Lagging the catalogue
ETL OrchestrationScheduling batch data jobs.502514Lagging the catalogue
Streaming DataEvent streams and message queues.503925Established

Top 40 products by GitHub stars

Search the directory

Profiled products only (96 of 172), ranked by the stars of their own repository. Products that live in a subfolder of a larger repository (common for skills) show that whole repository's stars and are ranked after them; products without a repository come last. Stars are popularity, not quality or installs.

Data Pipelines: top profiled products by GitHub stars, 2026-11-21
#ProductTypePublisherCapabilities GitHub stars Last push LicenceSave and compare
1kestraOrchestrate everything, everywhere. The open-source control plane for workflows across data, infrastructure, business processes and AI agents. Declarative YAML, event-driven, any language, Git-versioned, scalable to millions of executions.Agent projectkestra-io30k1d agoApache-2.0
2WindmillDeveloper platform that turns scripts in many languages into workflows, UIs and scheduled jobs, with AI steps and agents.Automation platformwindmill-labs19ktodayAGPL-3.0
3mcpReal-time Kubernetes network traffic visibility and API analysis for HTTP, gRPC, Redis, Kafka, DNS.MCP serverkubeshark12k11d agoApache-2.0
4cocoindexIncremental engine for long horizon agents 🌟 Star if you like it!Agent projectcocoindex-io12ktodayApache-2.0
5evidentlyEvidently is ​​an open-source ML and LLM observability framework. Evaluate, test, and monitor any AI-powered system or data pipeline. From tabular data to Gen AI. 100+ metrics.Infrastructureevidentlyai8.1k1mo agoApache-2.0
6unstractLLM-Driven Extraction of Unstructured Data — Built for API Deployments & ETL Pipeline WorkflowsMCP serverZipstack7.5k1d agoAGPL-3.0
7pinotApache Pinot: a real-time analytics database for low-latency SQL and vector search, serving user-facing and agent-facing applicationsAgent projectapache6.2ktodayApache-2.0
8JitsuManage Jitsu data pipelines: destinations, streams, connections, functions, live events.MCP serverjitsu.com5.1k1d agoMIT
9whodbWhere data access meets operational intelligenceMCP serverclidey5.1ktodayApache-2.0
10memgraphHigh-performance open-source in-memory graph database for GraphRAG, AI memory, agentic AI, and real-time graph analytics. Cypher-compatible, built in C++.Agent projectmemgraph4.7ktoday—
11daguSelf-hostable workflow orchestrator for teams whose main work isn't orchestration. Declarative YAML over your scripts, SSH commands, containers, etc; keep workflows separate from business logic. One binary, no database, runs on limited H/W resources. Alternative to Airflow / Cron / Job Scheduler.MCP serverdagucloud4.6ktodayGPL-3.0
12pythonOpen-source Python challenge courses — your AI coding agent teaches, verify.py decides when you're done.Agent projectflypythoncom4.2k11d ago—
13aws-data-analyticsData lake, analytics, and ETL workflows with S3 Tables, AWS Glue, and Athena.Pluginaws3.5ktodayApache-2.0
14hamiltonApache Hamilton helps data scientists and engineers define testable, modular, self-documenting dataflows, that encode lineage/tracing and metadata. Runs and scales everywhere python does.Infrastructureapache2.6k5d agoApache-2.0
15OpenMeterReal-time usage metering and billing for AI and API products, aggregating events into meters, entitlements and invoices.InfrastructureOpenMeter2.5ktodayApache-2.0
16GoNaviHigh-performance multi-data-source database client — ~30MB, AI & MCP ready, zero Electron bloat. | 高性能多数据源数据库客户端:约 30MB,AI 与 MCP 就绪,告别 Electron 膨胀。MCP serverSyngnat2.1k1d agoApache-2.0
17Seal Open DataFrench public open data (data.gouv.fr) cross-referenced by commune, EPCI and department.MCP serversealopendata.fr1.6k1d agoMIT
18OpenContractsThe open document intelligence platform for builders and hackers - DMS for the agentic worldInfrastructureOpen-Source-Legal1.6ktodayMIT
19NeumAINeum AI is a best-in-class framework to manage the creation and synchronization of vector embeddings at large scale.InfrastructureNeumTry8682y agoApache-2.0
20dbt-mcpA MCP (Model Context Protocol) server for interacting with dbt.MCP serverdbt-labs6243d agoApache-2.0
21components-contribCommunity driven, reusable components for distributed, durable, and agentic AI applications across all cloud service providersInfrastructuredapr6051d agoApache-2.0
22Power User for dbtMakes VS Code work seamlessly with dbt core and dbt cloud: auto-complete, preview, column lineage, AI docs generation, health checks, cost estimation etcIDE extensionAltimate Inc.586todayMIT
23FinSight-AIAI equity research agent with resilient workflows, evidence-grounded RAG, versioned reports, and automated quality evaluation.Infrastructurejuanjuandog5622mo agoMIT
24csv-data-summarizer-claude-skillA Claude Skill that automatically analyzes uploaded CSV files — generating summary statistics, detecting missing data, and creating quick visualizations using Python and pandas.Agent skillcoffeefuelbump4871y ago—
25huashu-excel数据分析与 Excel 全流程 skill:体检脏表、清洗、对齐需求、分析、对账、交付。让 AI 算出来的数字经得起追问。跨 agent 通用,依赖仅 openpyxl。Agent skillalchaincyf4776d agoMIT
26astronomer-data-agentsData engineering for Apache Airflow and Astronomer. Author DAGs with best practices, debug pipeline failures, trace data lineage, profile tables, migrate Airflow 2 to 3, and manage local and cloud deployments.Pluginastronomer4621d agoApache-2.0
27dataData engineering for Apache Airflow and Astronomer. Author DAGs with best practices, debug pipeline failures, trace data lineage, profile tables, migrate Airflow 2 to 3, and manage local and cloud deployments.Pluginastronomer4621d agoApache-2.0
28data-engineeringData engineering plugin - warehouse exploration, pipeline authoring, Airflow integrationPluginastronomer4621d agoApache-2.0
29data-agent-kitThis plugin provides a specialized suite of skills for data engineers and database practitioners working on Google Cloud. It acts as an expert assistant, allowing you to use natural language prompts in your preferred coding agent to architect complex data pipelines, transform data with dbt, write Spark and BigQuery SQL notebooks, and orchestrate end-to-end workflows across GCP's data ecosystem.PluginGoogleCloudPlatform2521d agoApache-2.0
30dexDex lets your agent build your analytics.Pluginexmergo141todayApache-2.0
31GoldenAnalysisRead-only cross-cutting analysis, metrics, and reporting across the Golden Suite.MCP serverbenseverndev-oss139todayMIT
32GoldenMatchFind duplicate records in 30 seconds. Zero-config entity resolution, 97.2% F1 out of the box.MCP serverbenseverndev-oss139todayMIT
33mcp-cn-commerce8 read-only MCP platform entries; support is operation-specific; merchant live acceptance pendingMCP serverTonyWang-hub1025d agoMIT
34strimzi-kafka-cliManage Apache Kafka on Kubernetes via Strimzi. Clusters, topics, users, connectors, MM2.MCP serverSystemCraftsman913mo agoApache-2.0
35mcpConnect your AI assistants to Keboola and expose your data, transformations, SQL queries, ...MCP serverkeboola.com84todayMIT
36Misata: synthetic data libraryDeclarative synthetic data: multi-table data that holds the facts you declare, checked by DuckDB.MCP serverrasinmuhammed7920d agoMIT
37dbtrailMySQL change tracking with instant row-level recovery and forensic attribution for compliance.MCP serverdbtrail.com73todayApache-2.0
38ClariLayerTurns selected past conversations into durable context your AI can use across projects and sessions.MCP serverclarilayer68today—
39streaming-skills-pluginSkills for streaming application developers, covering Kafka and Flink client libraries and Schema RegistryPluginconfluentinc669d agoApache-2.0
40oracle-ai-data-platform-workbench-databricks-migratorDrive the Oracle AI Data Platform (AIDP) Databricks Migration Toolkit in natural language. Plans and executes automated Databricks → AIDP migrations of notebooks, jobs, schedules, and catalog DDL — Pass-1 dependency rewrite + Pass-2 cell-by-cell execute/verify/fix on a live AIDP cluster with Claude.Pluginoracle-samples621d agoUPL-1.0

Most common integrations

Third-party services named by products in this node, whole catalogue.

IntegrationProducts Share
Apache Kafka2715.7%
PostgreSQL137.6%
MySQL63.5%
Redis63.5%
Elasticsearch52.9%
AWS31.7%
BigQuery31.7%
MongoDB31.7%
Snowflake31.7%
Databricks21.2%
Kubernetes21.2%
Azure10.6%
GitHub10.6%
Google Cloud10.6%
Google Sheets10.6%

Launches per month

New Data Pipelines products per monthSRC

Whole catalogue, by publication month, 2025-03 to 2026-11. The last month is partial; undated products are excluded.

New Data Pipelines products per monthWhole catalogue, by publication month, 2025-03 to 2026-11. The last month is partial; undated products are excluded.. Products published: 14 points, latest 27 products in 2026-11.
Show the data as a table
New Data Pipelines products per month. Whole catalogue, by publication month, 2025-03 to 2026-11. The last month is partial; undated products are excluded.
Products published (products)
2025-032
2025-091
2025-104
2026-014
2026-028
2026-035
2026-046
2026-0510
2026-0610
2026-0710
2026-0817
2026-0914
2026-1018
2026-1127

Related opportunities

Opportunity radar

Scored gaps for this node or anything beneath it. Scores rank hypotheses; they do not predict demand.

No scored opportunity here

Capability gaps are only scored inside subcategories with at least 20 indexed products, and only the top 80 are kept. None falls under this node.

Limits of these numbers

  • Products are placed here by whole-word keyword rules on their name, description and repository topics, and inherit every ancestor of a matched node. Products that do this without naming it are missed; ambiguous words cause false matches. Methodology
  • About half of the catalogue matches no node at all, so absolute counts understate every category.
  • Launch counts use the publication date each source reports; for auto-discovered repositories that is the repository’s creation date.
  • Stars and concentration cover only products with their own repository (not a subfolder of a larger one). They measure attention, not adoption.

Page data last updated .