Model Serving
Inference servers for production model hosting. As of 2026-11-21, 54 indexed products offer it (31 with full profiles), in 7 forms: 27 MCP servers, 11 Agent skills, 11 Infrastructure, 2 Plugins. 17 were published in the last 90 days, against 8 in the 90 days before. docker-android has the most GitHub stars, 24% of the total across 43 products with their own repository.
CapabilitySegment label: Lagging the catalogue DERScored as an opportunity
Available as
All in the directoryProfiled products grouped by product form. Within each form: own-repository products by GitHub stars, then products in a subfolder of a larger repository (they show that repository's stars), up to 10 each.
MCP servers 27 indexed · 4 profiled
Protocol-level reach: 36 of the 45 active platforms AgentSidekick tracks document support for MCP. DER Which platforms
| # | Product | Publisher | Capabilities | GitHub stars | Last push | Save and compare |
|---|---|---|---|---|---|---|
| 1 | docker-androidAndroid in docker solution with noVNC supported, video recording, mcp server and AI-agent | budtmo | 16k | 2d ago | ||
| 2 | ollayaJev-compatible API server for open decision models. Ollama for decision models: pull and serve 29 model families (Laya, pplx-decider, Clef, Winnow, coba...) locally or self-hosted behind /v1/systemone. The TypeSafe SDK works unchanged: set its base URL and pick an open model. CLI, desktop app, Docker, MCP server, public bench. | ollaya-dev | 1.5k | 2d ago | ||
| 3 | lmConnect Claude to any OpenAI-compatible LLM endpoint and offload routine work to a local model. | houtini-ai | 122 | 14d ago | ||
| 4 | model-composeDeclarative AI pipelines in one YAML file. Tokens, audio chunks, and video frames flow between isolated components. Compose 100+ components for models, agents, speech, vision, and live broadcast. Run local models, cloud APIs, or both. Inspired by docker-compose. | hanyeol | 118 | today |
Agent skills 11 indexed · 11 profiled
Protocol-level reach: 26 of the 45 active platforms AgentSidekick tracks document support for the Agent Skills format. DER Which platforms
| # | Product | Publisher | Capabilities | GitHub stars | Last push | Save and compare |
|---|---|---|---|---|---|---|
| 1 | AI-Research-SKILLsComprehensive open-source library of AI research and engineering skills for any AI model. Package the skills and your claude code/codex/gemini agent will be an AI research agent with full horsepower. Maintained by Orchestra Research. | Orchestra-Research | 15k | 1mo ago | ||
| 2 | llamaguardMeta's 7-8B specialized moderation model for LLM input/output filtering. 6 safety categories - violence/hate, sexual content, weapons, substances, self-harm, criminal planning. 94-95% accuracy. Deploy with vLLM, HuggingFace, Sagemaker. Integrates with NeMo Guardrails. | Orchestra-Research | — | 1mo ago | ||
| 3 | slime-rl-trainingProvides guidance for LLM post-training with RL using slime, a Megatron+SGLang framework. Use when training GLM models, implementing custom data generation workflows, or needing tight Megatron-LM integration for RL scaling. | Orchestra-Research | — | 1mo ago | ||
| 4 | openrlhf-trainingHigh-performance RLHF framework with Ray+vLLM acceleration. Use for PPO, GRPO, RLOO, DPO training of large models (7B-70B+). Built on Ray, vLLM, ZeRO-3. 2× faster than DeepSpeedChat with distributed architecture and GPU resource sharing. | Orchestra-Research | — | 1mo ago | ||
| 5 | hqq-quantizationHalf-Quadratic Quantization for LLMs without calibration data. Use when quantizing models to 4/3/2-bit precision without needing calibration datasets, for fast quantization workflows, or when deploying with vLLM or HuggingFace Transformers. | Orchestra-Research | — | 1mo ago | ||
| 6 | evaluating-llms-harnessEvaluates LLMs across 60+ academic benchmarks (MMLU, HumanEval, GSM8K, TruthfulQA, HellaSwag). Use when benchmarking model quality, comparing models, reporting academic results, or tracking training progress. Industry standard used by EleutherAI, HuggingFace, and major labs. Supports HuggingFace, vLLM, APIs. | Orchestra-Research | — | 1mo ago | ||
| 7 | sglangFast structured generation and serving for LLMs with RadixAttention prefix caching. Use for JSON/regex outputs, constrained decoding, agentic workflows with tool calls, or when you need 5× faster inference than vLLM with prefix sharing. Powers 300,000+ GPUs at xAI, AMD, NVIDIA, and LinkedIn. | Orchestra-Research | — | 1mo ago | ||
| 8 | serving-llms-vllmServes LLMs with high throughput using vLLM's PagedAttention and continuous batching. Use when deploying production LLM APIs, optimizing inference latency/throughput, or serving models with limited GPU memory. Supports OpenAI-compatible endpoints, quantization (GPTQ/AWQ/FP8), and tensor parallelism. | Orchestra-Research | — | 1mo ago | ||
| 9 | hugging-face-evaluationAdd and manage evaluation results in Hugging Face model cards. Supports extracting eval tables from README content, importing scores from Artificial Analysis API, and running custom model evaluations with vLLM/lighteval. Works with the model-index metadata format. | sickn33 | — | 1d ago | ||
| 10 | evaluation-lm-evaluation-harnessEvaluates LLMs across 60+ academic benchmarks (MMLU, HumanEval, GSM8K, TruthfulQA, HellaSwag). Use when benchmarking model quality, comparing models, reporting academic results, or tracking training progress. Industry standard used by EleutherAI, HuggingFace, and major labs. Supports HuggingFace, vLLM, APIs. | davila7 | — | today |
Infrastructure 11 indexed · 11 profiled
No compatibility evidence is collected for this form.
| # | Product | Publisher | Capabilities | GitHub stars | Last push | Save and compare |
|---|---|---|---|---|---|---|
| 1 | BentoMLThe easiest way to serve AI apps and models - Build Model Inference APIs, Job queues, LLM apps, Multi-model pipelines, and more! | bentoml | 8.9k | 4d ago | ||
| 2 | cube-studiocube studio开源云原生一站式机器学习/深度学习/大模型AI平台,mlops算法链路全流程,算力租赁平台,notebook在线开发,拖拉拽任务流pipeline编排,多机多卡分布式训练,超参搜索,推理服务VGPU虚拟化,边缘计算,标注平台自动化标注,deepseek等大模型sft微调/奖励模型/强化学习训练,vllm/ollama/mindie大模型多机推理,私有知识库,AI模型市场,支持国产cpu/gpu/npu 昇腾生态,支持RDMA,支持pytorch/tf/mxnet/deepspeed/paddle/colossalai/horovod/ray/volcano等分布式 | tencentmusic | 5.1k | 1mo ago | ||
| 3 | vllm-ascendCommunity maintained hardware plugin for vLLM on Huawei Ascend | vllm-project | 3.1k | today | ||
| 4 | cube-studiocubestudio开源云原生一站式机器学习/深度学习/大模型AI平台/MaaS/mlops/人工智能平台/训推平台,算法全链路流程,多租户,算力租赁平台,token中转,拖拉拽任务流pipeline编排,多机多卡分布式训练,超参搜索,推理服务,VGPU虚拟化,云边端协同,边缘计算,自动化标注平台,deepseek等大模型sft微调/奖励模型/强化学习训练,vllm/ollama/mindie大模型多机推理,私有知识库llmops智能体,AI模型市场,支持国产异构算力调度,昇腾/寒武纪/海光/摩尔/沐曦等,支持ib/roce/RDMA,信创支持 | data-infra | 2.6k | 1mo ago | ||
| 5 | envd🏕️ Reproducible development environment for humans and agents | tensorchord | 2.2k | 1d ago | ||
| 6 | aiciAICI: Prompts as (Wasm) Programs | microsoft | 2.1k | 1y ago | ||
| 7 | rtp-llmRTP-LLM: Alibaba's high-performance LLM inference engine for diverse applications. | alibaba | 1.4k | today | ||
| 8 | prometheus-evalEvaluate your LLM's response with Prometheus and GPT4 💯 | prometheus-eval | 1.1k | 1y ago | ||
| 9 | juliusSimple LLM service identification - translate IP:Port to Ollama, vLLM, LiteLLM, or 60+ other AI services in seconds | praetorian-inc | 256 | today | ||
| 10 | heronAgent and LLM API performance monitoring via network packet probe. Measures performance of OpenClaw, Claude, Codex, DeepAgents and more — deployed on the provider side, no SDK changes required. | Netis | 104 | 1mo ago |
Plugins 2 indexed · 2 profiled
2 of these are Claude plugins, which Claude Code documents (protocol-level). DER
| # | Product | Publisher | Capabilities | GitHub stars | Last push | Save and compare |
|---|---|---|---|---|---|---|
| 1 | amd-skillsAMD's verified Agent Skills in one plugin: route image/audio through local AI on Ryzen AI, serve LLMs on AMD Instinct GPUs with vLLM, optimize inference throughput with Hyperloom, and analyze GPU kernel and PyTorch trace performance. | amd | 509 | today | ||
| 2 | databricksDatabricks skills for the CLI, Apps, Lakebase, Model Serving, Lakeflow Jobs, Spark Declarative Pipelines, Declarative Automation Bundles (DABs), and classic-to-serverless migration. | databricks | 407 | today |
Agent projects 1 indexed · 1 profiled
No compatibility evidence is collected for this form.
| # | Product | Publisher | Capabilities | GitHub stars | Last push | Save and compare |
|---|---|---|---|---|---|---|
| 1 | OlaresOpen-Source Personal Cloud OS for Always-On Agents | beclab | 5.4k | today |
Frameworks 1 indexed · 1 profiled
No compatibility evidence is collected for this form.
| # | Product | Publisher | Capabilities | GitHub stars | Last push | Save and compare |
|---|---|---|---|---|---|---|
| 1 | taOSSelf-hosted AI agent OS. Your memory, chat, agents, and files stay on hardware you own, offline by default, cloud by choice. Offline AI memory (taOSmd), self-hosted multi-framework group chat, a full web desktop + app store, and auto-clustering across the consumer hardware you already have (Orange/Raspberry Pi, Mac mini, gaming PC). | jaylfc | 609 | 1d ago |
IDE extensions 1 indexed · 1 profiled
No compatibility evidence is collected for this form.
| # | Product | Publisher | Capabilities | GitHub stars | Last push | Save and compare |
|---|---|---|---|---|---|---|
| 1 | Vibe DGC — coding agent for your own modelsWork, ChatDGC and Pips in VS Code and Cursor, on your own local or cloud models or a signed-in Claude Code or Codex. | vibedgc | 1 | 10d ago |
Which platforms these forms can reach
All platformsProtocol-level: the platform documents support for the standard. No specific pairing has been tested.
MCP servers 27 indexed here
36 platforms document MCP support CUR.
ContinueOpenCodeClaude CodeCodexGemini CLIZedOpenHandsClineWarpgooseQwen CodeBolt (bolt.new)KiroAmazon Bedrock AgentCoreAmpApps in ChatGPT / Apps SDK (now ChatGPT Plugins)Augment CodeChatGPTClaude (claude.ai apps)Claude DesktopCursorDevinDevin Desktop (formerly Windsurf)Factory (Droid)Gemini (app)Gemini Enterprise Agent Platform (formerly Vertex AI / Vertex AI Agent Builder)GitHub CopilotGitHub Copilot cloud agent (formerly Copilot coding agent)Google AntigravityJetBrains AI AssistantJunieLovableMicrosoft Copilot StudioReplit AgentTabninev0
Agent skills 11 indexed here
26 platforms document Agent Skills support CUR.
OpenClawOpenCodeClaude CodeCodexGemini CLIOpenHandsWarpgooseBolt (bolt.new)KiroAmpApps in ChatGPT / Apps SDK (now ChatGPT Plugins)ChatGPTClaude (claude.ai apps)Claude DesktopCursorDevinDevin Desktop (formerly Windsurf)Factory (Droid)Gemini (app)GitHub CopilotGitHub Copilot cloud agent (formerly Copilot coding agent)Google AntigravityJunieReplit AgentTabnine
Platforms publishers name
| Platform | Profiled products naming it |
|---|---|
| Claude Code | 2 |
| Codex | 2 |
| Cursor | 1 |
| OpenClaw | 1 |
Integrations most often paired
Third-party services named by products with this capability, whole catalogue.
| Integration | Products | Share |
|---|---|---|
| 2 | 3.7% | |
| Android | 1 | 1.9% |
| Databricks | 1 | 1.9% |
| Grafana | 1 | 1.9% |
Keywords that trigger this capability
Matched as whole words, ignoring case, in a product's name, description and repository topics.
| Keyword | Products matched |
|---|---|
| model serving | 7 |
| vllm | 41 |
| inference server | 1 |
| text generation inference | 0 |
| triton inference | 0 |
| sglang | 5 |
Segment statistics
Whole catalogue (54 indexed products). Calculated 2026-11-21.
Lagging the catalogue. Mid-sized among capabilities, with launches growing more slowly than the catalogue’s. Size is ranked against all capabilities; momentum compares launches in the last 90 days with the 90 before, relative to the catalogue’s own ratio (62 or more = faster, 38 or less = slower; under 5 dated launches it is not measured). The HHI of 0.146 means stars are spread as if across about 7 equally popular projects — unconcentrated by the common 0.15 / 0.25 thresholds. Stars measure attention, not usage or quality.
Scored as an opportunity
Score breakdownCoverage gap: Model Serving: score 52.1 HEU, competition High, confidence Low. Only 54 indexed products match "Model Serving", against 388 in Local & Open Models overall.
Limits
- Classification is by the keywords above. A product that offers this capability without using one of them is not counted; one that uses a keyword in another sense is counted wrongly. Methodology
- Platform reach is protocol-level: it says a platform documents the standard, not that any product here was tested on it. Platforms named by publishers are their own claims.
- Stars belong to repositories. Products in a subfolder of a large repository show that whole repository’s stars and are excluded from the concentration figures.
- Launch counts use each source’s publication date; for auto-discovered repositories that is the repository’s creation date.
Page data last updated .