Screen Understanding
Parsing screenshots into actionable UI elements. As of 2026-11-21, 13 indexed products offer it (12 with full profiles), in 2 forms: 12 MCP servers, 1 Framework. 9 were published in the last 90 days, against 1 in the 90 days before. OpenGUI has the most GitHub stars, 57% of the total across 13 products with their own repository.
CapabilitySegment label: Emerging DERScored as an opportunity
Available as
All in the directoryProfiled products grouped by product form. Within each form: own-repository products by GitHub stars, then products in a subfolder of a larger repository (they show that repository's stars), up to 10 each.
MCP servers 12 indexed · 11 profiled
Protocol-level reach: 36 of the 45 active platforms AgentSidekick tracks document support for MCP. DER Which platforms
| # | Product | Publisher | Capabilities | GitHub stars | Last push | Save and compare |
|---|---|---|---|---|---|---|
| 1 | axuseAccessibility-first browser & computer use for AI agents — one MCP server for every harness (Codex, Claude Code, OpenCode, Antigravity). Instruments the DOM/accessibility tree via CDP instead of screenshot loops: diffed snapshots, stable element refs, trusted input, typed errors. Zero dependencies | ctoldigital | 804 | 18d ago | ||
| 2 | clawdcursorCross-OS desktop control for any AI agent: accessibility tree first, screenshots last. | AmrDab | 407 | 1d ago | ||
| 3 | iphone-cli (iph)Control any app on a real iPhone (iOS 27+) via accessibility tree. No Xcode or WebDriverAgent. | HyunSeob | 116 | 1d ago | ||
| 4 | bad-ass-mcpDesktop GUI automation on macOS, Windows and Linux through accessibility trees, not screenshots. | HoldMyBeer-gg | 8 | 1d ago | ||
| 5 | macos-vision-mcpLocal OCR & screen understanding for agents: read any window, find what to click. No uploads. | woladi | 8 | 2mo ago | ||
| 6 | ai-browserVisible Electron browser driven by a semantic accessibility tree: 17 MCP tools, no screenshots. | Glittering | 5 | 4d ago | ||
| 7 | win-mindSee and drive any Windows app via the accessibility tree - no screenshots - plus 200+ native tools | jhonpork1233-beep | 3 | 1mo ago | ||
| 8 | pyjab-mcpDrive Java desktop apps on Windows from an AI agent, through the JVM's own accessibility tree | gaozhao1989 | 1 | 5d ago | ||
| 9 | BrowsegrabToken-efficient browser agent for local LLMs. Playwright + accessibility tree + MarkGrab. | ArkNill | 0 | 7mo ago | ||
| 10 | oriel for WindowsComputer use for Windows: reads each app as text through the accessibility tree, not screenshots. | YordiLorenzo | 0 | 4d ago |
Frameworks 1 indexed · 1 profiled
No compatibility evidence is collected for this form.
| # | Product | Publisher | Capabilities | GitHub stars | Last push | Save and compare |
|---|---|---|---|---|---|---|
| 1 | OpenGUIOpenGUI is an Android GUI agent framework for phone-use AI that can see, plan, and operate real mobile apps through the GUI. | Core-Mate | 1.8k | 5d ago |
Which platforms these forms can reach
All platformsProtocol-level: the platform documents support for the standard. No specific pairing has been tested.
MCP servers 12 indexed here
36 platforms document MCP support CUR.
ContinueOpenCodeClaude CodeCodexGemini CLIZedOpenHandsClineWarpgooseQwen CodeBolt (bolt.new)KiroAmazon Bedrock AgentCoreAmpApps in ChatGPT / Apps SDK (now ChatGPT Plugins)Augment CodeChatGPTClaude (claude.ai apps)Claude DesktopCursorDevinDevin Desktop (formerly Windsurf)Factory (Droid)Gemini (app)Gemini Enterprise Agent Platform (formerly Vertex AI / Vertex AI Agent Builder)GitHub CopilotGitHub Copilot cloud agent (formerly Copilot coding agent)Google AntigravityJetBrains AI AssistantJunieLovableMicrosoft Copilot StudioReplit AgentTabninev0
Agent skills 0 indexed here
26 platforms document Agent Skills support CUR, but no product offering this capability ships in this form yet.
OpenClawOpenCodeClaude CodeCodexGemini CLIOpenHandsWarpgooseBolt (bolt.new)KiroAmpApps in ChatGPT / Apps SDK (now ChatGPT Plugins)ChatGPTClaude (claude.ai apps)Claude DesktopCursorDevinDevin Desktop (formerly Windsurf)Factory (Droid)Gemini (app)GitHub CopilotGitHub Copilot cloud agent (formerly Copilot coding agent)Google AntigravityJunieReplit AgentTabnine
Platforms publishers name
| Platform | Profiled products naming it |
|---|---|
| Claude Code | 1 |
| Google Antigravity | 1 |
| Codex | 1 |
| OpenCode | 1 |
Integrations most often paired
Third-party services named by products with this capability, whole catalogue.
| Integration | Products | Share |
|---|---|---|
| Android | 1 | 7.7% |
| Playwright | 1 | 7.7% |
| Xcode / iOS | 1 | 7.7% |
Keywords that trigger this capability
Matched as whole words, ignoring case, in a product's name, description and repository topics.
| Keyword | Products matched |
|---|---|
| screen understanding | 2 |
| screenshot analysis | 0 |
| ui grounding | 0 |
| screen parsing | 0 |
| omniparser | 0 |
| accessibility tree | 11 |
Segment statistics
Whole catalogue (13 indexed products). Calculated 2026-11-21.
Emerging. Among the smallest 30% of capabilities, but launches are outpacing the catalogue. Size is ranked against all capabilities; momentum compares launches in the last 90 days with the 90 before, relative to the catalogue’s own ratio (62 or more = faster, 38 or less = slower; under 5 dated launches it is not measured). The HHI of 0.411 means stars are spread as if across about 2 equally popular projects — highly concentrated by the common 0.15 / 0.25 thresholds. Stars measure attention, not usage or quality.
Scored as an opportunity
Score breakdownCoverage gap: Screen Understanding: score 57.7 HEU, competition Medium, confidence Low. Only 13 indexed products match "Screen Understanding", against 296 in Computer Use overall.
Limits
- Classification is by the keywords above. A product that offers this capability without using one of them is not counted; one that uses a keyword in another sense is counted wrongly. Methodology
- Platform reach is protocol-level: it says a platform documents the standard, not that any product here was tested on it. Platforms named by publishers are their own claims.
- Stars belong to repositories. Products in a subfolder of a large repository show that whole repository’s stars and are excluded from the concentration figures.
- Launch counts use each source’s publication date; for auto-discovered repositories that is the repository’s creation date.
Page data last updated .