AgentSidekick

Screen Understanding

Parsing screenshots into actionable UI elements. As of 2026-11-21, 13 indexed products offer it (12 with full profiles), in 2 forms: 12 MCP servers, 1 Framework. 9 were published in the last 90 days, against 1 in the 90 days before. OpenGUI has the most GitHub stars, 57% of the total across 13 products with their own repository.

CapabilitySegment label: Emerging DERScored as an opportunity

Available as

All in the directory

Profiled products grouped by product form. Within each form: own-repository products by GitHub stars, then products in a subfolder of a larger repository (they show that repository's stars), up to 10 each.

MCP servers 12 indexed · 11 profiled

Protocol-level reach: 36 of the 45 active platforms AgentSidekick tracks document support for MCP. DER Which platforms

Screen Understanding: MCP servers by GitHub stars
#ProductPublisherCapabilities GitHub stars Last push Save and compare
1axuseAccessibility-first browser & computer use for AI agents — one MCP server for every harness (Codex, Claude Code, OpenCode, Antigravity). Instruments the DOM/accessibility tree via CDP instead of screenshot loops: diffed snapshots, stable element refs, trusted input, typed errors. Zero dependenciesctoldigital80418d ago
2clawdcursorCross-OS desktop control for any AI agent: accessibility tree first, screenshots last.AmrDab4071d ago
3iphone-cli (iph)Control any app on a real iPhone (iOS 27+) via accessibility tree. No Xcode or WebDriverAgent.HyunSeob1161d ago
4bad-ass-mcpDesktop GUI automation on macOS, Windows and Linux through accessibility trees, not screenshots.HoldMyBeer-gg81d ago
5macos-vision-mcpLocal OCR & screen understanding for agents: read any window, find what to click. No uploads.woladi82mo ago
6ai-browserVisible Electron browser driven by a semantic accessibility tree: 17 MCP tools, no screenshots.Glittering54d ago
7win-mindSee and drive any Windows app via the accessibility tree - no screenshots - plus 200+ native toolsjhonpork1233-beep31mo ago
8pyjab-mcpDrive Java desktop apps on Windows from an AI agent, through the JVM's own accessibility treegaozhao198915d ago
9BrowsegrabToken-efficient browser agent for local LLMs. Playwright + accessibility tree + MarkGrab.ArkNill07mo ago
10oriel for WindowsComputer use for Windows: reads each app as text through the accessibility tree, not screenshots.YordiLorenzo04d ago

All 11 profiled MCP servers with this capability

Frameworks 1 indexed · 1 profiled

No compatibility evidence is collected for this form.

Screen Understanding: Frameworks by GitHub stars
#ProductPublisherCapabilities GitHub stars Last push Save and compare
1OpenGUIOpenGUI is an Android GUI agent framework for phone-use AI that can see, plan, and operate real mobile apps through the GUI.Core-Mate1.8k5d ago

Which platforms these forms can reach

All platforms

Protocol-level: the platform documents support for the standard. No specific pairing has been tested.

Platforms publishers name

PlatformProfiled products naming it
Claude Code1
Google Antigravity1
Codex1
OpenCode1

Integrations most often paired

Third-party services named by products with this capability, whole catalogue.

IntegrationProducts Share
Android17.7%
Playwright17.7%
Xcode / iOS17.7%

Keywords that trigger this capability

Matched as whole words, ignoring case, in a product's name, description and repository topics.

KeywordProducts matched
screen understanding2
screenshot analysis0
ui grounding0
screen parsing0
omniparser0
accessibility tree11

Segment statistics

Whole catalogue (13 indexed products). Calculated 2026-11-21.

13DERIndexed products12 with full profiles
9DERLaunched, last 90 days1 in the 90 days before
72DERMomentum vs catalogue50 = same pace as the catalogue
57%DERLargest project's share of starsAcross 13 products with their own repository
0.411DERConcentration (HHI)About 2 equal-sized projects’ worth; highly concentrated
5SRCMedian starsPer product with its own repository
85%DEROpen sourceMarked open source; unknown counts as not

Emerging. Among the smallest 30% of capabilities, but launches are outpacing the catalogue. Size is ranked against all capabilities; momentum compares launches in the last 90 days with the 90 before, relative to the catalogue’s own ratio (62 or more = faster, 38 or less = slower; under 5 dated launches it is not measured). The HHI of 0.411 means stars are spread as if across about 2 equally popular projects — highly concentrated by the common 0.15 / 0.25 thresholds. Stars measure attention, not usage or quality.

Scored as an opportunity

Score breakdown

Coverage gap: Screen Understanding: score 57.7 HEU, competition Medium, confidence Low. Only 13 indexed products match "Screen Understanding", against 296 in Computer Use overall.

Limits

  • Classification is by the keywords above. A product that offers this capability without using one of them is not counted; one that uses a keyword in another sense is counted wrongly. Methodology
  • Platform reach is protocol-level: it says a platform documents the standard, not that any product here was tested on it. Platforms named by publishers are their own claims.
  • Stars belong to repositories. Products in a subfolder of a large repository show that whole repository’s stars and are excluded from the concentration figures.
  • Launch counts use each source’s publication date; for auto-discovered repositories that is the repository’s creation date.

Page data last updated .