@amaster.ai/pi-computer-use

pi-coding-agent extension that wraps cua-driver-rs, exposing desktop automation tools with a computer_use_ prefix.

Features

• Zero external dependencies — pre-compiled cua-driver-rs binaries bundled for all platforms
• MCP stdio communication — spawns cua-driver mcp via StdioClientTransport, JSON-RPC over stdio
• Dynamic tool discovery — auto-discovers upstream MCP tools and registers with computer_use_ prefix; falls back to a built-in tool list when cua-driver fails to start
• Smart tool filtering — excludes non-essential tools (agent cursor, recording, config, raw screenshot), exposes 17 action tools + 1 vision tool
• Optional visual analysis — computer_use_analyze_screenshot via configurable vision model
• Cross-platform permission handling — detects platform-specific permission issues (macOS TCC, Windows UAC, Linux display server access) and returns actionable guidance
• Graceful degradation — tools are always registered even when cua-driver cannot connect; lazy reconnect is attempted on each tool call

Install

bun add @amaster.ai/pi-computer-use

Requires Node.js >= 20 and @earendil-works/pi-coding-agent >= 0.74.0.

Usage

Install the package and pi-coding-agent will automatically discover and load the extension. All tools are registered on session_start.

Configure via .pi/settings.json (project-level) or ~/.pi/agent/settings.json (user-level) under the "pi-computer-use" key:

{
  "pi-computer-use": {
    "mode": "bundled"
  }
}

Configuration

Option	Type	Default	Description
mode	'bundled' | 'path'	'bundled'	Binary resolution strategy
binaryPath	string	—	Custom cua-driver binary path (requires mode: 'path')
extraArgs	string[]	—	Extra CLI arguments passed to cua-driver
visionModel	VisionModelConfig	—	Enable visual screenshot analysis

Vision Model (Optional)

Enable computer_use_analyze_screenshot by referencing a model already configured in Pi's model registry (models.json):

{
  "pi-computer-use": {
    "visionModel": {
      "provider": "openai",
      "model": "gpt-4o"
    }
  }
}

The extension resolves API key, base URL, and headers from the model registry automatically — no need to duplicate credentials here.

Exposed Tools (17 + 1 vision)

Input

Tool	Description
computer_use_click	Left-click via element_index or x/y coordinates
computer_use_double_click	Double-click at x/y or on an AX element
computer_use_right_click	Right-click (context menu)
computer_use_type_text	Insert text via AX or CGEvent fallback
computer_use_press_key	Press and release a single key
computer_use_hotkey	Press a key combination (e.g. Cmd+C)
computer_use_scroll	Scroll by line or page in a direction
computer_use_drag	Press-drag-release gesture between two points
computer_use_set_value	Set value on UI elements (popups, sliders, steppers)

Query

Tool	Description
computer_use_get_screen_size	Get display dimensions and scale factor
computer_use_get_cursor_position	Get current mouse cursor position
computer_use_get_accessibility_tree	Lightweight desktop snapshot (apps, windows, bounds)
computer_use_get_window_state	Full AX tree of a window with actionable element indices
computer_use_list_windows	List all top-level windows with bounds and z-order
computer_use_list_apps	List running and installed apps with state flags

App Lifecycle

Tool	Description
computer_use_launch_app	Launch an app in the background without focus steal
computer_use_kill_app	Force-terminate a process by pid

Vision (requires visionModel config)

Tool	Description
computer_use_analyze_screenshot	Take a screenshot and analyze it with a vision model

Excluded Tools (16)

Agent cursor styling, recording/replay, config management, zoom, raw screenshot (use analyze_screenshot instead), and browser-specific operations are filtered out.

Permissions

On session_start, the extension checks permissions via cua-driver's check_permissions tool. Platform-specific guidance is provided:

Platform	Accessibility	Screen Capture
macOS	System Settings → Privacy & Security → Accessibility	System Settings → Privacy & Security → Screen & System Audio Recording
Windows	Run as Administrator / UI Automation access	Check DRM or security policy
Linux	AT-SPI accessibility service	PipeWire portal or X11 access

When cua-driver fails to connect (missing permissions, binary not found, etc.):

1. User is notified with a platform-appropriate warning
2. Tools are still registered using a built-in fallback schema
3. On each tool call, lazy reconnect is attempted; if it still fails, a friendly error with permission instructions is returned

Supported Platforms

Platform	Binary
macOS ARM64	bin/darwin-arm64/cua-driver
macOS x64	bin/darwin-x64/cua-driver
Linux x64	bin/linux-x64/cua-driver
Windows x64	bin/win32-x64/cua-driver.exe
Windows ARM64	bin/win32-arm64/cua-driver.exe

License

Apache-2.0