pi-agent-desktop-adapter

Pi extension that lets agents observe and operate macOS, Windows and Linux desktop apps through agent-desktop's accessibility-tree CLI.

Packages

Package details

extension

Install pi-agent-desktop-adapter from npm and Pi will load the resources declared by the package manifest.

npm report

$ pi install npm:pi-agent-desktop-adapter

Package: pi-agent-desktop-adapter
Version: 0.1.0
Published: Oct 31, 2026
Downloads: 357/mo · 357/wk
Author: mohebifar
License: MIT
Types: extension
Size: 69.1 KB
Dependencies: 1 dependency · 3 peers

Pi manifest JSON

{
  "extensions": [
    "./extensions"
  ]
}

Security note

Pi packages can execute code and influence agent behavior. Review the source before installing third-party packages.

README

pi-agent-desktop-adapter

A pi extension that lets the agent see and operate desktop apps on macOS, Windows and Linux through agent-desktop, Vercel Labs' accessibility-tree CLI.

The agent works in a loop: it takes a snapshot of an app, acts on an element's ref, then takes another snapshot to check the result. Actions are headless by default, so they go through the accessibility API without moving your pointer or stealing focus. Physical mouse and keyboard input happens only after you allow it.

Install

pi install npm:pi-agent-desktop-adapter

Or from GitHub:

pi install git:github.com/<you>/pi-agent-desktop-adapter

Or try it for one run from a checkout:

pi -e ./extensions/agent-desktop.ts

The agent-desktop npm package is a dependency and ships native binaries for every platform, so nothing else is needed. To use a different build, set AGENT_DESKTOP_BIN=/path/to/agent-desktop.

Permissions

macOS: grant Accessibility (and Screen Recording for screenshots) to the terminal app that runs pi. Run agent-desktop permissions --guide. The extension warns at session start if Accessibility is missing.
Windows: no grant is needed for apps at your terminal's integrity level.
Linux: needs a graphical session with at-spi2-core.

Tools

Tool	What it does
`desktop_snapshot`	Accessibility tree with qualified refs (`@s8f3k2p9:e5`). Supports `skeleton` plus `root` drill-down, surfaces (menu, sheet, alert…), bounds, and `wait_for`
`desktop_find`	Search by role, name, value, text, state or native id
`desktop_act`	Act on a ref: click, right-click, type, set-value, clear, focus, select, toggle, check, uncheck, expand, collapse, scroll, scroll-to (double-click and triple-click are headed)
`desktop_press`	Key combos (`cmd+s`, `escape`, …), optionally scoped to an app or window
`desktop_inspect`	Read one live property (`text`, `value`, `bounds`, …) or one boolean state (`enabled`, `checked`, …)
`desktop_screenshot`	Window or display screenshot returned as an image. On macOS it is downscaled with `sips`
`desktop_list`	Apps, windows, displays, surfaces or notifications
`desktop_app`	Launch, close, focus, resize, move, minimize, maximize or restore apps and windows
`desktop_wait`	Wait for an element predicate, window, text, menu, lifecycle event or notification
`desktop_mouse`	Physical hover, drag, move, click and wheel by ref or screen coordinates (always headed)
`desktop_clipboard`	Get, set or clear the clipboard
`desktop_batch`	Up to 64 commands run in order in one process, e.g. press then wait for `window-opened`
`desktop_cli`	Any other agent-desktop command, including `skills get desktop <reference>` for the bundled docs

The extension turns snapshots and find results into a compact text tree, one line per node:

snapshot id=s6feze89hjum4 app=Calculator window=w-26954 title="Calculator" refs=28 complete=true

window "Calculator"
  group
    button "7" @s6feze89hjum4:e10 {Click,SetFocus}
    button "Add" @s6feze89hjum4:e21 {Click,SetFocus}

Errors keep agent-desktop's code, suggestion and retry disposition (delivery=… retry=safe|unsafe), so the model knows when it is safe to repeat an action. Output larger than pi's 50 KB / 2000-line budget is truncated, and the full text is written to a private temp file.

Safety and configuration

Headed input. Physical input moves the real pointer, sends real keystrokes and can raise windows. This covers headed=true, desktop_mouse, double-click and triple-click. Under the default ask policy, pi asks once per session before any of it runs. With no UI to ask (print or JSON mode), headed input is refused.

/agent-desktop                    status: binary, version, permissions, policy, session
/agent-desktop headed allow|ask|deny

Risky actions. Force-sending a shortcut that agent-desktop flags as dangerous (desktop_press force=true) and force-quitting an app (desktop_app action=close force=true) need a confirmation when a UI is available.

Sessions. Each pi session gets its own agent-desktop session, an isolated snapshot namespace. It starts on first use and is sealed on shutdown.

Environment variable	Default	Effect
`PI_AGENT_DESKTOP_HEADED`	`ask`	`allow`, `ask` or `deny` physical input
`PI_AGENT_DESKTOP_SESSION`	`1`	`0` uses agent-desktop's global namespace instead of a per-session one
`PI_AGENT_DESKTOP_TRACE`	`0`	`1` records an agent-desktop trace (`agent-desktop trace show/export`)
`PI_AGENT_DESKTOP_CURSOR`	`0`	`1` draws agent-desktop's presentation-only cursor overlay
`AGENT_DESKTOP_BIN`	bundled	Path to an agent-desktop executable

Snapshots, screenshots and clipboard reads can contain private desktop content, and they reach your model provider like any other tool output.

Development

npm install
npm run typecheck
npm test

Layout: extensions/agent-desktop.ts (entry point: tools, command, lifecycle), src/cli.ts (binary resolution, envelope parsing), src/runtime.ts (session and headed policy), src/format.ts (tree rendering, truncation), src/tools.ts (tool schemas and argv mapping).