@lennix1337/pi-agent-desktop

Computer use for the Pi coding agent: see and control native Windows, macOS and Linux apps (click, type, read, screenshot) through the OS accessibility tree, powered by vercel-labs/agent-desktop.

Packages

Package details

extensionskill

Install @lennix1337/pi-agent-desktop from npm and Pi will load the resources declared by the package manifest.

npm repo home report

$ pi install npm:@lennix1337/pi-agent-desktop

Package: @lennix1337/pi-agent-desktop
Version: 0.1.2
Published: Nov 1, 2026
Downloads: 695/mo · 695/wk
Author: lennix1337
License: MIT
Types: extension, skill
Size: 17.6 KB
Dependencies: 2 dependencies · 1 peer

Pi manifest JSON

{
  "skills": [
    "./skills"
  ],
  "extensions": [
    "./src/index.ts"
  ]
}

Security note

Pi packages can execute code and influence agent behavior. Review the source before installing third-party packages.

README

pi-agent-desktop — computer use for Pi

Give the Pi coding agent eyes and hands on your desktop. It reads any native app on Windows, macOS and Linux through the operating system's accessibility tree, then clicks, types, fills fields, reads values and takes screenshots. Powered by vercel-labs/agent-desktop.

pi install npm:@lennix1337/pi-agent-desktop

No global install and no Python: the native binary for your platform ships inside the package.

What you can ask Pi

"Open Notepad, write the release notes and save them to my Desktop."
"Read the error text in that installer window and tell me what it wants."
"Fill this desktop form with the data from customers.csv."
"Take a screenshot of the Settings window and check the toggle is on."

How it works

Pi gets one desktop tool and a pi-agent-desktop skill that teaches the loop:

list-apps / list-windows find the target.
snapshot returns the accessibility tree as compact JSON with @ref ids (no pixels, cheap in tokens).
Act on a ref: click, set-value, type, press, launch, close-app...
Verify with get or a fresh snapshot; screenshot attaches a PNG when the tree is not enough.

Native apps snapshot in about 0.2 s. Electron/Chromium apps (Slack, VS Code) expose deep trees: snapshot them with --max-depth 3; the tool suggests it on timeout.

Safety

Semantic, headless delivery by default: no cursor movement, no focus stealing.
Physical input (mouse-*, drag, hover, double-click, key-down/up) is refused unless the call sets headed: true; --headed cannot be smuggled through args.
Commands run without a shell; batch, session, cursor-overlay and --debug are not exposed.
macOS: grant Accessibility (and Screen Recording for screenshots) to the terminal that runs Pi.

vs. other options

	pi-agent-desktop	Screenshot + vision models	Browser tools
Native desktop apps	yes	yes	no
Reads exact text/values	yes (accessibility tree)	OCR guess	web only
Token cost per step	small JSON	large images	small

Not to be confused with abcwyc/pi-agent-desktop, a desktop chat app for Pi. This package gives Pi a tool; it is not an app.

Develop

npm install
npm run check   # types
npm test        # unit + Pi loader test (runs the real binary's `version`)
npm run e2e     # real desktop: opens Notepad, writes, reads back, screenshots, closes (Windows)

Override the binary with PI_AGENT_DESKTOP_BIN.

Release

npm run release bumps the patch version, tags it and pushes. The Release workflow then runs the checks and publishes to npm with provenance through npm trusted publishing (no token, no OTP), and creates the GitHub release.