pi-gui-use

A GUI-use agent built on pi — a sibling of pi-coding-agent that drives a GUI instead of a codebase. A11y-first: the agent reads screens as numbered text element lists and asks for pixels only when text is not enough. Two drivers today — browser (playwright-core over CDP, the default) and android (adb against a running emulator); the GuiDriver seam keeps further desktop/mobile drivers addable without touching tools or prompts.

Install

Install the command globally:

npm install -g @mirofish-ai/pi-gui-use

Or add the library to a project:

npm install @mirofish-ai/pi-gui-use

Browser startup is automatic. In normal use pi-gui tries, in order:

1. PI_GUI_CDP_URL when set (treated as a recoverable hint),
2. an existing Chrome at http://127.0.0.1:9222,
3. the managed Chrome: adopt-or-launch on a persistent profile at ~/.pi-gui/chrome.

The managed browser is a device, not a subprocess. It is launched detached with --remote-debugging-port=0 and found again through the DevToolsActivePort file Chrome publishes into the profile, so a follow-up session adopts the running browser (milliseconds) instead of paying a launch (~0.7–1.5s) — logins, cookies and tabs carry over. Closing pi-gui disconnects; it never closes the browser. A leftover browser whose renderer froze is detected by a short attach deadline, killed by identity (only pids whose command line names our profile directory — the user's own Chrome is unreachable by construction) and replaced in about a second.

While the managed browser runs, its endpoint is published for observers at ~/.pi-gui/run/<profile>/endpoint.json ({cdpUrl, port, pid, profileDir, updatedAt}), modelled on Chrome's own DevToolsActivePort: a host that wants a live read-only view (screencast, debugging) attaches there without asking pi-gui.

Action points are published the same way, as a sibling actions.jsonl: one JSON line per successful action ({ts, kind, x?, y?, url}, plus direction for scrolls), where x/y is the page-CSS-pixel point the action landed on — the given point for coordinate actions, the element center for ref actions. A host compositing a live preview reads it to animate a pointer where each action lands. Best-effort and observer-only: appends are fire-and-forget, and the file is truncated once it exceeds 256 KB. Attached (non-managed) browsers publish nothing — they have no run dir.

Managed-browser knobs:

• PI_GUI_PROFILE_DIR — profile directory (default ~/.pi-gui/chrome).
• PI_GUI_CHROME — Chrome/Chromium binary, when discovery misses.
• PI_GUI_WINDOW=offscreen — park the window off-screen (for hosts that render their own preview). Off-screen, never minimised: a minimised window stops compositing, which kills screencast frames and hangs wheel input. Default is a visible window.

If no Chrome exists and none of the knobs point at one, the first browser action fails with the fix in hand (install Chrome / set PI_GUI_CHROME / set PI_GUI_CDP_URL) — never a silent degrade.

To drive an existing Chrome, start it with CDP enabled:

"/Applications/Google Chrome.app/Contents/MacOS/Google Chrome" --remote-debugging-port=9222

A stale PI_GUI_CDP_URL no longer blocks startup. For a strict endpoint that must not fall back to another browser, use pi-gui run --cdp <url> ....

Android (emulator)

The second driver (--driver android or PI_GUI_DRIVER=android) drives an Android device entirely through public adb — the mechanisms are ported from mirasim's simulator. Requirements: adb (platform-tools) and a booted device. Device DISCOVERY picks emulators only (never whatever phone is plugged in); an explicitly named serial — PI_GUI_ANDROID_SERIAL, tcp serials like a redroid container included — is honored verbatim as the caller's informed choice.

Lifecycle is a separate concern from driving, with its own verbs:

pi-gui android doctor        # adb/emulator/SDK/AVD/device state, with the fix for each gap
pi-gui android setup --yes   # install SDK pieces + create the profile's AVD
                             # (--profile plain|androidworld; androidworld = Pixel 6 /
                             #  API 33 / -grpc 8554 / cold boot — the benchmark's exact
                             #  environment; ≈1.5 GB from Google, explicit consent only)
pi-gui android start         # adopt a running emulator or boot one (window shown;
                             # --headless boots -no-window -gpu host); prints the serial
pi-gui android stop          # stop an emulator (emulator-<port> serials only)

An existing SDK (ANDROID_HOME / ANDROID_SDK_ROOT / the default install paths) is adopted, never duplicated; a fresh install lands in ~/.pi-gui/android-sdk. Setup — and only setup — needs a JRE 17+ (sdkmanager/avdmanager are JVM tools).

What works: observations come from uiautomator dump — every element named with exact bounds, which on Android is both cheaper and more precise than pixels; screenshots via screencap, downscaled to ≤1568px on the long side with the model's point coordinates mapped back to device pixels (vision APIs silently resize past that, which would break every tap); input via adb shell input (tap at ref centers or coordinates, swipe-based scrolling, text with shell escaping, keyevents for Enter/Back/Home/Tab/arrows); extra verbs swipe (explicit gesture), launch_app (resolved launcher activity via am start — never monkey, which lies about success), open_url (VIEW intent) and back. All three strategies (tree/hybrid/pixels) apply.

Knobs:

• PI_GUI_ANDROID_SERIAL — target device serial (default: the first emulator-* from adb devices). Any serial adb devices shows is accepted when named.
• PI_GUI_ADB — adb binary, when discovery (PATH, then the default SDK locations) misses.

One honest limitation inherited from Android itself: the OS has a single UiAutomation slot. While an accessibility service or instrumentation harness holds it, uiautomator dump cannot run — element observations fail with the cause and fix named, and the device can only be driven from screenshots (--strategy pixels) until the slot is released.

Use

pi-gui                       # interactive TUI — the full pi experience, GUI tools only
pi-gui -p "task..."          # non-interactive print mode (from the pi runtime)
pi-gui run --task "..." --json   # headless one-shot, JSONL events (integration contract)
pi-gui mcp                   # MCP stdio server for Claude Code / Codex
pi-gui eval setup android    # one command to a runnable eval harness (browser|android)

Inside the TUI, /eval lists every installed benchmark (WebArena, AndroidWorld) and /eval <benchmark> <task> runs a case in-session — environment staged automatically (an emulator boots itself if needed), every tool call visible, verdict from the same harness batch runs use. A benchmark needing the other driver answers with the exact relaunch command instead of half-working. Embedding pi-gui in another agent: docs/EMBEDDING.md.

• TUI / print / rpc run on the pi runtime (pi-coding-agent's exported main()) with an inline extension that registers the gui_* tools, narrows the toolset to GUI-only, and swaps in the GUI system prompt. Models, /login, sessions, and ~/.pi/agent auth are shared with pi. pi's native skills and CLAUDE.md/AGENTS.md discovery are disabled — prompt content comes only from pi-gui's own knobs. To use a skill library in GUI sessions, point PI_GUI_SKILL_DIR at it (:-separated dirs, SKILL.md format — ~/.pi/agent/skills and ~/.claude/skills load unchanged); skills are then listed by name in the prompt and fetched via the load_skill tool. PI_GUI_IDENTITY_FILE replaces the identity line of the system prompt (the operational rules always remain).
• run is our own thin runner on pi-agent-core: one task per process, events on stdout (assistant / action / observation / result), exit code 0 iff status is done. Model via --model provider/id or PI_GUI_MODEL (default openrouter/google/gemini-2.5-flash; API keys via provider env vars).
• mcp serves the tools to a host that brings its own agent loop:

  claude mcp add pi-gui -- pi-gui mcp
  

Model gateway (host embedding)

A host that routes model traffic through its own Anthropic-Messages-compatible endpoint (a local proxy, a relay, any LiteLLM-style gateway) sets one env knob — nothing is written to ~/.pi, and standalone use is unaffected when the variable is absent:

PI_GUI_GATEWAY_URL=http://127.0.0.1:PORT      # endpoint origin (anthropic-messages wire)
PI_GUI_GATEWAY_KEY=...                        # credential, sent as x-api-key
PI_GUI_GATEWAY_MODELS=google/gemini-3.5-flash,claude-sonnet-5   # wire ids, verbatim
PI_GUI_GATEWAY_PROVIDER=mirasim               # pi-local prefix, default "gateway"

This registers an in-memory provider on every entry (TUI/print/rpc and run), and the default model becomes PI_GUI_MODEL or the first roster entry. See src/agent/gateway.ts.

Already a pi user? The package is also a regular pi extension:

pi install npm:@mirofish-ai/pi-gui-use   # adds gui_* tools to your normal pi

Tools

Three tools, assembled per driver × strategy at session start:

tool	role
gui_targets	list operable surfaces (tabs / windows / apps) with stable ids
gui_observe	read state — what: elements (refs + form states) / text / screenshot / full
gui_act	every mutation: click, hover, set_text, keys, scroll, focus, wait + driver verbs (open_url, select_option, back, reload, ...); results report whether the screen changed

Strategies (--strategy or PI_GUI_STRATEGY): tree (text element lists, default), pixels (screenshots + coordinates), hybrid (refs + screenshot fallback).

Layout

src/platform/   the platform layer: shared types, abstract GuiDriver, the 3 tool factories,
                driver subclasses (browser + android today; macos/... later), registry
src/agent/      session bootstrap (profile, prompt assembly, skills) + our own loop: runner, pruning
src/entries/    the four ways in: cli, tui, mcp, extension (+ entries/commands/ bodies)
src/index.ts    library export

Repository maintainers can see CLAUDE.md for the design rules. It remains in the source repository and is intentionally excluded from the published npm package.

Controlling the pi version

pi packages are pinned exactly (pi releases in lockstep):

npm run pi:set 0.84.0   # rewrite pins + reinstall
npm test                # typecheck + smoke + agent test

Development

npm run dev             # run the CLI from source (tsx)
npm test                # typecheck + smoke (real headless Chrome) + agent test (faux LLM, no key)
npm run demo            # watch a headed Chrome driven live, agent view printed in lockstep
npm run build           # compile to dist/ (what the published bin runs)

Publishing to npm (maintainers)

The release script requires an active npm login. It checks that the Git working tree is clean, verifies that the version is unused, runs the test and build suites, rejects packages containing source/private files, performs an npm dry run, and asks for confirmation immediately before the public publish.

From the source repository, run:

npm run release:npm

Confirm the active account before publishing:

npm whoami

If this command is not authenticated, run npm login first. The logged-in account must have permission to create or update the package's scope.

The equivalent manual validation and publishing commands are:

npm whoami
npm ci
npm test
npm run build
npm pack --dry-run
npm publish --dry-run --access public
npm publish --access public
npm view @mirofish-ai/pi-gui-use version

Each npm version is immutable. Increment version in package.json before publishing a subsequent release.