@agimon-ai/po-computer-use

Session scoped semantic computer control through the Po Desktop capability.

Packages

Package details

extension

Install @agimon-ai/po-computer-use from npm and Pi will load the resources declared by the package manifest.

$ pi install npm:@agimon-ai/po-computer-use
Package
@agimon-ai/po-computer-use
Version
0.0.0-alpha.183
Published
Nov 22, 2026
Downloads
31.3K/mo · 5,136/wk
Author
agiflow-ai
License
BUSL-1.1
Types
extension
Size
1.1 MB
Dependencies
8 dependencies · 4 peers

Security note

Pi packages can execute code and influence agent behavior. Review the source before installing third-party packages.

README

@agimon-ai/po-computer-use

Session-scoped semantic computer control through the Po Desktop capability.

Requirements

  • Node.js 24 or newer
  • A supported packaged macOS Po Desktop with native Computer Use permissions
  • The Pi version declared by this package's peer dependencies

Use

In Desktop, enable Computer Use in host settings, then select the default-off Computer Use mode. When the agent needs control, it asks for a window or the whole desktop and a duration. Approve once from an owner-authenticated client. The computer-use Activity card monitors the live grant and offers Stop; there is no separate setup form. Only one session can hold a live Desktop run. Remote activation does not bypass Desktop opt-in or grant checks. To stop from another app on macOS, press Cmd+Shift+Esc. If macOS cannot register the shortcut, use Computer Use → Stop Computer Use in Po Desktop or stop from Activity.

The computer-use mode exposes Activate, Deactivate, and Doctor actions. /computer-use reports the current state.

While active, the package exposes three tools:

  • computer_state observes the authorized window and returns a semantic snapshot. saveScreenshot: true also keeps the screenshot in session artifacts under computer-use/<run>/ and returns its artifact: path.
  • computer_action performs one of the semantic actions advertised for an element in a current snapshot. Approved-window grants use the bundled, unmodified agent-desktop CLI and show a visual agent cursor. The cursor is not a separate keyboard or mouse. Whole-desktop grants continue to use the Swift backend for the existing cross-app, Dock and menu-bar behavior.
  • computer_exec executes a reusable JavaScript or erasable TypeScript function and its relative helpers in a restricted runtime, then returns a fresh semantic observation.

Po’s Swift helper records MP4 and audio. Finished recordings are copied into session artifacts under computer-use/ in bounded ranges. For a window grant, an approved request for extra trace capture also keeps bounded JSONL action metadata and PNGs of that window. The trace does not contain element refs, entered text, or raw CLI debug output. Recording and tracing stop before sensitive input can be sent. Treat these artifacts as private desktop content. Generated functions receive the authorized program API, JSON input, bounded logging, and cancellation. They do not receive Node or filesystem access. Script-internal observations omit screenshots by default, and includeScreenshot: true requests an image explicitly. Full Node execution requires both trusted: true and an exact host-configured PO_COMPUTER_USE_SCRIPT_PATHS allowlist. Trusted scripts remain unsandboxed, including their imported dependencies. See Author computer functions for the contract, limits, trust boundary, and examples.

With code mode, computer.targets() lists candidate windows and computer.request() asks for human approval. Hosts without code mode use computer_targets and computer_request. The state, act, run and stop members are available only during an active grant. The codemode-how-to-computer-use guide is listed whenever the mode is selected.

Disabling the mode or host opt-in, stopping control, expiry, or Desktop disconnection stops further actions. An approval does not change who can access the session. The owner can monitor or stop control from the same signed-in browser that approved it.

Public API

import { ComputerScriptRunner, createComputerUseSessionClient } from '@agimon-ai/po-computer-use';
import type { ComputerScriptRun } from '@agimon-ai/po-computer-use';

The session-scoped server plugin owns the agent and hub routes. Their host-issued context tokens remain opaque and session-bound inside the request broker.

Development

pnpm fixcode
pnpm typecheck
pnpm test
pnpm build

License

Business Source License 1.1: free for personal use, organisations under USD 1M revenue, non-profits and education. See LICENSE and https://agimon.ai/licensing.