Pi-Droid

Give your AI agent hands on an Android phone.

pi install pi-droid

Pi-Droid is a pi-agent extension that gives AI agents direct, real-time control over Android devices via ADB. Annotated screenshots with numbered element indices let the agent tap, swipe, and type without guessing pixel coordinates — works on any screen size, any device.

---

What it does

Category	Tools	Highlights
See	6 perception tools	Annotated screenshots with numbered indices, raw UI tree, OCR fallback
Touch	5 input tools	Tap by text/ID/coords, type with clear-first, swipe, scroll, key events
Navigate	3 app tools	Launch/stop apps, wait for elements, wait for activities
System	6 system tools	Battery, settings, processes, logcat, shell, APK install
Manage	3 device tools	Multi-device registry, preflight checks, WiFi ADB
Lock	4 lock tools	Query/set/clear PIN and pattern locks
Record	2 recording tools	Screen recording, gesture macro record/save/replay
Automate	3 automation tools	Wake+unlock, find-and-tap with scrolling, scroll-until-found
Extend	4 plugin tools	Discover capabilities, run plugin actions, health check, heartbeat cycle

36 tools total, organized into 4 skills that scope tool sets for focused agent behavior.

---

Prerequisites

Requirement	Notes
Node.js >= 18	ESM support required
ADB on PATH	adb devices should list your device
Android device	USB debugging enabled, USB or WiFi connected
ADBKeyboard	Required for Unicode text input via android_type
Tesseract OCR (optional)	Only needed for android_ocr tool

---

Installation

pi install pi-droid

Set your device serial (optional if only one device is connected):

export ANDROID_SERIAL=your_device_serial

Verify

npx tsx run.mts screen

Development

git clone https://github.com/ArtemisAI/pi-droid.git
cd pi-droid
npm install
npm run build
npm test

---

Tool Reference

Perception (6 tools)

Tool	Description
android_look	Annotated screenshot with numbered element index — primary perception tool
android_screenshot	Raw screenshot — use for failure diagnosis only
android_ui_dump	Raw UI tree XML for full element hierarchy
android_ocr	Tesseract OCR on current screen or saved screenshot
android_observe	Continuous screen state observation
android_screen_state	Current activity, package, orientation, lock state as JSON

Input (5 tools)

Tool	Description
android_tap	Tap by coordinates, text match, or resource ID; supports long press
android_type	Type text into the focused field; optional clear_first to replace
android_swipe	Swipe between two coordinates with configurable duration
android_scroll	Scroll up or down on the current screen
android_key	Press a key: back, home, enter, tab, or any KEYCODE_*

App and Navigation (3 tools)

Tool	Description
android_app	Launch, stop, or check status of an app by package name
android_wait	Wait for an element to appear (by text or resource ID) with timeout
android_wait_activity	Wait for a specific activity to reach the foreground

System (6 tools)

Tool	Description
android_device_info	Battery, network, and hardware info
android_settings	Read/write system settings (WiFi, Bluetooth, brightness, volume, etc.)
android_processes	List running processes, kill by PID or name
android_logcat	Capture, search, and clear logcat
android_shell	Execute arbitrary ADB shell commands
android_install	Install/uninstall APKs, check package versions

Device Management (3 tools)

Tool	Description
android_devices	List connected devices, register/unregister, set active device
android_preflight	Run device readiness checks (ADB, screen, battery, etc.)
android_wifi	Connect/disconnect WiFi ADB, auto-discover devices

Lock Management (4 tools)

Tool	Description
android_lock_status	Query current lock state and type
android_lock_clear	Remove existing lock
android_lock_set_pattern	Set a pattern lock
android_lock_set_pin	Set a PIN lock

Recording and Macros (2 tools)

Tool	Description
android_record	Start/stop screen recording, pull recordings
android_macro	Record, save, load, and replay gesture macros

Automation (3 tools)

Tool	Description
android_ensure_ready	Wake screen, unlock, dismiss overlays — call before any automation
android_find_and_tap	Search UI tree for element and tap it; retries with scrolling
android_scroll_find	Scroll until an element appears, then return it

Plugin System (4 tools)

Tool	Description
android_skills	Discover all loaded plugin capabilities and parameters
android_plugin_action	Execute a plugin action with approval gates for sensitive operations
android_plugin_status	Get health status of all loaded plugins
android_plugin_cycle	Run a plugin's autonomous heartbeat cycle

---

Skills

Pi-Droid defines four skills that scope tool sets for focused agent behavior. Skills are discovered automatically by pi-agent's package manager.

Skill	Description	Key Tools
android-screen	Perceive device state	look, screen_state, screenshot, ui_dump, ocr, observe
android-interact	Perform actions on the device	tap, type, swipe, scroll, key, app
android-automate	High-level automation sequences	ensure_ready, find_and_tap, scroll_find, wait, wait_activity, preflight
android-plugin	Manage and execute app plugins	skills, plugin_action, plugin_status, plugin_cycle

---

Plugin System

Pi-Droid's plugin system lets you add app-specific automation behind a standard interface. Plugins can declare approval gates for sensitive actions (sending messages, making purchases, posting content).

Building a plugin

Extend CliPlugin for CLI-backed apps, or implement PiDroidPlugin for full control:

import { CliPlugin } from "pi-droid";

export class WeatherPlugin extends CliPlugin {
  name = "weather";
  // ...
}

Loading plugins

Plugins are configured in config/default.json and loaded automatically on session start:

{
  "plugins": {
    "weather": {
      "enabled": true,
      "package": "@example/pi-droid-weather"
    }
  }
}

Distribute plugins as npm packages. See PLUGINS.md for the full plugin development guide, manifest schema, and marketplace integration.

---

Configuration

Pi-Droid reads configuration from config/default.json:

{
  "adb": {
    "serial": "your_device_serial"
  },
  "plugins": {},
  "routing": {}
}

Key	Purpose
adb.serial	Device serial (overridden by ANDROID_SERIAL env var)
plugins	Plugin configurations keyed by name
routing	Input router settings for deterministic action dispatch

Environment variables:

Variable	Purpose
ANDROID_SERIAL	Target device serial (takes precedence over config)

---

Programmatic Usage

Pi-Droid exports its full ADB layer for use in custom automation scripts, external plugins, or standalone tools:

import {
  Device,
  tap, swipe, typeText, keyEvent,
  takeScreenshot, annotatedScreenshot,
  getScreenState, waitForActivity,
  launchApp, stopApp,
  dumpUiTree, findElement, waitForElement,
  ensureReady, findAndTap, scrollToFind,
  getBatteryInfo, getDeviceInfo,
  adbShell,
} from "pi-droid";

// Connect to a device
const device = await Device.connect(process.env.ANDROID_SERIAL);

// Take an annotated screenshot with numbered elements
const annotated = await annotatedScreenshot();

// Find and tap an element by text
await findAndTap({ text: "Settings" });

// Wait for an activity transition
await waitForActivity("com.android.settings/.Settings");

// Run a raw ADB shell command
const result = await adbShell("dumpsys battery");

Available exports

• Device abstraction: Device
• Command execution: adb, adbShell, AdbError, listDevices, isDeviceReady
• Input: tap, swipe, typeText, keyEvent, pressBack, pressHome, pressEnter, scrollDown, scrollUp
• Screen state: getScreenState, waitForActivity, getActivityStack, isKeyboardVisible, getOrientation
• Screenshots and perception: takeScreenshot, screenshotBase64, annotatedScreenshot, dumpUiTree, findElements, findElement, waitForElement
• App management: launchApp, stopApp, getAppInfo, listPackages, wakeScreen, isScreenOn
• Monitoring: getBatteryInfo, getNetworkInfo, getDeviceInfo, isScreenLocked, getRunningApps
• Automation: ensureReady, findAndTap, scrollToFind, DefaultStuckDetector, createTaskBudget
• OCR: runOcrOnImage, runOcrOnCurrentScreen
• Plugin system: PluginManager, CliPlugin, TelegramPlugin, ApprovalQueue
• Types: AdbExecOptions, UIElement, Bounds, ElementSelector, StuckEvent, and more

---

Development

npm install            # Install dependencies
npm run build          # Compile TypeScript
npm run lint           # Type-check without emitting
npm test               # Run all tests (473+)
npm run test:unit      # Unit tests only (mocked ADB)
npm run test:ci        # CI-safe subset (unit + sandbox integration)
npm run test:integration # All integration tests
npm run test:device    # Full suite including device E2E
npm run dev            # Development mode (load as pi extension)

Project structure

src/
  index.ts          Extension entry point
  adb/              ADB primitives (app-agnostic, 28 modules)
  tools/            LLM tool registrations
  plugins/          Plugin system (loader, CLI base class, marketplace)
  notifications/    Notification channels and approval queues
skills/             Skill definitions (scoped tool sets)
tests/              473+ tests mirroring src/ structure
config/             Default configuration

---

Contributing

See CONTRIBUTING.md for development guidelines, coding standards, and the PR checklist.

Security

See SECURITY.md for vulnerability reporting and security considerations.

Changelog

See CHANGELOG.md for release history.

License

MIT -- ArtemisAI