pi-control-chrome

English · 简体中文

Codex-aligned Chrome and Edge browser control for Pi, Codex and DSH. It reuses the user's existing Chromium profile through a local WebSocket Bridge and a Manifest V3 extension.

The Stage 1/2 core implementation is complete. Later-stage capabilities remain tracked separately in FEATURES.md.

What it provides

• Reuses the current Chrome or Edge profile, login state, cookies, extensions, and tabs.
• Lets Pi inspect and claim explicitly selected existing tabs without moving them by default.
• Places Agent-created tabs in a dedicated blue Pi tab group.
• Tracks Agent ownership, sessions, handoff, and deliverable lifecycle states.
• Closes unmarked Agent temporary tabs at turn end, releases claimed user tabs without closing them, and preserves only handoff or deliverable tabs marked for the current turn.
• Provides DOM, accessibility, locator, coordinate, and native CDP controls.
• Returns bounded semantic page state by default: visible/actionable snapshot nodes, 20,000-character/200-node snapshot and DOM-CUA budgets, and a shared 12,000-character extract budget. Same-origin iframe text is included by default with bounded frames diagnostics; pass includeFrames: false for shell-only inspection. browser_snapshot keeps refs and snapshotId while Pi/DSH projections omit duplicate raw accessibility and frameTree data; the Bridge advertises capabilities.compactResponses and negotiates compact page-read responses, while unspecified mode remains raw-compatible for older consumers; when tab metadata already carries title and URL, projections do not repeat them inside page results, and full page text is reserved for browser_extract when no semantic state is available.
• Accessibility reads prefer the real Chromium AX tree and support full, incremental-diff, and unchanged states; actionable/focusable AX nodes may carry document-scoped aN refs, which require the matching snapshotId for operations. Sensitive values stay redacted in AX and locator results. If the Accessibility domain is unavailable, reads safely fall back to the DOM semantic tree. Pass disableDiffing: true only when a complete AX state is required. Snapshot, accessibility, extract, and DOM-CUA reads accept optional selector, includeFrames, and output budgets. Pi and DSH adapters omit blank optional selector, snapshotId, incarnation, and screenshot path fields before dispatch.
• Ordinary role/name, label, and accessible-text locator, wait, and interaction targets now try Chromium AX first. AX resolves the computed role/name/state, then maps the current backend DOM node under the existing document/frame fence; successful actions expose internal provenance as resolvedBy: "chromium_ax". AX ambiguity, incomplete trees, and unsafe DOM mapping fail closed. Only an unavailable Accessibility domain, or a text-only target whose custom clickable element is absent from AX, uses the bounded DOM fallback; selectors, test IDs, and placeholders remain DOM-driven.
• Bounds browser_evaluate results to depth 8, 2,000 array items, 200 object fields, and 200,000 characters before returning them through the Bridge.
• Supports screenshots, page extraction, Console, Network, JavaScript dialogs, file upload, downloads, and clipboard text. Console and Network listings retain at most 200 entries and 20,000 serialized characters per read; Console reads support bounded only: "errors" filtering and since/nextSince cursors, including pageerror events.
• browser_probe_interaction performs exactly one explicit page action and returns before/after document identity, action confirmation, post-action Console errors, settle status, and post-action target state for UI regression diagnosis. It never replays an uncertain side effect.
• Captures ordinary active-tab viewport screenshots without opening a DevTools debugger session; full-page and background-tab captures use a short session-owned debugger lease.
• Persists debugger lease identity across an MV3 worker restart. Ordinary cleanup reports an unverified old lease instead of detaching an untracked target; explicit stale recovery verifies the current tab fence and CDP target identity before detaching it.
• Fences tab handles with a tab fence and document identity. Snapshot refs and DOM-CUA node ids are live observations within their originating document: title/focus changes, user tab switching, later observations, and unrelated DOM churn do not invalidate them; an original detached node can be rebound once only when a strongly equivalent replacement is unique. Navigation, reload, and document replacement remain hard boundaries and reject old observations with BROWSER_DOCUMENT_CHANGED; tab closure and tab-fence changes reject them with BROWSER_TAB_CLOSED and BROWSER_TAB_FENCE_CHANGED; navigate(wait: false), back, forward, and reload return a tab marked transitionPending whose handle omits unstable URL/title and document-incarnation fields, so wait or re-observe before document-bound work. Read-only page reads re-observe a user tab and retry once when its document changes, while side-effecting operations that lose document identity return an uncertain result and are not replayed automatically.
• Keeps a Manifest V3 Bridge socket active with application heartbeats, preventing idle worker suspension from creating avoidable connection-generation churn during long browser waits.
• Includes a reusable pi-control-chrome Skill. Browser tool schemas are hidden until that Skill is explicitly loaded for the current session; the bundled CLI remains available for explicit human/developer workflows and tests.

Semantic page interaction

The common click and form tools accept a semantic target such as { "role": "button", "name": "Submit" }, { "label": "Email" }, { "placeholder": "Search" }, { "text": "Next" }, or { "testId": "submit-button" }. Semantic interactions wait for one visible match and fail closed when a target is missing, hidden, disabled, or ambiguous; use an explicit zero-based index only when multiple matches are intentional. Actions apply that index after visibility filtering, so hidden duplicate controls do not consume it. Role/name, label, and accessible-text targets prefer Chromium's computed AX semantics and return resolvedBy: "chromium_ax" on successful mapped actions; CSS selectors, test IDs, and placeholders continue through DOM resolution. Same-origin iframe AX nodes participate in the same bounded semantic read, while cross-origin frame content remains a diagnostic boundary. browser_snapshot returns document-scoped live eN refs and a snapshotId; pass that matching snapshotId with every ref-based interaction or element-state wait. The resolver uses the original connected element first and allows one unique, strongly equivalent same-document rebind; CSS selectors remain supported.

Text targets used by action and element-state locators project a matching text leaf to its nearest actionable ancestor, so isEnabled and browser_wait report the state of the control that receives the action. browser_wait supports load, url, text, text_gone, visible, hidden, and enabled. Text conditions use text; element conditions use the same semantic target accepted by interactions. url and urlIncludes can constrain every wait condition. hidden succeeds when the target has no visible matches, including an absent target or multiple hidden matches; visible and enabled require one visible match, and multiple visible matches fail closed. For visible and enabled, an explicit index is applied after visibility filtering. Browser page matching runs inside the selected tab, while the Bridge continues to enforce the existing browser target and connection-generation fence. If a side-effecting interaction loses its injected result after navigation, it reports an uncertain outcome and is never replayed automatically.

Architecture

Pi Extension
     ↕ WebSocket
127.0.0.1 Local Bridge
     ↕ WebSocket
Chrome / Edge Manifest V3 Extension
     ↕
Current Chromium Profile

The Bridge binds to loopback and requires a local pairing token for WebSocket operations. Its /pair bootstrap response is intentionally available to any caller on that loopback port so an unpacked extension can pair without reading the host token file; any local process or installed extension that can reach the port is therefore trusted with browser control. Keep the Bridge port off network proxies. Host-launched instances expose a non-secret instance id, launcher label and capability list; any paired DSH or Pi Host in the same local-user control domain may request a cooperative restart when the Bridge exposes capabilities.localUserRestart: true and has no pending browser request. The instance id prevents stale restart races, and a restart lock serializes concurrent requests. Unknown legacy Bridges remain untouched when they do not expose the local-user capability. Installing the extension and completing local pairing are the trust steps; normal browser operations do not request repeated per-action authorization.

Installation

From npm (recommended)

pi install npm:pi-control-chrome

From GitHub

pi install git:github.com/lyd123qw2008/pi-control-chrome

From a local checkout

pi install <path-to>/pi-control-chrome

The package registers the Pi extension and the bundled Skill through its package.json pi manifest. It requires Node.js 22 or newer.

DSH integration

This repository also contains the standalone @lyd123qw2008/dsh-tool-control-chrome package. It registers the same browser-control surface as model-facing DeepSeek Harness tools and routes calls through the local Bridge. Install the DSH package in the active DSH Profile:

dsh plugin --profile web add @lyd123qw2008/dsh-tool-control-chrome

Then merge the insert entry from its cordis.patch.yml into the existing cordis.patch.yml when installing by hand (dsh plugin --profile web add applies that bundle patch automatically), and keep Bridge settings in <DSH_HOME>/settings.yaml.

The DSH package reuses this project's Bridge and Manifest V3 extension. Its default lazyTools: true mode exposes only the pi-control-chrome Skill metadata initially; after a successful Skill load, all 43 browser_* tools are registered in that Agent and remain active across turns in the current Agent session. At turn end, the host closes unmarked Agent temporary tabs, releases claimed user tabs without closing them, and detaches the session debugger lease. Bridge, browser tools, and Browser binding remain available for later turns. A model must call browser_mark_handoff or browser_mark_deliverable to preserve an Agent tab through the current turn cleanup, and repeat the mark in a later turn when needed. Only an explicit user request to close temporary tabs, release claims, or clean the browser task may trigger browser_cleanup; it performs immediate task cleanup while retaining the lazy tools and healthy Bridge. browser_context_reset is the separate explicit user-requested operation that finalizes resources and deactivates lazy tools. Agent and plugin disposal retry final cleanup; failed recovery blocks a replacement Agent that reuses the same session ID until cleanup succeeds. Set lazyTools: false for eager visibility. Pi registers the same native tools once but hides them with its active-tool set until the explicit /skill:pi-control-chrome expansion or another successful Skill activation. Ordinary web search does not activate browser control. The DSH package also provides human-only /chrome status, /chrome targets, /chrome profile [browserId], /chrome connect, /chrome disconnect, /chrome doctor, /chrome restart, and /chrome tabs commands. It does not install browser extensions automatically, read Chrome Profile files, or expose the Bridge beyond loopback.

Codex CLI and Desktop

The repository includes a Codex plugin manifest and a local MCP stdio adapter. After installing a package version containing this adapter, register it with codex mcp add pi-control-chrome -- pi-control-chrome-codex, or point codex mcp add at codex/mcp-server.mjs in this checkout. The adapter reuses the existing Bridge and exposes the bounded Codex browser tool set, including browser_restart after explicit user confirmation; it does not open an additional MCP port. See codex/README.md for setup and safety rules.

Load the browser extension

Pi cannot install an unpacked browser extension automatically. Load the shared extension/ directory once in Chrome or Edge:

1. Open chrome://extensions or edge://extensions.
2. Enable Developer mode.
3. Select Load unpacked.
4. Choose the repository or installed package's extension/ directory.
5. Start or reload Pi.

Check the connection in Pi:

/chrome status
/chrome targets
/chrome profile <browserId>
/chrome tabs

The extension is shared by Chrome and Edge and uses Chromium Manifest V3 capability detection rather than separate browser-specific implementations. One Bridge can now keep multiple identified browser targets connected at once. Each target is keyed by its stable browserId; requests use target-qualified routing and connection-generation fencing, so a newer connection for Profile A cannot replace Profile B or satisfy Profile A's stale requests. When multiple targets are available, select one explicitly with /chrome profile <browserId> or browser_status/the Skill CLI --browser-id; the runtime never guesses from the newest connection or active window.

Skill CLI

The bundled scripts are for explicit human/developer workflows and automated tests. They connect to the Bridge directly and must not be invoked by a model shell as a substitute for the Skill-gated native tools:

node skills/pi-control-chrome/scripts/browser.mjs status --browser-id <browserId>
node skills/pi-control-chrome/scripts/browser.mjs tabs --browser-id <browserId> --json
node skills/pi-control-chrome/scripts/browser.mjs group --browser-id <browserId> --json
node skills/pi-control-chrome/scripts/browser.mjs view https://example.com --browser-id <browserId> --session example-session --turn 1 --screenshot "$env:TEMP\example.png"
node skills/pi-control-chrome/scripts/browser.mjs cleanup --browser-id <browserId> --session <session-id> [--recover-stale]

Managed CLI commands use explicit lifecycle identity: open and cleanup require --session <id>, and view requires both --session <id> and --turn <n> unless it is explicitly temporary. The CLI does not derive ownership from a process id, so a later invocation can address the same tabs safely. Use cleanup --recover-stale only for an explicit recovery decision after an extension runtime change; it forgets matching unknown-incarnation ownership records without closing those tabs and reports their ids in recovered for manual inspection.

The bundled scripts support these environment variables. The Pi extension also accepts PI_CONTROL_CHROME_BRIDGE_PORT when the local Bridge uses another loopback port; its host remains 127.0.0.1:

PI_CONTROL_CHROME_BRIDGE_HOST   # bundled CLI only
PI_CONTROL_CHROME_BRIDGE_PORT   # bundled CLI and Pi extension

The checked-in browser extension still connects to 127.0.0.1:17318 and its manifest CSP allowlist is fixed to that endpoint. Use a matching rebuilt extension and CSP before pointing the Pi extension or DSH client at another endpoint.

Use the native browser_* Pi or DSH tools for model browser work. They preserve the current Agent session, target identity, and tab ownership protections. User tabs remain valid browser targets, but their documents can be replaced while the user or DSH UI is active; read-only page reads re-observe a changing document once, while side-effecting operations retain strict document fencing. Automated browser verification should use an Agent-owned test tab or an isolated browser profile instead of the active DSH GUI tab.

Development and tests

cd <path-to>/pi-control-chrome
npm install
npm run check
npm test
npm run test:pi:lifecycle
npm run test:skill
npm run pack:check
npm run test:package-install

npm run test:skill requires a connected Chrome or Edge profile and the local Bridge. It creates temporary Agent tabs, so when multiple ready targets exist it skips rather than choosing one; set PI_CONTROL_CHROME_TEST_BROWSER_ID only for an explicitly authorized target. The high-coverage browser smoke test is:

npm run smoke:e2e

The multi-target acceptance test launches two isolated temporary profiles of the configured browser (Edge by default, or Chrome for Testing when selected) and verifies explicit routing, target disconnect isolation, and same-Profile reconnection fencing:

npm run smoke:e2e:multi-profile

The smoke test defaults to Edge. The current worktree has passed the same isolated coverage on Edge and Chrome for Testing 149.0.7827.55. Run it against Chrome for Testing with:

$env:PI_CONTROL_CHROME_BROWSER = "<path-to>\chrome-for-testing\chrome.exe"
npm run smoke:e2e

This requests a unique temporary --user-data-dir rather than the normal user profile. The harness fails if the browser process exits before the extension handshake, which detects common Windows singleton delegation; Chrome for Testing provides the most reliable isolated executable. The installed Google Chrome may reject command-line unpacked-extension flags; load extension/ manually from chrome://extensions for a normal-profile check.

Design principles

1. Install once, operate with low friction. Extension installation and local pairing are the user-facing trust steps.
2. The extension is the browser capability boundary. Pi communicates with it only through the local Bridge.
3. User tabs are preserved by default. Claiming a tab establishes control ownership; it does not move the tab into the Agent group.
4. Agent tabs are reclaimable. Agent-created tabs carry ownership and session metadata and are cleaned up according to lifecycle policy.
5. Chrome and Edge share one implementation. The extension uses Manifest V3 and capability detection.
6. Observable behavior is aligned without copying private Codex runtime code. Pi uses its own Bridge protocol and Extension API.
7. Capability layer, not a site adapter. The plugin understands page structure, identity, safety boundaries, and generic primitives; a product's DOM, workflow, and field names belong to the calling Skill. See Capability layer and personalization layer.

Documentation

• ARCHITECTURE.md — English architecture documentation.
• README-zh-CN.md — Simplified Chinese guide.
• FEATURES.md — feature scope and future work.
• CODEX-ALIGNMENT.zh-CN.md — Codex behavior alignment notes.
• DECISIONS.zh-CN.md — project decisions.
• docs/RELEASE-CHECKLIST.zh-CN.md — package/version, dependency, PR, npm publication and active DSH Profile release checklist.
• skills/pi-control-chrome-release/SKILL.md — executable release procedure for Pi and DSH packages.
• skills/pi-control-chrome/SKILL.md — bundled Pi Skill.
• CHANGELOG.md — release history.
• BROWSER-ACTIVATION-DESIGN.zh-CN.md — browser capability activation and on-demand Skill design.
• docs/BROWSER-LIFECYCLE-CODEX-ALIGNED.zh-CN.md — implemented Codex-aligned browser lifecycle.
• docs/BROWSER-OUTPUT-COMPACTION-DESIGN.zh-CN.md — implemented bounded model-facing browser output and AX revision design.
• docs/AGENT-BROWSER-RUNTIME-DESIGN.zh-CN.md — Agent-first live-ref runtime design, implementation status, and AX runtime status.

Current future scope

The following items are planned for later stages and do not block the current browser-control loop:

• WebMCP, GSuite export, and browsing history APIs.
• Dedicated media-download interfaces.
• Capability discovery beyond the current Bridge target and extension capability handshake.
• Chrome Web Store and Edge Add-ons release packages.
• Dedicated Brave and Chromium acceptance coverage.
• Cross-Bridge orchestration and simultaneous multi-target control within one session.

License

MIT. See LICENSE.