Pi Embark

Pi Embark lets Pi hand off remote coding tasks through a Cloudflare broker to a remote runner, with sync, attach, status, cancellation, and Docker-isolated execution.

Key surfaces: /embark, /embark status, /embark tasks, /embark attach, /embark sync, /embark cancel

Install

Published package:

pi install npm:pi-embark
pi update npm:pi-embark
pi -e npm:pi-embark

GitHub fallback:

pi install git:github.com/danialranjha/pi-embark
pi -e git:github.com/danialranjha/pi-embark

Install from a local checkout:

cd /path/to/pi-embark
pi install .
pi -e .

Local dev usage without publishing (global-style testing from the checkout):

cd /path/to/pi-embark
EMBARK_CLIENT_SECRET=dev-client-secret \
EMBARK_BROKER_URL=http://localhost:8787 \
pi -e .

Project-local install into the current repo’s .pi/ directory:

cd /path/to/your/project
pi install -l /path/to/pi-embark

30-second quick start

EMBARK_CLIENT_SECRET=dev-client-secret \
EMBARK_BROKER_URL=http://localhost:8787 \
pi --no-extensions --extension ./src/extensions/embark.ts

Then run:

/embark "echo hello world"
/embark status
/embark attach latest

Why install it

• hand off long-running Pi work while your laptop is away
• keep local dirty work in sync with remote execution
• attach back to results, status, and event streams from Pi
• run tasks with explicit provider/model intent and task-local Docker bundles

Highlights

• one client secret and one runner enrollment secret from env vars
• per-task callback tokens issued when a runner claims work
• runner polls assigned tasks and runs pi --mode rpc
• runner enforces explicit execution intent via execution.authProfile + provider + model
• runner resolves credentials from curated host-local config instead of ambient synced laptop auth state
• runner generates a task-local bundle per task and injects only that bundle into Docker
• runner polls its active task for broker-side cancellation and stops Pi promptly
• optional Docker isolation mounts each task at /embark/task and runs Pi in a named per-task container
• dirty local work syncs through a Git manifest + rsync overlay with fingerprints

This is still pre-production in the broader roadmap sense, but the execution-intent / runner-local-config / task-local-bundle rollout is now implemented and validated locally plus on the Hetzner host. Remaining broader production work is registry publication, Docker egress controls, cancellation artifact polish, and automatic Hetzner host provisioning.

Visual Tour

/embark help is grouped around the common loop first, with targeted and maintenance commands separated so it is easier to scan inside Pi.

/embark tasks opens an interactive picker in normal Pi TUI sessions. The rows show state, short task id, relative age, sync fingerprint, and prompt context; arrow keys move the selection and Enter attaches the highlighted task.

Selecting a task attaches to its event stream and brings the task result back into the Pi conversation, with a compact live widget at the bottom.

/embark status is focused on the attached/latest task instead of dumping the full task list, so the result and recent events are directly usable as context.

Local Phase 1 Loop

Install dependencies:

npm install

Run repeatable validation:

npm run typecheck
npm run smoke:pi-rpc
npm run smoke:runner-safety
npm run smoke:dirty-sync
npm run smoke:runner-cancel
npm run smoke:runner-image-config
npm run smoke:execution-contract
npm run smoke:runner-config
npm run smoke:task-bundle
npm run smoke:execution-failures
npm run smoke:docker-isolation

smoke:pi-rpc starts a temporary local Wrangler broker, loads the Embark extension through pi --mode rpc, and validates help output, submit/status output, scoped /embark sync cursor behavior, completed-result sync through the runner callback token path, task attach/list targeting, and session-start auto-sync on a reopened session. Use EMBARK_SMOKE_PORT=8799 npm run smoke:pi-rpc if port 8788 is busy.

smoke:runner-safety validates that the runner does not claim an inaccessible HTTPS repository, does not allow Git to prompt for a GitHub username, and leaves the task assigned with a clear repo_unavailable_before_claim log. Use EMBARK_RUNNER_SMOKE_PORT=8791 npm run smoke:runner-safety if port 8790 is busy.

smoke:dirty-sync validates the Phase 2.5 Git manifest + rsync overlay path: uncommitted tracked edits and nonignored untracked files reach the runner workspace, ignored files and node_modules do not, unchanged fingerprints reuse the cached overlay, and changed files invalidate the fingerprint.

smoke:runner-cancel validates that broker/user cancellation of a running task is observed by the runner and stops the active Pi child promptly.

smoke:runner-image-config validates the production runner image contract: the Dockerfile pins the Pi package/version, installs the expected task runtime tools, runs as a non-root user, keeps secrets/build noise out of the Docker context, and documents pulled-image pi --mode rpc verification.

smoke:execution-contract validates the Phase 1 execution contract and resolution semantics: supported auth profiles, valid and invalid provider/model/profile combinations, runner-default precedence, and the rule that explicit provider/model cannot remain advisory metadata.

smoke:runner-config validates the Phase 2 runner-local secret source design: config parsing, supported version, absolute approved-root policy, readable per-profile env/config files, and rejection of broken host config before task launch.

smoke:task-bundle validates the Phase 3 task-local bundle format: per-profile env/config materialization, task-unique bundle paths, non-secret metadata, and cleanup semantics.

smoke:execution-failures validates the Phase 5 negative path: malformed execution payloads are rejected, invalid profile/provider/model requests fail before Pi starts, missing bundle inputs fail explicitly, and no Docker container starts for those pre-launch failures.

smoke:docker-isolation validates the opt-in Docker runner mode with a fake Docker/Pi harness: each task gets a unique container name, a distinct task-directory volume mounted at /embark/task, the configured Docker image/network, interactive RPC stdin, and Docker isolation metadata in broker events. Separate real-Docker remote validation has also been completed on the Hetzner host; production egress restrictions remain a later hardening step.

Run the broker locally:

EMBARK_CLIENT_SECRET=dev-client-secret \
EMBARK_RUNNER_ENROLLMENT_SECRET=dev-runner-secret \
npm run dev:broker

Run the runner locally:

EMBARK_RUNNER_ENROLLMENT_SECRET=dev-runner-secret \
EMBARK_BROKER_URL=http://localhost:8787 \
EMBARK_SPACES_DIR=/tmp/pi-embark-spaces \
npm run dev:runner

Before running the runner against a private repo, validate Git credentials on that host:

GIT_TERMINAL_PROMPT=0 git ls-remote <repo-url> HEAD

If this fails, the runner will now skip the task before claiming it and log repo_unavailable_before_claim. For private GitHub repositories, SSH remotes with a loaded deploy key are the recommended near-term setup.

For manual testing, you can start or reuse the local broker and runner in the background, then launch Pi in a target repo with one command:

/Users/danial/ws/pi-embark/scripts/embark-local.sh /path/to/target/repo

The script writes logs, PID files, and Wrangler broker state under /tmp/pi-embark-manual by default. Each launch stops the previously tracked local broker/runner first, including Wrangler's project-owned workerd listener, then starts fresh services from the current checkout while preserving broker state. Set EMBARK_MANUAL_REUSE=1 if you explicitly want to reuse already-running tracked services, or EMBARK_MANUAL_RESET_STATE=1 when you want a clean local broker. Useful controls:

/Users/danial/ws/pi-embark/scripts/embark-local.sh status
/Users/danial/ws/pi-embark/scripts/embark-local.sh logs
/Users/danial/ws/pi-embark/scripts/embark-local.sh stop

You can override defaults with EMBARK_CLIENT_SECRET, EMBARK_RUNNER_ENROLLMENT_SECRET, EMBARK_BROKER_URL, EMBARK_BROKER_PORT, EMBARK_SPACES_DIR, EMBARK_MANUAL_STATE_DIR, and EMBARK_MANUAL_WRANGLER_STATE_DIR.

The manual launcher does not pass --no-extensions by default, so your normal Pi extensions such as model routing still load. Set EMBARK_PI_NO_EXTENSIONS=1 only when you want isolated extension validation.

For release capture and a low-fragility demo harness, see docs/release-demo.md.

Load the extension in Pi from a git repository:

EMBARK_CLIENT_SECRET=dev-client-secret \
EMBARK_BROKER_URL=http://localhost:8787 \
pi --no-extensions --extension ./src/extensions/embark.ts

--no-extensions disables discovered/global extensions while still loading the explicit Embark extension. Use it for Phase 1 validation so unrelated Pi extensions do not intercept startup, session, or UI events.

Then use:

/embark "echo hello world"
/embark status
/embark status latest
/embark status <task-id>
/embark status <number>
/embark list
/embark tasks
/embark sync
/embark sync all
/embark sync <task-id>
/embark sync <number>
/embark reconcile
/embark cleanup
/embark attach latest
/embark attach <task-id>
/embark attach <number>
/embark detach
/embark cancel latest
/embark cancel <task-id>

/embark status shows detailed state for the active attached task, or the latest tracked task if nothing is attached. You can also pass latest, a task id, a task id prefix, or a number from /embark tasks.

/embark sync pulls broker events newer than the local lastSyncedSeq cursor for the active attached task, or latest tracked task if nothing is attached. It no longer imports every broker-known task by default. Use /embark sync all only when you explicitly want a broad recovery sync.

/embark list shows a compact local/broker table for tasks tracked in the current session. /embark tasks opens an interactive task picker in normal Pi TUI sessions; selecting a task attaches to it and shows the latest result/events. In non-interactive or RPC mode, /embark tasks falls back to a numbered board you can use with /embark attach <number>, /embark status <number>, and /embark sync <number>. /embark reconcile imports broker-known tasks, refreshes local state against broker truth, and marks missing broker records as orphaned. /embark cleanup marks orphaned local records as abandoned without deleting session history.

/embark attach latest, /embark attach <task-id>, and /embark attach <number> poll a single task stream and remember a separate attach cursor, so repeated attach polls only show new events. /embark detach clears the local active attachment.

/embark cancel latest and /embark cancel <task-id> mark an assigned or running broker task as cancelled. The runner polls for cancellation while Pi RPC is active and stops the active Pi process or Docker container promptly.

If you reopen the same Pi session with pi -c, Embark uses the session's local embark-task entries. If you accidentally start a fresh Pi session, /embark list, /embark tasks, /embark reconcile, /embark sync all, /embark attach latest, and /embark attach <task-id> recover by importing broker-known tasks into the new local session.

Runner Notes

The runner expects git and rsync on the host. In the default process isolation mode it also expects pi on the host. In docker isolation mode it expects Docker on the host and a Pi-capable image. It clones the target repository into EMBARK_SPACES_DIR/<task-id>/workspace, checks out the exact commit submitted by the extension, writes the prompt to prompt.txt, starts Pi in RPC mode, and reports completion back to the broker.

Embark includes useful local work by default. The extension builds a Git manifest from git ls-files --cached --others --exclude-standard, excludes ignored/dependency/cache paths, fingerprints the file content and sync config, and sends that bounded payload with the task. The runner hydrates Git first, then uses rsync from a fingerprinted cache into the workspace. Identical fingerprints reuse the cached overlay.

Runner knobs:

• EMBARK_PI_RPC_TIMEOUT_MS sets the maximum wait for a Pi RPC agent_end event before failing the task. Defaults to 30 minutes.
• EMBARK_RUNNER_HEARTBEAT_MS sets the runner heartbeat interval while Pi RPC is active. Defaults to 30000.
• EMBARK_RUNNER_CANCEL_POLL_MS sets how often the runner checks broker state for cancellation while Pi RPC is active. Defaults to 5000.
• EMBARK_RUNNER_CANCEL_KILL_TIMEOUT_MS sets how long the runner waits after SIGTERM before SIGKILL when cancelling Pi. Defaults to 5000.
• EMBARK_RUNNER_ISOLATION=process|docker selects the Pi runtime. Defaults to process.
• EMBARK_RUNNER_DOCKER_IMAGE selects the image used in Docker mode. Defaults to pi-embark-runner:latest.
• EMBARK_RUNNER_DOCKER_NETWORK selects the Docker network for task containers. Defaults to bridge.
• EMBARK_RUNNER_DOCKER_USER optionally passes docker run --user, useful when the container UID/GID must match the host workspace owner.
• EMBARK_RUNNER_DOCKER_TASK_DIR selects the mounted task directory inside the container. Defaults to /embark/task.
• EMBARK_RUNNER_DOCKER_ENV is a comma-separated allowlist of host env var names to pass through with docker run --env.
• EMBARK_RUNNER_DOCKER_ENV_FILES is a comma-separated list of env-file paths to pass through with docker run --env-file.
• EMBARK_RUNNER_DOCKER_MOUNTS is a comma-separated list of additional Docker volume specs, intended for narrow read-only credential/config mounts.
• EMBARK_RUNNER_WORKSPACE_CLEANUP=never|success|terminal controls whether the runner removes workspace/ after task completion. Defaults to never; use success or terminal for production retention.
• EMBARK_RESULT_MAX_CHARS limits how much final assistant text the runner stores on the broker task. Defaults to 20000 characters.
• EMBARK_PI_NO_EXTENSIONS=1 starts remote Pi with --no-extensions. Leave it unset if your runner relies on local Pi extensions for model routing.
• EMBARK_SYNC_MAX_FILES caps manifest file count. Defaults to 500.
• EMBARK_SYNC_MAX_BYTES caps manifest payload bytes. Defaults to 2097152.
• EMBARK_SYNC_EXCLUDES adds comma-separated path-part excludes on top of .git, dependency folders, build outputs, and caches.
• PHASE3_HEARTBEAT_TIMEOUT_MS controls when broker liveness marks a running task stale. Defaults to 5 minutes.
• PHASE3_STALE_TIMEOUT_MS controls when a stale task becomes failed with stale_timeout. Defaults to 30 minutes.
• PHASE3_CLIENT_SILENCE_MS controls when broker abandonment cleanup marks a non-terminal task cancelled with client_abandoned. Defaults to 30 minutes and is capped at 24 hours.

The production Pi runtime image lives at docker/pi-runner/Dockerfile. See docs/runner-image.md for build commands and the credential/mount policy.