@parcha/pi-vision

pi-vision lets a pi agent read documents, look at pictures, scans and videos, and write down what a recording says. Each look (one question a vision model answers about a picture, a page or a window of film) comes back with its answer and its price, and a look that cannot be made says why.

Try it

Free, on scripted models: no key, no network. The demos are in this repository, not in the npm package: clone it, with Node 22.19 or later, and with ffmpeg, ffprobe, pdftoppm and pdftotext on your path (brew install ffmpeg poppler, or sudo apt-get install ffmpeg poppler-utils). Installing and building the whole workspace takes a few minutes:

git clone https://github.com/Parcha-ai/agentrun.git && cd agentrun
npm ci --ignore-scripts && npm run build
npm run demo:vision -- examples/document-check

examples/vision-agent is one agent that answers questions about a set of files. document-check asks it which of six marks a certified copy carries and where. On scripted models (the free run's stand-ins for every model) the agent takes one look at the first file and says its answer is scripted; the run still shows the whole path and writes the report, and prints the report's path: open that file in a browser. document-check commits its two PDFs and lake-mead its thirteen images, a map key and an elevation record, so they run as above. The six other examples fetch their files first: add --fetch-sources (video-chapters and hear-and-find 120 MB each, apollo-descent 211 MB, boulder-chapters 218 MB, market-street 227 MB, fair-robot 342 MB), as in npm run demo:vision -- examples/fair-robot --fetch-sources.

Go live

npm run demo:vision -- examples/document-check --live

The agent runs shell commands with your user's rights on your machine: use a container or a VM. A live run needs:

• GEMINI_API_KEY (aistudio.google.com), for the model that runs the agent and the model that looks;
• TYPESAFE_API_KEY (console.typesafe.ai/keys), for Jev, TypeSafe's judge model that answers yes-or-no, choice, count and box questions; needed unless a gateway (an OpenAI-compatible endpoint that serves every model, named by VIDEO_MODEL_BASE_URL with VIDEO_MODEL_API_KEY) passes TypeSafe through, and then the gateway's key is Jev's;
• FAL_KEY (fal.ai): optional. With it the agent can transcribe the film's speech; without it, it works from the picture alone;
• the same four programs as a free run.

One measured live document-check cost about $0.54 and took about 8 minutes; a run's cost and time vary. A run stops past $1 unless --max-usd <n> sets another cap. examples/vision-agent lists every flag and key.

It runs inside a pi-durable host

pi-vision is a library, not a command line: an extension for a pi-durable host, a process that opens an agent with pi-durable, chord and pi-ai and installs this package's extension. examples/reference is such a host, and what a host supplies says what each package asks of one. There is no pi coding-agent extension: pi install npm:@parcha/pi-vision@beta adds the five skills (find, look, report, transcribe, verify) and no tool, and the vision.see they call is not defined there.

What it does

A file becomes something a model can read or see: text for documents and spreadsheets, the image itself for pictures and scans, frames or a window for video, every image fitted to the model's own limits. What cannot be done is a miss, a failure with a code the model can act on (unreadable, over_budget), and every look a second model answers is filed as a receipt: the SHA-256 of its source, its question, its answer, the model and the price.

The host keeps what is its own: which model looks and its price, where receipts are filed, its rows, path custody. read keeps its name and its arguments (path, offset, limit, char_offset, ask, from_s, to_s, page), so hosts and their stored instructions do not change. No tool is added. For many pages, a long video or many images the agent writes code, in a host's code mode (a short script the host runs for it, with these functions defined): the look skill (skills/look/SKILL.md) shows it how to render pages and cut frames with bash and ask about them with vision.see (a vision model, the eyes, answers), have Jev judge them with typed answers (vision.judge: a choice, a score, a box, or a noul, a yes or no given as the probability of yes), or have a speech-to-text model write what a recording says, word by word with its second (vision.transcribe). Only the answers, where they looked or listened and the price enter the caller's context. For a long recording, the transcribe skill (skills/transcribe/SKILL.md) shows it how to cut the audio into pieces, transcribe each with vision.transcribe, write one timestamped transcript and search it with rg to the second. The find skill (skills/find/SKILL.md) finds every time a phrase is said in a long video, from its transcript, and the second a thing is first seen and its box, coarse to fine: the words narrow the search, Jev scores frames, the eyes name the second, Jev draws the box. The verify skill (skills/verify/SKILL.md) says whether a PDF has each mark a checklist file names, yes or no, with its box on its page. The report skill (skills/report/SKILL.md) ends a run with the agent's own report: one script-free HTML page, report/index.html, that leads with the answer and shows the evidence for every finding (the document page with the mark boxed and a close crop of it, the film moment's frame and a short clip of its window, the photo), and cites every claim to the file's own page, second or region (file.pdf#page=3, film.mp4#t=18,26, #xywh=percent:x,y,w,h) with the look it rests on.

The eight examples are questions and files for that one agent, and each has its own keys, download and cost: document-check (which marks a certified copy carries, and where), video-chapters (a film cut into chapters), hear-and-find (when a line is said and when a thing is first seen), apollo-descent (the landing minute by minute, from the transcript, the photographs and the film), boulder-chapters (a 1930s dam film in chapters), fair-robot (when a robot is first seen in a 1939 film, and where in the frame), lake-mead (when the water of Lake Mead fell the most, and where, from thirteen Landsat images and the lake's elevation record) and market-street (every vehicle that crosses in front of a 1906 cable-car film, boxed and counted by kind).

What it needs

• Node 22.19 or later, and ESM. The package is ESM only: import it. require("@parcha/pi-vision") is not supported.
• A pi-durable host, as above. A host whose read is its own calls handle.read and handle.fit with nodeVisionEnv, as the first read below does; it needs no pi-durable Harness for that.
• Nothing more for documents. A PDF's text, an office document and a spreadsheet read with nothing installed. Images want Photon (@silvia-odwyer/photon-node), video wants ffmpeg, and a look at an image or a scan that the model running the agent cannot see wants a model that sees, and its key.

Status

Published on npm under the beta tag, at the workspace version with this repository's other packages: install it with @beta. Everything this README describes is built; a part that is not throws NotBuiltError, never a half answer.

Known limits

• A recording source a host passes without owns() is readable by every conversation in the run; give it owns() when conversations must not share recordings. (A recording source, in VisionOptions.sources, lets read answer a question about a recording by its handle, such as a browser session's; the host's allow still applies to every read.)

A first read, with no key

A spreadsheet the test kit writes, read the way read reads it for a model: no eyes, no key, nothing optional installed. In an empty directory:

npm install @parcha/pi-vision@beta

read.mjs:

import { writeFile } from "node:fs/promises";
import { createVisionExtension } from "@parcha/pi-vision/durable";
import { nodeVisionEnv } from "@parcha/pi-vision/node";
import { imageSize, xlsxBytes } from "@parcha/pi-vision/testing";

await writeFile("ledger.xlsx", xlsxBytes());
const vision = createVisionExtension({
  evidence: { file: async () => null }, // where looks are filed: nowhere, and a read with no eyes makes none
  scope: () => ({ label: "first-read", deriveDir: "derived" }), // where a document's text is kept for the shell
});
const call = { conversationId: "first-read", callId: "1", model: undefined, signal: undefined };
const result = await vision.read(nodeVisionEnv({ cwd: process.cwd() }), { path: "ledger.xlsx" }, call);

// What the model is given, block by block.
for (const block of result.content) {
  if (block.type === "text") console.log(block.text);
  else { const size = imageSize(Buffer.from(block.data, "base64")); console.log(`[${block.mimeType}, ${size?.width}x${size?.height}]`); }
}

node read.mjs prints:

Cells of ledger.xlsx (1 sheet), read with office-js; the text is saved at derived/ledger.xlsx-5a119bb78efe62bd.txt. Cells only: formatting, charts, images and pivot views are not shown here. A faithful render needs LibreOffice (soffice) in the environment.

Sheet "ledger"
A1=id | B1=amount | C1=doubled | D1=running_total | E1=account_no
A2=1 | B2=1250.5 | C2=2501 {=B2*2} | D2=<no stored value> {=SUM(B$2:B2)} | E2=9007199254740993
A3=2 | B3=99.99 | C3=199.98 {=B3*2} | D3=<no stored value> {=SUM(B$2:B3)}

A cell shows its value, its formula and whether a value was stored; the 16-digit number stays digits. A PDF's text and an office document's read the same way with nothing installed. An image is fitted to the model's limits by Photon (@silvia-odwyer/photon-node) when it is installed, and without it the read is a needs miss that says so. model is the lead's model (the model that runs the agent) as pi-ai describes it, which sets the limits; with none, the defaults hold.

Entries

• @parcha/pi-vision: the host-neutral core. It imports no pi package.
• @parcha/pi-vision/durable: createVisionExtension(options), the pi-durable adapter.
• @parcha/pi-vision/node: the environment port over Node, for a host whose read is its own.
• @parcha/pi-vision/eyes/pi-ai: eyes on a pi-ai model the host builds.
• @parcha/pi-vision/transcribers: speech-to-text models for vision.transcribe: fal, ElevenLabs and OpenAI.
• @parcha/pi-vision/providers/<name>: unpdf, pdfium, office-js, photon, sharp, poppler, ffmpeg, soffice.
• @parcha/pi-vision/testing: fake eyes, fixture helpers and the provider conformance suite, public so a host's composition test uses the same fakes.

Importing an entry loads none of the dependencies: each is loaded the first time it is needed.

Example

The budgets for a route come from the model's own inputLimits (pi-ai's registry), with the host's cap on a request body over them. With @earendil-works/pi-ai installed (npm install @earendil-works/pi-ai@1.3.0; pi 1.4 is not supported yet, and a later release widens the range), in limits.mjs:

import { getModel } from "@earendil-works/pi-ai/compat";
import { citation, resolveLimits } from "@parcha/pi-vision";

const model = getModel("anthropic", "claude-sonnet-5-5"); // pi-ai's catalog entry for the lead's model
const limits = resolveLimits(model, { requestBytes: 20 * 1024 * 1024 });
console.log(JSON.stringify(limits.image));
console.log(limits.requestBytes);
console.log(citation("filing.pdf", { page: { from: 3, to: 3 } }));
console.log(citation("call.mp4", { t: { from: 60, to: 90 } }));

node limits.mjs prints the image limits of a current Claude model, the request cap (the smaller of the model's cap and 20 MiB, less 1 MiB of headroom) and two citations:

{"maxWidth":2000,"maxHeight":2000,"maxBytes":4718592,"jpegQuality":80,"native":true}
19922944
filing.pdf#page=3
call.mp4#t=60,90

Providers

A provider says what it can do and detects, once per environment, whether it can run there. Zero-install providers are always there; programs run in the environment the agent's shell runs in, never bundled.

@earendil-works/pi-durable and @earendil-works/chord (1.0.4 to 1.3.x) and @earendil-works/pi-ai (any version: the host's copy) are optional peers, for the adapter and the pi-ai eyes, and Photon is one for the default image processor. Install the pi ones at one version: npm install @earendil-works/pi-durable@1.3.0 @earendil-works/chord@1.3.0 @earendil-works/pi-ai@1.3.0. sharp is never resolved by this package: a host that wants it passes its own module (sharpImages(sharp)), so its LGPL code stays with the host's install. This package's own tarball carries none of the third-party code above: unpdf, @hyzyla/pdfium, fflate and @xmldom/xmldom are its dependencies, each installed as a package of its own under its own license.

For a host: vision.see in code mode

handle.see(env, call) is vision.see bound to one call, in the shape pi-codemode's globals take (the excerpt below imports it: npm install @earendil-works/pi-codemode@1.3.0, beside the adapter's peers named under Providers). Build it per call, so it reads paths through that call's environment and keeps its answers in that call's memos. This is a host's whole code-mode tool, a pi-durable extension next to vision.extension. tools is the host's own: the look skill's recipes render pages and cut frames with tools.bash, so a sandbox with no shell cannot run them:

import { execFile } from "node:child_process";
import { CodemodeSandbox, type CodemodeTool } from "@earendil-works/pi-codemode";
import type { Extension } from "@earendil-works/pi-durable";
import { callContext, createVisionExtension, piResult } from "@parcha/pi-vision/durable";
import { inheritedEnv } from "@parcha/pi-vision/node";

const vision = createVisionExtension({
  evidence: { file: async () => null }, // where looks are filed
  scope: () => ({ label: "run", deriveDir: "derived" }), // where derived files are kept for the shell
  eyesModel: { provider: "anthropic", modelId: "claude-sonnet-5-5" }, // looked through the host's pi-ai Models
});

// The shell the skill's recipes call as `tools.bash({ command })`: the command's output, or a rejection that carries it.
// Like every program a read starts, it gets `inheritedEnv()` and not the host's whole environment.
const bash: CodemodeTool = {
  name: "bash",
  description: "Run a shell command in the run's directory",
  execute: (args) => new Promise((resolve, reject) => {
    const command = String((args as { command?: unknown }).command);
    execFile("bash", ["-c", command], { cwd: process.cwd(), env: inheritedEnv() }, (error, stdout, stderr) => (error ? reject(new Error(`${stderr}${error.message}`)) : resolve(stdout)));
  }),
};

// The model's script runs in a sandbox that has the shell and `vision.see` for this call.
export const codeMode: Extension = {
  name: "code-mode",
  tools: [{
    name: "code",
    description: "Run a script that may call vision.see",
    parameters: { type: "object", properties: { code: { type: "string" } }, required: ["code"] },
    replay: "safe",
    execute: async ({ code }, api, context) => {
      const call = await callContext(api, context);
      const sandbox = new CodemodeSandbox({ tools: [bash], globals: [vision.see(api.env, call)] });
      try {
        const run = await sandbox.execute(code, { signal: context.abortSignal });
        return piResult({ content: [{ type: "text", text: JSON.stringify(run.ok ? run.value : run.error.message) }] });
      } finally { await sandbox.close(); }
    },
  }],
};
// The host installs both before it opens pi-durable's Harness: registry.install(vision.extension); registry.install(codeMode);

A global is not a tool: the model's tool list does not change, and the sandbox's declarations carry its doc comment and signature (VISION_SEE). Each eyes request is a look, filed and told through onLook as read's are; its answer is a memo before it is told, so a code-mode tool declared replay-safe and cut by a crash pays for each look and tells the host of it once when it runs again. The looks' price reaches the host through onLook, not the code-mode tool's result. A look that cannot be made throws a SeeMiss whose code is the miss code and whose message starts with it. There is no cap on pages, frames or price: a host that wants one wraps execute. The planner's own caps (plan(..., caps), priceLook(..., capUsd)) are also the host's to pass, with none by default. pi loads the skill from the package (pi.skills); a host that is not pi gives its agent skills/look/SKILL.md the way it gives any skill.

For a host: vision.judge, Jev in code mode

handle.judge(env, call) is vision.judge, built per call beside vision.see. It reaches Jev, TypeSafe's System One model, through VisionOptions.jev: the judge createJevRunner from @parcha/agentrun-jev returns, which owns the transport, the images header and the price, and the model to ask (jev-1.14.0 unless jev.model names another; it is the first Jev that answers box questions). pi-vision imports neither that package nor the TypeSafe SDK; it hands the judge each image as PNG or JPEG bytes under a reference written in the request. Install the Jev package with its beta tag (npm install @parcha/agentrun-jev@beta); the judge needs TYPESAFE_API_KEY:

import { createJevRunner } from "@parcha/agentrun-jev";
import { createVisionExtension } from "@parcha/pi-vision/durable";

const vision = createVisionExtension({
  evidence: { file: async () => null },
  scope: () => ({ label: "run", deriveDir: "derived" }),
  eyesModel: { provider: "anthropic", modelId: "claude-sonnet-5-5" },
  jev: { judge: createJevRunner({ apiKey: process.env.TYPESAFE_API_KEY }) },
});
// In the code-mode tool above: globals: [vision.see(api.env, call), vision.judge(api.env, call)]

One call is one Jev request. Every image is fitted to Jev's limits first (PNG or baseline JPEG, turned upright, each side 280 to 1,568 px: grown, or padded when it cannot grow, and shrunk), since Jev rejects an image out of range rather than resizing it. A box answer also has fragment, its region as file#xywh=percent:x,y,w,h of the image it was found in, mapped back through the fit and through the region a crop names. Each judgment is filed and told like a look (via: jev) at the price the judge gives (none known is filed as unknown, never $0), kept as a memo before it is told, and a miss throws a SeeMiss with its code; a box asked of a model that has none is an unsupported miss before any request. With no image at all a request is a text judgment: its text goes in state, it asks choice, noul or score (a box needs an image), and nothing but the state and the questions goes to Jev; it is filed and told like any other look, as a look of text, and its key follows the state, so another claim or other evidence is another judgment. Without jev, every judgment is a no_vision_route miss.

For a host: vision.transcribe, speech to text in code mode

handle.transcribe(env, call) is vision.transcribe, built per call beside vision.see: one audio or video file in, the words with their start and end in seconds out, in one shape whatever the provider gave. The host picks the provider in VisionOptions.transcriber and holds the key, the way the eyes hold theirs: each transcriber takes an apiKey function read at each request, and nothing the agent's code, the answers, the receipts or the misses see carries it.

Each transcriber has its own default model, since the providers name them differently; model names another. Never words without times: a model that answers with text alone is an unsupported miss that says so. OpenAI's gpt-4o-transcribe and gpt-4o-mini-transcribe give no word times, so at OpenAI's own address any model but whisper-1 is that miss before any request; any other model that answers without word times is that miss at its price.

A gateway that passes fal's queue through with its own bearer key is falTranscriber({ apiKey, queueUrl, auth: "bearer" }); fal's own addresses in its answers are rewritten to queueUrl.

import { createVisionExtension } from "@parcha/pi-vision/durable";
import { falTranscriber } from "@parcha/pi-vision/transcribers";

const vision = createVisionExtension({
  evidence: { file: async () => null },
  scope: () => ({ label: "run", deriveDir: "derived" }),
  eyesModel: { provider: "anthropic", modelId: "claude-sonnet-5-5" },
  transcriber: falTranscriber({ apiKey: () => process.env.FAL_KEY }),
});
// In the code-mode tool above: globals: [vision.see(api.env, call), vision.transcribe(api.env, call)]

A file over the provider's limit is an over_budget miss that names it, before any request; the agent cuts the file into pieces and transcribes each (the transcribe skill shows how). Each transcript is filed and told in the shape of a look, as tool transcribe (args the path, model and via the provider; kind audio or video; the words as rows of start, end and words in the receipt's body), at its list price per audio minute, the length from the provider or from ffprobe in the environment (none known is filed as unknown, never $0), and kept as a memo before it is told, so a file asked again, or after a cut call, costs nothing. usdPerMinute on a transcriber sets the price where the package names none. Without transcriber, every transcription is a no_vision_route miss.

For a host: what a program is started with

A read may start a program (poppler, ffmpeg, LibreOffice) on a file the host did not write. Such a program is never given the host process's whole environment: a host keeps credentials there, and a child's environment can be read by other processes of its user. By default a program gets only what inheritedEnv() of ./node copies from process.env: PATH, HOME, TMPDIR, TMP, TEMP, LANG, LC_*, TZ, the loader's library paths (LD_LIBRARY_PATH, DYLD_LIBRARY_PATH, DYLD_FALLBACK_LIBRARY_PATH) and, on Windows, SystemRoot, windir, PATHEXT, ComSpec and USERPROFILE.

LibreOffice converts a document in a profile of its own with macros, active content, linked content and link updates off; where the system allows unprivileged network namespaces (many Linux distributions; not Ubuntu 24.04 by default, not macOS, not a default Docker container) it also converts with no network, and elsewhere it converts with the network on, so run it in a container without network if your documents are untrusted.

In programs.mjs, which asks two environments whether a program started in them finds a name:

import { visionContext } from "@parcha/pi-vision";
import { inheritedEnv, nodeVisionEnv } from "@parcha/pi-vision/node";

const cwd = process.cwd();
const standard = nodeVisionEnv({ cwd }); // a program gets inheritedEnv() and nothing else
const withFonts = nodeVisionEnv({ cwd, env: { ...inheritedEnv(), FONTCONFIG_PATH: "/etc/fonts" } }); // one variable more

// Whether a program started in `env` finds `name` in its environment.
const sees = async (env, name) => (await env.exec(["sh", "-c", `test -n "$${name}"`], undefined, visionContext())).value.exitCode === 0;
for (const [label, env] of [["standard", standard], ["with fonts", withFonts]]) {
  console.log(`${label}: MY_SECRET ${await sees(env, "MY_SECRET")}, FONTCONFIG_PATH ${await sees(env, "FONTCONFIG_PATH")}`);
}

MY_SECRET=hunter2 node programs.mjs prints:

standard: MY_SECRET false, FONTCONFIG_PATH false
with fonts: MY_SECRET false, FONTCONFIG_PATH true

The adapter takes the same option. It holds for every program a read starts through it: handle.pdfText, the wrapped read on the conversation's pi environment, and handle.read on the environment a host hands it:

import { createVisionExtension } from "@parcha/pi-vision/durable";
import { inheritedEnv } from "@parcha/pi-vision/node";

export const vision = createVisionExtension({
  evidence: { file: async () => null },
  scope: () => ({ label: "run", deriveDir: "derived" }),
  env: { ...inheritedEnv(), FONTCONFIG_PATH: "/etc/fonts" },
});

env is the whole environment of those programs, not an addition to the default: a host that passes { PATH: "/usr/bin" } gives them that and nothing else. A call's own variables (a provider's) are added over it. pi's environments give a program all of process.env unless a call says not to, so the adapter says so on each call (inheritEnv: false) and passes this environment; nodeVisionEnv honours the same switch. So through the adapter, the adapter's env decides, whatever the environment a call runs in was built with: set it there. An environment's own env holds when a host uses it without the adapter. A host whose pi environment runs programs on another machine passes env for that machine, since the default is copied from this process.

Develop

npm run build -w @parcha/pi-vision
npm test -w @parcha/pi-vision                 # everything but the suites that need programs
npm run test:binaries -w @parcha/pi-vision    # test/binaries: needs ffmpeg and poppler; a missing one fails
npm run typecheck -w @parcha/pi-vision        # includes the type tests in test/types
npm run verify:tarball -w @parcha/pi-vision   # packs, installs and imports every entry (needs the registry)
npm run bench -w @parcha/pi-vision            # the free bench on synthetic files

Tests import the built package, so build before running one file with node --test. The package's tests build their fixtures, apart from one 20-second public-domain clip (ASSETS.md), and commit no document. (The examples do commit public inputs: document-check's pages of a certified copy from a public SEC filing.) On a busy machine, TEST_CONCURRENCY=N (or -- --concurrency N after either test script) caps how many test files run at once; unset, Node's default holds.

The live eyes check, two paid looks through pi-ai (about a cent) and never part of npm test or CI, runs from the repository root after a build: PI_VISION_LIVE_EYES=<provider>/<model id> node packages/pi-vision/test/live/eyes.mjs, with the provider's key in the variable pi-ai reads and PI_VISION_LIVE_BASE_URL set when a gateway serves the model.

License

Apache-2.0. See LICENSE and NOTICE.