pi-vision-tools

Lets non-multimodal models analyze images by delegating to a configured vision model. A single describe_image tool + /vision command.

Features

• One tool (describe_image) that sends an image + prompt to a vision-capable model and returns the text result to the calling model
• Calling model controls cost/quality per call: compress (on/off), thinkLevel (off through xhigh), and the prompt itself — no preconfiguration needed
• Fuzzy model selection — configure the vision model by name ("haiku", "qwen vl") or exact provider/id; only models that accept image input are considered
• Always follows the calling model's modality: if the current model already has image input, the tool disables itself; otherwise it's on
• Footer indicator (👁 provider/model) visible when the tool is active and a vision model is configured
• No /reload required: config changes take effect immediately

How it works

calling model  →  describe_image  →  vision model  →  text back to calling model
 (no vision)       (image+prompt)      (sees image)

1. The calling model invokes describe_image with an image and a prompt
2. The tool decodes the image, optionally compresses it with sharp, then calls the configured vision model
3. The vision model's text answer is returned as the tool result

Prefix cache

Model switching already invalidates the prefix cache (different provider or model name is part of the request). The setActiveTools add/remove of describe_image is simply part of the new model's system prompt — there is zero additional cache churn.

The tool list is deterministic per model (vision models never include describe_image, non-vision models always do). Switching to another model and back therefore produces the same prefix as before, and the original cache hit is preserved.

Install

pi install npm:@yandy0725/pi-vision-tools

Or add to ~/.pi/agent/settings.json:

{
  "packages": ["npm:@yandy0725/pi-vision-tools"]
}

Configuration

/vision command

Config is persisted to ~/.pi/agent/vision-tools.json and takes effect immediately — no /reload needed.

Activation is always automatic: describe_image is removed from the tool list when the calling model accepts image input, and enabled otherwise — there is no force switch. Legacy keys (enabled, defaultReasoning) are ignored on load; set your think level again after upgrading if you relied on the old key.

Model selection

The model entry is either an exact provider/id or a fuzzy name. Fuzzy matching scores every available model that accepts image input: exact id > id/full substring > display-name substring > all words present. So haiku, qwen vl, or anthropic haiku all resolve to a concrete model, as long as auth for it is configured. Models without image input are never chosen; when nothing matches, the error lists the available vision models.

/vision status, the /vision config model confirmation, and the footer indicator all show the resolved provider/id, so a fuzzy name always tells you what it picked:

{ "model": "haiku" }

Only model and defaultThinkLevel are read — there is no separate provider setting. To pin one specific provider, put it in the value: "model": "anthropic/claude-haiku-4-5".

Optional: sharp

Install sharp for automatic image compression before sending:

npm install sharp

sharp is optional. Without it, images are sent as-is (no error). Compression downsamples the longest edge to ≤1568px, removes alpha, and converts to JPEG.

Environment variables

Set compress: false on any call to skip compression for pixel-perfect needs (reading coordinates, inspecting tiny UI elements).

Tool reference

describe_image(image_path: string, prompt: string, compress?: boolean, thinkLevel?: string)

Example tool call

{
  "image_path": "/home/user/screenshot.png",
  "prompt": "Describe what you see in this screenshot.",
  "compress": true,
  "thinkLevel": "high"
}

Example prompts

Think levels

Image formats

Supported: PNG, JPEG, GIF, WebP, BMP.

Input can be:

• A file path (/path/to/image.png, ./relative.png, ~-prefixed)
• A data: URL (data:image/png;base64,...)
• Raw base64 (string >100 characters, auto-detected)

License

MIT