pi-vision-fallback

Pi extension that keeps your text-only coding model in the driver's seat while still letting it "see".

When a prompt or tool result contains images but the active model has no image input modality, this extension asks a configured vision model to describe each image and hands the text model a description (while keeping the image block so the TUI still shows it). The session model, transcript, and thinking level are never changed.

Install

pi install npm:pi-vision-fallback
# or from a git checkout:
pi install git:github.com/mforce/pi-vision-fallback

Configure

One-time, permanent — set it from the pi TUI:

/vision-model                       # opens a picker of all image-capable models
/vision-model openai-codex/gpt-5    # or pass "provider/modelId" directly
/vision-model clear                 # unset

This writes to the project config <cwd>/.pi/vision-fallback.json (per-repo), falling back to the global ~/.pi/agent/vision-fallback.json when no project dir applies. On read, the project value wins over the global, which wins over env vars — so you can set a global default and override it per repository.

The picker lists the image-capable models from your configured providers (the same set /model shows), sorted by provider. It's paged (PgUp/PgDn, Home/End), type-to-filter (e.g. start typing gpt), and marks ★ configured (the current setting, pre-selected) and ✓ active (your session model). Descriptions show the model name and context window.

Environment variables also work and are checked after the config file:

export PI_VISION_FALLBACK_MODEL="openai/gpt-4o"     # "provider/modelId" (or just "modelId" for the active model's provider)
export PI_VISION_FALLBACK_PROMPT="..."              # optional custom description prompt

(Legacy names VISION_FALLBACK_MODEL / VISION_FALLBACK_PROMPT still work.)

Without a configured model — or when the active model already supports images — the extension is inert.

The vision model's provider must be authenticated (API key or OAuth) like any other pi model.

Behavior

• Active model supports images → no-op.
• Active model is text-only and images are present → each image is sent alone to the vision model with a description prompt. The image block is kept (the TUI still renders it; pi's provider layer strips image blocks for text-only models) and a text description is added right before it for the parent model.
• Prompt-injection guard — each description is framed as UNTRUSTED DATA (the image itself may contain embedded instructions), so the text model treats it as content to analyze, never as commands to follow.
• Caching — a description is keyed by model + prompt + image bytes (in-memory LRU + ~/.pi/agent/vision-fallback-cache.json, 24h TTL). The same image isn't re-described across turns or restarts.
• Graceful failure — if the vision call fails, the image is swapped for a placeholder so the parent model moves on instead of retry-looping a dead API.
• A one-time notice is shown if the extension is needed but not configured, or if the vision provider has no credentials.

Limitations

• The main model receives a text description, not the raw image; fine-grained visual details can be lost.
• Uses pi's own stream dispatch (streamSimple), so any builtin provider API works.
• Keeping the original image block (so the TUI still shows it) relies on pi's provider layer stripping image blocks for text-only models. This extension only runs when the active model has no image modality.
• The cache lives under ~/.pi/agent/; delete vision-fallback-cache.json to force re-description.

Development

npm run check    # typecheck
npm test         # unit tests

Related

Filed against pi core as an enhancement: https://github.com/earendil-works/pi/issues/8815 (request-level routing to a visionModel setting would make this extension unnecessary).