Pi Vision Bridge — Vision for Text-Only Models

Give text-only models (DeepSeek, Llama, Qwen, local Ollama models) the ability to see images — without switching models mid-task.

English | 中文

---

Your coding agent runs on a text-only model and executes the whole task. When it needs to understand an image — a page screenshot, an error message, a UI mockup — it delegates to a vision model, gets a text description back, and keeps going. The active model never changes.

Features

• describe_image tool — the agent calls it at any point during a task: pass an image path and an optional question, get a text description back. Works with any vision model pi can talk to.
• Automatic image-to-text fallback — paste or attach an image while your model has no vision; it is described before reaching the model, so the conversation keeps working with no API errors.
• Automatic model discovery + fallback — finds a vision-capable model in your pi registry; if a candidate fails, it transparently tries the next one.
• Dual API support — OpenAI-compatible endpoints (/chat/completions) and Google Generative AI (generateContent) out of the box.
• Proxy-aware — honors the standard HTTPS_PROXY / HTTP_PROXY / PI_VISION_PROXY environment variables via a zero-dependency CONNECT tunnel.
• Zero runtime dependencies — no npm packages, no binary downloads.

How it works

Your agent (text-only, e.g. DeepSeek)
    │  executing a long task
    │  ── needs to see an image ──▶ calls describe_image(path, question)
    │                                    │
    │                                    ▼
    │                        vision model (Gemini / Qwen-VL / GLM / ...)
    │                                    │
    │  ◀── gets text description ────────┘
    │  continues the task, model never switched

Installation

pi install npm:@wuxiangru/pi-vision-bridge

Or install from git:

pi install git:github.com/wuxiangru915/pi-vision-bridge

Or try it without installing:

pi -e git:github.com/wuxiangru915/pi-vision-bridge

Note: pi extensions run with full system access. Review the source before installing.

Configuration

The vision model is resolved from pi's model registry (~/.pi/agent/models.json), so any provider pi can authenticate with works out of the box.

Option A: Automatic discovery (default)

With no explicit config, the extension picks the first available model that accepts images, has configured auth, and scores best on common chat formats — preferring your current provider, then the top candidates with automatic fallback on failure.

Option B: Explicit configuration (recommended)

Pin a specific provider/model with environment variables:

export PI_VISION_PROVIDER=google
export PI_VISION_MODEL=gemini-3-flash-preview

Both must be set; this takes priority over auto-discovery. The provider and model must be defined in ~/.pi/agent/models.json with "input": ["text", "image"]:

{
  "providers": {
    "google": {
      "baseUrl": "https://generativelanguage.googleapis.com/v1beta",
      "api": "google-generative-ai",
      "apiKey": "$GEMINI_API_KEY",
      "models": [
        { "id": "gemini-3-flash-preview", "input": ["text", "image"], "contextWindow": 1000000 }
      ]
    }
  }
}

Supported vision models

Any model pi can authenticate and call:

Provider	Example models	API format
Google Gemini	gemini-3-flash-preview, gemini-2.5-pro	google-generative-ai
Alibaba Qwen	qwen-vl-max, qwen2.5-vl	OpenAI-compatible
Zhipu GLM	glm-4v, glm-4v-plus	OpenAI-compatible
OpenAI	gpt-4o-mini, gpt-4o	OpenAI-compatible
Local	Ollama vision models (llama3.2-vision)	OpenAI-compatible

Proxy (optional)

If your vision model API requires a proxy, set the standard variables (the extension also honors PI_VISION_PROXY):

export HTTPS_PROXY=http://your-proxy:port

Usage

• Agent-driven — when a task involves images, the agent calls describe_image on its own. You can direct it: "Screenshot the page and check whether the design matches the requirements."
• User-driven — paste or attach an image directly in the chat. If your active model is text-only, it is described automatically.

Requirements

• pi v0.83+ (uses ctx.modelRegistry.getApiKeyAndHeaders)
• A vision-capable model in ~/.pi/agent/models.json (or set PI_VISION_PROVIDER / PI_VISION_MODEL)
• Network access to your vision model's API

License

MIT