pi-model-dynamic-router

Pi extension that routes model group names to concrete provider/model pairs. Auto-discovers models and pricing. Balances intelligence (GDPval), cost, and availability.

Fork of a-canary/pi-model-router — adds content-based dynamic routing (prompt classification → model group) on top of the upstream's price/quality/availability routing.

Architecture

The router uses a modular architecture with the following components:

Module	Purpose	Key Features
providers.ts	Provider definitions and mappings	25 supported providers, authentication patterns
types.ts	Type definitions	Config, Cache, Metrics, RateLimit, Group, Provider types
utils.ts	Utility functions	String manipulation, reference parsing
rate-limit.ts	Rate limit management	Key rotation, backoff, cost multiplier
discovery.ts	Discovery management	API key discovery, model scanning
metrics.ts	Metrics management	GDPval, throughput, latency tracking
cache.ts	Cache management	Persistent caching, versioning
routing.ts	Routing logic	Model selection, filtering, sorting
content-classifier.ts	Content classification	gemma4:12b-mlx primary, gemma2:2b fallback, cloud fallback
escalation.ts	Session escalation	Loop detection, level tracking, session-safe reset

This modular design enables better maintainability, testing, and extensibility.

Install

pi install npm:@anierbeck/pi-model-dynamic-router
# Or symlink for development
ln -s ~/pi-model-dynamic-router ~/.pi/agent/extensions/pi-model-dynamic-router

Then /reload in pi.

How It Works

Dynamic Routing

The dynamic routing feature automatically classifies user prompts and selects the optimal model group based on the task type. It uses Ollama (gemma4:12b-mlx primary, gemma2:2b fallback) for real-time classification and routes to one of the predefined groups: strategic, tactical, operational, scout, or fallback.

Categories for Classification

The system classifies prompts into the following categories:

• code_simple: Simple code changes (1–10 lines, syntax fixes, typos)
• code_complex: Complex code changes (refactoring, debugging, >50 lines)
• design: Architecture, system design, API design
• planning: Project planning, roadmaps, task breakdown
• exploration: Research, unclear requirements, brainstorming
• fallback: Unclear or multiple categories apply

Mapping of Categories to Model Groups

Each category maps to a specific model group:

Category	Model Group	Use Case
code_simple	operational	Simple coding tasks
code_complex	tactical	Complex coding tasks
design	strategic	High-level design decisions
planning	tactical	Project planning and coordination
exploration	scout	Research and exploration
fallback	fallback	Fallback for unclear requests

Dynamic Group

The dynamic group is a special group that uses Ollama (gemma4:12b-mlx primary, gemma2:2b fallback) to classify each prompt in real-time and automatically routes to the most appropriate model group (scout, operational, tactical, or strategic). This enables context-aware model selection without manual intervention.

Requirements for Dynamic Routing:

• Ollama must be installed and running locally.
• The gemma2:2b model must be available in Ollama (ollama pull gemma2:2b).

Auto-Discovery

Auto-Discovery

On startup, the router automatically:

1. Discovers API keys from env vars, ~/.pi/agent/auth.json, pass store, and CLI OAuth files (qwen, gemini)
2. Scans models from Chutes, OpenRouter, and direct provider APIs (Anthropic, OpenAI, Google, Mistral, DeepSeek)
3. Scrapes GDPval scores from Artificial Analysis with hardcoded fallbacks
4. Caches pricing per provider/model from APIs, with OpenRouter backfill for providers without pricing endpoints

All scanning is async and non-blocking.

Group Selection

Each group auto-discovers available models, filters by quality, and selects by billing preference:

Group	Method	Quality Filter	Use For
strategic	best	—	Best model available. Critical decisions.
tactical	tiered	≥75th percentile	Top quality, cost-optimized. Planning.
operational	tiered	≥50th percentile	Good quality, cheapest. Daily coding.
scout	tiered	≥25th percentile	Acceptable quality, cheapest. Exploration.
fallback	tiered	≥0th percentile	Any available. Last resort.
dynamic	dynamic	—	Auto-classifies prompts and routes to the best group.

No curated model lists. Groups draw from all discovered models automatically.

GDPval

GDPval is a composite quality score from Artificial Analysis that combines intelligence, throughput, and cost-efficiency into a single number. Higher = better overall value. The router scrapes scores once on first run and caches them; subsequent startups use the cache. Use /router scan to force a refresh. Hardcoded fallbacks from gdpval_builtin in the config are always loaded as a baseline.

Price Routing — how tiered works

1. Filter — discard any model below the group's GDPval percentile threshold.
2. Sort — rank survivors by billing tier first, then by effective cost within each tier:
   • Tier 0: free models
   • Tier 1: subscription (lowest rate-limit pressure first, then cost)
   • Tier 2: local (Ollama / LM Studio)
   • Tier 3: pay-per-token (ascending effective cost)
3. Select — pick the top-ranked model (cheapest within the preferred billing tier that clears the quality floor).

This means operational always uses the cheapest model that is at least median quality, while strategic always picks the single highest-scoring model regardless of cost.

costMux

After 4 consecutive HTTP 429s from a provider, the router applies a permanent cost multiplier penalty (costMux) to all its models. This pushes the provider to the back of the sorted list without blocking it entirely — useful when a provider is temporarily overloaded but still reachable. The penalty persists for the session and is reset on /router reload.

Rate Limits & Failover

On HTTP 429 the router works through three escalating responses:

1. Key rotation — immediately tries the next API key for the same provider; the exhausted key enters a 1-hour cooldown before rejoining the pool.
2. Model backoff — if all keys for a provider are cooling down, the model enters exponential backoff (1 min → 2 → 4 → … → 90 min cap) and the group falls over to its next-ranked candidate for the current request.
3. costMux penalty — after 4 consecutive 429s, the provider receives a permanent cost multiplier for the session (see costMux above), demoting all its models in future selections.

All three mechanisms are transparent to the user — the session continues with the next available model.

Stream Retry

When a streaming response fails mid-stream (empty body, connection drop, timeout), the group automatically retries with the next ranked candidate without requiring the user to resend the prompt. Soft failures are distinguished from hard errors: a 4xx response is not retried, but an interrupted stream or empty response is.

Configuration

Main Configuration File

router-config.json:

{
  "providers": {
    "anthropic": { "billing": "subscription", "keys": [{ "key": "!pass show api/claude/token" }] },
    "chutes": { "billing": "subscription" },
    "openrouter": { "billing": "pay_per_token" }
  },
  "model_groups": {
    "strategic": { "method": "best" },
    "tactical": { "method": "tiered", "min_gdpval_pct": 75 },
    "scout": { "method": "tiered", "min_gdpval_pct": 25 }
  },
  "model_metrics": {}
}

Groups need no models arrays — everything is auto-discovered.

Adding a Provider

Use the built-in skill: /skill:router-login

Or manually:

1. Set API key via env var, pass, or pi auth <provider>
2. Restart pi — the router discovers keys and scans models automatically

Supported Providers (25)

anthropic, openai, google, openrouter, chutes, mistral, groq, cerebras, xai, zai, huggingface, kimi-coding, minimax, minimax-cn, opencode, opencode-go, vercel-ai-gateway, azure-openai, deepseek, github-copilot, qwen-cli, gemini-cli, antigravity, ollama, lm-studio

Requirements for Dynamic Routing

To use the dynamic group, you need:

• Ollama installed and running locally (ollama serve)
• gemma4:12b-mlx pulled for best classification quality (ollama pull gemma4:12b-mlx)
• gemma2:2b pulled as fallback (ollama pull gemma2:2b) — used automatically if gemma4:12b-mlx fails
• Ollama accessible from your system (default: http://localhost:11434)

If both Ollama models are unavailable, the classifier falls back to cloud models (if configured), and finally to static keyword-based classification (only if allowStaticFallback is enabled) — otherwise the category fallback is returned.

Commands

Command	Description
/router	Overview: providers, groups, selections, rate limits
/router <group>	Detailed view of a group with ranked candidates
/router scan	Re-scan models and GDPval scores
/router reload	Hot-reload config and cache

Tools

Tool	Purpose
set_model_from_group	Switch session to best model from a group
resolve_model_group	Preview what a group would resolve to
update_model_metrics	Manual metric override

Dynamic Routing Tools

The dynamic group uses the following internal tools:

• classifyPrompt: Classifies user prompts into categories (via Ollama).
• getGroupForCategory: Maps categories to model groups.
• setupContentBasedRouting: PI hook for real-time classification and model switching.

Footer

strategic/anthropic/claude-opus-4-6 | int:1450 tps:80 | 12k/8k $1.43 62% | ⏱14m | ⌂ proj | ⎇ main | ⛔2

License

MIT