pi-model-speed
Measure real TTFT/TPOT in Pi, and automatically step off a model that is consistently slow within the same provider.
Package details
Install pi-model-speed from npm and Pi will load the resources declared by the package manifest.
$ pi install npm:pi-model-speed- Package
pi-model-speed- Version
0.3.0- Published
- Nov 21, 2026
- Downloads
- 343/mo · 50/wk
- Author
- xuqianhui
- License
- MIT
- Types
- extension
- Size
- 42.8 KB
- Dependencies
- 0 dependencies · 1 peer
Pi manifest JSON
{
"extensions": [
"./extensions/index.ts"
]
}Security note
Pi packages can execute code and influence agent behavior. Review the source before installing third-party packages.
README
pi-model-speed
Measure what your model provider actually delivers, and get off a model that is consistently slow — without leaving the provider.
What it does
Sampling. Every model request is timed:
- TTFT —
before_provider_request→ first content delta (reported in the status line and commands), - TPOT — ms per output token, derived from
message_endusage.
Two details that make the numbers trustworthy rather than decorative:
- The anchor is
before_provider_request, notmessage_start.message_startfires after the response headers, so counting from it reports a TTFT of ~0 for every request. - Timings are queued, so overlapping requests (retries, nested calls, compaction) do not clobber each other.
Samples stay in memory for /speed-stats and are appended to ~/.cache/pi-model-speed/speed-log.jsonl, which later sessions use for ranking.
Slow-streak failover. If the current model is slow — TTFT over the threshold or TPOT over the threshold, bucketed by prompt size — for window consecutive turns, the extension switches to the fastest other model of the same provider. It never crosses providers, because that changes pricing and capabilities behind your back.
Guards: once per cooldownMs, at most once per session, skipped while a manual /model is driving, the target must clearly be faster (minImprovementRatio / minImprovementMs) — a 13 s → 12 s "win" is not worth re-billing the prompt — and the target must fit the session's context (see below).
Fast is not the same as usable. A provider often mixes windows: a fast 256k model next to a slow 1M one. Letting speed alone pick the target hands a 300k-token session the 256k model, and the turn dies on a 400 ContextWindowExceededError (inside a Fabric child, the whole run dies). Targets whose declared contextWindow is below the session's current tokens (from getContextUsage() and the largest prompt this session has already sent) plus contextGuard.headroomTokens are skipped, and both the switch notification and /speed say so.
A warm session is expensive to leave. Switching models discards the gateway's prefix cache and re-pays for the whole prompt, so a session that has already run sessionBinding.warmTurns turns (or warmAfterMs of wall time, or one turn whose prompt is past thresholds.largePromptTokens) is bound: its slow thresholds are multiplied by boundThresholdMultiplier, TTFT never triggers a switch below boundMinTtftMs, and the switch log/notification says session bound. A bound session stays on a model that is 20% slower, and still walks away from one that stalls for a minute.
Never inside a Fabric child. A process launched by fabric_exec (it carries PI_FABRIC_LINEAGE / PI_FABRIC_DEPTH) keeps the model its parent assigned: Pi Fabric's model admission reads a mid-run switch as drift. Sampling still runs there, so child timings feed the shared ranking. PI_MODEL_SPEED_ALLOW_CHILD=1 opts back in for debugging.
Install
pi install npm:pi-model-speed
# or
pi install ./packages/model-speed
Restart Pi (or /reload).
Commands
| Command | Effect |
|---|---|
/speed |
Rank same-provider candidates by measured TTFT; runs the configured probe first if one is set |
/speed-stats |
This session's TTFT/TPOT medians per model |
/speed-auto on|off |
Toggle failover for this session |
Configuration
Optional file ~/.pi/agent/model-speed.json:
{
"auto": true,
"providers": ["my-gateway"],
"candidates": ["fast-model", "big-model"],
"thresholds": {
"ttftSmallMs": 12000,
"ttftLargeMs": 25000,
"largePromptTokens": 30000,
"tpotMs": 250
},
"window": 3,
"cooldownMs": 600000,
"sessionBinding": {
"warmTurns": 3,
"warmAfterMs": 600000,
"boundThresholdMultiplier": 3,
"boundMinTtftMs": 60000,
"minImprovementRatio": 1.6,
"minImprovementMs": 5000
},
"contextGuard": {
"enabled": true,
"headroomTokens": 32000
},
"probe": {
"command": ["bash", "/home/me/bin/model-probe.sh"],
"file": "/home/me/.cache/model-probe.json"
}
}
| Field | Default | Meaning |
|---|---|---|
auto |
true |
Enable failover |
providers |
[] |
Providers where switching is allowed; empty = any provider that offers 2+ models |
candidates |
[] |
Model ids or provider/id worth considering; empty = every model of the current provider |
thresholds.ttftSmallMs |
12000 |
Slow TTFT for prompts under largePromptTokens |
thresholds.ttftLargeMs |
25000 |
Slow TTFT for large prompts (long prompts legitimately take longer) |
thresholds.tpotMs |
250 |
Slow output rate (≈4 tokens/s) |
window |
3 |
Consecutive slow turns before switching |
cooldownMs |
600000 |
Do not switch more often than this |
sessionBinding.warmTurns |
3 |
Turns after which the session counts as bound (prefix cache already paid) |
sessionBinding.warmAfterMs |
600000 |
…or this much wall time into the session |
sessionBinding.boundThresholdMultiplier |
3 |
Bound sessions use thresholds × this |
sessionBinding.boundMinTtftMs |
60000 |
Bound sessions never switch on a TTFT below this |
sessionBinding.minImprovementRatio |
1.6 |
Target must be this much faster than the current model |
sessionBinding.minImprovementMs |
5000 |
…and at least this many ms faster |
contextGuard.enabled |
true |
Skip targets whose context window cannot hold the session (PI_MODEL_SPEED_CONTEXT_GUARD=0 turns it off) |
contextGuard.headroomTokens |
32000 |
Room kept after the current prompt: the next reply plus a few turns before the window is re-checked |
probe.command |
— | Optional external probe, run by /speed and never automatically |
probe.file |
— | That probe's output: { "ts": 1700000000, "models": { "<id>": { "ok": true, "ttft": 1.2 } } } (ttft in seconds) |
Environment overrides: PI_MODEL_SPEED=0 (disable), PI_MODEL_SPEED_DIR (state dir), PI_MODEL_SPEED_CONFIG (config path), PI_MODEL_SPEED_SLOW_MS, PI_MODEL_SPEED_SLOW_LARGE_MS, PI_MODEL_SPEED_SLOW_TPOT_MS, PI_MODEL_SPEED_WINDOW, PI_MODEL_SPEED_CONTEXT_GUARD=0.
Choosing a ranking source
Ranking uses, in order: the probe file (an out-of-band measurement, e.g. a cron job), then this extension's own sample history (median TTFT per model from speed-log.jsonl), then "unknown". Unknown models are never picked as a switch target — the extension says "no faster same-provider model is known yet" and leaves you where you are.
The probe is deliberately not part of the package: timing a model means spending tokens at your provider, and every gateway has its own way to do that. If you have one, point probe.command at it.
Notes and limits
- TTFT depends on prompt size, gateway load and caching; the history median is a heuristic, not a benchmark.
- If your provider names the deployment that answered in a header (
x-litellm-model-name/x-litellm-model-id), the sample records it asserved, which is often more useful than the requested id. - The extension never edits model configuration; it only calls
setModelon the live session.