pi-canary

A pi extension that silently verifies the agent's context awareness on every turn using hidden canary tokens.

Before answering your message, the agent must locate and return N canary tokens distributed across the conversation history. The entire verification exchange is invisible: no tokens in the thinking block, no tokens in the visible response, nothing in the history the agent uses to answer you. If the model can't find them, you get a warning — an early signal that its context is degrading (overflow, truncation, or a broken chat template) before it starts giving you confidently wrong answers.

Install

From npm:

pi install npm:pi-canary

Or from git:

pi install git:github.com/sebaxzero/pi-canary.git

Add -l to either form to install project-locally (adds to .pi/settings.json only).

How it works

Every time you send a message, the extension runs a hidden two-phase exchange before your question is answered:

Phase 1 — Verify

• N random 32-character canary tokens are generated (or reused if VARIANT=fixed).
• They are injected at the configured positions across the conversation history. The last token also carries a verification instruction.
• Your original question is temporarily suppressed.
• The agent is asked only to return the N tokens by name.
• The response is captured and checked. The exchange is hidden from the TUI.

Phase 2 — Respond

• The tokens and the verification exchange are stripped from context entirely.
• Your original question is restored.
• The agent answers normally, with no canary tokens anywhere in its view.

If verification fails, a warning notification appears in the TUI. A failure means the model could not recall content from its own context — a sign of degradation (context overflow, truncation, or a broken chat template). Consider compacting, or set FAIL_COMPACT to automate it. The turn still proceeds either way.

Commands

/canary                    — show current phase, failure count, and config
/canary set KEY=VAL        — override config for the current session only
/canary set KEY=VAL KEY=VAL ...
/canary save               — write the current config to canary.json

Example: /canary set COUNT=5 POSITION=equidistant VARIANT=variant

Configuration

Persistent configuration lives in extensions/canary.json next to the installed extension (auto-created on first load with defaults). You can ask the agent to edit it, or tune values live with /canary set.

{
  "COUNT": 3,
  "POSITION": "end",
  "VARIANT": "fixed",
  "FAIL_COMPACT": 0
}

Key	Default	Description
COUNT	3	Number of canary tokens injected per turn (0 disables the canary check entirely)
POSITION	end	Where tokens are injected: start, equidistant, or end
VARIANT	fixed	fixed = same tokens every turn (preserves KV cache); variant = new tokens each turn
FAIL_COMPACT	0	Compact context after N consecutive failures (0 = disabled)

POSITION=end + VARIANT=fixed (the defaults) is the cache-friendly mode for local model servers: the message prefix never changes and the injected suffix is always the same tokens, so the KV cache stays warm after the first turn. Use POSITION=equidistant + VARIANT=variant for maximum coverage at the cost of cache invalidation every turn.

Compatibility

Works alongside pi-loop-police. When loop-police aborts a turn, the canary check yields gracefully and does not fire its own recovery.

License

MIT