pi-jev-checkpoints

🚦 Jev-powered quality checkpoints for pi: catch ambiguous requests before the agent runs, and flag unverified answers after it finishes.

Jev is TypeSafe's System One model. It does not generate text; it returns typed decisions with calibrated probabilities in a few hundred milliseconds. pi-jev-checkpoints puts that judgement at two points of pi's lifecycle, each as its own extension you can switch on or off. Everything is redacted locally before it leaves the machine, every cited ID is validated against what was sent, and any Jev failure falls open so pi keeps working.

Third-party community package, not affiliated with pi or TypeSafe. checkpoint here means a quality gate in the agent lifecycle, not a model weight snapshot.

✨ What it does

• 🚧 clarity-gate (before the first change) — asks Jev whether the request is clear enough to act on, at the moment the agent is about to make its first edit, write or shell command. Read-only and Q&A turns never pay for it. Ambiguous requests get a short guidance message so the agent asks about the one point that matters instead of guessing; a strongly ambiguous one has its first change blocked until it asks. If ask_user_question is installed, the guidance tells the agent to use it
• 🏁 confidence-flag (after the answer) — asks Jev whether the answer's key claims are backed by this run's tool activity (files read, commands run) or merely recalled. Flags ⚠ partially verified and ✗ hallucination-risk answers, quoting the least reliable line. Answers given without touching a single tool are checked for confident claims about your workspace that were never looked up
• 🎛️ /checkpoints — one console for both: status, session and persisted toggles, config keys, a doctor connectivity probe
• 🔌 Two backends — TypeSafe direct or OpenRouter's Decisions API, auto-detected from your keys; a saved /login openrouter works too
• 🙈 Local redaction — private keys, sk-…, GitHub, Slack, AWS, Google tokens, JWTs, bearer headers, key=value secrets and URL credentials are masked before anything is sent
• 🧾 Cited evidence only — Jev picks fragment and line IDs from the submitted text; it never generates a quote, and unknown IDs count as no evidence
• 🪂 Fails open — no key, timeout, HTTP error or malformed answer means the gate passes and the flag stays silent; print/JSON modes are skipped entirely

📦 Install

Requires pi ≥ 0.85.0 and Node ≥ 22.19.

pi install npm:pi-jev-checkpoints

# or from GitHub
pi install git:github.com/wylu1037/pi-jev-checkpoints

# or try it for a single run without installing
pi -e npm:pi-jev-checkpoints

Provide a Jev key for either backend. With both present, TypeSafe is preferred. Three ways, highest precedence first:

# 1. environment of the shell that starts pi
export TYPESAFE_API_KEY=...        # TypeSafe direct
export OPENROUTER_API_KEY=...      # OpenRouter Decisions API

# 2. config file, from inside pi (writes ~/.pi/agent/jev-checkpoints.json)
/checkpoints set typesafeApiKey ts_...
/checkpoints set openrouterApiKey sk-or-...

# 3. pi's own saved login (OpenRouter only)
/login openrouter

Then check the wiring:

/checkpoints doctor

Each checkpoint is a separate extension file, so you can also turn one off for good with pi config.

🤝 Recommended companion

clarity-gate pairs well with @juicesharp/rpiv-ask-user-question, which gives the agent one tool — ask_user_question — that opens a structured, typed-option dialog instead of asking in prose.

pi install npm:@juicesharp/rpiv-ask-user-question

The two are complementary, not overlapping: rpiv is the asking mechanism (great dialog, but the model decides on its own whether to ask), while clarity-gate is the trigger and enforcement — Jev decides whether a question is needed, and a strong verdict blocks the change until one is asked. When the tool is active in the session, clarity-gate's guidance names it, so a blocked or steered change turns into a structured questionnaire rather than a wall of text. Together you get forced, well-shaped clarification right before the first irreversible change — something neither does alone. It is a strong nudge, not a hard contract: clarity-gate can point the agent at the tool, but Pi has no way to force the agent to call it.

🚧 clarity-gate

You cannot tell in advance whether a turn will be a two-second answer or a twenty-minute refactor, so the gate does not guess: it evaluates on the tool_call hook the first time a turn is about to change something (edit, write, bash, powershell by default — mutatingTools), once per turn. Jev receives the prompt, the prompt split into citable fragments, with includeContext the last six user/assistant turns, what the agent has read and searched so far this turn, and the change it is about to make. It answers four questions: is the request clear enough to make that change (probability), which kind of ambiguity (unclear_goal, missing_parameter, multiple_interpretations, scope_too_broad), which fragment most needs clarification (by ID), and whether the user explicitly said "just do it". A detail the agent already settled by reading the code does not count as ambiguity.

Set trigger to turn for the previous behaviour: evaluate on before_agent_start for every turn, before the agent has read anything, and inject the guidance as a message. blockOnStrongAmbiguity works with both triggers: it evaluates on input, and on a strong verdict shows a dialog before the run starts — refine the request (text returns to the editor), let the agent ask, or run anyway.

Guidance is given once per distinct prompt; resubmitting the same text is read as "go ahead". When the ask_user_question tool is active in the session, the guidance names it, so the question arrives as a structured dialog rather than prose. The guidance shows in the transcript as a ⚑ clarity-gate row and an audit entry records the verdict.

🏁 confidence-flag

Runs after agent_settled, never per tool call, and only when the run used at least one tool. Jev receives the final answer split into lines, the user request and, with includeToolData, the run's tool calls paired with their outputs. It answers: are the key claims supported by that evidence (probability), the evidence level (verified, partially_verified, mostly_inferred, no_evidence), the least reliable line (by ID), and whether the answer already admits something was not verified.

Answers that already hedge are downgraded or left alone, the same evidence is not flagged twice in a row, failed or aborted runs are skipped, and a verdict that arrives after you started a new turn is discarded. This is not a fact checker: it judges consistency between the answer and the evidence the agent actually gathered, and never rewrites the answer.

Because the measure is answer vs. this run's evidence, an answer given straight from the model's knowledge has no evidence to be consistent with. Those turns — the ones where hallucinated file paths, defaults and behaviours are most common — get a different, narrower question instead (recallMode: flag): does the answer assert specific facts about this workspace (what a file contains, what a function here does, the value of a setting, what a command would print in this project) that it could have checked by reading or running something, but did not? General knowledge (a git command, a concept, a snippet written from scratch), suggestions and answers that hedge are left alone. A hit is always ⚠, never ✗ — there is no counter-evidence, only the absence of a check:

⚠ confidence-flag: this answer states facts about the workspace without reading or running anything … Unverified: "<line>"

With annotateStyle: message the model sees the flag on the next turn and is asked to read or run the relevant thing before building on it; that is the recommended setting if you want the agent to self-correct rather than just warn you. recallMode: off restores the skip.

🔧 Options

Optional ~/.pi/agent/jev-checkpoints.json (or a "jevCheckpoints" object in settings.json; a trusted project's .pi/ versions override). Read on every evaluation, so edits apply without a restart. Defaults:

{
  "backend": "auto",
  "typesafeApiKey": "",
  "openrouterApiKey": "",
  "model": "",
  "endpoint": "",
  "timeoutMs": 2500,
  "telemetry": false,
  "clarityGate": {
    "enabled": true,
    "passThreshold": 0.8,
    "clarifyThreshold": 0.4,
    "includeContext": true,
    "blockOnStrongAmbiguity": false,
    "trigger": "first-mutation",
    "mutatingTools": "edit,write,bash,powershell",
    "backend": "inherit",
    "ambiguityTypes": { "unclear_goal": true, "missing_parameter": true, "multiple_interpretations": true, "scope_too_broad": true }
  },
  "confidenceFlag": {
    "enabled": true,
    "trustThreshold": 0.85,
    "flagThreshold": 0.5,
    "includeToolData": true,
    "annotateStyle": "inline",
    "recallMode": "flag",
    "recallThreshold": 0.7,
    "backend": "inherit"
  }
}

Environment variables JEV_CHECKPOINTS_BACKEND, _MODEL, _ENDPOINT, _TIMEOUT_MS, _TELEMETRY win over files, and JEV_CHECKPOINTS_DISABLED=1 turns both checkpoints off. Values of the wrong type or range fall back to the next layer. The config file is plain JSON: keep API keys in the global file (~/.pi/agent/), not in a project .pi/ that gets committed.

🕹️ Commands

Status and doctor output use ● / ○ for on / off, ✓ / ✗ for a working / missing backend, and → for hints. All symbols are single-width text, so they align in any terminal.

Every argument position has Tab completion: type /checkpoints to see the subcommands with a one-line description, then the checkpoint names or all after on/off, the config keys (with type and default) after set/unset/get, the allowed values after an enum or boolean key, and the --save / -l flags.

🔒 Data sent to Jev

• clarity-gate: the prompt (≤ 6 000 chars), its fragments and, with includeContext, the last six user/assistant turns (≤ 800 chars each, no tool output). With the first-mutation trigger also this turn's tool calls so far, with arguments (≤ 400 chars) and outputs (≤ 1 200 chars) inside a 16 000 char budget, and the arguments of the pending change
• confidence-flag: the final answer (≤ 8 000 chars), the user request (≤ 2 000) and, with includeToolData, the last 30 tool calls with arguments (≤ 400 chars) and outputs (≤ 1 200 chars) inside a 16 000 char budget. Recall checks send only the answer and the request

Tool arguments and outputs can carry source code and secrets the redaction patterns do not catch. Disable the extensions for sessions whose contents must not leave the machine: /checkpoints off, or JEV_CHECKPOINTS_DISABLED=1.

🩺 Troubleshooting

1. /checkpoints: is the checkpoint ON, and does the backend line show a key? "NO KEY" means both checkpoints are silently failing open.
2. /checkpoints doctor: a FAILED — http with 401 is a bad key; timeout means raise timeoutMs or check the network.
3. Nothing ever fires? Set telemetry: true and watch for skipped: … warnings after a turn. Print and JSON modes never evaluate, and with the default trigger clarity-gate only runs when a turn is about to edit, write or run a command.
4. Too chatty? Raise passThreshold / lower trustThreshold, switch off an ambiguity type, or set recallMode to off. Too quiet? Lower passThreshold, set trigger to turn, or set annotateStyle to message so the agent is asked to verify.

🛠️ Development

npm install
npm run check   # tsc --noEmit + node --test, no build step
pi -e ./extensions/clarity-gate.ts -e ./extensions/confidence-flag.ts -e ./extensions/checkpoints.ts

Module layout and the design patterns behind it: docs/architecture.md. Original suite design: docs/pi-jev-plugins-design.md.

License

MIT