Pi Chocklit — evidence-gated smart routing

An opt-in Pi extension for choosing Arc child models by working fit, not model names, release dates, or a universal “strongest” ranking. It can plan a route or submit one child through Arc's existing subagent tool. It does not switch the parent session's model.

Status: first implementation slice; offline tests only. No live model evaluations, provider probes, deployment, or performance-improvement claims. Native compatibility target: Pi 1.1.0; Node 22.19+. No npm dependencies. MIT-licensed; npm publication is an explicit release step.

What is different from Riverdale's weighted router?

Riverdale PR #74 is prior art, not a dependency. Its strength/currency ratings are useful operator preferences, but not measured task fit. This package does not modify Riverdale, its catalogs, Claude pins, or its Arc dispatch hooks.

The decision pipeline is:

1. Discover: read exact provider/model/effort identities from ARC_SUBAGENT_MODELS.
2. Qualify: intersect that snapshot with an explicit reviewed policy. Check capability, effort, context/output estimates, evidence age, sample size, quality floor, ceilings, and cooldowns.
3. Rank: compare task-specific evidence for the same evaluation suite and thinking level.
4. Explain: expose score components, rejection codes, unknown identities, and input hashes.
5. Optionally submit: recompute the decision, record it, and invoke one subagent using native ctx.executeTool. No retry or alternate model after a launch attempt.

A newly released model—or a small model that unexpectedly performs well—needs no name-table change. Once it appears in the host catalog, has an approved binding/capability review, and passes the task's evaluation contract, it competes normally. Until then it is unreviewed_identity, not silently presumed capable. There is no automatic paid exploration or evidence promotion.

npm publication

The manifest targets public npm publication at @electionbuddy/pi-chocklit; publishing requires an authorized npm account and an unused version. Inspect npm pack --dry-run --ignore-scripts before publishing. Package contents are public once published, regardless of GitHub visibility.

If npm reports EPRIVATE, remove a stale private: true from the publishing checkout. If you publish a .tgz, rebuild and reinspect it afterward: editing package.json does not change an already packed tarball. Use an exact version when configuring the package in Arc. Publication alone does not activate routing, supply model evidence, or rewire a vendored Riverdale consumer.

Load and configure

The operator must deliberately select a policy using an absolute path:

export PI_SMART_ROUTER_POLICY=/absolute/path/to/reviewed-router-policy.json
# Optional: observed unavailability/cooldowns from an approved health source.
export PI_SMART_ROUTER_HEALTH=/absolute/path/to/router-health.json

# Run in an Arc environment that already supplies ARC_SUBAGENT_MODELS and subagent.
pi -e /absolute/path/to/pi-chocklit/extension.mjs

Arc package attachment/tool enablement is a separate operator action. Loading this file locally is not proof of Arc deployment. Do not replace Arc's supplied catalog with invented identities. Standalone Pi without an Arc child catalog can load the extension but cannot resolve a route.

examples/policy.json is a safe, empty scaffold: it intentionally resolves nothing. Define profiles, review each model, then add records shaped like examples/model-evidence.json to models. That example is disabled, expired, fictional data; it must never be represented as measured evidence. examples/health.json illustrates the optional cooldown file.

Policy/catalog/health are reread on every call. Malformed or missing inputs fail; no fallback to host defaults. There is no background watcher, credential access, endpoint registration, parent model mutation, or extra classifier-model call. Use atomic file replacement when updating policy. The three inputs are separately read and hashed, not an atomic cross-file transaction.

Tools

smart_route — explain without launching

{
  "profile": "code-review",
  "inputTokens": 20000,
  "outputTokens": 4000,
  "requiredCapabilities": ["tools"],
  "maxEstimatedCostUsd": 0.25,
  "maxLatencyP95Ms": 60000
}

A profile supplies requirements, an exact thinking level, approved child tools, an evaluation suite revision, and scoring parameters. Caller requirements and ceilings can only tighten those constraints. Optional pin: {"provider": "...", "model": "..."} restricts selection to one exact identity; a pin never bypasses qualification and never falls back to a different model.

The result includes selected (or null), candidates with components, rejections, unreviewed identities, policy/catalog/health SHA-256 hashes, and a decision ID. Arrays are capped at 20 with explicit totals and truncated. Unknown cost/latency is null, never reported as zero. An unresolved plan is a semantic tool error (isError: true).

smart_dispatch — recompute and submit one child

{
  "request": {
    "profile": "code-review",
    "inputTokens": 20000,
    "outputTokens": 4000
  },
  "prompt": "Read src/widget.mjs. Report reproducible correctness defects with line references. Do not edit files or delegate. Stop after one review.",
  "tools": ["read", "grep"]
}

It calls ctx.executeTool('subagent', {agents: [node]}, {signal, onUpdate}) with the selected provider, model, exact thinking_level, and the supplied tool subset. This is not a parallel provider client or a direct invocation of the subagent implementation. The host remains responsible for authorization, approvals, credentials, child isolation, execution timeout, concurrency limits, and final model availability. The profile tool list can restrict what this wrapper requests; it is not a filesystem sandbox or a replacement for host enforcement. The mission receives a fixed text prefix because this Arc host passes it as a Pi CLI argument; this prevents a mission beginning with @file or --option from becoming an automatic file attachment or CLI flag instead of task text. Native usage aggregation remains Pi-owned so child usage is not counted twice.

Dispatch checks that the native nested API and subagent are available. It writes a smart-router-decision custom session entry before launching, with parent tool-call ID, requested tools, decision, and hashes, but no mission text. If recording fails, it does not launch. The native nested execution keeps parent attribution. No decision cache or branch-restored hidden model state is maintained. Pi owns persistence of session entries; this is not an exactly-once external-effect journal or a complete audit service.

Results retain the nested content and semantic error status. submitted only means the nested tool returned without a transport/tool error: the child can be pending, still running, or have failed its task. Inspect Arc's nested/background outcome. A thrown or malformed response returns outcome_unknown; a reported error returns failed_or_blocked. No automatic retries in either case. Reconcile the original call before attempting another launch. Caller cancellation propagates to the nested call; it does not guarantee already-started external effects were undone.

Existing native/extension hooks can modify or block child arguments. The decision records the binding this router requested, not a proven physical provider dispatch. Verify the host's actual child provider/model telemetry before using results as evaluation evidence. Do not infer successful task completion or evaluation scores from exit status, a fulfilled promise, or a model's self-reported confidence.

Scoring and evidence contract

Hard gates run before any scoring. With s successes in n trials:

p = s / n
z = 1.96
quality = (p + z²/(2n) - z × sqrt((p(1-p) + z²/(4n))/n)) / (1 + z²/n)
costUtility = costReferenceUsd / (costReferenceUsd + estimatedCostUsd)
latencyUtility = latencyReferenceMs / (latencyReferenceMs + latencyP95Ms)
score = wQuality × quality + wCost × costUtility + wLatency × latencyUtility

quality is the lower endpoint of the conventional 95% Wilson interval, used as a conservative ranking statistic. It is not a guarantee of reliability or a correction for biased samples, correlated trials, benchmark leakage, or selecting the best of many models. Evaluation sources must use representative held-out tasks, independent grading, a fixed success rubric, and a versioned suite. Review capability and limits for the exact Arc child runtime, not just vendor marketing. A source string is provenance supplied by the operator, not an independently verified signature. The extension does not run or authenticate evaluations.

Weights must sum to 1; quality weight must be positive. The scaffold's 80/10/10 allocation, floors, reference values, and TTL are explicit policy choices, not empirically optimized defaults. Stable reference values avoid scores changing merely because an unrelated candidate appeared. Ties break lexicographically by exact provider/model identity, independent of catalog order.

Each model review covers capabilities, child context/output limits, and pricing, and has an expiration. Each task evaluation declares suite, thinking level, successes/trials, source, and observation time. Expired/future evidence, wrong suite/effort, insufficient trials, and unmet quality floors cannot qualify. Models can have several task evaluations and entirely different rankings across profiles. A review alone without a task evaluation does not qualify a model.

Pricing and latency can be omitted only when their weights are zero and neither profile nor caller sets a corresponding ceiling. Otherwise unknown telemetry fails qualification. Estimated cost is uncached input/output token pricing for one inference. It excludes reasoning billing quirks, cache effects, tool charges, later turns, retries internal to the host, and other provider charges. It is not a child-run spending cap. Token estimates must include expected system/tool/history input; the profile reserve adds headroom. They are not tokenization, context reservation, output limit configuration, or proof the entire child run will fit. The extension does not lower native limits to simulate capacity.

Availability and failover boundary

• ARC_SUBAGENT_MODELS is the host process's eligibility snapshot, not a live health check, account quota guarantee, or capacity reservation. Rereading the environment does not refresh the broker's snapshot by itself. Empty means none; missing/malformed is an error.
• Optional health observations exclude an exact provider/model through retryAfter. States are unavailable and rate_limited; future-dated observations refuse that candidate. Expiry permits reconsideration, not a claim of recovery. Health source strings are reviewed input, not attestation.
• The extension does not scrape logs or guess failure classes from error prose. An approved host health adapter is needed to automate these observations; none is deployed by this package.
• The next eligible candidate may win before submission. There is no automatic post-submission fallback: a failed response does not prove a child performed no writes or other external effects.
• No provider/residency broadening: only identities explicitly enabled in the selected policy and present in the Arc catalog qualify. The host still rechecks live authority. Keep policies outside untrusted repositories, restrict who can edit them, and include only approved provider identities. Arbitrary file-edit permissions are not contained by this extension.

Verification

npm test                    # deterministic tests; native test skips without explicit SDK root
npm run check               # syntax checks
PI_NATIVE_ROOT=/usr/local/lib/node_modules/@earendil-works/pi-coding-agent npm run test:native

The native test uses a fake subagent and no provider traffic. It targets the installed Pi 1.1.0 contract; other versions need their own compatibility run. The test fixture values are synthetic, not a model quality report. There are no runtime dependency downloads; package-lock.json records the dependency-free package.

The installed Pi root SDK import failed during verification because its dependency did not export MCP_TYPESCRIPT_PREAMBLE. The native test therefore imports the actual internal SDK modules, loader, and nested runner, pinned to 1.1.0. Those paths passed; the root SDK defect was not repaired. The extension itself has no Pi root-SDK import and loads by explicit path. Do not assume automatic .mjs directory discovery or untested package-attachment behavior; use the explicit file path above.

Before production adoption: obtain reviewed task evidence and native child limits, exercise the exact Arc package/permission configuration, verify actual dispatch telemetry, integrate a trusted health source, and run bounded, separately authorized live trials. Automated exploration, learned policy updates, parent-session switching, full-run budgets, and retry reconciliation remain outside this first slice.

License

MIT — see LICENSE.