@sreetej510/pi-shipd-checks

A pi coding agent extension for benchmark-test analysis, prompt humanization, and Shipd quality checks.

What it does

For /checks --solve:

1. Snapshots only the current git HEAD into a throwaway temp directory; the source checkout is never modified.
2. Sends the exact contents of agent_prompt.md as each solver's sole first user message. No prompt file, task Dockerfile, test.patch, solution.patch, or uncommitted test files are overlaid into a solver workspace. The Dockerfile is used only by the harness to provision the Fargate environment.
3. Runs several configurable (default 3) independent Claude Code CLI solver invocations in parallel, each with its own isolated clean-HEAD workspace and only the Bash tool enabled. Captures each solver's stream-json trace of assistant and tool events (not private chain-of-thought). There is no test-driven repair loop, and the test patch remains unavailable until that solver exits.
4. After a solver finishes, the harness applies test.patch to its workspace, then runs bash test.sh --output_path <file> base and bash test.sh --output_path <file> new as an unprivileged user with a fresh writable HOME and cache. It captures both full logs.
5. After all solvers exit and their tests run, a read-only local Claude Code CLI reviewer (Read, Grep, Glob only; your local claude login, same runner as the auditor tool) receives an untouched clean-HEAD tree plus agent_prompt.md, all patch files, standalone copies of new test files, each solver's patch, and every solver's test logs. No patch is applied in the review workspace.
6. Runs the Claude CLI solvers inside one AWS ECS Fargate Spot task. A Spot interruption retries the task and resumes completed solver indexes. A local claude setup-token file is transferred through a private, server-side-encrypted S3 object and supplied to the Claude CLI through CLAUDE_CODE_OAUTH_TOKEN. A root-owned shell wrapper strips credential variables and drops every solver Bash command to its own unprivileged UID; the token is not available to those commands. The temporary token and run objects are deleted after the task stops.
7. Posts a chat summary and logs results in .pi/shipd-checks/<runId>/, including agent_steps.jsonl per solver, and shipd_report.json in your project root.

The agent-callable auditor tool replaces the old gap-finder and solution-precheck tools. It is a local replica of the hosted Auto Review, ported from the shipd-workflow rulebook, for the tests and the solution only (no description grading; test.sh is ignored). It takes no parameters:

1. Runs create_patches.sh when the project has one, to refresh test.patch and solution.patch from the working tree, then requires agent_prompt.md, test.patch and solution.patch.
2. Snapshots git HEAD into a temp workspace as the pinned base (repo/), next to review_assets/ holding agent_prompt.md (as problem.md), test.patch with its test.sh section removed, solution.patch, and standalone copies of new test files. Neither patch is applied.
3. Runs read-only Claude Code CLI reviewers on this machine (Read, Grep, Glob only; your local claude login), several at a time, one read each: Local Test Quality per-file fairness batches (one per ~60 KB of test files), one coverage read, the solution audit over ten lenses (paths, identity, ordering, variants, numeric, compose, metadata, lifecycle, behaviour, and the quality lens for solution betterment: repo conventions, dead code, duplication, unrelated changes, AI artefacts), and per-unit Solution Quality on every solution.patch hunk (12 hunks per reviewer). The embedded rules.md families the old tools used go into these prompts: fairness rules into per-test Test Quality, test-gap rules into coverage, solution-gap rules into the audit and per-unit reads. Prompts also carry hosted calibration distilled into general patterns (src/auditor-data.ts), not quotes from past tasks. A typical task needs about 6–8 sessions.
4. Runs synthesis (one session per 50 findings) over every finding with the hosted proof legs (failing case, wrong implementation, promise citation, load-bearing check, provenance, materiality, reference pass, exercised environment). A finding gates only when synthesis keeps it HIGH; findings synthesis rejects as wrong are dropped.
5. Returns the findings grouped as details.unfairTests, details.testGaps, details.solutionGaps and details.solutionImprovements, each with level (HIGH gates, MEDIUM is the fix-before-upload list), criterion, finding and the synthesis reasoning, plus details.verdict (BLOCK, REVIEW, PASS or INCOMPLETE), details.testQuality, details.solutionQuality, details.counterTests (not executed) and reviewer statuses. It writes only one file, the report .pi/shipd-checks/<runId>-auditor.md; reviewer prompts and answers are not kept.

The opt-in humanize-agent-prompt tool reads the complete current-project agent_prompt.md, requests 3 experimental rewrite drafts from the CoreWeave Nemotron LoRA, scores every draft through a configured HTTPS detector endpoint, and returns the top 2 as JSON (defaults are configurable). It never edits files: the local agent must review every source requirement before applying a draft. Detector scores are exploratory triage, not confidence, authorship, semantic-fidelity, or correctness evidence. The adapter is experimental and was trained on all 519 pairs, including records originally assigned to EVAL and HOLDOUT; treat all outputs as unapproved drafts.

The agent-callable quality-check tool runs create_patches.sh in the current working directory and reads agent_prompt.md, test.patch, and solution.patch. First, on every run (including a cached prompt pass), it opens the saved job link and syncs the Shipd draft: it fills only the fields whose saved content differs from the local files, waits until Shipd's Convex backend acknowledges every save and the header shows Saved, then reloads and confirms the draft matches (up to three attempts). Each page load waits until every quality row's status has loaded, since a row without its status would read as a current report. Only then does it compute a SHA-256 of the exact prompt content. The last passing prompt hash is stored as a custom entry in the current chat session. If the hash matches, it skips the Shipd Prompt Quality check entirely and reports a cached PASS; if it differs or is absent, it checks the refreshed status of the synced draft and runs only when the check is stale or has a Run button. Otherwise it uses the current report. A new hash is saved only after Prompt Quality passes. On failure, only promptQuality is returned; there are no Test/Solution results or stoppedReason. Only after Prompt Quality passes or has a matching cached pass does it run the Fargate patch precheck against a clean HEAD; only if that passes does it run Test Quality, and only if Test Quality passes does it run Solution Quality. A failing Test Quality (any unfair test or coverage suggestion) returns promptQuality and testQuality without running Solution Quality. Setting Test and Solution Quality to parallel in /checks --config starts both together after the precheck and always returns both reports, even when Test Quality fails. Each check runs fresh jobs, reruns stale reports, skips current reports, and waits for started jobs. It returns only the useful report data: details.testQuality.verdict, details.testQuality.coverageSuggestions, details.testQuality.unfairnessCount, and details.testQuality.tests filtered to blocks whose concerns array is non-empty or whose fairness is exactly "Not fair", details.promptQuality and its verdict, reason, and major non-AI comments (only category, quote, and suggestion; AI-category comments are omitted), plus details.solutionQuality.verdict and its evaluation fields with level and overall_feedback removed. Fairness-only blocks with an empty concerns array are marked Not fully fair. Test Quality is PASS only when there are no filtered unfair tests, reported unfairTestCount is zero (when present), and coverage suggestions are empty. Prompt Quality is PASS only when the completed report and evaluation verdicts are both PASS, requirementCount is not blocking, and the AI gate does not flag the prompt. The prompt is flagged as AI when a Prompt Quality comment category identifies AI (for example, ai_generated). AI-category quote and suggestion details are omitted; guidance recommends humanizer for a broader rewrite or a small targeted manual edit for a recent change. A cached Prompt Quality pass is marked cached: true. Test and Solution Quality return compact PASS results; failures include only actionable feedback. Prompt Quality failures return a concise reason or major comments. Returned quality results omit completed and skipped. Its compact UI is labeled Quality Checks, shows live elapsed time, and summarizes each stage; it does not print full pass criteria in the tool description. It takes no parameters. Transient status and report-dialog operations retry up to three times; after clicking Run, the tool verifies the stale state clears or the job shows running before it begins polling. It uses one fresh headless browser tab at a time: for Prompt Quality it checks after 2 minutes, then every 30 seconds; for Test and Solution Quality it checks after 5 minutes, then reopens every 90 seconds until those jobs finish, with no client-side overall timeout. If all checks are current, it uses existing reports without starting a job. Each browser context blocks images, fonts, and media to reduce memory use. It does not click the final orange challenge-submit button. Authentication comes from SHIPD_STORAGE_STATE or the saved state created by /shipd:auth. /shipd:auth launches the installed browser as a normal visible process and attaches over local CDP, which supports Google/SSO sign-in better than an automated browser launch.

The auditor tool is read-only apart from its report and create_patches.sh, takes a long time, and consumes Claude usage. Invoke it only when the user asks, never in parallel, and run repeated requests sequentially after applying each result. The prompt humanizer also requires an explicit user request and sends the prompt to CoreWeave and the configured detector endpoint. quality-check consumes Shipd tokens; do not run overlapping invocations for the same challenge.

Commands

/checks --solve runs the benchmark using agent_prompt.md; it requires create_patches.sh, that prompt file, Dockerfile, and test.patch (which supplies test.sh and the test files). Like the quality-check tool, it first runs create_patches.sh in the working directory to regenerate test.patch and solution.patch, and stops if the script fails. The solver sees only clean HEAD plus the prompt text; the harness applies test.patch after the solver exits. solution.patch is not required by the solver; when present, the local reviewer receives it as our reference solution: it stays context only for the fairness review, and the reviewer also compares its non-test code logic with every solver's patch and lists, under "Suggested improvements to our solution", edge cases or logical errors a solver handles correctly that ours does not (no tests, docs, or style). /checks --config is separate and configures the Claude solver, reviewer and auditor settings. The auditor and prompt-humanizer tools are enabled per project with /analyze:on.

Shortcut: Ctrl+Shift+X cancels an in-progress /checks run, stopping its active Fargate task or local Claude review. A cancelled run skips final report writing.

Configuration

/checks --config opens the row-based settings menu with three sections:

• Solver: Solver model (Claude: haiku, sonnet, or opus), solver effort (low, medium, high, xhigh, or max), one-shot timeout (10–90 minutes), parallel agent count, and the Claude reviewer model (default opus), effort (default high), and timeout (10–60 minutes; default 20), stored under solverReview in checks-config.json. Solver effort defaults to low when unset and is passed explicitly to Claude. Diffs, agent output, agent_steps.jsonl traces, base/new logs, and the local reviewer assessment are saved under .pi/shipd-checks/<runId>/ and summarized in shipd_report.json.
• Quality Checks: whether the quality-check tool runs Test and Solution Quality sequential (default; Solution Quality runs only after Test Quality passes) or parallel (both start together and both reports are returned), stored under qualityCheck in checks-config.json. The setting is read after the patch precheck on each run, so a change applies without restarting pi.
• Analysis Tools: whether the analysis tools are enabled for this project, and the auditor tool's Claude settings: model (haiku, sonnet, or opus; default opus), effort (default high), timeout per reviewer (10–90 minutes; default 40), and parallel reviewers (1–8; default 4), stored under auditor in checks-config.json (an autoReview block from earlier versions is still read). Fargate resources are selected automatically and are not configurable in this menu.

For each Claude solver, shipd_report.json records CLI-reported input/cache/output token totals, final and peak request-context token counts, context-window size/utilization, API-call count, and cost when available. When the CLI explicitly reports thinking tokens, totalTokens subtracts them from raw input-plus-output; otherwise it remains the raw total. The raw total is also recorded when a subtraction is made. The local Claude reviewer records the CLI-reported usage and cost. Per-agent, reviewer, and aggregate usage appears once under solverTokenUsage; solverRunSummary contains outcomes, aggregate test counts, and per-suite JUnit counts, while solverReview contains the assessment and reviewer model settings.

The humanizer candidate counts are configured separately in ~/.pi/agent/checks-config.json. candidateCount controls how many drafts are requested from CoreWeave (1–10); topCount controls the maximum returned (1–10, capped at candidateCount). Only drafts below the detector's reported Human-like cutoff are returned; if none pass, candidates is empty:

{
  "promptHumanizer": {
    "candidateCount": 3,
    "topCount": 2
  }
}

The detector runs as one on-demand FARGATE_SPOT task per humanizer invocation. The generated drafts are sent as one batch through short-lived, pre-signed S3 URLs; the task exits after scoring and the extension removes the run objects. There is no always-on detector task, API Gateway endpoint, or NLB in this execution path. The current task definition is 1 vCPU/2 GiB; the smoke test returned scores in about 83 seconds overall (the task ran about 27 seconds). The earlier 2-vCPU/4-GiB test took about 68 seconds; cold-start time varies. From the Pi agent directory, deploy the worker image/task definition with bash shipd-humanizer-assets/shipd-detector/deploy_fargate_spot_worker.sh before using the humanizer. The CoreWeave key can come from COREWEAVE_API_KEY, COREWEAVE_API_KEY_FILE, or the local path in promptHumanizer.coreweaveApiKeyFile; only the path, never the secret, is stored in checks-config.json. COREWEAVE_BASE_URL defaults to https://api.inference.wandb.ai/v1; COREWEAVE_PROJECT and NEMOTRON_MODEL default to the handoff's v1 LoRA. Output is intentionally compact: each candidate has rank, text, and a raw triage_score. It is exploratory triage, not a confidence percentage. Candidates that fail the detector's Human-like cutoff are discarded; passing candidates are ranked by ascending detector score, up to topCount. A CoreWeave finish_reason: "length" or reported completion count at the 900-token cap is rejected before scoring. If CoreWeave reports a normal stop despite an incomplete draft, there is no reliable automatic completeness check; compare each draft with the source. Detector scoring processes the full text in windows.

AWS credentials stay local. Configure the AWS CLI profile, then set AWS_PROFILE/AWS_REGION (or add fargate.awsProfile/fargate.region to checks-config.json). The runner discovers the default VPC, public subnets, security group, ECS cluster, and an account-scoped private S3 bucket unless explicit IDs are configured. It uses the FARGATE_SPOT capacity provider only; there is no On-Demand fallback. Spot interruptions are retried according to fargate.maxRetries.

Fargate setup

1. Create a dedicated, least-privilege IAM user in the AWS Console (do not use root), create an access key under Security credentials, and save it locally in %USERPROFILE%\\.aws\\credentials—no AWS CLI is required:

   [shipd-static]
   aws_access_key_id = YOUR_ACCESS_KEY_ID
   aws_secret_access_key = YOUR_SECRET_ACCESS_KEY

   Use "awsProfile": "shipd-static" in the checks config. The AWS SDK reads this file directly. Keep it outside the repository, rotate the key periodically, and never paste it into chat or commit it. The IAM user needs the runtime ECS, S3, EC2-discovery, STS, and iam:PassRole permissions for the configured task/execution roles.

   Alternatively, AWS CLI browser login/SSO profiles work and are automatically refreshed by the SDK; CLI setup is optional.

2. Create an ECS task role with this trust policy (ecs-tasks.amazonaws.com) and an inline S3 policy. Use an explicit bucket name so the policy stays narrow:

   {
     "Version": "2012-10-17",
     "Statement": [{
       "Effect": "Allow",
       "Action": ["s3:GetObject", "s3:PutObject"],
       "Resource": "arn:aws:s3:::BUCKET_NAME/runs/*"
     }]
   }

   The task role is recommended for long-running jobs because temporary presigned URLs can expire while dependencies or agents are running; it is needed if a run may exceed the URL lifetime.

3. Create an ECS execution role trusted by ecs-tasks.amazonaws.com, attach the AWS-managed service-role/AmazonECSTaskExecutionRolePolicy, and create the CloudWatch log group. The execution role is used for awslogs and image startup; the task role is used for S3 data.

4. Add the role ARNs and bucket to ~/.pi/agent/checks-config.json:

   {
     "fargate": {
       "awsProfile": "shipd-static",
       "region": "us-east-1",
       "bucket": "BUCKET_NAME",
       "taskRoleArn": "arn:aws:iam::ACCOUNT_ID:role/pi-shipd-checks-task",
       "executionRoleArn": "arn:aws:iam::ACCOUNT_ID:role/pi-shipd-checks-execution",
       "logGroup": "/aws/ecs/pi-shipd-checks",
       "resourceProfile": "medium",
       "adaptiveResourceProfile": true,
       "maxRetries": 1
     }
   }

   cluster, subnetIds, and securityGroupId are optional when a default VPC is available. With adaptiveResourceProfile: true, solver-gap runs use resourceProfile initially and adapt on later runs (unless projectProfiles already overrides the repository). They upgrade when normalized CPU is at least 95% for more than seven minutes, downgrade when CPU is at least 95% for under two minutes, and otherwise retain the profile. Only the selected next profile is written to projectProfiles; CPU telemetry is included in shipd_report.json for Claude benchmark runs, but no telemetry history is retained. Quality patch prechecks always use the medium profile and do not update adaptive sizing. Set projectProfiles to override solver-gap resources per repository: {"C:/path/to/repo":"large"}.

5. Generate a CI token with claude setup-token and save only the token in a local secret file. Set the top-level claudeOAuthTokenFile in checks-config.json to that file's path (or set CLAUDE_CODE_OAUTH_TOKEN_FILE). The runner transfers the token through a private, server-side-encrypted S3 object, does not place it in the ECS task definition or bootstrap JSON, and deletes the local temporary copy and run object after the task stops. Do not commit the token or put it in Pi's auth.json. Claude setup tokens are long-lived and must be rotated when they expire. Configure the Solver model (Claude), thinking/effort level, count, and timeout in /checks --config, then run /checks --solve. Projects need Dockerfile, agent_prompt.md, and test.patch, which should add test.sh with --output_path <file> base and --output_path <file> new modes. A task role is required for runs that may exceed the 55-minute pre-signed URL lifetime; configure it before using longer solver timeouts.

Use /analyze:on and /analyze:off to control the auditor and prompt-humanizer tools per project, like HPC. The enabled project list is stored alongside the other settings in ~/.pi/agent/checks-config.json.

The patch precheck always uses the medium Fargate profile. It applies test.patch, creates isolated workspaces with and without solution.patch, and runs the base suites concurrently in both workspaces followed by the new suites concurrently in both workspaces. It requires ./test.sh base to pass, requires every ./test.sh new testcase to fail or error individually (with no suite-level error) before solution.patch, then requires both base and new suites to pass after the solution patch. Pytest subtest totals are not treated as passing testcase records; emitted <testcase> statuses and explicit SUBPASSED records are used for the pre-solution invariant. A failed precheck aborts before Shipd is opened and includes the Linux platform, a human-readable phase, and failed/errored test names in the tool error.

The auditor tool returns review findings, not edits; the caller changes the tests or solution. Invoke them only when the user asks, never in parallel, and run repeated requests sequentially after applying each result. The humanizer does not modify agent_prompt.md; validate all requirements and apply any selected draft locally.

{
  "enabledProjects": ["/path/to/project"]
}

Use ↑/↓ to select a row and Enter/Space to change it. Settings are saved to ~/.pi/agent/checks-config.json; solver settings are nested under solverGap, the solve reviewer under solverReview, the quality-check mode under qualityCheck, and the auditor under auditor.

Install

npm install -g @sreetej510/pi-shipd-checks

Then add it to your pi settings.json:

{
  "packages": ["npm:@sreetej510/pi-shipd-checks"]
}

Or, for local development, point at the entry point directly:

{
  "extensions": ["/absolute/path/to/pi-extensions/extensions/pi-shipd-checks/src/index.ts"]
}

File layout

To change reviewer strictness or wording, edit prompts.ts. To change the rubric/fairness text itself, edit rubric.ts. Agent result-submission tools are defined in tools.ts and wired up in agents.ts; direct extension tools are registered from their feature modules and synchronized in tool-sync.ts.

Development

npm install
npm run --workspace @sreetej510/pi-shipd-checks check     # biome + typecheck
npm run --workspace @sreetej510/pi-shipd-checks format