pi-tool-stats

A pi extension that tracks how often you actually use your skills, prompt templates, extensions, and built-in tools — so you can prune the dead weight and keep pi light.

Quick Start

1. Install — add the extension to ~/.pi/agent/settings.json:

   {
     "extensions": ["/Users/<you>/code/pi-tool-stats"]
   }
   

   Then run /reload in pi (or restart pi).

2. Use pi normally — the extension silently records every skill load, slash command, tool call, and tool result to ~/.pi/agent/tool-stats.jsonl.

3. Check your usage — after a few sessions, run:

   /tool-stats
   

   You'll get a breakdown of everything you've used (and what you haven't) over the last 30 days, including a compact model/tool snapshot. Pass a number to change the window: /tool-stats 7 for the past week. For the detailed model/tool report, run /tool-stats model-tools 7.

That's it. No config, no setup wizard, no API keys.

What it records

Every event is appended as one JSON line to ~/.pi/agent/tool-stats.jsonl (durable, greppable, survives restarts):

Kind	Detected when
skill	the agent reads a SKILL.md, or you run /skill:<name>
template	you invoke a prompt-template slash command
ext-cmd	you invoke an extension-registered slash command
tool	any LLM tool call, with model attribution when available
tool-result	a tool finishes, recording success/failure and duration

Report

/tool-stats [days]              # overview + prune candidates; default 30 days
/tool-stats model-tools [days]  # detailed model/tool effectiveness
/tool-stats models [days]       # alias for model-tools
/tool-stats tools [days]        # alias for model-tools
/tool-stats mt [days]           # alias for model-tools

Use 0 days for all time.

The overview shows per-resource count + last-used date, sorted by usage, plus a compact MODEL / TOOL SNAPSHOT and a PRUNE CANDIDATES section listing everything that exists but had 0 hits in the window — by cross-referencing your live skill dirs, prompt templates, and extensions against the log.

The detailed model/tool report shows BY MODEL, BY TOOL, FAILURE HOTSPOTS, and a MODEL × TOP TOOLS matrix. ok% means tool-execution success, not proof of task success.

For skills, the report also calls out PROMPT-VISIBLE UNUSED SKILLS: skills present in Pi's <available_skills> prompt block but with no observed skill loads (read of the skill file or /skill:<name>) in the window. This is a prompt-cost signal, not proof that the capability is useless; package skills may also show related extension/tool usage.

Overview (/tool-stats)

Usage over last 30d  (log: 1281 events total, 1281 in window)

--- MODEL / TOOL SNAPSHOT ---
  model                        calls  ok  fail  unk  ok%  top tools         weak spot
  openai-codex/gpt-5.5            28  25     3    0   89%  read, bash, grep  read 2 fail
  zai/glm-5.1                     24  19     2    3   90%  bash, edit, read  bash 2 fail
  anthropic/claude-sonnet-4-6      8   8     0    0  100%  agent_browser     -
  details: /tool-stats model-tools [days]

--- SKILLS ---
  name          used  last
  land          6     2026-05-30
  pi-subagents  1     2026-05-30

--- PROMPT TEMPLATES ---
  (none)

--- EXTENSIONS ---
  name                                used  last
  @juicesharp/rpiv-todo               115   2026-05-30
  pi-lens                             22    2026-05-30
  @juicesharp/rpiv-advisor            13    2026-05-30
  pi-agent-browser-native             10    2026-05-30
  @juicesharp/rpiv-ask-user-question  9     2026-05-30
  @aliou/pi-processes                 6     2026-05-30
  pi-subagents                        3     2026-05-30
  pi-web-access                       1     2026-05-30

--- BUILT-IN TOOLS ---
  name      used  last
  bash      665   2026-05-30
  read      258   2026-05-30
  edit      82    2026-05-30
  grep      16    2026-05-30
  write     14    2026-05-30
  find      2     2026-05-30
  ls        1     2026-05-30

=== PRUNE CANDIDATES (0 hits in 30d) ===
  skills    : caveman, paseo-loop, ...
  templates : (none)
  extensions: (none)

=== PROMPT-VISIBLE UNUSED SKILLS (0 skill loads in 30d) ===
  These are in <available_skills> and cost prompt tokens, but no read or /skill use was observed.
  name      approx prompt tokens  related usage  file
  caveman   ~N                    -              /Users/you/.agents/skills/caveman/SKILL.md

Model / Tool Detail (/tool-stats model-tools)

Model Tool Effectiveness (last 30d)
Attempts: 60  Results: 57  OK: 52  Fail: 5  Unknown: 3  (log: 1281 events total, 1281 in window)
Note: ok% is tool-execution success (not proof of task success).

--- BY MODEL ---
  model                        calls  ok  fail  unk  ok%  tools  top tools
  openai-codex/gpt-5.5            28  25     3    0   89%      4  read, bash, grep
  zai/glm-5.1                     24  19     2    3   90%      4  bash, edit, read
  anthropic/claude-sonnet-4-6      8   8     0    0  100%      1  agent_browser

--- BY TOOL ---
  tool           calls  ok  fail  unk  ok%  best model                  worst model
  bash              18  15     3    0   83%  openai-codex/gpt-5.5        zai/glm-5.1 2 fail
  read              18  16     2    0   89%  zai/glm-5.1                 openai-codex/gpt-5.5 2 fail
  agent_browser     10  10     0    0  100%  anthropic/claude-sonnet-4-6  -
  edit               7   4     0    3  100%  zai/glm-5.1                 -
  grep               6   6     0    0  100%  openai-codex/gpt-5.5        -
  find               1   1     0    0  100%  openai-codex/gpt-5.5        -

--- FAILURE HOTSPOTS ---
  model                 tool      fail  calls  ok%
  openai-codex/gpt-5.5  read         2     12   83%
  zai/glm-5.1           bash         2      9   78%
  openai-codex/gpt-5.5  bash         1      9   89%

--- MODEL × TOP TOOLS ---
  model                        bash       read       agent_browser  edit       grep
                                       ok/calls    ok/calls      ok/calls    ok/calls  ok/calls
  openai-codex/gpt-5.5               8/9      10/12            -           -        6/6
  zai/glm-5.1                        7/9        6/6          2/2         4/7          -
  anthropic/claude-sonnet-4-6          -          -          8/8           -          -

The report opens in a scrollable selector and does not pollute the LLM context.

Install

Local directory (this checkout)

git clone <repo-url> ~/code/pi-tool-stats
cd ~/code/pi-tool-stats && bun install

Add to ~/.pi/agent/settings.json:

{
  "extensions": ["/Users/<you>/code/pi-tool-stats"]
}

Then /reload (or restart pi).

As a pi package

If published to npm or git, add to the packages array in settings.json:

{ "packages": ["npm:pi-tool-stats"] }

Development

bun install          # install dependencies
bun run typecheck    # tsc type-check
bun run build        # bun build to /tmp (verification only)

Tips

• Reset: rm ~/.pi/agent/tool-stats.jsonl
• Watch live: tail -f ~/.pi/agent/tool-stats.jsonl
• Caveat: an extension only shows usage if it exposes a tool or slash command. Purely passive extensions (event-only hooks) won't register hits — there is no per-event attribution API.

License

MIT