pi-compact

中文版：README.zh-CN.md

A long-term memory, deterministic session-compaction, and exact-history-recall extension for Pi.

The core of pi-compact is long-term memory. Compaction manages window capacity: history remains recoverable, and durable memory can be reconstructed after compactions, restarts, and context-window switches, then queried through tools on demand. The extension calls no LLM; the deterministic checkpoint still contains only pointers and audit metadata, while a separate session-scoped, rule-based work summary is extracted from original text with entry IDs. That summary can restore context but is not a semantic conclusion. Original session entries remain the source of truth and can be re-read through the recall tool or command.

Features

• Stores an append-only memory event log under .pi/pi-compact/; current memory is rebuilt by a pure projector, and older events cannot silently overwrite newer state.
• Users write authoritative memory with /remember, /memories, and /forget. Model proposals stay provisional and never become active or pinned by themselves.
• Models query history through tools on demand. After a successful compaction, the extension builds a session-scoped structured work summary from the primary records involved in that compaction, keeping excerpts, files, and entry IDs; the summary accumulates only along the current branch's compaction chain, can be injected on before_agent_start for the current session, and can be searched through the recall tool. It is an index of original text, not semantic inference or a verified conclusion. The extension registers no context hook. Authoritative long-term memory still injects only pinned/active records the user wrote with /remember.
• Takes over Pi's native /compact command and normal manual, threshold, and overflow compactions.
• Reuses Pi's own boundary calculation, token accounting, persistence, and recovery flow. The checkpoint is a deterministic pointer/audit extract, not primary memory.
• Records a WindowManifest (windowId, parent window, retained boundary, sourceCount, sourceHash, previousHash).
• Verifies that tool calls and tool results are fully paired before compacting; on unsafe boundaries it falls back to an earlier safe boundary first and only cancels when no safe boundary exists, to avoid breaking context.
• Produces a deterministic event ledger and never infers work completion or semantic conclusions. Rule-derived records extract files, commands, exit codes, and test counts, plus user sentences carrying explicit durability markers, and stay provisional with provenance. They are session-scoped and never shared across sessions.
• History records distinguish primary vs derived: compaction and branch_summary are derived and can be filtered with sourceClass; thinking remains in raw but not in default search text.
• Provides the pi_compact_recall tool with list/search/read, entry-ID lookup, keyword search, file-path and kind filters, pagination, and raw-entry replay. Long raw entries support offset/rawLimit.
• Provides the /pi-compact-recall command, which displays results through the UI without triggering a model turn.
• Provides pi_memory_search, pi_memory_read, pi_memory_propose, pi_memory_update, and pi_compact_new_context.
• Automatically creates a project-level .pi/pi-compact.json config without overwriting existing project or global configs.

Installation

The current source requires Pi >=0.87.1; development and integration tests pin 1.4.0, which requires Node.js >=22.19.0. The instructions below assume this repository lives at github.com/DavidEasden/pi-compact. The published v0.8.0 matches the current source.

Pi packages execute extension code, so review the source before installing.

Install from GitHub

Pin to a release tag (recommended — pinned refs are not moved by pi update --extensions or pi update --all):

pi install git:github.com/DavidEasden/pi-compact@v0.8.0

Or track the default branch without pinning:

pi install git:github.com/DavidEasden/pi-compact

SSH shorthand and raw HTTPS URLs work too:

pi install git:git@github.com:DavidEasden/pi-compact@v0.8.0
pi install https://github.com/DavidEasden/pi-compact@v0.8.0

Notes:

• The git: prefix enables host/user/repo and git@host:user/repo shorthands; without it, only protocol URLs (https://, http://, ssh://, git://) are accepted.
• v0.8.0 must be an existing tag or commit (create and push it once with git tag v0.8.0 && git push origin v0.8.0). To move to a newer tag later, re-run pi install git:github.com/DavidEasden/pi-compact@<new-tag>.
• Global installs are cloned to ~/.pi/agent/git/github.com/DavidEasden/pi-compact; with -l (project settings), the clone lives at .pi/git/github.com/DavidEasden/pi-compact and the project auto-installs any missing packages on startup after being trusted.
• Try the package without installing it:

pi -e git:github.com/DavidEasden/pi-compact

Install from npm

The same code is published to npm as pi-compact:

pi install npm:pi-compact@0.8.0

Or track the latest published version:

pi install npm:pi-compact

Try it without installing it:

pi -e npm:pi-compact

Notes:

• A versioned spec such as npm:pi-compact@0.8.0 is pinned, so pi update --extensions and pi update --all skip it.
• Global installs go under ~/.pi/agent/npm/; with -l (project settings) they go under .pi/npm/.

Try locally

Run from the project root:

npm ci
pi -e /absolute/path/to/pi-compact

You can also point directly at the entry file:

pi -e /absolute/path/to/pi-compact/index.ts

Install the local package

pi install /absolute/path/to/pi-compact

This writes to the global Pi settings by default. Use -l to write to the current project's .pi/settings.json:

pi install -l /absolute/path/to/pi-compact

The extension uses Pi's @earendil-works/pi-coding-agent API, verified against Pi 1.4.0 for context_edit, post-compaction projections, and the current extension lifecycle. Pi and typebox are peer dependencies provided by the Pi environment; the development dependency pins Pi 1.4.0 for reproducible host integration tests.

Configuration

On first startup, if there is no project config and no global config in the user directory, the extension creates:

.pi/pi-compact.json

Config lookup order:

1. Current project .pi/pi-compact.json
2. Global ~/.pi/agent/pi-compact.json
3. Built-in defaults

The extension uses the first existing config file and fills in missing fields with defaults; project and global configs are not merged field-by-field. If the project config exists but cannot be parsed as JSON, the extension falls back to the default config and does not read the global one.

Default config:

{
  "enabled": true,
  "overrideDefaultCompaction": true,
  "summaryMaxChars": 12000,
  "autoRecall": true,
  "autoRecallMode": "full",
  "autoRecallMaxChars": 5000,
  "recallMaxResults": 8,
  "recallMaxChars": 16000,
  "debug": false,
  "memory": {
    "enabled": true,
    "pinnedInjection": true,
    "proposalsProvisionalOnly": true,
    "hintMaxChars": 4000,
    "deriveOnCompact": true
  },
  "history": {
    "autoRecallPrimaryOnly": true,
    "excludeInContext": true
  },
  "window": {
    "manifest": true
  },
  "sessionSummary": {
    "enabled": true,
    "inject": true,
    "maxChars": 6000,
    "maxEvidence": 80
  }
}

Config fields:

Flat aliases such as memoryEnabled, memoryHintMaxChars, historyAutoRecallPrimaryOnly, and windowManifest are also accepted; nested objects win. Numeric fields must be positive safe integers; sessionSummary.maxEvidence is at least 2. The caps for summaryMaxChars, autoRecallMaxChars, recallMaxResults, recallMaxChars, memory.hintMaxChars, sessionSummary.maxChars, and sessionSummary.maxEvidence are 100000, 50000, 30, 100000, 50000, 30000, and 500 respectively; legal values above a cap are clamped, invalid values fall back to defaults, and unknown fields are ignored. Char budgets are not token budgets. Defaults enable memory and keep model proposals provisional; history and summaries are queried through tools on demand, and automatic injection is limited to a budgeted session excerpt index plus user-confirmed pinned/active memories. Legacy injection settings still cannot restore the removed context hook.

The automatic compaction threshold and retained-tail size are still controlled by Pi's own compaction settings; this extension does not set its own trigger threshold.

Usage

Compact now

Run Pi's native command:

/compact

Pi's native /compact command calls its normal compaction flow. When enabled and overrideDefaultCompaction are both true, this extension receives the session_before_compact event and replaces Pi's default LLM summary with its deterministic checkpoint. The extension does not register a separate compaction command. When enabled or overrideDefaultCompaction is false, /compact keeps Pi's default summary behavior.

Authoritative long-term memory

Durable memory lives in .pi/pi-compact/memory.jsonl under Pi's ctx.cwd workspace directory. Sessions started with the same cwd share workspace memories; a different cwd gets a separate store, even when the session IDs match. The extension does not infer a Git root. session memories are visible only in the session that created them. The public scopes are workspace and session; old project and user event scopes are projected as workspace without rewriting the append-only log. User writes are authoritative; model proposals are always provisional and become active/pinned only after the user confirms them with /remember. The extension cannot guarantee that the model will obey these memories.

Appends to memory.jsonl (and windows.jsonl) are guarded by a per-log synchronous file lock (.pi/pi-compact/memory.jsonl.lock, windows.jsonl.lock) so concurrent Pi processes cannot interleave the read-last → compute-seq → append sequence and lose events. PI_COMPACT_LOCK_TIMEOUT_MS is only a bounded wait timeout (mainly for tests), not a live-lock lease: a lock whose owner PID is still alive is never reclaimed because of age. Crash recovery deletes a lock only when the owner PID is known dead, when stored start time and the live process start time are both readable and differ (PID reuse), or when this process can prove from start time that it left the lock behind and does not currently hold it. Linux reads /proc/<pid>/stat; macOS and other available Unix platforms try synchronous ps with TZ=UTC; Windows tries standard OS facilities such as PowerShell Get-Process StartTime. Start-time identity is best-effort — not every platform can read it, and any failure is treated as unverified. Unparsable, empty, or otherwise invalid lock metadata is not deleted within a grace period (it may be a concurrent create/write window); after the grace period (PI_COMPACT_STALE_LOCK_GRACE_MS, default 1000 ms, mainly for tests) it is reclaimed as a crash leftover so an empty lock file cannot deadlock writes forever. Within the grace period, unverified identity waits until the timeout and throws a clear error that owner metadata could not be verified or the lock is still held, and that the write did not complete. Manual deletion of the .lock file remains the explicit recovery path. The lock is reentrant within a process and is always released when the transaction ends. Release and reclaim require the lock file's owner token to match, and also the pid/startTime when those fields are present; missing expected metadata or a mismatch never deletes another owner's lock. Write failures throw explicit errors that surface to the user or model — a write is never reported as successful after a failure.

When reading either log, events must form an unbroken chain: seq starts at 1 and increases by exactly one, prevHash must equal the previous accepted event's hash, and each event's own hash must be correct. Reading stops at the first invalid, duplicated, skipped, or tampered event and only replays the trusted prefix. Once such an event exists, further appends to that log are rejected with an explicit error: the user must repair the log by hand before writing again, and the extension never truncates or rewrites the log to hide the problem. An unparsable half-line at EOF is still tolerated — it is skipped on read and the next append seals it, so a crash mid-write remains recoverable.

/remember use pnpm instead of npm
/remember pin kind:constraint do not change the production database
/remember supersede:mem_abc the new authoritative statement
/memories
/memories status:pinned token
/forget mem_abc

/remember accepts pin, kind:, scope:, priority:, and supersede:<id>. /memories can filter by status:, kind:, and keywords. /forget marks a memory as resolved so it leaves default memory queries. These commands only perform deterministic local writes and UI notifications; they do not trigger a model turn.

Memory tools:

Rule-derived memories have author=rule and provenance, stay provisional, and are not user-confirmed facts. They use scope=session and are visible only in the session that produced them. Beyond files, commands, and test counts, the extractor only picks short user sentences carrying durability markers such as “prefer”, “by default”, “always”, or “in this project”, classifying them as preference, constraint, or decision. It does not analyze assistant narration, does not infer long-term preferences from ordinary requests, and skips suspected credential content. Without a valid session ID nothing is derived. Records stay provisional until the user confirms them and are never injected as authoritative memory; legacy rule-derived workspace provisional records are marked resolved on the next successful compaction, while user-written records are untouched.

Durable-memory writes are bounded. A memory body over 8000 characters is rejected rather than truncated. sourceEntryIds are deduplicated and capped at 64 entries / 4096 characters, with truncation reported in the tool result. Model proposals with identical content in the same scope (and session, for scope:session) are written once. Provisional records authored by the model or rules leave default lists and search after 30 days; explicit status:provisional queries still return them, and user-authored records never expire this way. Memory search and read output is capped at 16000 characters including JSON metadata and escaping; search excerpts are limited to 600 characters per body, and omitted content can be read by ID.

With debug=true, the extension logs summary/derivation candidate counts, characters and token estimates, elapsed time, failure categories, and — at Pi's before_provider_request boundary — payload and stable-prefix hashes and lengths; assistant usage records the provider-reported input/output/cacheRead/cacheWrite. Logs never contain prompt or memory body text. Hash/prefix stability only verifies the provider payload the extension actually observes; cacheRead/cacheWrite are provider-reported values, and whether they yield real cost savings still depends on the provider and server implementation, which the extension does not guarantee.

Inspect status

/pi-compact-status is a read-only diagnostic command. It reports the effective switches (enabled, memory, summary, debug), memory and window log event counts, row counts and file sizes, the current session's summary event and evidence counts, and process-local injection and compaction statistics (cache hits/misses, last injected characters and tokens, last compaction reason, duration, record count, checkpoint characters, and derived-record count). It notifies through the UI, sends nothing to the model, and triggers no model turn.

Manual recall

Run inside Pi:

/pi-compact-recall token refresh

Supported argument forms:

/pi-compact-recall file:src/auth/session.ts
/pi-compact-recall kind:tool_result timeout
/pi-compact-recall ids:entry-123,entry-456 raw
/pi-compact-recall scope:all old config page:2 limit:5
/pi-compact-recall raw:true file:src/auth.ts

Arguments:

Entry IDs in the examples are placeholders; use the real IDs shown by recall output.

Summaries are extracted by rules from the current session's primary original records and contain only excerpts, categories, files, and source entry IDs; they do not infer completion status, test-result meaning, or user preferences. Summaries accumulate along the current branch's parent compaction events; different sessions use separate hash-named logs. When the full content is needed, read the original by the entry IDs the summary cites.

When ids is given, records are selected by ID and scope and limit apply, while the keyword, file, kind, and page are ignored; results are returned in original session order. Without ids, a keyword or file is required unless action is list — an empty query or a kind-only query does not list the full history.

Keyword search is tokenized text matching; it does not support regular expressions or semantic search. File filtering only matches already-extracted paths and never reads files from disk. Paths mainly come from tool arguments and simple bash commands; paths inside message bodies are not necessarily indexed. The command splits arguments on whitespace and does not parse quoted paths with spaces; pass such paths via the tool's file field instead.

The command displays recall content through UI notifications without sending it to the model or appending session messages. The UI command supports scope:all for other branches; the model tool always stays on the current branch.

pi_compact_recall tool

The model can call it directly:

{
  "query": "token refresh",
  "file": "src/auth/session.ts",
  "kind": "tool_call",
  "scope": "active-lineage",
  "page": 1,
  "limit": 8,
  "raw": false
}

Available fields mirror the /pi-compact-recall arguments: query, entryIds, file, kind, scope, page, limit, raw, action, offset, rawLimit, and sourceClass. action:summary reads the session summary, and action:summary_search searches summary evidence and returns source IDs.

action is list/search/read/summary/summary_search. Summary actions read only committed summaries matching the current session ID and the current branch's compaction entries; search results carry entryId, so the original entry can be read with entryIds. Summary excerpts are untrusted historical data and must not be treated as verified conclusions or instructions. All model-tool output, including a single raw entry ID, obeys the recallMaxChars hard budget. An oversized single raw entry returns valid JSON containing entryId, offset, limit, totalChars, truncated, and body; use offset/rawLimit to continue reading. Only the UI command, with one explicit ID and raw and no slicing parameters, can return the complete original JSON without that budget.

Multiple raw hits include only complete entries that fit and list omitted IDs; JSON is never cut in the middle. Pretty output is clipped at complete result blocks. Pagination divides records; offset/rawLimit divides an individual raw entry. Raw data preserves stored tool arguments, outputs, timestamps, and parent links. Thinking stays out of default search text. compaction and branch_summary are derived and can be filtered with sourceClass.

Query history and memory on demand

The current version registers no context hook and does not auto-inject full history body text. On before_agent_start it can append a static compaction pointer, the current branch's structured work summary for the current session, and — when memory.pinnedInjection is enabled — pinned/active authoritative memories written by the user. The summary contains only rule-extracted excerpts of primary records, file lists, and entry IDs, is marked as untrusted JSON data, and is bounded by sessionSummary.maxChars. Provisional, rule-derived, and model-proposed long-term memories are never injected as authoritative memory. Historical detail stays reachable through pi_compact_recall.

The model's pi_compact_recall searches only the current branch. If the active lineage is unavailable, it does not expand to the whole session. Users can explicitly select scope:all in the /pi-compact-recall UI command. Tool results enter ordinary model context; UI command results only appear in the interface.

Stored originals remain available for exact retrieval. On-demand queries do not guarantee that the model recalls every relevant detail or follows every memory.

Compaction and data boundaries

pi-compact takes over only normal compactions and does not yet take over Pi's session_before_tree branch summaries.

During compaction:

1. Uses Pi's firstKeptEntryId, including context-invisible metadata and entries omitted by context_edit.
2. Converts the entries before the boundary from Pi's pre-compaction context projection into ID-bearing audit records. context_edit omissions affect the checkpoint's source count and hash; replacements affect the projected record content used for derived-memory inputs. The original raw entry remains available for explicit replay.
3. Generates a deterministic checkpoint with only compactor, sourceCount, and sourceHash. It copies no history body and is not primary memory.
4. Stores sourceEntryIds, sourceHash, sourceRecordCount, keptEntryId, omittedRecordCount, checkpointChars, summaryMaxChars, isSplitTurn, and estimatedTokensAfter (character count / 4, rounded up; not provider usage). Pi computes its own context-wide post-compaction estimate; the extension does not invent billed usage.
5. Previews each candidate compaction with Pi 1.4.0's buildSessionProjection(), checking the retained tail after omissions and replacements. Results must follow their calls, and ordinary calls must have results; the replaced prefix does not block compaction. Unsafe boundaries fall back only to earlier candidates in the current compaction window, after previewing each candidate. No safe boundary, failed projection, or an aborted request cancels compaction without falling back to LLM summarization. Pi's error/aborted assistant states may have calls without results. Once validation passes, checkpoint generation errors degrade to pointer JSON; a changed boundary gets a recomputed isSplitTurn.
6. Only after Pi commits compaction and emits success does the extension generate and append a window manifest under the log lock. Window IDs bind the session and actual compaction entry, so repeated events do not append duplicates. Rule-derived memories are also written after success, along with a session-scoped deterministic work summary that accumulates along the current branch's compaction chain and keeps excerpts, files, and entry IDs under .pi/pi-compact/sessions/ in a hash-named per-session log. Cancellation and failure leave no window, derived records, or summary. Legacy details.window remains readable; new manifests live in .pi/pi-compact/windows.jsonl.

Original session entries are the source of historical truth; a structured session summary is a deterministic excerpt index generated from primary records, carrying source entry IDs and the source hash — it makes no semantic inferences and does not verify statements. The user-written memory log is the sole source of authoritative durable memory. The checkpoint stores only counts and a hash; it is not a backup of raw text. Non-text content such as images shows only placeholder information in the text extraction.

The extension does not delete original session entries and does not create a separate backup; the raw text still relies on Pi's session storage. A finite context cannot display all history at once.

Development

Install dependencies:

npm ci

Run tests:

npm test

npm test runs unit and Pi SDK integration tests; npm run test:integration runs only the host lifecycle tests. Integration tests use a local Faux provider without API keys or online model requests, and write sessions and logs only to temporary directories.

Run the TypeScript type check:

npm run typecheck

Run the manual performance benchmark:

npm run bench

Inspect published contents:

npm pack --dry-run

The published allowlist is index.ts and src/; npm also automatically includes package.json and README files. Test files and tsconfig.json are not part of the package.

Project structure

index.ts                 Pi extension entry
src/config.ts            Config loading, normalization, and scaffolding
src/hooks.ts             Compaction projection validation and session hooks
src/recall.ts            Recall tool and /pi-compact-recall command
src/memory.ts            Memory commands and memory/new-window tools
src/status.ts            Read-only /pi-compact-status diagnostics
src/core/content.ts      Message text, files, thinking separation, and snippet boundaries
src/core/auto-recall.ts  Legacy gate pure functions, not connected at runtime
src/core/lock.ts         Synchronous log lock, owner-token reclaim, portable start-time identity
src/core/ledger.ts       Deterministic checkpoint and details
src/core/session.ts      Session entry conversion, sourceClass, search, and raw replay
src/core/jsonl.ts        Corruption-safe JSONL I/O
src/core/projector.ts    Pure-function memory event projection
src/core/store.ts        `.pi/pi-compact/` event logs
src/core/window.ts       WindowManifest and hash chain
src/core/summary.ts      Deterministic session-summary extraction, isolated storage, accumulation, and search
src/core/working.ts      Authoritative-memory hint rendering
src/core/memory-policy.ts Memory write limits, proposal deduplication, and provisional TTL
src/core/runtime-stats.ts Process-local injection and compaction counters
src/core/derive.ts       Non-semantic rule derivation
src/types.ts             Shared types
scripts/bench.mjs        Manual performance benchmark
tests/                   Unit tests

Verification scope

Verified so far:

• Deterministic record extraction, keyword and file search, raw record preservation, and hashing.

• Thinking stays out of default search; primary/derived filtering and current-branch enforcement for the model tool.

• Deterministic memory-event replay, supersede/pin, EOF half-line tolerance, and rejected appends after a chain break.

• Window hash chain, memory command/tool entry points, and sliced long-raw reads.

• Compaction boundary checks for tool calls and tool results.

• Config scaffolding and normalization of invalid configs, including memory/history/window.

• manual, threshold, and overflow compaction reasons plus aborted requests, via simulated hook events.

• Empty active lineage does not widen the search; no context hook is registered, and only bounded session excerpts with source IDs are injected according to configuration.

• before_agent_start compaction-pointer, session-summary, and authority-memory injection, including exclusion of provisional/rule/model records and the pinnedInjection/sessionSummary.inject off switches.

• Deterministic session-summary extraction, session isolation, hash-named per-session logs, branch accumulation, summary/summary_search recall, and rejection of untrusted prefixes in corrupted logs.

• Legacy auto-recall gates retain unit tests but are not connected at runtime; working.ts remains in use only to render the authoritative-memory hint, not the old pi-compact-memory-hint message.

• Log locks refuse token/metadata mismatches and do not reclaim live owners by age. Invalid metadata can be reclaimed after the grace period. Portable start-time identity (Linux /proc, Unix ps, Windows Get-Process) and conservative unverified fallback are unit-tested without changing process.platform or recycling real PIDs.

• Clean dependency installation, TypeScript type checking, and real Pi CLI extension loading.

• Pi 1.4.0 SDK metadata boundaries, full overflow retry, context_edit omissions/replacements and candidate projections, cancellation of orphan tool results, parallel/sequential tool-batch continuation, user abort, cancellation without side effects, success-event deduplication, and session persistence/restoration.

• Memory-tool disablement, session isolation, explicit historical status queries, long-content/JSON budgets, and search/read pagination.

Online model API responses and billing, manual TUI interactions, and long-running behavior remain outside these tests.

License

MIT