pi-okf-wiki

A Pi extension that turns any project into a local, agent-readable knowledge base using the Open Knowledge Format (OKF).

Drop documents into input/, run /wiki-update, and they become OKF concept files in wiki/. Then ask /wiki-query <question> and get an answer cited to the concepts that back it.

Commands

Command	What it does
/wiki-update	Ingest new documents from input/ into the wiki/ bundle, archive the originals, regenerate index.md / log.md, and show a summary.
/wiki-query <question>	Answer a question against the wiki, with every claim cited to a wiki/<concept-id>.md source.
/wiki-remove <path>	Move a concept (or a whole directory of concepts) into wiki/trash/, redirect links that pointed at it, and regenerate index.md / log.md.
/wiki-migrate	Rewrite legacy OKF v0.1 concepts to v0.2: timestamp becomes generated: { by, at }, the body # Citations list is lifted into the sources frontmatter, and the pre-v0.2 status values current / superseded become stable / deprecated. Deterministic (no agent turn); already-current concepts are untouched.

Folder layout

The extension operates on two folders at the project root (ctx.cwd):

.
├── input/          # drop new material here
└── wiki/           # the OKF knowledge bundle (concepts + index.md + log.md)
    ├── archive/    # originals land here after their concept exists in wiki/
    └── trash/      # concepts land here when they are removed from the wiki

Both archive/ and trash/ live inside the bundle so that references to them are bundle-relative paths, an OKF §6-sanctioned form for links and path-valued fields (§6.2). Neither is indexed, and .md files in them carry an outermost .orig suffix so they are not concept documents per §3.1 and the bundle stays conformant (§11).

Missing folders are created on the first /wiki-update.

Installation

From npm

pi install npm:pi-okf-wiki

From git

pi install git:github.com/Panzenbaby/pi-okf-wiki
# or pin a specific tag:
pi install git:github.com/Panzenbaby/pi-okf-wiki@<tag>

Local / development

Clone the repo, then either load it for a single run:

pi -e /path/to/pi-okf-wiki

…or add it to your project's .pi/settings.json so it auto-loads:

{
  "packages": ["./pi-okf-wiki"]
}

After installing, (re)start Pi in your project and the two commands are available. Reload after upgrading with /reload.

How /wiki-update works

/wiki-update classifies every file in input/ into one of three buckets:

1. Conformant — a .md (or .markdown) file with parseable YAML frontmatter and a non-empty type field. Taken over deterministically (no LLM): the file is written to wiki/<relative-path> and only then moved from input/ to archive/. Its concept ID is the path without the extension; a .markdown input is written as .md, since the wiki only loads .md.
2. Non-conformant — everything else worth reading: a .md lacking frontmatter or a non-empty type field; plain text and images read directly by Pi's read tool; and binary/structured documents which the extension pre-extracts to plain text into a temp input/.okf-extract/<relative-dir>/<stem>-extracted.txt and hands that to the agent (see Supported formats). These are handed to the agent, which reads every non-conformant file, clusters those describing the same real-world entity (matched on asserted name / resource / keywords, not on filename), and writes one OKF concept per cluster (frontmatter + structured body, cross-links, citations) to wiki/ before moving each original to archive/. The existing wiki structure — directories, types in use, and the full list of existing concept IDs — is passed to the agent so new concepts fit in and duplicate IDs are avoided. This bucket (and every /wiki-query) invokes an agent turn, i.e. uses the LLM; conformant files are ingested deterministically with no LLM call.
3. Ignored — unsupported file types, reserved filenames index.md / log.md placed in input/, and documents whose extraction failed (encrypted, no extractable text, or a lib error). Listed in the summary with a stable reason code (unsupported, reserved, encrypted, extraction_failed, empty, io_failed) and left in input/.

After the agent turn, the extension regenerates index.md, appends a dated entry to log.md, rewrites /archive/<input-relative-path> placeholder citation links in the agent-written concepts to the actual (collision-renamed) archive paths (so a UI can jump straight to the archived original even when it was renamed during the move), detects any files still left in input/ (the agent did not finish them), and renders a summary widget:

OKF /wiki-update summary
  Conformant imported:   1
  Agent-transformed:     3 (of 4)
  Ignored:               1
  Leftover (failed):     1
  Concepts created:      3
  Concepts updated:      0
  Wiki size:             5 -> 9
  Imported concept IDs:
    + tables/orders
  Created by agent:
    + playbooks/incident
  Ignored:
    - notes/old.doc (unsupported file type)
  Leftover in input/ (agent did not finish):
    ! reports/q3.md

Supported formats

Bucket	Extensions	How they reach the agent
Conformant (deterministic)	.md / .markdown with frontmatter type	Copied to wiki/ directly, no LLM.
Plain text (read directly)	.txt, .csv, .tsv, .json, .yaml/.yml, .toml, .dsl, .mmd/.mermaid, .puml/.plantuml, .dot/.gv, .rst, .adoc/.asciidoc, .org	Pi's read tool.
Images (vision)	.png, .jpg, .jpeg, .gif, .webp, .bmp	Pi's read tool.
Extracted to text	.pdf, .docx, .pptx, .xlsx, .odt, .ods, .odp, .html/.htm, .epub, .rtf, .jsonl/.ndjson, .ipynb	Pre-extracted to input/.okf-extract/<rel-dir>/<stem>-extracted.txt; the agent reads that. The original is archived; a copy of the extracted text is archived next to it.

The plain-text bucket is a deliberate allowlist, not a "does it decode as text?" sniff — sniffing would swallow lockfiles, keys and minified bundles, and unsupported is a more useful signal. .dsl covers diagram/architecture DSLs (including the text Miro's MCP tools read and write) and is intentionally not parsed: that grammar is served at runtime and versioned server-side.

Extraction libraries (runtime dependencies of the extension):

Format	Library
.pdf	unpdf
.docx	mammoth
.xlsx	exceljs (rendered as markdown tables)
.pptx / .odt / .ods / .odp / .epub	jszip + XML readers (.ods keeps rows/columns as markdown tables, .odp keeps slide boundaries)
.html	html-to-text
.rtf	dependency-free RTF stripper (control words + destination groups)
.jsonl / .ndjson	dependency-free record renderer (one pretty block per record)
.ipynb	dependency-free notebook reader (markdown + code cells, outputs dropped)

Large JSONL files are split

A .jsonl/.ndjson source is staged in parts of 1000 records, named <stem>-extracted.part01.txt, part02, … Records stay numbered continuously across parts, and unparseable lines are skipped with a warning rather than failing the whole file. The parts remain one input file everywhere else: one prompt entry, one archived original, and the agent is told to treat them as a single document. Every other format stages exactly one <stem>-extracted.txt, unchanged.

Extraction failures are reported with a stable code and never archive the original: encrypted (password-protected), empty (no extractable text), extraction_failed (library error), io_failed (read/write error). Unsupported types and reserved filenames are unsupported / reserved.

Safety invariant

An original is moved to archive/ only after its content exists as a concept in wiki/ — per file. If a run is interrupted, the file stays in input/ and is reprocessed on the next /wiki-update. Archive destinations are collision-free: if archive/<rel> is already taken, the new copy is timestamped (orders.2026-07-12-1305.md, then …-1, …-2 within the same minute) and existing archive files are never overwritten. This keeps every version of an original while making /wiki-update idempotent and abort-safe.

Finalization (snapshot diff, index.md/log.md regeneration, leftover detection, summary) runs in Pi's agent_end event, so it always sees the wiki state after the agent finished writing — not a racy pre-turn snapshot.

Conflict handling & versioning

When several inputs (or a new input and an existing concept) describe the same entity but disagree on a value, the agent does not silently pick one. Instead it records the disagreement inside the single concept body:

• A # Conflicts (or # Versions) table — one row per source with the differing attribute, its value, its source footnote, and the source date.
• Each source recorded as a sources frontmatter entry ({ id, resource, title }, §5.1) with per-claim attribution via footnotes keyed to the entry's id ([^spec-v2]). For an archived original, resource is a placeholder of the form /archive/<input-relative-path> using the ORIGINAL input relative path; the extension rewrites that placeholder to the actual (collision-renamed) archive path after the originals are moved, so the reference stays resolvable when a file was renamed on archiving. The top-level resource: frontmatter field stays a canonical URI (never an archive path).
• A canonical value chosen by temporal precedence: the value with the latest source date / “latest” marker wins and is stated in the description and # Schema; older values are marked superseded. If no source is clearly newer, all conflicting values stay in the table labelled unverified and no canonical value is declared.

The precedence graph across concepts uses status (the OKF §5.4 lifecycle: draft | stable | deprecated, absent meaning stable) together with the producer-defined supersedes: [/path/to/older.md]. A concept that is no longer current is deprecated; the one that replaced it lists it in supersedes.

How /wiki-query works

1. Load all concepts from wiki/.
2. TF-IDF cosine retrieval picks the top 10 matches for the question; the root index.md and the full wiki tree are always included as context.
3. The agent answers in the same language as the question, citing each claim with an inline [title](wiki/<concept-id>.md) link and a # Sources section at the end. It may use read/grep to explore the wiki further.
4. The retrieved concepts, index.md, and the wiki tree are injected into the system prompt via a before_agent_start hook, so the user message stays clean — just the question itself. This keeps the session readable and the question persistable. The agent is also told to open the whole concept body before answering (not just the retrieved snippet), to follow # Related Concepts / # Versions links, and to surface any conflicts it finds while exploring.

If wiki/ does not exist or has no concepts, /wiki-query tells you to run /wiki-update first instead of inventing an answer.

Trust, freshness, and attribution

Answers weigh the OKF v0.2 trust families rather than treating every concept as equally reliable:

• Trust tier derived from verified (§5.3): no verified means unverified, verification by non-human: actors only means machine-confirmed, and a human:<id> verifier means human-reviewed. When concepts conflict, the higher tier and the fresher generated.at win. If an answer rests on an unverified concept for a substantive claim, it says so.
• Freshness: a concept past its stale_after or marked status: deprecated is still used when it is the best available knowledge, but is flagged as possibly stale rather than presented as current.
• Attribution: when you ask where a fact comes from, the agent resolves the body footnote label ([^spec-v2]) through the concept's sources list to the actual source entry.

How /wiki-migrate works

Upgrades a legacy OKF v0.1 bundle to v0.2 in place. It is deterministic — no agent turn, no model call — so the result is reproducible and reviewable in a diff.

Per concept it applies the three v0.1 conventions OKF v0.2 supersedes:

v0.1	v0.2
timestamp: <ISO>	generated: { by: pi-okf-wiki/legacy, at: <ISO> } (§5.2)
body # Citations list	sources frontmatter entries { id, resource, title } (§5.1)
status: current / superseded	status: stable / deprecated (§5.4)

The legacy actor pi-okf-wiki/legacy is used because the model that originally wrote a v0.1 concept is not recorded anywhere, and generated.by is required within generated. No footnotes are fabricated: per-claim attribution cannot be reconstructed after the fact, and sources without body footnotes is fully conformant.

Guarantees:

• Concepts that are already v0.2 are left byte-for-byte untouched, and a run that changes nothing writes nothing at all — not even log.md.
• Unknown, producer-defined frontmatter keys survive the rewrite unchanged (§4.1), as does key order.
• # Citations inside a fenced code block is left alone — a concept that documents the v0.1 format is not mistaken for one that uses it.
• Afterwards index.md (now declaring okf_version: "0.2") and log.md are regenerated.

/wiki-migrate rewrites the files in wiki/ in place. Keep a copy if you want a way back.

Migrating is recommended but not required: v0.1 concepts stay readable either way (see OKF conformance).

OKF conformance

The extension targets OKF v0.2. Concepts are markdown files with YAML frontmatter; type is the only required field. Recommended fields: title, description, resource, tags. New and rewritten concepts carry the v0.2 trust and provenance families (§5): generated: { by, at } records who wrote the content and when, and the sources list records what it derives from, with per-claim attribution via markdown footnotes keyed to sources[].id. Existing v0.1 concepts (legacy timestamp, body # Citations) remain readable — consumers fall back per §13.1 — and can be upgraded in place with /wiki-migrate. index.md and log.md are reserved filenames maintained by /wiki-update. See the OKF spec for the full format.

The bundle declares its target spec version via okf_version: "0.2" in the root index.md frontmatter (§12 — the only index.md permitted to carry frontmatter). Per OKF §8, /wiki-update writes one index.md per qualifying directory (root + every directory that contains a concept, directly or transitively) for progressive disclosure; each lists only its own direct concepts and immediate child subdirectories. Orphan index.md files in directories that no longer contain any concept are pruned on the next /wiki-update (best-effort).

wiki/archive/ is a producer-specific OKF extension, not part of the spec's references/ model: it holds the raw original files (PDFs, DOCX, …) that back the concepts, so sources[].resource values pointing at them stay bundle-relative (/archive/<rel>) — one of the three §6.2-sanctioned path forms. It is tolerated by the spec because consumers ignore non-.md files (§11); archived .md originals carry an outermost .orig suffix so they never count as concept documents (§3.1 / §11). archive/ is never listed in any index.md.

Example concept (wiki/tables/orders.md):

---
type: BigQuery Table
title: Orders
description: One row per completed order.
tags: [sales, orders]
generated: { by: pi-okf-wiki/claude-sonnet-4, at: 2026-07-03T00:00:00Z }
sources:
  - id: orders-spec
    resource: /archive/notes/orders-spec.pdf
    title: Orders schema spec
---

# Schema

| Column        | Type   | Description                       |
|---------------|--------|-----------------------------------|
| `order_id`    | STRING | Unique order identifier.[^orders-spec] |
| `customer_id` | STRING | FK to [customers](/tables/customers.md). |

Part of the [sales dataset](/datasets/sales.md).

[^orders-spec]: Orders schema spec

Link styles: inside concept files, links are bundle-relative (/tables/orders.md — absolute, relative to the wiki/ bundle, the §6.1 recommended form). In index.md files, links are relative to that directory (orders.md from tables/, or tables/orders.md from the root) per the §8 example. These two styles mirror the spec's own conventions for concepts vs. index files, not two competing rules. In /wiki-query answers, links are repo-relative (wiki/tables/orders.md) because the answer renders outside the bundle, from the project root — a third rendering context.

Configuration

There is no configuration file yet. The extension always reads from input/, archive/, and wiki/ relative to ctx.cwd. /wiki-query answers and /wiki-update summaries follow the language of the question / transformed content automatically; edit src/prompts.ts to change that behavior.

Architecture

A TypeScript extension, strictly typed with no any. Filesystem, wiki, and extraction operations return a Result<T> (success/error) and never throw to callers. Per the project's AGENTS.md, document extraction follows the Repository pattern: each format family has a DocumentExtractorRepository that wraps a third-party library, converts the library's native Dto to the ExtractedText AppModel, and returns Result<ExtractedText>. The Dto never leaks outside its repository.

File	Responsibility
src/index.ts	Registers the /wiki-update, /wiki-query, /wiki-remove, and /wiki-migrate commands; owns the agent_end finalize hook and the before_agent_start query-context hook (both via the session registries).
src/types.ts	Result<T>, AppError, OKF domain models, and the IgnoreReason code union.
src/session.ts	Session interface and the generic SessionRegistry<T> that owns the single-slot handoff between a command handler and an event hook.
src/frontmatter.ts	YAML frontmatter parsing and serialization for OKF v0.2, backed by the yaml package (v0.2 needs nested maps and lists of maps). Parses permissively per §11: a malformed block is repaired and re-read rather than rejected, so a concept never disappears silently over a stray colon or asterisk. Also exports serializeDocument for deterministic writers.
src/files.ts	Filesystem helpers, all returning Result<T> (incl. copyFile, removeDir).
src/wiki.ts	Barrel re-exporting the wiki/ modules so the ./wiki.ts import surface stays stable for update.ts, query.ts, classifier.ts, and prompts.ts.
src/wiki/paths.ts	wikiPaths, conceptIdFromRelativePath, isConceptFile, relativePosix, ARCHIVE_DIR, TRASH_DIR, WikiPaths.
src/wiki/concepts.ts	Concept loading (loadConcept, loadAllConcepts), snapshot/diff (snapshotWiki, diffSnapshots, WikiDiff).
src/wiki/index-log.ts	index.md / log.md generation (generateIndexMd, writeIndexMd, appendLogMd, buildLogEntry).
src/wiki/retrieval.ts	Structure preview and TF-IDF cosine retrieval. The Retriever interface is the seam injected into /wiki-query; TermFrequencyRetriever is the default implementation (it replaces the former retrieveConcepts free function, which is kept as a thin wrapper). IDF is computed on the fly from the loaded concepts so common terms are downweighted in any language — no hardcoded stopword list. Also exports: tokenize, renderConceptForPrompt, renderWikiTree, displayTitle, buildStructurePreview.
src/prompts.ts	Agent prompt builders for ingestion and query.
src/links.ts	Pure link rewriters (no IO). compileArchiveRewriter / rewriteArchiveCitationLinks rewrite /archive/<input-relative-path> placeholders to the actual (post-rename) archive path across the WHOLE document — in v0.2 those placeholders live in sources[].resource frontmatter values, not only in body links. compileRemovedConceptRewriter / conceptIdFromLinkTarget resolve any spelling of a concept reference (root-relative, wiki/-prefixed, or relative to the citing file) to a conceptId and redirect removed ones to their /trash/ path, in body links and frontmatter resource: values alike. collectConceptReferences is the read-only counterpart used by the removal preview, so preview and rewriter cannot drift apart.
src/migrate.ts	/wiki-migrate logic: deterministic v0.1→v0.2 concept rewriting (timestamp→generated, body # Citations→sources, legacy status values→the §5.4 lifecycle). No agent turn; already-current concepts stay byte-identical and a no-op run writes nothing.
src/remove.ts	/wiki-remove logic: planRemoval (what would be affected, incl. incoming references from bodies and from frontmatter resource: values — no mutation) and removeFromWiki (move to trash, redirect links, regenerate index.md, append the Removal log entry, collapse emptied directories). Deterministic, no agent turn.
src/update.ts	/wiki-update command logic and the IntakeSession (finalize) that owns the agent-handoff state, including the post-agent citation-link rewrite (rewriteArchiveCitationsInConcepts).
src/classifier.ts	InputClassifier that owns the full input→bucket pipeline AND the deterministic conformant intake: tentative dispatch by extension, the extraction pass (staging extracted text), and pass 3 — read + verify frontmatter + write to wiki/ + archive original — for conformant .md files. Emits the three final buckets (conformantImported / forAgent / ignored) once, in input order.
src/query.ts	/wiki-query command logic and the QuerySession that owns the pending question. Both buildWikiQueryContext and runQuery take an optional Retriever (default TermFrequencyRetriever) so the scoring strategy is injectable.
src/extract/types.ts	ExtractedText AppModel (one or more text parts), DocumentExtractorRepository interface, extraction-failure cause codes.
src/extract/pdf.ts	PdfRepository (unpdf).
src/extract/docx.ts	DocxRepository (mammoth).
src/extract/sheet.ts	SheetRepository (exceljs), rendering worksheets as markdown tables.
src/extract/office-xml.ts	PptxRepository, OdtRepository, OdsRepository, OdpRepository, EpubRepository (shared jszip + XML helpers, EPUB spine-order).
src/extract/html.ts	HtmlRepository (html-to-text).
src/extract/rtf.ts	RtfRepository (dependency-free RTF stripper).
src/extract/jsonl.ts	JsonLinesRepository (.jsonl/.ndjson), the only extractor that splits its output into parts.
src/extract/notebook.ts	NotebookRepository (.ipynb), markdown + code cells, outputs dropped.
src/extract/registry.ts	Format taxonomy + ExtractorRegistry dispatch.
src/extract/service.ts	Extraction-to-temp-file orchestration and the .okf-extract/ lifecycle.
src/extract/util.ts	Shared Result<T> failure + error-message helpers for repositories.

Development

npm install          # installs peer/dev deps for type-checking
npm run check        # tsc --noEmit (strict, noUnusedLocals)

The extension imports @earendil-works/pi-coding-agent only as type-only imports (erased at runtime), so it has no bundled runtime dependencies.

License

MIT — see LICENSE.