pi-web-kit

Give Pi current web knowledge, authoritative library docs, and real-world code examples without flooding model context.

pi-web-kit combines search, page reading, version-aware documentation, and code research behind five agent-ready tools with bounded, cache-aware output.

Features

• Research the live web — search current information and read single or multiple pages without leaving Pi.
• Use docs that match the task — resolve libraries and retrieve focused, version-aware documentation with code examples.
• Find proven implementation patterns — search practical usage, setup, migrations, and error context across real code.
• Spend context wisely — compact search results, chunked page reads, bounded output, and fetch caching keep research useful without overwhelming the model.
• Compose research in scripts — typed results let codemode filter and join data without parsing model-facing text.
• Choose your providers — mix Exa, TinyFish, Brave, Firecrawl, markdown.new, and Context7 based on coverage, cost, and credentials.

Installation

Install from npm:

pi install npm:pi-web-kit

Install project-locally with Pi's -l flag:

pi install -l npm:pi-web-kit

During local development from this monorepo:

pi install /path/to/pi-mono/packages/pi-web-kit

For a one-off test run without installing:

pi -e /path/to/pi-mono/packages/pi-web-kit --web-provider-fetch markdown_new --print "Fetch https://example.com"

This is an npm-compatible TypeScript Pi package. Bun is not required. Use Pi 1.1.0 or newer and Node.js >=22.19.0 for the structured tool contracts.

Quick usage

Search:

Find recent documentation for the Pi extension API.

Multi-query search:

Search for recent docs on Pi extensions and Pi tool schemas.

Fetch a page:

Read https://example.com and summarize it.

Fetch a long page in chunks:

Fetch https://example.com/long-doc with limit 8000, then continue with offset 8000.

Pi chooses web_search or web_fetch automatically when the request calls for it. You can also mention provider settings explicitly in prompts, but provider changes usually require config or CLI flags.

Providers

Defaults: provider_search = "exa", provider_fetch = "exa".

Tool schemas are tailored to the configured providers at startup/reload, so only supported provider-specific fields are exposed. Restart/reload Pi after changing provider config.

Provider-native efficiencies are used as follows:

Configuration

Resolution order: defaults < environment variables < global config < trusted project config < CLI flags. Project config is ignored unless Pi trusts the current project, including in print, JSON, and RPC modes.

Environment variables

PI_OFFLINE=1        # disables install/update telemetry
PI_TELEMETRY=0      # disables install/update telemetry
PI_WEB_KIT_PROVIDER_SEARCH=exa|tinyfish|brave|firecrawl
PI_WEB_KIT_PROVIDER_FETCH=exa|tinyfish|markdown_new|firecrawl
EXA_API_KEY=...          # enables Exa provider and code_search
CONTEXT7_API_KEY=...     # enables library_search and library_docs
TINYFISH_API_KEY=...
BRAVE_SEARCH_API_KEY=...
FIRECRAWL_API_KEY=...

Config files

Config files, in increasing precedence:

The global path follows Pi's getAgentDir() resolver. If PI_CODING_AGENT_DIR is set, pi-web-kit.json is read from that directory instead of the default.

Example:

{
  "provider_search": "firecrawl",
  "provider_fetch": "markdown_new",
  "apiKeys": {
    "firecrawl": "...",
    "context7": "...",
    "exa": "..."
  },
  "markdownNew": {
    "method": "auto",
    "retainImages": false
  }
}

Do not commit config files containing secrets. Project .pi-web-kit.json is ignored by this repo's .gitignore, but other repositories may need their own ignore rule.

CLI overrides

pi -e . --web-provider-search firecrawl --web-provider-fetch markdown_new --print "Search and fetch docs"

Provider CLI flags are temporary for the Pi process. Restart/reload Pi after changing provider config so registered tool schemas match the active provider.

Tools

Terminal result views

library_search, library_docs, and code_search show compact completion summaries instead of raw JSON in the interactive transcript. Expand a tool result with Pi's expand-tools action (default: Ctrl+O) to inspect ranked library candidates, available versions, documentation sources, or code source titles and URLs. Long fields are clipped to the terminal width; version previews show at most two entries plus the remaining count. Code-search summaries flag reduced output; expanded views show omitted-source and context counts when present. Library documentation also shows omitted code/text snippet counts and refinement guidance. All five tools remove terminal control sequences from untrusted display text before applying theme styling.

These views change presentation only. Model-facing JSON and typed codemode results retain the same bounded, redacted data, including snippets and code. Final print, JSON, and RPC tool results are unchanged.

web_search

Searches with the active search provider and returns compact results grouped by query.

numResults controls source breadth. contextTokens controls grounding depth. The tool automatically keeps native/included Exa highlights and Brave LLM Context. Explicit contextTokens enables extra extraction for TinyFish and Firecrawl; this can add provider calls or provider cost. Exa always requests highlights and prefers them over text or summaries. Use web_fetch for complete pages, exact passages, or missing context. Search results keep a compact snippet plus ranked content and contentFormat, with one shared context budget and the existing 50KB tool-output limit.

Provider caps are Exa 100, Brave 50, and Firecrawl 100. TinyFish is paginated internally through its service maximum of page 10 and may make up to 11 search requests for one query. Search output reports requested, effective, returned, and omitted result/context counts. Brave maxUrls remains as a deprecated alias for numResults.

Other provider-specific parameters are exposed only for the configured provider. These include Exa date/domain filters; TinyFish domain, date, geography, language, and publication filters; Brave locale, freshness, spellcheck, Goggles, and LLM Context controls; and Firecrawl scrape/search options.

TinyFish preserves one-sided date/year bounds. If both endpoints are supplied in reverse order, they are reordered; an upper-only bound never becomes a lower bound.

Output limiting preserves complete source URLs. It reduces evidence or omits whole results rather than shortening citation targets.

Exa modes, filters, and notices

• type: "auto" is the default. type: "lite" is an explicit lower-cost option that serves content from Exa's regularly refreshed index without query-time page fetching. When content is requested, Lite can exclude sources with no cached content. Use Auto when coverage or live freshness matters. Lite remains available for general web_search, but is never available in code_search.
• Lite accepts the exposed domain/path, publication-date, category, result-count, and purpose controls. Every maxAgeHours value, including -1, is rejected locally with guidance to use Auto. Unsupported overrides are not silently removed. Legacy search types remain accepted.
• startCrawlDate and endCrawlDate are deprecated and rejected: Exa ignores them. Publication dates describe a different constraint and are not substituted. The company and people categories reject publication-date filters; people also rejects excludeDomains.
• Exa query strings, including each queries entry, are limited to 2,000 characters. type and category are limited to 64 characters. Each domain filter accepts at most 1,200 entries of up to 2,048 characters each. Publication-date strings are limited to 128 characters and must be ISO 8601 dates. Blank query, purpose, mode, category, and domain/path strings are rejected. These local limits are declared in the tool schema.
• Each query group can include notices: [{ code, message, path? }] that explain changes to retrieval, such as relaxed query restrictions. At most 8 notices with 512-character messages are returned; omittedNoticeCount reports omitted entries. Notice codes are an open set. Arbitrary notice details are excluded.
• Ordinary highlights: true is the stable baseline. Dynamic Highlights is not enabled automatically: the announcement and current API guide disagree about its GA/beta status. The manual evaluation below tests the documented beta request separately; no beta header is sent by the production provider.

Exa /search calls emit best-effort pi-web-kit:exa-diagnostics events through pi.events, outside model output. Consumers can subscribe to capture request IDs, duration in milliseconds, provider searchTime and costDollars.total when present, known search modes, warning codes/field paths, and controlled failure categories/HTTP status. Warning messages, queries, content, headers, and credentials are not included. At most 8 warning identifiers are emitted, with omittedWarningCount for the rest. No diagnostic file or remote log is created by this package.

Purpose, Search, Deep Search, and Agent

purpose is task intent, not a research-mode switch. Exa's Search best-practices guide recommends a required objective in model-facing tools, including ordinary Auto searches. Keep it short: name the implementation goal, exact dependency version, and useful sources. The tool sends it as objective in the same request. It does not send outputSchema, start Agent, or make a second API call.

Exa lists no separate objective charge. That is not a claim of zero processing overhead or identical latency. Auto is currently $7 per 1,000 requests up to 10 results; Deep types have separate prices. See pricing. JSON returned by this extension is a local structured tool contract, not Exa's paid synthesis feature.

web_fetch

Fetches page content with the active fetch provider. Results are cached in memory by canonical URL plus provider/config/fetch-affecting options.

Provider-specific parameters are exposed only for the configured provider. TinyFish supports purpose, format, links/images, selectors, per-URL timeout, and a seconds-based ttl alias. Exa supports the hours-based maxAgeHours alias. markdown.new supports method / retainImages. Firecrawl supports format, waitFor, mobile, structured location, and its existing maxAge alias. refresh: true also requests live content from Exa, TinyFish, and Firecrawl instead of only bypassing the local cache.

For Exa, explicit maxAgeHours must be an integer from -1 to 720, in both search and fetch calls. Larger maxAgeMs values remain valid: their provider-side hour conversion is capped at 720, which requests content at least as fresh as the caller requires. Sub-hour values round down, possibly to zero. The original millisecond limit still governs the local cache. Explicit invalid hour values are rejected even when refresh: true would otherwise override them.

Exa Contents responses are matched by requested document IDs before canonical URLs. Per-URL error statuses trigger fallback retrieval for only the failed or missing pages; they cannot borrow another URL's result or cache it under the failed URL. Failed pages are not cached. Identity-less legacy redirect results use positional matching only for a complete batch or a single unambiguous pair.

TinyFish and Firecrawl also retain the requested URL across redirects, including when a redirect lands on another URL in the same batch. url is the requested identity and fetchedUrl is the returned final URL.

Exa fallback retrieval uses TinyFish, then Firecrawl when their keys are configured, then markdown.new. Only structured failures or empty content trigger fallback, never error-like phrases in ordinary page text. Explicit Exa freshness is translated to TinyFish ttl or Firecrawl maxAge, rounding toward fresher content. A freshness-constrained request skips markdown.new because it has no equivalent control. Cached-only Exa (maxAgeHours: -1) never uses another provider; refresh: true overrides that mode to request live content without populating its cached-only cache key.

TinyFish links: true and imageLinks: true expose links and imageLinks arrays of absolute HTTP(S) URLs in each successful result. Invalid, credential-bearing, duplicate and excess links are omitted. Each array retains at most 50 URLs, with one shared 12,500-byte link budget across the call. omittedLinkCount and omittedImageLinkCount include validation and output-budget omissions. Link URLs remain complete, including useful fragments; these fields are also available on cached reads. Fetch content limiting preserves both URL identities and updates range.nextOffset for continuation.

library_search

Resolves packages, frameworks, SDKs, APIs, CLIs, and libraries to canonical library IDs.

Both Context7 tools prefer provider-enforced strict argument schemas when supported. Unsupported providers or schemas retain best-effort tool calling; local validation always applies. In Pi 1.4.0, Anthropic falls back because its strict schemas do not support these numeric bounds.

library_docs

Fetches current docs and code snippets for a library. Provide libraryId, or provide libraryName and the tool resolves the best match first.

Documentation output keeps complete snippets, including their code and token metadata. If the serialized result exceeds 50,000 bytes, whole snippets are omitted; omittedCodeSnippetCount and omittedInfoSnippetCount report exact counts. Smaller later snippets can still be returned. A message asks you to refine the query or lower limit; an empty snippet array with omission counts means the output budget was reached, not that no documentation exists. Library search similarly omits whole oversized candidates with omittedResultCount, preserving IDs and versions.

code_search

Finds practical code examples, implementation context, setup snippets, migrations, usage patterns, and error-message research through Exa Search Auto. The endpoint is /search, not the undocumented legacy /context.

Code search always sends:

{
  "type": "auto",
  "contents": {
    "highlights": true,
    "maxAgeHours": 0,
    "livecrawlTimeout": 15000
  }
}

There is no Lite, legacy fast/instant, Deep, cached-only, historical snapshot, beta, or Agent override. No summary, outputSchema, or synthesized answer is requested. It uses the Exa key regardless of the providers selected for general web search and fetch. Search responses are not locally cached.

maxAgeHours: 0 requests freshly fetched page content under Exa's documented contract. It does not guarantee that every newly published page is indexed, that a source matches the project's version, or that the provider's freshness can be independently attested: Search does not return a per-page verified crawl timestamp. publishedDate is an estimated publication date, not proof of a fresh crawl. No publication-date filter is imposed on code search, because current evergreen references can have old publication dates.

Only highlight evidence is returned. Metadata-only results and responses with only text/summary fallbacks are omitted and counted in omittedNoContentCount; the tool never substitutes cached-only retrieval when crawling fails. Errors surface without automatic retry or endpoint/mode fallback. The HTTP deadline is 45 seconds, including JSON body consumption; the provider's individual live crawl timeout is 15 seconds.

Results preserve complete source URLs, titles, excerpts, and publication dates when present, with exact result/context omission accounting within the 50,000-byte JSON limit. Character counts use JavaScript UTF-16 string length; prefixes do not split surrogate pairs. If evidence is incomplete, narrow the query or use web_fetch with refresh: true on a cited URL. Excerpts are not guaranteed to contain complete compilable functions.

Breaking migration: provide purpose for both Exa search tools, replace tokensNum/"dynamic" with contextTokens, and read results[].content instead of the old combined response. Legacy response/token metadata and aliases are removed rather than silently translated.

Cache and limits

web_fetch uses an in-memory cache for the current Pi process.

Cache keys include the provider, canonical URL, fetch-affecting parameters, relevant provider defaults, and an opaque HMAC-SHA-256 API-key/account scope with a random process-local key. Internal cache keys are never returned in tool output. refresh: true bypasses and replaces the cached entry.

Exa cache scopes also include fallback accounts and markdown.new defaults. Raw provider metadata is not retained by the fetch cache; only page fields and validated, requested link fields are stored. Oversized entries are served for the current call but are not cached.

Provider bodies are streamed through the incoming byte limit before JSON parsing or cache insertion. Requests that exceed it fail instead of returning partial documents. Text and JSON request deadlines include body consumption; markdown.new's 45-second deadline also applies when used as an Exa fallback. Provider API redirects are rejected, not followed with credentials.

Structured results and discovery

All five tools declare outputSchema and return objects through structuredContent. Codemode receives those objects directly; remove old JSON.parse(await tools.…()) wrappers. Direct calls still return useful JSON text with exactly the same bounded, redacted value. Renderer details remain compact and are not the script API.

Every result also allows the top-level fallback { truncated: true, message } when metadata alone cannot fit. Check this before reading ordinary fields. Both text and structured payloads stay within the same 50,000-byte JSON budget. Fetch slices retain range continuation; search results report omitted counts. Missing/invalid optional provider fields are omitted. Invalid required fields fail the call rather than yielding a guessed success object.

const search = await tools.web_search({ query: "Pi extension API", purpose: "Implement a tool with the installed Pi version; prefer official extension docs", numResults: 3 });
if ("truncated" in search) throw new Error(search.message);
const urls = search.queries.flatMap(q => q.results.map(r => r.url));
if (urls.length) {
  const pages = await tools.web_fetch({ urls: urls.slice(0, 10) });
  if ("truncated" in pages) throw new Error(pages.message);
  text(pages.results.filter(r => !("error" in r)).map(r => ({
    url: r.url, excerpt: r.content.slice(0, 500), nextOffset: r.range.nextOffset
  })));
}

Validation, whole-request/provider, malformed-output and cancellation failures throw: direct calls become Pi tool errors and scripts must use try/catch or Promise.allSettled. Individual fetch failures are successful result data with an error field, not isError: true. These tools do not return structured success objects alongside isError: true. A trusted result hook can replace that contract; annotations do not bypass hooks or approvals.

The namespace is web, not a name prefix: calls remain tools.web_fetch(…). await describeNamespace("web"), await describeTool("web_fetch"), and await searchTools("page content", { namespace: "web" }) work even with codemode.inlineBudget: 0. Independent reads can run in parallel. The advisory hints are read-only, non-destructive, idempotent, and open-world: repeated reads can return different content and incur provider charges.

Direct exposure remains the default, and codemode is optional. No package-level deferred/codemode exposure switch is added: it would make inactive direct tools nested-callable and change the meaning of disabling them. Use Pi's codemode.mode (on or only) and inlineBudget to reduce declarations while retaining the active-tool boundary. Optional research tools still require their provider keys. Explicit defaultTools -name entries are honored at late registration; reload restores Pi's active selection rather than re-enabling manually disabled tools. New optional tools activate when credentials first become available on reload, including existing positive defaultTools selections. Previously registered tools are not force-reactivated, even if settings still positively select them. The extension records only previously registered tool names in branch-local pi-web-kit:registered-tools session metadata, outside model context; it does not store activation choices or credentials. The record is refreshed before reload so navigating to older branch history cannot revive a manually disabled tool. Older sessions without this metadata preserve inactivity on their first reload, so they may require explicit activation or a restart. Malformed/unknown-version metadata uses the same conservative fallback. CLI exclusions remain host-enforced.

Privacy and security

pi-web-kit sends search queries and fetched URLs to the configured provider. Developer-search tools send library/doc queries to Context7 and code-context queries to Exa when those tools are enabled. Fetch providers may also receive provider-specific options. API keys are read from environment variables or local config files and are used only for provider requests.

The extension rejects non-HTTP(S) URLs and URLs with embedded username/password credentials. Returned data is projected to declared public fields; raw provider metadata, Context7 rules, cache keys and HTTP error bodies are not returned. Known API keys, URL credential patterns and Basic/Bearer authorization-header values are masked in text, structured data, renderer details and progress. This is not a general sensitive-data scanner: page content remains untrusted, and Pi retains caller-supplied arguments in its own transcript. Do not put credentials in queries or URLs.

Report security issues privately. See SECURITY.md.

Troubleshooting

Development

Requirements:

• Node.js >= 22.19.0
• npm

Common commands:

npm install
npm run check
npm test
npm audit --omit=dev
npm run pack:dry-run

Context7 strict-sampling checks

The SDK regression tests cover strict-capable OpenAI requests, unsupported routes, Anthropic's numeric-bound fallback, optional argument defaults, and local rejection before Context7 access:

node --import tsx --test packages/pi-web-kit/tests/strict-sampling.test.mjs

To repeat against an installed Pi host instead of the pinned development SDK, set PI_WEB_KIT_TEST_RUNTIME to its directory containing the pi-ai and pi-coding-agent packages:

PI_WEB_KIT_TEST_RUNTIME=/path/to/node_modules/@earendil-works \
  node --import tsx --test packages/pi-web-kit/tests/strict-sampling.test.mjs

For a small live smoke test, configure Context7 and model credentials, select a strict-capable model, and run from the monorepo root:

pi -ne -e ./packages/pi-web-kit --no-session --no-context-files \
  --tools library_search,library_docs --print \
  "Call library_search for Node.js with limit 1. Then call library_docs for the returned ID about node:fs readFile with limit 1. Report whether both calls succeeded."

Confirm that both calls succeed without schema errors. This is a paid model and Context7 smoke test, not a guarantee of provider support on every route.

Manual terminal smoke test

From the monorepo root, with Context7 and Exa credentials configured:

pi -ne -e ./packages/pi-web-kit

Ask Pi to run library_search for a library with multiple matches, then library_docs and code_search for a specific API question. Check that:

• Each running tool shows a short pending message without a percentage.
• Each completed result shows two summary rows rather than raw JSON.
• The expand-tools action (default: Ctrl+O) reveals candidate IDs and short version previews, the resolved docs library and sources, or ranked code URLs.
• Narrowing the terminal clips fields rather than adding wrapped output rows.
• Empty results and provider failures do not show a success heading.

Repeat in regular and fullscreen terminal modes when changing renderer layout. Automated tests cover mocked provider calls through the registered renderers, data-contract preservation, errors, output limits, Unicode, and narrow widths; this manual check covers the host's interactive expansion and redraw behavior.

Manual Exa comparison and smoke test

From the monorepo root, with EXA_API_KEY in the environment:

node --import tsx packages/pi-web-kit/tests/exa-evaluation.mjs --live

This opt-in, paid run sends 12 requests: three fixed public coding queries (Node.js streams, Pydantic v2 migration, React 19 actions) through production fresh-only code search, general Auto highlights, general Lite highlights, and Auto Dynamic Highlights with the documented Exa-Beta: dynamic-highlights-2026-08-28 header. It makes no retries or follow-up fetches. Lite and Dynamic are comparison paths for general web search, never code-search configuration options. All paths request 5 results with an 8,000-character local search-context budget, followed by the same public output projection and byte limiter used by the tools.

The JSON output includes latency, reported cost when available, raw/returned character counts, serialized bytes, and source excerpts for human inspection. approximateTokens is characters divided by four, not a tokenizer measurement. Review version/operation match, usable code, citations, and whether a follow-up fetch is needed. This is a small retrieval smoke test, not a downstream answer-quality benchmark or proof that a returned page matches a dependency version. The earlier legacy-Context comparison did not test this fresh-only policy and is not a release gate for migration away from that API.

The deterministic tests exercise all 16 general search/fetch provider combinations, every documented general search mode with freshness controls, strict code-search override rejection, source/budget boundaries, stale-text poisoning, credential masking, notices, HTTP failures, cancellation, and a stalled response body after headers. Root contract tests exercise real Pi direct and codemode dispatch, including rejection before provider access.

Contract references: Search, Highlights, and Code & Docs. The live OpenAPI is linked from those pages; check it again before changing beta defaults.

This package is source-distributed. Pi loads the TypeScript extension files directly via its extension loader.

Contributing

Contributions are welcome. See CONTRIBUTING.md for development workflow and pull request guidelines.

License

MIT. See LICENSE.