pi-web-access

npm · Pi package directory

Search the web, fetch pages as Markdown and inspect cached results in the Pi coding agent, without supplying search API keys. Revisit long results in chunks or find a passage without fetching the page again.

No GitHub clone, PDF, video or curator UI.

Security: Pi extensions run with your full system permissions. Search and content requests hit the public internet from this machine. A local SSRF gate blocks private/loopback/link-local/special-use IPs, URL credentials, and non-http(s), rechecks every redirect, and pins DNS lookup to the connecting socket (no second resolve). Install only from a source you trust.

Quick example

After installing, ask Pi:

Search nodejs.org for the Node.js test runner documentation.
Fetch the relevant documentation page as Markdown, then use
get_search_content with the fetch responseId to find "node --test".
Include the source URL in your answer.

The tool sequence is search → fetch → inspect the cached text:

1. web_search:

   {
     "query": "Node.js test runner documentation",
     "domainFilter": ["nodejs.org"],
     "numResults": 3
   }

2. fetch_content, using a relevant result URL (for example):

   { "url": "https://nodejs.org/api/test.html" }

3. get_search_content, using the ID returned by that fetch:

   { "responseId": "REPLACE_WITH_FETCH_RESPONSE_ID", "findText": "node --test" }

Replace the placeholder with the actual responseId. These are example arguments, not captured results. Search and fetch need public network access; the final lookup reads the local cache, whose entries expire after one hour. Provider availability and limits still apply even without API keys.

Design trade-off: HTTP fetching and Readability extraction avoid launching a browser, but do not execute page JavaScript. Use pi-browser when content needs interaction or rendering. See fetch handling, HTML extraction and extraction tests.

Install as a Pi extension

Requires Node.js 22.19+ and Pi; native loading is tested with Pi 1.4.0. This README tracks repository source. npm packages and Git tags may be behind it. Version 0.3.0 removes the standalone MCP server and its executable. Finish old gateway work and remove its web service/package entries before loading this extension; do not load both backends. Existing cached results remain usable.

pi install npm:@kvidzibo/pi-web-access

Git:

pi install git:github.com/kvidzibo/pi-web-access

Local checkout — Pi adds the path only; it does not run npm install for local sources:

cd /absolute/path/to/pi-web-access
npm install --omit=peer
pi install /absolute/path/to/pi-web-access

pi install of npm or git sources installs the HTML extraction dependencies. Pi supplies @earendil-works/pi-coding-agent and typebox; no compilation or standalone server is needed. Exa's outbound search protocol is still used directly over HTTP; no local MCP server or MCP SDK is involved.

Do not also list this path in settings.json extensions — package load is enough.

Then /reload (or restart Pi).

HTTP handling

• Unsupported final HTTP status codes (outside 200–599), protocol upgrades, and response-conversion failures become request errors instead of terminating Pi. HEAD, 204, 205, and 304 responses have no body.
• Gzip, deflate, and Brotli are decoded as streams (up to three stacked encodings). The 2,000,000-byte response limit applies after decoding. Invalid encodings and truncated bodies are errors; unused streams are canceled.
• The 30-second request timeout and cancellation include waiting for DNS. A canceled lookup cannot start an HTTP connection afterward.
• Redirects discard cross-origin credentials and stale Host headers. POST-to-GET redirects also discard the body and its headers; HEAD remains HEAD on a 303.

Readable Markdown resolves relative links and image sources against the final fetched URL, including a valid HTML <base> element. Fragment anchors and non-HTTP links remain links only; extraction never fetches them.

Search and errors

Use queries for batches of up to 4 searches (500 characters per query), or urls for up to 20 fetched URLs (2,048 characters per URL). numResults accepts 1–20 results per query, defaulting to 5.

Domain filters are sent to both providers and enforced on returned URLs; multiple allowed domains use OR. DuckDuckGo uses its date filter for recencyFilter; Exa receives a relative-date query hint, not a guaranteed date constraint. Empty matching results are identified explicitly.

A failed query no longer discards successful queries from the same batch. A provider timeout ends that query without fallback; later queries still run. Caller cancellation stops the batch and does not cache canceled results. All-failed batches and invalid tool arguments throw errors in Pi; partial results remain available. All-failed batch errors include a cache response ID for inspection with get_search_content.

Cache

~/.pi/agent/web-access-cache (dir 0700, files 0600). Entries expire after 1 hour. Expired and over-quota files are deleted when the extension loads and on store/get. Inactive leftover files can remain until the next load.

get_search_content returns up to 30,000 characters per page; use offset and limit to continue. To search instead, pass findText without offset or limit.

Caps: 128 entries, 128 MiB. Case-insensitive and fuzzy match offsets refer to the original UTF-16 text, even when Unicode lowercasing changes its length.

Development

index.ts is the native Pi entry point; src/tools.ts registers tools. Search providers and fallback live in src/search/; URL policy, pinned transport, redirects, and bounded body reading are separated in src/network/. Page extraction and caching live in src/pages.ts, src/html.ts, and src/cache.ts. Tests are in tests/, with Pi loader checks in tests/load/.

npm ci --omit=peer
npm test          # unit + native factory load (needs `pi` on PATH)
npm run test:unit # no Pi required

CI runs both suites with Pi 1.4.0. The default suite uses fixtures and loopback servers, never the public network. Optional live smoke:

WEB_ACCESS_LIVE=1 node --test --experimental-strip-types tests/live.test.ts