@nbafrank/pi-webfetch

Keyless web search and fetch tools for pi (an @earendil-works coding agent).

Defaults need no API key, no login, no Ollama:

• Search → DuckDuckGo (HTML parser, with a JSON fallback).
• Fetch → Jina Reader (r.jina.ai), with a raw GET + HTML→text fallback.

Higher-quality providers (SearXNG, Tavily, Brave) are used automatically when their config is present. Everything is implemented on Node built-ins only (fetch, AbortSignal, URL, URLSearchParams) so the core runs under node --experimental-strip-types and is fully unit-testable with zero deps.

---

Architecture

src/core.ts        dependency-free engine  (search / fetch / providers / parsing)
src/extension.ts   pi layer               (registers web_search + web_fetch tools)
scripts/smoke.ts   offline + live smoke test
tsconfig.json      strict typecheck of core + smoke (no external deps)
tsconfig.extension.json   typecheck the extension against pi's real types

core.ts has no pi/typebox imports, which is what lets it be tested and reused outside pi. extension.ts is the only place that imports @earendil-works/pi-coding-agent and typebox.

---

Providers

Search (best-then-fallback, in this order)

provider	when enabled	key required
searxng	SEARCH_SEARXNG_URL set	no*
tavily	TAVILY_API_KEY set	yes
brave	BRAVE_API_KEY set	yes
duckduckgo-html	always (default)	no
duckduckgo-json	always (fallback when HTML fails)	no

* SearXNG format=json may need your instance to allow JSON / a shared secret.

Fetch

provider	when used	key required
jina	default	no
direct	fallback when Jina unavailable or fails	no

---

Environment variables

var	purpose	default
SEARCH_SEARXNG_URL	SearXNG base URL	— (disabled)
SEARCH_SEARXNG_SECRET	SearXNG anonymous-search secret	—
TAVILY_API_KEY	Tavily key	— (disabled)
BRAVE_API_KEY	Brave Search key (X-Subscription-Token)	— (disabled)
SEARCH_PROVIDER	force a search provider (duckduckgo|searxng|tavily|brave)	auto
FETCH_PROVIDER	force a fetch provider (jina|direct)	auto
SEARCH_TIMEOUT_MS	per-request timeout (ms)	15000
FETCH_TIMEOUT_MS	per-request timeout (ms)	30000
MAX_FETCH_CHARS	max chars returned by fetch	20000

---

Retry & backoff

httpGet retries transient failures with capped exponential backoff:

• network errors and HTTP 429/5xx → retry, delay retryBaseMs * 2**attempt (default base 400ms, default retries=3 → 3 attempts total).
• honors the Retry-After response header, capped at 8 s.
• abortable: a cancelled AbortSignal stops mid-backoff and surfaces an abort error.
• hard errors (404, 422, auth failures) and JSON-parse failures do not retry — they fail fast so the orchestrator falls through to the next provider.

---

Core API

import { search, fetch, type SearchItem, type FetchResult } from "./core.ts";

const results: SearchItem[] = await search({
   query: "node fetch api",
   maxResults: 5,             // default 5
   provider: "searxng",       // optional: "duckduckgo" | "searxng" | "tavily" | "brave"
   signal,                    // optional AbortSignal
   onLog: (stage, ok) => {},  // optional progress: "search:brave" | "fetch:jina" ...
});
// SearchItem = { title: string; url: string; content: string }

const page: FetchResult = await fetch({
   url: "https://example.com",
   provider: "jina",          // optional: "jina" | "direct"
   signal,
});
// FetchResult = { title: string; content: string; links: string[] }

Search results are de-duplicated by URL (deduped by origin+path, self-hosts like duckduckgo.com filtered out) and snippets are bounded per result to avoid cross-block contamination.

---

pi extension (tools)

src/extension.ts registers two model-facing tools that wrap the core:

• web_search — { query, max?, provider? } → numbered results (title, url, bounded snippet).
• web_fetch — { url, provider? } → page title + readable text + linked pages.

Both pass through the agent's abort signal, cap their context size, and return graceful "failed" text on error instead of throwing.

Install

From npm (published package):

# user-global    (all pi sessions)
pi install npm:@nbafrank/pi-webfetch

# or project-local    (this repo only) → writes to .pi/settings.json
pi install -l npm:@nbafrank/pi-webfetch

# try it for the current run only, without installing
pi -e npm:@nbafrank/pi-webfetch

pi installs the package (and runs its npm install for dependencies), then auto-discovers its pi.extensions on the next session. Manage it with pi list, pi update npm:@nbafrank/pi-webfetch, or pi remove npm:@nbafrank/pi-webfetch.

Or, develop locally without publishing — drop / symlink src/extension.ts into a pi extensions directory for auto-discovery + hot-reload (pi supports both .ts and .js):

# user-global  (all pi sessions)
ln -s "$(pwd)/src/extension.ts" ~/.pi/agent/extensions/webfetch.ts

# or project-local  (this repo only)
ln -s "$(pwd)/src/extension.ts" .pi/extensions/webfetch.ts

That's it for running it: pi's extension loader (jiti) compiles the .ts on the fly and resolves @earendil-works/* + typebox from the running pi install automatically, so you do not need npm install or the link:pi symlinks to use the tools — the link step below is only for a local type-check.

---

Build, typecheck, test

Requires Node ≥ 22.6 (--experimental-strip-types).

npm install                         # devDeps: typescript, @types/node

npm run typecheck                   # strict TS check of core + smoke (no external deps)
npm run typecheck:extension         # type-checks the extension vs pi's REAL types
npm run smoke                       # offline assertions + live keyless DDG search + fetch

typecheck:extension needs the @earendil-works/* + typebox types. They are not on npm (they ship inside pi-coding-agent), so a pre-hook (pretypecheck:extension -> scripts/link-pi.mjs, also npm run link:pi) links them from your global pi install. npm install prunes these extraneous symlinks, so the hook rebuilds them automatically before the check.

npm i -g @earendil-works/pi-coding-agent    # if pi isn't installed globally yet
npm run link:pi                              # (re)link types from the global install