@laboon85/pi-web-access

Web search and web fetch for the pi coding agent, built around a tiered provider fallback chain.

Packages

Package details

extension

Install @laboon85/pi-web-access from npm and Pi will load the resources declared by the package manifest.

$ pi install npm:@laboon85/pi-web-access
Package
@laboon85/pi-web-access
Version
0.9.1
Published
Nov 17, 2026
Downloads
556/mo · 556/wk
Author
laboon85
License
MIT
Types
extension
Size
213.8 KB
Dependencies
4 dependencies · 2 peers
Pi manifest JSON
{
  "extensions": [
    "./dist/index.js"
  ]
}

Security note

Pi packages can execute code and influence agent behavior. Review the source before installing third-party packages.

README

@laboon85/pi-web-access

Web search and web fetch for the pi coding agent, built around a tiered provider fallback chain.

Both tools share one orchestrator. A provider that fails — quota, auth, transport, timeout, malformed response, or an unusable result — hands off to the next one in line. The chain also understands cooldown: a provider marked as cooling down is demoted to the tail of its own tier without ever being dropped. The shipped build does not yet mark providers, so today the configured order always stands.

Tools

web_search

Argument Required Meaning
query yes Search text
max_results no Results to return; capped globally

web_fetch

Argument Required Meaning
url yes Public HTTP(S) URL

Returns a text block for textual content, an image block for image/png, image/jpeg, image/gif and image/webp (image/jpg is accepted as an alias), and a content-type error for anything else. What textual HTML becomes — GFM markdown or the untouched body — is decided by the fetch provider chain, not by the call.

Providers

Provider Search Fetch Key
local — yes none (HTML → GFM markdown, in-process)
pandoc — yes none (pandoc on PATH; fidelity upgrade)
raw — yes none (body handed back untouched)
exa-free yes — none (Exa hosted MCP, rate limited per IP)
bing yes — none (HTML results)
searxng yes — none (SEARXNG_URL)
bocha yes — BOCHA_API_KEY
tavily yes yes TAVILY_API_KEY (fetch side is Extract)
exa yes — EXA_API_KEY
brave yes — BRAVE_SEARCH_API_KEY
jina yes yes optional JINA_API_KEY (keyless on fetch)
firecrawl yes yes optional FIRECRAWL_API_KEY (keyless on fetch)
scrapfly — yes SCRAPFLY_API_KEY
scrapingant — yes SCRAPINGANT_API_KEY
scrapingbee — yes SCRAPINGBEE_API_KEY
scraperapi — yes SCRAPERAPI_API_KEY
zenrows — yes ZENROWS_API_KEY

Defaults with no configuration at all: search uses exa-free, fetch uses local.

The three keyless fetch providers read the URL themselves — no third party sees it — and differ only in what HTML becomes:

  • local renders markdown in-process (DOM parse, GFM tables, headings, links resolved against the final URL). Always available; the floor of every chain.
  • pandoc converts with the pandoc binary for fidelity the in-process converter cannot reach, complex tables especially. The binary must already be on PATH — this package never downloads or installs one — and a missing or failing binary is an ordinary provider failure the chain falls back from.
  • raw hands back the body exactly as the server sent it. It replaces the old raw flag on the tool: original markup is a chain decision now.

A chain that prefers fidelity and keeps raw markup reachable looks like "fetchProviders": ["pandoc", "local", "raw"].

The remote readers come prewired — endpoint, auth header or query-parameter key, and the response path that carries the markdown — so adding one to the chain is a one-word edit. Firecrawl and Jina serve a keyless tier: they work with no FIRECRAWL_API_KEY / JINA_API_KEY at all, and setting one raises the limits (their keys are optional; every other reader needs its key). Reachability is yours to judge — some networks block some of these hosts, and a reader that cannot be reached fails like any other provider while the chain moves on.

Configuration

One JSON file. $PI_PKG_CFG_DIR wins if set; otherwise <agent dir>/pi-pkg-cfg/pi-web-access/config.json, where <agent dir> is $PI_CODING_AGENT_DIR or ~/.pi/agent.

{
  // Results returned by web_search. Both the default and the ceiling: the model
  // may ask for fewer, never more. 1..20, default 10.
  "maxResults": 10,

  // Search chain, in preference order. Grouped by tier internally.
  "searchProviders": ["tavily", "exa", "firecrawl", "exa-free", "bing"],

  // Fetch chain, in preference order. Default: ["local"]. "pandoc" needs the
  // binary on PATH and falls back when it is missing; "raw" skips conversion.
  "fetchProviders": ["pandoc", "local", "raw"],

  "apiKeys": {
    "tavily": "tvly-...",
    "exa": "exa-...",
    "firecrawl": "fc-...",
  },

  // Optional per-capability tier overrides. Providers default to tier 1 except
  // the keyless search fallbacks (exa-free, bing), which default to tier 2.
  "tiers": {
    "search": { "bing": 3 },
    "fetch": { "jina": 1 },
  },

  // HTTP(S) proxy for every outbound request. Falls back to HTTPS_PROXY,
  // HTTP_PROXY, ALL_PROXY; honours NO_PROXY. SOCKS is not supported.
  "proxy": "http://127.0.0.1:7890",
}

Environment variables win over apiKeys for the keys listed above. Configuration is re-read on every call, so an edited chain takes effect immediately. Only maxResults is fixed at load time, because it is part of the tool schema.

Declaring your own provider

customProviders takes one entry per capability, so each side keeps its own rules. A search entry describes a JSON API; a fetch entry points at a remote reader.

"customProviders": {
  "search": {
    "my-engine": {
      "url": "https://api.example.com/search",
      "method": "POST",
      "auth": { "header": "authorization", "prefix": "Bearer " },
      "keyEnv": "MY_SEARCH_KEY",
      "body": { "q": "{query}", "limit": "{maxResults}" },
      "resultsPath": "data.items",
      "fields": { "title": "name", "url": "link", "snippet": "summary" },
      "tier": 1,
      "maxResults": 10
    }
  },
  "fetch": {
    "my-reader": {
      "url": "https://reader.example.com/{url}",
      "auth": { "header": "x-api-key", "prefix": "" },
      "keyEnv": "MY_READER_KEY",
      "fetch": { "textPath": "data.markdown", "titlePath": "data.metadata.title" },
      "tier": 2
    }
  }
}

Placeholders: {query}, {queryEncoded} (for a path segment), {maxResults}, {url} (percent-encoded), {urlRaw} (for a JSON body field, where encoding would corrupt the target), {apiKey} (for vendors that take the key as a query parameter; a missing key empties the slot), {baseUrl} (taken from baseUrls).

Rules, all enforced by name so one bad entry does not take the file down:

  • a name may not shadow a built-in provider of the capability it declares for — the fetch side ships firecrawl, jina, tavily, scrapfly, scrapingant, scrapingbee, scraperapi and zenrows, so those names are taken there — and the same custom name may not be declared for both capabilities
  • auth requires keyEnv; a custom provider reads the environment only, never apiKeys, so a key in this file can never be sent to an endpoint that did not ask for it
  • a search entry requires resultsPath; a POST entry requires body
  • tier is any positive integer; tiers need not be contiguous

Built-in vendors use exactly this shape in code rather than living in this file: their field mappings follow the vendors' APIs, so a renamed field is fixed by a release instead of silently rotting in every copy.

/web-access-status prints the active chain — each member's tier and key source, never the key itself; --all adds the providers outside the chain.

Notes

  • web_fetch refuses private, loopback and otherwise non-public addresses, and re-checks every redirect hop (at most five).
  • When a proxy is configured the DNS-level check is skipped for that transport: the proxy resolves names itself, so the local pre-resolution result would not describe the address actually contacted. Redirects are still re-checked.
  • Only http:// and https:// proxies work.
  • Every provider except the keyless local readers (local, pandoc, raw) sends your URL or query to a third party. Adding one to a chain is an explicit opt-in.

Development

npm install
npm run typecheck     # tsc --noEmit
npm run lint          # eslint
npm run format        # prettier --write
npm run test          # vitest, no network access
npm run build         # tsc -p tsconfig.build.json -> dist/
npm run smoke         # import dist/ and assert the default export
npm run pack:check    # assert the npm tarball contents
npm run build -- --watch   # keep dist/ fresh while using a local install

The published artifact is compiled JavaScript plus .d.ts. pi loads dist/index.js, so a local install needs a build first.

Releasing

Publishing uses npm trusted publishing: the workflow exchanges a GitHub OIDC token for a short-lived credential, so no npm token is stored in the repository, and provenance is attached automatically.

Once per package, on npmjs.com:

  • Release workflow must be registered as a GitHub Actions trusted publisher with workflow filename release.yml, and npm publish must be explicitly allowed — new configurations default to staged publishing only.
  • A new configuration has to complete its first publish within two days, and the package must already exist to be registered, so the very first release is published manually:
npm login
npm publish --access public

After that, releases are tagged and published by the workflow:

git tag v0.7.0
git push origin v0.7.0

The workflow refuses to publish when the tag and package.json disagree, and prepublishOnly re-runs the whole gate before the registry sees anything.

Acknowledgements

Portions of this project are derived from pi-web-search by byte, licensed under the MIT License. Copyright (c) 2026 byte. The full MIT text is reproduced in LICENSE.

Thanks also to the pi project for an extension surface that makes a package like this possible, and to the maintainers of every service listed under Providers for keeping a usable free tier alive.