pi-lightpanda-web

Pi Lightpanda Web: web_search and web_fetch powered entirely by the local Lightpanda browser

Packages

Package details

extension

Install pi-lightpanda-web from npm and Pi will load the resources declared by the package manifest.

$ pi install npm:pi-lightpanda-web
Package
pi-lightpanda-web
Version
0.1.0
Published
Nov 14, 2026
Downloads
338/mo · 338/wk
Author
yugimob
License
AGPL-3.0-only
Types
extension
Size
285.9 KB
Dependencies
0 dependencies · 3 peers
Pi manifest JSON
{
  "extensions": [
    "./index.ts"
  ]
}

Security note

Pi packages can execute code and influence agent behavior. Review the source before installing third-party packages.

README

Pi Lightpanda Web

A pi extension that provides web_search and web_fetch, powered entirely by a local Lightpanda browser. It is a fork of pi-unsloth-webtools: the extraction pipeline, HTML→Markdown conversion, entity decoding, PDF layer, settings, and tests are derived from that project, but the network stack is replaced with Lightpanda.

There is no direct Node HTTP transport for HTML, no wreq-js browser-fingerprint tier, and no third-party rendering service. Every remote HTML page — whether fetched by web_fetch or used as a search result page — is loaded by the local browser. PDFs are the one documented exception: a narrow direct downloader fetches them for MuPDF extraction, and it can be disabled.

The package name is pi-lightpanda-web. The npm package pi-lightpanda is a different, unrelated project, so this package uses the -web suffix to stay unique.

Requirements

  • Node.js >= 22.19.0.
  • A Lightpanda binary. The extension checks at session_start and, when no renderer is configured, runs scripts/install-lightpanda.sh in the background; the first tool call waits for that install. There is no postinstall. Set PI_LIGHTPANDA_BIN or webRender.lightpandaPath / webRender.lightpandaCommand to use your own binary and suppress the automatic install.
  • Optional: mupdf for the full PDF engine. Without it, PDF extraction falls back to a limited regular-expression extractor.

Install

From a checkout:

pi install /path/to/pi-lightpanda-web

Once the package is published to npm, pi install npm:pi-lightpanda-web installs it under the unique -web name. Either way, add the path (or package name) to the packages array in ~/.pi/agent/settings.json. From the checkout:

npm install

The extension downloads Lightpanda automatically at the next Pi startup. To pre-install it yourself, run:

bash scripts/install-lightpanda.sh

Automatic renderer installation

At session_start, the extension checks whether a runnable Lightpanda is already available. If none is configured, it runs scripts/install-lightpanda.sh --no-configure in the background; the first web_fetch or web_search call waits for the same install rather than racing it. There is no postinstall and no bundled binary.

Explicit configuration is respected and suppresses auto-install: PI_LIGHTPANDA_BIN, webRender.lightpandaPath, webRender.lightpandaCommand, or webRender.lightpandaEnabled: false. The installer supports Linux (glibc; musl needs a source build), macOS, and Windows through WSL2. Native Windows has no Lightpanda binary. On Linux the installer can bundle a newer glibc into the install directory when the system one is too old, so Lightpanda 1.0.x runs without touching system libraries.

Tools

Both tools show their target in the pi TUI: web_search "query" and web_fetch <url>.

web_search

web_search renders each engine's result page with Lightpanda, parses the dump, and fuses the results with weighted reciprocal-rank fusion (k=60):

  • Default engines: Brave and Ecosia.
  • Results are deduplicated by canonicalized href (utm_* and tracking parameters stripped, fragments dropped, default ports removed), then capped per registrable domain (webSearch.maxPerHost, off by default).
  • A transient empty render is retried once inside the sweep budget. Engines still in flight are aborted once enough results have arrived.
  • websitePolicy is enforced before rendering and again on every result URL; when a policy restricts domains, the query is scoped with rotating site: filters and the sweep overfetches.
  • Failure messages are stable and documented below.

The engine set was verified live with Lightpanda 1.0.4 from a normal residential connection. Lightpanda identifies itself honestly and --user-agent rejects any value containing Mozilla, so some engines cannot be used:

Engine Status Backend
Brave Default Brave's own independent index
Ecosia Default Google-provided results (each result is labelled provider="GOOGLE"); may answer with a 403 bot challenge
Yahoo Opt-in Bing's index with Yahoo's ranking; in our runs it was much better than Bing for developer queries
Bing Opt-in Bing's index; in our runs it returned generic entity pages for developer queries (for example python.org instead of the 3.14 release notes, linux.org instead of io_uring documentation)
DuckDuckGo Unavailable Anomaly / bot challenge
Startpage Unavailable Anubis proof-of-work challenge
Google Unavailable Consent interstitial
Mojeek Unavailable Captcha

npm run engine:eval measures the engines against a hand-labelled developer query set. On that set: top-5 URL overlap (mean Jaccard) is 0.37 Brave/Ecosia, 0.35 Brave/Yahoo, 0.09 Bing/Ecosia, and 0.17 Bing/Yahoo. Brave and Ecosia are different indexes with moderate overlap; Bing's low overlap comes from returning generic pages, not from adding useful coverage. Yahoo is Bing-backed but ranked much better than Bing through Lightpanda, so it remains available as an opt-in fallback.

The same set exposes Bing's specificity problem. topic@5 (top-5 results whose URL or title contains a query-specific term such as 3.14, hnsw, or io_uring) is Brave 0.97, Yahoo 0.97, Ecosia 0.72, and Bing 0.25. Bing's domain-level precision (0.42) looks close to the others only because broad entity pages such as python.org still land on an expected domain; it is not answering the query. Ecosia answered 6 of 8 queries; the other two came back as 403 BotChallenge, and the sweep falls back to Brave for those. That is why the defaults are Brave + Ecosia, with Bing and Yahoo opt-in.

Use /search-config to enable or disable engines; Bing and Yahoo are available as opt-ins. At least one engine must stay on.

web_fetch

  • Renders the requested URL with lightpanda fetch --json --dump html --wait-until networkidle. --json supplies the real HTTP status, response headers, and final URL, so errors are reported as Failed to fetch URL: HTTP 404 Not Found instead of returning an error page as content.
  • Converts HTML to Markdown with the ported Studio pipeline: <article>/<main> scoping, hidden-element stripping, header link-density stripping, boilerplate removal, and page metadata (Title:, Author:, Date:, Site:).
  • Adds URL: and Rendered via Lightpanda (local browser; nothing sent to a third party). to every rendered page.
  • Local files are read directly (file:// URLs, absolute paths, ~/, ./): HTML is converted, text is returned raw, local PDFs are extracted with MuPDF, binary files are rejected. Lightpanda itself does not support file://, so this is the only non-browser read path.
  • Results are cached for 1 hour (128 entries, 128 MiB); a stale entry from the last 24 hours is served when a render fails. The cache is never used for blocked URLs or local files.
  • Remote PDFs are the one direct-network exception. When a URL looks like a PDF (.pdf, /pdf/, or format=pdf) or Lightpanda reports application/pdf, web_fetch downloads it with a DNS-pinned, policy-checked, redirect-validated direct request and extracts it with MuPDF. The downloader has no page or byte cap, handles gzip/deflate/brotli, and sends a normal browser User-Agent. Set webFetch.allowRemotePdf: false to disable the exception; PDFs then report (PDF content cannot be read by the local renderer) again. The PDF downloader uses a direct connection and does not apply proxy environment variables.
  • There is no character or page cap on web_fetch output. HTML, local files, and PDFs are returned in full; the tool does not truncate them. Only web_search has a result-count limit (maxResults).

Local and private access

Private/loopback/link-local targets are allowed by default, so http://localhost:3000 and LAN hosts work. Lightpanda blocks unsafe ports (for example port 1) and file://; local files still work through the fs path. Set webFetch.allowPrivateAddresses: false to pass --block-private-networks to Lightpanda and restore the SSRF guard; the direct PDF downloader then validates every resolved address with the same guard. Set webFetch.allowLocalFiles: false to refuse local paths.

Proxies

HTTPS_PROXY, HTTP_PROXY, and ALL_PROXY are honored when they name an HTTP(S) proxy: the URL is passed to Lightpanda's --http-proxy. NO_PROXY exclusions are applied before launching. SOCKS5 proxies (including Tor) are not supported by this fork, because Lightpanda's proxy flag is HTTP-only. The direct PDF downloader is an exception to this proxy handling: it connects directly and does not read proxy environment variables.

Configuration

Settings live in ~/.pi/agent/settings.json or .pi/settings.json (project overrides global).

{
  "piLightpanda": {
    "maxResults": 5,
    "timeoutMs": 60000,
    "allowPrivateAddresses": true,
    "allowLocalFiles": true,
    "allowRemotePdf": true
  },
  "webFetch": {
    "timeoutMs": 60000,
    "allowPrivateAddresses": true,
    "allowLocalFiles": true,
    "allowRemotePdf": true
  },
  "webSearch": {
    "engines": ["brave", "ecosia"],
    "engineWeights": { "brave": 1, "ecosia": 1 },
    "maxPerHost": 0
  },
  "webRender": {
    "lightpandaEnabled": true,
    "lightpandaPath": "/home/you/.local/share/pi-lightpanda-web/lightpanda/lightpanda-run"
  }
}
Key Default Description
piLightpanda.maxResults / webSearch.maxResults 5 Default maxResults for web_search (clamped 1–20)
piLightpanda.timeoutMs / webFetch.timeoutMs 60000 fetch, 300000 search Default timeoutMs when the tool param is absent (>= 1000)
piLightpanda.allowPrivateAddresses / webFetch.allowPrivateAddresses true Set false to pass --block-private-networks and refuse localhost/LAN targets
piLightpanda.allowLocalFiles / webFetch.allowLocalFiles true Set false to refuse file://, absolute, ~/, and ./ paths
piLightpanda.allowRemotePdf / webFetch.allowRemotePdf true Set false to disable the direct PDF downloader and keep Lightpanda as the only network transport
webSearch.engines / piLightpanda.engines ["brave","ecosia"] Enabled engines; unknown names are ignored, an empty match falls back to defaults
webSearch.engineWeights / piLightpanda.engineWeights all 1 Positive per-engine RRF weight
webSearch.maxPerHost / piLightpanda.maxPerHost 0 (no cap) Most results one registrable domain may contribute
webRender.lightpandaEnabled / piLightpanda.lightpandaEnabled true Disable the local renderer entirely
webRender.lightpandaPath / piLightpanda.lightpandaPath installed launcher, else lightpanda on PATH Path to the Lightpanda binary
webRender.lightpandaCommand / piLightpanda.lightpandaCommand none Command prefix, for WSL (["wsl.exe","-e","<path>"]) or containers; overrides lightpandaPath

Environment overrides:

  • PI_LIGHTPANDA_BIN — renderer binary path.
  • PI_LIGHTPANDA_CACHE_DIR (or legacy PI_UNSLOTH_CACHE_DIR) — fetch cache directory.
  • PI_CODING_AGENT_DIR / PI_AGENT_DIR — global settings directory.
  • PI_LIGHTPANDA_E2E=1 / PI_LIGHTPANDA_E2E_BIN — enable and point the live test suite.

Troubleshooting

Symptom Exact string Action
Empty query No query provided. Pass query.
Cancelled Search cancelled. / Failed to render URL: cancelled. Retry only if unintended.
Search timeout Search failed: the search engines (...) did not respond within N seconds. Retry, narrow the query, or raise timeoutMs.
No results No results found. Rephrase, or enable another engine.
Policy filtered everything No results found within the website access limits. Widen websitePolicy.
Blocked URL Blocked: ... Respect it.
Missing renderer (Lightpanda is not installed; run bash scripts/install-lightpanda.sh to enable web_fetch) Install Lightpanda or set PI_LIGHTPANDA_BIN.
Local render failure Failed to render URL: Lightpanda exited with code N. Check the binary by hand.
Bot challenge Failed to render URL: the page served a bot challenge instead of content. Hard anti-bot walls cannot be bypassed by Lightpanda's honest user agent.
HTTP failure Failed to fetch URL: HTTP 404 Not Found Fix the URL.
Remote PDF disabled (PDF content cannot be read by the local renderer) Set webFetch.allowRemotePdf: true, or use a local PDF.
Cache fallback Served from cache / STALE cache from YYYY-MM-DD The render failed; output is the cached copy.

Development

npm install
npm run typecheck
npm run test:unit
PI_LIGHTPANDA_E2E=1 PI_LIGHTPANDA_E2E_BIN="$(bash scripts/install-lightpanda.sh --print-path)" npm run test:live
npm run engine:eval
npm run lint
npm run check:package

npm run test:unit runs the offline suite with stubbed renderer processes. npm run test:live needs a real Lightpanda binary and skips itself unless PI_LIGHTPANDA_E2E=1 is set; it also exercises the direct PDF downloader. CI runs both on Linux, with the unit suite also on Windows and Node 22.19/24/26.

License

The ported logic derives from Unsloth Studio (AGPL-3.0-only), so this package is released under the same AGPL-3.0-only license.