pi-unsloth-webtools

A pi extension providing web_search, web_fetch, and web_render tools. It began as a port of the Unsloth Studio codebase (unslothai/unsloth, studio/backend/core/inference/); the engine, extraction, and PDF layers are still derived from it, but the package is no longer behavior-identical to Studio — it enables local file and private-address fetching by default and adds a fetch cache, Wayback fallbacks, page metadata, a third-party rendering tool (web_render), and other behavior Studio does not have. See Known differences from Studio. The unsloth in the name marks provenance, not affiliation.

Install

pi install npm:pi-unsloth-webtools

or add npm:pi-unsloth-webtools to the packages array in ~/.pi/agent/settings.json.

To install from source instead:

pi install /path/to/pi-unsloth-webtools

What it does

All three tools display their target in the TUI tool row: web_search "query", web_search <url> in url mode, web_fetch <url>, and web_render <url>.

web_search

Mirrors Unsloth Studio's web_search tool:

• Searches exactly like Studio's pinned ddgs==9.14.4 DDGS.text(): the same seven engines (duckduckgo, brave, google, mojeek, yahoo, yandex, wikipedia; bing is disabled upstream), the same provider deduplication, href-dedupe aggregator with frequency ordering (hrefs are canonicalized first — utm_*/tracking parameters and fragments are dropped and the URL is re-serialized, collapsing host-case, default-port, and trailing-slash variants — so the same page found via different tracking links collapses), and the same SimpleFilterRanker re-ranking. Formats results identically: Title: / URL: / Snippet: blocks separated by ---, ending with the hint to pass {"url": "<URL>"} to read a full page.
• Accepts an optional url parameter; when given, fetches that page's text instead of searching (optionally truncated with maxChars). An HTTP 403 on that fetch falls back to web_render when the tool is enabled.
• Rate-limit, timeout, and empty-result messages mirror Studio's _search_failure_message.
• Transient engine failures (network errors or null responses) are retried once with a short backoff inside the same timeout budget (a retry that cannot fit in the remaining budget is skipped); timeouts and cancellations are never retried.
• Sweeps stop as soon as enough results are gathered: engines still in flight are aborted instead of being allowed to run to their timeout.

web_fetch

Port of Studio's _fetch_page_text / _fetch_url_raw pipeline:

• URL scheme normalization (bare hosts like google.com become https://google.com).
• URL validation: http/https only, no credentials or encoded hostnames, hostname/port checks (any port 1–65535 is permitted; SSRF protection is enforced at the resolved-IP layer, not by port allowlists). Canonical public IPv4 literals are accepted like IPv6 literals; private literals are blocked at the resolved-IP layer unless webFetch.allowPrivateAddresses is enabled (the default; non-canonical numeric encodings like 0x7f.0.0.1 / 013.0.0.1 / 2130706433 are rejected by URL validation before DNS either way).
• Local access by default: private/loopback/link-local targets (localhost dev servers, LAN hosts) and local files (file:// URLs, absolute, ~/, and ./ paths) are fetched directly — local PDFs run the same MuPDF extraction, local HTML gets the same Markdown conversion and metadata prefix, and the same byte caps and truncation notices apply. Opt out with webFetch.allowPrivateAddresses: false to restore the resolved-IP SSRF guard, or webFetch.allowLocalFiles: false to refuse local paths; both are settings, not tool parameters.
• DNS resolution with SSRF protection: every resolved address is validated against private/loopback/link-local/CGNAT/documentation/multicast/reserved ranges (unless webFetch.allowPrivateAddresses is enabled, the default), then the validated IP is pinned for the connection (custom lookup + SNI servername), so DNS cannot rebind between validation and fetch; resolution shares the caller's abort signal and the overall deadline, so a stuck resolver cannot outlive the fetch. When a host publishes both IPv4 and IPv6 addresses, IPv4 is preferred (broken IPv6 routes cannot stall a fetch), and a connection failure falls back to the next validated address for the same host before giving up.
• Transient DNS failures (EAI_AGAIN, resolver timeouts, connection refusals) are retried once with a short backoff inside the same deadline, so a brief resolver hiccup does not fail a fetch; the deadline abort cuts a retry short when no budget remains.
• Proxy environment variables are honored when they name a SOCKS5 proxy: HTTPS_PROXY / HTTP_PROXY / ALL_PROXY (with NO_PROXY exclusions) tunnel the pinned connection, so a Tor-mode agent routes web_fetch and web_search url mode through its exit. socks5h is treated like socks5: the host is resolved locally for the guard and the pinned IP is what the proxy connects to. Other proxy schemes are ignored (direct connection).
• GitHub repo root pages are rewritten to the unauthenticated README API (Accept: application/vnd.github.raw+json), falling back to the raw README URL (raw.githubusercontent.com, no API rate limit) and then to the HTML page on failure. returning error-page content.
• Up to 5 total requests (initial fetch + up to 4 HTTP redirects or <meta http-equiv="refresh"> refreshes), each re-validated and re-resolved against the same rules.
• 512 KiB download cap (10 MiB for PDFs), overall deadline + per-hop socket timeouts, abort-aware (signal cancels mid-flight). Fetches cut off by a download cap are marked with a trailing truncation notice, so a partial page is not mistaken for a complete one.
• Responses sent with a Content-Encoding of gzip, deflate, or brotli (servers that ignore the Accept-Encoding: identity request) are decompressed while streaming, so the download caps bound the decoded page text (a gzip'd PDF still gets the 10 MiB PDF budget via magic sniffing on the decoded head) and a stream cut by the cap or a mid-stream decode failure returns the readable decoded prefix with the truncation notice instead of a binary-content error. A 64 MiB decoded-output cap bounds a compressed bomb.
• Long fetches and searches report a short progress note to the session before they start, so slow tool calls are not silent.
• PDF text extraction via the official MuPDF.js engine (the same C library pymupdf wraps): object streams, all filters, ToUnicode fonts, encryption detection, and a pymupdf4llm-style markdown layer (headings, bold/italic, code fences, links, tables), running header/footer and page-number stripping, with Studio's corrupted/incomplete fallback to plain text.
• Content sniffing: MIME allow/deny, binary magic signatures, PDF magic detection, and charset decoding (BOM sniffing for UTF-8/16/32 first, then the declared charset, <meta charset> sniffing for CJK and Windows/ISO encodings, cp1252 rescue for mislabeled single-byte pages).
• HTML → Markdown conversion ported from Studio's dependency-free _html_to_md.py: headings, links, emphasis, lists, tables, blockquotes, code fences, entity decoding; hidden-element stripping (hidden, aria-hidden, inline styles); <article>/<main> main-content scoping with link-density header stripping; boilerplate-line removal.
• No page-size budget: fetched pages and PDFs are returned in full (Studio's window-aware cap is deliberately dropped; the optional maxChars parameter still truncates when given, on web_fetch and on web_search's url mode). The 512 KiB / 10 MiB download caps still bound the raw fetch.
• HTML entity decoding replicates CPython's html.unescape (full 2,231-entry HTML5 table, longest-prefix rule, Windows-1252 numeric mappings), matching Studio byte-for-byte.
• Fetched HTML pages are prefixed with the decoded document <title>, so the model can see which page it is reading. Author: (meta name=author / article:author / dc.creator), Date: (article:published_time / dc.date / date) and Site: (og:site_name / application-name) lines are added when declared, so the model can judge recency and provenance.
• A direct fetch refused with HTTP 403 is retried through web_render automatically when that tool is enabled; the rendered page is prefixed with a note saying so. When web_render is disabled or the render also fails, the original Failed to fetch URL: HTTP 403 ... is returned.

web_render

Renders a public page to Markdown through the third-party Jina Reader (r.jina.ai) for pages that need JavaScript to render:

• Every target is validated and resolved locally first: http/https only, and any private, loopback, link-local, or otherwise non-public address is refused. Local files are refused on this path regardless of webFetch.allowPrivateAddresses / webFetch.allowLocalFiles, because the URL is sent to Jina.
• unslothWebTools.jinaApiKey (or webRender.jinaApiKey, or the JINA_API_KEY environment variable) raises the Reader's rate limits; without a key it still works at Jina's free limits.
• Output is Markdown prefixed with Title: / URL: lines and a Rendered via the Jina Reader provenance line. An optional maxChars truncates, like web_fetch.
• Keyless Reader requests are rate-limited per outgoing IP; see Companion: rotating exit IPs.
• Enabled by default. Disable it with /webtools-config (webRenderEnabled in ~/.config/pi-unsloth-webtools/config.json), which deactivates the tool for the session.
• Also used automatically when a web_fetch or web_search url-mode fetch is refused with HTTP 403.

Known differences from Studio

• Local access: Studio validates every resolved address against non-public ranges and fetches public web content only. This port permits private/loopback/link-local targets and local files (file:// URLs and absolute, ~/, ./ paths) by default; opt out with webFetch.allowPrivateAddresses: false and webFetch.allowLocalFiles: false to restore Studio's behavior.
• Third-party rendering: the extra web_render tool asks the Jina Reader (r.jina.ai) to fetch the page, so the target URL leaves the machine. Studio has no third-party rendering path. This path always refuses local files and non-public addresses, regardless of the local-access settings. A direct fetch refused with HTTP 403 is retried through it automatically when enabled, so those targets also leave the machine in that case.
• PDF styling: MuPDF.js exposes one font per line, so mixed-style lines style the whole line instead of per-span; superscript, subscript, underline, strikeout, and highlight markers are not emitted. Tables use a conservative text-grid detector: aligned text tables are detected, drawn-rule-only tables are not.
• Running headers and footers: lines repeated at the same page-edge position on at least half the pages (two pages minimum) are dropped from the markdown layer, as is any numeric-only line at a fixed edge position where page numbers appear on at least half the pages (so a one-off number sharing that position is dropped too, while fused labels like Page 3 of 12 survive). Studio and pymupdf4llm return them verbatim.
• Search engines: Node's fetch TLS fingerprint differs from ddgs's primp impersonation, so Google/Brave/Yahoo/Yandex may block or serve consent pages more aggressively (a blocked engine simply contributes no results). User agents are a fixed browser set plus ddgs's Android Google UA generator, not fake_useragent's database.
• Empty sweeps: ddgs 9.14.4 raises the last engine exception; this port reports a timeout whenever any engine timed out, so the timeout message is not masked by later generic engine failures. The timeout budget bounds the entire sweep: per-engine timeouts shrink as the budget is consumed, so the reported timeout matches the worst-case wall time.
• Proxies: Studio routes through environment proxies; this port resolves and pins the target IP and tunnels that connection through HTTPS_PROXY / HTTP_PROXY / ALL_PROXY when the proxy is a SOCKS5 proxy (NO_PROXY exclusions respected; DNS stays local for the guard). Other proxy schemes fall back to a direct connection. The search and web_render paths use the process-wide fetch, so an agent-level proxy dispatcher applies there too — see Companion: rotating exit IPs.
• Dedup and titles: the aggregator keys on canonicalized hrefs (utm_*/tracking parameters and fragments stripped, then the URL re-serialized); fetched HTML pages are prefixed with the document <title>. Studio keys on raw hrefs and returns the converted body alone.
• Upstream drift: current ddgs ships ten backends (adding bing, startpage, grokipedia), requires a vqd token for DuckDuckGo, and exposes an extract() mode. This port deliberately pins the Studio snapshot — seven engines, bing disabled upstream, no vqd, no pagination — so engine behavior matches Studio rather than ddgs head.

When to use alternatives

This package keeps Studio's deterministic, zero-dependency pipeline and test parity with unsloth/studio, then layers local access, third-party rendering, and other Studio-independent behavior on top. The SSRF guard is real and thoroughly tested, but it is opt-in: allowPrivateAddresses defaults to true, and local files are readable unless allowLocalFiles is false. For other tradeoffs, prefer:

Need	Use
Browser-like TLS/HTTP fingerprinting to unblock bot-defended pages	pi-smart-fetch (wreq-js chrome_145)
Headless Chrome for JS-rendered SPAs/YouTube/Reddit threads	Built-in web_render (Jina Reader) first; georgebashi/pi-web-fetch (puppeteer + trafilatura) when the Reader falls short
Hosted search with semantic ranking and no scraping	Brave Search API / Tavily / Exa via pi-ollama-web-search
Prompt-focused page distillation to save context	pi-web-fetch prompt -> sub-agent or Claude Code WebFetch(url,prompt)
Batch fetching many URLs concurrently	pi-smart-fetch batch_web_fetch or call web_fetch in parallel

Mixing is supported: pi install npm:pi-unsloth-webtools npm:pi-smart-fetch lets the model choose the best tool per URL. No need to fork this package to add those features.

Companion: rotating exit IPs

pi-tor-proxy routes pi's in-process fetch traffic through Tor (it downloads and manages its own Tor binary) and gives each pi instance its own circuit and exit IP. The search sweep and web_render both use fetch, so they leave through the current Tor exit, and Jina rate-limits keyless Reader requests per outgoing IP — /tor-cycle swaps the exit those limits are counted against, while /tor-country and /tor-exclude constrain which exits are used.

pi install npm:pi-unsloth-webtools npm:pi-tor-proxy

web_fetch and web_search url mode also route: they resolve and pin the target IP, then tunnel the connection through the SOCKS5 proxy named by HTTPS_PROXY / HTTP_PROXY / ALL_PROXY (with NO_PROXY exclusions, so localhost and local files stay direct). DNS is still resolved locally for the SSRF guard, and the proxy connects to that pinned IP. Caveats: Tor mode supports Linux and macOS only, adds latency, and many search engines and Cloudflare-fronted services challenge or block Tor exits, so cycling helps with per-IP limits but is not a guarantee.

Configuration

Optional settings in ~/.pi/agent/settings.json or .pi/settings.json (project overrides global):

{
  "unslothWebTools": {
    "maxResults": 5,
    "maxChars": 50000,
    "timeoutMs": 60000
  },
  "webFetch": {
    "maxChars": 50000,
    "timeoutMs": 15000
  }
}

Key	Default	Description
unslothWebTools.maxResults	5	Default maxResults for web_search (clamped 1-20)
unslothWebTools.maxChars / webFetch.maxChars / smartFetchDefaultMaxChars	tool param	Default maxChars for web_fetch and web_search url mode
unslothWebTools.timeoutMs / webFetch.timeoutMs / smartFetchDefaultTimeoutMs	60000 fetch, 300000 search	Default timeoutMs when the tool param is absent (>=1000). Fetch and web_search url mode fall back to 60000; web_search query mode falls back to 300000
webSearch.maxResults / smartWebSearch.resultsPerQuery	same as above	Legacy aliases for maxResults
websitePolicy	none	Not read from settings. Tools run unrestricted by default; websitePolicy is a programmatic option the host passes to webSearch / fetchPageText
unslothWebTools.allowPrivateAddresses / webFetch.allowPrivateAddresses	true	Opt out to restore the resolved-IP SSRF guard: private/loopback/link-local hosts (localhost, LAN IPs) are refused again. Non-canonical numeric IP encodings stay blocked either way
unslothWebTools.allowLocalFiles / webFetch.allowLocalFiles	true	Opt out to refuse local files in web_fetch and web_search url mode (file:// URLs, absolute, ~/, or ./ paths); when enabled, PDFs are extracted and HTML converted
unslothWebTools.jinaApiKey / webRender.jinaApiKey	none (JINA_API_KEY fallback)	API key for web_render's Jina Reader; raises its rate limits. Settings keys win over the environment variable

Settings window

/webtools-config opens an interactive settings window (↑↓ navigate, space toggle, q close). Settings persist across sessions in ~/.config/pi-unsloth-webtools/config.json, created when a setting is first changed:

{
  "webRenderEnabled": true
}

Key	Default	Description
webRenderEnabled	true	When false, the web_render tool is deactivated for the session and the automatic HTTP 403 fallback in web_fetch / web_search url mode is disabled

On non-Windows platforms the directory honors XDG_CONFIG_HOME when set (falling back to ~/.config); on Windows it always uses ~/.config.

Tool params always win over file defaults. Search dedup also strips default ports, so https://example.com:443/a and https://example.com/a collapse.

Environment overrides: PI_UNSLOTH_CACHE_DIR changes the fetch cache directory, PI_UNSLOTH_WEBTOOLS_STATS opts into append-only sweep stats JSONL, PI_CODING_AGENT_DIR / PI_AGENT_DIR change the global settings directory, and JINA_API_KEY supplies the web_render key when no settings key is set. Cache entries live 1 hour and stale copies are served only after a network failure. SOCKS5 proxies named by HTTPS_PROXY, HTTP_PROXY, or ALL_PROXY are honored on every fetch (NO_PROXY exclusions apply).

Troubleshooting

Match on the exact prefix. Do not retry blocked hosts with spelling tricks.

Symptom	Exact string	Action
Empty query	No query provided.	Pass query or url.
Cancelled	Search cancelled. / Failed to fetch URL: cancelled.	Retry only if the abort was not intentional.
Search timeout	Search failed: the search engines (...) did not respond within N seconds.	Retry, narrow the query, or raise timeoutMs.
No results	No results found.	Rephrase the query.
Policy filtered everything	No results found within the website access limits.	Widen websitePolicy, do not work around it.
Blocked URL	Blocked: ...	Respect it. Includes non-http schemes, credentials, invalid hosts, non-canonical numeric IP encodings, and policy denials.
Private address blocked	Blocked: refusing to fetch the non-public address ...	The SSRF guard is active (allowPrivateAddresses: false); remove it or set true to reach localhost/LAN, and write the scheme explicitly (http://localhost:3000).
Local file blocked	Blocked: the URL has an invalid hostname or port. for paths	Local files are disabled: remove allowLocalFiles: false to read file://, absolute, ~/, or ./ paths.
File read failed	Failed to read file: ...	Check the path exists and is a regular file.
HTTP failure	Failed to fetch URL: HTTP ...	Fix the URL. A 404 automatically tries a Wayback snapshot; a 403 retries through web_render when enabled.
Proxy failure	Failed to fetch URL: SOCKS5 proxy ...	The SOCKS5 proxy refused or failed (for example Tor is stopping). Check the proxy, or unset the proxy variables for a direct fetch.
Non-text / binary	(non-text content: / (binary content,	Not readable as text by design.
PDF without text	(PDF contains no extractable text) / (PDF content could not be read as text...)	Scanned or encrypted PDF.
Download cap hit	... (page truncated at the download limit)	Raw fetch hit 512 KiB (10 MiB for PDFs).
maxChars cut	... (truncated, N chars total)	Raise maxChars for the full text.
Empty page	(page returned no readable text)	Page had no extractable text; try web_render, which renders JavaScript pages.
Render blocked (local file)	Blocked: web_render cannot fetch local files.	Use web_fetch, which reads local files by default.
Render blocked (private host)	Blocked: refusing to fetch the non-public address ...	web_render only reaches public hosts; use web_fetch for localhost/LAN.
Render failure	Failed to render URL: ...	Jina rejected or failed the request (rate limit, bad key, unreachable page). Retry, or check unslothWebTools.jinaApiKey / JINA_API_KEY.
GitHub rewrite	README of ... (fetched via the GitHub README API):	Expected repo-root rewrite, not the HTML chrome.
Cache fallback	Served from cache / STALE cache from YYYY-MM-DD	Network failed; output is the cached copy with its date.
Wayback fallback	Fetched from Wayback Machine snapshot (YYYY-MM-DD) for ...	Original 404'd; output is the archived copy with its date.

Development

npm install
npm run typecheck
npm test
npm run test:unit
npm run test:smoke

Tests

npm test runs the full suite. npm run test:unit skips the live-network smoke tests, and npm run test:smoke runs only those.

The suite ports Unsloth Studio's own tests for these tools:

• test/html-to-md.test.ts: hidden-element stripping and main-content scoping (from test_web_fetch_extraction.py)
• test/header-strip.test.ts: the header link-density suite plus article-vs-main selection and boilerplate cases (from test_web_fetch_extraction.py)
• test/binary-guard.test.ts: the MIME/magic/charset/PDF matrix (from test_web_fetch_binary_guard.py)
• test/web-search-policy.test.ts: policy filtering, overfetch, and failure messages (from test_web_access_policy.py)
• test/fetch-flow.test.ts: GitHub README rewrite, deadline/cancellation, HTML sniffing (from test_web_fetch_extraction.py; the fetch client is injected via seams)
• test/engines.test.ts: the ddgs engine port, normalizers, the XPath subset, the aggregator, the ranker, and the Wikipedia engine with a stubbed fetch
• test/pdf-parity.test.ts: MuPDF engine capabilities, PDF 1.5 object streams, ASCII85Decode, font /Differences encodings, pymupdf4llm-style headings/links/tables
• test/entities.test.ts: decodeHtmlEntities parity with CPython html.unescape, legacy refs, longest-prefix rule, Windows-1252 numeric mappings, invalid codepoints
• test/smoke.test.ts: live network checks against real hosts, including a per-engine result-health sweep (at least two engines must return well-formed results; engines that block or reset connections from datacenter IPs count as unhealthy, not failures)
• test/web-render.test.ts: Jina Reader rendering with a stubbed fetch and DNS, the public-only guard, policy enforcement, error mapping, truncation, cancellation, and API key precedence

The seams (seams.resolve / seams.request / rawFetch) replace the network stack with fakes, mirroring how the Studio suite monkeypatches _validate_and_resolve_host and build_opener.

License

The ported logic derives from Unsloth Studio (AGPL-3.0-only), so this package is released under the same AGPL-3.0-only license.