pi-webaio

All-in-one web access tools for pi: search, fetch, crawl, extract, map, cache, chunk, and render web content for AI agents.

What It Does

pi-webaio registers eight pi tools:

• aio-websearch — search DuckDuckGo, Brave, Yahoo, Bing, TinyFish, and FireCrawl in parallel, with default Google (via a local CDP broker) and an opt-in Reddit CDP companion (reddit: true, requires Chrome). Returns in ~2.9s with live per-provider TUI progress (spinner rows, result counts with latency, an elapsed-vs-target bar) and a stable final view showing every engine's count and timing
• aio-webfetch — fetch one or many URLs into markdown or structured formats, with an opt-in heading outline, query-focused answer mode, and multi-source cited answers
• aio-webcontent — retrieve cached content by URL (with opt-in section-level diff)
• aio-webresult — retrieve cached results by response ID
• aio-webmap — discover site pages or map GitHub repositories without fetching
• aio-webpull — crawl/pull sites into local markdown files
• aio-webquery — BM25 search over a locally-pulled corpus (offline, no re-fetching)
• aio-webresearch — single-round research bundle: search → rank → fetch → cited evidence bundle on disk

It includes anti-bot TLS fingerprinting, browser fallback, 21 API-first extractors (GitHub, YouTube, npm/PyPI and other package registries, Context7, DeepWiki, and more), RAG chunking, TUI progress rendering, phase-aware errors, opt-in paywall bypass support, and keyless providers (FireCrawl) that work without API keys.

Extra Providers

FireCrawl Keyless (free, 1k credits/month, no API key needed):

• Search: works out of the box as a search provider
• Fetch: firecrawl: true on aio-webfetch — delegates to FireCrawl Scrape API

TinyFish (free, requires API key in ~/.piwebaio/config or ~/.piwebaio/.env):

• Search: unlimited free search results
• Fetch: tinyfish: true on aio-webfetch — delegates to TinyFish Fetch API

Both providers run in parallel with the HTTP engines during search. For fetch, the recommended chain is FireCrawl → TinyFish → normal pipeline.

Google Search uses the local CDP broker by default (faster cold start, tighter p95, 100% Google success under concurrency — see speed.md). Google ignores the deprecated num param and renders ~8–10 organic results per SERP page, so the broker paginates through ?start=10, ?start=20, … (the same mechanism Google's own "Next" links use), merging and URL-deduping pages up to max until the lane is satisfied, the SERP runs out of new organics, or the lane budget is exhausted. The Google lane carries a hard 3-second cap measured from when its search starts, so it never gates the tool's 7-second overall deadline — even on a full multi-page pagination. If a page-2+ navigation or extraction fails, the lane degrades gracefully to the results it had already collected and annotates googleStatus accordingly (e.g. ok (an extra SERP page failed…)); a total fresh failure still surfaces as an error. Set PI_WEBAIO_CDP_BROKER=0 to force the legacy extractor. The manual, live-only benchmark is npm run bench:google-cdp -- --live --query "..." --samples 3. It reports total/startup measurements; detailed CDP phase timings are not yet instrumented, and no speedup is inferred.

For the full public tool path (HTTP engines + Google + Reddit under the response target), use scripts/bench-full-search.mjs:

node --experimental-strip-types scripts/bench-full-search.mjs broker 10 3000 "query"

<legacy|broker> picks the Google path; sample 1 measures cold start and samples 2–n are warm; the third argument is inter-sample spacing in ms. On the 2.9s response-target path the tool returns at the budget by design — recent runs: p50 ≈ 2.90s both modes, HTTP success 10/10 (see speed.md for full tables and environment caveats).

Install

pi install npm:pi-webaio

Or from git:

pi install git:github.com/apmantza/pi-webaio

Documentation

• Features — overview, extraction pipeline, GitHub/YouTube handling, output formats, chunking, errors, and search ranking
• Custom vertical extractors — add your own site extractors (company wikis, niche sites) without forking
• Usage guide — common pi prompts and examples
• Tools reference — tool names, parameters, and defaults
• Architecture — build, TUI rendering, FetchError system, CI, and security notes
• MCP server — use the tools from Claude Code, Claude Desktop, and other MCP clients without pi (npx -y pi-webaio-mcp)
• PageMap inspiration — future extraction and structured-output ideas

Contributing

We especially welcome contributors for new vertical extractors, search engines, site-specific fetch fixes, anti-bot/paywall resilience, and docs. See CONTRIBUTING.md for setup, testing, and contribution checklists.

Contributors

Thanks goes to these wonderful people:

If you land a pull request or report an issue that gets fixed, we'll add you here.

License

pi-webaio is released under the MIT License.