pi-webaio

All-in-one web access tools for pi: search, fetch, crawl, extract, map, cache, chunk, and render web content for AI agents.

What It Does

pi-webaio registers eight pi tools:

• aio-websearch — search DuckDuckGo, Brave, Yahoo, and Bing in parallel, with default Google (via a local CDP broker) and automatic Reddit CDP companions when Chrome is available
• aio-webfetch — fetch one or many URLs into markdown or structured formats, with an opt-in heading outline, query-focused answer mode, and multi-source cited answers
• aio-webcontent — retrieve cached content by URL (with opt-in section-level diff)
• aio-webresult — retrieve cached results by response ID
• aio-webmap — discover site pages or map GitHub repositories without fetching
• aio-webpull — crawl/pull sites into local markdown files
• aio-webquery — BM25 search over a locally-pulled corpus (offline, no re-fetching)
• aio-webresearch — single-round research bundle: search → rank → fetch → cited evidence bundle on disk

It includes anti-bot TLS fingerprinting, browser fallback, 21 API-first extractors (GitHub, YouTube, npm/PyPI and other package registries, Context7, DeepWiki, and more), RAG chunking, TUI progress rendering, phase-aware errors, and opt-in paywall bypass support.

Google Search uses the local CDP broker by default (faster cold start, tighter p95, 100% Google success under concurrency — see speed.md). Google ignores the deprecated num param and renders ~8–10 organic results per SERP page, so the broker paginates through ?start=10, ?start=20, … (the same mechanism Google's own "Next" links use), merging and URL-deduping pages up to max until the lane is satisfied, the SERP runs out of new organics, or the lane budget is exhausted. The Google lane carries a hard 3-second cap measured from when its search starts, so it never gates the tool's 7-second overall deadline — even on a full multi-page pagination. If a page-2+ navigation or extraction fails, the lane degrades gracefully to the results it had already collected and annotates googleStatus accordingly (e.g. ok (an extra SERP page failed…)); a total fresh failure still surfaces as an error. Set PI_WEBAIO_CDP_BROKER=0 to force the legacy extractor. The manual, live-only benchmark is npm run bench:google-cdp -- --live --query "..." --samples 3. It reports total/startup measurements; detailed CDP phase timings are not yet instrumented, and no speedup is inferred.

Install

pi install npm:pi-webaio

Or from git:

pi install git:github.com/apmantza/pi-webaio

Documentation

• Features — overview, extraction pipeline, GitHub/YouTube handling, output formats, chunking, errors, and search ranking
• Custom vertical extractors — add your own site extractors (company wikis, niche sites) without forking
• Usage guide — common pi prompts and examples
• Tools reference — tool names, parameters, and defaults
• Architecture — build, TUI rendering, FetchError system, CI, and security notes
• MCP server — use the tools from Claude Code, Claude Desktop, and other MCP clients without pi (npx -y pi-webaio-mcp)
• PageMap inspiration — future extraction and structured-output ideas

Contributing

We especially welcome contributors for new vertical extractors, search engines, site-specific fetch fixes, anti-bot/paywall resilience, and docs. See CONTRIBUTING.md for setup, testing, and contribution checklists.

Contributors

Thanks goes to these wonderful people:

If you land a pull request or report an issue that gets fixed, we'll add you here.

License

pi-webaio is released under the MIT License.