pi-web v0.5

Pi extension for unified web search, content extraction, site crawling, and page capture.

Auto-selects the best backend from SearXNG (self-hosted), Brave Search, Firecrawl, and Crawl4AI — so agents don't have to know which backend to use. Search selection is adaptive: broad discovery prefers self-hosted SearXNG, while precision-sensitive searches and inline content prefer Brave.

Install

pi install npm:@bacnh85/pi-web

From this repository:

pi install ./extensions/pi-web
# or test directly
pi -e ./extensions/pi-web

Configuration

Environment lookup order:

1. Process environment
2. Current working directory .env.local
3. Current working directory .env
4. Pi global config ~/.pi/agent/.env.local
5. Pi global config ~/.pi/agent/.env

Variables:

Variable	Required	Default	Notes
BRAVE_API_KEY	No (1)	—	Brave Search API key
SEARXNG_BASE_URL	No	http://172.30.55.22:8888	Self-hosted SearXNG
FIRECRAWL_API_URL	No	https://api.firecrawl.dev/v2	Self-hosted or hosted
FIRECRAWL_API_KEY	No (2)	—	Required for hosted Firecrawl
CRAWL4AI_API_URL	No	http://172.30.55.22:11235	Self-hosted Crawl4AI
CRAWL4AI_API_TOKEN	No (3)	—	Required if Crawl4AI auth enabled

(1) At least one search backend (SearXNG, Brave, or Firecrawl) must be configured for web_search. (2) Required for hosted Firecrawl; optional for self-hosted instances without auth. (3) Required for Crawl4AI v0.9+ default config.

Secrets are never printed; web_status reports only presence/source.

Tools

web_search — Unified search

Searches the web. Auto-selects backends adaptively: SearXNG for broad self-hosted discovery, Brave for precision-sensitive queries and include_content, Firecrawl as last resort.

web_search query="ansible podman quadlet" count=5
web_search query="ansible documentation" backend=brave count=10
web_search query="latest python release" engines="google,github"
web_search query="riven media" include_content=true

Parameters:

Parameter	Type	Default	Description
query	string	—	Search query
count	number	5	Number of results (max 20)
freshness	string	—	Time filter: pw, pm, py, or YYYY-MM-DDtoYYYY-MM-DD
country	string	US	Two-letter country code
backend	string	auto	Force backend: auto, searxng, brave, firecrawl
engines	string	—	SearXNG engine override, e.g. google,github
include_content	boolean	false	Fetch page content alongside results
content_chars	number	5000	Max content chars per result

Auto-selection behavior:

1. SearXNG — first for broad/general discovery, especially when engines is supplied.
2. Brave — first for precision-sensitive queries (site:, quoted phrases, docs/API/source lookups, short proper-name queries) and whenever include_content is true. Requires BRAVE_API_KEY.
3. Firecrawl Search — last resort. ⚠️ Poor semantic accuracy on domain-specific/ambiguous queries (e.g., "Riven" returns League of Legends results). Prefer SearXNG or Brave for precision.

Tool output includes search diagnostics showing attempted backends and the selected backend.

Use backend parameter to force a specific backend when needed.

web_extract — Unified content extraction

Extracts readable content from a URL. Auto-selects backend: static (JSDOM) → dynamic (Firecrawl) → full (Crawl4AI), with extraction diagnostics showing fallback attempts.

web_extract url="https://docs.ansible.com/..."
web_extract url="https://riven.tv/" mode=static
web_extract url="https://example.com" mode=dynamic prompt="Extract pricing plans"

Parameters:

Parameter	Type	Default	Description
url	string	—	URL to extract
mode	string	auto	auto, static, dynamic, or full
prompt	string	—	Prompt for JSON extraction (dynamic mode)
schema	any	—	JSON schema for structured extraction (dynamic mode)
content_chars	number	20000	Max content chars
wait_for	number	—	Milliseconds to wait for Firecrawl dynamic rendering. Crawl4AI /md full mode may ignore this.
mobile	boolean	false	Emulate mobile viewport (dynamic mode)

Mode behavior:

Mode	Backend	Best for	API key needed
static	JSDOM+Readability	Simple static pages, blog posts, docs	No
dynamic	Firecrawl Scrape	JS-rendered pages, dynamic content	Maybe
full	Crawl4AI	JS-heavy SPA, complex rendering	Maybe
auto (default)	static → dynamic → full	Unknown page type	Maybe

In auto mode, fallbacks are noted in the output (e.g., [Extraction fell back to Firecrawl Scrape (dynamic mode)]). If static extraction fails, the tool gracefully escalates to heavier backends.

⚠️ Note on Firecrawl Scrape: Fails on bot-protected sites (Ansible docs, many CDN-backed doc sites). Falls back to full mode (Crawl4AI) in auto mode.

web_map — Site URL discovery

Discovers URLs from a site using Firecrawl Map. Best on base domains; may return fewer results on sub-paths.

web_map url="https://riven.tv"
web_map url="https://docs.example.com" sitemap=only

Parameters: url, limit (default 100), include_subdomains, search, sitemap, use_index, ignore_cache.

web_crawl — Site crawl

Crawls pages from a site. Two modes:

• light (default): Firecrawl Crawl — conservative, docs-focused, single URL.
• full: Crawl4AI Crawl — headless browser, rendered data, media, links, up to 100 URLs.

web_crawl url="https://docs.example.com" limit=10          # Firecrawl light mode
web_crawl urls=["https://a.com","https://b.com"] mode=full  # Crawl4AI full mode
web_crawl url="https://example.com" mode=light poll=true    # Poll for completion

web_screenshot — Page screenshot

Captures a full-page PNG screenshot using Crawl4AI. Returns base64-encoded PNG.

web_screenshot url="https://example.com"
web_screenshot url="https://example.com" wait_for=5 wait_for_images=true

web_pdf — Page PDF

Generates a PDF document using Crawl4AI. Returns base64-encoded PDF.

web_pdf url="https://example.com/article"

web_status — Provider status

Shows all provider configuration status and Crawl4AI server health.

web_status

Typical output:

{
  "brave": { "apiKeyFound": true, "apiKeySource": "process.env" },
  "searxng": { "baseUrl": "http://172.30.55.22:8888", ... },
  "firecrawl": { "baseUrl": "http://172.30.55.22:3002/v2", ... },
  "crawl4ai": {
    "baseUrl": "http://172.30.55.22:11235",
    ...
    "health": { "status": "healthy", "version": "0.5.0", ... }
  }
}

Library structure

Module	Contents
lib/config.ts	Environment loading, config helpers for all providers
lib/format.ts	Text sanitization, truncation, crawl/scrape result formatting
lib/content.ts	Readable content extraction (JSDOM + Readability + Turndown)
lib/retry.ts	Retry with exponential backoff for transient HTTP failures
lib/brave.ts	Brave Search API fetch client (internal)
lib/searxng.ts	SearXNG metasearch fetch client (internal)
lib/firecrawl.ts	Firecrawl API fetch client with v2→v1 fallback (internal)
lib/crawl4ai.ts	Crawl4AI Docker API fetch client (internal)
lib/search.ts	Unified search orchestrator — probes backends, fallback chain
lib/extract.ts	Unified extraction orchestrator — mode-based backend selection

Migration from 0.3.x

v0.4 replaces the 14 individual backend-specific tools with 7 unified tools:

v0.3 tool	v0.4 replacement
brave_search	web_search with backend: "brave"
searxng_search	web_search with backend: "searxng"
firecrawl_search	web_search with backend: "firecrawl"
web_content	web_extract with mode: "static"
firecrawl_scrape	web_extract with mode: "dynamic"
crawl4ai_scrape	web_extract with mode: "full"
firecrawl_map	web_map (same behavior)
firecrawl_crawl	web_crawl with mode: "light"
crawl4ai_crawl	web_crawl with mode: "full"
crawl4ai_stream	(removed — use web_crawl with mode: "full")
crawl4ai_screenshot	web_screenshot (same behavior)
crawl4ai_pdf	web_pdf (same behavior)
crawl4ai_status	Merged into web_status
web_status	web_status (enhanced with Crawl4AI health)

All v0.3 tool names were removed in v0.4. Update any agent instructions or skills that reference the old names.

Development

# Run all tests
npm test

# Run only unit tests
npm run test:unit