pi-websearch-crawl4ai

A pi extension that lets the agent fetch content from the web via a running Crawl4AI server.

Intended use: running pi with the bash tool disabled (so curl / wget are unavailable) while still letting the model read and crawl web pages.

What it gives the LLM

Six tools, all talking to a Crawl4AI server:

Tool	Endpoint	Purpose
web_fetch	POST /md	Fetch a URL → clean Markdown (filters: fit/raw/bm25/llm)
web_fetch_html	POST /html	Sanitized HTML for DOM-aware tasks
web_crawl	POST /crawl	Multi-URL crawl with typed BrowserConfig/CrawlerRunConfig
web_execute_js	POST /execute_js	Run JS snippets on a page and read back JSON
web_screenshot	POST /screenshot	Full-page PNG screenshot (returned inline)
web_ask	GET /ask	Query the Crawl4AI library's own docs (for configuring it)

Plus commands: /crawl4ai-status, /crawl4ai-url <url>, /crawl4ai-token <tok>.

Prerequisites

You need a Crawl4AI server reachable from where pi runs. The fastest path:

docker run -d \
  -p 11235:11235 \
  --name crawl4ai \
  --shm-size=1g \
  unclecode/crawl4ai:latest

# Sanity check
curl http://localhost:11235/health

See the Crawl4AI Docker guide for GPU, LLM keys, config.yml, JWT auth, etc.

Install as a pi extension

# project-local
mkdir -p .pi/extensions
ln -s "$(pwd)/pi-websearch-crawl4ai" .pi/extensions/crawl4ai

# or global
mkdir -p ~/.pi/agent/extensions
ln -s "$(pwd)/pi-websearch-crawl4ai" ~/.pi/agent/extensions/crawl4ai

pi auto-discovers index.ts via the "pi".extensions field in package.json.

Alternatively, for a one-off test:

pi -e ./pi-websearch-crawl4ai/index.ts

Configuration

Precedence: CLI flag > env var > default.

Setting	Env	Flag	Default
Base URL	CRAWL4AI_BASE_URL	--crawl4ai-url <url>	http://localhost:11235
Auth token	CRAWL4AI_TOKEN	--crawl4ai-token <tok>	(none)

At runtime you can also:

• /crawl4ai-status — show current config + /health
• /crawl4ai-url http://host:11235 — change base URL for this session
• /crawl4ai-token <jwt> — set bearer token (empty clears it)

Example use

Running pi with only read-only tools and this extension:

pi --tools read,write,edit,web_fetch,web_crawl
> "Read https://example.com and summarize it."

The model will call web_fetch instead of reaching for bash/curl.

How web_crawl typed configs work

Crawl4AI accepts configuration objects shaped as {"type":"ClassName","params":{...}}. Example you (or the model) can pass:

{
  "urls": ["https://example.com", "https://httpbin.org/html"],
  "browser_config": { "type": "BrowserConfig", "params": { "headless": true } },
  "crawler_config": {
    "type": "CrawlerRunConfig",
    "params": { "cache_mode": "bypass", "stream": false }
  }
}

If you need to remind the model what's available, it can call web_ask with a query like "CrawlerRunConfig parameters" to pull the Crawl4AI library docs.

Security note

Extensions run with your user's full permissions. The tools here can fetch arbitrary URLs via your Crawl4AI server. If that's a problem, run Crawl4AI with rate limiting / allowlists configured in its config.yml, and/or restrict which tools pi activates via --tools.