@pi-unipi/web-api

Web search, page reading, and content summarization for the agent. The read path uses a local smart-fetch engine by default — free, no API key, browser-grade TLS fingerprinting that bypasses Cloudflare.

wigolo is the default search and read provider when installed — a local-first engine with multi-engine search, rank fusion and on-device reranking, at $0/query with no API key. Paid providers (SerpAPI, Tavily, Firecrawl, Perplexity) are available as fallbacks, and DuckDuckGo and Jina work out of the box.

wigolo (optional, recommended)

wigolo is ranked first for both search and read but is not bundled — it is an AGPL-licensed project while UniPi is MIT, so it is an optional dependency loaded at runtime only if you installed it. Set it up once:

npm install -g wigolo
npx wigolo init      # downloads the browser engine + on-device models (~1.5 GB)
npx wigolo doctor    # verify

Until then, web calls fall through automatically to the next-ranked provider, so nothing breaks if you skip it. Disable it entirely in /unipi:web-settings.

Commands

Command	Description
/unipi:web-settings	Configure providers, API keys, and smart-fetch defaults
/unipi:web-cache-clear	Clear all cached web content

Special Triggers

Workflow skills detect web-api and inject web tools for research-type commands:

Skill	What Changes
research	Full web search, read, summarize
gather-context	External documentation lookup
consultant	Industry best practices research
subagents (explore)	Web research in parallel

The footer and info-screen don't display web-api data — it's a tool package, not a state package.

Agent Tools

Tool	Description
web_search	Search the web via provider
multi_web_content_read	Extract content from URLs (smart-fetch or provider)
web_llm_summarize	Summarize web content via LLM

web_search

# Auto-select cheapest provider
web_search(query: "TypeScript generics")

# Use specific provider
web_search(query: "latest AI research", source: 5)  # Tavily

multi_web_content_read

# Single URL (smart-fetch engine by default)
multi_web_content_read(url: "https://example.com/article")

# Batch URLs
multi_web_content_read(url: ["https://example.com/a", "https://example.com/b"])

# Provider fallback (wigolo)
multi_web_content_read(url: "https://example.com/article", source: 1)

# Custom options
multi_web_content_read(url: "https://example.com/article", format: "json", maxChars: 10000)

web_llm_summarize

web_llm_summarize(url: "https://example.com/long-article")
web_llm_summarize(url: "https://example.com/research", prompt: "Extract key findings")

Smart-Fetch Engine

Local content extraction pipeline — no API key required:

Component	Purpose
wreq-js	Browser-grade TLS fingerprinting (bypasses Cloudflare)
defuddle	Intelligent content extraction from HTML
linkedom	Server-side DOM parsing

Outputs clean markdown with metadata (title, author, site, word count). Supports batch concurrent fetching with progress.

Providers

Search

Provider	Rank	Cost	API Key
wigolo (local)	1	Free	No
DuckDuckGo	2	Free	No
Jina AI Search	3	Freemium	Optional
SerpAPI	4	Paid	Required
Tavily	5	Paid	Required
Perplexity	6	Paid	Required

Read

Provider	Rank	Cost	API Key
Smart-Fetch Engine	0	Free	No
wigolo (local)	1	Free	No
Jina AI Reader	2	Freemium	Optional
Firecrawl	3	Paid	Required
Perplexity	4	Paid	Required

Summarize

Provider	Rank	Cost	API Key
Perplexity	1	Paid	Required
LLM Summarize	2	LLM tokens	No

Configurables

API Keys

Configure via /unipi:web-settings (interactive TUI) or environment variables:

export SERPAPI_KEY="your-key"
export TAVILY_API_KEY="your-key"
export FIRECRAWL_API_KEY="your-key"
export PERPLEXITY_API_KEY="your-key"
export JINA_API_KEY="your-key"

Providers auto-enable when you add a valid API key.

Smart-Fetch Defaults

Configure browser profile, OS, max chars, timeout via /unipi:web-settings → "Smart Fetch Defaults".

Settings Files

• Auth: ~/.unipi/config/web-api/auth.json (API keys, gitignored)
• Config: ~/.unipi/config/web-api/config.json (provider settings, smart-fetch defaults)

Cache

• Default TTL: 1 hour
• Cache location: ~/.unipi/config/web-api/cache/
• Automatic for all read operations

Troubleshooting

No provider available: Run /unipi:web-settings and add API keys or enable a free provider.

Smart-fetch fails: Try a different browser profile (browser: "chrome_133") or a provider fallback (source: 1).

Rate limiting: Add an API key for higher limits, use smart-fetch (no limits), or try a different provider.

License

MIT