@pi-unipi/web-api

Web search, page reading, and content summarization for the agent. The read path uses a local smart-fetch engine by default — free, no API key, browser-grade TLS fingerprinting that bypasses Cloudflare.

Paid providers (SerpAPI, Tavily, Firecrawl, Perplexity) are available as fallbacks. DuckDuckGo and Jina work out of the box for search.

Commands

Command	Description
/unipi:web-settings	Configure providers, API keys, and smart-fetch defaults
/unipi:web-cache-clear	Clear all cached web content

Special Triggers

Workflow skills detect web-api and inject web tools for research-type commands:

Skill	What Changes
research	Full web search, read, summarize
gather-context	External documentation lookup
consultant	Industry best practices research
subagents (explore)	Web research in parallel

The footer and info-screen don't display web-api data — it's a tool package, not a state package.

Agent Tools

Tool	Description
web_search	Search the web via provider
multi_web_content_read	Extract content from URLs (smart-fetch or provider)
web_llm_summarize	Summarize web content via LLM

web_search

# Auto-select cheapest provider
web_search(query: "TypeScript generics")

# Use specific provider
web_search(query: "latest AI research", source: 4)  # Tavily

multi_web_content_read

# Single URL (smart-fetch engine by default)
multi_web_content_read(url: "https://example.com/article")

# Batch URLs
multi_web_content_read(url: ["https://example.com/a", "https://example.com/b"])

# Provider fallback (Jina Reader)
multi_web_content_read(url: "https://example.com/article", source: 1)

# Custom options
multi_web_content_read(url: "https://example.com/article", format: "json", maxChars: 10000)

web_llm_summarize

web_llm_summarize(url: "https://example.com/long-article")
web_llm_summarize(url: "https://example.com/research", prompt: "Extract key findings")

Smart-Fetch Engine

Local content extraction pipeline — no API key required:

Component	Purpose
wreq-js	Browser-grade TLS fingerprinting (bypasses Cloudflare)
defuddle	Intelligent content extraction from HTML
linkedom	Server-side DOM parsing

Outputs clean markdown with metadata (title, author, site, word count). Supports batch concurrent fetching with progress.

Providers

Search

Provider	Rank	Cost	API Key
DuckDuckGo	1	Free	No
Jina AI Search	2	Freemium	Optional
SerpAPI	3	Paid	Required
Tavily	4	Paid	Required
Perplexity	5	Paid	Required

Read

Provider	Rank	Cost	API Key
Smart-Fetch Engine	0	Free	No
Jina AI Reader	1	Freemium	Optional
Firecrawl	2	Paid	Required
Perplexity	3	Paid	Required

Summarize

Provider	Rank	Cost	API Key
Perplexity	1	Paid	Required
LLM Summarize	2	LLM tokens	No

Configurables

API Keys

Configure via /unipi:web-settings (interactive TUI) or environment variables:

export SERPAPI_KEY="your-key"
export TAVILY_API_KEY="your-key"
export FIRECRAWL_API_KEY="your-key"
export PERPLEXITY_API_KEY="your-key"
export JINA_API_KEY="your-key"

Providers auto-enable when you add a valid API key.

Smart-Fetch Defaults

Configure browser profile, OS, max chars, timeout via /unipi:web-settings → "Smart Fetch Defaults".

Settings Files

• Auth: ~/.unipi/config/web-api/auth.json (API keys, gitignored)
• Config: ~/.unipi/config/web-api/config.json (provider settings, smart-fetch defaults)

Cache

• Default TTL: 1 hour
• Cache location: ~/.unipi/config/web-api/cache/
• Automatic for all read operations

Troubleshooting

No provider available: Run /unipi:web-settings and add API keys or enable a free provider.

Smart-fetch fails: Try a different browser profile (browser: "chrome_133") or a provider fallback (source: 1).

Rate limiting: Add an API key for higher limits, use smart-fetch (no limits), or try a different provider.

License

MIT