pi-better-web-search

One tool. Every provider. Raw results.

A pi extension that adds exactly one model-callable tool: web_search. It uses provider-native search APIs, analyzes URLs when a usable provider supports it, automatically falls back across authenticated providers, and returns provider data as-is.

简体中文 · Architecture · Providers · Testing · Source

The npm registry may lag GitHub releases. If the version you want is not yet published to npm, install the tagged GitHub release below instead.

pi install npm:pi-better-web-search

Then start a new pi process, or run /reload in an existing session.

GitHub and local installations also work:

pi install git:github.com/zhexusun10/pi-better-web-search@v0.1.9
pi install ./path/to/pi-better-web-search

Do not load this package together with pi-web-search. Both register a tool named web_search, and the second registration replaces the first.

Requirements: pi 1.3+ and Node.js 22.19+.

What it fixes

pi-better-web-search started as an extraction of the web_search tool from pi-web-search, then changed the parts that made it unreliable:

• One tool, not two. There is no separate url_context tool. URL analysis is the optional urls field on web_search.
• Correct exposure. The tool is hidden when the current model has no compatible search interface and no compatible provider is authenticated. A login or API key for any supported provider is enough to expose it.
• Automatic fallback. If the current model cannot search, or a provider call fails, the extension tries other authenticated provider interfaces until one succeeds.
• Remembered interfaces. The successful search and fetch providers are cached under pi's agent directory. The next call uses the remembered interface when the current model cannot search.
• Fresh discovery. Adding or removing a provider, login, API key, or available model changes the cache fingerprint, so old preferences are not blindly reused.
• Dynamic URL field. urls is exposed only when at least one usable provider supports URL analysis. urls can be used alone or with query.
• Raw results. Provider data is returned in a JSON wrapper without deduplication, URL normalization, junk filtering, re-ranking, summarization, citation-marker insertion, or appended Sources sections. A final character guard applies structured truncation only to oversized output.
• Correct semantics. The tool description is exactly Search the web or analyze specified URLs. The schema makes clear that urls does not require query.
• No system prompt changes. The extension does not add prompt snippets, guidelines, loadout hooks, commands, providers, MCP servers, or other tools.
• Native DeepSeek Search/Fetch. DeepSeek uses the documented /v1/tools/search and /v1/tools/fetch endpoints with Bearer auth.

Tool semantics

The runtime schema is logically:

{
  query?: string;
  urls?: string[];
}

• query searches the web. Outer whitespace is trimmed; search operators are otherwise passed through unchanged. If DeepSeek is selected and the query exceeds 128 Unicode characters, the tool returns an error with the actual length and limit, asking the model to shorten the query and retry. It does not send the query or silently truncate it. Other providers may accept longer queries.
• urls analyzes the listed pages. URLs can be used alone or together with query. A query plus URLs performs a search and then analyzes the URLs.
• At most 20 URLs may be supplied.
• At least one non-empty query or one valid URL is required. This rule is enforced at runtime rather than with a top-level anyOf, for compatibility with all model providers.
• The urls field is hidden when no usable interface supports URL analysis. In that configuration query is required.
• Search and URL analysis may use different providers in the same call.

Invalid input with no usable query or URL, a missing URL-capable provider for supplied URLs, or a call where every search/URL-only provider fails returns a tool error (isError: true) instead of silently dropping work. Individual invalid URLs alongside valid work are reported in the fetched entries.

Provider support

pi-better-web-search natively supports 10 distinct provider classes (plus Anthropic Messages), automatically detecting authenticated credentials and routing each call through native provider protocols:

URL analysis is only exposed when at least one usable interface supports it. For example, a current OpenAI model with only OpenAI credentials gets a query-only schema, while the same session with DeepSeek, Ollama Cloud, or Google credentials also gets urls.

Provider detection follows the authenticated model's provider/API:

1. deepseek provider (DeepSeek native search/fetch)
2. Antigravity or Google Generative AI (Google Gemini & Google Antigravity)
3. Ollama Cloud (ollama-cloud provider or exact ollama.com host)
4. xAI with openai-responses (xAI Grok)
5. OpenAI-compatible Responses APIs (OpenAI, Azure OpenAI, OpenAI Codex, GitHub Copilot, OpenCode)
6. Anthropic Messages (Claude)

See Providers for request shapes, auth quirks, and URL support per family.

Automatic selection and fallback

For a search:

1. The current conversation model is tried first when its provider family supports web search and neither the provider nor that model is cooling down after a recent failure.
2. A remembered successful provider from the preference cache is tried next when the cache fingerprint is current.
3. The remaining provider families are tried in this fixed order: DeepSeek → Ollama → Google → OpenAI → xAI → Anthropic.
4. Within a family, brand order comes first: DeepSeek models, then OpenAI, then Claude, then Gemini/Gemma, then other gateway brands. For brands without named tiers, variants order flash → base → pro; within a variant, newer versions come first. GPT and Claude keep version-first ordering, with their capability tier as a tiebreaker: OpenAI tiers are luna → terra → sol → astra and Claude tiers are haiku → sonnet → opus → fable. Ties are deterministic.
5. Alternate providers are considered before additional models of an already represented provider; siblings are then tried round-robin across families. A provider-wide failure skips that provider, while a model-specific failure (including a model-specific 403) can fall back to a sibling.

The search limit is 12 actual attempts; skipped candidates do not consume it. URL fetching follows the same idea with a separate limit of 6 actual attempts. Model ranking applies only to supported APIs: for example, the built-in opencode-go DeepSeek models use openai-completions, which this extension cannot search through and does not reinterpret as Responses.

The current model is normally tried first when it supports search, unless its provider or model is in a failure cooldown. The cache does not replace a supported current model; it mainly lets an unsupported current model use a remembered compatible provider immediately on later calls.

If a provider errors, the extension records the attempt and moves to the next interface. An overlong query rejected by DeepSeek instead returns immediately without trying another provider or updating its health state. Aborts stop immediately. Each attempt has a timeout (60 seconds by default, configurable with PI_BETTER_WEB_SEARCH_TIMEOUT_MS). If every attempt fails, the tool returns an error whose message contains the provider errors. Provider/model routing remains in details.attempts. For a URL-only call, the error also reports the per-URL failures. Provider error bodies and model-facing error text are bounded to prevent an oversized error from filling the context.

Provider-wide failures (auth, balance, network, timeout, malformed request) and model-specific failures are remembered for the life of the extension process. On later calls the failed provider or model is moved behind healthy candidates for search and fetch independently. The cooldown starts at 60 seconds, doubles when a subsequent attempt after expiry fails, and is capped at 10 minutes. Concurrent failures during the same cooldown do not extend it; a successful search or a fetch with at least one successful URL and no provider transport error clears it. Set PI_BETTER_WEB_SEARCH_FAILURE_COOLDOWN_MS=0 to disable this behavior. Calls already in flight may still try the same candidate at the same time.

Cache fingerprint invalidation

The preference cache is stamped with a fingerprint of all authenticated chat models known to pi, derived from sorted provider/model@baseUrl values. Adding or removing a provider, login, API key, or available model changes the fingerprint. A changed fingerprint means:

• the old cached provider is not used as the first choice;
• provider discovery starts again from the current model and family priority;
• the cache is rewritten only after the next successful search/fetch;
• stale, unverified preferences are not carried into the new fingerprint.

A cache entry is only a hint. Credentials are resolved again on every call, and the referenced provider must still be available and support the operation.

Cache

Default path:

~/.pi/agent/better-web-search.json

The path respects pi's PI_CODING_AGENT_DIR setting. Override it directly with:

export PI_BETTER_WEB_SEARCH_CACHE=/path/to/better-web-search.json

Example contents:

{
  "version": 1,
  "fingerprint": "0123456789abcdef0123456789abcdef",
  "search": { "provider": "deepseek", "model": "deepseek-chat" },
  "fetch": { "provider": "ollama-cloud", "model": "gpt-oss:120b" }
}

Only provider and model identifiers plus the fingerprint are stored. API keys, OAuth tokens, and other credentials are never written to this file.

To clear the remembered preferences, delete the file:

rm ~/.pi/agent/better-web-search.json

A missing, unreadable, or corrupt cache is treated as empty and never breaks a tool call.

Raw provider results

Provider data is returned in a JSON wrapper without cleanup. A search looks like this:

{
  "query": "the query",
  "search": {
    "answer": "provider text when available",
    "results": "the provider's raw results",
    "citations": "the provider's raw citations",
    "searchQueries": ["provider query"],
    "groundingMetadata": "raw provider metadata"
  },
  "fetched": [
    {
      "url": "https://example.com",
      "data": "the provider's raw fetch payload"
    }
  ]
}

• Duplicate URLs are not deduplicated.
• Tracking parameters and junk-looking URLs are not normalized or filtered.
• Results are not re-ranked or summarized.
• Citation markers are not inserted, and no "Sources" or "Additional results" sections are appended.
• Redirects are not resolved and sources are not derived.
• Unknown provider fields are preserved when the result fits the size guard.

A final output-size guard limits model-facing results and errors to 96,000 Unicode code points by default. Normal-sized results retain their original fields and formatting. If indentation alone exceeds the limit, compact JSON is returned without dropping data or setting the truncation flag.

When even compact JSON is too large, the guard allocates space across sibling fields and array entries, redistributing unused space. Recognized link and identity fields (url, uri, href, link, title, name, id, and common aliases) stay complete; if they cannot fit, the associated entry or branch is omitted instead. Standalone URLs and entries in link/citation arrays also stay whole. When a text cut would split a recognizable embedded URL token (such as https://, mailto:, or www.), the incomplete token is omitted. Relative links or bare domains in unrecognized fields cannot always be identified. Other long strings may be shortened, and fields or trailing array entries omitted; results are not re-ranked or summarized.

The JSON portion remains valid, followed by a single [truncated] line. No output-size truncation metadata or per-string markers are added to the JSON. Because of the trailing line, the complete text is not itself valid JSON. details.truncated is also set to true when data is actually removed.

Change the limit before starting pi:

export PI_BETTER_WEB_SEARCH_MAX_CHARS=100000

In PowerShell, use $env:PI_BETTER_WEB_SEARCH_MAX_CHARS = "100000". The limit is in code points, not tokens, and does not adapt to the current model's context window. The marker counts toward the limit. If truncation is needed and the limit is 11–13 characters, only [truncated] is returned; below 11 characters the marker does not fit and only a minimal JSON value is returned. details.truncated remains true in both cases. There is no continuation or pagination cache: retry individual URLs or raise the limit if more content is needed.

Exposure rules

web_search is exposed when either:

• the current conversation model supports a provider-native search interface; or
• at least one authenticated available model maps to a supported provider family.

For an unsupported current model with no compatible login or API key, the tool stays hidden (removed from the active tool set). The extension never calls an unauthenticated provider. When the set of available providers or credentials changes, the extension re-evaluates and re-registers the same web_search name with the appropriate schema variant.

DeepSeek native Search/Fetch API

This package supports DeepSeek's native search and fetch APIs. The older project's Anthropic-compatible DeepSeek chat route is replaced by this native API; DeepSeek models are routed to it even when their underlying chat API is Anthropic Messages compatible.

POST https://api.deepseek.com/v1/tools/search
POST https://api.deepseek.com/v1/tools/fetch
Authorization: Bearer <DeepSeek API Key>
Content-Type: application/json

Search uses query, count: 10, and need_content: true. DeepSeek limits query to 128 Unicode characters. Longer queries return an error before any DeepSeek request is sent; the model should shorten the query and retry. For example: Search query is too long (129 Unicode characters). The maximum is 128 Unicode characters. Shorten the query and retry. The raw results array is returned unchanged, including rerank_score, title, url, snippet, summary, content, site_name, site_icon, date, authority, and any unknown fields.

Fetch sends { "url": "..." } and preserves the raw { title, content } response. URLs must be at most 8192 characters, use HTTP/HTTPS, include a host, and omit credentials.

DeepSeek errors use the OpenAI-compatible error shape. Search errors and global fetch errors switch providers; 403 and 404 fetch failures are reported against the individual URL so the orchestrator can retry only that URL elsewhere.

See Providers for the full request, response, and error details.

Security and privacy

• Credentials are resolved through pi's model registry on each call. The extension does not store API keys or OAuth tokens.
• The cache stores only provider/model identifiers and a fingerprint.
• Search queries and requested URLs are sent to the selected provider's API.
• The extension has no telemetry, analytics, or third-party service of its own.
• A call may use an authenticated provider other than the current conversation model, which can incur usage on that provider.
• Provider error text and tool results can contain request identifiers or page content. Redact them before sharing logs or session exports.

See SECURITY.md for reporting vulnerabilities.

Verified

• Unit and integration tests covering exposure, schema variants, provider request shapes, SSE parsing, raw-result preservation, fallback, cache invalidation, URL handling, truncation, and abort behavior.
• Real DeepSeek Search/Fetch run with a real API key: 10 raw search results, summaries and content, valid fetch payloads for normal pages, and documented errors for 404 and invalid URLs.
• Real pi process end-to-end run: a mock unsupported current model plus DeepSeek credentials exposed web_search, called it, auto-switched to the native DeepSeek API, returned 10 raw results, and answered with the tool result attached.
• Other provider families are covered by unit/integration tests with mocked requests and streams; live credentials for every provider are not required.

The opt-in live DeepSeek test is reproducible from a repository checkout:

DEEPSEEK_API_KEY=... npm run test:live:deepseek

Compatibility

• pi 1.3+. The extension uses the current extension tool and model registry APIs.
• Node.js 22.19+. Pi loads the TypeScript extension directly; there is no compile step.
• Pi provides @earendil-works/pi-coding-agent, @earendil-works/pi-ai, and typebox as host packages. They are declared as peer dependencies and are not bundled.
• Upstream provider API changes may require adapter updates. Unknown provider fields are preserved rather than rewritten.

Development

npm ci --ignore-scripts
npm run check
npm test
npm run verify:package
npm run build

npm run build runs the checks and tests, verifies the npm file list, and packs dist/pi-better-web-search-0.1.9.tgz. Nothing is published automatically.

The test suite uses plain .mjs files that import the .ts sources directly. It mocks globalThis.fetch, uses a temporary cache file, and exercises exposure, fallback, cache invalidation, raw-result preservation, truncation, URL handling, and provider switching.

See Testing for live-provider tests and the real pi end-to-end procedure. See Publishing for the release checklist.

Contributing

Use English for issues, pull requests, and primary documentation. Chinese translations are welcome. Read CONTRIBUTING.md before changing provider behavior, exposure rules, or output handling.

The extension keeps one invariant above all others: trust the provider's result. Do not add cleanup that rewrites, filters, re-ranks, summarizes, or reconstructs provider data.

License

MIT. An independent community package, not an official Pi product.

The web_search extraction is derived from the MIT-licensed pi-web-search project. The repository structure and documentation conventions are inspired by pi-trim.