pi-webfetch

pi-webfetch is a standalone TypeScript pi package that registers exactly one tool: webfetch.

The tool fetches a single public URL and returns AI-friendly markdown by default, with optional plain text or cleaned HTML output.

Install

Local path

If you are installing from a source checkout, run npm install first so the compiled extension exists under dist/extensions.

pi install /path/to/pi-webfetch

Git

pi install https://github.com/langgengydst/pi-webfetch

npm

pi install npm:@lydst/pi-webfetch

Direct extension loading

After building the package locally:

npm run build
pi --no-extensions -e ./dist/extensions/webfetch.js

Tool

Name

webfetch

Description

Fetch a public web page and return AI-optimized content in markdown, text, or html.

Arguments

Argument	Type	Default	Notes
url	string	required	Fully-qualified http or https URL
format	"markdown" | "text" | "html"	"markdown"	Markdown is the default output
timeout	number	20000	Milliseconds, min 1000, max 120000
maxChars	number	25000	Applied after conversion, min 1000, max 200000
mainContentOnly	boolean	true	Prefer readable article/main content
includeMetadata	boolean	true	Includes optional source metadata such as title and description
includeLinksSummary	boolean	false	Adds discovered links to details.links
includeImagesSummary	boolean	false	Adds discovered images to details.images

Behavior

• Accepts only http and https URLs.
• Uses fetch with AbortController timeout enforcement.
• Follows redirects and records the final URL in metadata.
• Accepts text/html, text/plain, and text/markdown.
• Converts HTML to cleaned markdown by default.
• Preserves code blocks during markdown conversion.
• Applies maxChars to the final returned text, including the source header.
• Truncates after conversion with a visible suffix when maxChars is exceeded.
• Returns stable error codes such as INVALID_URL, UNSUPPORTED_PROTOCOL, TIMEOUT, HTTP_ERROR, and UNSUPPORTED_CONTENT_TYPE.

Result shape

Successful calls return content like:

{
  "content": [{ "type": "text", "text": "Source: https://example.com/..." }],
  "details": {
    "url": "https://example.com/start",
    "finalUrl": "https://example.com/final",
    "format": "markdown",
    "status": 200,
    "ok": true,
    "contentType": "text/html",
    "fetchedAt": "2026-04-03T10:00:00.000Z",
    "truncated": false,
    "originalLength": 1234,
    "returnedLength": 1234
  }
}

Failures still return a tool result with readable text plus details.errorCode and details.isError.

When the fetched source is already text/plain or text/markdown, details.format reflects the actual returned representation.

Usage examples

• Fetch https://example.com and summarize it.
• Use webfetch to get https://example.com/blog/post as markdown.
• Fetch https://example.com/changelog as text with a 10000ms timeout.
• Use webfetch on https://example.com/docs with format html and maxChars 12000.

Development

npm test
npm run build
npm pack --dry-run

The package builds ESM output into dist/ and exposes the compiled extension directory through:

{
  "pi": {
    "extensions": ["./dist/extensions"]
  }
}

Limitations

• No JavaScript browser automation or rendered-page execution.
• No login-protected or session-authenticated pages.
• No multi-page crawling.
• No PDF or image OCR.
• No guaranteed robots.txt enforcement. This package does not claim robots enforcement in v1.