scrapy-spider-gen

A Pi extension that generates Scrapy spiders from web pages. Downloads HTML, strips the noise, and feeds the cleaned DOM to the LLM for automatic selector extraction and spider generation.

The problem

A typical e-commerce page weighs 500KB–2MB of raw HTML. Most of it is noise for an LLM trying to identify CSS/XPath selectors on product content:

• <style> / <link> blocks
• <script> tags (analytics, tracking, hydration)
• Dozens of data-* attributes
• Inline SVG icons
• HTML comments

How it works

Raw HTML (2MB)
      │
      ▼
[scrapy-spider-gen extension]
  ├── Strip <script>, <style>, <svg>, <noscript>, <iframe>, …
  ├── Filter parasitic attributes (style, onclick, data-tracking…)
  ├── Keep only structural DOM
  └── Optionally extract product zone only (main, article…)
      │
      ▼
Cleaned HTML (20–50KB)
      │
      ▼
[LLM via Pi]  ←  spider template + item fields
      │
      ▼
Completed Scrapy spider

Install

pi install npm:scrapy-spider-gen

Or from source:

git clone <this-repo>
cd pi-vapalape
npm install
pi install ./

Looking for omp.sh (oh-my-pi) support? See scrapy-spider-gen-omp — a sibling package that targets the @oh-my-pi/pi-coding-agent runtime.

Usage

Command: /spider-create

/spider-create url=<url_or_path> [template=<path>] [level=light|normal|aggressive] [prompt=<text_or_path>]

Parameter	Description	Default
url	HTTPS URL or local file path to the HTML page	required
template	Path to a Scrapy spider template file	—
level	Cleaning intensity	normal
prompt	Custom prompt (replaces the default). Inline text or path to a prompt file	—

Examples:

/spider-create url=https://shop.example.com/product/42
/spider-create url=https://shop.example.com/product/42 template=./spider.py level=aggressive
/spider-create url=./pages/product.html level=light
/spider-create url=https://shop.example.com/product/42 prompt=./my_instructions.txt

Command: /spider-help

Shows extension usage, available commands, clean levels, and custom prompt variables.

Custom prompt variables

When using prompt=<file>, the following variables are resolved in the prompt content:

Variable	Value
{source}	URL or file path of the HTML page
{level}	Cleaning level
{templateClause}	, template="<path>" or empty string

Tool: spider_clean

LLM-callable tool that fetches, cleans, and returns structural HTML for Scrapy selector analysis. The LLM can also call it directly when generating or completing a spider.

Clean levels

Level	What it does
light	Removes <script>, <style>, <svg>, <noscript>, <iframe>, <canvas>, <video>, <audio>, <link>, <meta>, <template>
normal	light + filters parasitic attributes (style, onclick, data-tracking…), keeps only structural attributes (id, class, href, src, alt, data-product-*, Schema.org…)
aggressive	normal + extracts only the product zone using heuristics (Schema.org markers, .product-detail, main, article, [role='main']…)

Safety

Cleaned HTML is capped at 400KB to stay within LLM context limits. If the output exceeds this threshold, it is truncated with a warning.

Architecture

src/
├── index.ts          — Entry point: /spider-create command + spider_clean tool
├── arg-parser.ts     — Parses /spider-create command arguments
├── fetcher.ts        — Downloads HTML from URL or reads local file
├── html-cleaner.ts   — Cleans HTML: removes noise, filters attributes, extracts product zone
└── *.test.ts         — Co-located tests (vitest)

Development

npm install
npm test            # vitest run
npm run check       # biome lint + format
npm run format      # auto-fix formatting
npm run build       # tsc
npm run ci          # check + build + test

License

MIT