SELFMEND

A Playwright scraper that mends its own broken selectors.

The problem

Most scrapers die the moment a site renames a CSS class. Three pains selfmend addresses:

1. Brittle selectors. A <div class="price"> becomes <div class="product-price"> and your pipeline silently breaks.
2. Maintenance tax. Across N scrapers, you spend hours every month patching the same trivial bugs.
3. Schema drift. Fields get renamed or change type; downstream consumers get garbage and you don't notice for days.

What it does

• Self-healing selectors. When a selector fails, the page's accessibility tree is sent to an LLM, which proposes a new locator. Validators (type / regex / range) gate the proposal — only verified fixes go in the cache.
• Selector cache. SQLite, scored by ✓/✗. Best selector wins; broken ones drift to the bottom and get retried last.
• Stealth + behavioral layer. patchright with anti-detection patches, plus human-like mouse and scroll.
• Schema-drift detector. Flags missing / new / type-changed fields between runs.
• Replay store. Every heal saves the page snapshot for offline debugging.

"I don't want to write YAML"

You don't have to. selfmend init-interactive opens a real browser; you click each field on the page; selfmend infers stable selectors, auto-detects the repeating row container, and writes the spec for you.

selfmend init-interactive https://books.toscrape.com/ -o books.yaml

Or have an LLM draft it from a plain-English description:

selfmend init https://books.toscrape.com/ \
  --want "title, price in pounds, availability" \
  -o books.yaml

Run

pip install -e .
patchright install chromium
export OPENAI_API_KEY=sk-...

selfmend run books.yaml -o data.json
selfmend stats          # selector health table

Architecture

spec.yaml ─► Runner ─► Playwright (patchright stealth + bezier mouse)
              │
              ├─ Selector cache (SQLite, scored by ✓/✗)
              ├─ Healer ─► OpenAI ── a11y-tree snapshot, NOT raw HTML
              │     └─► validator gate ─► cache only if it parses
              ├─ Schema-diff detector
              └─ Replay store (HTML + a11y per heal)