SiftCrawl

Paste a URL and SiftCrawl auto-detects the repeating listing pattern and the pagination. You map CSS selectors to typed fields, optionally follow every item to its detail page, and export the results as CSV, JSON, XLSX or SQLite. Cheerio parses the HTML, with Playwright (Chromium) as the fallback for JavaScript-rendered pages. A graph view shows how the fields relate.

It started as a single-user local tool and I hardened it for multi-tenant SaaS use. Every fetch goes through an SSRF guard that resolves DNS and blocks private, loopback, link-local, CGNAT and cloud-metadata ranges with Node’s built-in BlockList. User-supplied regex is checked for catastrophic patterns at save time, then runs in a worker thread that is killed after one second, so a (a+)+$ pattern can’t hang the server.

Supabase Auth protects every page and API route, with per-user tenancy scoping on every query, and every request body is validated with zod. robots.txt is actually respected now, proxies work on the HTTP path too, and a 100-page cap is enforced.

For throughput, detail pages are fetched concurrently (bounded at 4) instead of one by one, which made jobs about 4× faster. The job queue is durable on SQLite, claims jobs atomically, runs two at a time and survives restarts. Requests retry with exponential backoff and jitter, back off per domain on HTTP 429 (honouring Retry-After), and use an HTML cache with a 6-hour TTL.