SSE Real-time

Live crawl events streamed from server

Zero limits

Unlimited pages & recursive depth

Smart checking

HEAD → GET fallback, 429 retry backoff

Robots.txt aware

Optionally respect crawl disallow rules