A URL frontier hands URLs to fetcher workers, one queue per host so no site is hammered. Fetched pages are stored, parsed for links, and new links are checked against a Bloom filter of everything already seen before going back into the frontier. Content fingerprints catch the same page under different URLs, and a scheduler re-queues pages when they're due for a recrawl.
Functional requirements
- Start from seed URLs and follow links to discover new pages
- Respect robots.txt and per-site rate limits (politeness)
- Skip URLs already crawled and pages whose content is a duplicate
- Store raw pages for indexing and analysis
- Recrawl pages based on how often they change
Non-functional requirements
- Billions of pages per month on a horizontally scalable cluster
- Never overload a single website
- Resilient to slow, broken or malicious sites (timeouts, size limits, traps)
- Workers can fail without losing frontier state
Extensions
- JavaScript rendering for client-side apps (headless browser pool).
- Focused crawling for one topic or set of domains.
- Sitemap and RSS ingestion to find new pages faster.