It’s not that hard. I’ve done this. The list of URLs for a crawl is several hund... | Hacker News

Hacker Newsnew | past | comments | ask | show | jobs | submit

binarymax 48 days ago | parent | context | favorite | on: An update on residential proxies and the scraper s...

It’s not that hard. I’ve done this. The list of URLs for a crawl is several hundred gigs. Easily fits in a lookup index on a single instance.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact