Philosopher turned self-taught developer. I manage a team of devs at Apify, and still love going deep into programming concepts. I like gardening, sports, and dogs.
Scaling web scrapers horizontally across multiple containers for optimal concurrency. Tips and advice for large-scale scraping projects using Puppeteer or Playwright.
When scraping very large or complex websites, a normal scraping workflow won't get the job done. Read on to find out how to overcome the challenges of large-scale scraping.
Unleash the full potential of Apify automation tools! Integrate your Apify tools with Google spreadsheets to upload lists of URLs, save crawled data for later processing, and more.