Monitoring

Learn how to effectively monitor and optimize your large-scale web scraping projects with our expert tips and tutorials. Maximize your scraping efficiency.

6 Posts

Monitoring

Why you need to monitor long-running large-scale scraping projects

A great deal can go wrong in long-running large-scale web scraping projects. We go through a few of the common problems you’re likely to encounter, which can be mitigated or prevented by monitoring.

Theo Vasilis

Apr 6, 2023

Tutorial

How to set up a content change watchdog for any website in 5 minutes

Learn how to automate website monitoring and get notified whenever a website changes.

Natasha Lekh

Mar 13, 2023

Engineering

How we monitor infrastructure that processes 1 petabyte of data every month

What happens at Apify when AWS or DockerHub go down? Find out about our monitoring and alerting architecture. How we use New Relic, LogDNA, Sentry, and PagerDuty to keep our systems up and running.

Marek Trunkát

Feb 21, 2022

Use Cases

How Apify uses traffic monitoring to ensure fair billing

Discover the importance of efficiently monitoring the traffic users generate and how Apify ensures that customers are fairly billed for the resources they consume.

Matej Hamas

Dec 6, 2021

Tutorial

Keep up with Medium responses

Enables the set up of notifications across multiple Medium members, checks for new responses on posts, sends email or Slack notification.

Lucie Burisin

Aug 12, 2020

Tutorial

How to never miss a beat on ever changing websites

The web is constantly changing. Posts in discussions are being deleted, online stores are changing prices on hourly bases and news sites are updating their articles even weeks after publication.

Marek Trunkát

Nov 9, 2017