Some websites, typically in the area of e-commerce, provide a feature called a watchdog that you can use to get a notification when something changes (usually a product price). This feature is really handy - you don’t need to remember to regularly open the same page to see if something changed. Our Content Checker lets you set up such a watchdog for any website and get an email notification whenever the content you want to monitor changes.
Just think about how often you’ve checked the same page every evening to see if tickets are already available or on sale, a product price is below what you want to pay, your competitor changed their homepage, or there’s a new version of a library your work depends on. This is the issue that Content Checker is created to solve and we’ll show you how to set up this watchdog to keep you in the loop before everyone else.
Applications of a watchdog program
Content Checker is one of the most popular actors on the Apify platform. The reason for this is simple: it allows you to monitor specific content on any web page and sends a notification with before and after screenshots whenever that content changes. This allows automating monitoring anything from prices to availability which can be useful both for business and personal use.
This makes the Content Checker a truly universal solution because you can use it to:
- create your own watchdog for retail prices, keep an eye on product updates and sales in any industry, from real estate to e-commerce
- check the availability of appointments and tickets - from concert venues to hotel listings
- track your competitors’ activity and carry out well-rounded market research - from prices to product names
- receive the freshest news updates - from specific categories to general ones
Basically, the scraper enables you to automate tracking changes in any content that you want to keep an eye on. It can be literally any parameter that gets updated - that’s the best part of the idea behind web automation. You don’t have to spend time doing things manually if you can get a bot to do it!
How Content Checker works
The logic behind the website checker is simple as well: it searches for specific text in a chosen CSS selector and compares the current results to the original state. If there is any change, Content Checker launches another actor to send an email or a Slack notification and also makes and sends screenshots before and after the change occurred on the website.
Step-by-step guide on how to set up your own website watchdog program
1. Go to Apify’s website: http://apify.com
2. Sign in at the top-right corner using your email account, Google, or GitHub.
4. When in Apify Store, just type Content Checker in the search bar. You'll see your scraper show up in the drop-down list. Click on it. You can also find that scraper among others in the automation category.
5. Now you're on the Content Checker page. Scroll down or explore tabs to get familiar with its parameters and possibilities. The Readme and Input tabs will tell you everything you should know about how to set up this scraper to fit your unique needs.
6. Click on the Try me button. You’ll be redirected to the Apify Console with Content Checker ready to be set up. Apify Console is the main tab where we’ll be working from now on.
8. In this example, we’ll be scraping a website full of event announcements. We want to scrape it from time to time and get notified every time there’s a new jazz concert announcement in our area (New York). So we go to the upper Searcher bar, set up the City, Genres, Sort by sections and click GO. It is crucial to have your search sorted by Newly Announced since we want only the latest, freshest concert announcements.
9. Now just copy the URL that you’ve got from your search. Go to the Apify Console tab with the Content Checker page open, and paste the copied link into the first field - URL to check.
10. The next step is finding the right selector on the concert website. In our case, we need the very first box that gets refreshed with a new entry when a new concert is added. So right-click on the page, choose Inspect and target the first box. You’ll see what it’s called in the code. In our example, this element is called event_row. This is all we need to know from this inspection. Back to Apify Console.
11. Now type into the .event_row:nth-of-type(1) into both selector fields. This combination will let the Content Checker know which selector to look for and check when scraping that concert web page. You can read more about the nth-of-type (1) command in our SDK library. Combined with the URL we put above, with this command, we're simply telling the scraper to only watch for the first box that appears at the top of the concert’s webpage - only the newly announced concerts.
12. Fill in your email address in the corresponding box, add the message you want to receive when getting an email from your scraper and click the Save and run button. If the actor shows the status Succeeded after the run - it’s all ready.
The interesting thing is that most likely, your scraping run will show you zero results. However, collecting results is not the point of this scraper - its purpose is only to notify you if a change occurred. So the moment changes on the website happen, you will receive a notification, and get handy before and after screenshots in your email inbox.
13. Last but not least - you can schedule your scraper to check the website every now and then on a regular basis. You can set that up in the Schedules tab. You can tell Content Checker to check on your chosen selector hourly, daily, and so on - and can even customize the schedule based on different time zones.
And that’s it - you’re all set. The pattern of the website checker will run every day, scrape the content of your selector, compare it with the previous run and send you a notification if the content changes (along with the previous and current content and screenshots).
Our APIs - both the free and the paid ones - can also be used in conjunction with each other, so if you prefer getting Slack notifications rather than emails - you can pair the Content Checker scraper with a Slack notification one.
We have another scraper that can be customized in a similar way - Web Scraper. It is even more adjustable and flexible than Content Checker, so if you feel like challenging yourself with scraper customization, here’s a detailed tutorial of how to set it up according to your unique scraping needs.
And that’s everything you need to know to get started using this actor. Be sure to check out other actors in Apify Store - they're just as great, although most are definitely easier to adapt. If you come up with any interesting ways to use Content Checker, we’ll be happy to hear about it from you on Twitter :)