Did you know that, technically, basic web crawling and SEO monitoring share the same background? The reason for this is simple: in order to identify broken links, you need to crawl and parse the website first. It's impossible to detect bad links without first crawling the content. So making smart SEO moves is a logical extension of web scraping. In this tutorial, we'll show you how, with a basic link-checking tool, web scraping can take your search engine optimization to a whole new level.
🔗 What is a Broken Link Checker?
Broken Link Checker is an online tool that specializes in uncovering those elusive culprits known as broken links. Bad hyperlinks, dead links, broken links, bad URLs—whatever you choose to call them, they have no place in the seamless digital landscape of your website. While it can't perform the miraculous feat of fixing these wayward links for you, what it can do with unmatched efficiency is track them down. It tirelessly scours your website, meticulously pinpointing the exact locations of these troublesome links, leaving you with the valuable information needed to take swift corrective action.
🚧 Why is it bad to have broken links on a website?
Let's get the obvious things out of the way: there are two main reasons why such a seemingly small thing as a few broken links are damaging to the SEO health of a website. For a user, it's frustrating; for a website creator, it's potentially dragging down their work on the mysterious Google charts. This is why battling 404 error issues, and similar errors on web pages is a priority task for a website owner and following a guide to managing backlinks effectively can help maintain and improve your site's SEO health.
🧑💻 User experience
The web is constantly in flux, so as time passes, there's an increasing chance for the backlinks to be replaced whether you acquired them through backlinks service providers or found placements yourself. The same goes for internally produced URLs such as a catalog of the services, product descriptions, and content in general expanding and transforming. So it is natural that when the domain gets old enough, the dead links might and will appear more often.
Now, should they? For a user who doesn't need to care for the internal URL kitchen of your domain, coming across a page with a dead link might suggest poor maintenance and a general UX feeling of being dated. So users will most likely just vote with their feet (or cursors). A decision to leave a web page because of a broken link might seem minor on a small scale such as a blog, for example. But for businesses in such industries as e-commerce, several broken links of products during a promotional season might be all the difference between them and the competitors.
⚙️ Search engine algorithms
In a perfect internet world, a website visitor should never see a Page Not Found message. An all-seeing search engine should enjoy the same experience - to be able to help your website show up in the ranks. While a website user may just shrug and walk away from a website (bad enough as it is), the Googlebot crawler will take digital notes and cast down the website's rating - thus lowering its chances of being found in the first place. There's a whole art to keeping the Google algorithms happy, and running regular checks of broken links on web pages is a part of mastering it. Consider leveraging a backlinks service to help you enhance your website's SEO and improve its overall performance.
These days, nobody has time to manually check links every other week just to see if all of them work. Using an online tool to monitor broken links is one of the easiest ways to maintain a good user experience, keep SEO indicators at a decent level, and generally provide the value the user expects from the website. There are many tools to check links and keep track of their performance, so naturally, some are better than others, some are more expensive than others, and some are free! Here's our free Broken Link Checker and instructions on how to use it.
👷♂️ How this free link checker works
This tool starts the link check with an original URL and then moves down the hierarchy of URLs. So, for instance, if the crawler starts at https://www.example.com/something, then you can expect it to also check all the pages down the line, such as:
https://www.example.com/something/index.html
https://www.example.com/something/else
https://www.example.com/something/even/more/deeper/file.htmlIt digs down every page and checks their status codes until it finds those that are not 200 OK – a.k.a the broken pages. Another thing that the link checker will do is analyze whether the links to other pages are working or not. For example, if the page contains a link to .../another/page#anchor, the crawler will do a triple check: open the page .../another/page, confirm whether it loads, and make sure it includes the #anchor part. If the link doesn't match this little checklist, the link checker will flag it and include it in your report.
🛠 What a basic broken link checker should be able to do
- Carry out a basic SEO audit of the website, an SEO checklist might be helpful off page SEO checklist
- Scrape the main domain and subdomains
- Monitor both external links and internal links (inbound and outbound links)
- Check all links and find all the dead links
- Identify broken fragments of each dead link separately
- Automate link monitoring by crawling the website daily, weekly, or monthly
- Present link-checking results and website authority evaluation in various formats
All of these are possible with this Borken Link Checker tool.
🚧 What can I do using the Broken Links Checker?
Armed with all that SEO knowledge, here's a small action plan that you as one of the website owners or maintainers can carry out using this link checker:
- Make a basic SEO audit or dig deeper into the subdomains.
- Check the whole website or zoom in on particular pages.
- Get a comprehensive report containing an assessment of all links or just the broken ones.
- Do an auto-check or be able to launch the site check manually, on-demand.
- Quickly zero in on broken URL fragments and fix them.
As an extension of these practical things, here are some strategic things you can accomplish:
- Prevent link rot and loss of link equity at early stages
- Speed up the content audits
- Replace broken backlinks with active ones to maintain your website's authority
- Ensure that all forms and registration pages are accessible and working correctly
- Identify broken link-building opportunities: find broken links in your competitor's web resources and use them to your advantage
And that's just for starters! You can easily customize our Broken Link Checker to suit your SEO-related needs. No need to install a plugin, some random browser extension, or activate the downloaded software; no credit card required. Just create a free Apify account and let's try how this simple dead link checker works, step-by-step.
⛓ How to use Broken Link Checker
Step 1. Find free Broken Link Checker
Go to the Broken Links Checker page among the SEO tools in Apify Store and click the ▶️ Try for free button.
If you're not signed in, you'll land on the sign-up page. You can sign up using your email account, Gmail, or GitHub, and get your free Apify account without providing credit card details. You will be redirected to Apify Console – your workspace to create tasks for your scrapers, crawlers, actors, and integrations.
Step 2. Add a domain to check broken links
Broken Link Checker only requires a few input fields:
- URL of the website to be checked for bad links. In our example, we're crawling Apify Blog.
- Two toggle buttons to keep in mind: Save only broken links will exclude thousands of healthy links from the report (recommended), and Crawl subdomains will dive deeper into if needed.
- Your email address to receive the report with identified links needing your attention.
- You can also limit the number of crawled pages in Max pages field unless you want a thorough website check.
Once you're all set, click the Start ▶️ button. Notice that your task will change its status to Running 🏃🏻♀️, so wait for the crawler's run to finish. It might take some time for the crawler to go through all the links before seeing the status turn into Succeeded 🏁. For instance, our Apify Blog check took around an hour to complete.
Step 3. View the results
Now click on the Output tab to see how many broken links you've got. You can download the list of problematic links either as a machine-readable JSON report or an easy-to-read HTML table. You can also preview the data by clicking the ⤴️ Preview button.
Step 4. Download the full report
You can download the SEO report onto your computer for further use.
Step 5. Check your email
Don't forget to check your inbox! If you also included your email address in the input field, the full broken links report will appear there as soon as the run finishes 🚀
Step 6. Schedule crawls for broken links
Bonus: schedule the scraper to run automatically once per a defined period: daily, weekly, monthly. You will receive the broken links notification paired with a report in your inbox once the job is done.
Watch this short video to learn how to schedule your scrapers, crawlers and automation tools
You can get started with Broken Link Checker right now - the trial is free and you get $5 credits on top to test it out and do a broken links check online. Not bad for a free tool!
💡 Other free SEO tools to track website performance
With regular monitoring, purging, and replacements of dead links, the day may come soon when our link-checking tool will show no results for the broken links on your website. Automation tools are the foundation of keeping the website's SEO health score and link equity on optimal level and combining a few of them will improve it. Here are several other free SEO monitoring tools in the Apify stack:
- SEO Audit Tool 🔗 for a more comprehensive SEO inspection.
2. Google Search Results Scraper 🔍 for monitoring your website's position on Google SERPs.
3. Content Checker ☑️ for notifying you whenever website content changes (any part of the website).
4. GPT Scraper 📚 to improve meta descriptions of your pages.
All of these small SEO tools are free to try and make up a great combo to make your SEO journey easier. They will crawl the website data and deliver it to you in machine-readable formats such as Excel, JSON, CSV, or XML. Those file formats are made to fit the apps, data analysis tools, and data projects.
