Announcing Crawlee: the web scraping and browser automation library
Crawlee is an open-source web scraping and automation library that helps you build reliable scrapers. Fast.
4 min read
Web Crawling
Web crawling is what fuels search engines, such as Google. Learn about Apify's web crawlers to help you with your crawling projects.
13 posts
Latest Posts
Bots and automation tools: are they friends or enemies?
BOTS is a four-letter word. But do bots deserve all the bad rap? We explain the difference between good and bad bots, how to use the good ones, and how to protect yourself against the bad ones.
5 min read
Top 11 open source web crawlers - and one powerful web scraper
Learn about free software libraries, packages, and SDKs that can get your web crawling journey started in no time.
6 min read
What are web crawlers and how do they work?
What is web spidering? What do Google crawlers do? And how do they affect SEO? We give you the answers to these and other questions about web crawlers.
4 min read
Powerful tool to check broken links on any website
Read on how and why using a free broken links checker can keep your SEO score healthy.
7 min read
What is the difference between web scraping and crawling?
Want to learn about web scraping and how it differs from web crawling? Read on to understand web crawling vs. web scraping and how web scraping can create an API for any website.
5 min read
Pros and cons of web scraping
If you've ever heard of web scraping or web crawling, this post is for you. In this article, we will talk about the pros and cons of using web scraping tools and technologies.
13 min read
Apify Crawler product to be replaced by an Apify actor
On September 30, 2019, we’ll be retiring the legacy Apify Crawler product together with Apify API version 1. Please migrate your crawlers to tasks of the new apify/legacy-phantomjs-crawler actor, which provides the same functionality, configuration options and results as Apify Crawler.
9 min read
Why the world needs a universal web scraping library for JavaScript
We’ve released the Apify SDK — an open-source Node.js library for scraping and web crawling. There was one for Python, but until now, there was no such library for JavaScript, THE language of the web.
1 min read
One Crawler to Audit Them All
Read about how Agency Analytics only needed a single generic Apify crawler to power their SEO Audit Tool. They were also able to set it up themselves, with just a little support from the Apify team.
2 min read
Dedicated Followers of Fashion: Apify and KNOMI
Apify crawlers work behind the scenes to help KNOMI collect product data from the best independent and luxury boutiques across London.
2 min read
Apifier is now Apify!
Today we are launching the largest upgrade of Apifier to date —a new product called Actor together with a complete redesign of the website. Oh, and Apifier has been renamed to Apify! Note that all existing crawlers are safe and sound — nothing has changed for them.
5 min read