Web scraping: the beginner's guide

What is web scraping?

Web scraping is the process of automatically extracting data from websites. Any publicly accessible web page can be analyzed and processed to extract information – or data. These data can then be downloaded or stored so that they can be used for any purpose outside the original website.

What is the point of web scraping?

The web is the greatest repository of knowledge and data in the history of humanity. But that information was designed to be read by human beings, not machines. Web scraping enables you to create rules for computers to access those data in an efficient and machine-readable way. It is already impossible for humans to process even a fraction of the data on the web. That's why web scraping is becoming essential. We need machines to read that data for us so that we can use it in business, conservation, protecting human rights, fighting crime, and any number of projects that can benefit from the kind of data that the Internet is so good at accumulating.

To ignore the potential of web scraping is to ignore the potential of the web.

What is web scraping used for?

Web scraping allows you to collect structured data. Structured data is just a way to say that the information is easy for computers to read or add to a database. Instead of relying on humans to read or process web pages, computers can rapidly use that data in lots of unexpected and useful ways. To illustrate the difference, imagine how long it might take you to manually copy and paste text from 100 web pages. A machine could do it in less than a second if you give it the correct instructions. It can also do it repeatedly, tirelessly, and at any scale. Forget about 100 pages. A computer could deal with 1,000,000 pages in the time it would take you to open just the first few.

Ways web scraping can benefit business

Web scraping gives you access to a lot of data. Those data can be:

• loaded into databases
• added to spreadsheets
• used in apps
• repurposed in surprising and unexpected ways

Here are just some of the ways web scraping can help your business be more efficient and profitable:

• Price tracking
  Be more competitive by tracking the prices of your competitors in real time and with the ability to adjust your own prices on the fly. You can even tell your own customers what your competitors are up to so that they see the advantages of buying from you instead.
• Lead generation
  Generate smart leads by scraping publicly available contact information and social media platform profiles to find new customers and potential business leads.
• Content aggregation
  Aggregate content to create new uses for data, make data easier to read or add value by notifying users when prices or content changes.
• Market analysis
  Gain market insights by scraping data about your business, customer demand, feedback in the wild, or even identify opportunities in the real world by analyzing demographic changes and trends.
• SEO
  Improve your SEO by monitoring keywords, popularity, and trends across the web.

If you would like to read more about other businesses and industries that use web scraping, check out our use cases and success stories. You’ll find examples of how retailer price monitoring, machine learning, copyright protection, and even moms returning to work can benefit from web scraping.

Web scraping can also benefit humanity

Web scraping isn’t only used for financial gain. Organizations around the world are using web scraping to help.

Find missing animals

Combat human trafficking networks

Encourage forest restoration

Track COVID-19

Advantages of web scraping

Speed
Web scraping is the fastest way to get data from websites and it means that you don’t have to spend time manually collecting that data. On top of that, you can scrape multiple websites at the same time. No more copying and pasting data. You set up your scrapers and they tirelessly and rapidly gather data whenever you need it. Want to extract all pricing and listing information on thousands of products in minutes? No problem.

Data at scale
Web scraping tools provide you with data at much greater volume than you would ever be able to collect manually. Robots win over humans every time when you’re dealing with huge amounts of information. Scrapers will supply you with terabytes of data in seconds, sorted, organized, and ready to use. There is no other solution that can deliver the mind-boggling amount of data that modern scraping makes possible.

Cost-effective
Think you need a complex system to scrape? Think again! You’ll often find that a simple scraper can do the job, so you don’t need to invest in more staff or worry about development costs. Scraping tools are all about the automation of repetitive tasks, but those tasks are often not that complicated. Even better, you might not even need to create or order a new scraper, because there are so many ready-made tools out there.

Modifiable and flexible
Scrapers are even more cost-effective because they are completely customizable. Create a scraper for one task and you can often retrofit it for a different task by making only small changes. And they aren’t hard-coded solutions that can’t be changed as your circumstances or challenges change. Scraping bots are tools that can adjust and adapt to your workflow as you grow.

Accurate, reliable, and robust
Set up your scraper correctly and it will accurately collect data directly from websites, with a very low chance of errors being introduced. Humans aren’t good at monotonous, repetitive tasks. We get bored, our attention wanders, and we have limits on how fast we can work. Bots don’t have those problems, so if you get the initial setup right, you can be sure that your scraper will give you reliable and accurate results for as long as you need it.

Low maintenance costs
The cost of maintaining a scraping solution is low because of the inherent flexibility of scrapers. Websites change over time, with new designs, categories, and layouts. A scraper needs to be updated so that it can react to those changes. But these kinds of changes can usually be accommodated by slightly tweaking the scraper. The maintenance of a scraper might just be a matter of changing a single variable or updating a single field, so you don’t need a whole team of developers to keep your scrapers up and running.

Automatic delivery of structured data
Computers like to be given information that has structure so that they can easily read and sort it. This just means that each piece of data has to be organized into what would look like a spreadsheet to us humans. Scraped data arrives in a machine-readable format by default, so simple values can often immediately be used in other databases and programs. If you set up your scraping solution correctly, you will get structured data that will work seamlessly with other tools.

Disadvantages of web scraping

Web scraping has a learning curve
It can be intimidating to think about the programming that goes into creating a scraper. But most companies that use scrapers don’t need to think about that, as there are ready-made solutions that work for many different use cases. Sure, if you decide to create your own scraper from scratch, it can be time-consuming, but there are also great communities you can turn to for help, along with extensive documentation to guide you.

Web scraping needs perpetual maintenance
No web scraping solution can be set and forgotten forever. Because your scraper depends on an external website, you have no control over when that website changes its structure or content, so you need to react if the scraper becomes outdated. That will mean paying regular attention to your results and making sure that your data remains relevant and accurate. Maintenance might be a fact of life for web scrapers, but that’s an unavoidable truth about most solutions that give you value.

Data extraction is not the same as data analysis
This is mostly a question of setting realistic expectations. No matter how good the scraping tool you’re using, it is designed to do a simple task. It collects data, sorts it into a structured format, and delivers it to your computer or database without any data loss. The data will arrive in a structured format, but more complex data will need to be processed so that it can be used in other programs. This process can be quite resource-intensive and time-consuming, so you should be prepared for it if you’re up against a big data analysis project.

Scrapers can be blocked
Some websites just don’t like to be scraped. This might be because they believe that scrapers are consuming their resources, or just because they don’t want to make it easy for other companies to compete with them. In some cases, access is blocked because of the origin of the scraper, so that a request coming from a particular country or IP address is not permitted. This kind of IP blocking is often solved by the use of proxy servers or by taking measures to prevent browser or device fingerprinting. But as web scraping has become a more widespread tool for many businesses, websites are becoming less suspicious of scraping and lowering some of their resistance to it. So even if a website has blocked scrapers in the past, that may change over time.

Is web scraping legal?

Web scraping is just a way to get information from websites. That information is already publicly available on the internet, but it is delivered in a way that is optimized for humans. Web scraping simply optimizes it for machines. Web scraping is not hacking, and it is not intended to cause problems for the websites that are scraped.

Web scraping is legal, but it's all a matter of what you scrape and how you scrape it. It’s like taking pictures with your phone. Most of the time it will be legal, but taking pictures of an army base or confidential documents could get you in trouble. Web scraping is the same. There is no law or rule banning web scraping. But that doesn't mean you can scrape everything.

Here are some good rules of thumb to follow when creating a scraper:

• Avoid scraping large amounts of personal data unless you know the rules.
• Don't overload the servers of the website you're scraping.
• Only scrape publicly available information.
• Don't scrape or use copyrighted content.

How does the web work?

Before you start getting into the world of web scraping, it might help to understand more about how the Internet and the web work.

The Internet was born during the Cold War in the 1960s, but the web came into being many years later when Sir Tim Berners-Lee proposed a networked hypertext system to his boss at CERN.

That idea eventually led Berners-Lee to create three important technologies:

• Hypertext Transfer Protocol (HTTP). This enables computers to retrieve linked resources across the web.
• Hypertext Markup Language (HTML). The markup language of the web. Allows text to be formatted so that it can be displayed correctly.
• Uniform Resource Locator (URL). Otherwise known as a “web address”. Used to identify all the resources on the web.

Put those together and you have the vital building blocks of what eventually became known as the World Wide Web.

Decentralization was fundamental to the early web as envisaged by Berners-Lee, as was universal compatibility and making it simple to share information. Over time, standards were established through a transparent and participatory process by the World Wide Web Consortium (W3C). These open standards are one of the cornerstones that have made it possible for the web to grow.

Berners-Lee still firmly believes that it is vital to “defend and advance the open web as a public good and a basic right” and created the World Wide Web Foundation just over ten years ago to ensure digital equality and transparency for everyone.

That vision of an open web is just as important now as it was then. And making data accessible to everyone is part of keeping the web open. That’s where web scraping comes in.

What is a web browser?

You’re using a web browser to view this web page. A web browser is just software, or a computer program, that enables you to access, view and interact with web pages.

How do web browsers work?

Your browser retrieves information from the web and displays it on your computer or mobile device. It uses the Hypertext Transfer Protocol (HTTP) to retrieve the content of websites and Hypertext Markup Language (HTML) to determine how to render the content. The final result is that you see a web page on your device, and you can interact with that web page. Underlying the web page can be a multitude of other technologies, such as HTML, CSS, JavaScript, etc.

How do I start web scraping?

We find that web scraping works best if you pause and ask yourself these three questions before you start coding or ordering a solution:

1. What information are you looking for? What data do you want to get?

2. Where can you find the data? What’s the website and what’s the URL?

3. What will you do with the data? What format do you need it?

Once you’ve answered these questions, you can start thinking about how you will scrape the data you want.

Basic scraping terminology

Web scraping: The process of automatically extracting data from websites. Also known as screen scraping, web data extraction, web harvesting.

Web scrapping: This is just a really common and easy-to-make typo!

Web crawling: Web crawlers are spiders or spider bots that systematically browse the web and index it. Search engines use these bots to make it easier for us to search the web.

Structured data: Information that is organized and formatted in such a way that it is easy for computers to read and store in databases. A spreadsheet is a good example of how data can be organized in a structured way.

Hypertext Transfer Protocol (HTTP): Enables computers to retrieve linked resources across the web.

Hypertext Markup Language (HTML): The markup language of the web. Allows text to be formatted so that it can be displayed correctly.

Uniform Resource Locator (URL): A “web address”. Used to identify all the resources on the web.

Cascading Style Sheets (CSS): The design language of the web. It enables web page authors to style content and control presentation across an entire website.

JavaScript: A programming language used all over the Internet to control the behavior of websites and enable complicated interaction between user and web page.

IP address: An Internet Protocol address is a number assigned to every device on the Internet. These numbers allow devices to communicate with each other.

Proxy: A proxy server is a device that acts as an intermediary between other devices on the Internet. Proxies are commonly used to hide the geographical location of a particular device, often for privacy reasons.

Application Programming Interface (API): A computing interface that makes it possible for multiple different applications to communicate with each other. An API operates as a set of rules to tell the software what requests or instructions can be exchanged and how data are to be transmitted. Apify got its name from API 😉

Software Development Kit (SDK): A package that enables developers to create applications on a particular platform. An SDK can include programming libraries, APIs, debugging tools and utilities designed to make it easy for a developer to use the platform. Apify has its own SDK.

Web scraping companies and tools

So you want to start web scraping, you know what you want to scrape, and you’ve decided to explore the ways you can start.

There are lots of methods and companies out there involved in web scraping. To help you choose, let’s split the web scraping world into four different categories.

Enterprise consulting companies

These provide high-end turnkey “data-as-a-service” solutions to large companies. They will carry out scraping at any scale, but at a price.

Examples: Import.io, Mozenda, Apify.

Point-and-click tools

Allow you to go to a website and just click on the elements you want to scrape. These are good enough for simple use cases, but not so good for more complicated projects.

Examples: Dexi.

Programming platforms

A platform is designed for developers and offers a lot of flexibility. Instead of building the infrastructure for scraping, you use an existing system that was specifically designed for the task.

Examples: Zyte, Apify.

AI knowledge extractors

These companies take an AI approach and attempt to extract data from websites automatically. It works for standardized pages, but is not flexible enough to cover a variety of use cases.

Examples: DiffBot.

Read about the Top 10 free web scraping tools for data analysts.

Take a look at the other web scraping companies and tools you might have heard of on our Apify alternatives page.

Web scraping with Apify

Apify offers several different ways to scrape. You can start from scratch with your own solution, build upon existing tools, use ready-made tools, or get a solution created for you.

Enterprise solution

Enterprise customers can order a more specialized web scraping or automation solution at any scale from a dedicated Apify data expert. We will work with you all the way to project completion and can continue to provide maintenance once it is up and running.

Order a custom solution

Developing your own web scrapers or web automation robots can take a lot of time and effort. With Apify, you can delegate this job to experts who will deliver a turn-key solution just for you.

Use a ready-made tool

Apify Store has existing solutions for popular sites. This is the quickest way to get your data as the tools are already optimized for particular use cases. Our tools are designed to be easy for even those with no previous coding experience and our support team is always ready to help.

The output data will contain the following information about each movie or series that you have listed in the input schema of the IMDb scraper:

[
	{
		title: "The Queen's Gambit",
		original title: "",
		runtime: 395,
		certificate: "TV-MA",
		year: "",
		rating: "8.6",
		ratingcount: "250392",
		description: "Orphaned at the tender age of nine, prodigious
		introvert Beth Harmon discovers and masters the game of
		chess in 1960s USA. But child stardom comes at a price.",
		stars: "Anya Taylor-Joy, Chloe Pirrie, Bill Camp",
		director: "",
		genre: "Drama, Sport",
		country: "USA",
		url: "https: //www.imdb.com/title/tt10048342"
	}
]

Code it yourself

You can use our generic scrapers and customize them with just a bit of JavaScript. Or you can use Apify SDK to create your own scraping solution.

1. Inspect the source of your data, in other words this link (remember that you just have to right-click on the page and select “Inspect” at the bottom of the menu), and find and select the information you want to scrape. For our example, the code will look like this:

<span itemprop="ratingValue">8.6</span>

2. Create a task for Web Scraper on the Apify platform by clicking on Try for free.

3. Paste the URL to the Queen's Gambit IMDb page into the Start URLs field and replace the code in the Page function field with the code below. Remove the Link selector and Pseudo-URLs fields.

async function pageFunction(context) {
  const $ = context.jQuery;
  return {
    url: context.request.url,
    rating: +$('[itemprop="ratingValue"]').text().trim(),
    ratingCount: +$('[itemprop="ratingCount"]').text().replace(/[^\d]+/g, '') || null,
    title: $('.title_wrapper h1').text().trim(),
  };
}

4. Click Save and run and then check the dataset with the final result.

{
  url: "https: //www.imdb.com/title/tt10048342"
  rating: "8.6",
  ratingcount: "250392",
  title: "The Queen's Gambit",
}

Tip: for a more detailed explanation, check out our extensive tutorial for this scraper.
If you still can’t decide which option is right for you, read more on choosing the right solution or just email us at hello@apify.com for free expert advice on your use case.
Not sure which web scraping solution is right for you? Compare the benefits of using Apify side by side with its alternatives to help you decide.

Learn web scraping

Now that you know the basics of web scraping, you might want to explore the topic further. To save you time, we’ve collected a few courses and tutorials suitable for all levels. We recommend these as a great way to quickly get up to speed on web scraping.

Courses for beginners

Udemy has a course for beginners to introduce you to web scraping in 60 minutes.

Pluralsight has a course on web scraping with Python for more experienced beginners.

Coursera has a guided project on scraping with Python and Beautiful Soup, for much more advanced users.

Guides for beginners

Our own Apify blog has general articles to inspire you and also several step-by-step guides to scraping popular websites.

• What’s the difference between web scraping and crawling?
• How to scrape any website for absolute beginners.
• How to scrape Facebook pages.
• Scraping Google Maps locations.