Does Google even need an introduction? If our phones have become extensions of our hands, then Google is one of the main reasons for that evolution. These days, Google is the synonym for answers, speed, and accessibility - but also simply for billions. Billions of dollars, users, accounts, visitors, devices, clicks, minutes, gigabytes, searches - you name it - and of course, data measured in billions of terabytes. And that data can be extracted automatically and effortlessly, if you know the right methods and have the right tools at hand.
In this brief how-to article, we’re going to show you exactly how to scrape the biggest library in the world by using a ready-made tool on the Apify platform called Google Search Results Scraper. This is your step-by-step guide to how to scrape any information available from Google, including organic and paid results, ads, queries, People Also Ask boxes, prices, and reviews. Let’s get started!
What are Google SERPs?
A Google SERP is the page containing the list of search results that Google displays to you when you type in your query and hit Enter. SERP in this case stands for Search Engine Results Page, and you’ll find SERPs not only on Google, which controls 90% of the search engine market, but also on other search engines, such as Bing, Yahoo, Yandex, and others. We need to know this term in order to know how to use web scraping on the Google Search Engine. You can consider the terms Google page, Google search page, and Google SERP to be equal and interchangeable, but we’ll stick with Google SERP in order to remain technically correct.
Google SERPs have changed a lot over the years, with the most prominent features being those infoboxes we all know too well - Knowledge Graphs, as well as the Carousel - something so ubiquitous these days that we can’t imagine the Google SERP interface looking any other way. Both of those now-classic Google SERP features were part of the Hummingbird algorithm release in 2013.
It’s a far cry from the 2003 version of Google results. Does this prehistoric SERP interface ring a bell? Luckily, we’re not there anymore.
Google SERP structure - how to scrape Google
In order to work out how to scrape Google, we first need to understand how it sees and prioritizes our searches. What you see when you search for things on Google is not just an index of pages with URLs, or so-called organic search. While it used to be like that in the past, as we’ve seen, the main purpose and driving force of Google - or any search engine for that matter - has always been to have your queries answered as quickly and efficiently as possible, and in a way that will attract your attention and be easy on the eyes.
That’s why over time the search results have become much more multilayered, including the results of different complexity and formats, sort of like a huge cake. And that cake-like structure isn’t going away any time soon, with voice command search, apps, and mobile search introducing their significant corrections into the way we google stuff. Today, Google Search results consist of various levels, depending on the complexity and type of search, as you can see in this example of a string theory query:
As you can see, the Google search page is now packed with various content: featured snippets, so-called snap packs, ads, and organic results. Additional types may also show up: product ads, related searches, and multiple snap pack types (Wikipedia, Maps, videos, etc).
Google SERP API
Now why would you need an API to extract data from Google? Technically, you can fish out some insights into the way Google works and displays results without the need to use any specific SEO tools: just google your keyword and see what you get. But there are two problems with this approach: first, the process is pretty time-consuming to do manually and at scale - an inefficient monkey job, essentially. Second, the results you get can’t be considered objective. At the beginning of the 2000s, when the Google SERPs were first introduced, they looked much the same to each user for the localized Google version per each country. Now Google algorithms give out customized results tailored to each user, taking into account many factors, such as:
- Type of device: If a user is searching using their smartphone, the search results will look different, since starting from 2015 Google prefers showing web pages that are mobile-optimized.
- Registration: if a Google user is logged into their account, what they see on SERPs will be aligned with their history and user behavior, provided that's allowed within their data-related settings.
- Browser history: if a user rarely empties their browser cache, Google will include that information concerning previous search queries with cookies, and adjust the results.
- Location: if the geolocalization option is activated, Google aligns the SERPs with the user's location. That's why search results for the sushi takeaway query in Prague will be different from those in Los Angeles. If we're talking about local search, the results will be a combination of data from Google Search and Google Maps.
The solution to both manual work and this lack of objectivity is an automated crawler that is simple enough to use, but also complex enough to scrape such a massive website as Google. In other words, a SERP API - that’s a lot of Caps, but essentially it’s a program that will automatically collect data from Google SERP for you to analyze and use. This is exactly what our Google Search Result Scraper is created for. Our SERP API supports the extraction of all data on:
- organic and paid results
- People Also Ask
Why scrape Google?
Google is the main entry point to the internet for billions of people. This makes appearing in Google Search results a key factor for almost every business. And Google reviews and ratings have a massive impact on local businesses’ online profiles. Marketing agencies, especially those with a large number of clients from various industries, rely heavily on obtaining reliable SEO tools. They are not only a means of effectively performing various tasks, but also a means of successful management and analysis of results. You can look for things like how the top-ranking pages are writing their page titles, the keywords they're targeting, how they format their content, or take it a stage further and do some deeper link analysis.
Typical use cases for Google Search scraping are, among thousands of others:
- Search engine optimization (SEO) — monitor how your website performs in Google for certain queries over a period of time
- Analyze ads for a given set of keywords
- Monitor your competition in both organic and paid results
- Build a URL list for certain keywords. This is useful if you, for example, need good relevant starting points when scraping web pages containing specific phrases
And if you’re out of ideas of what to do with all that extracted data, visit our Industry pages for inspiration, with clear examples of how to use the results of web scraping for business and research.
What about the official Google Search API?
That’s a funny question. Google doesn’t provide its own SERP API for web search - so Google doesn’t make it that easy to extract data from Google at scale. Moreover, only a limited subset of information available on any search results page can be provided to you via Google services such as Google Ads or Google Analytics. The two official methods suggested by Google for getting data are Google Custom Search API (deprecated in April 2018) or scraping by URLFetch method.
Now that we’ve covered all the aspects and reasons for scraping Google, let’s get started with the tutorial itself. Promise it won’t take long :)
Step-by-step guide to scraping Google SERPs
1. Go to Apify’s website: https://apify.com
2. Sign in at the top-right corner using your email account, Google, or GitHub.
3. When you log in, you’ll be redirected to your Apify Console. Find the Google Search Scraper card and click on it.
4. Now you're on the page for the Google Search Results Scraper. Scroll down to get familiar with its parameters and possibilities.
5. Come back to the top of the page and click the blue Create new task button. It will redirect you to new input parameters of your scraper. Note that your scraper can be found in the Actors tab on the left, because that's what we call our customized scrapers. You can read more about those terms here.
6. You'll find yourself on a new page with plenty of options for your first scraping session. Once you've figured out all the input parameters such as Country, Language, Results per page, etc. just click the green Run button.
7. Our actor will start running its task and will change its status to Running.
8. When the scraper finishes its run, you will see its status change to Succeeded as well as how many results your scraping has brought you.
9. To preview your results, head over to the Dataset tab. This tab contains your data in lots of versatile formats, including HTML table, JSON, CSV, Excel, XML, and RSS feed. You can see them by clicking on View in another tab or Preview. Let's preview our results in JSON format.
10. To download your dataset results, pick the format and click Download. You can then share that data or upload it anywhere you like. Use it in spreadsheets, other programs or apps, or your own projects. Congrats, you've just completed your first scraping session :)
Google SERP proxies
Apify has proxies designed specifically for SERPs. Our proxies will make your scraping much faster and you’ll be able to dynamically switch between countries, so that you can get search information from any location. If you sign up for a free Apify account, you get a 30-day free trial of our SERP proxy service.
Now that you’re all ready, go ahead and start your first month with Apify by using our free Google Search Scraper on the Apify Store. Don't forget to send us a tweet if you do something interesting with all that data :)
If you need to scrape other parts of the Google giant, we have two other amazing scrapers: our Google Trends Scraper and Google Trending Searches Scraper are ready to help you keep track of emerging trends and ideas. Scraping Google Trends can be useful both for research, business, and personal interest and entertainment. If you need some inspiration on how to use these and our other SEO tools, check out 5 powerful scrapers to add to your SEO tool kit.