Does Google even need an introduction? If our phones have become extensions of our hands, Google is one of the main reasons for that evolution. Google is a synonym for answers, speed, and accessibility - but also for billions of dollars, users, clicks, searches, and billions of terabytes of data. That data can be extracted automatically and effortlessly if you know the proper methods and have the right tools at hand.
In this brief how-to article, we’re going to show you exactly how to scrape the most extensive library in the world by using a ready-made tool on the Apify platform called Google Search Results Scraper. This is your step-by-step guide to how to scrape any information available from Google, including organic and paid results, ads, queries, People Also Ask boxes, prices, and reviews. Let’s get started!
Is it legal to scrape Google?
Google search results fall into the category of publicly available data, scraping which is legal. But there is still some data you should not be accumulating, such as personal information or copyrighted content. Learn more about regulations and laws connected to scraping at our legality article.
What are Google SERPs?
A Google SERP is a page containing the list of search results that Google displays to you when you type in your query and hit Enter. SERP, in this case, stands for Search Engine Results Page, and you’ll find SERPs not only on Google, which controls 90% of the search engine market, but also on other search engines, such as Bing, Yahoo, Yandex, and others. We need to know this term to understand how to use web scraping on the Google Search Engine. You can consider the terms Google page, Google search page, and Google SERP to be equal and interchangeable, but we’ll stick with Google SERP for the sake of being technically correct.
Google SERPs have changed a lot over the years, with the most prominent features being those infoboxes we all know too well - Knowledge Graphs and Carousel - so ubiquitous these days that we can’t imagine the Google SERP interface looking any other way. Those now-classic Google SERP features were part of the Hummingbird algorithm release in 2013.
It’s a far cry from the 2003 version of Google results. Does this prehistoric SERP interface ring a bell? Luckily, we’re not there anymore.
Google SERP structure - how to scrape Google
To scrape Google, we first need to understand how it sees and prioritizes our searches. When you search for things on Google, what you see is not just an index of pages with URLs or so-called organic searches. While it used to be like that in the past, as we’ve seen, the primary purpose and driving force of Google - or any search engine for that matter - has always been to have your queries answered as quickly and efficiently as possible, and in a way that will attract your attention and be easy on the eyes.
That’s why over time, the search results have become much more multilayered, including the results of varying complexity and formats, like a giant layer cake. And that cake-like structure isn’t going away any time soon, with voice command search, apps, and mobile search introducing their significant corrections into the way we google stuff. Today, Google Search results consist of various levels, depending on the complexity and type of search, as you can see in this example of a string theory query:
As you can see, the Google search page is now packed with various content: featured snippets, so-called snap packs, ads, and organic results. Additional types may also show up: product ads, related searches, and multiple snap pack types (Wikipedia, Maps, videos, etc.).
Google SERP API
Now, why would you need an API to extract data from Google? Technically, you can fish out some insights into the way Google works and displays results without the need to use any specific SEO tools: just google your keyword and see what you get. But there are two problems with this approach: first, the process is pretty time-consuming to do manually and at scale - an inefficient monkey job, essentially. Second, the results you get can’t be considered objective. At the beginning of the 2000s, when the Google SERPs were first introduced, they looked much the same to each user for the localized Google version per each country. Now Google algorithms give out customized results tailored to each user, taking into account many factors, such as:
- Type of device: If a user searches using their smartphone, the search results will look different, since starting from 2015, Google prefers showing mobile-optimized web pages.
- Registration: if a Google user is logged into their account, what they see on SERPs will be aligned with their history and user behavior, provided that's allowed within their data-related settings.
- Browser history: if a user rarely empties their browser cache, Google will include that information concerning previous search queries with cookies and adjust the results.
- Location: if the geolocalization option is activated, Google aligns the SERPs with the user's location. That's why search results for the sushi takeaway query in Prague will be different from those in Los Angeles. If we're talking about local search, the results will be a combination of data from Google Search and Google Maps.
The solution to both manual work and this lack of objectivity is an automated crawler that is simple enough to use and complex enough to scrape such a massive website as Google. In other words, a SERP API - that’s a lot of Caps, but essentially it’s a program that will automatically collect data from Google SERP for you to analyze and use. This is precisely what our Google Search Result Scraper is created for. Our SERP API supports the extraction of all data on:
- organic and paid results
- People Also Ask
Why scrape Google?
Google is the main entry point to the internet for billions of people. This makes appearing in Google Search results a key factor for almost every business. And Google reviews and ratings have a massive impact on local businesses’ online profiles. Marketing agencies, especially those with a large number of clients from various industries, rely heavily on obtaining reliable SEO tools. They are not only a means of effectively performing multiple tasks but also a means of successful management and analysis of results. You can look for things like how the top-ranking pages are writing their page titles, the keywords they're targeting, how they format their content, or take it a stage further and do some deeper link analysis.
Typical use cases for Google Search scraping are, among thousands of others:
- Search engine optimization (SEO) — monitor how your website performs in Google for specific queries over a period of time
- Analyze ads for a given set of keywords
- Monitor your competition in both organic and paid results
- Build a URL list for specific keywords. This is useful if you, for example, need good relevant starting points when scraping web pages containing specific phrases
And if you’re out of ideas of what to do with all that extracted data, visit our Industry pages for inspiration, with clear examples of how to use web scraping results for business and research.
What about the official Google Search API?
That’s a funny question. Google doesn’t provide its own SERP API for web search - so Google doesn’t make it that easy to extract data from Google at scale. Moreover, only a limited subset of information available on any search results page can be provided to you via Google services such as Google Ads or Google Analytics. The two official methods suggested by Google for getting data are Google Custom Search API (deprecated in April 2018) or scraping by URLFetch method.
Now that we’ve covered all the aspects and reasons for scraping Google, let’s get started with the tutorial. Promise it won’t take long :), but if you would prefer a short video tutorial, we have one for you right here:
1. Go to the actor's page, and click the Try for free button. You will be redirected to Apify Console, which is your workspace to run tasks for your scrapers. If you already have an Apify account and are logged in, go to Step 3.
2. If you are not signed in, you’ll find yourself on the sign-up page (if you are already signed in, skip to Step 3). Sign up using your email account, Google, or GitHub. You will be redirected to the scraper’s page on your Apify Console.
3. It's now time to fill in the input fields. You can provide keywords or URLs to scrape, and limit the number of results per page. Take your time exploring your options and shaping your search.
4. Once you are all set, click the Start button. Notice that your task will change its status to Running, so wait for the scraper's run to finish. It will be just a minute before you see the status switch to Succeeded.
5. Once the actor has finished running, move to the Dataset tab to see the results of your scraping. Explore the Dataset tab containing your scraped data in many formats, including HTML table, JSON, CSV, Excel, XML, and RSS feed.
6. Preview the data by clicking the Preview button or viewing it in a new tab if the dataset is too large. You can choose to download it onto your computer for further use as spreadsheets or in other apps and your projects.
Google SERP proxies
Apify has proxies designed specifically for SERPs. Our proxies will make your scraping much faster, and you’ll be able to dynamically switch between countries so that you can get search information from any location. If you sign up for a free Apify account, you get a 30-day free trial of our SERP proxy service.
Now that you’re all ready, go ahead and start your first month with Apify by using our free Google Search Scraper on Apify Store. Don't forget to send us a tweet if you do something interesting with all that data :)
If you want an even easier Google scraper, we have a great alternative for you: Easy Google Scraper.
If you need to scrape other parts of the Google giant, we have other impressive scrapers: our Google Maps Scraper is great for automating the extraction of contact data such as phone numbers, emails, addresses, and website names of companies in the general vicinity. Google Trends Scraper and Google Trending Searches Scraper are ready to help you keep track of emerging trends and ideas. Scraping Google Trends can be useful for research, business, personal interest, and entertainment. If you need some inspiration on how to use these and our other SEO tools, check out 5 powerful scrapers to add to your SEO tool kit.