Visibility on Google translates directly into market share, as every click, ranking shift, and featured snippet is an opportunity. SERP data is the foundation of competitive intelligence and strategic decision-making, but scraping Google isn’t straightforward.
Google offers no public API for search results, and direct scraping attempts quickly run into limits - CAPTCHAs, IP blocks, or shifting HTML structures. As a result, most companies turn to third-party SERP APIs or specialized web scraping solutions to obtain the data they need. In this tutorial, we’ll explore how to scrape Google search results with a ready-made solution from Apify.
Why is Google SERP hard to scrape?
Google search result pages weren’t designed to be machine-readable. Unlike structured pages (e.g. Wikipedia), Google’s SERP is a constantly moving target - Google tweaks its layout constantly. In the end, the result pages are far more complex than a list of <a> tags, containing usually:
- Standard web results
- Sponsored web results
- Knowledge panels
- Featured snippets
- People Also Ask boxes
- Images, videos, shopping results, maps, etc.
Search result page with featured snippets and panels
Each of these elements has its own custom HTML structure, and some load dynamically when clicked. Parsing them requires lots of conditional logic, and the results differ depending on geolocation, language settings, or browser history.
A solution for SERP data collection: Google Search Results Scraper
Apify’s scraper for Google SERP is a straightforward tool that you can run immediately, without code: It will get your search request, conduct the search on google.com, and extract raw data from the result pages. It covers key SERP elements such as organic + paid results, People Also Ask boxes, related queries, prices, and reviews.
With Apify’s free plan, you get $5 in usage credits every month, allowing you to scrape over 1,000 pages for free.
Step-by-step guide to scraping Google SERPs
Step 1. Go to Google Search Results Scraper
Click the Try for free button to sign up for a free Apify account and start using Google Search Results Scraper.
If you don’t have an Apify account yet, it’s easy to sign up with your GitHub or email account from any provider. You’ll enter Apify Console, a workspace to run or build web scraping tools.
Step 2. Insert the keyword you want to scrape
Now fill in the input fields. You can provide keywords or Google Search URLs - either will work. We’ll use art cinema as our search term.
Step 3. Customize your search
Google Search Results Scraper allows you to specify language preferences, result geolocation, and other parameters. It’s a good idea to set how many Google pages you want to scrape and how many results you want to see on each page. If you’re interested in the first two pages and 50 results on each, then you'll scrape 100 pages in total:
You can also specify the country and the language of the results. In our example, we’ll use Canada and English.
For paying users, there is also an option to include the Business leads enrichment feature. With this add-on, the scraper uncovers contact details from the companies’ websites. This can include emails, job titles, etc. You can even filter the information by department, such as C-Suite, HR, or Marketing:
Google Search Results Scraper offers another interesting feature for your results: Google AI Mode. This add-on is great for Generative Engine Optimization (GEO) or Answer Engine Optimization (AEO), as well as competitive analysis. Once you run your scraper, it will generate structured AI-generated responses either on its own or along your organic search results, with a description, related links, and key information.
Here’s what you get after a successful run:
Step 4. Run the scraper by clicking Start
Once you're ready, click the Start button to begin the data extraction. The status will change to Running, and you’ll need to wait until the process is complete. Once the status changes to Succeeded, you can export your dataset.
Step 5. Download your data
Once you're ready to download Google data, use the Export button to see the results of your scraping. Now you can download your scraped data in many formats, including HTML table, JSON, CSV, or Excel.
You can choose to download all results or, e.g., only organic or paid ones. You can also narrow down your dataset to a few specific fields that you want to keep or discard.
That’s it, you’re done. Now that you know how it works, you can give it a go yourself - simply choose your search terms, and run the scraper with a single click.
Using data extracted from Google
Google’s Search Engine Results Pages reflect what real customers search for and how Google interprets the market. It’s a great source for competition analysis or identifying gaps in your marketing communication strategy. Typical use cases include:
- Keyword discovery: Find what search queries people use to refine SEO and paid ads.
- Competitor tracking: See which competitors rank, what titles/descriptions they use, and how they position products.
- Content gap analysis: Identify topics where competitors rank but your business doesn’t.
- Featured snippets & People Also Ask: Learn what questions people ask, then create content that answers them.
- SERP algorithm tracking: See how rankings shift over time to spot rising competitors.
- Content trend spotting: Scrape news and trending stories from Google News SERPs to anticipate industry shifts.
FAQ
Can you scrape Google Search results?
Google search results fall into the category of publicly available data, so scraping Google search results is legal. But there is still some data you should not be accumulating, such as personal information or copyrighted content. Learn more about regulations and laws connected to scraping in Is web scraping legal?.
Is there an official Google Search API?
Currently, no. At first glance, Google has an "army" of public APIs for its various services, typically structured around JSON and RESTful requests. These APIs are openly accessible via Google's API GitHub repository. But you won't find a SERP API among them. Custom Search API is the closest to an official solution, but it serves a different purpose. It won't crawl Google and return a comprehensive list of SERP results in JSON format, as you might expect. Instead, it can help you to create a mini search engine for your site by applying the logic of Google SERPs.
Moreover, this alternative, often confused with Google SERP API, introduces two significant restrictions: a limit on search queries and the scope of searchable websites. It allows a limited number of searches daily (10K), with a fee of $5 per 1K queries for broader searches through the official Google API (see SERP API pricing page).
Can I use API with this Google Search Scraper?
Yes. Google Search Scraper can be used as an API of its own. Once you scrape the Google SERP data, you can transfer it via an API directly to your projects and apps. You can use direct API Endpoints, Node.js, or Python Client.
Can I use AI to scrape Google?
AI is currently unable to scrape websites directly, but it can help generate code for scraping Google if you prompt it with the target elements you want to scrape. Note that the code may not be functional, and website structure and design changes may impact the targeted elements and attributes.
Do I need proxies for scraping Google SERPs?
Yes. Apify has proxies designed specifically for SERPs. Our proxies will make your scraping much faster, and you'll be able to dynamically switch between countries so that you can get search information from any location. If you sign up for a free Apify account, you get a 30-day free trial of our SERP proxy service.
Need more tools for scraping Google services?
Apify Store offers a wide choice of Google scrapers, designed specifically to extract images, contact data, trending searches, and more. They are all free to try out:
