Unofficial Google Search API from Apify

Marek Trunkát
Marek Trunkát

We have just released a new version of Apify actor called Google Search Results Scraper (apify/google-search-scraper) to extract data from Google Search result pages. You can try it for free.

Google is the main entry point to the internet for hundreds of millions of people. This makes appearing in Google Search results a key factor for almost every business.

https://www.google.com/

Google doesn’t provide an API for web search. Moreover, only a limited subset of information available on any search results page is provided to you via Google services such as Google Ads or Google Analytics. To get the full picture, you need to scrape the data from Google’s web pages. Typical use cases for Google Search crawling are, among thousands of others:

  • Search engine optimization (SEO) — monitor how your website performs in Google for certain queries over a period of time.
  • Analyze ads for a given set of keywords.
  • Monitor your competition in both organic and paid results.
  • Build a URL list for certain keywords. This is useful if you, for example, need good relevant starting points when scraping web pages containing specific phrases.

To address these use cases, we have published a new open-source actor called Google Search Results Scraper (apify/google-search-scraper) in Apify Store to extract data from Google Search. The new actor uses Apify Proxy’s Google SERP feature, which automatically routes requests through a proxy server from the selected country and returns the raw HTML data of the Google Search result pages.

Data

As shown in the following screenshot, Google Search results consist of various parts:

Google search results

You can see ads, featured snippets, snap packs, and organic results. Additional types may also appear: product ads, related queries, and multiple snap pack types (Wikipedia, map, videos, …). Apify’s new Google Search scraping actor currently supports:

  • Organic results
  • Ads
  • Product ads
  • Related queries

If you need additional attributes, you can also include a short JavaScript code to extract additional attributes from the HTML. Need something else? Just submit an issue on GitHub.

How to use the actor

  1. Open Google Search scraper in the Apify Store: https://apify.com/apify/google-search-scraper

2. Click the Try for free button

Public actor in the Apify store

3. Log in to your Apify account, or create a new one (if you’ve just signed up, you’ll need to verify your email address).

4. Automatically the task will be created and you can fill in the input schema for the scrape you want to proceed with.

Actor input schema

5. You can update search queries and select the country and language for the results. Then click the Run button to start scraping. For more information on configuration options, see the actor input documentation in Apify Store.

6. After you click the Run button, crawling will start, and in about a minute you will see results popping up.

7. You’ll then see the following console for running the actor:

Console log Apify platform

8. The results are located under the Dataset tab:

Output in datasets

You can download the output data in different formats like HTML, JSON, CSV, XLS, RSS.

Or just preview if you scraped the right results.

JSON data preview
HTML data preview

If you open the results in JSON format, you’ll get the following dataset:

[{
  "searchQuery": {
    "term": "Hotels in Prague",
    "page": 0,
    "type": "SEARCH",
    "countryCode": "cz",
    "languageCode": "en",
    "locationUule": null,
    "resultsPerPage": "10"
  },
  "url": "http://www.google.com/search?gl=cz&hl=en&num=10&q=Hotels%20in%20Prague",
  "hasNextPage": false,
  "resultsTotal": 138000000078,
  "relatedQueries": [
    {
      "title": "cheap hotels in prague",
      "url": "https://www.google.com/search?hl=en&gl=CZ&q=cheap+hotels+in+prague&sa=X&sqi=2&ved=2ahUKEwjem6jG9cTgAhVoxlQKHeE4BuwQ1QIoAHoECAoQAQ"
    },
    {
      "title": "best hotels in prague old town",
      "url": "https://www.google.com/search?hl=en&gl=CZ&q=best+hotels+in+prague+old+town&sa=X&sqi=2&ved=2ahUKEwjem6jG9cTgAhVoxlQKHeE4BuwQ1QIoAXoECAoQAg"
    },
    ...
  ],
  "paidResults": [
    {
      "title": "2280 Hotels in Prague | Best Price Guarantee | booking.com‎Book apartments and moreMore than just hotels",
      "url": "https://www.booking.com/go.html?slc=h3;aid=303948;label=",
      "displayedUrl": "www.booking.com/",
      "description": "Book your Hotel in Prague online. No reservation costs. Great rates. Bed and Breakfasts. Support in 42 Languages. Hotels. Motels. Read Real Guest Reviews. 24/7 Customer Service. 34+ Million Real Reviews. Secure Booking. Apartments. Save 10% with Genius. Types: Hotels, Apartments, Villas.£0 - £45 Hotels - up to £45.00/day - Book Now · More£45 - £90 Hotels - up to £90.00/dayBook Now£130 - £180 Hotels - up to £180.00/dayBook Now£90 - £130 Hotels - up to £130.00/dayBook Nowup to £45.00/dayup to £90.00/dayup to £180.00/dayup to £130.00/day",
      "siteLinks": [
        {
          "title": "Book apartments and more",
          "url": "https://www.booking.com/go.html?slc=h3;aid=303948;label=",
          "description": "Bookings instantly confirmed!Instant confirmation, 24/7 support"
        },
        {
          "title": "More than just hotels",
          "url": "https://www.booking.com/go.html?slc=h2;aid=303948;label=",
          "description": "Search, book, stay – get started!Hotels when and where you need them"
        }
      ]
    },
    {
      "title": "Hotels In Prague | Hotels.com™ Official Site‎",
      "displayedUrl": "www.hotels.com/Prague/Hotel",
      "description": "Hotels In Prague Book Now! Collect 10 Nights and Get 1 Free. Budget Hotels. Guest Reviews. Last Minute Hotel Deals. Luxury Hotels. Exclusive Deals. Price Guarantee. Photos & Reviews. Travel Guides. Earn Free Hotel Nights. No Cancellation Fees. Types: Hotel, Apartment, Hostel.",
      "siteLinks": []
    },
    ...
  ],
  "paidProducts": [],
  "organicResults": [
    {
      "title": "30 Best Prague Hotels, Czech Republic (From $11) - Booking.com",
      "url": "https://www.booking.com/city/cz/prague.html",
      "displayedUrl": "https://www.booking.com › Czech Republic",
      "description": "Great savings on hotels in Prague, Czech Republic online. Good availability and great rates. Read hotel reviews and choose the best hotel deal for your stay.",
      "siteLinks": []
    },
    {
      "title": "The 30 best hotels & places to stay in Prague, Czech Republic ...",
      "url": "https://www.booking.com/city/cz/prague.en-gb.html",
      "displayedUrl": "https://www.booking.com › Czech Republic",
      "description": "Great savings on hotels in Prague, Czech Republic online. Good availability and great rates. Read hotel reviews and choose the best hotel deal for your stay.",
      "siteLinks": []
    },
    ...
  ],
  "customData": {
    "pageTitle": "Hotels in Prague - Google Search"
  }
},
...

Integration

Apify is an open platform that allows actors to work together and also be integrated with external services. You can, for example, use the lukaskrivka/google-sheets-export actor to upload your crawled data to Google Sheets.

Another option is to schedule the Google Search scraper actor to run daily and use webhooks to notify your system that data has been scraped and is ready to be downloaded.

You can also call this actor from your system via Apify actor API.

Have fun crawling Google!



Great! Next, complete checkout for full access to Apify
Welcome back! You've successfully signed in
You've successfully subscribed to Apify
Success! Your account is fully activated, you now have access to all content
Success! Your billing info has been updated
Your billing was not updated