2020 marks 25 years since the founding of Amazon as an Internet marketplace specializing in selling books to customers. What was then a radical idea, grew into an e-commerce giant that now dominates retail as we know it, both online and offline, with 9 out of 10 consumers reporting having used the site to price check a product. One of the most surprising developments in recent years has been in Amazon’s business model, allowing third-party retailers to sell items on the Amazon store with fulfilment handled entirely by Amazon, known as FBA. In fact, more than 50% of Amazon sales now come from third-party sellers on the site.
With the online retail sector booming in the era of COVID-19, retailers are looking closely at Amazon, particularly at sellers, and wondering how to get a slice of this ever-expanding pie. In the fast-paced world of e-commerce, retailers need to look closely at their competitors, particularly their best selling products, prices and their reviews. With all eyes on Amazon, keeping track of all of this information across the over 12 million products listed on the site can quickly become a full time job. This is why top e-commerce sellers are investing in technology to keep ahead of the competition, turning to web scraping to automate the collection of best selling Amazon products, prices, and reviews.
What is an Amazon web scraper?
An Amazon web scraper is a tool that can visit Amazon, search for products and extract all the relevant information, including the item price, delivery times, product photos, and the strength of the product’s reviews. Web scrapers act in the same way as a user visiting the website and extracting the information manually, except they can be programmed to do this completely automatically, visiting thousands of pages in a single day. These tools can visit Amazon at a scheduled time each day or week, tracking hundreds of products in order to give the user an edge in pricing and positioning their products. But building a web scraper like this for Amazon can be a challenge, requiring a development team to build, test, maintain, and run the scraper. That is why many in the industry are turning to web scraping platforms like Apify.
Why scrape Amazon?
Scraping Amazon for Best Sellers
The Amazon Best Sellers list features the top selling items across Amazon, which tells us a lot about the top trends in e-commerce. Competing directly against these products can be difficult, but the best seller list can be a source of inspiration for new products and help e-commerce retailers stay ahead of the competition. Getting your item into the best seller list and keeping it there is one of the surest ways to guarantee sales for your business. Once a product reaches the best seller list, e-commerce retailers increasingly turn to web scraping, to track up and coming products, and adjust their products to compete.
Scraping Amazon for pricing and reviews
Amazon ranks products using a range of metrics, the two most common being price and reviews. The more competitive the price compared to other similar items and the more favorable the reviews, the more traffic a listing will get and the more profitable it will become.
Tracking pricing and the quality of reviews on Amazon is therefore critical to competing against the hundreds of other listings on the website.
How does Apify make it easy to scrape Amazon?
Apify is a web scraping platform that allows e-commerce retailers to build and run web scrapers without having to hire a development team and spend money on expensive servers. Apify Store contains pre-built web scrapers (called actors on the Apify platform) that can be run without writing a single line of code, allowing anyone to get started right away, including a readymade, free Amazon web scraper.
Apify allows you to carry out data extraction on a schedule, handles the entire scraping process from start to finish, and then allows you to export the data into useful formats such as Excel or JSON, or integrate it directly into your technology stack using the API. Using Apify lets you avoid a lot of the expensive mistakes that e-commerce retailers often make when building their own Amazon web scrapers. Instead, you can get started right away.
Building your own Amazon web scraper using Apify
To set up your own web scraper to extract Amazon product data, visit Apify Store. After creating your Apify account, go to the Amazon Scraper and click “Try for free”. You’ll be directed to a pre-prepared Amazon Scraper task and you can begin configuring it for your products.
To begin, select your Amazon market from the dropdown: amazon.com for the US, amazon.co.uk for UK Amazon, or any other Amazon marketplace. Select your main market first and if you need to compare listings between marketplaces, you can run a separate web scrape to extract this later. Next select the Store Department, if you need to filter products by a particular Amazon department such as Books, Computers or Electronics. If you don’t need to filter by department, leave this as All Departments.
Next, select the type of search you want to run — keywords, Direct URLs or ASINs (Amazon Standard Identification Numbers). If you want to search across Amazon for products, select keywords, or, if you want to track known competitors’ products using their product URLs or ASINs, select either of these two options. Depending on the type of search you want to run, enter the search terms in the box below separated by commas for multiple terms. These will be a list of keywords, ASINs or URLs that you want Apify to search, visit, and extract data from.
In the second section, you have configuration options for the number of results, your proxy settings, number of reviews to extract, and the delivery location. If you want Apify to only deliver a certain number of products during the search, or you want to filter products based on their number or reviews or delivery location, set these options here. To complete the configuration, select the proxy you would like to use for the extraction. Apify Proxy lets you extract information from Amazon without being blocked, so for large scraping jobs, select from a proxy group or use the Apify proxies automatically assigned. New users get to try Apify Proxy for free for one month. Now that you have set up your scraper, hit Save & Run to begin the scrape.
As the web scraper runs, you can monitor its progress using your Apify Dashboard, viewing the status, number of results extracted, number of pages left to extract in the request queue as well as the duration. If there are any errors or warnings as the scraper completes the scraping job, this will show in the log below.
Once complete, you can view your results in the dataset tab, downloading the results to popular export formats, such as Excel, CSV, JSON or provide a HTML table or an RSS feed to extract the data from. In the output file you will find the following, if available;
- Seller Details
- Delivery Details
- Stock Levels
- Links to Product Pictures
Start scraping Amazon with Apify
There has never been a better time to build an Amazon web scraper. Sign up with Apify for free and use the pre-built Amazon web scraper from Apify Store. Get a competitive edge on Amazon and you’ll never look back.
Bonus: here’s a short video of me explaining how I use Apify to scrape Amazon!