Have you ever needed to check which Kickstarter projects launched last week in the US, or which projects from your area have successfully raised enough money? It’s easy to look all of this up on the Kickstarter website - just find the search bar, filters, and off you go. But what if you need to aggregate all that data and get it neatly onto your computer for further data analysis and visualization? Well, in that case, we have a free scraping solution for you. But first, let’s look at why getting the project data manually is not a viable option.
What about the Kickstarter API?
There are many reasons why a developer might need raw data: for example, if they need it in a machine-readable format to carry out calculations. There are also many reasons why an analyst might need it: for example, if an application needs a way to access data from a different source.
The Kickstarter website still doesn’t offer a public API, so performing Kickstarter data analysis without it can be a pretty tricky task. But our Kickstarter Scraper should solve all of these problems and help you organize your own little Kickstarter scraping lab.
Why scrape Kickstarter
Kickstarter website is a place where crowdfunding projects make their way from an idea to its realization, enriching both the communal and individual lives. Here’s just a fraction of the ways you can use bulk data extracted from Kickstarter:
- Follow existing and future Kickstarter projects in your area.
- Keep track of past successful projects in your area.
- Track projects participating in the same category anywhere in the world.
- Track your own, favorite, or saved projects and get notifications into your inbox whenever their receive funding.
- Support your own crowdfunding campaigns with real-time data.
- Back up your Kickstarter investment analysis with up-to-date data.
- Compare the success rate of crowdfunding campaigns in different cities
- Discover projects most likely to receive public support and evaluate their chances of success.
There’s so many more of course - you can easily incorporate that data into NGO projects, for instance. If you think of something we haven’t mentioned here, tell us on Twitter and we’ll be happy to hear about your use case. But read on for now and we’ll show you how to scrape Kickstarter in 10 simple steps.
The shortest guide to the Kickstarter website structure
The Kickstarter website isn’t badly constructed or anything - it actually has various filters available: from locations and project categories to percentage of funds aimed for and raised. For example, this is what you see when you try to look up all STEM projects near you, with no extra filters applied.
That looks great if your goal is to simply browse through the sea of projects. However, you’ll quickly notice that you cannot preview any of the project details, the descriptions are cut off in the middle, plus you need to scroll to the end of the page to load more projects. So what if you want to calculate how much money for projects has been raised in your city? Or summarize and show the 5 newest projects from the Community Gardens category on your website? With those filters, it will be either very hard or time-consuming to do this manually. So if you want to actually extract that data and work with that crowdfunding dataset - then this way of filtering doesn’t really suit your needs.
Kickstarter Scraper is an efficient little program that any user of the Apify platform can use for free and search Kickstarter for quicker, easier, and machine-readable results. It has all the same parameters as the search function on the Kickstarter website, but instead of nice-looking boxes with a picture, styled text, and hidden details, it will provide you with unadorned, useful data. Anything from the most recent projects of any category, any location to the amount of money aimed for, pledged, and collected plus sorted by status, recency, or popularity.
The scraped Apify datasets can be exported to many different formats so you can carry out any data transformations you like with it. You can keep them on your computer or simply upload them into the data visualization tool of your choice.
Step-by-step guide to scraping Kickstarter
1. First, head over to the apify.com website. Click on the very first tab on the top called Solutions and choose the Apify Store button. Apify Store is our library full of scraping and automation tools: they are easy to customize, ready to use, and free to try out.
2. When you're in Apify Store, type Kickstarter into the Search bar. As you type, the scraper will show up in the drop-down list, so just click on it. You will be redirected from Apify Store to the scraper page.
3. Once you’re on the Kickstarter Scraper page (which will look like this), just click the green Try for free button. You'll be redirected from our website to your Apify Console.
4. Now log in or sign up for free using your email, gmail or GitHub, if you haven’t already.
5. Now that you’re in Apify Console, it’s time to configure your first scraping Task. Notice that a new Task will be created for your chosen Scraper automatically. No worries, it won’t start until you add some input.
6. Let’s say your Task is to get all successful video game projects ever submitted to Kickstarter in the US, with the amount pledged up to $10,000. You also want them ranked from the most funded to the least funded. In that case, this is what your input will look like:
6. Once you’re set with all of the parameters, click on the green Save & Run button.
7. Now your Task has changed its status to Running. This scraping task is actually so easy for our Scraper that it will be finished in under 2 minutes. Once the run finishes, you’ll see the status change to Succeeded.
8. Next on your list is to click on the Dataset tab to see the results. Download the Kickstarter dataset in any format: HTML table, JSON, CSV, Excel and XML. You can choose to upload it onto your computer or to be used in your own Kickstarter data science projects.
9. Finally, preview the data before downloading as HTML or JSON by clicking the Preview 👁 button.
Apify datasets can be exported to many different formats. You can keep them on your computer or simply upload them into the data visualization tool of your choice.
Kickstarter search will only ever show you the first 200 pages of results. This restriction in its turn puts some limitations on our scraper as well: for the time being, at most, you can get a maximum of 2,400 results for any query. In practice, this means that even though you might see 6,000 projects for your search on the website itself, the scraper will not get you all of them in one go. This limit is hard to overcome, so the remaining 3,600 items can only be scraped by using more specific filters.
This is how by doing a little bit of coding you can work directly with the data. These days, you can also learn how to scrape your Kickstarter projects with Python. Happy scraping!