Have you ever needed to check which Kickstarter projects launched last week in the US, or which projects from your area have successfully raised enough money? It’s easy to look all of this up on the Kickstarter website - just find the search bar, filters, and off you go. But what if you need to aggregate all that data and get it neatly onto your computer for further data analysis and visualization? Well, in that case, we have a free web scraping solution for you. But first, let’s look at why getting the project data manually is not a viable option.
What about the Kickstarter API?
There are many reasons why a developer might need raw data: for example, if they need it in a machine-readable format to carry out calculations. There are also many reasons why an analyst might need it: for example, if an application needs a way to access data from a different source.
The Kickstarter website still doesn’t offer a public API, so performing Kickstarter data analysis without it can be a pretty tricky task. But our Kickstarter Scraper should solve all of these problems and help you organize your own little Kickstarter scraping lab.
Why scrape Kickstarter
Kickstarter website is a place where crowdfunding projects make their way from an idea to its realization, enriching both communal and individual lives. Here’s just a fraction of the ways you can use bulk data extracted from Kickstarter:
- Follow existing and future Kickstarter projects in your area.
- Keep track of past successful projects in your area.
- Track projects participating in the same category anywhere in the world.
- Track your own, favorite, or saved projects and get notifications into your inbox whenever their receive funding.
- Support your own crowdfunding campaigns with real-time data.
- Back up your Kickstarter investment analysis with up-to-date data.
- Compare the success rate of crowdfunding campaigns in different cities
- Discover projects most likely to receive public support and evaluate their chances of success.
There’s so many more of course - you can easily incorporate that data into NGO projects, for instance. If you think of something we haven’t mentioned here, tell us on Twitter and we’ll be happy to hear about your use case. But read on for now and we’ll show you how to scrape Kickstarter in 10 simple steps.
The shortest guide to the Kickstarter website structure
The Kickstarter website isn’t badly constructed or anything - it actually has various filters available: from locations and project categories to percentage of funds aimed for and raised. For example, this is what you see when you try to look up all STEM projects near you, with no extra filters applied.
That looks great if your goal is to simply browse through the sea of projects. However, you’ll quickly notice that you cannot preview any of the project details, the descriptions are cut off in the middle, plus you need to scroll to the end of the page to load more projects. So what if you want to calculate how much money for projects has been raised in your city? Or summarize and show the 5 newest projects from the Community Gardens category on your website? With those filters, it will be either very hard or time-consuming to do this manually. So if you want to actually extract that data and work with that crowdfunding dataset - then this way of filtering doesn’t really suit your needs.
Kickstarter Scraper is an efficient little program that any user of the Apify platform can use for free and search Kickstarter for quicker, easier, and machine-readable results. It has all the same parameters as the search function on the Kickstarter website, but instead of nice-looking boxes with a picture, styled text, and hidden details, it will provide you with unadorned, useful data. Anything from the most recent projects of any category, any location to the amount of money aimed for, pledged and collected plus sorted by status, recency, or popularity.
The scraped Apify datasets can be exported to many different formats so you can carry out any data transformations you like with it. You can keep them on your computer or simply upload them into the data visualization tool of your choice.
Step-by-step guide to scraping Kickstarter
Find your actor
1. First, go to the scraper's page, and click the Try for free button. You will be redirected to Apify Console, which is your workspace to run tasks for your scrapers. If you already have an Apify account and are logged in, go to Step 3.
Log in or sign up
2. If you are not signed in, you’ll find yourself on the sign-up page (if you are already signed in, skip to Step 3). Sign up using your email account, Google, or GitHub. You will be redirected to the scraper’s page on your Apify Console.
3. Now you can fill in the input fields, such as keywords to scrape for and the maximum number of results you would like to obtain.
4. Once you are all set, click the Start button. Notice that your task will change its status to Running, so wait for the scraper's run to finish. It will be just a minute before you see the status switch to Succeeded.
Results
5. Move tothe Dataset tab to see the results of your scraping. Explore the Dataset tab containing your scraped data in many formats, including HTML table, JSON, CSV, Excel, XML, and RSS feed. Preview the data by clicking the Preview button or viewing it in a new tab if the dataset is too large.
6. You can choose to download it onto your computer for further use as spreadsheets or in other apps and your projects.
Apify datasets can be exported to many different formats. You can keep them on your computer or simply upload them into the data visualization tool of your choice.
2400 caveat
Kickstarter search will only ever show you the first 200 pages of results. This restriction in its turn puts some limitations on our scraper as well: for the time being, at most, you can get a maximum of 2,400 results for any query. In practice, this means that even though you might see 6,000 projects for your search on the website itself, the scraper will not get you all of them in one go. This limit is hard to overcome, so the remaining 3,600 items can only be scraped by using more specific filters.
This is how by doing a little bit of coding you can work directly with the data. These days, you can also learn how to scrape your Kickstarter projects with Python. Happy scraping!
