Search Kickstarter quicker, easier and with machine-readable results
Have you ever needed to check which Kickstarter projects launched last week, or which projects from your city have successfully raised enough money? It’s possible to look all of this up on the Kickstarter website, but Kickstarter does not have a public API, so consuming the data is pretty tricky.
That’s great if you just want to scroll through the projects, but if you want to work with the data on the page, then it quickly becomes quite a pain. For example, for backed projects you cannot see any useful details, the descriptions are cut off and you need to scroll to the end of the page to load more projects. What if you want to sum up how much money projects in your city raised? Or show the five newest projects from a certain category on your website? That’s very hard to do. But what if you could see when each project was funded, how much money is needed until it reaches its goal, and much more…
To solve this problem, we’ve created a new actor (check it out at jaroslavhejlek/kickstarter-search) that any user of the Apify platform can use for free. It has all the same parameters as the search function on the Kickstarter website, but instead of nice-looking boxes with a picture, styled text, and hidden details, it will give you useful data.
Each project is represented as a single JSON object:
And all projects are in a single JSON array, so you can carry out any data transformations you like with it.
You can see the example run of this exact search on the actor’s library page. And this link points to the results of the same query for country music from Nashville as shown in the screenshot above, only now you can see all the projects at once and you if do a little bit of coding you can work directly with the data.
How to use the actor
The actor is public, so any user of the Apify platform (if you don’t use Apify yet, sign up for a free account first!) can get started with it right away. Just go to the Apify library and click on the “Try actor” button. You should see a page with a form that looks like this:
Here you have all the available options for searching for Kickstarter projects. Change them to whatever you wish and press “Run” to start the actor. Once the actor finishes, go to its dataset tab to see the results.
Creating a Kickstarter API
There are many reasons why a developer might need an API, such as if you want to get machine-readable data and carry out calculations from it or if an application needs a way to access data from a different source.
The new Kickstarter search actor should help you with both problems. A complete tutorial on how to use actors is beyond the scope of this blog post (check out our docs for more info on using actors), but I will at least try to give you an example of how to solve a typical use case:
Instructions — API which returns the 10 most-backed projects in the United States:
- Go to the actor console and configure the input parameters to anything you want; in our example, these would be:
- Category: All
- Location: United States
- Sort: most_backed
- Maximum results: 10
2. Click on the “Create task” button above the form. This will bring you to the task configuration page. Here, fill in the name of the dataset that you wish to use. In our example, it will be “ks-most-backed-usa”.
3. Switch to the settings tab and change the name. Let’s call it “ks-most-backed-usa”.
4. Now you can run it to test that everything works.
5. Once the run finishes, you can go to storage and find the dataset with the name
ks-most-backed-usa. It should contain all 10 projects.
6. So all you need to do in any application that needs data from this project is to send an API request to the following URL:
https://api.apify.com/v2/datasets/<YOUR USERNAME>~<DATASET NAME>/items?token=<YOUR API TOKEN>
7. Now any application can have access to the 10 most-backed projects on Kickstarter. If you need to refresh the data, just go to the schedules section on the platform and set up a schedule for the task.
Kickstarter will only ever show you 200 pages of results on their website, so at most, you will get 2400 results for any query. So even though you might see that there are 6000 projects for your search, you will never actually see the remaining 3600. This limit is hard to overcome and can only be worked around by using more specific filters.
You can look up any location you want by its name, but to get the ID of the location on Kickstarter, the actor needs access to Apify Proxy. If you do not have access to this product, you need to provide the Kickstarter ID of the location directly instead of its name.
To get the ID of the location, go to Kickstarter, use the search page and switch location to wherever you need. You should see the ID of the location as part of the address of the website:
Next time we’ll take a look at an actor that gets content from the detail page of a project, enabling you to track any changes in the project.
Congratulations, you’ve just APIfied Kickstarter! Happy crawling with the new Kickstarter API!