Content
Apifier (later renamed Apify) is a hosted web crawler for developers that enables them to extract data from any website using a few simple lines of JavaScript.
You might wonder, why build another web scraper when there are already so many of them. Indeed, that’s exactly what we thought about a year ago when one of our customers asked us to setup a system to regularly and reliably extract large amounts of data from a variety of websites. After a lot of research we realized we basically had the following options:
- tools that prefer UI instead of coding
- tools that enable scripting of your own web browser or desktop apps
- tools that treat websites as a set of plain HTML documents
- consulting companies that “do it for you”
but what we actually needed was a tool that:
- can crawl websites with arbitrarily complex or irregular structure
- is a cloud service that could interface with apps
- can crawl dynamic websites (i.e. understand JavaScript)
- be simple to use and affordable enough even for small companies
After a lot of trial and error, it was clear than none of the available tools were good enough for the job so we decided to build our own. The goal was to make scraping a website as simple as possible without compromising the ability to scrape extremely complex websites. Therefore UI-based scrape definition was out of question. We are programmers and we already use JavaScript for client-side development, so why not use it for scraping?
Today, we’re bringing you what we built for ourselves. With Apifier, you can define your own crawler or web scraping tool in just a few minutes. There is no need to setup any servers, proxies, cron jobs, databases and it is easily programmable using simple JavaScript. We hope you’ll like it!
There are a ton of features that didn’t make it into this first release, but stay tuned, a lot more is coming soon.
Jan & Jakub
If you've landed on this page, you might like to know that Apifier became Apify, and many years later is now the only full-stack web scraping and automation platform on the market 💪🚀
