Software is eating the world and there are now software services, websites, and APIs for almost anything imaginable. The web has become the largest source of information in the history of humankind and most people increasingly spend more of their productive time in front of computers. By automating manual work on computers, taking advantage of the vast amounts of data on the web and combining existing services and APIs in novel ways, people can greatly increase their productivity and build new tools on top of existing ones. In this way, companies can improve their decisions, products, sales, and marketing and thrive in the digital economy of the 21st century.
But there’s a catch. Even today, most of the web is only available in a form accessible by humans, but not by automated systems. Most websites don’t have an API and even those that have one might not provide their complete data through it. To make automation possible and strive towards the goal of the open web as a public good and a basic right for everyone, people need tools to extract structured data from the web and automate workflows on it.
In principle, there are three ways to extract data or automate operations on the web: artificial intelligence, point-and-click tools, and custom code. Artificial intelligence and point-and-click tools can get you far, but have their limits and cannot handle complex websites or scenarios. Custom code, on the other hand, gives you full control and unlimited flexibility. In fact, this is true not only for web automation, but for every aspect of human interaction with a computer. A man with code and a machine is always more powerful than the machine itself (and must remain so in the future!)
Unfortunately, most people that could benefit from automation are either not able or not willing to write code. In order to let them fully benefit from automation, they must be able to purchase custom code and tools for their automation jobs. The principles of the access economy have revolutionized how low-skill services are delivered. Purchasing accommodation or transportation from random strangers on the other side of the planet has become so common that we rarely think about it. This trend is already bubbling up to higher-skill services and in the future, the access economy model will also revolutionize the development of custom software.
To make web automation available to everyone, we need two things.
First, we need a cloud computing platform where developers can easily develop, run, showcase, deliver and sell novel software tools or services built on top of existing websites, APIs or other services. The platform needs to be flexible enough so that developers are not limited in what they can build with it, while it needs to be simple enough so that even non-developers can use the tools running on top of it. The platform must enable developers to publish their software in such a way that people can immediately understand what it is and use it without any installation or setup. While existing cloud computing platforms are certainly flexible, they lack the other required qualities.
Second, we need to have a functioning marketplace where people can buy custom software developed by others. The entry barrier needs to be low enough so that even novice developers can quickly start delivering value for customers. On the other hand, high-cost customers looking for quality need to be served by master developers, who are not forced to underbid their prices and compromise on quality. The marketplace must encourage developers to learn and improve their skills and give customers sufficient guarantees about the quality of the software they are buying.
To make this work, both the cloud computing platform and the marketplace need to be closely interconnected, in order to give both developers and customers a full end-to-end experience. Traditional marketplaces for purchasing freelance work come without the computing platform and are not designed for the specifics of software development. They are a lot like renting accommodation on Craigslist — there are many options for how the goods will be delivered, which creates a lot of uncertainty. The same way that Airbnb gave structure to the short-term rental of accommodation and made it a commodity, having a unified marketplace and computing platform will bring structure to the way custom cloud software can be purchased and delivered.
This is where Apify comes in.
Apify has already established itself as one of the world’s most innovative companies in web automation and data extraction. Our platform makes it possible to develop, build and run long-running software jobs called Actors, which are extremely useful for web automation tasks. Users can publish their Actors and, thanks to auto-generated user interfaces, Actors can be used even by non-developers. Additionally, we’ve built a number of tools especially suited to web automation, such as Apify Proxy, various types of specialized data storages and open-source libraries such as the Apify SDK.
We’ve also built the initial version of the marketplace. Users can submit their request for a custom web scraping or automation solution, get a quote, approve it and then get the project delivered on the Apify platform by an external developer. Web scraping and automation jobs seem well-suited for outsourcing to a marketplace — they are usually small, well-defined and isolated.
However, we still have a long way to go. Now our main focus is to fire up enthusiasm in the developer community about Apify and encourage developers to start building, publishing and sharing Actors. Without developers, we have nothing. To do that, the platform needs to be open, cool, approachable, simple to use and we need to have great onboarding and learning materials, guides and examples. Ultimately, each newly published Actor increases our surface area for search engines and helps bring more users to the platform. Once these users come in, they will either use an existing Actor, build a new one themselves or buy it on the marketplace. This operation needs to become a viral feedback loop and an engine of exponential growth. Besides that, we need to continue building the marketplace, in order to make it fully autonomous and make it work for customers and developers alike. We should take inspiration from existing marketplaces wherever possible, but we also need to be bold and innovative.
In order to support our mission of bringing web automation to everyone, we will continue to provide high-end consulting services for enterprise clients. The purpose of this is twofold. First, to improve the platform and products. Any challenging project that pushes the boundary of what Apify can do is worth pursuing. Second, these projects need to generate enough profit to finance the development of the platform and the marketplace. If a consulting project doesn’t decisively contribute to any of these two goals, there is no point in doing it in-house and it should be delivered on the marketplace, which will, in turn, help us to improve the marketplace itself.
Once the combination of the platform and marketplace works well for both customers and developers in the niche of web scraping and data extraction, it will gradually grow into more general software automation. We already see a clear indication of that in the projects requested by customers. As more developers and customers come in and the capabilities of the platform grow, we want Apify to become a true social cloud computing and development ecosystem where anyone can find, build, buy, sell, run and share cloud software.
In summary, the Apify master plan is:
- Build an awesome cloud computing platform where developers can build, share and make money on new cloud services built on top of existing ones.
- Make it such that it’s also usable by non-developers, who can either find an existing solution or find someone who can build it for them on the marketplace.
- Let the model grow into general software development and create a social cloud computing and development ecosystem.
Our goal is extremely ambitious, but it can be done. We’ll pursue it relentlessly.
