What does Apify have in common with the 2020 Nobel Prize?

Since the launch of Apify Marketplace, the purpose has always been to provide the most cost-effective and frictionless data extraction experience possible. To this end, our team has been hard at work optimizing the process of ordering web scraping and web automation bots. Unlike your generic freelancing platform, Apify is committed to maintaining a pool of expert developers. These certified developers bid on projects and the prospective customer may then choose the offer that suits them best. For everything related to the bids, we decided to let market forces do their thing — unhindered. Over time, we noticed that several projects ended up receiving offers which were either equal or very close to each other. This is despite the fact that:

1. There was no attempt to educate the developers on how to price their bids.

2. The number is left floating up to the smallest decimal.

3. Developers are blinded to each other’s bids. They are only informed when their own bid has been accepted or rejected.

Inspired by the 2020 Nobel prize in Economics, we set out to analyze the bids that the developers placed for all projects since the launch of Apify Marketplace earlier this year. For every project which received two bids or more, we calculated the standard deviation of all the bids and plotted that value versus time. Over the course of a few months, the standard deviation of the bids more than halved (see graph below). That is, bids spontaneously converged towards the market price. And while this phenomenon is bound to emerge in any environment where market forces are unconstrained, the strength of the effect and the speed with which we were able to identify it were surprising. The points on the x-axis (standard deviation equals zero) represent the projects with equal offers. This spontaneous convergence of the bids towards the market price increases consumer surplus. And thanks to the latter, the reliability of the platform, and the world-class service customers receive, about 15% of project requests come from users who already use Apify Marketplace (one of our customers executed 34 projects in a little under 4 months!).

Plot showing that bids are converging over time — **Standard deviation of bids versus time**

As a moderated online marketplace, we intervene in each project to ensure smooth communication and high data quality. Because of this, our satisfaction rate is close to 100%. Since the beginning of the year, we have processed close to 1,000 project requests. All Apify Marketplace developers are carefully vetted, bound by strict non-disclosure agreements and undergo a thorough onboarding process.

The way Apify Marketplace is able to position itself as the most competitive solution in this space is by leveraging the entire Apify ecosystem in service of anyone looking for custom scraping/automation solutions. Apify provides an SDK that supports state-of-the-art scraping and automation technologies. This SDK is regularly updated to provide tools for bypassing the latest bot mitigation measures. Once developed, the solution is automatically deployed onto Apify’s scalable cloud platform. As the name of the company suggests, interacting with the solution is then possible through dedicated API endpoints (or through a simple interface from a web browser).