Freelance web scraping on Apify Store towards financial freedom

The Apify platform enables developers to create and monetize their web scraping services and products easier and faster. Check out what it can do for you.

Caleb David, a talented freelance software engineer from Nigeria, has been coding for fun and business for the past five years. In 2022, he shifted his focus to web scraping specifically, working on various projects, gaining expertise, and testing his skills in the tough art of extracting data from websites.

It was just a matter of time he discovered the Apify platform. Now, over 50 Actors later — both in Apify Store and as custom projects — Caleb has a few things to say about his journey: the inspirations, the financial advantages, and the community. So let's hear his story.

🌱 Beginnings

“If I'm being honest. Apify has opened a lot of doors for me, especially regarding my path to financial freedom”.

It was through his work with a Dutch web archiving company Stillio that Caleb first stumbled upon Apify. The company at that time used Apify to automate capturing website screenshots, and Caleb was intrigued by the platform's capabilities. Although he wasn't actively involved in web scraping at the time, as a software engineer he recognized the potential and decided to explore it further.

Two streams of revenue: custom projects and paid Actors

Inspired by the possibilities offered by web scraping, Caleb began building his own scrapers on the platform. He drew inspiration for building scrapers from various sources, including existing projects on ProductHunt or Apify's Ideas page — a place where anyone can submit an idea for an Actor solution.

Once he got better at building scrapers for his own needs, he gained the confidence to take on custom web scraping projects that were posted on Apify Discord. One thing led to another, and eventually, working on these projects, he became a full-time freelancer. Caleb has by now built and maintained over 50 custom Actors, including ones for coupon sites and news websites; some of them are an integral part of his customers' workflows. He built a strong network of clients and until this day maintains long-term relationships with them. However, while working on these private projects, he also thought of making his own solutions public on Apify Store, to see if he could attract any subscription-based users.

Publishing his code on Apify has made it possible for Caleb (known as microworlds on Apify Store) to not only to make his web scrapers searchable, but also to monetize them and secure another stream of revenue. One of Caleb's greatest achievements is the development of Twitter Scraper, which at this point has exceeded 10,000 users, with over half of them regularly active, and is generating stable profit. However, that success didn't come overnight, as Caleb faced challenges along the way.

Challenges along the way

“When I started, I thought my Twitter scraper was the most outstanding Actor, and had the potential to create a significant amount of revenue. But even with that, there were months where it didn't generate anything. During those times, you might feel as if all your efforts have been watered down. This can be discouraging”.

For a long time, there seemed to be no changes. He felt like he put a lot of work in and the number of users just wasn't increasing. Despite the ups and downs, Caleb remained committed to his craft. He believed in the value he provided to his clients and the potential of his Actors, and continued working on the custom projects.

🚀Advantages gained

Community

“The Apify community is a huge plus for me, not just financially but also in terms of the networking spirit it brings. It's something I admire, and I believe it's a huge benefit for anyone looking to try building scrapers”.

One aspect of Apify that Caleb found invaluable was its developer community on Discord. No matter whether it was when he was just starting out or a few years into scraping, the support he received from fellow developers was always there. He could count on somebody's willingness to share tips or untangle a use case. And he has often contributed help himself. This sense of community made Caleb feel motivated and connected to the rest of the IT world.

Not getting hung up on technical challenges

“Apify handles all of those behind-the-scenes scraping hurdles seamlessly and without much effort“.

At some point, Caleb wanted to scale up his scraping to a few million pages. That's no small feat. To achieve that, his scraping solution would have to be able to maintain multiple full browsers on a server at once. With the autoscaling feature [of Crawlee, the open-source web scraping library from Apify], scraping those pages became a breeze. Caleb can easily configure it in maxconcurrency settings.

Financial opportunities

“Being a web scraping service provider is now a big part of my financial progress but also a source of my professional networking opportunities that web scraping on Apify has provided me are unparalleled. It is my greatest achievement (so far)”.

Last month, Apify hit a milestone — we paid out more than $100K in total to community developers of Apify Store. We're thrilled at Caleb's success and look forward to more developers making the Apify platform part of their plan for financial freedom.

💭 Tips for web automation developers

Last but not least, here's Caleb's greatest tip to developers who are considering publishing their own Actors on Apify's marketplace: the most important thing is to remain focused, solve real problems, and stay resilient in the face of challenges.

“If you're solving a problem and you see a potential use case for it, you shouldn't give up. Keep pushing forward, as eventually, with time, you'll have what they call a 'unicorn' – an Actor that everybody wants to pursue. Perseverance and persistence are key. Additionally, luck also plays a vital role here. I know that there were some Twitter Actors in particular, that were outperforming mine, but at a certain point, their position has changed, and mine took the lead. So, one should be resilient in their approach and keep going”.