Automating due diligence to fight financial crime
Diligent AI is building the future of financial crime fighting. The London and Berlin-based Y Combinator company helps fintechs and banks automate customer due diligence - the critical compliance work that stands between criminals and the financial system.
Every year, trillions of dollars flow through banks from drug cartels, terrorist organizations, and corruption networks. Compliance teams spend their days on repetitive verification work when they should be investigating real threats. Diligent AI uses purpose-built AI agents to automate the routine work, freeing compliance professionals for what matters: sophisticated risk assessment.
For Ahmed Gaber, Co-Founder and CTO, this mission needed reliable web scraping infrastructure operating globally across dozens of countries and languages. But Diligent AI couldn't afford to have engineers maintaining scrapers instead of building their core product.
After trying to build scrapers in-house and testing other solutions, Ahmed found his answer in Apify. His assessment is direct:
I don't see a situation where you shouldn't use social media scrapers from Apify. Given the cost and quality.
-- Ahmed Gaber, Co-Founder & CTO, Diligent AI
The build vs. buy dilemma
Diligent AI's product assesses business risk by gathering online data - websites, social media profiles, news mentions, reviews - and stitching it all together to verify it actually belongs to that business.
They needed scraping infrastructure, but the cost-benefit math wasn't simple.
Should I buy or build? That was a bit tricky because we might use excessively. So if I buy it fully, it may put my microeconomics in jeopardy. But at the same time, I can't just build it because knowing how scraping could be, there's a lot of management around it.
-- Ahmed Gaber, Co-Founder & CTO, Diligent AI
The team experimented with building their own scrapers. Headless scraping proved unworkable within days. They spent more time on a custom Facebook scraper, which became particularly challenging - it barely reached 80% accuracy and required constant maintenance.
As the company grew, more issues emerged:
- Global complexity: Supporting businesses from any country with any language created numerous edge cases
- Maintenance overhead: Social media scrapers required ongoing work managing cookies, sessions, proxies, and anti-bot measures
- Infrastructure fragility: When one proxy provider went down, everything fell down. The entire platform was offline for hours - no customer onboardings, no risk assessments, nothing
Open source to production
Ahmed researched open source frameworks and discovered Crawlee. After comparing several options, Crawlee stood out for its comprehensive approach - proxy optimization, speed management, support for Playwright and Cheerio. "Within days, I got something up and running," he says.
That launch got their MVP out the door. But as their needs evolved, so did their Apify usage:
- Needed more proxy traffic → started using Apify's Proxies
- Required residential proxies → available through Apify
- Added Google search data → Apify's Google Search Scraper costs one-third the price of alternatives
- Needed social media data → Apify's pre-built social media scrapers were more reliable than custom solutions
Apify's credit-based pricing encouraged experimentation - Ahmed could try different Actors without committing upfront. Integration took about an hour: "The API works well. It allows you different sync and async ways to integrate, which is great and was rip and replace."
That low friction made it easy to run a direct comparison. Ahmed tested their custom Facebook scraper against Apify's Facebook Pages Scraper side-by-side.
"We had our own scraper for Facebook reviews," Ahmed recalls. "It was so hard to maintain. I was hardly getting to 80%." The constant tinkering - managing cookies, rewriting sessions, fighting anti-bot defenses - meant engineers spent their time on infrastructure instead of the product. When Ahmed switched to Apify's Actor, it worked reliably and retrieved more reviews than their custom implementation ever had.
Results: engineering time where it matters most
Today, Diligent AI runs approximately 40,000 Apify Actor runs per month: ~20,000 Google Search Scraper runs, ~10,000 Instagram Scraper runs, and ~10,000 Facebook Scraper runs.
Cost savings
- 30% cost reduction after switching from SERP API to Apify's Google Search Scraper
- Tens of thousands saved in engineering opportunity costs by eliminating social media scraper maintenance
Reliability
- 99% data quality across their global customer base
- Apify proxies serve as their high-quality fallback after the hours-long outage with another provider
- Failures are rare: "I don't even remember seeing them failing. If they do, it's probably something minuscule that I never noticed."
Speed to market
When Diligent AI wanted to test whether customer reviews could predict fraud, they needed data fast - reviews from Facebook, TikTok, Google, across multiple platforms. "If I would’ve done this manually or on my own, it would have taken days and days," Ahmed says. With Apify's Actors, the data gathering was done immediately. "Now we can focus on what we're trying to experiment."
The same speed applies across experiments: test hypotheses across multiple platforms without developing new scrapers each time.
Team focus
"At our stage, engineering time is more valuable than revenue," Ahmed explains. Every minute spent maintaining scrapers is a minute not spent building fraud detection models.
- Engineers spend time on core product features rather than scraping infrastructure
- No need to manage proxies, anti-bot defenses, or broken scrapers
The data Apify provides has become foundational to Diligent AI's fraud detection models. Without these data sources, their core risk assessment checks cannot run.
Apify’s just production ready, and it's a platform that can get you to 99% easily.
-- Ahmed Gaber, Co-Founder & CTO, Diligent AI
Spending more time fixing scrapers than building features? Ahmed got Crawlee running in days, then scaled to 40,000 monthly runs without adding engineering overhead. Start with Crawlee's open-source framework or skip straight to production for free with ready-to-use Actors.
