In December 2025, Apify and The Web Scraping Club joined forces to ask their communities how they’re scraping today, what has changed compared to last year, and what the outlook is for 2026.
The result is the state of web scraping report 2026, in which we share insights directly from our survey, extracted from thousands of answers to our questions.
The report is based on responses from hundreds of web scraping professionals, and covers 4 main areas:
- Proxy usage
- Infrastructure
- Bot detection
- AI scraping
The professionals who answered the survey represent a mature and diverse industry. 35.8% of respondents work as freelancers, while 49.1% work in startups and small or medium-sized businesses.
Web scraping is widely used across various industries, with e-commerce and social media making up the most targeted types of websites.
Operationally, the biggest shifts are cost-related. 65.8% of respondents reported increased proxy usage, and 58.3% said their proxy spending increased year over year.
Infrastructure costs followed a similar trend, with more than 62% reporting increased spending, driven largely by stronger anti-bot protections.
The biggest surprise is in the answers related to the use of AI in web scraping. 54.2% of respondents stated that they don't use AI in their scraping workflows, and only 45.8% of surveyed web scraping experts are leveraging AI in their projects.
The reasons for not using AI for web scraping were:
- Lack of trust in the results.
- High costs.
- Technical challenges when integrating or using these tools.
- Unreliable performance on certain websites.
- Uncertainty about practical benefits or use cases.
Despite this skepticism, when asked whether they plan to try AI-assisted web scraping tools, 66.2% of respondents said yes, while only 33.8% indicated they don't plan to use them even in the future.
No less interesting is that among those using AI, 72.7% report that AI delivers productivity advantages, and 100% claim that they plan to increase the use of AI-assisted scraping tools in the future.
The obvious conclusion is that developers who have embraced AI in web scraping not only find it useful but also expect its adoption to grow steadily, signaling that AI is here to stay.
