Back to all posts

Apify Blog: Web crawling

Web crawling is what fuels search engines, such as Google. Learn about Apify's web crawlers to help you with your crawling projects.

5 Scrapy alternatives: Playwright, Crawlee, Selenium, BeautifulSoup, MechanicalSoup

Comparison

Scrapy

Crawlee

Alternatives to Scrapy for web scraping in 2025

Mar 20, 2025

Web crawling in Python

Python

Web crawling

Tutorial

How to build a Python web crawler step-by-step

Satyam Tripathi

Mar 18, 2025

Anonymous proxies

Proxy

Web crawling

Web scraping

What is an anonymous proxy? A detailed guide for 2025

Michelle Macias

Mar 17, 2025

AI data collection: Website Content Crawler

AI

LLMs

Web crawling

AI data collection (how to feed your LLM)

Feb 13, 2025

Best open-source web crawlers and scrapers

Web crawling

Web

Crawlee

11 best open-source web crawlers and scrapers in 2025

Dec 15, 2024

What is web scraping? Complete guide.

Web scraping

Data extraction

Web crawling

What is web scraping?

Oct 15, 2024

Lovgren Consulting Group, LLC and Apify

Case studies

Web crawling

Web scraping

How a Salesforce consultant took control of his business to get better leads

Sep 5, 2024

Customer story with deecoob and Apify logos

Case studies

Web crawling

Harmonizing data with deecoob and Apify

Sep 4, 2024

Python caching

Python

Web crawling

Web scraping

Python cache: a complete guide

Satyam Tripathi

Aug 29, 2024