What is web scraping? Complete guide.
Intro to the basics of web scraping: what it is, how it works, real-world use cases, and how to start.
Category
Web crawling
Web crawling is what fuels search engines, such as Google. Learn about Apify's web crawlers to help you with your crawling projects.
23 Posts
LLM web scraping: using large language models to talk to websites
Discover a new way to interact with websites using large language models. Website Content Crawler makes it easy to ingest and process web content, turning any website into a conversational partner for your AI chatbot or LLM.
Crawlee web scraping tutorial
Give your scraping skills a boost with the most popular full-stack web scraping library for Node.js
The definitive guide to text scraping
How to extract text data from URLs and PDFs for text mining and analysis.
Custom GPTs: how to add a knowledge base
Collect and upload web data to your GPT's knowledge base in 5 easy steps.
Alternatives to Scrapy for web scraping in 2024
Scrapy compared with 5 alternatives for web scraping in both Python and JavaScript. Which one will you try?
What is an anonymous proxy? A detailed guide for 2024
Learn the intricacies of anonymous proxies, their types, benefits, and risks, and how to set them up on various devices.
Concurrency vs. parallelism
Distinctions between concurrency and parallelism in multithreading and their uses in web scraping.
How to check broken links on any website
Broken links on your website can really damage the performance of your website. Lucky for you, we have a free broken links checker that can help you keep your SEO score healthy.
Web crawling vs. web scraping: what's the difference?
Explore the key differences between web crawling and web scraping, two essential tools for web data extraction. This guide breaks down their unique features, use cases, and how they can work together.
Web scraping for AI: how to collect data for LLMs
A tutorial that shows you how to crawl, extract, and process web data to feed, fine-tune, or train large language models.
What are web crawlers and how do they work?
Learn what web crawlers are, how they work, their role in indexing the internet for Google, and how to build your own. Includes essential web crawling tips.
Top 11 open source web crawlers - and one powerful web scraper
Free software libraries, packages, and SDKs for web crawling? Or is it a web scraper that you need?
Announcing Crawlee: the web scraping and browser automation library
Crawlee is an open-source web scraping and automation library that helps you build reliable scrapers. Fast.
Bots and automation tools: are they friends or enemies?
BOTS is a four-letter word. But do bots deserve all the bad rap? We explain the difference between good and bad bots, how to use the good ones, and how to protect yourself against the bad ones.
Pros and cons of web scraping
If you've ever heard of web scraping or web crawling, this post is for you. In this article, we will talk about the pros and cons of using web scraping tools and technologies.
Apify Crawler product to be replaced by an Apify Actor
On September 30, 2019, we’ll be retiring the legacy Apify Crawler product together with Apify API version 1. Please migrate your crawlers to tasks of the new apify/legacy-phantomjs-crawler actor, which provides the same functionality, configuration options and results as Apify Crawler.
