Back to all posts

Apify Blog: Web crawling

Web crawling is what fuels search engines, such as Google. Learn about Apify's web crawlers to help you with your crawling projects.

What is retrieval-augmented generation (RAG)?

AI

LLMs

Data

What is retrieval-augmented generation (and why use it for LLMs)?

Jul 10, 2024

How to send a GET request with cURL

cURL

Web crawling

Data extraction

How to send a GET request with cURL: step-by-step guide

Jun 7, 2024

How to build a GPT with a knowledge base

AI

LLMs

ChatGPT

Custom GPTs: how to build a GPT with a knowledge base

Jun 5, 2024

LLM web scraping: illustration showing code snippets from using website content crawler

AI

LLMs

Web crawling

Using large language models to talk to websites

Apr 17, 2024

Crawlee web scraping

Crawlee

Tutorial

Web scraping

Crawlee web scraping tutorial

Percival Villalva

Apr 2, 2024

Text scraping

Web crawling

Web scraping

Data extraction

The definitive guide to text scraping

Mar 8, 2024

Edge AI vs Cloud AI

AI

Machine learning

Comparison

Edge AI vs. Cloud AI

Nov 23, 2023

Concurrency and parallelism in web scraping

Programming

Comparison

Web scraping

Concurrency vs. parallelism

Oct 30, 2023

How to check broken links on your website? Let Broken Links Checker do the heavy lifting.

SEO

Web crawling

Tutorial

How to check broken links on any website

Sep 10, 2023