Crawl4AI vs. Website Content Crawler

I wanted to build a support chatbot on top of Shopify's help center. It needed to give visitors clear instructions without losing important conditions and exceptions that matter.

For example, if a merchant asks how to add inventory quantities, the chatbot needs to know that stores with more than 50 locations must save the product first.

Apify's Website Content Crawler and Crawl4AI fetched every page I gave them. Neither reported a failed request. Yet with the initial cleaning settings, both left out that instruction.

A successful crawl tells you a page was retrieved. It doesn't tell you whether every detail survived. A run can look perfect and still leave gaps in the information your application relies on.

How the two crawlers differ in practice

Website Content Crawler is an Actor you configure in Apify Console. You supply URLs, choose whether to follow links, and set rules for what content to keep. Apify runs it, stores the results in a dataset, and keeps a record of each run.

Crawl4AI is an open-source Python library. You write the code that controls page loading and extraction, then run it on your own machine or infrastructure. Crawl4AI Cloud offers a hosted API, but this benchmark tested the Python library.

What I tested

I tested Crawl4AI's open-source Python library against Apify's Website Content Crawler using the same Shopify Help Center URLs. I ran each tool twice at separate scales: 20 articles covering 15 topics, before expanding to 1,000 URLs.

I checked content quality on a sample of the 1,000 pages. I chose two details from each of the first 20 articles that the collected text needed to preserve. After adjusting the crawlers, I checked one detail in each of 20 additional articles. That gave me 60 content checks across 40 pages.

To keep the comparison consistent, both tools fetched HTML without running JavaScript or following links. Each was allowed two concurrent requests, one retry, and a 60-second timeout per page.

The results

Round 1: initial cleaning settings, 20 URLs

Both tools returned all 20 pages without reporting a failed request. But Apify's output missed 7 of the 40 checked details, and Crawl4AI's missed 6. Both omitted the exception for stores with more than 50 locations. Apify also left out later steps for connecting a domain and some section headings.

Crawl4AI's run summary explains why its number came out where it did, and it records the hardware caveat that applies to every timing in this article.

Where the missing details went

I used Crawl4AI's pruning filter to remove navigation and other page clutter. It saved the unfiltered Markdown in .raw.md and the pruned version in .fit.md.

All 40 checked details appeared in the raw files, but only 34 remained after pruning. The crawler had fetched those six details; the filter removed them.

Turning off the filter alone could bring back menus and other clutter. Shopify keeps the help article inside an article.article section, so I could select that section directly instead of relying on a filter to decide what mattered.

Apify's cleaned output had also missed details, so I made the same change to both setups and checked the results in the 1,000-URL crawl.

Round 2: targeted extraction, 1,000 URLs

The first round passed 33 of 40 checks for Apify and 34 of 40 for Crawl4AI. After the adjustment, both passed all 60 checks across 40 reviewed pages in the larger crawl.

Three other results stood out:

• Speed: Apify finished both 1,000-URL runs sooner, taking about 15% less time on average.
• Retries: Apify recorded 45 retries in its first large run and 22 in its second. Crawl4AI recorded none.
• A page that needed a closer look: The inspection flagged the same pos-hardware-support URL in both outputs. The request returned HTTP 200, but neither record contained an article body.

What the runs cost

Apify's two 1,000-URL runs cost $0.249 combined, well within its Free plan's $5 monthly credit; Crawl4AI had no software licence fee when I ran it on my computer, though I managed the script myself and would pay extra for hosting or proxies.

Crawl4AI Cloud's pricing page lists basic page fetches at 0.5 credits, with each credit worth $0.001. Browser-based fetches and harder sites cost more.

Which crawler should you choose?

With the adjusted settings, both tools returned 1,000 records and passed all 60 content checks; the choice then comes down to how you want to run and manage future crawls:

Crawl4AI makes sense if you want crawling built into a script you already maintain, on infrastructure you already run. Apify is best if your collection needs to keep running without a developer attached to it, with scheduled reruns, run history, shared access, and no server to maintain.

Methodology: reproduce the benchmark yourself

Everything above is reproducible. The benchmark ran on the files below — the exact URL lists, the content checklists I scored against, and the scripts. Your numbers may differ, because Shopify's pages and network conditions change.

The checklists guide your review; they don't score the crawlers automatically. A check passes only if all its required details are present and clear enough to follow. You'll also create prepare_apify.py along the way to format the URL lists for Apify.

Step 1. Set up your workspace

Install Python 3.12 and PyCharm. VS Code works too.

Create a folder for the benchmark and put the files from the table above in it. In PyCharm, choose File → Open and select that folder. In the Project panel, double-click urls-20.json. These are the 20 addresses both crawlers get.

Click the Terminal icon near the bottom-left. On Windows, if the prompt starts with PS, type cmd and press Enter. Then:

py -3.12 -m venv .venv
.venv\Scripts\activate.bat
python -m pip install -r requirements.txt

On macOS or Linux:

python3.12 -m venv .venv
source .venv/bin/activate
python -m pip install -r requirements.txt

After installing you should see (.venv) at the start of the prompt. Leave this terminal open.

Step 2. Collect the first 20 pages with Apify

Apify expects addresses and crawl settings in a specific format. A short script prepares that input, so you don't format 20 addresses by hand now and 1,000 later.

In PyCharm's Project panel, right-click the folder containing urls-20.json and choose New → Python File. Name it prepare_apify.py, then paste:

import argparse
import json
from pathlib import Path

parser = argparse.ArgumentParser()
parser.add_argument("urls", choices=["urls-20.json", "urls-1000.json"])
parser.add_argument("--article-only", action="store_true")
args = parser.parse_args()

urls = json.loads(Path(args.urls).read_text(encoding="utf-8-sig"))
count = len(urls)
if count not in (20, 1000) or len(set(urls)) != count:
    raise ValueError("Use the unchanged list of 20 or 1,000 unique URLs.")

settings = {
    "includeUrlGlobs": [{"glob": "https://help.shopify.com/en/manual/**"}],
    "crawlerType": "cheerio",
    "maxCrawlDepth": 0,
    "initialConcurrency": 2,
    "maxConcurrency": 2,
    "requestTimeoutSecs": 60,
    "maxRequestRetries": 1,
    "maxSessionRotations": 1,
    "respectRobotsTxtFile": True,
    "proxyConfiguration": {"useApifyProxy": True},
    "htmlTransformer": "readableText",
    "saveMarkdown": True,
    "saveHtml": True,
    "useSitemaps": False,
    "summarize": False,
    "startUrls": [{"url": url} for url in urls],
    "maxCrawlPages": count,
    "maxResults": count,
}

if args.article_only:
    settings["keepElementsCssSelector"] = "article.article"
    settings["htmlTransformer"] = "none"

suffix = "-article-only" if args.article_only else ""
output = Path(f"apify-input-{count}{suffix}.json")
output.write_text(json.dumps(settings, indent=2) + "\n", encoding="utf-8")
print(f"Saved {output} with {count} URLs")

Save, then run:

python prepare_apify.py urls-20.json

When you see Saved apify-input-20.json with 20 URLs, open the new file and copy its entire contents.

Open Website Content Crawler, click Try for free or Open in Console, and sign in. In the input editor, switch to JSON and replace the existing input with what you copied.

Switch back to the input form and confirm HTML transformer is set to Readable text, the default cleaning option for this first run. Then open Run options:

Type 0.3.98 into the Build field to match my test. If you use version-0, record the build number from your first run and keep it for all remaining tests.

Click Start. When it finishes, open Storage → Dataset, pick an article, and read its markdown field. Then choose Export → JSON → Download, leaving Select fields and Omit fields empty so the saved HTML is included. Save it as apify-20-1.json.

Run it a second time with the same input and save as apify-20-2.json. Record both durations, since two runs give you a range rather than a single figure.

Step 3. Run Crawl4AI and score both outputs

In the terminal, run these one at a time, waiting for each to finish:

python run_crawl4ai.py --urls urls-20.json --output crawl4ai-20-1
python run_crawl4ai.py --urls urls-20.json --output crawl4ai-20-2

Expand crawl4ai-20-1 and open summary.json. The elapsed_seconds field gives the duration; results lists each URL and its saved files. Do the same for crawl4ai-20-2.

Now score. Use quality-checks-initial-40.json against the original Shopify articles. For Apify, read the markdown field in apify-20-1.json. For Crawl4AI, use the filtered .fit.md files. Give each crawler a score out of 40 and record the IDs of failed checks.

When a detail is missing from a .fit.md file, open the matching .raw.md to see the text before filtering. That comparison is what showed the pruning filter was responsible. Keep your score based on the filtered output, since that is what you would ship.

Step 4. Target the article and turn the cleaning off

Shopify wraps each article in an <article> element with the class article. Open any help article, right-click a paragraph, choose Inspect, and find that element in the developer tools. Hover it to confirm the region it covers.

urls-1000.json contains the original 20 pages plus 980 more, so the adjusted run re-checks the same details at scale. Generate the new Apify input:

python prepare_apify.py urls-1000.json --article-only

Open apify-input-1000-article-only.json, copy its contents, and paste them over the existing input in Apify's JSON editor. Back in the input form, open the HTML Extraction section and set Keep HTML elements (CSS selector) to article.article and HTML transformer to None.

Under Run options, keep the same build and memory but raise the timeout to 1,800 seconds. Leave the other limits alone.

Step 5. Run the larger crawls and verify

Run Apify twice with the new input, one at a time, saving the full JSON exports as apify-1000-1.json and apify-1000-2.json. Then run Crawl4AI with the same list. The --article-only option applies the same selector and disables pruning:

python run_crawl4ai.py --urls urls-1000.json --article-only --output crawl4ai-1000-1
python run_crawl4ai.py --urls urls-1000.json --article-only --output crawl4ai-1000-2

The script already caps these runs at 1,800 seconds. Record durations from Apify's run pages and from elapsed_seconds in each Crawl4AI summary.json.

Check retries and failures. For Apify, open each run's Log tab, click View full log, press Ctrl+F, search for Final request statistics, and use the last match near the bottom. Look for:

• retryHistogram: the first number counts requests that needed no retries; add the rest to get the retried total. [968, 32] means 968 requests needed no retry and 32 needed one.
• requestsFailed: requests that still failed after the allowed retries. 0 means none.

For Crawl4AI, summary.json reports retries and failed_results directly.

Check that records actually contain articles. This step is what caught pos-hardware-support:

python inspect_shopify_articles.py --apify apify-1000-1.json --urls urls-1000.json --output apify-1000-1-inspection.json
python inspect_shopify_articles.py --crawl4ai crawl4ai-1000-1 --urls urls-1000.json --output crawl4ai-1000-1-inspection.json

Compare record_count with structurally_valid_article_count in each inspection file. If they differ, open the entries under flagged_urls and read the saved records.

Score the content again. Use quality-checks-initial-40.json for the original 20 pages and quality-checks-additional-20.json for 20 more pages. That gives you 60 checks across 40 pages. This time, review Apify's exported markdown and Crawl4AI's .raw.md files, since pruning is off.

Compare your findings with the results above. If you want to try the same approach on your own collection, find a scraper in Apify Store, start with a few pages, and check the output before scaling up. Apify's Free plan includes $5 in monthly usage credit and requires no credit card.