What is data ingestion for large language models?

Getting data for a large language model sounds straightforward. Scrape some websites, download datasets, gather documents, etc. Surely the hard part comes later?

Until you open those files and find that some repeat the same text several times. A scanned PDF contains images of pages instead of selectable text. Product reviews are labeled “positive” in one dataset and “good” in another, but it’s unclear whether the labels mean the same thing.

You see, collecting data is just the start. You also need to decide what to use and prepare it for the task ahead. If you’re training a model, you need examples it can learn from. If you’re using reference documents to help it answer questions, you need to make the right information easy to find.

Data ingestion is the foundation that feeds this process. Let’s look at the steps, potential problems you might encounter, and the tools that can help.

How data ingestion works for LLMs

Data ingestion is the process of collecting data from various sources and bringing it into a system for use. For LLMs, that might be a training pipeline or a collection of reference material the model can draw on when answering questions.

An ingestion pipeline may also extract text, check its quality, prepare it, and store it. The exact scope varies by platform. Microsoft’s documentation, for example, includes extraction, transformation, and loading.

Data can come from webpages, books, code, databases, or examples created for a specific task. Models that work with images or audio also need to undergo transformation. This article, however, focuses on text used for training or as reference material for large language models.

Start with what the data is for

Before choosing tools or extracting files, decide how the model will use the data. Three common uses are for pretraining, fine-tuning, and retrieval. Each calls for a different kind of collection.

Pretraining: building a model’s foundations

Pretraining builds a model’s initial capabilities from large amounts of data, which may include prose, technical writing, or code. Data ingestion gathers and prepares that material. The languages, subjects, and sources you choose — and the quality of the text you collect — affect what the model outputs.

Hugging Face’s FineWeb is one example of a pretraining dataset. Preparing it involved extracting text from webpages, filtering for language and quality, removing duplicates, and masking certain personal identifiers.

Fine-tuning: teaching an existing model a task

Fine-tuning trains an existing model further for a specific goal. In supervised fine-tuning, you prepare examples that pair an input with the desired response.

To classify product reviews as positive, negative, or mixed, for example, you would collect varied reviews and label them consistently. Other tasks might use conversations, instruction-and-response pairs, or examples in a required output format.

The structure depends on the training method. Supervised fine-tuning tools support several formats, so there’s no single template for every dataset.

Retrieval: preparing information the model can look up

With retrieval-augmented generation (RAG), an application finds relevant information and gives it to an LLM along with the user’s question. For example, when an employee asks an internal AI assistant about the company’s travel policy, it can find the relevant passage and use it to answer.

Ingestion prepares reference material so it can be found. Retrieval supplies information when a question is asked; it doesn’t change the model’s parameters. A fine-tuned model can use retrieval too.

How to prepare data for an LLM

The first steps are similar whether you’re preparing data for training, fine-tuning, or retrieval: choose your data sources, extract the content, clean it, and check the results. Further work depends on the task and the results you need from the LLM.

1. Choose sources that fit the task

Start with the goal. A multilingual pretraining dataset needs an intentional mix of languages. A review classifier needs examples of every category it should recognize. A reference collection needs documents that can answer the questions people will ask.

You might download a dataset, use an API, connect to a database, or scrape webpages. Dataset-loading tools can help with files stored locally or online. As you collect the data, record its source, version, and any conditions on its use. That makes it easier to trace errors, repeat the work, and check whether the data is suitable for your purpose. A dataset card can keep those details together.

2. Extract the content without losing its meaning

Webpages mix useful text with menus and adverts. PDFs pose different problems: scanned pages may need optical character recognition (OCR) to turn images into text, while columns and tables need careful extraction so the reading order and headings still make sense.

Take a product price table. If extraction separates the prices from the product names, the words and numbers are still there, but you can’t tell which price belongs to which product. A tool like Docsumo helps extract text and tables from PDFs and scanned documents.

3. Clean the data and verify what remains

Cleaning matters because errors in the source can carry through to training or search. But an overly broad cleaning step can also remove something important, so you need to check the result against your original.

Fix unreadable text, remove irrelevant records, identify duplicates, and handle sensitive information where needed. Look for missing text and empty records, and check whether you have the expected number of usable records. If the data has labels, make sure their meanings remain consistent.

To monitor recurring runs, track how many documents were processed, failed, or produced empty records. You can review those numbers in a pipeline dashboard or use embedded analytics tools to display them inside your application.

Preparing data for training

Once the source material is usable, turn it into examples that fit the training method.

1. Format examples and set aside evaluation data

For pretraining, you might collect passages of ordinary writing or code. The model learns by repeatedly predicting what comes next in the text and checking its prediction against the passage. You don’t need to write a separate answer for each one. Hugging Face’s guide shows how passages are prepared for training.

Fine-tuning for a specific task, however, needs a different kind of example. If you want a model to classify reviews, each review needs a label showing the answer it should learn:

{"review":"The headphones sound great, but the battery barely lasts an hour.","label":"mixed"}

Always set aside some examples to check the model as you develop it, and a separate group for the final test. Make sure those groups don’t contain copies or near-copies of the training examples. If a review appears in both training and testing, the model may get it right because it has seen it before. That makes the test score less reliable as a measure of how it handles new reviews.

2. Tokenize and store the examples

Before a model can process text, a tokenizer breaks it into pieces called tokens and assigns each piece a numerical ID. If you’re fine-tuning an existing model, use the tokenizer and input format it expects. Check what happens when an example is too long. If the software cuts it off, it could remove the answer you want the model to learn or the context needed to understand it. Tokenizer settings control how inputs are shortened or padded.

Save the examples in a format your training software accepts, such as JSONL or Parquet. If the dataset is too large to fit in memory, you can split it across files or stream it in smaller parts. Preparing the data gets it ready for training; the model changes only when training updates it.

Preparing data for retrieval

For retrieval, you prepare documents so an application can find relevant passages when someone asks a question and pass them to the model. The documents stay in a reference collection rather than becoming training examples.

1. Split documents and preserve context

Chunking divides documents into passages that an application can find and pass to the LLM. If an employee asks whether they can book a hotel outside the company’s travel portal, the application should find the passage explaining that manager approval is required.

To make that possible, split documents so a rule stays with its conditions and exceptions. Section boundaries can help, as can keeping table headings with their data. The right chunk size and method depend on the documents and the questions people are likely to ask.

Once you have the passages, keep source details with each one. That way, the application can tell which policy a passage came from and whether it belongs to an older version. For example:

{
  "text": "Hotel bookings outside the travel portal require manager approval.",
  "document": "Travel policy",
  "section": "Hotel bookings",
  "effective_date": "2026-01-01",
  "source": "https://example.com/travel-policy"
}

The passage contains the approval rule. The other fields show where it came from and when the policy took effect, helping the application distinguish it from an older version.

2. Index the passages and test the search

Put the passages in a search index so the application can find them when someone asks a question. It can search by keywords, use embeddings to find related passages even when the wording differs, or combine both methods. These are called keyword, vector, and hybrid search. Whichever method you choose, the application must be able to retrieve the passage text to give the LLM. A dedicated vector database is one option, but it isn’t required for every setup.

Test the search with questions people would actually ask, and note which passages should come back. If it retrieves last year’s travel policy, look at the search results and how policy versions are handled. If it retrieves the current policy but the answer is wrong, look at how the LLM used that passage. Checking retrieval and answers separately helps you locate the problem.

Tools for LLM data ingestion

The right tools depend on what you’re collecting and what the data is for. A crawler can gather webpages for use in training or retrieval, while other tools help parse documents, prepare training datasets, or index passages so they can be found and passed to the LLM.

You don’t need every tool in the table. Choose the ones that match your data sources and what you want to do with the data.

Start small, then expand

Before processing a full collection, test a small sample that includes the different kinds of content you plan to collect. Compare the results with the originals to catch missing text or incorrect labels. For training, inspect the examples and keep evaluation data separate. For retrieval, check that important details survive extraction and that realistic questions find the right passages.

For a retrieval test, you could collect 15 pages from a university admissions website with Apify Website Content Crawler. Include an FAQ, an entry requirements table, and pages with different deadlines. Then check whether the extracted text preserves those details and whether your search returns them when asked.

Once the sample works, expand the collection and repeat the checks. If you need data from other kinds of sites, explore the scrapers in Apify Store.