What is a document ingestion pipeline?

It is the system that takes raw files in every format an organization receives (PDF, CSV, JSON, scanned images) and turns them into one normalized, queryable store, generating vector embeddings along the way so an LLM can retrieve from them. It sits upstream of every RAG feature.

Should embeddings be generated during ingestion or at query time?

During ingestion. Embedding on ingest means retrieval is a fast vector lookup instead of an on-demand embedding call. The cost is paid once per document rather than once per query, and every downstream agent shares the same retrieval layer.

How do you handle scanned images and PDFs without a text layer?

Run OCR as an ingestion step, attach a confidence score to the extracted text, and route low-confidence documents to a review queue instead of letting them into the store silently. The confidence score becomes a filter you can use at retrieval time.

How do you keep an ingestion pipeline maintainable as formats change?

Make ingestion idempotent and replayable. Store the raw source file, version the parser, and key records by a content hash so you can re-run the whole corpus through a new parser without duplicating data or losing history.

How to Build a Document Ingestion Pipeline for RAG

Quick answer: A document ingestion pipeline for RAG has four stages. Land the raw file unchanged, parse and normalize every format into one schema, generate embeddings during ingestion, and write to a store keyed by content hash so the run is idempotent and replayable. Get those right and retrieval-augmented generation becomes a lookup problem instead of a reliability problem.

Most retrieval-augmented generation projects fail in the same place. Not the model, not the prompt. The layer underneath, where raw documents are supposed to become something an LLM can retrieve from. When that layer is a pile of one-off scripts, every new source breaks something, and nobody trusts the answers because nobody can say what is actually in the index.

This is the architecture we reach for when the goal is an ingestion layer that stays reliable as sources multiply: one normalized store, embeddings generated on the way in, every run replayable.

What does a document ingestion pipeline actually do?

It converts received files into retrievable knowledge. Concretely, it takes an inbound PDF, a CSV dump, a JSON payload, or a scanned image, and produces normalized records in a single schema plus vector embeddings for each chunk. Everything downstream (agent retrieval, search, analytics) reads from that one store instead of re-parsing source files.

The pipeline is not the RAG feature. It is the thing that makes every RAG feature possible, which is why it deserves to be designed rather than accreted.

The four stages

1. Land the raw file first, untouched

Before any parsing, write the original bytes to object storage and record where they came from, when, and a content hash. This one habit pays for itself repeatedly. When a parser has a bug, you re-run against the originals. When someone asks where an answer came from, you have the source. When a format changes, you can diff old and new.

2. Normalize every format into one schema

This is the hard part and the whole point. A spreadsheet export, a PDF table, and a JSON API response should land as the same shape of record. Define that target schema deliberately (source, document type, extracted fields, full text, extraction confidence, timestamps) and write one adapter per input format that maps into it.

Keep the adapters small and boring. Format quirks belong isolated in the adapter, never leaking into downstream code. If you find yourself special-casing a source three stages later, an adapter failed to do its job.

3. Embed during ingestion, not at query time

Chunk the normalized text, generate embeddings, and store the vectors alongside the records. Doing this on ingest rather than per query changes the economics:

	Embed on ingest	Embed at query time
Retrieval latency	Vector lookup only	Embedding call plus lookup
Cost model	Once per document	Once per query, forever
Consistency	One shared retrieval layer	Varies by caller
Re-embedding	Batch job, replayable	Not really possible

The tradeoff is that changing your embedding model means re-running the corpus. That is fine, because stage 1 and stage 4 made the corpus replayable.

4. Write idempotently, keyed by content hash

Key each record by a hash of its normalized content plus a parser version. Re-ingesting the same file is a no-op. Re-running everything through an improved parser updates records in place instead of duplicating them. The pipeline becomes something you can run a hundred times without fear, which is what makes it maintainable.

Which architecture should run it?

For anything beyond a single machine, event-driven beats a monolithic batch job. A file lands, an event fires, a worker parses it, another event fires, a worker embeds it. In practice that maps cleanly onto managed primitives: object storage, a message queue, serverless functions, and a managed database, on GCP, AWS, or Azure alike.

Event-driven gives you three things that matter in production. Each stage retries independently. You can reprocess one document without touching the rest. Throughput scales by adding workers rather than rewriting the job. For regulated environments it also keeps the whole pipeline inside the client’s cloud boundary, which is often a hard requirement. See running LLMs in air-gapped environments.

How do you know the pipeline is working?

Track four numbers: percentage of documents that parse without error, extraction confidence distribution, embedding coverage (records with vectors versus total), and retrieval hit rate on a fixed evaluation set. When a new source degrades any of them, you find out before the answers get worse instead of after. Retrieval quality itself is a separate discipline, covered in how to evaluate a RAG system.

The takeaway

RAG is only as good as what it retrieves from, and what it retrieves from is only as good as the ingestion pipeline. Build that layer as a real system (raw-first, one schema, embed on ingest, replayable) and the model stops being the bottleneck.

If you are standing up RAG on messy real-world documents and the ingestion layer is where it hurts, that is the work Eximo does.