Multimodal RAG

Multimodal RAG extends retrieval-augmented generation beyond plain text, so a system can retrieve and reason over images, charts, scanned pages, tables and sometimes audio or video, not only paragraphs. It matters because much of the information businesses care about lives in documents where the layout carries the meaning.

This article explains the three common architectures, when each one works, how to evaluate the result, and what goes wrong. It assumes you know the basic RAG loop: embed documents, retrieve the closest ones for a query, and pass them to a language model to write the answer. That loop comes from the 2020 paper Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. We build document and vision pipelines of this kind under our computer vision and NLP and LLM application services.

Where text-only RAG fails

A typical text pipeline extracts text from a PDF, splits it into chunks and embeds them. That works until the page is not mostly text. A bar chart's meaning is in the bars. A table flattened into a line of text loses which value belongs to which column. A scanned form depends on its layout. A slide deck has an image with three words on it. Text extraction either drops that content or scrambles it, and the retriever never sees what the user is asking about.

Approach 1: convert everything to text

The simplest design keeps the text-only pipeline and adds a preprocessing step. Each non-text element is turned into text first, and from then on the system is ordinary RAG.

• Images and charts are described by a vision-language model, and the description is indexed.
• Tables are extracted into a structured form such as Markdown or CSV and indexed with their headers and caption.
• Scanned pages go through OCR or a layout-aware parser.
• Audio is transcribed.

Strengths. One index, one embedding model, a mature toolchain, and easy debugging, because you can read what was indexed.

Weaknesses. The description is a lossy summary written without knowing the future question. If a chart caption says "revenue by region" but the user asks about a specific quarter in one region, the summary may not contain that value. Quality is capped by the converter, and ingestion becomes slower and more expensive because every image needs a model call. Keep a link from each text chunk back to the original image or page so the generator can be shown the original when needed.

Approach 2: a shared embedding space

Here a single model embeds both text and images into the same vector space, so a text query can retrieve images directly. The well-known example is CLIP, described in Learning Transferable Visual Models From Natural Language Supervision: it was trained on 400 million image and caption pairs to match each image with its text, which lets you describe visual categories in plain language.

Strengths. Natural for photo-like content: product images, scenes, diagrams of objects. You can search images with text or with another image, and there is no captioning cost at ingestion.

Weaknesses. General image-text models are weaker at dense text inside images, such as a page full of small print or a detailed table. Retrieval returns an image, but the generator still needs a vision-capable model to read it. Text-to-image similarity scores are also harder to calibrate than text-to-text scores, so choosing a cutoff takes testing on your own data.

Approach 3: embed page images directly

For visually rich documents, a newer approach skips text extraction. Each page is rendered as an image and embedded by a vision-language model. The ColPali paper describes this: it embeds page images directly, produces multi-vector embeddings, and matches queries to pages through late interaction, which compares query tokens against many vectors per page rather than one. The authors also introduced a benchmark, ViDoRe, for page-level retrieval over visually rich documents.

Strengths. It removes the fragile parsing step. Layout, tables, charts and figures are all part of what is embedded, and one ingestion path handles any page.

Weaknesses. Multi-vector indexes are larger than single-vector ones, since each page stores many vectors, and not every vector database supports that style of scoring, so check support before committing. Embedding page images needs GPU capacity. The retrieved unit is a whole page, so you may pass more than you need to the generator. Evaluate it on your own documents, since published benchmark gains do not guarantee the same result on your content.

Choosing between them

Hybrids are common. A practical system might index a text conversion for keyword and semantic search and page-image embeddings for visual pages, then merge the results. Whichever route you take, build the simple text conversion first. It is a baseline, and it tells you whether the added complexity buys anything.

The generation step

Retrieval is half the system. For generation you have two options.

• Text-only generator. Pass the converted text and any captions. Cheaper, and works with more models, but only sees what the converter produced.
• Vision-capable generator. Pass the retrieved images or pages along with the question. It can read the original chart or table, at a higher cost per request and with limits on how many images fit into one call.

In both cases, instruct the model to answer only from the supplied material and to cite which page or image it used. Citations let a person verify the answer, and they make evaluation tractable.

Evaluating a multimodal RAG system

Evaluate retrieval and generation separately, as you would for text RAG.

1. Build a question set from your real documents, including questions whose answer sits in a chart, a table cell, or a figure caption, not just in body text. Record the page or image that contains each answer.
2. Measure retrieval: for each question, was the correct page or element in the top few results? This tells you whether the indexing approach works before any model writes a word.
3. Measure answers: correctness against a reference, faithfulness to the retrieved material, and whether the cited source actually supports the claim. Use human review on a sample to calibrate any automated judge.
4. Compare approaches on the same set. Try text conversion against page embeddings on your questions. The difference, if any, is the evidence for choosing.

For ongoing operations, the monitoring ideas in LLM monitoring and observability apply directly: trace retrieval and generation separately, and turn failed questions into regression tests.

Where it goes wrong

• Silent extraction failures. OCR returns garbage on a low-quality scan and nobody notices. Sample and review the converted output, and flag pages where the extracted text is suspiciously short.
• Charts without numbers. A description such as "line chart showing growth" is useless. If exact values matter, extract the underlying data when you can or pass the original image to a vision model.
• Stale indexes. Documents change and the image index is not rebuilt. Track a document version or hash with each indexed item.
• Cost surprises. Captioning every image or embedding every page is a per-item cost. Estimate it on the full corpus before ingesting everything.
• Sensitive content in images. Screenshots and scans carry personal data that text filters do not catch. Run detection on images too; see privacy in LLM applications.
• Access control. Retrieval must respect who may see which document. Apply permissions as a filter at query time, not afterward in the prompt.

A sensible build order

1. Collect 50 to 100 real questions and their source pages before writing any code.
2. Build the text-conversion baseline and score it.
3. Identify the failure categories: charts, tables, scans, layout.
4. Add the approach that targets those failures, for example page-image retrieval for the layout-heavy documents.
5. Add a vision-capable generator only where the text path cannot answer.
6. Instrument, evaluate on a schedule, and keep the question set growing.

If you have a document corpus where text-only search keeps missing the answer, request a quote and we will scope a multimodal retrieval pipeline around your documents.