Multimodal RAG — Intuitively and Exhaustively Explained | by Daniel Warfield

Artificial Intelligence | Retrieval Augmented Generation | Multimodality

Modern RAG for modern models.

“Multicolored Team” by Daniel Warfield using Midjourney. All images by the author unless otherwise specified. Article originally made available on Intuitively and Exhaustively Explained.

Multimodal Retrieval Augmented Generation is an emerging design paradigm that allows AI models to interface with stores of text, images, video, and more. Essentially, multimodal RAG allows a model to reference rich and diverse information about the world.

First, we’ll cover what retrieval augmented generation (RAG) is, the idea of multimodality, and how the two are being combined to make modern multimodal RAG systems. Once we understand the fundamental concepts of multimodal RAG, we’ll build a multimodal RAG system ourselves using Google Gemini and a CLIP style model for encoding.

Who is this useful for? Anyone interested in modern AI.

How advanced is this post? Even though multimodal RAG is at the forefront of AI, it’s intuitively simple and accessible. This article should be interesting to senior AI researchers, while simple enough for a beginner.

Pre-requisites: None