Until around May of last year, I was building an AI-powered internal chat system.
AI is evolving so fast these days that even a short break makes it feel like ancient history. So before I forget, I wanted to write down the architecture I was working with at the time.
Note: I can't write about the actual business domain as-is, so I'm framing it as an "internal company system." The reality was a bit different :)
This post walks through the general structure of RAG (Retrieval-Augmented Generation), based on the system I was actually building.
Put simply, RAG is a mechanism that lets the AI read company documents before answering.
A typical LLM (the kind that powers ChatGPT) knows general knowledge from the internet, but obviously has no idea about company-specific information.
For example:
None of this is in the training data.
That's where RAG comes in.
The setup was pretty orthodox.
The flow looked like this: