Retrieval-Augmented Generation (RAG)
Retrieval-Augmented Generation hands a language model passages pulled from your own data, so it answers from your content. On IBM i, retrieval is the hard half.
How RAG works
- RetrievePull the passages most related to the question from your own data.
- Add to the promptSend those passages to the large language model along with the question.
- AnswerThe model answers from your current content, not only from what it learned in training.
It is the standard way to get AI to answer questions about your documents, your runbooks and your history.
What RAG looks like on IBM i
Who supplies each piece
| Piece | Where it comes from |
|---|---|
| Embeddings | watsonx.ai embedding models and its text embeddings API |
| Search index | Outside Db2 for i. IBM points to OpenSearch on Power |
| System of record | Db2 for i |
| Storing the vectors | Your job, in a database you provide, per IBM's documentation |
The Bob Premium Package for i lists RAG workflows among its skills. IBM has not published where those workflows keep their embeddings.
When you do not need RAG
Exact, structured questions do not need RAG. Send those to Db2 for i through the IBM i MCP Server, and save RAG for questions no single row can answer.
Related Terms
Vector Store
A vector store holds embeddings and finds the ones closest to a query. It is the retrieval half of RAG, and Db2 for i has no vector store as of 2026.
OpenSearch
OpenSearch is the open-source hybrid search engine IBM has positioned as the vector, text and structured search layer for RAG on Power, since Db2 for i has none.
Db2 for i
The relational database management system built into every IBM i operating system, tightly integrated with IBM i rather than a separately installed database product.
IBM i MCP Server
An open-source, IBM-published server implementing the Model Context Protocol (MCP), letting AI agents and LLMs query Db2 for i data and take defined actions on IBM i using natural language. In beta since October 2025.
Large Language Model (LLM)
A machine learning model trained on large volumes of text, used as the underlying technology behind generative AI tools including IBM Bob and general-purpose assistants like ChatGPT and Claude.
Foundation Model
A large, general-purpose AI model trained on broad data and then adapted, through prompting, tuning, or fine-tuning, to specific tasks such as RPG code generation.