Retrieval-Augmented Generation (RAG)

Retrieval-augmented generation, almost always shortened to RAG, retrieves the passages most relevant to a question from your own data and includes them in the prompt sent to a large language model, so the answer is grounded in current, specific content rather than only in what the model absorbed during training. It is the standard design for getting an AI system to answer questions about your documents, your runbooks and your history.

On IBM i the generation half is straightforward and the retrieval half is where the work is. Retrieval needs a vector store, and Db2 for i has none as of September 2026, so the index lives outside the database. IBM's stated direction is OpenSearch running on Power as the hybrid vector, text and structured search layer, with Db2 for i staying the system of record. watsonx.ai supplies the embedding models and the text embeddings API that create the vectors, and IBM's documentation is explicit that storing them is your job, in a database you provide. The Bob Premium Package for i lists RAG workflows among its IBM i skills, but IBM has not published where those workflows keep embeddings.

One thing worth deciding early: exact, structured questions do not need RAG at all. Route those to Db2 for i through the IBM i MCP Server and save retrieval for questions no single row can answer.