watsonx Orchestrate AI Gateway
The watsonx Orchestrate AI Gateway is where an IBM i shop points IBM watsonx Orchestrate at a large language model it already has, instead of hosting a model next to the agents. IBM's Agent Development Kit documentation lists 14 tested providers, among them watsonx.ai, Red Hat OpenShift AI and Ollama, and a registered model is added with a single orchestrate models command.
What is the watsonx Orchestrate AI Gateway?
Think of a hotel switchboard. A guest picks up any phone in any room and asks for a line out. The switchboard knows which outside line to use, and the guest never has to know. The AI Gateway is that switchboard for AI models. An agent in watsonx Orchestrate asks for a model. The gateway holds the list of models you have registered, the account each one belongs to, and the key that opens it. IBM's own tutorial puts the payoff plainly: every agent can use the most fit model for its task without needing to change the agent code.
IBM has a name for a model you register this way. It calls it a virtual model, which its documentation defines as an external AI model from a supported provider that you register with watsonx Orchestrate. The name you give it follows a fixed shape, virtual-model/provider/model-name. The provider part has to match IBM's list exactly, and if you leave the first part off, watsonx Orchestrate adds it for you. LLM stands for large language model, the kind of model that reads and writes text.
Which model providers can it reach?
- OpenAI, provider ID openai
- watsonx.ai, provider ID watsonx
- Groq, provider ID groq
- Anthropic, provider ID anthropic
- Google Gen AI, provider ID google
- Gemini Enterprise Agent Platform, provider ID vertex-ai
- Azure AI, provider ID azure-ai
- Azure OpenAI, provider ID azure-openai
- AWS Bedrock, provider ID bedrock
- Mistral, provider ID mistral-ai
- OpenRouter, provider ID openrouter
- x.ai, provider ID x-ai
- Ollama, provider ID ollama
- Red Hat OpenShift AI, provider ID redhat-ai
The list is of tested providers, not of every model each one sells. IBM states that not every model and provider combination is supported or tested, and that not every model from those providers is tested or verified to work. One known limit sits on the list already: IBM's release notes say a Gemini Enterprise Agent Platform API key only works with Express Mode endpoints, which the AI gateway does not support, so that provider wants a service account JSON file instead. API stands for application programming interface, the way one program calls another.
A built-in model against a virtual model you register yourself
| Question | Built-in model | Virtual model |
|---|---|---|
| Who supplies it | The platform, out of the box | You, from your own provider account |
| Does it need your own key | No | Yes, held in a connection you create first |
| An example IBM names | gpt-oss-120b on Groq, the default on AWS and IBM Cloud | Any tested provider and model you choose to name |
| How sure is the fit | It ships as a default, so IBM has tested it | The provider is tested, the exact model may not be |
IBM's optimized list names three combinations: gpt-oss-120b on AWS Bedrock, gpt-oss-120b on Groq, and gpt-oss-120b on watsonx.ai for AWS GovCloud only. The ADK documentation adds that on an on premises install a default model is required before you can create agents. ADK stands for Agent Development Kit. Table rows are not linked because this site's content blocks render as plain text.
Why does this matter on IBM Power?
Because it separates the brain from the plumbing. An IBM i partition holds the data and the business rules. A large model wants different hardware and a different budget. The gateway lets the two stay apart. IBM's tutorial spells out the arrangement: you can install watsonx Orchestrate without GPUs, place foundation models on a remote cluster or external providers, and register those models through AI Gateway. GPU means graphics processing unit, the chip most large models run on. So the agent layer can stay small and close to your shop while the model sits wherever you already pay for it, including a watsonx.ai account or a Red Hat OpenShift AI cluster.
The gateway covers the model half of an agent only. The tools an agent can call, such as a query against Db2 for i, come from somewhere else, usually a Model Context Protocol server registered as a toolkit. One more thing to know before an on premises install: IBM Software Hub 5.4 documentation says that if your watsonx Orchestrate installation uses foundation models on a remote or external cluster, you must store your AI gateway model configuration in the internal vault, as a Base-64 encoded payload under a secret named sfc-wxo-secret, before you run the service functions check. An install with local foundation models can skip that step.
How do you register a model?
Four steps, following IBM's own tutorial and the ADK documentation
- Create the connectionorchestrate connections add names the connection, then orchestrate connections set-credentials puts the provider key in it. The key lives in the connection, not in the agent.
- Import the modelorchestrate models import reads a YAML specification file, or orchestrate models add passes the details straight on the command line. YAML is a plain text layout for settings.
- Let it validatewatsonx Orchestrate checks the model after you register it, unless you pass the flag that skips validation. The check includes tool calling, because an agent that cannot call a tool cannot do the work.
- Assign it to an agentorchestrate models list shows what is registered, and from there you pick a model for each agent. Different agents can use different models with no change to the agents themselves.
Commands read from the ADK page Managing custom LLMs with the AI gateway on November 17, 2026. The step order follows the IBM Developer tutorial Extend Your AI Agents with External LLMs Using watsonx Orchestrate and AI Gateway, published September 16, 2025 by Ela Dixit and Ahmed Azraq, which registered a GPT-4o model on OpenAI and a Llama 3.2 model on a local Ollama install. The same family of commands also offers remove, validate and export.
Related Terms
watsonx Orchestrate
IBM's AI agent and automation platform for orchestrating multi-step business workflows across enterprise applications. Since August 2026 it can launch IBM Bob from its Create an agent dialog through a Build with Bob preview, but it still ships no IBM i or Db2 for i connector.
watsonx Orchestrate ADK
The watsonx Orchestrate ADK is the Python package and command line tool that builds and ships IBM watsonx Orchestrate agents from a terminal instead of a web form, and it is where an IBM i shop attaches a Model Context Protocol (MCP) toolkit so an agent can reach Db2 for i. It installs with pip install ibm-watsonx-orchestrate, carries an MIT licence, and its toolkit command accepts either a local MCP server or a remote web address plus a transport, which is exactly what IBM's open source IBM i MCP Server serves.
watsonx.ai
The studio component of watsonx for building, tuning, and deploying foundation models and generative AI applications, including retrieval-augmented generation (RAG) against enterprise data.
Red Hat OpenShift AI
Red Hat's add-on platform for serving and governing AI models on a Red Hat OpenShift cluster, supported self-managed on ppc64le, the IBM Power architecture that also runs IBM i. It lets a Power shop host open weight models on its own hardware instead of calling an outside service.
vLLM
An open source server that keeps a large language model loaded and answers requests over an OpenAI style API, with PowerPC CPUs on its supported hardware list. That lets an IBM i shop run a model in a Linux partition on the same IBM Power server as its production workload, and Red Hat ships a generally available vLLM container image for ppc64le in Red Hat AI Inference 3.5.0.
IBM i MCP Server
An open-source, IBM-published server implementing the Model Context Protocol (MCP), letting AI agents and LLMs query Db2 for i data and take defined actions on IBM i using natural language. In beta since October 2025.
Large Language Model (LLM)
A machine learning model trained on large volumes of text, used as the underlying technology behind generative AI tools including IBM Bob and general-purpose assistants like ChatGPT and Claude.