llama.cpp
llama.cpp is an open source C and C++ engine that runs a large language model on ordinary processors, with no graphics card and no cloud account. IBM packages it for IBM i as the llama-cpp RPM in the base repository that IBM i 7.3 and later systems already use, so a yum install puts llama-cli and llama-server inside the IBM i partition itself.
What does llama.cpp do?
A large language model, or LLM, is the kind of AI that reads plain words and answers in plain words. Most ways of using one send your question out to a company over the internet, or need an expensive graphics card. llama.cpp is a third way. It loads the model file on the machine you already own and does the math on that machine's regular processors. Think of a cookbook on your own kitchen shelf instead of phoning a chef every time you want dinner.
The project says its main goal is to enable LLM inference with minimal setup and state-of-the-art performance on a wide range of hardware. Inference just means using a trained model to answer something. The model itself arrives as one file in a format called GGUF. The package also includes llama-server, which keeps the model loaded and takes questions over an HTTP connection. Its own notes say that connection is compatible with the OpenAI API, so programs written for OpenAI can usually talk to it without being rewritten.
Can it run inside IBM i?
Yes, and that is what makes it unusual. Most model servers, vLLM included, are Linux software, so an IBM i shop runs them in a separate Linux partition on the same IBM Power server. llama.cpp runs in PASE, the AIX style runtime that sits inside IBM i itself. IBM builds the package, puts it in its own repository and labels it for IBM i 7.3 and later, so the model can answer from the same partition that runs the production work.
How llama.cpp arrived on IBM i
- 2025-05-05The llama.cpp project tags upstream build b5281.
- 2025-10-06IBM builds its IBM i package from that build and names it llama-cpp-b5281-1.ibmi7.3.ppc64.rpm.
- 2025-10-17The file appears in IBM's base RPM repository, repo-base-7.3, which the ibmi-repos package enables by default on IBM i 7.3 and later.
- 2026-08-11An IBM Community post by Amit Kumar wires five llama-server endpoints to IBM Bob through a Model Context Protocol server, giving the assistant a list of local models it can ask.
The first three dates come from IBM's own package data on public.dhe.ibm.com and the project release tag, both read on 11 November 2026. The IBM Community post does not say which operating system hosts its five servers, so treat it as a working pattern rather than proof of an IBM i install.
How do you get a model running on it?
Four steps from empty partition to an answer
- Install the engineRun yum install llama-cpp from a PASE shell. The files land under /QOpenSys/pkgs/bin, the same place every other IBM i open source package goes.
- Get a model in GGUF formGGUF is the one file format the engine reads. The package includes a converter script, convert_hf_to_gguf.py, that turns a model downloaded from Hugging Face into a GGUF file.
- Match the byte orderIBM i is a big endian machine, which means it stores the bytes of a number in the opposite order from most computers. Published GGUF files are usually little endian, so the conversion needs the converter bigendian option. Its help text reads: model is executed on big endian machine.
- Start the server and point a program at itllama-server keeps the model loaded and answers over an OpenAI compatible API, so an RPG or SQL integration that already calls a cloud model can usually be aimed at this one instead.
No model ships with the package. The download is about 6 MB and its files unpack to about 63 MB, so every byte of a model is on top of that.
What comes in the package?
Files the IBM i RPM installs in /QOpenSys/pkgs/bin
| File | What it is for |
|---|---|
| llama-cli | Send a prompt from the command line and read the answer. |
| llama-server | Keep the model loaded and answer over an HTTP API shaped like the OpenAI one. |
| llama-quantize | Shrink a model file so it needs less memory, at some cost in accuracy. |
| convert_hf_to_gguf.py | Turn a downloaded Hugging Face model into the GGUF file the engine reads. |
| llama-run, llama-simple, llama-simple-chat | Smaller runners and example programs included in the same build. |
File list read from IBM package data for llama-cpp-b5281-1.ibmi7.3.ppc64.rpm.
What should you check before you lean on it?
IBM shipping the package is not the same as IBM supporting the software. IBM open source support page says a software maintenance agreement covers installing IBM delivered RPM packages, while actual usage or defect support needs community help or a commercial contract, which it names as OpenLogic by Perforce. That page was last changed on 21 April 2023.
The work also runs on the server processors and nothing else. IBM source package asks only for OpenBLAS, which is a math library, and the OpenMP library that spreads work across processor cores. There is no accelerator card in its requirement list, so the answer speed you get is the speed of the cores you give it. Watch what that does to the rest of the partition before you let it loose on real users.
One more thing to weigh is age. The build IBM ships is b5281, tagged on 5 May 2025, and the project tags new builds nearly every day. If you need something newer you build it yourself, and IBM own source package lists what that takes: a gcc 12 C and C++ compiler, cmake, git, binutils and the OpenBLAS development files.
Related Terms
Large Language Model (LLM)
A machine learning model trained on large volumes of text, used as the underlying technology behind generative AI tools including IBM Bob and general-purpose assistants like ChatGPT and Claude.
IBM PASE for i
PASE for i is the AIX runtime built into IBM i, and it is where the open-source software behind most AI work on the platform actually runs. It ships as option 33 of 5770-SS1, has been a no-charge option since V5R2M0, and it hosts the yum-installed Node.js and Python builds under /QOpenSys/pkgs, including the Mapepire server that IBM's IBM i MCP Server connects through on port 8076.
vLLM
An open source server that keeps a large language model loaded and answers requests over an OpenAI style API, with PowerPC CPUs on its supported hardware list. That lets an IBM i shop run a model in a Linux partition on the same IBM Power server as its production workload, and Red Hat ships a generally available vLLM container image for ppc64le in Red Hat AI Inference 3.5.0.
IBM Granite
IBM's family of open-weight foundation models, released under the Apache 2.0 license, that underpins watsonx.ai and serves as the base model fine-tuned for RPG and IBM i in watsonx Code Assistant for i and IBM Bob.
DbToo
DbToo lets RPG programs and Db2 for i data call AI models and outside services such as watsonx, Ollama and OpenAI-compatible endpoints. It is an experimental open-source SDK from IBM, and IBM says not to run it in production yet.