vLLM

What vLLM does

vLLM calls itself a fast and easy-to-use library for LLM inference and serving. A large language model, or LLM, is the kind of model that reads and answers in plain words. Serving means the model stays loaded and ready, so other programs can send it a question and get an answer back right away. Think of a kitchen that stays open all day instead of cooking one meal and closing. vLLM answers over an API shaped like OpenAI's, so a program written for one can usually talk to the other. An API, or application programming interface, is just the doorway one program uses to ask another program for something.

Its headline feature is called PagedAttention. The vLLM project describes it as efficient management of attention key and value memory, and names it as a key feature behind the serving throughput it claims. Throughput here means how many requests one machine can handle at once.

Can an IBM i shop use it?

Not inside IBM i itself. vLLM is Linux software, and its supported hardware list names NVIDIA GPUs, AMD GPUs and x86, ARM and PowerPC CPUs. PowerPC here means ppc64le, the 64-bit IBM Power architecture that an IBM i machine already runs on. So the usual setup is a Linux partition on the same Power server, sitting beside the partition that runs IBM i. The model and the data stay on hardware you already own.

IBM has a project aimed at exactly that. Project PIM, short for Power Inference Microservices, describes itself as deploying an AI solution on IBM Power servers managed by a Hardware Management Console for IBM i and Linux users, by way of bootc and one button deployment. A Hardware Management Console, or HMC, is the console an administrator already uses to create and manage partitions on a Power server. The project supports IBM i 7.5 and later alongside Linux, and its featured example deploys the vLLM inference engine and exposes an OpenAI API server on a partition. The sample model it loads is IBM Granite 3.3 8B Instruct.

How do you build it on Power 11?

What does Red Hat support on Power?

Red Hat sells a supported build and says its inference stack is powered by vLLM and llm-d. Two Red Hat listings matter if you want a vendor to call when it breaks, and they do not carry the same promise.

The 3.5.0 image name carries Spyre in it. Red Hat lists that release as providing container images that optimize inference for NVIDIA CUDA, AMD ROCm, Google TPU, Intel Gaudi and IBM Spyre AI accelerators, with multi-architecture support for s390x, meaning IBM Z, and ppc64le, meaning IBM Power.