Getting a large language model to respond is easy. Getting it to respond well, repeatedly, on the hardware you already own is the part that can take a week. Precision, how work is distributed across cores, how memory is laid out across NUMA nodes, and which instruction sets the engine uses all depend on both the model and the processor beneath it. Change the model and the good settings change. Move to a different generation of silicon and they change again.
Intel® is announcing Intel® Inference Microservices. Each microservice is a container that serves one model behind an OpenAI compatible API and configures itself for the Intel® processor it detects at startup, using settings Intel® tuned and validated for that model on that silicon. You pick the model. The container picks the settings.
This first release runs on Intel® Xeon® processors, and it is the first step toward a single inference interface across Intel® hardware. A microservice serves one model on one host. When you need a fleet of them, shared across teams with security and routing already handled, the Intel® AI for Enterprise Inference toolkit provides that layer.
Try It Now
The example below serves Qwen3-4B. You need a 4th, 5th, or 6th Gen Intel® Xeon® processor and Docker:
docker run --rm -p 8000:8000 \
--cap-add SYS_NICE
--shm-size=2g \
intel/inference-xeon-qwen-qwen3-4b:0.1.0That is the only command. On startup the container inspects the processor it landed on, selects the best available configuration profile for that model on that hardware, and applies it for you. There is nothing for you to tune and nothing to work out first.
The first start takes a few minutes while the weights download. Wait for /health, then send it a request straight from the terminal:
curl http://localhost:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "Qwen/Qwen3-4B",
"messages": [{"role": "user", "content": "Hello!"}]
}'That is the whole integration. No custom protocol, no Intel® SDK, no Python environment to build, and no engine to install.
Most AI frameworks already speak the OpenAI API, so pointing one at that base URL is all it takes: the OpenAI SDK, LangChain, LlamaIndex, LiteLLM, and Haystack all work unchanged, streaming included.
What It Changes for the Business
- Faster time to production, at the tuned configuration. Tuning, benchmarking, and adjusting for each model on each platform is replaced by a pull and a run, and what you pull is the configuration that performed best in Intel® Hardware.
- No new hardware to procure. Inference becomes a scheduling decision against Intel® Xeon® capacity you already own, not a capital purchase, so a project can start this quarter rather than after a procurement cycle.
- A wider pool of developers can ship AI. Application developers change a base URL. Deploying a model no longer needs an inference optimization specialist, which is the bottleneck most enterprise AI programs actually hit.
- One pattern to standardize on. A validated profile plus a pinned tag means the same model behaves the same way on every matching host, under Docker and under Kubernetes alike.
- What you build now carries forward. The interface is the contract, not the silicon. This release runs on Intel® Xeon®, and as Intel® Inference Microservices extend across the rest of the Intel® hardware portfolio the API in front of them stays the same, so moving a workload onto different Intel® hardware later is a deployment change rather than an application rewrite.
- Your data stays put. Models run in your environment, on your hardware.
What You Get
| Category | What You Get |
|---|---|
| What you run | One container per model |
| What it exposes | An OpenAI compatible API on a port of your choosing, 8000 by default: chat completions, completions, models, health, and Prometheus metrics |
| Engine | vLLM, with additional engines planned |
| What Intel® Inference Microservices sets for you | Precision, core and work distribution, and memory layout, for your model on this processor |
| Hardware | 4th, 5th, or 6th Gen Intel® Xeon® processors, 32 GB RAM suggested. Intel® GPU support planned |
| Clientes that work unchanged | OpenAI SDK, LangChain, LlamaIndex, LiteLLM, Haystack. Change the base URL only |
Performance is the reason the profile exists. Precision, how work is spread across cores, and how memory is laid out are what decide how much a host delivers, and finding the right combination is normally a week of benchmarking for every model on every platform. Each container ships with the best configuration Intel® measured for that model on that processor, so the tuned path is the one that runs by default.
Automatic Profile Selection
Figure 1. Your application talks to an OpenAI compatible API. The microservice container sits between that API and the Intel® processor underneath it.
At startup the container inspects the host it landed on processor generation, how many cores it was given, how many NUMA nodes the machine has, and which built in accelerators the processor offers. That last one is where the real acceleration comes from. Recent Intel® Xeon® cores carry dedicated engines for exactly the math inference spends its time on, Intel® Advanced Matrix Extensions (Intel® AMX) for the matrix multiplication at the heart of a model and the vector extensions (AVX-512, AVX2, and VNNI) for the work around it. They are already in the silicon you own, and they only pay off if the serving engine knows they are there and is configured to use them.
Those findings select a validated profile, the runtime resolves the tuned engine parameters, weights are cached, and the optimized inference engine starts behind the API on the port you mapped, 8000 by default. In this release that engine is vLLM. Two hosts with similar product names can legitimately get different settings if their capabilities differ, which is the point. Constrain the container and it configures for the constraint, not for the host.
What happens when nothing matches depends on which container you run. The model specific images ship with the general profile fallback disabled, so an unrecognized host surfaces as an error at startup rather than as a quietly untuned deployment you discover later in a benchmark. That is the conservative default on purpose: the only configurations those images will run are ones Intel® measured for that model. The base image, the one you point at any model on Hugging Face, defaults the other way with profile fallback `enabled`. When no model specific profile matches, selection uses a compatible general profile instead, one that is model agnostic and tuned to the hardware rather than to the model.
You can see what it found without serving anything, and dry-run will print the resolved profile and the exact vLLM command it would execute:
docker run --rm intel/inference-xeon-base:0.1.0 detect-hardware
docker run --rm intel/inference-xeon-qwen-qwen3-4b:0.1.0 dry-run --format jsonA First Step Toward Unified Inference on Intel®
This first release targets Intel® Xeon® processors with vLLM at BF16 across 4th, 5th, and 6th Gen, on Docker or Kubernetes 1.24+. Tool calling and structured output are reachable today as vLLM pass throughs rather than validated capabilities. A Helm chart, a Kubernetes Operator, additional inference engines, and additional precisions are planned for later releases.
Each model carries its own license, shown on the model's listing in the Intel® Software Catalog. Users are responsible for complying with the license terms of the model they deploy.
Intel® Xeon® is where this starts, not where it stops. The contract a microservice presents is deliberately independent of the silicon behind it: one container per model, an OpenAI compatible API on a known port, and a runtime that detects the platform it lands on and applies the profile Intel® validated for that combination. Nothing in that contract is specific to a CPU. As the portfolio broadens to Intel® GPUs, including the data center inference GPU codenamed Crescent Island , the same microservice model extends to them. The profile that gets selected changes. The interface your application talks to does not.
That is the point of unifying on one interface. Build against Intel® Xeon® today, on processors already in the fleet and available at ubiquitous scale, then move the same workload to an Intel® GPU when model size or economics call for it, without rewriting the application, tuning it again for the new platform, or standing up a second serving stack beside the first. The choice of silicon becomes a deployment decision rather than an engineering project, and it stays a reversible one. A single API spanning general-purpose CPUs and dedicated inference GPUs provides a consistent portability path across Intel® platforms.
That the interface can stay this stable is not a packaging trick. It holds because the work underneath it goes upstream: native integration with the standard frameworks and open inference engines teams already run, with Intel®'s optimizations contributed to those projects rather than locked inside a vendor-specific toolkit. A tuned container stays a standard OpenAI compatible endpoint instead of becoming an SDK you have to adopt. Intel® Inference Microservices are where that blueprint becomes something you pull and run, and the full picture is laid out in Software Connects the System to the Open Model: An Open Blueprint for Production AI.
Get Started
Intel® Inference Microservices ships as software you run on your own infrastructure. There is no hosted endpoint and no API key to create. Pull an image and run it.
- Browse the models to find an image name, and token requirement.
- Read the documentation for the QuickStart, the Kubernetes guide, and the supported platform list.
- Learn more about Intel® Inference Microservices.
- Explore the Intel® Software Catalog for the rest of the developer software.