Fix vLLM CUDA OutOfMemoryError in Kubernetes

If your vLLM pod dies at startup with torch.cuda.OutOfMemoryError: CUDA out of memory, the model plus its KV cache needs more VRAM than the GPU allocated to that pod can give. The fastest fix is to cap two things: pass --gpu-memory-utilization 0.85 and --max-model-len 4096 on the serve command, then redeploy. If you have more than one GPU in the pod, add --tensor-parallel-size N to shard the weights across them. Those three flags resolve most of these crashes. The rest of this guide explains when each one matters and the Kubernetes-specific traps that make the error worse than it looks.

This is a GPU error, not a pod OOMKill

The first thing to get straight: torch.cuda.OutOfMemoryError is not the same failure as an OOMKilled pod. An OOMKill happens when your container exceeds its host RAM limit and the kernel sends Exit Code 137. A CUDA OOM happens entirely inside the GPU’s own memory, which the Linux OOM killer and your pod memory limit know nothing about. You can have gigabytes of free node RAM and still hit this. If you are chasing an Exit Code 137 instead, the diagnosis path is different and covered in debugging OOMKilled pods.

The full error usually looks like this:

torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 2.00 GiB.
GPU 0 has a total capacity of 39.38 GiB of which 224.00 MiB is free.

vLLM does a memory profiling run at startup. It loads the weights, runs a forward pass to measure peak activation memory, then claims the remaining GPU memory (up to gpu_memory_utilization) as a static KV cache pool. The crash happens when weights plus activations already exceed what is free, or when the KV cache target overcommits memory that another process on the GPU is holding.

Confirm what is actually on the GPU

Before changing flags, look at the GPU from inside the running or crash-looping pod. Guessing wastes redeploys.

$ kubectl exec -it deploy/vllm-server -- nvidia-smi

Read two numbers from the output: total GPU memory and current used memory. If used memory is already high before vLLM starts, something else is sharing the card. That is common on shared or MIG-partitioned GPUs, where gpu_memory_utilization of 0.9 (the vLLM default) is a fraction of the full physical card, not of your slice, so the target quietly overcommits.

Then confirm how many GPUs the pod was actually granted:

$ kubectl get pod -l app=vllm -o jsonpath='{.items[0].spec.containers[0].resources.limits}'

If this shows nvidia.com/gpu: "1" but you set --tensor-parallel-size 2, vLLM will try to place shards on GPUs that were never scheduled to the pod, and you get an OOM (or a hang) instead of a clean error. GPUs reach the pod through the Kubernetes device plugin, so the limit you request is the hard ceiling vLLM sees. The tensor parallel size must equal the GPU count in the resource limit.

The four flags that fix it

Each flag trades a different resource. Reach for them in this order.

Flag	Default	What it does
--gpu-memory-utilization	0.9	Fraction of GPU memory vLLM may claim for weights plus KV cache. Lower it to leave headroom on a shared card.
--max-model-len	model max	Caps context length. KV cache size scales with this, so a 128k model capped at 8k frees a large block.
--max-num-seqs	256	Max sequences batched at once. Fewer concurrent requests means a smaller KV cache.
--tensor-parallel-size	1	Shards model weights across N GPUs in the pod. The main lever for models too big for one card.

Start by trimming the KV cache, since that is where most waste lives:

$ vllm serve meta-llama/Llama-3.1-8B-Instruct \
    --gpu-memory-utilization 0.85 \
    --max-model-len 4096 \
    --max-num-seqs 64

If the model weights alone do not fit on one GPU, no amount of KV cache trimming helps. That is when you shard:

$ vllm serve meta-llama/Llama-3.1-70B-Instruct \
    --tensor-parallel-size 4 \
    --gpu-memory-utilization 0.90 \
    --max-model-len 8192

A rough sizing check: a model in FP16 needs about 2 GB of VRAM per billion parameters just for weights, before any KV cache. A 70B model is roughly 140 GB, so it will not fit on a single 80 GB A100 no matter how you tune the cache. Shard it across GPUs or quantize it with --quantization fp8 to halve the weight footprint. If you are still deciding which serving engine to run, the tradeoffs are compared in choosing an LLM serving engine.

Kubernetes-specific traps

The same flags behave differently under Kubernetes than on a bare workstation. Three traps account for most repeat incidents.

1. Shared memory is too small for tensor parallelism

vLLM uses shared memory for inter-GPU communication when --tensor-parallel-size is greater than 1. Containers default /dev/shm to 64 MB, which is far too small, and the symptom is often a confusing OOM or NCCL hang rather than a clear message. Mount a memory-backed volume:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: vllm-server
spec:
  template:
    spec:
      containers:
      - name: vllm
        image: vllm/vllm-openai:latest
        resources:
          limits:
            nvidia.com/gpu: "4"
        volumeMounts:
        - name: dshm
          mountPath: /dev/shm
      volumes:
      - name: dshm
        emptyDir:
          medium: Memory
          sizeLimit: 8Gi

2. Memory fragmentation on long-running pods

If the error says a large amount is reserved but unallocated, the allocator has fragmented the pool. Set the PyTorch allocator to use expandable segments so freed blocks can be reused:

        env:
        - name: PYTORCH_CUDA_ALLOC_CONF
          value: "expandable_segments:True"

3. CUDA graph capture spikes memory

vLLM captures CUDA graphs at startup for lower latency, and the capture itself needs extra memory. If the OOM lands during capture rather than during the profiling run, disable it with --enforce-eager. You lose some throughput but the pod starts:

$ vllm serve meta-llama/Llama-3.1-8B-Instruct \
    --gpu-memory-utilization 0.85 \
    --enforce-eager

Verify the fix

After redeploying, watch the startup logs for the KV cache report. A healthy start prints the number of GPU blocks it allocated:

$ kubectl logs -f deploy/vllm-server | grep -i "kv cache"

If that line appears and the readiness probe passes, the crash is resolved. Then send a real request so a full sequence actually fills the cache, since a crash can still surface under load rather than at boot:

$ kubectl port-forward deploy/vllm-server 8000:8000 &
$ curl http://localhost:8000/v1/completions \
    -H "Content-Type: application/json" \
    -d '{"model": "meta-llama/Llama-3.1-8B-Instruct", "prompt": "ping", "max_tokens": 16}'

Once serving is stable, keep an eye on GPU memory over time. A slow climb points to fragmentation or a KV cache sized too close to the limit, and wiring GPU metrics into your dashboards early makes that obvious before the next crash. See LLM observability on Kubernetes for the metrics worth tracking, and the top LLMOps tools for the wider serving stack.

The vLLM documentation on conserving memory lists the full set of flags and their interactions. For most Kubernetes deployments, though, the pattern is consistent: cap the KV cache first, shard the weights only when a single GPU genuinely cannot hold the model, and give tensor parallelism the shared memory it needs.