Serving LLMs in Production with vLLM

When you need to serve an LLM yourself, vLLM is typically the first option that comes to mind. You might think loading a model with Hugging Face transformers and calling .generate() is good enough — but once you need to handle concurrent requests from multiple users in a production environment, that approach falls short. vLLM is an LLM inference engine designed to achieve both high throughput and low latency simultaneously, and it ships with an OpenAI-compatible API server out of the box.

This post covers vLLM's core concepts, common mistakes to avoid, and tuning tips you can actually use in production.

---

Why vLLM Is Fast

PagedAttention

The secret to vLLM's performance is PagedAttention. During inference, a Transformer model needs a KV cache to store the keys and values for all previously processed tokens. The conventional approach pre-allocates a contiguous block of GPU memory equal to the maximum sequence length for each request — memory that stays reserved and wasted even when it isn't fully used.

PagedAttention takes inspiration from OS virtual memory paging: it splits the KV cache into fixed-size blocks that can be stored in non-contiguous memory. This dramatically reduces memory fragmentation and allows far more concurrent requests to fit in the same VRAM. In practice, results show 60–80% reduction in KV cache memory waste compared to the conventional approach.

Continuous Batching

Traditional static batching waits for every request in a batch to complete before starting the next batch. vLLM's Continuous Batching inserts new requests into the batch the moment an existing request finishes. This minimizes GPU idle time and significantly increases overall throughput.

---

Installation and Basic Usage

pip install vllm

Starting an OpenAI-Compatible API Server

vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --host 0.0.0.0 \
  --port 8000

Once the server is up, you can use it directly with the OpenAI SDK or curl.

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="dummy",  # vLLM does not validate API keys by default
)

response = client.chat.completions.create(
    model="meta-llama/Llama-3.1-8B-Instruct",
    messages=[{"role": "user", "content": "안녕하세요!"}],
)
print(response.choices[0].message.content)

Using vLLM Directly from Python

from vllm import LLM, SamplingParams

llm = LLM(model="meta-llama/Llama-3.1-8B-Instruct")

prompts = [
    "한국의 수도는 어디인가요?",
    "파이썬의 장점을 설명해주세요.",
]

sampling_params = SamplingParams(temperature=0.7, max_tokens=512)
outputs = llm.generate(prompts, sampling_params)

for output in outputs:
    print(output.outputs[0].text)

---

Key Configuration Options

--gpu-memory-utilization (default: 0.9)

At startup, vLLM reserves a fixed fraction of GPU memory for the KV cache. The default is 0.9 (90%). If you're sharing the GPU with other processes, lower this value. If you're hitting OOM errors, try 0.85.

vllm serve ... --gpu-memory-utilization 0.85

Note: Setting this too low reduces KV cache space, which decreases the number of requests you can handle concurrently.

--max-model-len

Caps the maximum context length supported by the model. Even if a model supports 128K context, there's no reason to allocate that much if your actual requests are much shorter. Reducing this value saves KV cache memory.

vllm serve ... --max-model-len 8192

--tensor-parallel-size (TP)

Enables tensor parallelism, which shards model weights across multiple GPUs. Use this when a single GPU isn't enough to hold the model. Typically set this to match the number of GPUs on a single node.

# Distribute the model across 4 GPUs
vllm serve ... --tensor-parallel-size 4

--pipeline-parallel-size (PP)

Enables pipeline parallelism, which distributes model layers sequentially across multiple GPUs or nodes. Primarily used in multi-node setups; typically set to match the number of nodes.

# 4 GPUs × 2 nodes = 8 GPUs total
vllm serve ... --tensor-parallel-size 4 --pipeline-parallel-size 2

When combining TP and PP: total GPU count = tensor-parallel-size × pipeline-parallel-size

---

Performance Optimization Tips

1. Enable Automatic Prefix Caching (APC)

If you're running a chatbot service where all requests share a common system prompt, prefix caching can have a large impact. vLLM caches the KV values for identical prefixes (e.g., the system prompt) and reuses them for subsequent requests. This is enabled by default in vLLM V1.

# Explicitly enable in vLLM V0
vllm serve ... --enable-prefix-caching

To verify it's working, monitor prefix_cache_hit_rate in the server logs. A higher cache hit rate translates directly to lower TTFT (Time To First Token).

# Pin the system prompt at the front to maximize prefix caching benefits
messages = [
    {"role": "system", "content": "당신은 친절한 한국어 AI 어시스턴트입니다."},  # this gets cached
    {"role": "user", "content": user_input},
]

2. Reduce Memory Footprint with Quantization

Quantizing the model significantly reduces VRAM usage and frees up more space for the KV cache.

AWQ / GPTQ (Weight Quantization)

# Use a pre-quantized AWQ model (available on Hugging Face)
vllm serve TheBloke/Llama-2-7B-AWQ \
  --quantization awq

FP8 KV Cache

Storing the KV cache in FP8 cuts memory usage in half compared to FP16. Accuracy loss is minimal, but memory efficiency improves significantly.

vllm serve ... --kv-cache-dtype fp8

Tip: Prefer pre-quantized checkpoints provided by the model vendor when available. On-the-fly quantization via --quantization increases startup time and consumes additional CPU/GPU resources.

3. Speculative Decoding

Speculative decoding uses a small draft model (or n-gram predictor) to speculatively generate several tokens ahead, which the larger target model then verifies in a single forward pass. This is especially effective for short, predictable output patterns.

vllm serve meta-llama/Llama-3.1-70B-Instruct \
  --speculative-model meta-llama/Llama-3.2-1B-Instruct \
  --num-speculative-tokens 5

Note: Speculative Decoding can be used alongside Prefix Caching, but in some versions the cache hit rate statistics may not be recorded accurately.

4. Chunked Prefill

Processing a long prompt in one shot blocks decoding for other requests during that time. Chunked Prefill breaks long prefill work into smaller chunks and interleaves them with decoding steps, reducing TTFT variance.

vllm serve ... --enable-chunked-prefill

5. Optimization Level (-O flag)

vLLM provides optimization levels that control the degree of CUDA graph capture and kernel fusion.

vllm serve ... -O 3  # maximum performance, longer startup

---

Common Mistakes

❌ Mistake 1: Not Enough CPU Cores

vLLM internally spawns separate processes for the API server, the engine core, and one worker per GPU. You need at least 2 + number of GPUs physical cores. In virtualized environments or containers with too few allocated CPU cores, you'll see slow responses or timeouts.

# Explicitly allocate CPU cores when running with Docker
docker run --gpus all --cpus="8" ...

❌ Mistake 2: Unnecessary Multi-GPU Initialization with TP=1

Even when --tensor-parallel-size 1 is set, vLLM may detect and attempt to initialize all GPUs on the system. This can cause unnecessary data transfers between GPUs on different NUMA nodes, actually hurting performance.

# Explicitly specify which GPU(s) to use
CUDA_VISIBLE_DEVICES=0 vllm serve ...

❌ Mistake 3: Trusting Synthetic Benchmarks

Randomly generated prompts or overly simple benchmarks are poor predictors of real-world performance. Prefix caching behavior, KV cache eviction patterns, and more can look completely different from your actual traffic. Always load-test with data that resembles your real production requests.

# Use vLLM's built-in benchmarking tool
python -m vllm.entrypoints.benchmark_serving \
  --dataset-name sharegpt \
  --model meta-llama/Llama-3.1-8B-Instruct

❌ Mistake 4: Setting max_tokens Too High

Setting max_tokens to an arbitrarily large value causes KV cache blocks to be held for a long time, reducing the number of requests that can be processed concurrently. Determine the actual maximum output length your service needs and set a reasonable limit.

# Bad
SamplingParams(max_tokens=32768)  # wasteful if you rarely generate that much

# Good — constrain to what your service actually needs
SamplingParams(max_tokens=1024)

❌ Mistake 5: Scaling GPU Count Without a Replica Strategy

Adding more GPUs and cranking up TP on a single instance isn't always the right move. Think about your replica strategy instead.

Recommended approach: Start with the minimum number of GPUs needed to fit the model and run as many replicas as possible. Measure performance at each concurrency level. Then gradually reduce replica count and increase GPU count per instance to find the optimal operating point.

For example, with an 8B model and 8× A100s:

• Poor choice: TP=8, 1 replica
• Good choice: TP=1, 8 replicas (when optimizing for throughput)

---

Monitoring

Prometheus + Grafana

vLLM exposes Prometheus metrics out of the box.

vllm serve ... --enable-metrics

Navigate to http://localhost:8000/metrics to see the key metrics:

Monitoring KV Cache Utilization

If vllm:gpu_cache_usage_perc is consistently near 100%, it's a sign that requests are piling up in the queue. At that point, you should either add more replicas or reduce --max-model-len to improve KV cache efficiency.

---

Production Deployment Checklist

✅ Limit context length to match real service needs with --max-model-len
✅ Account for memory sharing with other processes via --gpu-memory-utilization
✅ Specify target GPUs explicitly with CUDA_VISIBLE_DEVICES
✅ Allocate sufficient CPU cores (at least 2 + number of GPUs)
✅ Enable --enable-prefix-caching if using a shared system prompt
✅ Configure Prometheus metrics collection
✅ Complete load testing with traffic patterns that reflect real usage
✅ Evaluate quantized models or FP8 KV cache
✅ Configure health check endpoint (/health)

---

Closing Thoughts

vLLM is an actively developed project, and each release brings significant new features and changes. On top of its solid foundation of PagedAttention and Continuous Batching, features like Prefix Caching, Speculative Decoding, and Disaggregated Prefill continue to be added. Make it a habit to check the official documentation and GitHub release notes regularly.

The most important thing is to benchmark with your actual workload. Numbers measured on synthetic data can differ substantially from production behavior, so optimize your configuration based on real service traffic patterns.