vLLM From Zero to OpenAI-Compatible API Serving

vLLM is an LLM inference server that manages GPU memory efficiently using the PagedAttention algorithm. A single vllm serve command immediately exposes an OpenAI-compatible REST API, so you can switch to local LLM serving by changing only the endpoint — no changes to existing OpenAI SDK-based code required. This post walks through the process step by step, from installation to receiving your first response.

---

What is vLLM — and Why Use It

The core of vLLM is PagedAttention (Kwon et al., 2023). It applies OS virtual memory and paging concepts to KV cache management, solving the problem where traditional LLM serving systems waste 60–80% of their KV cache. PagedAttention reduces that waste to under 4% and achieves up to 24× higher throughput than HuggingFace Transformers on the same GPU.

vLLM is a good fit in these situations:

• API servers with high concurrency: batches requests from multiple users to maximize GPU utilization.
• Existing OpenAI API-based codebases: change only base_url — no client code changes needed.
• RAG pipelines: embedding generation and text generation can run on the same server, simplifying infrastructure.
• Mixed serving of chat, embedding, and reranking models on a single server: supports multi-task configurations.

---

Setting Up the Environment (CUDA, Python, pip/Docker)

Minimum Requirements

• Python 3.10–3.13
• Linux (WSL2 recommended on Windows)
• NVIDIA GPU with compute capability 7.5 or higher (T4, RTX 20xx, A100, L4, H100, etc.)

pip Installation

The default installation uses pre-built wheels targeting CUDA 12.9.

pip install vllm

For other CUDA versions, specify the appropriate wheel index via --extra-index-url.

Docker Image

Using the official image is the simplest approach in a Docker environment. vllm/vllm-openai:latest is NVIDIA CUDA-based.

docker run --runtime nvidia --gpus all \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    -p 8000:8000 \
    vllm/vllm-openai:latest \
    --model Qwen/Qwen3-4B

For AMD ROCm environments, substitute vllm/vllm-openai-rocm:latest.

Minimum GPU Memory Requirements by Model Size

In BF16, weight size is roughly parameters × 2 bytes. Accounting for KV cache, you'll need at least the figures below in practice.

---

Serving Your First Model — Basic vllm serve Usage

Here is the basic command to bring up a server using Qwen3-4B as an example.

vllm serve Qwen/Qwen3-4B \
  --dtype bfloat16 \
  --max-model-len 32768 \
  --port 8000

When the server starts successfully, the following lines appear at the end of the log.

INFO:     Application startup complete.
INFO:     Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)

Key Arguments

--dtype

The data type for model weights. bfloat16 is the recommended default for modern GPUs (A100, H100, H200). Older GPUs that don't support bfloat16, such as the T4, should use float16. Setting this to auto lets vLLM choose automatically based on the GPU.

--max-model-len

The maximum token length the server will process. Larger values increase KV cache reservation and VRAM usage. Keeping this at or below what your actual usage patterns require — rather than the model's maximum — saves memory. Qwen3-4B supports up to 128K, but 32K is sufficient for most practical tasks.

--gpu-memory-utilization

The fraction of GPU VRAM vLLM is allowed to use. Defaults to 0.9. When running multiple processes on the same GPU, lower this value so VRAM is shared appropriately.

--served-model-name

An alias for the model name used in API requests. Without this, the model path is used as the name. Useful when serving from a local path but wanting clients to reference a short, clean name.

vllm serve /data/models/Qwen3-4B \
  --served-model-name qwen3-4b \
  --dtype bfloat16 \
  --max-model-len 32768 \
  --gpu-memory-utilization 0.9

In a multi-GPU setup, set --tensor-parallel-size to the number of GPUs. For two GPUs, add --tensor-parallel-size 2.

---

Calling the OpenAI-Compatible API

The vLLM server exposes the /v1/chat/completions endpoint in the same format as the OpenAI API. Here are examples using three different approaches.

curl

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen/Qwen3-4B",
    "messages": [
      {"role": "user", "content": "vLLM을 한 문장으로 설명해줘"}
    ]
  }'

OpenAI Python SDK

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="dummy"  # no authentication in the default configuration; any value works
)

response = client.chat.completions.create(
    model="Qwen/Qwen3-4B",
    messages=[{"role": "user", "content": "vLLM을 한 문장으로 설명해줘"}]
)
print(response.choices[0].message.content)

LangChain OpenAI Wrapper

from langchain_openai import ChatOpenAI

llm = ChatOpenAI(
    model="Qwen/Qwen3-4B",
    openai_api_base="http://localhost:8000/v1",
    openai_api_key="dummy"
)

response = llm.invoke("vLLM을 한 문장으로 설명해줘")
print(response.content)

All three approaches work as long as the server address and model name are correct. For existing OpenAI API code, replacing just base_url (or openai_api_base) is all that's needed.

---

Serving an Embedding Model Alongside

vLLM serves embedding models with the same vllm serve command used for chat LLMs. Dedicated embedding models are detected automatically, but when using an LLM for embedding purposes, pass --task embed explicitly.

vllm serve intfloat/multilingual-e5-large-instruct \
  --task embed \
  --port 8001

Once the server is up, request vectors from the /v1/embeddings endpoint.

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8001/v1", api_key="dummy")

response = client.embeddings.create(
    model="intfloat/multilingual-e5-large-instruct",
    input=["RAG 파이프라인 첫 번째 문장", "비교할 두 번째 문장"]
)

print(response.data[0].embedding[:5])  # first 5 dimensions of the first sentence's embedding

Connecting an embedding server in LangChain works the same way.

from langchain_openai import OpenAIEmbeddings

embeddings = OpenAIEmbeddings(
    model="intfloat/multilingual-e5-large-instruct",
    openai_api_base="http://localhost:8001/v1",
    openai_api_key="dummy"
)

vector = embeddings.embed_query("검색할 문장을 넣는다")

When running a chat LLM and an embedding model on the same GPU, use different ports and make sure the sum of --gpu-memory-utilization across both processes does not exceed 1.0. For example, 0.75 for the LLM and 0.15 for the embedding model gives a total of 0.90 per GPU.

---

Common Mistakes and Troubleshooting

1. OOM (Out of Memory)

torch.cuda.OutOfMemoryError: CUDA out of memory.

There are two main causes. First, --max-model-len is set too high, causing KV cache reservation to encroach on the space needed for weights. Second, when loading multiple models onto one GPU, the combined --gpu-memory-utilization values exceed 1.0.

Resolution order:

1. Reduce --max-model-len (e.g., 131072 → 32768).
2. Reduce --gpu-memory-utilization (e.g., 0.9 → 0.8).
3. Add --enforce-eager to reduce the CUDA graph memory budget.

2. Missing chat_template Error

jinja2.exceptions.TemplateError: chat_template is not defined

This occurs when the model's tokenizer_config.json has no chat_template. Specify a Jinja2 template file path with --chat-template, or re-download the latest tokenizer config from HuggingFace.

vllm serve my-model \
  --chat-template /path/to/chat_template.jinja

3. Tokenizer Mismatch

ValueError: The model's tokenizer does not match the model's vocabulary.

This happens when the model weights and tokenizer are from different versions. Specify the correct tokenizer path separately using --tokenizer, or download both the model and tokenizer from the same checkpoint.

4. HuggingFace Gated Model Access Failure

OSError: You are trying to access a gated repo.

Some models such as Llama and Gemma require accepting a HuggingFace license and providing an API token. Set the token as an environment variable.

export HF_TOKEN=hf_xxxxxxxxxxxxx
vllm serve meta-llama/Llama-3.1-8B-Instruct

5. Port Conflict

OSError: [Errno 98] Address already in use

Another process is already bound to the same port. Use lsof -i :8000 to identify and kill the occupying process, or specify a different port with --port.

---

What's Next — Multi-Model, Quantization, and Production Serving

Once you're comfortable with the basic serving workflow, here are the natural next topics.

Argument deep-dive: vLLM 0.21.0 Serving Guide — vllm serve Arguments by Model Type covers per-type serving configurations and argument combinations for thinking models, tool calling, OCR, and multimodal models.

Multi-model deployment: vLLM 0.21.0 Serving Guide Part 2 — Running Three Models Simultaneously on Two H200s covers VRAM budget planning when serving LLM + embedding + OCR models on the same server, CUDA_VISIBLE_DEVICES isolation strategies, and CUDA Graph troubleshooting.

Topics planned for upcoming posts in this series:

• Reducing GPU memory requirements with AWQ, GPTQ, and INT4 quantization
• Production serving configurations including load balancing and Prometheus monitoring
• Multi-GPU deployments combining pipeline parallelism and tensor parallelism