Serving Custom LLMs with vLLM and VastAI GPU Cloud

The Case for GPU Serving

For serving the custom LLM in PlateeRAG-1, we used vLLM with VastAI GPU cloud.

Serving Architecture

graph TD
    A[XGen Frontend] --> B[FastAPI Gateway]
    B --> C{모델 라우팅}
    C -->|OpenAI| D[OpenAI API]
    C -->|Custom| E[vLLM Server]

    E --> F[VastAI GPU]
    F --> G[A100 80GB]

    E --> H[OpenAI 호환 API]
    H --> B

vLLM Server Configuration

# vLLM 서버 시작
python -m vllm.entrypoints.openai.api_server \
    --model "custom-model-path" \
    --host 0.0.0.0 \
    --port 8000 \
    --tensor-parallel-size 1 \
    --gpu-memory-utilization 0.9

Renting GPUs on VastAI

flowchart TD
    A[VastAI 마켓플레이스] --> B{GPU 선택}
    B --> C[A100 80GB - $1.2/hr]
    B --> D[RTX 4090 - $0.4/hr]
    B --> E[H100 - $2.5/hr]
    C --> F[Docker 이미지 배포]
    D --> F
    E --> F
    F --> G[SSH 접속]
    G --> H[vLLM 서버 시작]
    H --> I[API 엔드포인트]

Why vLLM

Cost Optimization

1. Preemptible instances: 50%+ cost reduction, with interruption risk
2. Model quantization: 4-bit quantization to reduce GPU memory footprint
3. Batch inference: Maximize throughput by batching requests
4. Auto-scaling: Adjust instance count based on traffic

This setup was used in the XGen project to serve both the custom model and OpenAI models simultaneously.