VLLM Model Serving Integration: Qwen3 and Gemma3 Support

What is vLLM

vLLM is a high-throughput serving framework for LLM inference. It maximizes memory efficiency through PagedAttention, and can batch multiple requests simultaneously. At XGen, we serve our in-house fine-tuned models with vLLM and built a UI to manage the serving infrastructure from the frontend.

Model Serving Management UI

We needed to be able to manage vLLM servers from the frontend. This required features like model loading, unloading, status monitoring, and configuration changes.

interface VLLMServerConfig {
  modelPath: string;
  tensorParallelSize: number;
  maxModelLen: number;
  gpuMemoryUtilization: number;
  quantization?: 'awq' | 'gptq' | 'none';
  dtype: 'float16' | 'bfloat16' | 'auto';
}

interface VLLMServerStatus {
  isRunning: boolean;
  loadedModel: string | null;
  requestsProcessed: number;
  avgLatency: number;
  gpuMemoryUsed: number;
  gpuMemoryTotal: number;
}

Model Selection UI

Supported models are displayed as cards, each showing the model size, recommended GPU, and performance benchmarks.

const ModelCard: React.FC<{ model: ModelInfo }> = ({ model }) => (
  <div className="border rounded-xl p-5 hover:shadow-md transition-shadow">
    <div className="flex items-center gap-3 mb-3">
      <ModelIcon provider={model.provider} />
      <div>
        <h3 className="font-semibold">{model.name}</h3>
        <p className="text-sm text-gray-500">{model.parameterCount}</p>
      </div>
    </div>
    <div className="space-y-2 text-sm">
      <div className="flex justify-between">
        <span>권장 VRAM</span>
        <span>{model.recommendedVram}GB</span>
      </div>
      <div className="flex justify-between">
        <span>양자화</span>
        <span>{model.supportedQuantization.join(', ')}</span>
      </div>
    </div>
    <Button className="w-full mt-4" onClick={() => deployModel(model.id)}>
      서빙 시작
    </Button>
  </div>
);

Qwen3 and Gemma3 Model-Specific Configuration

Each model has its own optimal serving configuration. Qwen3 supports long contexts, so max_model_len can be set high. Gemma3 is memory-efficient and can run on smaller GPUs.

Serving Status Monitoring Dashboard

We also built a dashboard for real-time monitoring of the vLLM server. GPU memory utilization, tokens generated per second, and request queue length are all displayed as live charts. This data can also inform autoscaling decisions.