Building a Vast.ai GPU Instance Management System

Why We Needed a GPU Cloud

GPU is a hard requirement for AI model training and serving. When our on-prem GPUs weren't enough, we turned to GPU cloud marketplaces like Vast.ai. This post documents how we built a system inside XGen to manage Vast.ai instances directly.

Vast.ai API Integration Architecture

Vast.ai exposes a REST API. We route requests through the XGen backend as a proxy so the frontend can manage instances without talking to Vast.ai directly.

[프론트엔드] → [XGen 백엔드 API] → [Vast.ai API]

GPU Offer Search UI

We built a UI for browsing available GPU instances, with filters for GPU type, price, VRAM, and network speed.

interface GpuOfferFilter {
  gpuName?: string;           // 'RTX 4090', 'A100', 'H100'
  minVram?: number;           // GB
  maxPrice?: number;          // $/hr
  minUploadSpeed?: number;    // Mbps
  minDownloadSpeed?: number;  // Mbps
  minGpuCount?: number;
  region?: string;
}

interface GpuOffer {
  id: string;
  gpuName: string;
  gpuCount: number;
  vram: number;
  pricePerHour: number;
  uploadSpeed: number;
  downloadSpeed: number;
  reliability: number;
  location: string;
}

Instance Management Modal

Once an instance is rented, a management modal lets you check its status and control it.

const InstanceManagementModal: React.FC<{ instance: GpuInstance }> = ({ instance }) => (
  <Modal title="인스턴스 관리">
    <div className="space-y-6">
      <StatusSection>
        <StatusBadge status={instance.status} />
        <MetricRow label="GPU" value={`${instance.gpuName} x${instance.gpuCount}`} />
        <MetricRow label="가동 시간" value={formatDuration(instance.uptime)} />
        <MetricRow label="누적 비용" value={`$${instance.totalCost.toFixed(2)}`} />
      </StatusSection>
      
      <HealthCheckSection instanceId={instance.id} />
      
      <ActionButtons>
        <Button variant="outline" onClick={() => restartInstance(instance.id)}>
          재시작
        </Button>
        <Button variant="destructive" onClick={() => destroyInstance(instance.id)}>
          인스턴스 종료
        </Button>
      </ActionButtons>
    </div>
  </Modal>
);

Health Check System

We implemented periodic health checks to verify that the vLLM server deployed on each instance is functioning correctly. The checks cover ping latency, GPU temperature, and memory utilization, and send an alert when anything looks unhealthy.

Cost Tracking Dashboard

Because hourly costs accumulate quickly, tracking spend is critical. The dashboard displays daily, weekly, and monthly cost trends as charts and supports budget alerts. Somewhat surprisingly, this feature got the best reception from users.

Adding the Network Speed Filter

The initial version had no network speed filter. After repeatedly getting stuck on slow instances while downloading large models, we added one. It seems like a minor detail, but in practice it turned out to be one of the most important filters.