Request Cancellation in LLM Serving: GPU Waste and Mitigation

When GPU utilization comes in lower than expected, the usual suspects are batch size and KV cache size settings. But when cancellation rates exceed 10%, tuning those two parameters never addresses the root cause — because the holes left by cancelled requests keep filling the gap.

Orca (OSDI 2022) introduced iteration-level scheduling, allowing requests to enter and leave a batch at iteration boundaries. But the departure condition Orca assumed was "when an EOS token is generated." Clean handling of client disconnects and early stop-sequence hits was never baked in as a default. Most serving engines patched in those cases after the fact, and the traces of that are still visible in the code and bug trackers today.

Three Ways a Request Can End Early

From the server's perspective, there are three paths to early termination.

HTTP stream cancellation — When a user closes the page mid-generation or a frontend timeout fires, the TCP connection drops. The server learns about this when a packet it was trying to send comes back as EPIPE or ECONNRESET. On FastAPI/uvicorn-based servers, if request.is_disconnected() isn't being polled, the moment the disconnect is actually propagated to the engine varies by implementation — and in some cases generation doesn't stop until the request completes fully.

Early stop sequence hit — When patterns like </s> or \n\nHuman: are specified, the server stops generation when those tokens appear. Since this is a server-initiated termination it resolves faster than a disconnect, but it produces the same structural gap between resources the scheduler has already reserved and what actually gets used.

Early max_new_tokens cutoff — If max_new_tokens=512 is set but the actual response finishes at 80 tokens, the KV slots for the remaining 432 tokens were already reserved and then returned unused. While they're reserved, no other request can use that memory.

All three paths cause the same structural problem: a gap between the slots the scheduler has already allocated and the slots actually needed.

The Hole Cancelled Slots Leave in Continuous Batching

vLLM's AsyncLLMEngine.abort(request_id) does not remove a request immediately. Internally it queues the abort request into RequestTracker, and the actual processing happens when the engine's background loop executes the next engine_step() (vLLM docs v0.5.5):

# vllm/engine/async_llm_engine.py (v0 engine)
async def abort(self, request_id: str) -> None:
    if not self.is_running:
        raise AsyncEngineDeadError(...)
    return self._abort(request_id)

def _abort(self, request_id: str) -> None:
    self._request_tracker.abort_request(
        request_id,
        exception=asyncio.CancelledError,
        verbose=self.log_requests,
    )

# actual processing inside engine_step()
new_requests, aborted_requests = (
    self._request_tracker.get_new_and_aborted_requests()
)
if aborted_requests:
    await self._engine_abort(aborted_requests)

At minimum one iteration elapses between the abort() call and the moment the sequence group is actually removed from the scheduler. If the GPU has already started a forward pass on a batch that includes that request, that iteration is wasted.

This delay looks small, but the cost changes when you're close to the maximum batch size. If 10 cancellations occur in a fully packed batch of 100 slots but processing is deferred by one iteration, 10 new incoming requests can't be admitted to the batch during that iteration — they sit in the queue waiting. The core benefit of continuous batching — admitting new requests the instant a slot frees up — breaks down.

The v1 engine in vLLM behaves differently from v0. According to issue #20362, in v1 there are cases where calling abort() alone doesn't guarantee termination unless the generate coroutine itself is cancelled. In v0, calling abort from another coroutine caused the generate coroutine to receive asyncio.CancelledError and exit; in v1, that path changed.

KV Cache Return Delays and Fragmentation

vLLM manages KV cache in fixed-size pages (blocks) based on PagedAttention. When a request is aborted, the blocks it held should be returned to the free block pool — but with prefix caching enabled, this release may not happen immediately.

Prefix caching keeps KV blocks for common prompt prefixes in cache for reuse. If a cancelled request's KV blocks are judged likely to match a future request's prefix, those blocks aren't freed immediately; they stay in an eviction-pending state (the cached pool). The result is that the number of free blocks visible to the scheduler stays lower than what's actually available, and new requests spend more time queued.

A bug was reported in SGLang where KV cache wasn't being returned properly, causing token usage to accumulate until the scheduler could no longer accept new requests (SGLang issue #6778). If you observe TTFT suddenly spiking under heavy load and not recovering, free block exhaustion is a likely cause.

The Reservation-to-Usage Gap from Early Stop Sequences

Computing the gap between KV reservation and actual usage for Llama-3 8B makes the scale concrete.

• Layers: 32
• GQA KV heads: 8
• Head dimension: 128
• Data type: FP16 (2 bytes)
• Block size: 16 tokens

KV size of one block (16 tokens, all 32 layers):

2(K+V) × 8 KV heads × 128 dim × 2 bytes(FP16) × 16 tokens × 32 layers
= 2 × 8 × 128 × 2 × 16 × 32
= 2,097,152 bytes ≈ 2 MB

Setting max_new_tokens=512 reserves 512 / 16 = 32 blocks → 32 × 2 MB = 64 MB reserved.

If that request hits a stop sequence and finishes at 80 tokens, actual blocks used are ⌈80 / 16⌉ = 5 blocks → 10 MB used. 54 MB and 27 blocks were reserved and returned unused.

With tens of requests running concurrently, the sum of these gaps reaches several GB. The scheduler sees this reserved memory and decides there's no room to admit new requests, so the queue grows. This is why bumping max_num_seqs or gpu_memory_utilization has no effect in these situations.

Throughput Impact by Cancellation Rate

Measuring three scenarios with different cancellation rates on the same hardware using a ShareGPT input/output length distribution yields the following pattern.

(Llama-3 8B, A100 80GB, max_num_seqs=256, synthetic benchmark on vLLM)

At 10% cancellation rate, throughput drops 12% and SM utilization falls 9 percentage points. At 30%, throughput loss reaches 29%. Notable: the SM utilization drop is larger than the throughput loss. Forward passes for cancelled slots already ran on the GPU but their outputs were discarded, so SM cycles were spent producing no valid output.

A 30% cancellation rate might sound rare. But chatbot services using streaming responses see users frequently typing the next question mid-generation, closing tabs, or hitting regenerate. If you've never checked the finish_reason=abort rate in your production logs, now is a good time.

Operational Responses and Measurement

Early blocking at the gateway is the most effective approach — and should come before any internal engine optimization. When nginx or an API gateway detects a client disconnect, propagating the abort to the upstream immediately prevents the engine from including that request in the next iteration. If you're running FastAPI in front of vLLM, the streaming endpoint should periodically call await request.is_disconnected() and call abort when it's true.

Cancellation handling differences by framework:

None of the three implementations support immediate exit within an iteration. This isn't an implementation oversight — the cost of interrupting an in-flight GPU kernel is higher than waiting for the iteration to complete and handling it then.

Three metrics that expose cancellation cost — adding these to your monitoring stack lets you isolate the cause quickly.

• abort_rate: count of requests with finish_reason=abort / total requests. vLLM exposes this via RequestOutput.finish_reason.
• completion_ratio: actual tokens generated / max_new_tokens. A value close to 1.0 means requests ran to their full reservation.
• useful_token_ratio: total tokens generated by successfully completed requests / sum of max_new_tokens across all requests. This is the batch-aggregate version of completion_ratio.

Example middleware to collect useful_token_ratio with Prometheus:

from prometheus_client import Counter, Gauge

_useful = Counter("llm_useful_tokens_total", "Tokens from completed requests")
_reserved = Counter("llm_reserved_tokens_total", "Sum of max_new_tokens across all requests")
useful_token_ratio = Gauge("llm_useful_token_ratio", "Useful tokens / reserved tokens")

def record_completion(output):
    generated = len(output.token_ids)
    reserved = output.max_new_tokens

    if output.finish_reason != "abort":
        _useful.inc(generated)
    _reserved.inc(reserved)

    ratio = _useful._value.get() / max(_reserved._value.get(), 1)
    useful_token_ratio.set(ratio)

When useful_token_ratio drops below 0.5, half the KV cache is being reserved but never actually used. Tracking this metric alongside TTFT and throughput lets you isolate how much cancellation cost is contributing to batching inefficiency, separate from other causes. Without this instrumentation, you typically discover that cancellation traffic was the culprit only after trying and failing to fix it by increasing batch size or expanding KV cache.