As long-context requests grow more common, more teams are adopting KV cache eviction to reduce memory usage and increase throughput — a reasonable call on its face. What many teams don't know until they measure it, though, is that the choice of eviction policy can push perplexity from the tens into the thousands on the same KV budget. The variable that drives that gap is a single one: whether the Attention Sink tokens were evicted or preserved.
Asymmetry in the Attention Distribution
Visualizing transformer attention scores layer by layer reveals a striking pattern. The first one to four tokens — typically the BOS token or the first newline character — capture an overwhelming share of attention across every layer. According to the analysis in Xiao et al. (2023), on a 4,096-token sequence the attention score directed at the first token exceeds half the total sum in most layers.
During training, the model anchored these positions as reference points (sinks) for its processing flow. Regardless of text content, these slots always play that role.
The softmax formulation is what produces this behavior. The attention score that token $i$ assigns to token $j$ in layer $l$, head $h$ is:
$$\alpha_{ij}^{(l,h)} = \frac{\exp(q_i^T k_j / \sqrt{d_k})}{\sum_{j'} \exp(q_i^T k_{j'} / \sqrt{d_k})}$$
The denominator always normalizes to 1.0. Once the model develops a tendency to concentrate attention on a particular slot, the mere presence of that slot stabilizes the rest of the distribution as a reference point. Remove the sink token and that anchor disappears, leaving the attention distribution over the remaining tokens unstable.
How Much Does Perplexity Shift When the Sink Is Evicted?
The StreamingLLM paper quantifies this directly. On Llama-2-13B with a sliding window that retains only the most recent 1,024 tokens — the simplest approach, which just drops old KV entries with no sink preservation — perplexity on a 20K-token sequence spikes to 5,158. On the same budget, pinning the KV entries for the first four tokens brings perplexity down to 5.40. The oracle baseline — recomputing the full sliding window from scratch at every step — also lands at 5.40.
Adding just four sink tokens produces a perplexity difference of over 1,000×.
That gap is comparable to, or larger than, the quality loss from dropping one quantization level (W8 → W4). Teams make quantization decisions explicitly; eviction policy, by contrast, often gets left at its default and the problem only surfaces later.
The failure mode from losing the sink is also distinctive: repeated phrases, and context breaks where names or conditions mentioned early in the prompt are no longer referenced. Sensitivity to the number of sink slots, per Table 2 of the paper, shows incomplete recovery with one or two slots, convergence at four, and no meaningful improvement beyond eight.
How Three Strategies Handle the Sink
StreamingLLM, H2O, and SnapKV take fundamentally different approaches to the sink.
StreamingLLM is explicit. The KV entries for the first N tokens (default: 4) are hard-pinned, and the rest of the window is filled with the most recent tokens. The implementation is simple and there is no risk of sink eviction. The downside is that nothing outside the sink tokens and the recent-N window is retained — important content appearing in the middle of a long document simply disappears. This approach suits workloads where what matters is the flow of the current moment: real-time streaming and continuous dialogue.
H2O preserves the sink indirectly. For each head, it dynamically selects tokens with the highest cumulative attention score (the Heavy Hitter score). Because the first token participates in every attention computation, its cumulative score naturally accumulates near the top of the ranking — the sink survives on its own. The advantage over StreamingLLM is that H2O also retains intermediate tokens relevant to the query. Per the H2O paper, on OPT-30B at 20% KV budget it achieves COPA 84.0 (vs. full-cache 85.0), PiQA 78.45 (vs. 78.51), and Winogrande 69.06 (vs. 70.24) — nearly lossless. By contrast, a Local strategy that retains only recent tokens on the same budget is reported to drop up to 37% in some scenarios.
H2O's weak point is at extremely low KV budgets. When the cumulative-score competition intensifies, the sink can be knocked out of the Heavy Hitter ranking on certain heads. In that regime, the risk of sink eviction outweighs the benefits of dynamic selection.
SnapKV finalizes KV selection before generation begins. It examines what each head attends to in the end of the prompt (the observation window) and uses that pattern to decide which positions to retain. Sink tokens are already identified as high-score positions during the observation phase. The advantage is query-aware selection. The SnapKV paper reports a 3.6× speedup and 8.2× memory efficiency on 16K-token inputs, with a single A100-80GB GPU handling 380K context. The limitation is that selection is committed after a single pass over the prompt, so it cannot adapt when tokens that become important later during a long generation. It fits RAG patterns and long-document QA well.
Strategy Comparison
| Strategy | Sink Preservation | Mid-sequence Token Retention | Dynamic Updates | Best-fit Workloads |
|---|---|---|---|---|
| StreamingLLM | Explicit, pinned (4 tokens) | None (recent N only) | Not needed | Real-time streaming, long conversations |
| H2O | Implicit (top cumulative score) | Heavy Hitter–based | Every token | Medium-length, information extraction |
| SnapKV | Pre-determined via prompt observation | Query-based selection | None | Long-document QA, RAG |
| Random / FIFO | None | Random | — | Not recommended |
At 50% KV budget, both H2O and SnapKV are stable. As the budget drops below 25%, the risk of H2O evicting the sink grows, and in that regime StreamingLLM's explicit pinning is the safer choice.
Sink Pinning Composes with Any Policy
The cost of pinning the sink is four slots. Regardless of which strategy you use, reserve four slots from the total KV budget upfront and run Heavy Hitter selection or observation-based selection on the remainder. StreamingLLM's sink-pinning idea is most valuable not as a standalone strategy, but as a plugin layered on top of other policies.
There has been ongoing discussion about integrating H2O into vLLM (vllm-project/vllm#3532), and in practice most production deployments use custom implementations. Any implementation that does not include sink preservation deserves a close look. It may appear stable at 50% KV budget, then collapse in quality when context grows longer or the budget is cut further — a failure that is easy to miss without an A/B test.