1 ↩︎

1.0.1 Introduction↩︎

Transformer-based large language models (LLMs) have achieved remarkable performance across a wide range of natural language understanding and generation tasks [1]–[3]. However, their inference-time memory consumption remains a significant challenge, particularly for application requiring long-context information. A major source of this overhead is the Key-Value (KV) cache, which stores the key and value vectors corresponding to previously generated tokens. These vectors are used to compute attention outputs without recomputing intermediate states, but their memory footprint grows linearly with sequence length. For instance, as highlighted by [4], LLM-8B, which generats a sequence of 2M tokens, can consume up to 256GB of GPU memory while storing the KV cache.

Typically, KV cache is populated in two stages: during the prefill phase, where a long-context input is encoded in parallel and all token-level KV vectors are stored; and during the generation phase, where tokens are produced autoregressively and KV entries are appended one step at a time. In particular, the prefill phase dominates memory usage, especially when handling large input contexts. This has motivated extensive research [5] into KV compression techniques that reduce the number of cached entries while preserving output quality. Existing KV compression methods prioritize tokens based on attention scores. For instance, SnapKV [6] and H2O [7] estimate token importance by accumulating attention weights across heads and layers. While such heuristics are computationally cheap, they only capture query-key alignment and neglect the downstream contribution of the value vectors. Moreover, these methods mostly overlook the fact that tokens with low attention scores can still carry rich semantic information through their value vectors. Recent work formalizes this mismatch through the concept of eviction loss [4], which quantifies the performance degradation from evicting specific KV entries. Critically, eviction loss is not always correlated with attention score (c.f. Figure 1), highlighting the need for value-aware selection strategies.

In this work, we introduce CurDKV(CUR Decomposition for KV Compression), a principled and efficient KV compression method that circumvents these limitations by leveraging the structure of the original value and key matrices rather than relying on attention scores. CurDKV assigns importance to each key and value based on its leverage score, computed as the squared \(\ell_2\) norm of the corresponding left-singular vector of the matrix in consideration. These scores reflect each token’s contribution to the attention output and naturally align with the goal of minimizing eviction loss. To improve scalability, we also introduce a fast approximation based on random Gaussian projections, where the value and key matrices are projected into a lower-dimensional subspace before computing leverage scores. This maintains relative importance across tokens while significantly reducing computational cost. We further cast the KV selection problem as a CUR decomposition task on the attention matrix product, selecting key and value vectors that best reconstruct either the attention matrix \(QK^\top\) or the final output \(\mathrm{softmax}(QK^\top)V\). Unlike prior work, our formulation directly targets the fidelity of the attention output, leading to a more semantically consistent compression strategy. As shown in Figure 2, CurDKVachieves significantly lower reconstruction loss on both the \(QK^\top\) matrix and the final \(\mathrm{softmax}(QK^\top)V\) product, demonstrating improved preservation of contextual semantics post-compression.

We evaluate CurDKVon two popular long-context benchmarks – LongBench [8] and Ruler [9], spanning 24 tasks with LLaMA-3.1-8B-Instruct [2] and Mistral-7B-Instruct [3]. Our experiments demonstrate that CurDKVconsistently outperforms existing attention-based KV compression baselines such as SnapKV [6], ChunkKV [10], and Streaming LLM [11]. In particular, CurDKVachieves significantly lower degradation in generation accuracy at aggressive compression ratios (e.g., \(90\%\) cache reduction). With LLaMA, CurDKVsurpasses SnapKV by up to \(9.6\%\) in average task score, and achieves higher fidelity (similarity as the full cache model) across all types of long-context-dependent tasks. Similar gains are observed with Mistral, where CurDKVmaintains over \(95\%\) average fidelity even under constrained memory budgets. These results validate that our value-centric, CUR-based strategy more effectively preserves semantic attention outputs than methods relying solely on query-key attention scores.¹

1.0.2 Related Work↩︎

1.0.2.0.0.1 KV Cache compression via attention heuristics and top-k eviction.

A widely used class of KV compression methods rank cached tokens by heuristics and retain the top-\(k\) scoring entries. H2O [7] uses observed attention scores from specific query-key interactions to estimate per-token importance and prunes less relevant tokens online. However, it requires access to attention weights, which breaks compatibility with efficient kernels like FlashAttention [12]. SnapKV [6] computes cumulative attention scores and selects the highest-ranked tokens, assuming that frequently attended tokens are most informative. Scissorhands [13] further introduces temporal windows to detect consistently attended tokens. PyramidKV [14] proposes a hierarchical scoring scheme across layers and heads, combining multi-level top-\(k\) selection with scaling-aware normalization. ChunkKV [10] compresses the cache by segmenting it into chunks and retaining only top-ranked entries per chunk to ensure temporal coverage. While these approaches are efficient and easy to deploy, they focus on approximating \(QK^\top\) and neglect the downstream contribution of value vectors in \(\mathrm{softmax}(QK^\top)V\), the final attention output propagated to the subsequent layers. Beyond static heuristics, adaptive methods attempt to allocate compression budgets more intelligently across heads or layers. AdaKV [4] computes attention statistics such as entropy and average weights to estimate the relative importance of each head and distributes the token retention budget accordingly. While more flexible, these approaches still rely on attention-derived metrics and do not explicitly account for the downstream impact of compression on the attention output.

1.0.2.0.0.2 Compatibility with efficient attention implementations.

A critical practical consideration is compatibility with modern inference backends such as FlashAttention [12], which avoids materializing the full attention matrix by fusing softmax and matrix multiplications. Methods such as H2O [7], which rely on observed attention weights, are incompatible with FlashAttention and similar kernels that trade explicit attention computation for efficiency. Another major limitation with the majority of the existing KV compression methods is their inability to work with Group Query Attention (GQA). GQA [15] has been widely adopted in recent LLM architectures such as LLaMA [2] and Mistral [3] due to its ability to reduce the memory footprint of the KV cache by sharing keys and values across multiple heads. However, most existing KV cache compression techniques, including SnapKV and PyramidKV, do not support GQA natively. These methods redundantly replicate the compressed KV cache across heads within a group, thus forfeiting GQA’s inherent efficiency. To address this limitation, [4] proposed a GQA-compatible eviction framework that computes average attention weights within each group to identify important tokens. This enhancement enables state-of-the-art KV compression methods to function effectively in GQA-enabled models and achieve significant cache size reductions while preserving generation quality.

1.0.3 Methodology↩︎

1.0.3.1 Preliminaries

1.0.3.1.0.1 KV Cache.

The generation process of an LLM is autoregressive in nature, i.e., each generation step relies on the outputs of all previous steps. Suppose \(X\in\mathbb{R}^{n\times d}\) be the matrix containing the hidden states of a layer in an LLM upto a time step. Further, suppose \(x\in\mathbb{R}^{1\times d}\) be the hidden state of the last generated token, which is used as an input for the current generation step. Suppose, there are \(h\) attention heads. For a head \(i\in[1, h]\), the query, key and value matrices, \(W_i^Q\), \(W_i^K\) and \(W_i^V\) (all in \(\mathbb{R}^{d\times d_h}\)) map hidden states \(X\) to query, key and value states; specifically, the following are computed: \[\begin{align} Q_i = XW_i^Q, K_i = XW_i^K, V_i = XW_i^V \end{align}\] For the current time step, the states \(K_i\) and \(V_i\) constitute the KV cache elements for head \(i\). KV caching refers to caching \(K_i\) and \(V_i\) in memory to avoid recomputing them at every generation step. After computing the output for the current time step, the KV cache is updated accordingly: \[\begin{align} K_i = Cat[K_i : xW_i^K], V_i = Cat[V_i : xW_i^V] \end{align}\] KV cache compression techniques aim to retain only the most relevant key and value states from \(K_i\) and \(V_i\), without significant loss in generation quality. Particularly, we show that the loss on the attention output is upper bounded by the Frobenius distance between the original value matrix and it’s compression:

Note that for higher compression levels, the second term (\(\lVert V'\rVert_F\)) becomes negligible; in that case, the role of the approximation \(V'\) to \(V\) becomes significant. On the other hand, the contribution of \(Q\), \(K\) and \(K'\) is absorbed by the \(\text{softmax}\) operator. This motivates us to design a value-guided compression technique.

1.0.3.1.0.2 Low-rank decompositions.

Let \(A\in\mathbb{R}^{n\times d}\) be a matrix. The problem of low-rank decomposition of \(A\) seeks to factor \(A\) as a product of low-rank matrices, and is a well-studied problem with wide applications in mathematical modeling and data compression [16]. The optimal solution to this problem is the well-known singular value decomposition (SVD), which represents \(A\) in the form \(A = U\Sigma V^*\), with \(U\) and \(V\) being semi-unitary matrices, and \(\Sigma\) a square diagonal matrix of dimension equal to the rank of \(A\). While the SVD is widely adopted in data analysis problems, when applied to the context of the KV cache compression problem, it presents an inherent challenge; namely, the column/row vectors of \(U\) and \(V\) are not representative of the column/row vectors of \(A\) itself. Another suboptimal low-rank decomposition, called the CUR decomposition, resolves this limitation: it is an approximation of the form \(A \approx CUR\), where \(C\) is a subset of columns of \(A\), \(R\) is a subset of rows of \(A\) and \(U\) is a low-rank matrix. Here, while the matrices \(C\) and \(R\) are representative of the underlying data, the matrix \(U\) optimizes the approximation to \(A\). The CUR decomposition is known to be a good low-rank approximation to the original matrix [16]–[18].

1.0.3.1.0.3 Computation of CUR and Leverage Scores.

We utilize leverage scores, a popular technique used to compute CUR factorizations based on identifying key elements of the singular vectors of a matrix [18]. As before, let \(A = U\Sigma V^*\) be the SVD of \(A\in\mathbb{R}^{n\times d}\), where \(U\in\mathbb{R}^{n\times r}\), \(\Sigma\in\mathbb{R}^{r\times r}\) and \(V\in\mathbb{R}^{d\times r}\) with \(r = \text{rank}(A)\). The idea of leverage scores is to compute importance scores for each row and column of \(A\), which are then used to sample the rows and columns that then constitute the matrices \(R\) and \(C\) of the CUR decomposition respectively. Particularly, to rank the importance of rows of \(A\), we define the row leverage scores \(l_{r, j}\) (for the \(j\)th row of \(A\)) as \[\begin{align} \label{eq:row95lev95score} l_{r, j} := \lVert U(j, :)\rVert_2 \end{align}\tag{1}\] Similarly, to rank the importance of columns of \(A\), the column leverage scores \(l_{c, j}\) (for the \(j\)th column of \(A\)) are defined analogously by \[\begin{align} \label{eq:col95lev95score} l_{c, j} := \lVert V^*(:, j)\rVert_2 = \lVert V(j, :)\rVert_2 \end{align}\tag{2}\] Since \(U\) and \(V\) contain the left and right singular vectors of \(A\), it easily follows that \(\sum_{j = 1}^n l^2_{r, j} = n\) and \(\sum_{j = 1}^d l^2_{c, j} = d\). In turn, these scores lead to the distributions \(\{\frac{l^2_{r, j}}{n}\}\) and \(\{\frac{l^2_{c, j}}{d}\}\) from which the matrices \(R\) and \(C\) can be sampled. Alternatively, static methods use the top-\(k\) scores from either distribution to sample the corresponding matrices.

1.0.3.2 CurDKV: Algorithm Design

We now introduce CurDKV, our proposed KV cache compression method, which uses combined leverage scores of key and value matrices to compress the KV matrix. CurDKV  can be applied during the pre-filling as well as the token generation phase, is easy to implement, and seamlessly integrates with inference acceleration methods such as FlashAttention [12] and GQA [15]. Algorithm 3 provides the pseudocode of CurDKV. In the following discussion, we assume the attention setup as in GQA [15], wherein a group of attention heads uses a common key and value matrix. Particularly, we assume that there are \(g\) groups, with the key and value matrices of group \(i\), denoted by \(K_i\in\mathbb{R}^{n\times d}\) and \(V_i\in\mathbb{R}^{n\times d}\), respectively. As usual, \(n\) represents the number of keys (or values) in consideration, and \(d\) is the dimension of the space of vectors. The combined key and value matrices across all groups is denoted by \(K\in\mathbb{R}^{g\times n\times d}\) and \(V\in\mathbb{R}^{g\times n\times d}\).

As shown in Algorithm 3, CurDKV  takes as input the key matrix \(K\in\mathbb{R}^{g\times n\times d}\) across all attention groups, the corresponding value matrix \(V\in \mathbb{R}^{g\times n\times d}\), the group budget \(k\in [n]\) representing the level of compression, a projection dimension \(r\) and the number of initial attention sinks \(s\) to be preserved. It outputs compressed key and value matrices \(K'\in\mathbb{R}^{g\times k\times d}\) and \(V'\in\mathbb{R}^{g\times k\times d}\), which are then used to compute the output of the attention module. At a high-level, CurDKV  computes approximate leverage scores for each row of the key and value matrices \(K_i\in\mathbb{R}^{n\times d}\) and \(V_i\in\mathbb{R}^{n\times d}\) within a single group \(i\) (lines 3-5). The leverage scores for corresponding keys and values are then combined, followed by normalization (lines 6-7). Finally, the computed leverage scores are then used to retrieve the top-\(k\) keys and values from \(K_i\) and \(V_i\) (lines 9-11). In addition, CurDKV  also preserves the initial \(s\) tokens (lines 8, 10) due to the prevalence of attention sinks [11].

1.0.3.2.0.1 Approximate leverage scores and Gaussian projections.

The computation of exact leverage scores as in Equations 1 and 2 requires the computation of the SVD of the matrices in consideration, which turns out to be a bottleneck for the KV compression problem. To that end, CurDKV  projects \(K_i\) and \(V_i\) to matrices \(K_iG\) and \(V_iG\), respectively, where \(G\in\mathbb{R}^{d\times r}\) is a random matrix, each of whose entries are sampled from a normal \(\mathcal{N}(0, \frac{1}{r})\) distribution; in our implementation, we use \(r = 20\) ³. After this projection step, the norms of the rows of \(K_iG\) and \(V_iG\) are used as proxies for the actual leverage scores (lines 3-5 of Algorithm 3).

1.0.3.2.0.2 Adaptive budget allocation for CurDKV.

Along with CurDKV, we also implement AdaCurDKV, an adaptive variation of CurDKV. AdaCurDKV  adaptively achieves head-specific compression by choosing the top-\(k\) key and value vectors across all heads in a layer, where the selection is based on the computed leverage scores. As in the standard adaptive implementation introduced by  [4], a safeguard parameter \(\alpha\) is used to ensure a minimum fraction of key/value vectors are preserved for each head (a default value of \(\alpha = 0.20\) is used in our implementation).⁴

1.0.4 Experiments and Results↩︎

1.0.4.1 Experimental Settings

For the empirical analyses, following the contemporary studies, we use two instruction-tuned LLMs – LLaMA-3.1-8B-Instruct [2] and Mistral-7B-Instruct-v0.3 [3]. All the evaluations are done on two widely popular long-context benchmarks – LongBench [8] and Ruler [9]. LongBench benchmark contains a total 16 tasks, covering various task domains including single-document QA, multiple-document QA, summarization, few-shot learning, synthetic tasks and code completion. From Ruler benchmark, we consider a total eight needle-in-a-haystack tasks with a maximum context length of 16K. Details of the LongBench and Ruler tasks and the prompt templates can be found in the appendix. For all these 24 tasks, we evaluate only on the more challenging question-agnostic setting [19], where the questions are omitted during compression and only the context is compressed. As argued by [4], this setup mimics more challenging scenarios, where the compression method is unaware of the questions being passed to the model during inference.

We compare CurDKVwith various competitive KV compression baselines, including SnapKV [6], ChunkKV [10], LLM Streamline [11] and Ada-SnapKV [4]. We consider an additional baseline, KNorm [19] that uses key norm as a proxy to determine which keys to evict during compression. H2O [7] is purposefully omitted from the evaluation as it throws out-of-memory error when run on a single NVIDIA A100-80GB GPU card (as it does not naively support flashattention, therefore requiring \(4\times\) more cache memory on long sequences). All these baselines were run for different compression (eviction) ratios \(\{30\%, 50\%, 70\%, 90\%\}\). Following  [11], we use attention sinks of size \(s=4\) for all the baselines (all the baseline numbers are produced in-house).

1.0.4.2 Results on LongBench

r0.6

We report the results obtained with full cache (0% compression) for LLaMA-8B and Mistral-7B in Table 1. Table 2 summarizes the results obtained with LLaMA-8B and Mistral-7B with KV cache compressed at 30% and 90% with different baselines.

On LLaMA-8B, CurDKVoutperforms SnapKV by +3.6% at 30% compression (48.9% vs. 45.3%) and by +2.0% points at 90% (35.7% vs. 33.7%). On Mistral-7B, CurDKVshows a similar trend, surpassing SnapKV by +2.9% at 30% (45.6% vs. 42.7%) and by +0.5% at 90% (33.2% vs. 32.7%). These gains are robust across task types, including multi-hop QA, summarization, and code completion. Norm-based heuristics such as Knorm and ChunkKV fail to consistently preserve semantic fidelity, often underperforming even under moderate compression. This is most evident in Mistral-7B’s few-shot learning tasks (e.g., TriviaQA, SAMSum), where CurDKVmaintains over 89% performance, other baselines trail by 5–10%.

We compare AdaCurDKV, our adaptive compression strategy with its static variant (CurDKV) and the attention-based adaptive baseline, AdaSnapKV. Across models and retention levels, AdaCurDKVconsistently outperforms AdaSnapKV, while performing competitively with CurDKV. With LLaMA-8B under 30% KV compression, AdaCurDKVachieves the highest average score (49.1%), outperforming AdaSnapKV (45.5%) by +3.6% and CurDKV(48.9%) by +0.02%. At 90% retention, AdaCurDKVmaintains a strong average score of 35.1%, closely trailing CurDKV(35.7%) and surpassing AdaSnapKV (34.2%). For Mistral-7B, AdaCurDKVachieves 45.2% performance at 30% retention, outperforming AdaSnapKV (42.1%), and remaining close to CurDKV(45.6%). Although AdaCurDKV’s average score drops to 29.1% at 90% retention, it remains competitive and stable across all task types. While CurDKVtypically attains the best or second-best overall score, AdaCurDKVprovides additional robustness by allocating head-wise budgets proportional to leverage score mass, leading to improved performance on head-sensitive tasks, especially in summarization and multi-hop QA.

We conduct ablations (c.f. Figure 4) to assess two core design choices in CurDKV: the source of leverage scores and the use of random projections. At 30% compression, different leverage modules (key, value, key-value product) perform similarly, but at 90%, value-centric and combined variants show better robustness than key-only. Random projections offer marginal gains overall, with slightly improved stability at high compression (more elaborated discussion in Appendix 1.0.8).

1.0.4.3 Results on Ruler

r0.34

We further evaluate CurDKVand AdaCurDKVon the Ruler benchmark, focusing on needle-in-a-haystack (NIAH) subtasks, which measure a model’s ability to recover rare, high-salience facts embedded in long distractor contexts. These tasks are particularly sensitive to the retention of semantically critical key-value tokens. Full cache results are highlighted in Table 3.

As shown in Table 4, CurDKVand AdaCurDKVachieve the highest average score (98.7% and 97.7%) under 30% compression, outperforming all baselines. CurDKV substantially outperform norm-based (ChunkKV: 87.0%, Knorm: 71.8%) and attention-based approaches (SnapKV: 80.4%, StreamingLLM: 68.2%). On particularly hard retrieval subtasks like MK-3 and MQ, AdaCurDKVscores 93.6% and 100.0% respectively, indicating its strong capacity to retain tokens that encode key factual content. In S-3 – a shallow, multi-hop setting, AdaCurDKVscores 88.2% compared to 82.4% (AdaSnapKV) and 94.1% (CurDKV), reflecting robustness in both adaptive and static modes. At 90% compression, performance drops sharply for most baselines, with norm-based methods like Knorm (13.9%) and StreamingLLM (10.1%) failing to preserve fidelity. In contrast, CurDKV(34.7%) and AdaCurDKV(39.1%) maintain significantly higher performance, with AdaCurDKVagain achieving the best overall average. Notably, on MQ and MV, AdaCurDKVreaches 56.7% and 52.0% respectively, far ahead of all other methods, highlighting its advantage on tasks requiring deeper contextual aggregation. On S-2, AdaCurDKVoutperforms both AdaSnapKV and CurDKV(63.2% vs. 36.8% and 57.9%), showcasing the benefit of adaptive head-wise allocation in early-token regimes. Albeit the strong performance across different subtasks, we observe marginal underperformance on a subset of NIAH tasks with Mistral at \(90\%\) compression ratio. This can be attributed to Mistral’s use of sliding-window attention which reduces head specialization and makes head-local token selection less effective. In such cases, heuristics like ChunkKV, which retain tokens based on uniform position or locality, can incidentally preserve critical information. Nonetheless, CurDKVcontinues to show superior performance on more challenging retrieval subtasks, indicating its robustness under complex attention patterns.

1.0.4.4 Ablation Study

We perform two ablation studies to investigate the key design decisions behind CurDKV: (1) the module on which leverage scores are computed, and (2) the effect of using randomized projections for score approximation.

We first ablate the module on which leverage scores are computed. Figure 4 shows the average performance (task-level scores reported in Table in Appendix) when using key-only, value-only, and the default key-value product (elementwise product of key and value leverage scores). All three variants perform similarly at 30% compression. However, at 90% compression, the differences become more pronounced: key-only scores degrade significantly, while value and key-value product maintain higher median accuracy and robustness. This confirms that value information is more predictive of output quality, and that combining key and value structure yields the most reliable token selection strategy.

Next, we evaluate the role of randomized leverage score approximation. This determines whether the key and value matrices are projected to a low-rank space via Gaussian random matrices before computing leverage scores. we observe marginal differences between the two variants. At 30% compression, both versions perform similarly in median accuracy, though random projection slightly improves the worst-case performance. At 90% compression, however, random projections result in modest improvements in median and lower-quartile accuracy. These gains suggest that random projections can help stabilize leverage score estimation under extreme compression, but their effect is secondary to the choice of leverage module.

1.0.4.5 Computational Efficiency of CurDKV

To evaluate the practical benefits of KV compression beyond accuracy, we analyze CurDKVin terms of memory usage, prefill latency, and generation latency across varying sequence lengths and compression ratios. As expected, CurDKVyields (c.f. Figure 6a) a near-linear reduction in KV cache size with increasing compression ratio. For example, at 80% compression, memory usage drops from 15.6 GB to under 3 GB for a 128K-token context. This linear trend holds consistently across all tested sequence lengths. Since CurDKVcompresses the cache on a per-layer and per-head basis, these reductions are directly proportional to the retained key-value tokens, enabling predictable memory scaling in long-context scenarios. Figure 6 (b) shows a slight upward trend in prefill latency at higher compression ratios, especially for longer sequences. This is because CurDKVcomputes leverage scores and performs token selection during the prefill phase, introducing modest computational overhead. For instance, with 128K tokens, prefill time increases from approximately 10 seconds (no compression) to around 14–15 seconds at high compression levels. However, the added cost remains constant beyond 40–60% compression, indicating amortization of overhead due to batch-efficient scoring. Interestingly, generation latency (Figure 6 (c)) decreases with compression for all sequence lengths, particularly for longer inputs. This is attributed to the smaller number of cached tokens being read during autoregressive decoding. With 128K-token contexts, generation time reduces from 10 seconds (no compression) to under 6 seconds at 80% compression. The effect is more muted for shorter sequences, but consistently observable. These properties make CurDKVa practical and scalable solution for deployment-time KV cache optimization in LLMs.

1.0.5 Conclusion↩︎

In this paper we proposed CurDKV, a value-centric KV cache compression method based on CUR decomposition, and AdaCurDKV, its adaptive variant. Unlike prior approaches that rely on attention scores, our method selects key-value tokens using leverage scores derived from the value matrix, with efficient approximations via random projections. Experiments on LongBench and Ruler benchmarks showed that CurDKVconsistently outperformed existing methods across tasks and compression ratios, while remaining compatible with FlashAttention and Grouped Query Attention.

1.0.5.0.0.1 Limitations and future work.

While CurDKVoffers a principled and effective approach to KV cache compression, it relies on static token selection during the prefill phase, which may limit its responsiveness to dynamically emerging information needs during generation. Future work could explore query-aware token selection, hybrid token-chunk strategies, or lightweight learned components to improve performance in semantically sparse settings. Extending CurDKVto support dynamic compression during generation is another promising direction to better adapt to evolving query demands.

1.0.6 Proof of Lemma 1↩︎

1.0.7 Datasets, Task Descriptions and Task Templates↩︎

1.0.7.1 The LongBench benchmark

LongBench [8] is a set of multitasks designed to assess the ability of LLMs to handle long-context problems requiring deep understanding and reasoning. Tasks include single-document question-answering, mutli-document question answering, summarization, few-shot learning, synthetic tasks and code generation. Below we present a detailed overview of all tasks:

• Single-Doc QA: Models are required to obtain the answer from a single source document. Test samples are derived from a variety of datasets, including NarrativeQA [20], Qasper [21], and MultiFieldQA [22]. Templates for these tasks can be found in Table 5.

• Multi-Doc QA: Models are required to extract and combine information from multiple source documents to obtain the answer. Data samples are derived from three multi-hop QA datasets: HotpotQA [23], 2WikiMultihopQA [24] and MuSiQue [25]. Templates for these tasks can be found in Table 6.

• Summarization: These tasks require a comprehensive understanding of the context. The data samples are obtained from the GovReport [26] and QMSum [27] datasets. Templates for these tasks can be found in Table 7.

• Few-shot Learning: These tasks have classification, summarization and reading-comprehension tasks integrated within them to maintain task diversity. The TREC dataset [28] is used for classification problems. The SAMSum dataset [29] is used for summarization tasks. Finally, TriviaQA [30] is utilized for comprehension tasks. Templates for these tasks can be found in Table 8.

• Synthetic Tasks: These tasks assess a model’s ability to handle specific scenarios and patterns. In LongBench, the PassageRetrieval-en and PassageCount datasets are used. Templates for these tasks can be found in Table 9.

• Code Completion Tasks: These tasks are meant to assist users by completing code based on a user’s input and context. The LCC dataset from the original Long Code Completion dataset [31] as well as the RepoBench-P dataset [32] is used. Templates for these tasks can be found in Table 10.

1.0.7.2 The Ruler benchmark

Ruler [9] is a collection of synthetic examples to evaluate long-context language models, containing four task categories to test behaviours beyond simple retrieval from context. Here is a detailed description of each task category:

• Singe Needle-In-A-Haystack (S-): In these tasks, a keyword sentence, called the "needle", is embedded within a lengthy text, called the "haystack". The objective of the task is to retrieve the needle from the context.

• Multi-keys NIAH (MK-): These tasks are similar to S- tasks, with the difference being the presence of multiple "needles" inserted into the "haystack". However, the task is to retrieve only one of them.

• Multi-values NIAH (MV-): Here, multiple "needles" in the form of key-value pairs are hidden in the "haystack". The task is to retrieve all values associated to a single key.

• Multi-queries NIAH (MQ-): Here, multiple "needles" in the form of key-value pairs are inserted into the "haystack". All "needles" corresponding to the keys specified in the queries are to be retrieved.

Templates for these tasks can be found in Tables 11 and 12 respectively.

1.0.8 Additional Results↩︎

1.0.8.1 Results on LongBench

We evaluate the performance of CurDKVand AdaCurDKVagainst baseline compression methods at intermediate compression ratios of 50% and 70%, using both the LLaMA-3.1-8B and Mistral-7B in Table 13 and Table 14, respectively.

On LLaMA, CurDKVachieves the highest average accuracy (48.3), outperforming all non-adaptive baselines such as SnapKV (44.3) and Knorm (43.1), as well as Streaming LLM (37.7). The adaptive variant AdaCurDKValso performs competitively, with an average score of 47.9, surpassing AdaSnapKV (44.7). At 70% compression, CUR-based approaches continue to perform favorably. On LLaMA, CurDKVagain achieves the highest average (45.4), ahead of SnapKV (41.9) and Knorm (36.4), confirming its robustness across moderate compression levels. AdaCurDKVfurther improves upon AdaSnapKV (42.4) with an average of 46.1, marking consistent gains across adaptive strategies.

On Mistral, CurDKVagain delivers the best average score (43.8), outperforming SnapKV (41.2) and ChunkKV (41.0). AdaCurDKV(43.7) also surpasses AdaSnapKV (41.9), particularly benefiting from adaptive budget reallocation on heterogeneous tasks such as TREC (72.0) and SAMSum (45.4). Across both models, CUR-based methods show superior stability, particularly under structured reasoning and information-dense settings.

1.0.8.2 Results on Ruler

Table 15 and 16 report the results of different KV compression models with LLaMA and Mistral models, respectively, at 50% and 70% compression ratios.

With LLaMA, at 50% compression, CurDKVachieves a strong average of 83.1, outperforming all non-adaptive baselines, including ChunkKV (79.2) and SnapKV (67.0). The adaptive variant, AdaCurDKV  further improves performance to 89.6, benefiting from its dynamic budget allocation across heads. Notably, AdaCurDKVachieves perfect accuracy (100.0) on 6 out of 8 subtasks, demonstrating its ability to preserve high-salience tokens even at moderate compression. At 70% compression, the relative advantage of CUR-based methods persists. CurDKVscores an average of 62.0, substantially ahead of SnapKV (48.0) and Knorm (32.7), however, falling short from the best non-adaptive baseline ChunkKV (64.1). AdaCurDKVcontinues to perform best overall with an average of 69.8, highlighting its robustness in token-starved regimes. These results reinforce the effectiveness of CUR-based compression for preserving semantically critical tokens required for pinpoint retrieval.

On Mistral at 50% compression, CurDKVachieves an average score of 54.2, considerably higher than SnapKV (23.9), Knorm (12.0), and Streaming LLM (41.3). AdaCurDKVpushes this further to 56.1, showing consistent gains across nearly all subtasks.

Albeit the minor performance drops, these experiments confirm that CUR-based selection, especially when adaptively guided—provides a robust and effective mechanism for maintaining retrieval fidelity under tight memory constraints, outperforming both norm- and attention-based KV selection across two architectures and multiple compression levels.

1.0.8.3 Ablation Study

We perform two ablation studies to investigate the key design decisions behind CurDKV: (1) the module on which leverage scores are computed, and (2) the effect of using randomized projections for score approximation. While in our algorithm we use combined leverage scores computed using both key and value vectors (line 5 of Algorithm 3), in our ablations we study the affect of using only the key or value vectors to compute leverage scores. We also run experiments wherein leverage scores are computed on the original key/value matrices without the use of the Gaussian projection (i.e., we drop lines 3-4 of Algorithm 3)

We first ablate the module on which leverage scores are computed. Figure 4 shows the average performance (task-level scores reported in Table 17) when using key-only, value-only, and the default key-value product (elementwise product of key and value leverage scores). All three variants perform similarly at 30% compression. However, at 90% compression, the differences become more pronounced: key-only scores degrade significantly, while value and key-value product maintain higher median accuracy and robustness. This confirms that value information is more predictive of output quality, and that combining key and value structure yields the most reliable token selection strategy.

Next, we evaluate the role of randomized leverage score approximation. This determines whether the key and value matrices are projected to a low-rank space via Gaussian random matrices before computing leverage scores. We observe marginal differences between the two variants. At 30% compression, both versions perform similarly in median accuracy, though random projection slightly improves the worst-case performance. At 90% compression, however, random projections result in modest improvements in median and lower-quartile accuracy. These gains suggest that random projections can help stabilize leverage score estimation under extreme compression, but their effect is secondary to the choice of leverage module.

1.0.8.4 Runtime Analysis

We analyze the runtime behavior of CurDKVand SnapKV under varying compression ratios and input lengths, as illustrated in Figures 6 and 7. We focus on three key metrics: KV cache size, prefill time, and generation time.

1.0.8.4.0.1 Cache Size.

Both methods exhibit nearly linear reduction in KV cache size as compression increases. For example, with a 128K token input, the cache size reduces from approximately 16 GB at 0% compression to under 2 GB at 80% compression for both CurDKVand SnapKV (Figures 6a and 7a). This confirms that CUR-based token selection inherits the cache sparsity advantages of heuristic methods like SnapKV.

1.0.8.4.0.2 Prefill Time.

SnapKV maintains flat prefill latency regardless of compression. For instance, at 128K tokens, prefill time remains consistently around 10 seconds across all compression levels (Figure 7b). CurDKV  by contrast, incurs higher latency due to leverage score computation, rising from 10 seconds at 0% compression to about 14-15 seconds at 60–80% compression for 128K tokens (Figure 6b). For shorter sequences like 4K or 16K, the difference is marginal, under 1 second for both methods.

1.0.8.4.0.3 Generation Time.

CurDKVdemonstrates a stronger reduction in generation time, particularly for long sequences. For 128K-token inputs, generation latency drops from \(\sim\)​12 seconds at 0% compression to just \(\sim\)​4.5 seconds at 80% compression (Figure 6c). SnapKV also sees a drop from \(\sim\)​15 seconds to \(\sim\)​6 seconds, but exhibits irregular behavior for mid-length sequences (e.g., 64K shows a flat profile between 40–80% compression in Figure 7c).

While SnapKV is faster to prefill, CurDKVyields larger reductions in generation time, especially for long sequences (e.g., 128K tokens) and high compression levels. Its additional prefill cost roughly 3–5 seconds at worst is offset by its smoother and more aggressive runtime scaling, making it a more efficient choice for long-context inference when accuracy and latency matter.

References↩︎