1 Introduction↩︎

Chain-of-thought reasoning [1]–[3] has endowed LLMs with powerful capabilities for solving complex tasks. However, reasoning models often generate long traces, which lead to a substantial growth of KV cache during decoding and impose significant memory overhead [4], [5]. For example, caching 128K tokens in Qwen3-14B with FP16 precision requires approximately 20 GB memory. This overhead becomes especially pronounced in large-batch serving settings, where the rapidly growing KV cache increases decoding latency and causes requests to wait for available memory resources, thereby reducing overall output throughput.

To address these issues, KV cache compression [6]–[8], an inference-time approach, has garnered increasing attention. By selectively retaining important KV pairs, KV cache compression can reduce memory usage while keeping performance degradation within an acceptable range (e.g., within a 1% performance drop). Existing works [8], [9] have shown strong performance on multiple tasks while retaining only a small fraction of KV cache (e.g., 30\(\%\)). Early compression methods [10], [11] relied on heuristic techniques such as attention sinks to preserve KV pairs at fixed positions, ignoring the varying importance of different tokens and resulting in suboptimal performance. More recently, research has shifted toward score-based KV cache compression [6], [8], [12], [13], which assigns importance scores to multi-head KV pairs to determine cache retention. These methods typically maintain a query cache of recent query states and use them to score historical KV pairs via attention with their key states. The score-based compression methods often achieve state-of-the-art performance, as they leverage the internal information of LLMs to identify critical KV pairs [14].

Despite their success, we identify two key limitations in existing KV cache compression methods.

• These methods mainly rely on a threshold-triggered compression policy, which can degrade both reasoning performance and inference efficiency. Under the threshold-triggered compression policy, the KV cache is compressed back to a smaller target length once it reaches a predefined threshold, after which this grow-and-compress pattern repeats. While effective in low-frequency or personal deployment settings, this policy can cause a concurrency–throughput inversion effect. Specifically, when the number of decoding sequences becomes comparable to the gap between the threshold and the post-compression cache length, compression may be triggered frequently, and the associated computational overhead can decrease output throughput even as the number of concurrently decoded sequences increases. In addition, since compression is applied over the entire KV cache, previously compressed regions of the sequence may be compressed again, which can further worsen information loss and poison reasoning quality. To empirically validate these challenges, we conduct a simple experiment using the vLLM² framework [15] with a representative decoding-time compression method (SnapKV[8]) on the MATH-500 dataset [16] with distilled Llama-8B models [17]. As shown in Figures 1 and [fig:F2], the output throughput of vLLM with SnapKV decreases even as the number of concurrently decoded sequences increases, and the corresponding KV pairs of earlier tokens are almost completely removed as compression is applied repeatedly. Detailed experimental settings are provided in Appendix 7.

• These methods retain either isolated KV pairs or fixed-size chunks with rigid boundaries, failing to preserve important flexible-sized chunks at arbitrary token positions. Mainstream KV cache compression methods select isolated KV pairs for retention based on scores, which can lead to significant loss of semantic information. To address this issue, several methods further propose retaining contiguous KV pairs to form chunks. For example, ChunkKV [18] partitions the historical KV cache into chunks of the same length with fixed boundaries. However, such retention paradigms are often rigid and fail to capture flexible-sized KV information distributed at arbitrary positions across the sequence.

To overcome these limitations, we propose Kara, a sliding-window KV cache compression method that performs decoding-time compression only on the recently generated context. Through empirical analysis, we find that the accumulated bidirectional attention received by each KV pair serves as an effective indicator of its importance and informativeness. Building on this observation, Kara uses the accumulated bidirectional attention score as the importance measure to identify discrete candidate KV pairs within the window. To preserve flexible-sized contiguous KV pairs at arbitrary positions, we introduce a simple yet effective Token2Chunk module. Given the candidate discrete KV pairs, Token2Chunk treats every two consecutive candidate indices as the endpoints of a chunk, and filters chunks according to their width and the importance scores of the endpoint KV pairs. Finally, we preserve the candidate discrete KV pairs as well as all KV pairs inside the selected chunks. In summary, Kara produces a compressed KV cache that combines both isolated KV pairs and chunk-level KV pairs within the sliding window. Furthermore, we adapt Kara to PagedAttention [15] and develop KvLLM³, a vLLM-based inference framework equipped with a periodic compression policy. To avoid frequent compression triggering during decoding, the periodic compression policy compresses the trailing blocks of the selected sequences every fixed number of decoding steps. Extensive experiments demonstrate that the proposed Kara and KvLLM framework can reduce KV cache memory usage while improving throughput and concurrency, and further preserving reasoning quality.

In summary, our main contributions are as follows:

• We propose Kara, a KV cache compression method that uses sliding-window bidirectional attention to identify candidate discrete KV pairs and employs a Token2Chunk module to generate chunks with contiguous KV pairs.

• We adapt the proposed method to PagedAttention and design KvLLM inference framework with a periodic compression policy, which can effectively improve throughput, concurrency, and reasoning quality compared with existing compression methods.

• We conduct extensive experiments, including reasoning and NIAH (Needle-in-a-Haystack) evaluations, to demonstrate the superiority and effectiveness of the proposed method and inference framework.

2 Preliminary↩︎

2.0.0.1 Notation.

We consider an autoregressive Transformer-based LLM [19], [20] that generates a token sequence denoted by \(\mathbf{X}=(x_1,x_2,\dots,x_S)\), where \(S\) is the sequence length. The model consists of \(L\) Transformer layers, each with \(H\) attention heads and head dimension \(d\). For the \(h\)-th attention head, we denote the projected query, key, and value states as \(\mathbf{Q}^{l,h},\mathbf{K}^{l,h},\mathbf{V}^{l,h}\in\mathbb{R}^{S\times d}\), respectively. The scaled dot-product attention probabilities are defined as: \[\mathbf{P}^{l,h}=\text{softmax}\!\left(\frac{\mathbf{Q}^{l,h}(\mathbf{K}^{l,h})^\top}{\sqrt{d}}\right),\] where \(\mathbf{P}^{l,h}_{i,j}\) denotes the attention probability from the \(i\)-th query token to the \(j\)-th key token in layer \(l\) and head \(h\). For simplicity, we use \(\mathbf{Q}\) and \(\mathbf{K}\) to denote the query and key states of a specific layer \(l\) and head \(h\).

2.0.0.2 LLM Inference.

During autoregressive decoding, the model maintains a key cache \(\mathcal{K}\) and a value cache \(\mathcal{V}\) to store the key and value states corresponding to historical tokens. Compared with computing attention over the entire prefix, KV caching substantially reduces the quadratic computation cost and improves inference efficiency [21].

In practice, the decoding efficiency of LLM inference is commonly measured by TPOT (Time Per Output Token), which denotes the average time required to generate one output token during decoding. Beyond TPOT, throughput is another important metric that measures the number of output tokens generated per unit time, and is commonly used to reflect the concurrency and latency. In general, more decoding sequences enables higher throughput In long-context reasoning scenarios, the inference overhead may be dominated by memory access to the KV cache, leaving the inference memory-bound. The detailed formulas for TPOT and throughput are presented in Appendix 8.

To improve the serving efficiency of LLM inference, frameworks [15], [22] such as vLLM have been widely developed, they aim to maximize memory utilization and sustain efficient decoding under high-concurrency and long-context generation. The core idea of vLLM is PagedAttention, which views the KV cache as a group of fixed-size blocks. Each block stores a contiguous segment of KV pairs (e.g., 256 tokens), and memory allocation and release are managed at the block level during inference. However, in memory-constrained settings, the available memory can be rapidly consumed as decoding proceeds, which may force the inference framework to reduce the number of running requests and thereby limit overall output throughput (shown in Figure 1 (b)).

2.0.0.3 KV Cache Compression.

KV cache compression [6], [10], [23]–[25] reduces the memory footprint of LLM inference by evicting unimportant ones. Most compression methods [6], [8], [26] estimate an importance score \(\mathbf{A}_i^{l,h}\) for the KV pair of token position \(i\) in attention head \(h\) and layer \(l\), and then select the most important KV pairs under a cache budget when the cache length reaches a predefined threshold. A representative score-based compression paradigm can be formulated as: \[\label{Eq:Topk} \mathcal{I}^{l,h} = \textrm{TopK}(\{\mathbf{A}_i^{l,h}\}_{i=1}^{C_t},\, N)\tag{1}\] \[(\mathcal{K}^{l,h},\mathcal{V}^{l,h})\leftarrow(\mathcal{K}^{l,h},\mathcal{V}^{l,h})[\mathcal{I}^{l,h}],\quad \text{when } C_t \ge C_{\max}, \label{eq:threshold95triggered95compression}\tag{2}\] where \(\mathcal{I}^{l,h}\) is the index of selected KV pairs of head \(h\) and layer \(l\), \(N\) is the predefined KV budget, \(C_t\) denote the number of KV pairs currently stored in the cache (i.e., the cache length) at decoding step \(t\), \(C_{\max}\) denote the predefined threshold. Once compressed, the cache length is constrained within the interval \([N,\, C_{\max}]\) as it grows from \(N\) back to \(C_{\max}\) before the next compression. In the following, we omit the notation \(l\) and \(h\) as all operations performed in the layer and attention head are same in our method. Although the compression paradigm in Equation 1 and 2 is widely adopted, it suffers from two key limitations:

• Threshold-triggered compression may degrade both reasoning performance and inference efficiency. Since the paradigm operates compression over the entire KV cache, it may fully remove all KV pairs within a long contiguous span of context (e.g., \(x_i \sim x_{i+256}\) in Figure [fig:F2]), potentially exacerbating information loss. Furthermore, when the decoding batch size increases toward the gap between \(C_{\max}\) and \(N\) (i.e., \(C_{\max}-N\)), compression can be triggered frequently across concurrently decoded sequences, making the overall compression overhead more pronounced and potentially reducing throughput (shown in Figure 1).

• Retention granularity is often either isolated KV pairs or rigid fixed-size chunks. Mainstream score-based methods select isolated KV pairs for retention, which can lead to the loss of semantic information. Although some methods retain contiguous KV pairs to form chunks, such chunks are often extracted with rigid boundaries (e.g., fixed spans like \(\mathbf{K}_{1}\sim\mathbf{K}_{256}\)), which may fail to capture flexible-sized cache information distributed at arbitrary positions across the sequence.

Given the flaws of existing methods, we argue for using sliding-window bidirectional-attention scores to guide KV-pair retention within the window, together with a retention strategy that incorporates both token-level and chunk-level KV pairs. We further adapt our method to PagedAttention and develop the KvLLM framework with a periodic compression policy, which aims to improve throughput under high concurrency while maintaining reasoning quality.

3 Method↩︎

3.1 Sliding-Window Bidirectional Attention for KV Importance Estimation↩︎

During decoding, we maintain a sliding window that dynamically covers the recently generated tokens, and we apply KV cache compression only to KV pairs within the window. Once compression is performed, the window slides forward to cover newly generated tokens that is uncompressed, and the same procedure is applied repeatedly. Inspired by [6] and [8] , we additionally reserve a buffer at the end of the window to protect the latest generated tokens from being compressed, which helps preserve reasoning quality.

To obtain important discrete KV pairs, we first extract the query states of all tokens inside the window and compute bidirectional attention to the key states of tokens inside the window except those in the buffer. We then accumulate the attention weights received by each KV pair to estimate its importance score. Finally, we select the TopK important KV pairs inside the window as retention candidates. In summary, the sliding-window bidirectional attention scoring can be formulated as: \[\label{eq:kara95importance} \mathbf{A}_{j} = \sum_{i\in\mathcal{W}} \frac{ \exp(\langle \mathbf{Q}_{i}, \mathbf{K}_{j}\rangle / \sqrt{d}) }{ \sum\limits_{j' \in \mathcal{W}\setminus\mathcal{U}} \exp(\langle \mathbf{Q}_{i}, \mathbf{K}_{j'}\rangle / \sqrt{d}) }, \qquad j\in \mathcal{W}\setminus\mathcal{U},\tag{3}\] \[\mathcal{I}=\textrm{TopK}( \{\mathbf{A}_{j}\}_{j\in \mathcal{W}\setminus\mathcal{U}}, \left\lceil r (|\mathcal{W}|-|\mathcal{U}|)\right\rceil). \label{eq:kara95topk}\tag{4}\] where \(\mathbf{A}_{j}\) is the estimated importance score of the KV pair from token \(x_{j}\), \(\mathcal{W}\) denotes the token indices of the current sliding window, and \(\mathcal{U}\subset\mathcal{W}\) denotes the buffer at the end of the window (\(|\mathcal{U}| < |\mathcal{W}|\)), \(\mathcal{W}\setminus\mathcal{U}\) is the compressible region in the window with length \(|\mathcal{W}| - |\mathcal{U}|\). \(r\in(0,1]\) is the retention ratio, and \(\mathcal{I}\) denotes the index set of candidate discrete KV pairs to be retained within the current window.

Since tokens in \(\mathcal{U}\) are excluded from compression, Equation (3 ) serves as an approximation to bidirectional attention over all tokens within the window. The importance score \(\mathbf{A}\in \mathbb{R}^{|\mathcal{I}|}\) can also be viewed as combining bidirectional attention scores over the compressible region with causal-attention scores induced by the query states in the buffer. After compression, the window moves forward by \(|\mathcal{W}|-|\mathcal{U}|\) steps once there are \(|\mathcal{W}|-|\mathcal{U}|\) newly generated uncompressed tokens beyond the current window.

3.1.0.1 Empirical Analysis.

We conduct simple experiments to empirically demonstrate the superiority of bidirectional attention in importance estimation. Given a sequence, we first compute both the causal attention and bidirectional attention probability matrices over all token pairs, and visualize the cumulative attention received by each token’s key state from all query tokens. As shown in Figure [fig:F4], most accumulated casual attention scores concentrate in the prefix region of the window, as earlier positions are accessible to more queries than later ones. Then, we visualize the specific token’s bidirectional attention probability distribution over all tokens in the sequence, as well as its causal attention distribution over its preceding tokens. Finally, we examine the correlation between the bidirectional attention probability from former token \(x_i\) to later token \(x_j\) (\(j>i\)) and the causal attention probability from later token \(x_j\) to former token \(x_i\). As shown in Figures 2, we observe that 1) token \(x_i\)’s causal attention distribution largely matches its bidirectional attention distribution on preceding positions, and 2) when former token \(x_i\) assigns a higher bidirectional attention weight to future token \(x_j\), token \(x_j\) also tends to assign a higher causal attention weight to former token \(x_i\).

The above analyses suggest that bidirectional attention can identify important preceding context information similar to causal attention, and highlight future tokens that are likely to attend strongly to the current token. Building on this observation, we use bidirectional attention to identify KV pairs that are important for the current window context. It also selects KV pairs whose corresponding query states are highly active during causal-attention computation, and these KV pairs often carry rich contextual information.

3.1.0.2 Implementation Details.

We can obtain the query states required for KV scoring via recomputation while reusing the existing KV cache. Specifically, at each compression, we run an additional forward pass on the tokens in the current window and compute only the query projections, the attention computation and the FFN (feed-forward network). The recomputation avoids maintaining an explicit query cache during decoding. The sliding-window mechanism can also be applied to compress the prompt KV cache before decoding. Specifically, during the prefill stage, we can partition the prompt into multiple windows of length \(|\mathcal{W}|\) and perform window-wise compression on the KV pairs corresponding to positions inside each window in parallel. After compression, the window moves forward by \(|\mathcal{W}|-|\mathcal{U}|\) steps once there are \(|\mathcal{W}|-|\mathcal{U}|\) uncompressed tokens beyond the current window during decoding.

3.2 Token2Chunk Module↩︎

Building on the discrete KV indices \(\mathcal{I}\) obtained from sliding-window bidirectional attention, we introduce a lightweight Token2Chunk module to preserve flexible-sized contiguous KV pairs at arbitrary positions within the window. Specifically, Token2Chunk takes as input a discrete retained indices \(\mathcal{I}\) and the corresponding importance scores \(\mathbf{A}\in \mathbb{R}^{|\mathcal{I}|}\). We first define the chunk budget that controls maximum additional KV pairs can be introduced by this module, together with a maximum chunk length. We then sort the indices in \(\mathcal{I}\) in ascending order and divide the compressible region into chunks by treating every two consecutive indices as the endpoints of a candidate chunk. Each candidate chunk indicates a contiguous span, whose interior positions correspond to KV pairs that can be additionally retained as a chunk. Next, we filter candidate chunks by the maximum chunk length, and score each remaining chunk using both its length and the importance scores of the two endpoint KV pairs. Finally, we preserve all KV pairs inside the selected chunks, and union them with \(\mathcal{I}\) to obtain the final retained indices. For the \(k\)-th chunk (\(1\le k\le|\mathcal{I}|-1\)), Token2Chunk can be formulated as: \[\label{eq:chunkscore} \mathbf{R}_k = (\mathbf{A}_{\mathcal{I}[k]} + \mathbf{A}_{\mathcal{I}[k+1]}) \times (\mathcal{I}[k+1]-\mathcal{I}[k]-1),\tag{5}\] \[\label{eq:set95topk} \mathcal{P} = \mathrm{TopK}( \{\, \mathbf{R}_k \,\}_{k = 1}^{|\mathcal{I}|-1},\; \left\lfloor \frac{\alpha}{\gamma}\right\rfloor), \quad \text{s.t. }\;\mathcal{I}[k+1]-\mathcal{I}[k]\le \gamma\tag{6}\] \[\label{eq:mergeset} \hat{\mathcal{I}} = \mathcal{I} \;\cup\; \bigcup_{k\in \mathcal{P}} \left\{\, j \;\middle|\; \mathcal{I}[k]< j < \mathcal{I}[k+1] \,\right\}.\tag{7}\] where \(\mathbf{R}_k\) is the \(k\)-th chunk’s importance score, \(\mathcal{I}\) is the discrete retained indices obtained from sliding-window bidirectional attention (sorted in ascending order), and \(\mathcal{I}[k]\) denotes the \(k\)-th smallest element in \(\mathcal{I}\). \(A_{\mathcal{I}[k]}\) is the importance score of the KV pair at index \(\mathcal{I}[k]\). \(\gamma\) is the predefined maximum chunk length, and \(\alpha\) is the predefined chunk budget controlling the number of additional KV pairs introduced by Token2Chunk. \(\mathcal{P}\) denotes selected chunk indices returned by \(\mathrm{TopK}(\cdot)\), and \(\hat{\mathcal{I}}\) denotes the final retained KV pair indices after adding all KV pairs inside the selected chunks. We then use \(\hat{\mathcal{I}}\) to compress the KV cache of the current sliding window: \[(\mathcal{K},\mathcal{V}) \leftarrow (\mathcal{K},\mathcal{V})[\hat{\mathcal{I}}],\] where \((\mathcal{K},\mathcal{V})\) denotes the KV cache within the current sliding window.

Equation (5 ) assigns each candidate chunk a score that associated with both its length and the importance of its two endpoints. Intuitively, a longer chunk preserves more contextual information, while important endpoints indicate that the chunk is likely to be important for the window context. Equation (6 ) selects the TopK important chunks under the maximum-length constraint.

3.2.0.1 Implementation Details.

During decoding, Token2Chunk is applied to the compressible region immediately after the bidirectional-attention scoring stage. Token2Chunk is a lightweight module, and its additional complexity is \(O(n)\), where \(n = |\mathcal{I}|\). Equation (5 ) and Equation (6 ) ensure that the number of additional KV pairs is bounded by the chunk budget \(\alpha\). If the selected chunks introduce fewer than \(\alpha\) additional KV pairs, we fill the chunk budget by selecting discrete KV pairs from the unselected positions in \(\mathcal{W}\setminus\mathcal{U}\) according to score \(\mathbf{A}_i\).

3.3 KvLLM: Periodic Compression with PagedAttention↩︎

We further adapt Kara to PagedAttention and develop KvLLM, an inference framework that supports applying KV cache compression during serving to improve decoding efficiency. To mitigate the concurrency-throughput inversion effect observed in Figure 1, KvLLM adopts a periodic compression policy that is aligned with Kara’s sliding-window design. Concretely, KvLLM maintains a global decoding-step counter from the beginning of inference and applies a fixed periodic compression schedule. At each scheduled compression, KvLLM selects a subset of running sequences and treats the trailing PagedAttention blocks of each selected sequence as a compression window, on which Kara is applied to compress the KV cache. The compression period is set to be longer than the window span, so that each invocation operates on newly generated, previously uncompressed tokens. By using periodic compression, KvLLM avoids frequent compression triggering across concurrent requests. In practice, this policy can further reduce the KV-cache memory footprint while improving overall output throughput.

4 Experiments↩︎

4.0.0.1 Benchmarks.

We evaluate on three mathematical reasoning benchmarks: MATH-500[16], AIME24[27], and AMC23. We use decoding andreport accuracy of zero-shot pass@1, computed as the fraction of problems answered correctly. We set the maximum generation length to 16,384 tokens for MATH-500 and AMC 2023, and 32,768 tokens for AIME 2024. We also include a long-context evaluation on Needle-in-a-Haystack (NIAH) benchmark.

4.0.0.2 Baselines.

We compare Kara against representative KV cache compression baselines, including SnapKV [8], ChunkKV [18], StreamingLLM [10], PyramidKV [28], and AdaKV [7]. For each baseline, we use its best-performing configuration as recommended by the original paper.

4.0.0.3 Configurations.

We conduct experiments on DeepSeek-R1-Distill-Llama-8B (R1-Llama-8B) [1], Qwen3-14B, and Qwen3-4B [3], using BF16 precision. For Kara, we sweep the window length \(|\mathcal{W}|\in\{256,384,512\}\) and set the buffer length \(|\mathcal{U}|\in\{32,64\}\). For Token2Chunk, we use a fixed maximum chunk size \(\gamma\) of \(8\) and set the additional chunk budget \(\alpha\) to \(16\) or \(32\). We evaluate retention ratios \(r\in\{20,30,40,50,60\}\%\), and apply the same configuration to all layers and attention heads. Since Kara does not strictly enforce a fixed compressed cache length, different sequences may result in various cache sizes. We ensure a fair comparison by first running Kara to obtain the compressed KV cache length for each sequence and setting the per-sequence KV budget of each baseline to match Kara under the corresponding setting. Note that we evaluate Kara’s reasoning pass@1 without the periodic compression policy for a fair accuracy comparison, since this policy is primarily designed to improve throughput in practical application, and we only activate it in throughput experiments.

4.1 Main Results↩︎

r0.43 0.1cm

0.008

4.1.0.1 Reasoning Performance.

Figure 3 reports the reasoning performance of Kara compared with KV cache compression methods under various retention ratios. For a fair comparison, we first run Kara to obtain the realized compressed KV cache length for each sequence, and then set the per-sequence KV budget of each baseline to match Kara under the corresponding setting. We observe that: 1) Kara preserves accuracy with no or only minor degradation relative to the vanilla LLM without compression. For example, on MATH-500 with Qwen3-14B, Kara maintains nearly unchanged accuracy at a 30% retention ratio. 2) Kara generally outperforms other compression methods across retention ratios. For example, on AMC23 with Qwen3-4B, Kara remains better than the baselines at a 20% retention ratio. 3) Kara outperforms AdaKV and PyramidKV in most cases, which allocate different memory budgets across layers and heads. In contrast, we use a unified configuration and retention ratio for all layers and heads, which is architecturally more compatible with tensor parallelism and PagedAttention in inference frameworks. Overall, the results demonstrate the effectiveness of Kara for decoding-time KV cache compression.

4.1.0.2 NIAH Performance.

To evaluate Kara’s ability to preserve context information under KV cache compression, we conduct Needle-in-a-Haystack (NIAH) experiment. In this experiment, a short needle statement is inserted into a long context at varying depths, and the model is queried to retrieve the needle. Figure 4 compares Kara with ChunkKV and AdaKV across different context lengths and insertion depths. We observe that Kara achieves higher retrieval accuracy than the baselines across most depths and context lengths. In particular, Kara reduces the accuracy drop that some methods exhibit at mid-context depths, indicating a less information loss. This behavior is consistent with Kara’s joint retention of discrete KV pairs and flexible-sized contiguous chunks within the window, which helps preserve relevant context that may be distributed across positions. Overall, the NIAH results support Kara’s effectiveness in retaining contextual information under compression.

4.2 Ablation Studies↩︎

4.2.0.1 Effects of Bidirectional Attention Scoring.

To evaluate the effectiveness of the sliding-window bidirectional attention scoring in Equation (3 ), we remove bidirectional attention and scores KV pairs in the compressible region using only the query states from the protected buffer \(\mathcal{U}\) (denoted as w/o Bi-Attention). As shown in Table 1, remove bidirectional attention consistently reduces reasoning accuracy. These results indicate that utilizing all query states in the window with bidirectional attention provides a stronger importance signal than buffer-only scoring, leading to better preservation of informative context under the same KV retention ratio.

4.2.0.2 Effects of Token2Chunk.

To assess the effectiveness of the Token2Chunk module in Equation (5 ), we remove Token2Chunk and retain KV pairs using only sliding-window bidirectional-attention scoring, selecting a target number of discrete KV pairs under the same retention ratio (w/o Token2Chunk). As shown in Table 1, removing Token2Chunk leads to lower performance under compression. These results show that expanding a subset of selected discrete KV pairs into flexible-sized contiguous chunks at arbitrary positions helps preserve additional semantic context and improves compression performance.

5 Related Work↩︎

KV cache compression[4], [9] aims to reduce inference-time memory usage while keeping performance degradation within an acceptable range, typically by retaining important KV pairs and evicting unimportant ones. Early compression methods, such as StreamingLLM [10], preserve KV pairs at fixed prefix positions, ignoring the varying importance of different tokens. Mainstream KV cache compression methods [6], [8], [24], [29], [30] leverage LLM-internal query signals to score historical KV pairs and typically achieve state-of-the-art compression performance. Nevertheless, we observe two limitations of score-based KV cache compression. First, most methods rely on a threshold-triggered policy for decoding-time compression, where compression is executed once the cache length reaches a predefined threshold. This policy may degrade throughput even when compression increases the number of concurrently running sequences. Moreover, most methods perform eviction over the entire KV cache, which may fully remove KV pairs from a contiguous span of context and exacerbate information loss. Some works mitigate issues induced by this policy by adopting delayed compression. For example, DMS [12] scores KV pairs within a recent sliding window and executes eviction after tokens leave the window. TRIM-KV [31] further introduces a time-decay mechanism on KV importance to avoid immediate eviction after leaving the window. However, these methods may still require frequent eviction operations under concurrency and may still fail to avoid eliminating KV pairs from long contiguous context spans. Second, existing methods often fail to preserve flexible-size context information distributed at arbitrary positions across the sequence. Most methods retain either isolated KV pairs or fixed-size chunks with rigid boundaries. Different from the above works, we score KV pairs using sliding-window bidirectional attention and introduce Token2Chunk to expand selected discrete KV pairs into flexible-sized chunks at arbitrary positions. Our sliding-window bidirectional attention scoring also shares a similar intuition with DeepSeek-V4’s heavily compressed attention [32], which directly merges the KV pairs of contiguous tokens within a long span into a single entry. We also develop the KvLLM inference framework equipped with a periodic compression policy to reduce KV cache overhead and improve throughput.

6 Conclusion↩︎

We propose Kara, a decoding-time sliding-window KV cache compression method that scores and retains KV pairs in the recent context using bidirectional attention and augments discrete retention with a Token2Chunk module for flexible-size chunks. We further integrate it into PagedAttention and develop KvLLM framework with periodic trailing-block compression to reduce KV-cache memory usage and improve throughput. In the future, we plan to combine KV offloading and KV retrieval to enable lossless KV cache compression, maintaining decoding performance with substantially smaller cache footprints.

7 Detailed Experimental Settings for Figure 1 and [fig:F2]↩︎

This section provides detailed settings for the simple experiment shown in Figure 1 and [fig:F2]. We run all experiments on the MATH-500 dataset using DeepSeek-R1-Distill-LLaMA-8B. In Figure 1, we vary the maximum number of decoding sequences (i.e., the maximum concurrency) from 16 to 256. On MATH-500, the model’s average generation length is approximately 3,000 tokens [6]. For vLLM, We use tensor parallelism with size 4, the maximum number of tokens per batch is set to 16,384 and the PagedAttention block size is set to 256 tokens and the maximum generation length per sequence is set to 8,192 token. These experiments are conducted on RTX 5090 GPU. For Figure [fig:F2], we visualize results from a certain layer and attention head.

8 Formula of TPOT and Throughput↩︎

Considering a simplified decoding setting with decoding batch size \(B\)⁴, and BF16 KV cache, and an FFN intermediate size set to four times the hidden size, then TPOT can be approximated as \[\label{Eq:TPOT} \text{TPOT} \approx \max \left( \frac{L(12BD^2 + 2BSD)}{\rho}, \frac{4LBSD}{\beta} \right),\tag{8}\] where \(\rho\) denotes the effective GPU compute throughput, \(D = Hd\) is the hidden dimension and \(\beta\) denotes the effective HBM bandwidth, and modern GPUs generally have a high compute-to-bandwidth ratio. The first term in Equation (8 ) corresponds to the computation cost of one decoding forward step, while the second term captures the memory access overhead of KV cache. The output throughput can be formulated as: \[\label{Eq:throughput} \text{Throughput} \approx \frac{B}{\text{TPOT}},\tag{9}\] From Equation 8 and 9 , we can observe that in long-context reasoning scenarios, the inference overhead may be dominated by memory access to the KV cache, leaving the inference memory-bound, and higher \(B\) generally enables higher throughput.

9 Detail Settings of Experiments↩︎

Kara experiments are running on 8 H200 GPUs and 8 H100 GPUs.

References↩︎