1 Introduction↩︎

Large language models (LLMs) have been widely adopted across a broad range of applications, including question answering [1]–[3], personal assistants [4], [5] and knowledge-intensive reasoning systems [6]. In practical deployments, these applications are increasingly powered by in-context learning (ICL) paradigms such as retrieval-augmented generation (RAG) [7]–[10], agent systems with long-term memory [11], and few-shot learning [12]. A common characteristic of these workloads is the continuous growth of input context length [13]–[15], as each request injects multiple long documents into the model input.

As context lengths increase, LLM inference becomes prohibitively expensive. Given an input sequence of length \(L\), inference incurs a compute-intensive prefill stage with \(O(n^2)\) complexity to construct key-value (KV) caches, followed by autoregressive decoding where each generation step attends to all cached tokens, resulting in \(O(L)\) memory storage and bandwidth costs. Consequently, long-context inference simultaneously stresses GPU compute, memory capacity, and memory bandwidth, motivating the need for effective KV cache reuse mechanisms in LLM serving systems.

KV cache reuse has therefore attracted significant attention. Existing approaches can be broadly categorized into three classes. 1) Prefix-based reuse exploits identical prefixes across requests and enables exact KV reuse under standard causal attention; however, its applicability is severely limited in ICL workloads where repeated documents rarely appear at the same position. 2) To go beyond prefix matching, training-free non-prefix methods attempt to concatenate independently computed KV caches with limited recomputation to mitigate context mismatch, reducing prefill computation while preserving the original model. 3) In contrast, training-based approaches modify the attention structure or introduce auxiliary tokens during pretraining to enable more flexible reuse or compression, at the cost of altering the base model or introducing information bottlenecks.

While existing KV cache reuse techniques are effective at reducing recomputation, they largely overlook a growing bottleneck in long-context LLM inference: the cost of storing and accessing KV caches. In practical deployments, KV caches are often stored outside high-bandwidth accelerator memory and transferred on demand, making KV cache size and memory bandwidth critical factors as context length increases. As a result, previous works overlook a critical challenge in real-world LLM serving systems: the dominant bottleneck has shifted to KV cache storage and memory bandwidth.

This naturally motivates incorporating KV compression [16]–[20] into non-prefix KV reuse. However, this exposes two fundamental challenges. First, most existing compression schemes produce KV representations that are tightly coupled to their original context and thus non-composable, leading to significant accuracy degradation when reused or concatenated across different contexts. Second, there exists an architectural mismatch: long-context serving requires sequence-level compression, whereas many token-eviction methods rely on layer- or head-dependent selection, preventing consistent alignment once contexts are recomposed. As a result, even under the most reuse-compatible setting, naive compression remains fundamentally incompatible with non-prefix KV reuse, motivating a composability-aware design.

To resolve this tension, we propose \(\text{C}^2\text{KV}\), a unified framework for non-prefix KV reuse that jointly optimizes the KV extraction and inference-time concatenation, as shown in Figure 1. Unlike prior methods that directly compress or cache the original KV states, which are inherently context-dependent, \(\text{C}^2\text{KV}\) constructs a Composable and Compressed KV cache manifold. This manifold is specifically designed to be position-agnostic, allowing KV segments of various documents to be flexibly “plugged and played” at arbitrary positions within a long-context prompt.

The core of \(\text{C}^2\text{KV}\) is its \(\text{C}^2\)Extractor, a lightweight sidecar module that operates independently of the frozen base LLM. Unlike traditional fine-tuning, this “attached” module performs extraction without perturbing the model’s original parametric knowledge. To achieve efficient reuse, the \(\text{C}^2\)Extractor injects learnable \(\text{C}^2\)Tokens as compressed memory slots, whose interactions are governed by a novel Structured Attention Flow. This mechanism serves a dual purpose: first, it acts as a high-fidelity semantic aggregator that maximizes information retention via localized grouping; second, it ensures the resulting KV representations are modular and position-independent. This structured flow inherently facilitates seamless downstream concatenation while eliminating semantic cross-talk.

To further fortify the robustness of these compressed representations, we employ a Compression-Concatenation Co-training strategy. By supervising the model on generation tasks conditioned on concatenated KV segments, we implicitly steer the \(\text{C}^2\)Extractor to produce “merge-ready” states that maintain semantic coherence even when reordered. Consequently, \(\text{C}^2\)KV enables aggressive compression with minimal fidelity loss, significantly alleviating the storage and bandwidth bottlenecks of long-context inference.

Our main contributions are summarized as follows:

• We show that the key challenge of KV cache reuse lies in storage and memory transfer overheads, whereas existing approaches only reduce recomputation and ignore the storage overhead.

• We propose a unified training formulation that jointly optimizes KV extraction and inference-time KV concatenation, explicitly aligning the extracted representations with their reuse behavior under non-prefix inference.

• We introduce a new attention information flow through a carefully designed attention mask to preserve generation accuracy when concatenating compressed KV caches.

• Extensive experiments show that our method significantly reduces KV cache storage and transfer cost, achieving up to 17\(\times\) inference speedups over existing approaches, without degrading the inference quality.

2 Preliminaries and Motivation↩︎

2.1 KV Cache in LLM Inference↩︎

The Transformer-based Large Language Model (LLM) generates tokens auto-regressively. To avoid redundant computations, Key-Value (KV) caching stores the intermediate attention states. Formally, for a sequence \(\{x_1, \dots, x_t\}\), the KV cache at layer \(l\) is:

\[\mathbf{K}_l = [k_1, \dots, k_t], \quad \mathbf{V}_l = [v_1, \dots, v_t].\]

During the generation of \(x_{t+1}\), the model only computes the KV pair for the new token and concatenates it with the existing cache. While accelerating standard inference, the sequential dependency of self-attention poses a challenge for reusing KV blocks in different contexts.

2.2 Generalized KV Cache Reuse↩︎

Existing LLM serving systems primarily utilize Prefix Caching (e.g., vLLM [21], RadixAttention [22]), which allows the reuse of KV states if the leading sequence of a new request exactly matches a previously cached one. This is achieved by simple position-aligned mapping. However, in complex scenarios such as multi-document RAG [23] or modular tool-use [24], reusable content (e.g., \(\text{Doc}_\text{A}, \text{Doc}_\text{B}\)) may appear in arbitrary orders or be interleaved with dynamic instructions. To address this, we move beyond prefix matching and consider a Non-Prefix Caching paradigm.

We define a generalized two-stage framework for non-prefix reuse. Given a request with a static context \(\mathbf{x}^{\text{s}}\), several independent cached documents \(\{\mathbf{x}^{\text{d}}_1, \dots, \mathbf{x}^{\text{d}}_K\}\), and a query \(\mathbf{x}^{\text{q}}\), the process is formalized as:

• Encoding: An encoder \(\mathcal{M}\) (typically the base LLM itself or a specialized module) extracts KV representations from documents: \[\mathbf{KV}^{\text{reuse}}_k = \mathcal{M}(\mathbf{x}^{\text{d}}_k).\]

• Blending: A blender \(\mathcal{B}\) adapts and integrates these pre-computed states into the new request context: \[\mathbf{KV}^{\text{total}} = \mathcal{B}(\mathbf{x}^{\text{s}}, \mathbf{KV}^{\text{reuse}}_1, \dots, \mathbf{KV}^{\text{reuse}}_K).\]

• Inference: The LLM computes the final output: \[y = \mathcal{F}([\mathbf{KV}^{\text{total}}, \mathbf{x}^{\text{q}}]).\]

Based on the implementation of \(\mathcal{M}\) and \(\mathcal{B}\), current methods fall into two categories:

• Training-Free Methods: These methods (e.g., CacheBlend [25], KVShare [26]) use the vanilla LLM as the encoder and employ selective recomputation as the blender to reconcile mismatched KV states.

• Training-Based Methods: These methods (e.g., Block-Attention [27], KVLink [28]) modify the model’s architecture or fine-tune its weights so that it natively learns to process disjoint or reordered KV blocks.

2.3 Limitations of Existing Reuse Methods↩︎

Despite their potential, both categories of methods face critical bottlenecks that prevent efficient and lossless serving.

2.3.1 The KV Deviation Gap in Training-Free Methods.↩︎

In training-free methods, the blender \(\mathcal{B}\) identifies a subset of tokens \(M(\mathbf{x})\) to recompute. However, since the remaining unrecomputed KV states were generated in a different context, there exists an inherent KV deviation (\(\Delta \mathbf{KV}\)): \[\Delta \mathbf{KV} = \bigl\lVert \mathbf{KV}^{\text{full}} - \mathbf{KV}^{\text{blend}} \bigr\rVert\;/\;\bigl\lVert \mathbf{KV}^{\text{full}}\bigr\rVert \ge \delta > 0.\] As shown in Figure 2, this deviation monotonically increases as the recomputation ratio decreases, creating a rigid trade-off between serving latency and model accuracy.

2.3.2 Generalization Collapse in Training-Based Methods.↩︎

Training-based methods attempt to eliminate this gap by altering the self-attention mechanism, which suffers from:

• Capability Loss: Altering attention patterns often leads to “catastrophic forgetting”. Our empirical study shows a Llama3.1-8B model fine-tuned with Block-Attention suffers a performance drop of up to \(10.4\%\) on LongBench.

• Inflexibility: The high cost of per-model fine-tuning makes it difficult to adapt these methods to the rapid iteration of open-source base models.

2.3.3 Storage and Bandwidth Bottlenecks.↩︎

Crucially, prior methods primarily focus on reducing blending computation, while overlooking the fact that modern long-context serving is increasingly memory-bound. As context length \(L\) grows, the KV cache size scales linearly, making the transfer of uncompressed KV states between memory hierarchies (e.g., from DRAM to HBM) the dominant latency factor. As illustrated in Figure 3, existing reuse methods reduce prefill cost by loading cached KVs from memory, but still do not address this storage pressure; moreover, naively applying KV compression to reused blocks leads to severe fidelity loss, as standard compressed representations are not designed to be composable or position-agnostic.

3 \(\text{C}^2\text{KV}\) Framework↩︎

3.1 Framework Overview↩︎

We propose a non-prefix KV cache reuse framework \(\text{C}^2\text{KV}\) that enables long documents to be compressed, stored, and directly reused at arbitrary positions. \(\text{C}^2\text{KV}\) augments a frozen base LLM with a lightweight trainable module, together forming the \(\text{C}^2\)Extractor, which produces compact and composable KV representations.

The pipeline consists of two stages. First, given a set of documents, the \(\text{C}^2\)Extractor encodes each document independently into a compressed, position-agnostic KV cache, referred to as \(\text{C}^2\)KV. The \(\text{C}^2\)Extractor employs Structured Information Flow to preserve the base model’s semantics while enabling compression. Second, at inference time, the requested \(\text{C}^2\)KV caches are retrieved, assigned positional embeddings according to their designated position, and directly concatenated to condition generation.

To ensure the validity of the extracted KV caches after concatenation, the \(\text{C}^2\)Extractor is trained using Compression-Concatenation Co-Training, jointly optimizing compression and generation under a single supervised objective. This design decouples document encoding from query-time generation, enabling efficient non-prefix reuse with reduced KV cache size and minimal impact on generation quality. An overview of the pipeline is illustrated in Figure 4.

3.2 \(\text{C}^2\)Extractor↩︎

The \(\text{C}^2\)Extractor is a lightweight module attached to a frozen base language model. It takes a document as input and produces a compressed, composable key-value representation, whose length is reduced by a \(k\!:\!1\) ratio. Given a document token sequence, the \(\text{C}^2\)Extractor outputs per-layer key-value pairs, referred to as \(\text{C}^2\)KVs, which are stored and later retrieved for downstream generation.

3.2.1 \(\text{C}^2\)Tokens as Memory Slots↩︎

Compression is achieved by introducing a sequence of auxiliary tokens, referred to as \(\text{C}^2\)Tokens, into the document token stream. Given a compression ratio k, the number of \(\text{C}^2\)Tokens is determined as m = \(\lceil n / k \rceil\). The \(\text{C}^2\)Tokens are logically interleaved with the document tokens such that one \(\text{C}^2\)Token is associated with each consecutive block of \(k\) document tokens. All \(\text{C}^2\)Tokens share the same learnable embedding vector and do not correspond to any lexical content.

3.2.2 New QKV Projection Heads↩︎

The core design of the \(\text{C}^2\)Extractor is a set of new, per-layer QKV projection heads that operate exclusively on the \(\text{C}^2\)Tokens. For each Transformer layer \(\ell\), the base model’s original QKV projections are kept frozen and are applied only to the original document tokens. In contrast, \(\text{C}^2\)Tokens are projected using separate trainable matrices: \[Q^{(\ell)}_{\text{C}^2} = W^{(\ell)}_{{\text{C}^2},Q} h^{(\ell)}_{\text{C}^2}, \quad K^{(\ell)}_{\text{C}^2} = W^{(\ell)}_{{\text{C}^2},K} h^{(\ell)}_{\text{C}^2}, \quad V^{(\ell)}_{\text{C}^2} = W^{(\ell)}_{{\text{C}^2},V} h^{(\ell)}_{\text{C}^2},\] where the projection dimensions exactly match those of the base model, ensuring compatibility with downstream attention operations.

The projected \(\text{C}^2\)QKV tensors are concatenated with the frozen QKV tensors of the original document tokens and passed into the standard attention operator. Only the parameters of the \(\text{C}^2\)Token embeddings and their associated QKV projection heads are trainable; all base model parameters remain frozen. This design adds around \(10\%\) of additional parameters to the base model for Qwen3-4B and Llama3.1-8B.

3.2.3 Residual Mean-Pooling Variant↩︎

We also consider a variant of the Extractor that incorporates residual semantic aggregation, denoted as \(\text{C}^2\)KV-Residual. In this variant, at the first Transformer layer, we compute the mean of the hidden states of the k document tokens associated with a given \(\text{C}^2\)Token and add it to the \(\text{C}^2\)Token’s hidden state: \[h^{(\ell)}_{\text{C}^2} \leftarrow h^{(\ell)}_{\text{C}^2} + \frac{1}{k} \sum_{i \in \mathcal{B}} h^{(\ell)}_i,\] where \(\mathcal{B}\) denotes the corresponding document token block. The scaling factor is fixed to \(1.0\). This residual pathway provides an explicit aggregation of local document semantics and improves robustness under compression.

3.2.4 Output and Reuse↩︎

After extraction, only the KV pairs associated with the \(\text{C}^2\)Tokens are retained as \(\text{C}^2\)Extractor KVs. During inference, the retrieved KVs are applied new positional embeddings according to their placement in the final input sequence, where positional offsets are accumulated across concatenated segments and each compressed KV segment is treated as occupying the same effective span as its original uncompressed tokens. The compressed KV segments are then directly concatenated and used to condition downstream generation, without reintroducing the original tokens.

3.3 Structured Information Flow↩︎

The goal of Structured Information Flow (SIF) is to enable semantic extraction and compression without altering the behavior of the frozen base model. To this end, we impose a block-structured attention pattern that strictly separates the information pathways of Original Tokens and \(\text{C}^2\)Tokens during extraction.

3.3.1 Token layout and block structure↩︎

Given a document token sequence \(x_{1:n}\), we partition it into contiguous blocks of size \(k\). For each block \(\mathcal{B}(j)=\{(j-1)k+1,\ldots,\min(jk,n)\}\), we introduce exactly one \(\text{C}^2\)Token \(c_j\). Conceptually, \(c_j\) is placed after all tokens in its corresponding block, so that standard causal attention suffices to enforce block-local visibility. For implementation convenience, all \(\text{C}^2\)Tokens are physically grouped after the document tokens, and logical block boundaries are enforced solely through the attention mask. Each \(\text{C}^2\)Token is assigned the same position index as the last token in its block. This positional assignment is used only during extraction; the resulting key-value pairs are stored before positional embeddings are applied.

3.3.2 Attention constraints↩︎

At every layer, we apply an attention mask that enforces the following constraints.

• Original-token invariance. Original Tokens are allowed to attend only to other Original Tokens under the standard causal mask. Attention from Original Tokens to any \(\text{C}^2\)Token is strictly disallowed: \[M_{o,c}=-\infty \quad \forall\, o\in\mathcal{O},\; c\in\mathcal{C}.\] As a result, the hidden states and KV caches of Original Tokens are identical to those produced by the frozen base model.

• Block-local semantic extraction. Each \(\text{C}^2\)Token \(c_j\) may attend only to the Original Tokens in its own block \(\mathcal{B}(j)\) and the sink block \(\mathcal{B}(0)\): \[M_{c_j,o}=0 \;\; \forall\, o\in\mathcal{B}(j)\cup\mathcal{B}(0),\; M_{c_j,o}=-\infty \;\; \forall\, o\notin\mathcal{B}(j)\cup\mathcal{B}(0).\] Since \(c_j\) is conceptually positioned at the end of the block, causal masking guarantees that it can attend to all tokens within \(\mathcal{B}(j)\) while preventing access to tokens beyond the block boundary. This enforces a \(k\!:\!1\) structure and prevents global information leakage during extraction.

• Causal accumulation across blocks. To allow compressed representations to accumulate document-level context, \(\text{C}^2\)Tokens may attend to preceding \(\text{C}^2\)Tokens causally. This enables later blocks to incorporate information from earlier compressed segments, while preserving autoregressive consistency.

3.3.3 Comparison with Anchor Token-based Compression.↩︎

Figure 6 contrasts our Structured Attention Flow with anchor token-based compression methods, which also interleave auxiliary tokens into the input sequence. While both approaches introduce extra tokens, they serve fundamentally different purposes. Anchor tokens are designed as semantic bottlenecks for context compression and rely on bidirectional attention between original tokens and anchor tokens. As a result, original token representations are modified through anchor tokens, and the resulting KV cache is inherently entangled with the full input context.

In contrast, \(\text{C}^2\)Tokens are introduced as latent KV carriers rather than semantic bottlenecks. Information flows only from original tokens to \(\text{C}^2\)Tokens in a block-local and causal manner. This asymmetry is critical: it allows the extracted \(\text{C}^2\)KVs to be context-independent, composable, and directly reusable across requests. Consequently, although anchor token-based methods are effective for compressing context within a single input, they are fundamentally incompatible with non-prefix KV cache reuse. Our design replaces semantic compression with structural KV extraction, enabling reuse by construction rather than approximation.

3.4 Composable KV Learning via Compression-Concatenation Training↩︎

The objective of compression-concatenation training is to ensure that document-level compression and KV cache concatenation are , so that independently extracted KV caches remain valid and useful after being composed in arbitrary multi-document contexts.

3.4.1 Training objective↩︎

The entire model is trained using a single supervised fine-tuning objective on the generated answer. Let \(\mathbf{K}_{\text{cat}}, \mathbf{V}_{\text{cat}}\) denote the concatenated document KV cache, and let \(q_{1:m}\) and \(y_{1:T}\) denote the query and answer tokens. The training loss is the standard autoregressive language modeling loss: \[\mathcal{L}_{\text{SFT}} = - \sum_{t=1}^{T} \log p\bigl(y_t \mid y_{<t}, q_{1:m}, \mathbf{K}_{\text{cat}}, \mathbf{V}_{\text{cat}}\bigr).\] No auxiliary reconstruction loss, alignment loss, or document-level supervision is introduced. Gradients are propagated exclusively to the parameters of the \(\text{C}^2\)Extractor, including the \(\text{C}^2\)Token embeddings and their dedicated QKV projection heads; all base model parameters remain frozen.

3.4.2 Why concatenation is learned↩︎

Although documents are extracted independently, supervision is applied only after concatenation. Each document’s \(\text{C}^2\)KV cache is optimized based on its contribution to answering the query in the presence of other documents. Since document order and the number of documents vary across training samples, the Extractor is forced to project document semantics into a KV cache manifold that remains valid under different compositions. Importantly, compression and concatenation are not optimized in isolation. A document KV cache that compresses too aggressively but fails after concatenation will incur higher generation loss, as will a KV cache that only functions in a fixed global context. Through end-to-end supervision, the Extractor learns a representation that balances both requirements and maps document tokens into a compression concatenation enabled KV manifold.

4 Evaluations↩︎

4.1 Experiment Setup↩︎

4.1.1 Dataset↩︎

We train the \(\text{C}^2\)Extractor using multi-document supervised fine-tuning (SFT) question-answering data, while keeping all Base Model parameters frozen. The training corpus is constructed from the training splits of three publicly available QA datasets: HotpotQA [29], 2WikiMultiHopQA [30], and LongMagpie [31]. Together, these datasets cover a wide range of multi-document reasoning scenarios, making them suitable for learning document-level KV extraction under non-prefix reuse settings.

Each instance is formatted using the target model’s official chat template, including a system prompt, documents, user question, and reference answer. To improve formatting robustness and reduce verbosity, both system prompts and QA instructions are randomly sampled from a small prompt pool. During extraction, document tokens do not attend to system tokens; system KVs are concatenated only at generation time.

For evaluation, we use the LongBench [32] benchmarks. For the single-document subsets in LongBench, we partition each document into natural paragraphs and treat them as multiple documents, enabling evaluation of non-prefix KV reuse.

4.1.2 Metrics↩︎

We adopt task-appropriate evaluation metrics following prior works. For question-answering tasks, we report the F1 score, which measures token-level overlap between the predicted answers and the ground-truth references. For summarization tasks, we report ROUGE-L, which evaluates sequence-level similarity based on the longest common subsequence between the generated and reference summaries.

4.1.3 Models↩︎

We conduct experiments using three widely adopted instruction-tuned large language models: Qwen3-4B-Instruct-2507 [33], Llama-3.1-8B-Instruct [34], and Qwen2.5-7B-Instruct [35]. These models cover different architectures and parameter scales, allowing us to evaluate the robustness of our method across model families.

4.1.4 Baselines↩︎

We compare \(\text{C}^2\text{KV}\) against 5 baselines:

• Full KV Recompute (FR): The model performs a full prefill for every request without any KV reuse, serving as an accuracy and computation upper bound;

• Full KV Reuse (Reuse): KV caches of reusable documents are directly reused without correction or retraining, representing an aggressive but error-prone lower bound;

• EPIC [36]: A selective recompute reuse method that choose tokens by a static pattern;

• CacheBlend [25]: A selective recompute reuse method that recomputes tokens by value differences to mitigate errors.

• Block-Attention [27]: A training-based baseline by continue-training with no cross-attention between documents.

4.2 TTFT-Accuracy Trade-off↩︎

4.2.1 Experiment setup.↩︎

In this experiment, we focus on Time-to-First-Token (TTFT) under a document-reuse serving scenario, where a document corpus is available and document KV caches can be pre-extracted offline. TTFT is defined as the time elapsed from receiving a user request to the moment when the language model has its past KV cache fully prepared on GPU and is ready to start decoding the first output token.

Under this definition, different methods incur TTFT from different sources:

• Full Recompute (FR) must prefill the entire document context online and thus its TTFT corresponds to the full-context prefill latency.

• Non-prefix reuse methods load precomputed document KVs from host memory and additionally perform blending or structural transformations before decoding.

• \(\text{C}^2\text{KV}\) loads compressed document KVs from host memory and directly reassigns positions without any blending, reducing TTFT to a load-only operation.

Although different reuse methods may employ different KV compression ratios, this difference reflects their inherent design goals. In particular, \(\text{C}^2\text{KV}\) jointly performs KV compression (4\(\times\)) and reuse.

4.2.2 TTFT-Accuracy Trade-off↩︎

Figure 7 shows the trade-off between time-to-first-token (TTFT) and task accuracy across seven LongBench tasks and three instruction-tuned LLMs. Each point corresponds to the average TTFT and task score of a method on the full test set, where lower TTFT and higher accuracy are preferable. This analysis reveals how different KV reuse strategies balance startup latency against generation quality:

  While FR consistently achieves the highest accuracy by prefilling the entire context online, it suffers from prohibitive TTFT. This establishes a performance ceiling for accuracy but highlights the critical need for efficient KV reuse in long-context workloads.

  Prior non-prefix reuse approaches reduce TTFT (shifting leftward in the plot) but introduce substantial accuracy instability. Methods like EPIC and CacheBlend, which rely on KV blending, require complex preparation and exhibit sensitivity to architectural changes, leading to degraded performance in multi-document reasoning.

  \(\text{C}^2\text{KV}\) achieves minimal TTFT and competitive accuracy simultaneously, occupying the optimal upper-left region of the trade-off plot. Its learnable, composable KV representation transforms preparation into a load-only operation, eliminating errors from blending. While the \(\text{C}^2\text{KV}\)-Residual variant improves stability on specific tasks, the vanilla \(\text{C}^2\text{KV}\) delivers the best overall efficiency-performance balance.

Finally, we note that this experiment isolates TTFT, which dominates user-perceived latency for long-context requests. Together with the decode-time results presented in the next section, these findings show that \(\text{C}^2\text{KV}\) improves both request startup latency and steady-state generation efficiency.

4.3 Decode-Time Efficiency↩︎

We evaluate decode-stage efficiency using average time per token (TBT), isolating memory bandwidth costs by excluding prefill. All methods use identical decoding settings with KV reuse applied before decoding, and results are shown in Figure 8. Our findings are summarized as follows.

  As context length grows from 16k to 128k tokens, per-token decode latency increases for all methods due to longer attention contexts. Full-length KV caches exhibit near-linear scaling, indicating severe memory bandwidth pressure. In contrast, \(\text{C}^2\text{KV}\) compresses the KV footprint and largely flattens the decode-time curve, showing only a mild increase even at 128k tokens across all models. Shrinking the effective KV cache size directly translates into lower time-between-tokens (TBT) under long-context decoding.

  Beyond eliminating redundant prefill computation, \(\text{C}^2\text{KV}\) delivers sustained performance benefits during decoding, where KV access dominates user-perceived latency. These results highlight that non-prefix KV reuse, when combined with effective KV compression, yields system-level gains throughout the entire inference pipeline.

4.4 Compression Ratio Scaling↩︎

We further evaluate how \(\text{C}^2\text{KV}\) scales under different KV compression ratios on Llama3.1-8B. Table 1 compares \(\text{C}^2\text{KV}\) with existing non-prefix reuse methods under both fixed and dynamic compression settings.

  As the compression ratio increases from \(4\times\) to \(16\times\), \(\text{C}^2\text{KV}\) exhibits graceful degradation and remains substantially more robust than existing reuse methods. Even under aggressive compression, \(\text{C}^2\text{KV}\) preserves competitive performance across both QA and summarization tasks, demonstrating that the extracted KV representations remain stable and composable under large sequence reduction ratios.

  We further train a single model by randomly sampling compression ratios from \(\{4\times, 8\times, 16\times\}\) at each training step. The resulting model generalizes effectively across different inference-time compression budgets, including the unseen \(10\times\) setting. Compared with fixed-ratio training, dynamic-ratio training consistently improves robustness and achieves stronger overall performance across varying compression levels.

Overall, these results demonstrate that \(\text{C}^2\text{KV}\) provides a unified framework for composable KV reuse and scalable KV compression, while maintaining strong accuracy across a wide range of compression budgets.

4.5 Long-Context Evaluation on RULER↩︎

We further evaluate \(\text{C}^2\text{KV}\) on the RULER benchmark under 4\(\times\) KV compression. To support long-context evaluation, each input context is split into segments of approximately 4k tokens before extraction and KV reuse. Figure 9 reports the average retrieval score across different context lengths and answer depths.

  \(\text{C}^2\text{KV}\) maintains stable retrieval performance as the context length increases from 4k to 64k tokens, demonstrating effective long-range information preservation under compressed KV reuse.

  Despite operating with 4\(\times\) compressed KV caches, \(\text{C}^2\text{KV}\) achieves only limited degradation compared with the original models, showing that composable KV compression can preserve strong long-context reasoning capability.

4.6 Scaling to Larger Models↩︎

To evaluate whether \(\text{C}^2\text{KV}\) scales to larger model sizes, we further conduct experiments on Qwen3-14B and report the results in Table 2. We compare \(\text{C}^2\text{KV}\) against existing non-prefix reuse baselines under multiple compression ratios.

The results show that \(\text{C}^2\text{KV}\) consistently maintains strong performance across all tasks while enabling substantial KV compression. In particular, \(C^2\)KV-4x-Res achieves accuracy close to or exceeding the full-context baseline on MuSiQue, SAMSum, and MultiNews, while significantly outperforming existing reuse methods on multi-document QA tasks such as WikiMQA. Moreover, the performance degradation of \(\text{C}^2\text{KV}\) remains gradual as the compression ratio increases from \(4\times\) to \(16\times\), demonstrating favorable scaling behavior under aggressive compression settings.

These results indicate that the composable KV representations learned by \(\text{C}^2\text{KV}\) generalize effectively to larger LLMs and remain stable under high compression ratios.

4.7 Does Compression Alone Suffice?↩︎

To validate the claim that naively combining KV reuse with generic compression degrades accuracy, we conduct a motivating experiment. Table 3 reports LongBench accuracy when a generic KV compression method, SnapKV (\(4\times\) compression on all layers), is applied prior to KV reuse strategies. Across all reuse methods, directly stacking SnapKV with KV reuse results in severe accuracy loss. This pattern is consistent across tasks and models, showing that generic KV compression, while reducing cache size, is incompatible with non-prefix KV reuse without composability-aware adaptation.

These results motivate our design: compressing KV caches naively further distorts already non-composable KV representations, compounding errors during reuse. KV compression and reuse must therefore be jointly designed within a unified framework that produces KV representations both compressible and composable.

4.8 Ablation Study↩︎

We conduct ablation studies to validate the key designs of \(\text{C}^2\text{KV}\). Table 4 shows that replacing Structured Information Flow with bidirectional Anchor Tokens causes large accuracy degradation, indicating that unrestricted information mixing breaks KV composability. Directly using the untrained Extractor also fails severely, showing that composable KV representations must be learned.

  \(\text{C}^2\text{KV}\)(4x) Fixed-Pos. disables reuse-time positional re-rotation and consistently reduces accuracy, validating the necessity of position re-alignment after KV concatenation.

  Relaxing block-local constraints through Info-Leakage or Global-Info attention harms performance across all tasks. These results show that stable KV reuse requires strict structural isolation and localized information aggregation.

Overall, the results support the core principles of \(\text{C}^2\text{KV}\): structured extraction, learned composable representations, and reuse-time positional alignment.

5 Related Work↩︎

 Efficient KV cache reuse is critical for reducing Time-to-First-Token (TTFT) and computational redundancy. Prefix-based methods, such as PromptCache [37], PagedAttention [21] and Radix-Attention [22], cache KV states of common prefixes in a tree structure to avoid recomputation across multiple requests. To handle non-prefix scenarios, training-free approaches like CacheBlend [25] and Cache-Craft [38] enable arbitrary text chunk reuse via selective recomputation and KV state alignment, though they may suffer from slight accuracy degradation. Training-based methods, including TurboRAG [39] and Block-Attention [27], modify model architectures or employ fine-tuning to support modular cache stitching (e.g., independent attention blocks). While these offer better generation quality, they require significant training overhead and risk compromising the base model’s general capabilities.

  To mitigate LLM memory bottlenecks, KV cache compression reduces per-token storage. Quantization methods like CommVQ [40] or MiniKV [41] achieve ultra-low bit-precision (e.g., 2-bit) while preserving accuracy. Eviction strategies, such as SnapKV [42] and RocketKV [43], identify and retain crucial tokens via attention-driven scores or reconstruction importance. Furthermore, Merging and Low-rank approaches like MiniCache [44] or ReCalKV [45] consolidate redundant KV states across layers or attention heads. These techniques maximize memory efficiency and, when paired with reuse, significantly scale long-context processing.

6 Conclusion↩︎

In this work, we identify KV cache storage and memory bandwidth as the key bottlenecks in non-prefix KV reuse for long-context LLM inference. We show that existing reuse and compression methods are fundamentally limited by the non-composability of standard KV representations. To address this, we propose \(\text{C}^2\text{KV}\), a unified framework that learns a composable and compressed KV cache manifold through a lightweight extractor and structured attention, while keeping the base model frozen. Extensive experiments demonstrate that \(\text{C}^2\text{KV}\) enables efficient KV reuse and compression without degrading generation quality, substantially reducing inference overhead under long contexts.

Acknowledgements↩︎

This work was sponsored in part by China NSF grant No. 62441236, 62472278, 62432007, 62332014, 62332013, and 62372296, and Shanghai QiYuan Innovation Foundation. This work was partially supported by Alibaba Group through the Alibaba Innovation Research Program and in part by SJTU Kunpeng & Ascend Center of Excellence. The opinions, findings, conclusions, and recommendations in this paper are those of the authors and do not necessarily reflect the views of the funding agencies or the government.

7 Appendix Outline↩︎

In this appendix, we report the following sections as supplementary materials for the main paper.

• Appendix 8 shows the dataset setup for training and evaluation.

• Appendix 9 shows the detailed TTFT measurement protocol.

• Appendix 10 gives full results for accuracy evalution.

• Appendix 11 lists the limitations and future work.

8 Dataset Setup↩︎

8.1 Training Setup↩︎

We train the \(\text{C}^2\)Extractor using supervised fine-tuning on multi-document instruction-following data sampled from HotpotQA, 2WikiMQA, and LongMagpie. We sample 40k instances from each dataset, resulting in 120k total training samples, and train for one epoch only. For LongMagpie, long passages are segmented into multiple documents using sentence- or paragraph-aware splitting.

The base LLM remains frozen throughout training. Only the C\(^2\) token embeddings and per-layer QKV projection heads are trainable. We use AdamW with cosine learning rate scheduling under the following settings: Learning rate: 5e-5, Warmup ratio: 0.06, Weight decay: 0.1, Global batch size: 32, Gradient accumulation: 4 and Precision: BF16.

8.2 Evaluation Setup↩︎

We evaluate \(\text{C}^2\text{KV}\) on LongBench datasets covering multi-document QA and long-form summarization, including HotpotQA, 2WikiMQA, MuSiQue, MultiNews, SAMSum, QMSum, and GovReport.

For QA tasks, we report F1 following official evaluation protocols. For summarization tasks, we report ROUGE-L. All methods use greedy decoding, with maximum generation lengths of 32 tokens for QA and 512 tokens for summarization.

9 Detailed TTFT Measurement Protocol↩︎

This section formalizes the protocol used to measure Time-to-First-Token (TTFT) in our system-level evaluations.

9.1 Definition and Timing Boundary↩︎

TTFT is defined as the elapsed wall-clock time from when a user request arrives to when the language model has its past key–value cache fully prepared on GPU and is ready to start decoding the first output token.

We explicitly exclude the prefill cost of system prompts and query tokens from TTFT for all methods, as their lengths are short and negligible relative to long-context document processing. Offline document KV extraction time is also excluded. All timing measurements are synchronized using torch.cuda.synchronize() to ensure accurate GPU timing.

9.2 Method-Specific TTFT Components↩︎

Under this definition, TTFT is measured as follows:

• Full Recompute (FR): TTFT equals the wall-clock time of performing a full prefill over the entire document context to construct the KV cache, since non-prefix document reuse is not supported.

• Load+Blend Methods: TTFT is the sum of loading precomputed document KVs from host memory to GPU and executing GPU-side KV blending or structural transformation kernels. These components are profiled separately and summed.

• \(\text{C}^2\text{KV}\): TTFT includes loading compressed document KVs from host memory to GPU and lightweight positional reassignment and RoPE re-rotation. No blending or recomputation is required.

10 Full Results↩︎

10.1 Accuracy Evaluation↩︎

We evaluate non-prefix KV cache reuse across multiple models and tasks under a unified setup. KV reuse is applied only to document tokens, while query and generation tokens are always computed online. Block-Attention results are reported only for Llama-3.1-8B-Instruct due to checkpoint availability.

As shown in Table 5, \(\text{C}^2\text{KV}\) maintains strong accuracy under \(4\times\) KV compression, consistently approaching Full KV Recompute (FR) across models and tasks. This demonstrates that non-prefix KV reuse can remain accurate when using composable KV representations.

In contrast, directly concatenating standard KV caches causes severe degradation on multi-document QA tasks, revealing the non-composability of vanilla self-attention KV representations. Training-free reuse methods such as EPIC and CacheBlend partially improve over naive reuse but exhibit unstable performance across models and datasets.

Under the same compression ratio, \(\text{C}^2\text{KV}\) achieves the best or near-best accuracy among reuse-based approaches on most tasks. Compared with generic KV compression methods such as SnapKV, \(\text{C}^2\text{KV}\) attains comparable or better accuracy while additionally supporting composable KV reuse.

11 Limitations and Future Work↩︎

While \(\text{C}^2\text{KV}\) effectively enables compressed and composable non-prefix KV reuse, it has several limitations that suggest directions for future work. First, \(\text{C}^2\text{KV}\) focuses on document-level reuse and assumes that reusable content can be identified and extracted offline. Extending the extractor to support online or incremental KV extraction is an important direction. Second, our current design applies a uniform compression ratio across documents and layers. Future work could explore adaptive or content-aware compression strategies that preserve the composability property while allocating compression budgets more flexibly.

References↩︎