1 Introduction↩︎

LLM serving is becoming increasingly memory-dominated. As workloads evolve from single-turn prompts to long-context, multi-turn, and agentic applications [1]–[4], inference cost is shaped not only by computation, but also by the memory capacity [5]–[9] to retain and reuse growing context state. The memory-capacity pressure created by modern LLM serving translates into expensive GPU and DRAM provisioning and lower serving efficiency. Specifically, KV cache reuse reduces repeated context-processing work by retaining the intermediate state generated for previously processed tokens, but it also turns the KV cache into reusable data whose capacity, movement, and sharing increasingly influence serving efficiency across turns, sessions, and workers.

This paper presents HyMCache, a KV-cache framework that integrates CXL-hybrid memory for multi-turn LLM serving. CXL-hybrid memory, hereafter also referred to as CXL-HM, is a CXL device that combines a small amount of in-device DRAM with large SSD-backed capacity. By seamlessly integrating CXL-HM into a modern LLM serving stack, HyMCache enables DRAM-scale KV-cache efficiency at SSD-level cost.

The memory-capacity pressure is already pushing LLM serving toward shared, storage-based context infrastructure. TB-scale and beyond reusable context capacity is difficult to support economically with HBM or DRAM alone, so publicly disclosed frontier systems are beginning to incorporate lower-cost, high-capacity storage media into the inference memory hierarchy. NVIDIA CMX (Context Memory Storage) introduces a dedicated context-memory tier for long-context and multi-agent inference [10], while DeepSeek Context Caching on Disk shows that disk-backed context reuse is being adopted in commercial LLM APIs [4]. These efforts show that KV-cache reuse is increasingly becoming a memory-tiering problem. Storage media provide the capacity and cost efficiency needed to retain large reusable contexts, while optimizations such as fast staging, prefetching, and data-movement control are required to meet the latency demands of context reuse in LLM serving systems.

To build such a shared context layer, the ecosystem is exploring both storage-centric and memory-centric designs. On the storage side, networking and DPU vendors such as NVIDIA [11], [12], Broadcom [13], Marvell [14], [15], Xsight Labs [16], and Chelsio [17] are combining RDMA/NVMe-oF, SmartNICs, and DPUs with dense SSDs from storage vendors such as SK hynix [18] and Samsung [19] to build remote flash tiers [20]–[23]. On the memory side, memory vendors, cloud providers, and startups, including SK hynix [24], [25], Samsung [26], [27], Alibaba [28], Liqid [29], UniFabriX [30], and XCENA [31] are developing CXL-based memory expansion, rack-scale memory pooling, and CXL memory appliances.

Despite this broad momentum, the most practical architecture remains unclear. Storage-backed tiers provide large capacity at lower cost, but still require storage-node software to stage SSD-resident KV blocks through DRAM buffers before remote access. DPU-based offload can reduce part of this overhead, but adds software complexity and non-trivial appliance cost. In contrast, DRAM-based memory pools provide a cleaner low-latency memory abstraction, but scaling them to tens or hundreds of terabytes is costly.

As a middle ground, major memory vendors such as SK hynix [32] and Samsung [33] have proposed CXL-hybrid memory, or CXL-HM, as a compromise between DRAM-only memory expansion and SSD-based capacity expansion. CXL-HM exposes the SSD-backed capacity to the host through a memory interface (i.e., CXL.mem), while using the device-side DRAM transparently as an internal cache. This makes it an attractive substrate for remote KV caching: the SSD-backed region provides TB-scale capacity at lower cost, while the internal DRAM can serve latency-critical accesses when managed appropriately.

However, we find that generic internal resource management including cache-management policies, such as the LRU policy used in commercial CXL-HM devices, are insufficient for agentic LLM serving. We therefore design and prototype a CXL-HM tailored to the workload characteristics of multi-turn, agentic LLM serving. To exploit the read-dominant, predictable, and cross-turn reuse patterns of these workloads, HyMCache provides an API that allows the serving stack to prefetch KV blocks into CXL-HM’s internal DRAM, thereby hiding retrieval latency from the SSD-backed region. We evaluate HyMCache under both single-aggregator and Prefill-Decode (PD) disaggregated serving configurations. With the same DRAM budget, HyMCache improves serving performance over local LMCache by 3.0\(\times\) in single-node serving and 1.45\(\times\) in PD-disaggregated serving with 64 GB per worker, 256 GB in total. Compared with 1 TB distributed-DRAM Mooncake, HyMCache trades a 30% performance gap for a 16\(\times\) reduction in DRAM usage.

2 Background and Motivation↩︎

2.1 LLM Serving and KV-Cache Tiers↩︎

LLM inference and KV cache. LLMs generate output tokens autoregressively: each new token is produced based on the input prompt and all previously generated tokens. In modern LLM serving systems, inference is commonly divided into two phases, prefill and decode [5]. During prefill, the model processes the input prompt and materializes key/value (KV) tensors for all prompt tokens. During decode, the model generates one token at a time and attends to the KV tensors produced for earlier tokens. The KV cache stores these intermediate tensors so that later attention computation can reuse them instead of regenerating them. Within a request, this avoids recomputing KV tensors during decode; across requests or turns, it also enables reusable prefixes to skip repeated context processing.

LLM serving stack. Modern LLM serving stacks combine a GPU-side serving engine with a distributed serving framework. Serving engines such as vLLM [5] manage request batching, scheduling, and GPU KV-cache allocation for high-throughput inference. vLLM organizes the GPU KV cache into fixed-size blocks and uses PagedAttention to improve memory utilization. Distributed frameworks such as Dynamo [34] build on these runtimes to support multi-worker serving, including Prefill–Decode (PD) disaggregation, request routing, and inter-worker KV transfer. In PD-disaggregated serving, prefill workers process input contexts and produce KV blocks, while decode workers continue token generation after receiving the required KV blocks. Dynamo uses NIXL [35] for the P–D KV-transfer path, and NIXL can use transport backends such as UCX [36] to run over RDMA-capable networks in cross-node deployments.

KV-cache tiers. KV-cache reuse is commonly implemented as a tiered memory hierarchy. One useful framing is NVIDIA’s G1–G4 context hierarchy [10]: G1 is GPU HBM, where serving engines such as vLLM manage resident KV blocks through GPU KV-cache allocation and prefix caching; G2 is system DRAM, used by systems such as LMCache and Mooncake to extend KV capacity within or across workers [6], [7]; G3 is local SSD, used as a lower-cost capacity tier through KV offloading or DRAM–SSD caching; and G4 is shared remote storage, where larger context data can be accessed through storage/network fabrics such as NVMe-oF, RDMA, or DPU-assisted data paths. Recent designs also introduce intermediate tiers and optimizations between these levels, such as G2.5 or G3.5 context-memory tiers, to bridge the latency–capacity gap between DRAM and storage. Overall, KV-cache tiering exposes a trade-off among latency, capacity, cost, energy, and data movement.

2.2 Available CXL Solutions and Trade-offs↩︎

Compute Express Link (CXL) has emerged as a memory interconnect that extends server memory capacity beyond local DIMMs while preserving a memory-like access model. The CXL specification has evolved from single-device attachment in CXL 1.1 to switch-based memory pooling in CXL 2.0 and fabric-oriented multi-level switching in CXL 3.0. Its sub-protocols separate device management and I/O semantics through CXL.io, cache-coherent accelerator access through CXL.cache, and load/store memory access through CXL.mem [37].

CXL creates a distinct latency–capacity point for KV-cache tiering, sitting much closer to DRAM than to storage while allowing memory capacity to scale beyond local DRAM. Importantly, commercial CXL platforms provide memory access latencies comparable to conventional NUMA systems. Local DRAM access is typically around 80–140 ns [38], remote NUMA access is roughly 2\(\times\) higher, and CXL memory expansion modules have been reported to be around 170–250 ns [38]–[40]. More recently, commercial CXL 2.0 switches have expanded the practical scope of CXL beyond single-node memory expansion toward large-scale shared memory pools. For example, the XConn (acquired by Marvell) CXL switch supports up to 256 lanes [41], while prior work reports roughly 750 ns minimal 64-byte I/O latency for an XConn-based CXL switch fabric [42]. Pooled-memory prototypes such as SK hynix Niagara further show that shared CXL memory can remain sub-microsecond, reporting 600 ns access latency [25].

The design of CXL memory modules can vary significantly depending on system and application requirements. For large shared KV-cache tiers, existing CXL memory solutions can be broadly organized into three design points: DRAM-based CXL memory expansion, switch-based CXL memory pooling, and CXL-HM.

A CXL memory expander is a DRAM-based CXL memory module that targets latency- and bandwidth-sensitive workloads by expanding memory capacity beyond local DIMM channels while preserving memory-semantic access [24], [26]. For KV-cache reuse, such devices can provide a low-latency capacity tier for KV blocks that no longer fit in GPU memory but still require fast access. A CXL switch-based memory pool extends this model to rack-scale environments by aggregating multiple CXL memory devices into a pooled memory appliance, allowing memory capacity to be flexibly allocated and shared across hosts [25], [27], [42]. This model is attractive for distributed LLM serving because reusable KV states can be shared across workers rather than being replicated in each server. In contrast, CXL-HM combines internal DRAM with high-capacity SSD, exposing SSD-backed capacity through a memory interface while using internal DRAM as a staging/cache layer [32], [33]. This design point is especially relevant for large KV-cache tiers because it provides much larger capacity than DRAM-only CXL devices at lower cost.

6pt

\(^{\dagger}\)The

cost is a rough estimate for four FPGA-based prototypes.

Recent literature and industry efforts have explored remote KV-cache tiers based on DRAM-backed CXL memory expansion and pooling, as illustrated in Figure 1 (a). Such systems aggregate multiple CXL memory expanders through a CXL switch, memory box, or fabric, allowing hosts to dynamically expand and share memory capacity. This direction is increasingly relevant for LLM serving, where large KV-cache states are retained and reused across requests to improve throughput and reduce redundant computation [5]–[7], [50], [51]. It has been pursued by memory vendors, cloud providers, startups, and system vendors, including SK hynix [24], [25], Samsung [26], [27], Alibaba [28], Liqid [29], UniFabriX [30], and XCENA [31], as well as research systems such as TraCT and Beluga [42], [52]. However, building large shared KV-cache tiers entirely from DRAM-based CXL memory expanders remains costly at scale. Although these designs provide high-performance memory-semantic access, Table 1 shows that provisioning a DRAM-only tier at roughly 15 TB capacity requires hundreds of thousands of dollars in RDIMM cost.

An alternative deployment path is to place CXL-HM behind an existing RDMA-capable datacenter network, as shown in Figure 1 (b). In this model, the CXL-HM device is attached to a remote node, while LLM workers access reusable KV blocks through RDMA rather than through a CXL fabric. This deployment does not require CXL switches or CXL fabric support on every worker node, making it more compatible with existing cluster infrastructure. Unlike conventional remote storage, the CXL-HM node exposes SSD-backed capacity as a memory-addressable tier, allowing workers to access KV blocks through RDMA operations rather than storage I/O.

For CXL-HM, even with the FPGA/controller overhead, Table 1 illustrates the potential of CXL-HM to expose large SSD-backed capacity through a memory interface without provisioning the same amount of DRAM. As reusable KV states scale to tens or hundreds of terabytes across serving clusters, the cost overhead of DRAM-only tiers becomes difficult to ignore. This motivates considering CXL-HM for KV-cache tiering, which retains a memory interface while using SSD-backed capacity to reduce the cost of large-capacity remote KV tiers.

2.3 CXL-HM for Multi-Turn KV-Cache Tiering↩︎

CXL-HM provides an opportunity to build large-capacity remote memory tiers at lower cost than DRAM-only CXL memory pools. Unlike DRAM-based CXL memory expanders, which are composed entirely of DRAM, CXL-HM combines a small amount of internal DRAM with high-capacity SSDs. CXL-HM exposes SSD-backed capacity as a host-addressable memory tier through a memory-semantic interface (i.e., CXL.mem), while internally staging data between internal DRAM and the SSD-backed region. As a result, CXL-HM can expose large SSD-backed capacity as a CPU-less NUMA node and allow applications to use it as expanded memory.

Figure 2 illustrates an example deployment of CXL-HM as a remote memory tier for multi-turn LLM serving. The key observation is that this memory abstraction naturally matches the access pattern of multi-turn workloads: as shown in Figure 2 (a), prefix KV blocks generated in earlier turns are repeatedly reused by later requests, while newly generated KV blocks are appended after each turn. In a remote serving configuration, workers can fetch reused prefix KV blocks through RDMA reads and store newly generated KV blocks through RDMA writes, allowing KV reuse to be handled through a remote memory rather than a storage-oriented path.

To scale capacity beyond a single device, the CXL-HM node manages multiple CXL-HM devices as a unified remote KV address space, as illustrated in Figure 2 (b). Each device is exposed as a CPU-less NUMA node backed by its SSD capacity, and the CXL-HM node aggregates these NUMA regions into one remote memory tier visible to LLM workers. Figure 2 (c) shows the internal access path of each CXL-HM device: the SSD-backed region provides the large visible capacity, while internal DRAM is used transparently as an internal cache. DRAM hits are served through the fast path, whereas misses require staging data from SSD into DRAM before completion. Therefore, the effectiveness of CXL-HM depends critically on managing this limited internal DRAM to keep predictable, latency-critical KV reuse on the fast path.

In summary, the opportunities CXL-HM brings can be summarized as follows:

• Cost-efficient SSD-backed capacity. By relying on high-density TLC or QLC NAND SSDs attached through commodity PCIe Gen4/Gen5 NVMe interfaces, CXL-HM can provide TB-scale capacity at much lower cost per GB than DRAM-only CXL memory pools.

• Exploiting predictable KV reuse in CXL-HM. Recent agentic LLM traces show that runs can span dozens to hundreds of turns, grow up to one million tokens, and reuse more than 95% of tokens across rounds [53]. This creates a highly predictable, read-heavy prefix KV access pattern: most reused KV blocks are known before execution, while only newly appended tokens generate new KV states. This predictability enables coordinated prefetching and internal DRAM staging in CXL-HM.

2.4 CXL-HM vs. DRAM–SSD tiering↩︎

At this point, it is useful to clarify the difference between using CXL-HM and building a remote tier from conventional DRAM and SSDs, since the two approaches rely on similar underlying components. Figure 3 compares the remote access paths of the two approaches. To read a remote KV block, for both approaches, an LLM worker first queries the metadata server to identify the storage node containing the requested block. Then, in conventional DRAM–SSD tiering, the request is forwarded to the storage node, which performs a second metadata lookup to determine whether the block resides in DRAM or SSD. If the block is cached in DRAM, the corresponding RDMA-accessible address is returned. Otherwise, the block is first staged from SSD into an RDMA-accessible DRAM buffer, which will elongate the response, before the newly allocated DRAM address is returned to the worker. In contrast, CXL-HM exposes the entire SSD-backed capacity as a host-visible memory space. Therefore, the metadata server directly returns the final \(\langle node_id, addr\rangle\) for the requested KV block, allowing the worker to immediately issue RDMA reads or writes without additional storage-node address resolution or software-mediated staging. This shortens the control path and simplifies metadata management.

3 CXL-HM for LLM serving↩︎

This section presents our design of CXL-HM, a key component we develop for HyMCache. Unlike general-purpose CMM-H products, our CXL-HM is designed specifically for multi-turn LLM inference. We therefore begin in Section 3.1 by characterizing the relevant workload properties found in multi-turn LLM inference. Based on this analysis, Section 3.3 derives the design directives for CXL-HM, followed by a discussion of the CXL-HM prototype developed for this study in Section 3.4.

3.1 Characterizing Multi-Turn Workloads↩︎

Multi-turn and agentic LLM serving workloads frequently reuse expanding prompt prefixes [5]–[7], [51], [53]. Prior work reports that agent trajectories often span dozens to hundreds of turns, allowing more than 95% of the tokens to be reused across turns [53]. When these prefix states are distributed across a hierarchical KV cache, the execution bottleneck shifts from compute-heavy prefill operations to read-intensive metadata lookups and block-loading loops spanning GPU memory, host DRAM, and remote tiers.

To understand these workload characteristics, we analyze KV-cache accesses in an idealized remote-DRAM setting. Our setup consists of a single aggregator with one NVIDIA A100 GPU worker connected to a 256 GB remote-DRAM tier over a 100 Gbps RDMA path. To expose remote KV-cache behavior, we intentionally constrain the GPU-side KV-cache capacity, forcing accumulated prefix KV blocks to be served from remote DRAM rather than remaining resident in GPU memory.

For the workload, we use Llama-3.1-8B with vLLM’s 128-token KV block size. Even this relatively small model generates 16 MB KV blocks, while larger models produce substantially larger blocks, such as 32 MB for Qwen2.5-32B and 168 MB for OPT-30B. As a result, KV-cache movement manifests as large, block-granular transfers rather than small cache-line accesses. We run chat-based multi-turn requests from the LMSYS dataset [54] to study how the remote KV-cache footprint grows across turns.

Figure 4 shows aggregate KV-cache traffic generated by a single prefill node running 512 concurrent requests across multiple conversation turns, under two decode lengths: MAX_TOKENS=1 and MAX_TOKENS=128. This traffic reflects a single prefill node. In deployments where multiple prefill nodes share the same remote tier, the aggregate demand on that tier can increase roughly in proportion to the number of attached prefill nodes. We highlight four read- and write-side characteristics of this traffic.

• Observation : Read traffic grows across turns. As the accumulated context grows, each turn needs to reload more prefix KV blocks. This causes both instantaneous read traffic and cumulative read volume to increase over turns.

• Observation : KV reuse is read-heavy and read-only. Previously generated KV blocks are reused as prefix state and read back in later turns, but they are not modified during reuse. From the perspective of the memory tier, these KV objects remain clean, that is, they are not updated, and only serve read requests.

• Observation : Sequential KV reads with weak locality. While not directly observable in Figure 4, we find that KV blocks are read sequentially in context order, and this sequential access yields little short-term reuse, that is, most blocks are read once within a turn and are rarely revisited soon after. This pattern resembles a one-hit-wonder workload [55], a scan-like access stream that pollutes the cache and neutralizes recency-based locality. For example, in our setup the prefix-read footprint added between Turn 6 and Turn 7 alone exceeds 64 GB. As each turn extends the context, the KV-cache footprint grows monotonically with conversation length. As a result, increasing the DRAM-cache capacity only defers, rather than eliminates, cache pollution. Such scan-like access pattern inherently lacks the temporal reuse that recency-based caching policies rely on. This effect is amplified when multiple prefill nodes share the same remote tier, since their long prefix reads interleave and further dilute whatever locality remains.

• Observation : Writes are append-only of newly generated KV blocks. Each turn writes only the KV blocks produced for its newly generated output tokens. As a result, when output lengths are similar, write volume remains relatively stable across turns.

3.2 Memory-Tier Takeaways for Multi-Turn LLM Serving↩︎

Taken together, the observations above expose a structural mismatch between how multi-turn KV traffic behaves and how a generic memory tier is managed. We distill three takeaways that any memory tier backing long-context, agentic LLM serving must address.

Takeaway : KV reuse is predictable, not recency-driven. Each turn re-reads the accumulated prefix sequentially in context order, so reuse is long-range and schedulable—the blocks needed next turn are known before the turn begins—yet carries almost no short-term locality within a turn. Demand-driven recency/frequency policies (LRU/LFU) are therefore the wrong abstraction: they retain already-consumed blocks while the upcoming prefix misses, and because the scan-like pattern lacks short-term reuse in the first place, no realistic DRAM budget can recover locality that was never there. The implication is that internal DRAM should be managed from the request’s turn structure rather than from observed access recency—informed management, not a transparent demand cache.

Takeaway : Writes are deferrable and should be isolated from reads. Newly generated KV blocks are append-only, stable in volume across turns, and off the correctness path of the current request—they exist solely to accelerate future turns. This makes writes safe to buffer in fast memory and flush asynchronously in large batches, and safe to deprioritize under read pressure, trading a small future hit-rate loss for current-turn read latency.

Takeaway : Mixed read/write traffic is especially costly on SSD-backed tiers. Because the backend capacity is SSD-backed (e.g., CXL-HM, NVMe-oF), interleaving performance-critical reads with writebacks induces SSD-internal contention that depresses read bandwidth and inflates tail latency. This effect compounds when multiple prefill workers share the same remote tier and their long prefix reads interleave. Read/write isolation is thus a first-class requirement, reinforcing the asynchronous, batched flush path of Takeaway .

These takeaways converge on a single conclusion: device-managed, transparent DRAM caching is insufficient for LLM serving. The internal DRAM should instead be governed by a workload-aware policy that exploits the predictability of prefix reuse and isolates deferrable writes—the mechanism we develop in Section [sec:internal-dram-management].

3.3 CXL-HM under Multi-Turn KV Traffic↩︎

CXL-HM is an attractive substrate for a cost-efficient remote memory tier, but its performance depends critically on how device-side DRAM is managed. Existing CXL-HM prototypes and products, typically referred to as CMM-H devices, normally use this DRAM as a transparent cache for SSD-backed capacity, governed by LRU-like replacement policies [31]–[33]. This choice is reasonable for generic memory expansion, but it is a poor match for the long-context and agentic LLM serving access pattern observed in Section 3.1. As was shown, these workloads repeatedly scan the accumulated prefix across turns, generating large, predictable reads with little short-term locality. Consequently, under LRU, the limited DRAM is populated with already-consumed KV blocks, while the prefix blocks needed in the next turn remain in the SSD-backed region.

To validate this behavior, we run a multi-turn KV-cache microbenchmark on a commercial CMM-H device whose 256 GB of device-side DRAM is managed as a transparent LRU cache for various representative KV block sizes. In these experiments, a remote worker accesses the CMM-H address space over a 200 Gbps RDMA path, issuing 64 concurrent requests that each append 32 KV blocks per turn. Turn 1 writes only the newly generated blocks. From Turn 2 onward, each request first reads the prefix written in previous turns and then appends its new blocks. Thus, the per-turn write footprint scales with KV-block size, from 32 GB for 16 MB blocks to 336 GB for 168 MB blocks, while read footprints grow monotonically across turns. Figure 5 shows the results, which we now discuss.

We observe that as long as the active footprint fits in device-side DRAM, bandwidth remains close to the RDMA line rate, as shown for Llama-3.1-8B. However, as the footprint grows beyond the effective DRAM-cache capacity, bandwidth begins to collapse toward the SSD-backed regime, reaching roughly 5 GB/s under mixed reads and writes. In Figure 5, this collapse occurs between Turns 2 and 3 for 168 MB blocks and between Turns 3 and 4 for 64 MB blocks. For smaller blocks, the same effect is delayed, but once the accumulated footprint eventually fills the cache, bandwidth is expected to degrade in the same manner.

We attribute this collapse to the conventional cache-management policies used in CMM-H, particularly LRU-induced refills and dirty evictions, which we verify internally and which are consistent with prior CMM-H microbenchmark results [33]. LRU-induced refills cause one-shot-wonder blocks to remain in the cache even after they have been consumed, occupying space that could otherwise hold upcoming prefix blocks that we know, from Section 3.1, will soon be needed. Dirty eviction is more subtle. As discussed above, write traffic is stable and bounded, but newly written KV blocks are quickly propagated to persistent flash media according to the device’s internal policy to protect against data loss. Consequently, read traffic becomes interleaved with backend writebacks inside the CMM-H device, degrading the effective read bandwidth.

Thus, our central challenge is to design CXL-HM for HyMCache to overcome the limitations of conventional CMM-H devices and directly support the observed workload characteristics of multi-turn LLM serving.

3.4 LLM Targeted CXL-HM↩︎

The workload observations above motivate a different way of using CXL-HM for LLM serving. Rather than treating the internal DRAM as a transparent LRU cache, our prototype uses it as an explicitly managed staging space for KV objects. The key idea is to use the predictable KV-object access order revealed by multi-turn prefix-cache lookup, allowing CXL-HM to stage upcoming KV objects before the foreground remote reads arrive.

Figure 6 shows the overview of our LLM-targeted CXL-HM prototype. LLM workers access the remote KV tier as remote memory through RDMA or a CXL switch-based fabric; our prototype uses RDMA as the example path. The remote tier is exposed as a unified memory address space backed by one or more CXL-HM devices. A read request is served directly from internal DRAM if the requested KV object is already staged; otherwise, the object is first copied from SSD-backed capacity into internal DRAM before being returned. The prototype focuses on two design points for LLM serving: KV-object prefetching and latency-hiding staging window.
KV-object prefetching. Multi-turn prefix-cache lookup reveals the ordered list of KV objects that an inference request will consume. The serving runtime passes this object-level access order to CXL-HM, allowing upcoming KV objects to be prefetched before foreground remote reads arrive. CXL-HM provides an explicit prefetch interface for user-level control of KV-object staging. We describe the detailed API and execution protocol below.
Latency-hiding staging window. A larger internal-DRAM staging window can provide more slack to hide a slower SSD-backed path, but increasing device-side DRAM directly increases cost. Our prototype therefore keeps the internal-DRAM staging space bounded and instead scales the SSD-to-DRAM refill path. This design accommodates the growing prefix KV footprint across turns without proportionally scaling internal DRAM. Rather than using internal DRAM as a capacity cache for the accumulated prefix, CXL-HM streams KV objects through a fixed-depth staging window: upcoming objects are staged ahead of foreground remote reads, consumed by workers, and then released for later objects. As long as the refill pipeline keeps this window populated, bounded internal DRAM can support large and growing prefix KV caches. Thus, SSD bandwidth becomes a practical knob for sustaining prefetching, while internal DRAM remains a latency-hiding staging window rather than a cache sized to the total KV footprint.

Read-Prioritized Write-Isolated KV Flushing. KV writes correspond to newly generated tokens and affect only future reuse, not the correctness of the current request. CXL-HM therefore isolates write handling from the latency-critical read path. The device reserves a small write-side buffer in internal DRAM and admits newly generated KV objects only when buffer space is available. If the buffer is full, the runtime can skip or retry the write later. Admitted KV objects are deferred and flushed asynchronously to the SSD-backed region in large batches, preferably when read pressure is low. This avoids interleaving background writes with foreground prefix reads on the SSD-backed path, preserving read-side prefetch bandwidth and reducing random read/write interference.

User-Level Prefetch Coordination CXL-HM can receive object-level prefetch commands in the order that the objects will be consumed. For each prefetch command, CXL-HM stages the corresponding KV object from SSD-backed capacity into an internal-DRAM region. Later foreground remote reads can then be served from internal DRAM if the object has already been staged. To support KV-object prefetching in CXL-HM, we extend the user-directed prefetch interface of the CXL-HM prototype with an explicit issue--wait--release protocol. The interface provides three object-level lifecycle APIs:

• chm_prefetch_object(ptr, size) prefetches an object into an internal-DRAM staging region and returns a request handle.

• chm_prefetch_wait(req) waits until the prefetched object associated with the request handle is ready to be consumed.

• chm_prefetch_release(req) releases the internal-DRAM staging region associated with the request after the object has been consumed.

For remote KV-cache reads, CXL-HM keeps up to a configured prefetch depth of KV objects in flight. Each prefetch stages an object from SSD-backed capacity into an internal-DRAM region. Once the object is consumed by a foreground remote read, the region is released back to the free pool. This bounded issue–consume–release protocol overlaps SSD-to-DRAM staging with remote reads while limiting internal DRAM usage.

Design Requirements. Motivated by the workload characteristics above, CXL-HM should manage its internal DRAM differently from a conventional LRU-style cache. Our design follows two key principles.

• Backend bandwidth provisioning.

• Internal DRAM Butget for KV Object Staging.

Prefetch Coordination.

• chm_prefetch_object(ptr, size) prefetches a structured object, such as a KV block, into an internal-DRAM staging region and returns a request handle.

• chm_prefetch_wait(req) waits until the prefetched object associated with the request handle is ready to be consumed.

• chm_prefetch_release(req) releases the internal-DRAM staging region associated with the request after the object has been consumed.

For remote KV-cache reads, the remote KV manager issues prefetch requests for upcoming KV objects up to the configured prefetch depth. Inside the CXL-HM device, the prefetch engine stages these objects from SSD-backed capacity into internal DRAM. The worker waits until each prefetched object is ready, consumes it through RDMA reads, and then releases the associated internal-DRAM region back to the free pool. By keeping at most the prefetch depth requests in flight, HyMCache bounds the internal DRAM staging footprint while allowing CXL-HM to overlap SSD-to-DRAM staging with worker-side RDMA reads.

Write Path: Opportunistic KV Buffering. CXL-HM reserves a small fraction of internal DRAM for write-side buffering, proportional to the total internal-DRAM capacity. The read/write partition is configurable at deployment time, allowing the device to prioritize KV-read prefetching while reserving enough write-buffer space to absorb newly generated KV blocks. During execution, however, this partition remains fixed. Although unused read-side DRAM could be dynamically reassigned to write buffering, doing so would complicate hardware tracking and arbitration while providing limited benefit, because write-back is off the critical path. We therefore use a fixed read/write partition at runtime.

Figure 8 illustrates the KV write process with a fixed-capacity write buffer. Before accepting a newly generated KV block, the CXL-HM node checks whether enough write-buffer space is available in internal DRAM. Since multiple requests may issue writes concurrently, this check-and-reserve step is serialized at the CXL-HM node. If there is not enough space, the node returns a write failure instead of blocking foreground execution. For example, KV7 may fail to be stored in Turn 1 due to insufficient write-buffer space, but it can be retried in Turn 2 when space becomes available. This retry-based behavior is acceptable for prefix caching because preserving and serving earlier KV blocks as cache hits is more important than forcing newly generated suffix blocks to be written immediately.

• Read-prioritized write admission. New KV writes are admitted only when enough write-buffer space is available in internal DRAM. If the write buffer is full, the CXL-HM node rejects the write and allows the serving system to retry it in a later turn. This prevents background KV writes from interfering with latency-critical reads.

• Endurance-aware append-only placement. Newly generated KV blocks are written to new locations rather than updated in place. This spreads write traffic across the SSD-backed region and avoids repeated overwrites to the same locations.

This write path is opportunistic and does not affect the correctness of the current inference request. Delaying, skipping, or later invalidating a KV write may reduce future cache hit rate, but the current response remains unchanged.

4pt

Hardware Overhead.

4 HyMCache↩︎

4.1 HyMCache Overview↩︎

To run within a practical LLM serving stack, HyMCache integrates with Dynamo and vLLM, as shown in Figure 9. On the hardware side, CXL-HM serves as the Tier-3 remote KV-cache device. LLM workers can access this tier either through a CXL-based memory fabric, such as a CXL switch or CXL memory box, or through existing RDMA-capable datacenter networks. Our prototype focuses on the RDMA-based deployment because it can run on existing infrastructure without requiring a CXL fabric.

On the software side, the key distinguishing feature of HyMCache is that it enables CXL-HM to provide remote-DRAM-like performance while exposing SSD-scale capacity. Specifically, HyMCache aims to ensure that KV blocks residing in the SSD component of CXL-HM are prefetched in time to be present in CXL-HM’s internal DRAM when needed. To this end, we add four modules to the existing serving stack: Master, Lookup, KV Connector, and KV Manager. In the following, we describe the roles of the first three modules, while the KV Manager, which is part of CXL-HM, is described in Section 4.2.

HyMCache Master. The HyMCache Master module is responsible for metadata lookup and remote address resolution, which is done by maintaining global metadata for remote KV blocks across CXL-HM nodes. It maps each prefix/KV identifier to the remote access information needed for data transfer, including the target CXL-HM node, remote address, and length. The metadata footprint is small relative to the remote KV capacity and scales with the number of KV objects rather than raw capacity. With 16 MB KV blocks, 10 TB of remote KV capacity corresponds to approximately \(6.5\times10^5\) KV objects. Assuming a compact 64 B metadata entry per object, covering the KV identifier, target CXL-HM node, offset, length, and state/version fields, the metadata footprint is about 40 MB for 10 TB of remote KV capacity.

More specifically, for writes, Master selects a target CXL-HM node at request granularity and returns the remote address where the KV block can be written. Keeping KV blocks from the same request on the same target node preserves ordered access opportunities, while different requests can be distributed across nodes for load balancing. After RDMA writes complete, the written KV blocks are registered with Master and added to the global metadata.

For reads, Master is queried with prefix/KV identifiers and Master returns the corresponding remote access information for blocks stored in HyMCache. The returned metadata allows the worker-side KV Connector to issue RDMA reads directly to the target CXL-HM node.

HyMCache Lookup. The Lookup module is responsible for early remote-hit matching, batched metadata lookup, and CXL-HM prefetch coordination before the vLLM worker reloads KV blocks into the GPU KV cache. After GPU and optional upper-tier cache lookup, Lookup immediately sends the currently available unmatched prefix blocks to HyMCache Master, for up to 32 blocks per request by default, without waiting for the request to become full. Once the first remote hit identifies the target CXL-HM node, prefetch requests are issued for the remaining candidate suffix blocks on that node while Lookup continues to discover additional remote hits. The collected remote-hit metadata is then passed to the vLLM worker, so the latency-critical read path performs only data movement through batched RDMA reads.

HyMCache KV Connector. HyMCache integrates with vLLM through a lightweight KV Connector attached to vLLM’s existing KV-transfer callbacks. With HyMCache disabled, vLLM would run unchanged. When enabled, KV Connector is invoked after vLLM’s GPU prefix-cache lookup and any optional upper-tier DRAM-based KV-cache lookup, such as LMCache or Mooncake. For the remaining unmatched prefix blocks, KV Connector uses the remote-hit metadata produced by Lookup to issue RDMA reads into the GPU KV cache before attention, allowing the latency-critical read path to focus on RDMA data movement. For newly generated KV blocks, KV Connector obtains write locations from Master, performs RDMA writes to the target CXL-HM node, and registers the written blocks in batch. To avoid blocking inference due to transient delays in remote KV retrieval, HyMCache falls back to normal recomputation if the remote KV blocks are not ready within a 10 second timeout, following Mooncake’s default configuration [7].

KV Connector also maintains a host-side CPU staging buffer for RDMA transfers, similar to existing KV-transfer systems such as LMCache [6]. This buffer is used as temporary transfer space between RDMA and the GPU KV cache, rather than as an additional cache tier. We use a 5 GB staging buffer by default, matching common practice in existing systems. For higher concurrency, this buffer should be scaled with the number of in-flight KV transfers.

4.2 Remote KV Management on CXL-HM↩︎

HyMCache KV Manager. The HyMCache KV Manager module coordinates CXL-HM prefetching across concurrent requests and manages prefetch resources for each request. After receiving the remaining suffix block list, KV Manager performs remote-hit matching in the same request order as Lookup and invokes the CXL-HM prefetch API (Section 3.3) for matched KV objects. As both modules process the same suffix list in the same order, the prefetch order is aligned with the subsequent RDMA read order.

To balance latency hiding and DRAM utilization, KV Manager dynamically allocates the prefetch window according to the prefix-hit footprint of each request while enforcing a bounded maximum prefetch window. Based on our prototype evaluation, allocating up to approximately 128 MB of prefetched KV data per request provides sustained performance. Consequently, KV Manager adjusts the prefetch window according to the KV-block size, up to eight blocks for 16 MB KV blocks and four blocks for 32 MB KV blocks. Requests with larger KV blocks or fewer prefix hits use a smaller prefetch window, down to double buffering when appropriate.

Each prefetched block is managed by KV Manager using the CXL-HM prefetch API and an opaque read token. For each matched KV object, KV Manager submits a prefetch request to the CXL-HM prefetch API, which stages the object into the CXL-HM’s internal DRAM before the actual RDMA read. Once the prefetch request completes, KV Manager makes the corresponding descriptor available to the vLLM worker, indicating that the worker can fetch the object from the returned CXL-HM address. The worker then posts RDMA reads from the returned CXL-HM addresses into its local staging buffer and observes completion through its local RDMA completion queue. After the reads complete, the worker sends the consumed read tokens back to KV Manager, which releases the corresponding prefetched objects via the CXL-HM release API.

Unified CXL-HM Address Space. On each remote CXL-HM node, each device appears as a CPU-less NUMA node, creating a one-to-one device-to-NUMA-node mapping. The HyMCache KV Manager uses NUMA allocation interfaces [56] to allocate memory from selected device-backed NUMA nodes for storing KV blocks, and registers the allocated memory as an RDMA memory region (MR). This RDMA registration provides an MR-level rkey, while per-block metadata records only the offset and length of each KV block within the registered memory region.

HyMCache uses request-level NUMA assignment rather than page-level or fine-grained interleaving across devices. Across requests, the HyMCache KV Manager selects NUMA nodes using simple policies such as round-robin or load-aware selection to balance capacity and bandwidth. After assigning a request to a NUMA node, the HyMCache KV Manager allocates all prefix KV objects for that request from the selected node, placing them on the corresponding CXL-HM device. This enables ordered request-level prefetching through a single device-side staging path, without splitting the prefix stream across multiple CXL-HM devices. This NUMA-level placement remains internal to the HyMCache KV Manager, while the rest of HyMCache treats the remote CXL-HM capacity as a large memory space.

Support for Heterogeneous KV Object Sizes. Because HyMCache manages KV placement at object granularity, KV objects from different models, workers, and workloads can coexist in the same unified CXL-HM address space. Each request carries its model-specific KV object size, and its prefix list preserves the order in which KV objects are consumed. KV Manager uses this object size to determine the prefetch window and internal DRAM staging budget independently for each request. This allows concurrent LLM workloads with different KV object sizes to share the same CXL-HM tier without assuming a single global object size, while still balancing capacity and bandwidth across CXL-HM regions.

4.3 A Walkthrough Example↩︎

Figure 9 summarizes how HyMCache coordinates CXL-HM prefetching, batched metadata operations, and RDMA data movement. On the read path, after GPU-side prefix-cache matching and optional upper-tier KV-cache lookup, the remaining prefix blocks are handled by the HyMCache Lookup module, which 1 sends prefetch hints to the HyMCache KV Manager so that upcoming KV objects can be staged in CXL-HM internal DRAM and issues a batched remote-hit query to the HyMCache Master to obtain remote descriptors for matched KV objects. The matched descriptors are then passed to the vLLM worker, so the latency-critical read path performs only RDMA GETs into the GPU KV cache. On the write path, the worker sends a batched register request to the HyMCache Master to obtain remote write descriptors, and writes the generated KV objects to the assigned CXL-HM addresses using RDMA PUTs.

5 Experimental Setup↩︎

System configurations. We evaluate HyMCache in a vLLM-based serving environment where workers access a remote CXL-HM KV tier over an RDMA-capable network. Our experiments cover two settings: PD-disaggregated serving, where prefill and decode execution are separated, and single-node serving, where they are co-located on the same worker.

For the PD-disaggregated setup, we use six Dell PowerEdge R770 servers, each equipped with dual Intel Xeon 6730 CPUs, 256 GB of DDR5 DRAM, and an NVIDIA A100 GPU with 80 GB of HBM. The remote CXL-HM tier is built using an FPGA-based CXL-HM prototype configured with 64 GB of device-side DRAM and two 1 TB Gen5\(\times\)​4 backend SSDs. exposed as a 2 TB shared memory tier to the host. We map the serving roles as a 4P–1D–1S configuration: four servers run prefill workers, one server runs the decode worker and benchmark/control components, and one server runs the CXL-HM remote storage node. Each prefill server dedicates one 100 Gbps NIC port to the NIXL-based PD path and the other 100 Gbps NIC port to HyMCache traffic. The decode and storage nodes each provide up to 200 Gbps of aggregate bandwidth to the prefill workers. This setup stresses KV-cache tier utilization under prefill-heavy, multi-worker access to reusable prefix KV blocks.

The single-node setup co-locates prefill and decode execution on one compute node and directly connects it to the CXL-HM remote KV tier. This removes PD scheduling effects and allows us to compare remote CXL-HM performance against local DRAM-based KV caching.

HyMCache is implemented in vLLM 0.19.0 on top of the Dynamo v1.1.0 container image. Following Dynamo’s LMCache integration model, we add a HyMCache connector to vLLM’s KV-transfer path during prefill execution. The connector coordinates remote KV lookup, prefetch requests, and RDMA-based KV transfer through the HyMCache master and remote KV manager. When HyMCache is disabled, vLLM follows its default execution path.

Baselines. We compare HyMCache against baselines that represent different levels of the KV-cache memory hierarchy:

• Recomputation. No remote KV tier is used. Missing prefix KV blocks are recomputed through the default vLLM.

• GPU Prefix Caching (T1). vLLM reuses KV blocks that are already resident in the GPU KV cache.

• Local LMCache [6] (T1 + local DRAM). Each of the four workers is configured with 64 GB of local DRAM for KV caching.

• Distributed Mooncake [7] (T1 + remote DRAM). Mooncake uses a distributed DRAM tier across prefill nodes, providing 1 TB of memory capacity for KV caching.

• Mooncake NVMe-oF [7] (T1 + remote SSD). KV blocks are stored in a remote SSD-based KV tier and accessed through an RDMA-based NVMe-oF fabric.

• HyMCache (T1 + remote CXL-HM). HyMCache serves reusable prefix KV blocks from a CXL-HM remote KV tier that combines internal DRAM with SSD-backed capacity.

Among the baselines, Distributed Mooncake and Mooncake NVMe-oF are the primary points of comparison for HyMCache, as both expand their KV-cache capacity through a remote tier. However, the remote-tier components differ across these systems, leading to different performance and cost characteristics. We discuss the performance implications in the next section; here, we briefly estimate cost, focusing on the remote-tier components used in our experiments.

The FPGA-based CXL-HM prototype include an Agilex 7 development board [49], 64 GB of device-side DRAM, and two Gen5 SSDs for a rough cost of $10.5K–$11.0K. In comparison, Distributed Mooncake uses a 1 TB DDR5 RDIMM DRAM tier, which costs roughly $30K–$40K. Thus, even with FPGA prototyping overhead, CXL-HM is about 2.8–3.8\(\times\) cheaper. The FPGA board alone accounts for roughly 91–95% of the prototype cost, indicating that the estimate is dominated by prototyping overhead rather than by the SSDs or DRAM. Mooncake NVMe-oF, on the other hand, can be considerably cheaper than CXL-HM by leveraging existing remote SSD infrastructure. However, practical deployments typically require integrating the remote NVMe devices through a host-side file system and storage software stack, introducing additional software overhead and latency, as discussed in Section 6.1. While these overheads may be reduced through DPU-based storage offloading, such deployments require additional accelerator hardware, which incurs additional costs.

Models and Workloads. We evaluate HyMCache using LLMs with different KV-cache footprints. Unless otherwise stated, we use FP16/BF16 KV cache and a vLLM block size of 128 tokens. We select Llama-3.1-8B and Qwen2.5-32B to cover different KV-object sizes used in our experiments. In our configuration, Llama-3.1-8B generates 16 MB KV blocks, while Qwen2.5-32B generates 32 MB KV blocks.

For workload generation, we use the synthetic benchmark provided by Dynamo AIPerf [57], [58], with mooncake trace [7], followed by the documented guideline from NVIDIA [59]. AIPerf allows to benchmark LLM with controlled input and output length, prefix sharing ratio, request concurrency, and the number of prefill workers. We also use the LMSYS dataset [54] to evaluate HyMCache under realistic multi-turn conversation patterns. To isolate the effect of KV-cache reuse, we set the maximum number of decode tokens to 1, minimizing decode-side computation. We report serving metrics including TTFT, end-to-end latency, remote-tier bandwidth, and KV-cache hit rate.

6 Evaluation↩︎

6.1 PD-Disaggregated Serving Performance↩︎

6.1.1 Overall Performance Comparison↩︎

Figure 10 shows the results under PD-disaggregated serving on Qwen2.5-32B using AIPerf synthetic inputs. This workload stresses cache capacity because the reusable prefix state quickly exceeds the capacity of GPU-only prefix caching and limited local DRAM caching, while generating approximately 1.5TB of reusable KV-cache footprint across turns.

Compared to Recomputation, GPU prefix caching, and LMCache-local caching, HyMCache improves performance by retaining a larger fraction of reusable KV blocks and serving them from the remote CXL-HM tier. Figure 10 (c) shows the cache-hit behavior at Turn 7. By this point, local cache baselines such as GPU prefix caching and LMCache local CPU caching exhibit low hit rates because their effective cache capacities are exceeded, and LRU eviction can break the contiguous prefix chain required for reuse. Distributed Mooncake is more robust as it retains reusable prefix blocks in the remote DRAM tier, but its 1 TB capacity covers only about two-thirds of the 1.5 TB reusable KV footprint. In contrast, HyMCache uses larger SSD-backed CXL-HM KV capacity to preserve more reusable prefix blocks, improving the hit rate by roughly 10% over Mooncake and reducing prefill recomputation. This benefit is slightly limited by the read-prioritized CXL-HM policy, which delays write-buffer draining and skips more than 15% of potential KV insertions under this workload.

Compared to HyMCache, Distributed Mooncake with 1 TB of DRAM achieves roughly 30% higher performance. This advantage comes primarily from its lower DRAM fetch latency and shared distributed-cache pool, which can reduce recomputation by preserving reusable KV blocks across workers in this workload. In contrast, HyMCache incurs additional remote CXL-HM access latency and RDMA transfer overhead.

However, this gap should be interpreted in the context of the remote-tiers and their cost, where Distributed Mooncake is estimated to be much more costly, as discussed earlier. More importantly, the performance gap narrows as the workload progresses to later turns. As the reusable KV footprint grows, capacity becomes increasingly important, allowing HyMCache to retain reusable blocks that smaller caches would evict. This trend is visible in the figure where the gap between HyMCache and the distributed-DRAM baseline decreases as the turn number increases. Furthermore, the CXL-HM approach improves scalability because DRAM-only expansion is limited by DIMM slots, socket-level memory channels, and high capacity cost. In contrast, CXL-HM can expand the remote KV tier by adding more CXL-HM devices or by increasing the SSD-backed capacity per device, with a much lower marginal cost per additional GB.

Overall, HyMCache targets a practical middle point between high-cost remote DRAM and slow remote storage. It provides a much larger effective cache capacity than local memory, while avoiding the high recomputation cost of capacity-limited caches by approaching distributed-DRAM performance with a lower-cost CXL-HM tier.

6.1.2 Comparison with NVMe-oF Remote Storage↩︎

We now compare HyMCache with an NVMe-oF-over-RDMA remote storage baseline using Mooncake. This baseline has the same setup as CXL-HM in that it uses the same ConnectX-6 RNIC and two Gen5\(\times\)​4 SSDs and thus highlights the features of CXL-HM that have been tailored for multi-turn LLM.

To support multiple prefill nodes in this baseline, the storage node first stripes the SSDs with RAID. To simplify our experimental setup, we then use LVM to partition the striped capacity and expose a separate NVMe-oF block volume to each prefill worker. Each remote volume is mounted with XFS, adding file-system overhead beyond the raw NVMe-oF transport.

Figure 11 shows that using an NVMe-oF storage tier substantially slows end-to-end serving compared with HyMCache. In this configuration, remote KV fetches frequently exceed the timeout and fall back to normal recomputation, making the observed performance close to the recomputation baseline rather than benefiting from remote KV reuse. We attribute this degradation mainly to the software I/O path. We observe that the raw NVMe-oF block-device path reaches about 23 GB/s on each prefill node, close to the 200 Gbps network limit, whereas mounting XFS on top of the same NVMe-oF device reduces the measured sequential-read bandwidth to roughly 6.5 GB/s in our setup. This suggests that simply attaching high-bandwidth NVMe-oF storage is not sufficient for remote KV-cache serving; the storage path often requires additional optimization to deliver predictable bandwidth on the critical read path. Such optimizations may include careful I/O-path design, file-system bypass, DPU-assisted storage, or high-performance distributed file systems, but they add cost and system complexity. In contrast, HyMCache leverages the memory-like semantics of CXL-HM to manage SSD-backed capacity as a TB-scale remote memory tier. By directly controlling KV-cache placement and prefetching over this tier, HyMCache avoids placing a general-purpose file-system stack on the critical read path.

6.1.3 Transfer-Buffer Sensitivity in Distributed DRAM Caching↩︎

Although distributed DRAM caching is a high-performance reference point for remote KV reuse, its performance also depends on worker-side transfer-buffer capacity. For both Mooncake and HyMCache, remote KV reuse follows a memory-based transfer path: KV blocks are received into CPU-side staging buffers and then copied into GPU-resident KV cache blocks. Sufficient staging space overlaps remote-to-host transfers with host-to-GPU copies, keeping remote reads ahead of GPU consumption. With limited staging space, remote reads are throttled by buffer availability, overlap is reduced, and cache-hit latency appears on the critical path. The required buffer also scales with KV-loading bandwidth; 100–400 Gbps links or multiple NICs need more staging space to keep KV reads in flight.

In the following set of experiments, we isolate this transfer-buffer effect and examine how much local host DRAM is needed to sustain memory-based remote KV reuse by varying the per-node transfer-buffer budget for both HyMCache and Distributed Mooncake. Figure 12 shows their end-to-end serving performance. With the same 10 GB of additional host DRAM per node, Mooncake suffers from severe transfer-buffer pressure. Many remote KV insertions fail to complete in time, lowering effective KV reuse and often breaking the contiguous prefix chain in later turns. As a result, performance approaches the recomputation baseline, even with a longer transfer timeout.

HyMCache is less sensitive to this constraint because it adopts an opportunistic remote-insertion policy. If the remote CXL-HM write buffer (Section 3.3) is full and cannot accept additional RDMA writes, HyMCache skips the insertion and retries it in later turns rather than stalling the serving path. As the transfer-buffer budget increases, Mooncake recovers, showing that optimizing a distributed DRAM cache requires not only sufficient remote DRAM and RDMA bandwidth, but also additional local transfer-buffer DRAM at each worker. HyMCache, however, shows similar performance beyond the 10 GB transfer-buffer budget.

6.2 Single-Node Serving Performance↩︎

6.2.1 Overall Performance Comparison↩︎

Figure 13 compares the single-node serving performance of HyMCache against baseline caching methods. In this configuration, prefill and decode execution are co-located on the same LLM worker, allowing us to isolate the latency and hit-rate effects of remote KV caching. We evaluate Llama-3.1-8B using the LMSYS dataset with 512 multi-turn conversations, a concurrency of 32, and up to 10 turns per conversation. We set the maximum number of decode tokens to 1 to focus on prefix reuse and prefill-side latency.

The key advantage of HyMCache is that it extends KV-cache capacity with a remote CXL-HM tier, enabling large-capacity prefix reuse and reducing recomputation. Recomputation shows steadily increasing latency as turns progress because the accumulated prefix becomes longer at each turn. GPU prefix caching and LMCache local CPU caching improve performance in early turns, but their benefits decrease once the accumulated prefix working set exceeds their effective cache capacity. After this point, GPU prefix caching begins to lose effective reuse around Turn 3, and LMCache local CPU caching around Turn 4. This is because LRU eviction can remove earlier prefix blocks as the conversation grows. Since prefix reuse depends on a contiguous chain of cached blocks from the beginning of the prompt, evicting an early block truncates the reusable prefix, and later cached blocks no longer translate into effective prefix hits. Consequently, the cache hit rate drops sharply, more of the accumulated prefix is recomputed, and latency gradually approaches the recomputation baseline. In contrast, HyMCache preserves reusable prefix KV blocks in the CXL-HM tier and maintains a hit rate close to 90%, avoiding this recomputation latency growth across turns.

6.2.2 Comparison with LMCache Remote Redis Tier.↩︎

We further compare HyMCache against LMCache configured with a remote Redis tier [60]. We evaluate two Redis configurations: Redis-only 64 GB remote caching and LMCache with a 64 GB local DRAM cache plus an additional 64 GB Redis-backed remote cache tier accessed over TCP. For this experiment, we use Qwen2.5-32B and choose a workload footprint that fits within the combined 128 GB cache capacity of the LMCache and Redis configuration.

Figure 14 (a) shows serving performance across Turns 1–6. When Redis is used alone as the remote cache tier, frequent store failures break the reusable prefix chain and cause requests to fall back to recomputation. Adding LMCache local DRAM caching makes the configuration more stable, as prefix KV blocks can be retained and reused through the combined local and Redis-backed cache.

Figure 14 (b) shows the latency CDF at Turn 6. Although LMCache and Redis benefits from a memory-backed local/remote cache combination, HyMCache maintains a similar latency distribution by preserving reusable prefix KV blocks in the remote CXL-HM-backed tier. This shows that HyMCache can sustain effective prefix reuse using remote CXL-HM capacity, even when compared with a memory-based cache hierarchy.

6.2.3 Impact of HyMCache Prefetching↩︎

We analyze the impact of CXL-HM prefetching by comparing execution with and without HyMCache prefetching. To better isolate the benefit of prefetching under read-intensive access patterns, we first design a microbenchmark that issues heavy random-read requests to the remote CXL-HM tier managed by HyMCache. As shown in Figure 15 (a), enabling prefetching improves performance by 37% compared to the prefetch-off configuration.

Figures 15 (b) and 15 (c) further evaluate prefetching using the LMSYS dataset. To create a read-heavy scenario, we select requests with input lengths longer than 2K characters and run 512 conversations with a concurrency of 64. We report the results at Turn 6, where the external cache hit rate exceeds 90%. Figure 15 (b) shows the average and peak remote KV loading bandwidth. With prefetching, the peak bandwidth approaches the 200 Gbps network limit, while the average bandwidth improves by more than 35% over the prefetch-off case. As a result, Figure 15 (c) shows that prefetching reduces end-to-end execution time by 14%. These results show that HyMCache prefetching is effective for read-heavy multi-turn workloads because it pipelines data staging within the CXL-HM tier and RDMA-based KV transfer to the worker.

7 Related work↩︎

CXL-based Memory Expansion and Pooling. CXL has been explored as a substrate for memory expansion, pooling, and disaggregation. Prior work has studied CXL-based memory expansion for improving memory capacity beyond local DIMM channels [61]–[65], as well as CXL-aware tiered memory management for heterogeneous memory systems [38]. Due to the limited availability of commercial CXL hardware, many early studies relied on simulation, emulation, or software-based modeling to evaluate CXL memory behavior [66]–[68]. More recent commercial and prototype systems demonstrate several CXL design points, including DRAM-based CXL memory expanders [24], [26], switch-based CXL memory pools and appliances [25], [27], [42], and CXL-HM devices that combine internal DRAM with SSD-backed capacity [32], [33].

HyMCache differs from these efforts in both target workload and system interface. Rather than using CXL memory as a generic memory expansion tier, HyMCache targets remote KV caching for multi-turn and agentic LLM serving. In this setting, KV accesses are large, read-heavy, and predictable because prefix KV blocks are repeatedly reused across turns. HyMCache exploits this property by coordinating serving-level KV metadata with CXL-HM prefetching and device-side DRAM staging. This allows SSD-backed CXL-HM capacity to act as a cost-efficient remote KV tier while keeping latency-critical reused KV blocks on the fast internal DRAM path.

Multi-tier KV Cache and Prefetching. The large memory footprint of KV caches has motivated work on memory tiering and cache reuse for LLM inference. General inference systems such as FlexGen [8] and DeepSpeed Inference [69] improve throughput by partitioning model data and intermediate states across GPU, CPU, and NVMe storage, while LLM in a Flash [70] and PowerInfer [71] further explore flash-based offloading for memory-constrained inference. For KV-cache management, PagedAttention [5], Pensieve [72], and CachedAttention [73] improve paging, placement, and context reuse, while LMCache [6] and Mooncake [7] reduce recomputation through KV sharing and disaggregated KV-cache architectures.

HyMCache is complementary to these software-level systems but targets a different layer of the hierarchy. Rather than proposing another GPU/CPU/NVMe cache policy, HyMCache provides a CXL-HM based remote memory backend for large reusable KV states. Its key distinction is that it couples serving-level knowledge of prefix KV reuse with device-level DRAM staging: the serving system identifies reusable KV blocks before they are consumed and conveys this information to CXL-HM through prefetch commands. This allows HyMCache to use SSD-backed capacity for scale while staging latency-critical KV reads in device-side DRAM, making CXL-HM an efficient lower-tier backend underneath existing KV-cache management policies.

DPU-based Storage and Remote KV Caching. DPU and SmartNIC based storage disaggregation has enabled high-density remote flash systems and JBOF-style storage appliances [74]. Prior work and products such as LEED [21], Gimbal [22], NVMe-oF target offload [75], [76], Ditto [77], and FORD [78] reduce host CPU overhead or improve remote storage access by offloading indexing, storage processing, and RDMA/NVMe-oF data movement.

HyMCache takes a different approach. Rather than optimizing a storage-oriented path to remote flash, HyMCache builds a memory-addressable remote KV tier using CXL-HM In this design, SSD-backed capacity is exposed as remote memory, while device-side DRAM is managed as a staging layer for latency-critical KV reads. This avoids relying on DPU cores or storage-node software to mediate every SSD-resident KV access, and instead uses CXL-HM as a cost-efficient remote KV-cache backend for predictable KV reuse in multi-turn LLM serving.

8 Conclusion↩︎

This paper presents HyMCache, a KV-cache framework that integrates CXL-hybrid memory for multi-turn LLM serving. Leveraging the read-dominant, predictable, and append-only access patterns of multi-turn KV caches, HyMCache rethinks DRAM management within CXL-HM through request-level prefix prefetching and opportunistic write buffering. This allows latency-critical reusable KV blocks to be staged in device DRAM while relying on SSD-backed capacity for TB-scale KV reuse. Our evaluation on a real CXL-HM prototype under both single-aggregator and PD-disaggregated serving shows that HyMCache enables scalable and cost-efficient remote KV-cache capacity for future LLM serving systems.

References↩︎