Abstract

Memory tiering is a well established technique to tackle the increasing server memory total cost of ownership (TCO) and growing memory demands of data center workloads. A tiering solution dynamically places frequently accessed data (hot) to a small, fast, expensive memory tier known as near memory and infrequently accessed (cold) data to a low-cost, slow, capacity memory tier known as far memory. Precise identification of hot and cold data is the key to efficient tiering. However, in a virtualized setup where the host uses huge pages, host-level tiering can be inefficient when (i) hot base pages are scattered across guest physical address space, mapping to many host huge pages, or (ii) accesses within a guest huge page are skewed to a few base pages. In both cases, the host sees entire corresponding mapped huge pages as hot, and places them in near memory, wasting the precious resource.

We propose GPAC, a guest-side technique that exploits the two-level address translation to consolidate scattered and skewed hot base pages into a compact set of guest physical address ranges. This transforms many skewed hot huge pages into a few densely hot huge pages at the host level, enabling existing host-based tiering solutions to use near memory effectively. GPAC is host-agnostic, requires no hardware or hypervisor modifications, and is compatible with any host-based tiering solution. Our evaluation with DRAM as near memory and CXL as far memory shows 50–70% reduction in near memory consumption at similar performance levels with standalone real-world benchmarks and state-of-the-art host-based tiering. At scale, with multiple guests, GPAC improves performance by 4–11% with similar memory TCO.

1 Introduction↩︎

Memory tiering [1]–[8] on a heterogeneous memory system has emerged as a practical way to address the increasing cost of memory in data centers, which now accounts for 33–90% of the total cost of ownership (TCO) [2], [9], [10] and is projected to grow further with demands of modern data-hungry applications [11]–[13]. Memory tiering solutions aim to strike a balance between performance and TCO savings. A tiering solution periodically identifies frequently accessed (hot) data pages and places them in a small, fast, and expensive “near memory” tier (HBM, DRAM). It also moves infrequently accessed (cold) data pages to a large, slow, and cost-effective “far memory” tier (CXL, NVMM).

Figure 1: In a virtualized setup with host and guest using huge pages, \approx52% of Redis’s total memory footprint (running inside the guest) is incorrectly identified as hot by an host-based tiering technique using ideal page tracking, and is placed in the near memory(fast memory). — Figure 1: In a virtualized setup with host and guest using huge pages, \(\approx\)52% of Redis’s total memory footprint (running inside the guest) is incorrectly identified as hot by an host-based tiering technique using ideal page tracking, and is placed in the near memory(fast memory).

In modern data centers operating in a virtualized setup, memory tiering is employed at the host level. Cloud Service Providers (CSPs) enable huge pages at the host due to their performance benefits [14], weiwei?, while guests are free to use either huge pages or base pages. The guest physical address (GPA) has a one-to-one mapping with the host virtual address (HVA), and host-level tiering tracks hotness at HVA granularity, which corresponds to a huge page at the host.

However, host-based tiering can be rendered ineffective when in a guest using huge pages, accesses within a guest huge page are skewed to a few base pages or in a guest using base pages, hot base pages are scattered in the guest physical address space. Since guest pages are mapped to host huge pages, any access within the huge page boundary, either skewed or scattered, will make the corresponding huge page at host appear hot [6], [15], resulting in several skewed hot huge pages at the host level. As memory tiering techniques are based on page hotness, skewed hot huge pages are placed in near memory, leading to under-utilization of costly near memory resources [15], [16] and reducing opportunities to place other guests’ hot huge pages in near memory. As shown in Figure 1, for Redis workload running inside a VM and the host always using huge pages, \(\approx 52\%\) of the total memory footprint is incorrectly identified as hot by a host-based tiering technique using ideal page tracking. All the identified hot pages are moved to near memory.

In a non-virtualized setup, [6] [6] addresses this problem by using Intel PEBS [17] to track memory access at byte level and splits skewed hot huge page into base 4 KB pages. However, in a virtualized setup, hardware-assisted byte-level access tracking is not available for guests [15], [18], and host-level tiering solutions must use page table-based memory access tracking, which only tracks at page granularity. The state-of-the-art, HugeScope [15], proposes identifying skewed hot huge page by splitting huge pages. Although this solves the issue of skewed hot huge page, it loses the benefit of huge pages at the host level. Moreover, HugeScope requires extensive modifications at the hypervisor level for efficient huge page splitting and coalescing.

We propose Guest Physical Address Consolidation or GPAC, a novel, configurable guest-side technique that enhances memory tiering at the host level by significantly reducing the number of skewed hot huge pages at the host while maintaining huge pages at the host level. GPAC is agnostic to the host-level tiering solution and memory tiers, as well as the telemetry technique inside the guest. It requires no support from the host and works without any hardware or hypervisor modifications. GPAC identifies skewed hot huge pages inside the guest and only demotes them within the guest if the guest is using huge pages; huge pages at the host are never demoted, allowing both the host and the guest to continue benefiting from huge pages.

GPAC exploits the two-level address translation and consolidates hot base pages from multiple skewed hot huge pages into a few densely hot huge pages. This enables memory tiering techniques in the host to effectively place “actual" hot huge pages in near memory, reducing per-VM near memory consumption with minimal performance degradation. GPAC only modifies the GVA to GPA mapping; the HVA to HPA mapping remains untouched, and the host continues to get benefits of huge pages. To control the aggressiveness of consolidation and limit its impact on performance, GPAC introduces a tunable parameter called Consolidation Limit or CL that determines whether a hot huge page inside the guest is classified as skewed. A higher CL favors near memory savings at the cost of performance, while a lower CL prioritizes performance over memory savings.

Our main contributions are as follows:

We propose GPAC, a host-agnostic, guest-side consolidation technique that reduces the number of skewed hot huge pages at the host without splitting huge pages at host level. GPAC requires no hardware or hypervisor modifications.
GPAC introduces a tunable consolidation limit (CL) that allows users to control the trade-off between near memory savings and performance.
Our evaluation with state-of-the-art host-level tiering solutions shows that GPAC reduces near memory consumption by 50–70% for a single VM with minimal performance overhead, and improves throughput by 4–11% at scale with multiple VMs at similar memory TCO.

2 Background and Related Work↩︎

2.1 Two Dimensional Address Translation↩︎

In a bare metal setting, translating a virtual address (VA) to a physical address (PA) requires a single-level page table walk on a TLB miss. However, in a virtualized environment, a 2-dimensional (2D) walk or nested walk is required to convert the guest virtual address (GVA) to the host physical address (HPA). The first level walks the guest page table or GPT to translate GVA \(\xrightarrow{GPT}\) GPA, while the second level walks the host page table or HPT to translate HVA \(\xrightarrow{HPT}\) HPA. In virtualized environments such as QEMU/KVM [19], the GPA has a one-to-one linear mapping with the host virtual address (HVA). To reduce the translation overhead, hardware-assisted virtualization solutions like Extended Page Table (EPT) [20] or Rapid Virtualization Indexing (RVI) [21] were developed to reduce address translation overhead. However, in the worst case, GVA to HPA translation requires 24 memory accesses with a 4-level page table and 35 in 5-level page table [22], [23].

Translation with huge pages: A huge page is a contiguous group of base pages, virtually and physically. The supported huge page sizes depend on CPU and operating system support ( for example, huge pages of sizes 2 MB and 1 GB on Intel x86 [24]). In a virtualized setting, huge pages increase TLB reach and reduce TLB miss latency by eliminating the last level of the page table walk.

2.2 Telemetry — Access Profiling↩︎

Generating an accurate access profile (page addresses and corresponding access counts) is the key to effective memory tiering [25]. State-of-the-art telemetry techniques either rely on page table entries [26]–[28] or hardware counters [17] to build the access profile.

Access Bit tracking: These methods [25], [27], [29] rely on the manipulation and monitoring of ACCESSED (\(A\)) bit in the page table, which is automatically set by the hardware upon a page table walk [25]. A software daemon periodically clears this bit and checks it again after a certain time window. If the bit is set, that indicates that the page was used; otherwise, not.

NUMA Hints: Another set of methods [4], [28] rely on the PROT_NONE or \(P\) bit to capture the access profile. A daemon sets the \(P\) bit temporarily marking pages as inaccessible. A subsequent access to those pages triggers a minor fault. The access is recorded while serving the minor fault and later used to identify hot data pages.

Hardware counters: In hardware-based telemetry, the hardware collects statistics about the memory access pattern based on certain events such as LOAD and STORE instructions. It collects the corresponding virtual memory addresses and reports them to the OS or the software. Intel PEBS [17] is a hardware-based mechanism to collect the memory access pattern of an application and has been used by different memory tiering solutions [3], [5], [6].

2.3 Memory Tiering↩︎

Memory tiering is a widely adopted solution to address the growing cost of memory in data centers. CSPs employ a heterogeneous memory system with a fast, costly, and small-capacity memory tier such as DRAM or HBM, and a slow, cost-effective, and large-capacity memory tier such as CXL-attached memory [30], [31] and NVMM [32], henceforth referred to as near memory and far memory, respectively. A tiering solution dynamically places the data across these different memory tiers based on the access pattern of the application such that most of the LLC misses are served from a near memory while ensuring that cold data is placed on far memory. There is a plethora of memory tiering solutions proposed by academia and industry [1]–[8].

These tiering solutions differ based on three parameters: the telemetry technique used to collect data page access information (§2.2), the choice of memory tiers, such as compressed memory [2], NVMM [3], [5]–[8], or CXL-attached memories [3], [4], [6], and the policy used to promote or demote pages across tiers, which may rely on access counts [1], [5], fault counts [3], [4], [28], or a combination of events [8].

Memory tiering in virtualized environment: In a virtualized setup, a memory tiering solution

3 Motivation↩︎

3.1 Access Skewness in a Huge page↩︎

Figure 2: Address translation in a virtualized environment and the issue of Skewed hot huge pages. Host has four skewed hot huge pages: two from Guest1 and two from Guest2. — Figure 2: Address translation in a virtualized environment and the issue of *Skewed* hot huge pages. Host has four skewed hot huge pages: two from Guest1 and two from Guest2.

Huge pages alleviate pressure on the TLB and page caching structures, and are also recommended in virtualized environments (both guest and host) to improve application performance, as they reduce TLB pressure and Extended Page Table (EPT) walks on TLB misses [22], [33]. However, they introduce challenges for memory tiering in the presence of skewed hot huge page, where a few hot base pages can make the entire huge page hot [6]. A hot huge page is considered skewed if the number of hot base pages is below a defined skewness threshold. In a virtualized environment, CSPs typically enable huge pages at the host due to the aforementioned benefits of huge pages. Guests can either use base pages or huge pages which are backed by huge pages at the host.

As shown in Figure 2, a skewed hot huge page in the host context can be due to either a skewed hot huge page inside the guest (mapped to a host huge page) or scattered hot base pages in the guest (mapped to different host huge pages). Figure 2 explains skewness in detail with an example of two guests, Guest1, which uses huge pages, and Guest2, which uses only base pages. Guest1 frequently accesses one guest huge page A and two base pages within guest huge pages B and C, resulting in a total of three hot huge pages in both guest and host due to one-to-one GPA to HVA mapping. Out of the three hot huge pages, A is an actual hot huge page, whereas B and C are skewed hot huge pages. Guest2 frequently accesses two base pages (D and E) mapped to different host huge pages leading to two skewed hot huge pages at the host level. A host-based memory tiering solution will place all five identified hot huge pages in the near memory, even though only one (A) is actually hot and remaining four are skewed hot huge page, wasting precious resources.

3.1.1 Skewness and Impact on tiering↩︎

Figure 3: CDF of the number of accessed 4 KB base pages within a 2 MB huge page for a set of real-world workloads. The dots represent the knee points and are discussed in Section §4.4.2 to be used by GPAC.

We measure the amount of skewness present in a set of real world applications. Workload details and setup are provided in Table 2 and Section 5.1, respectively. Figure 3 shows the CDF of the number of 4 KB base pages accessed within each 2 MB huge page across different workloads. For a given value on the x-axis, y-axis indicates the fraction of huge pages that have that many or fewer base pages accessed. In Memcached [34] (with Memtier benchmark [35]), for \(\approx85\%\) of 2 MB huge pages, fewer than 100 of their 512 4 KB base pages are accessed. Hence, due to this skewness, 85% of the total 2 MB huge pages are marked hot even though less than 400 KB of data within those individual huge pages is accessed. In contrast, workloads such as Liblinear [36] and SPEC 654.Roms [37] have most of their huge pages densely hot.

Impact on tiering: Host-based memory tiering techniques identify hot data pages of all the guest instances and place them in the fast memory tier. As these solutions cannot differentiate between a dense hot huge page and a skewed hot huge page, they end up placing skewed hot huge pages in the limited-capacity and costly near memory tier. Our evaluation using Redis [38] shows that \(\approx52\%\) of host huge pages placed in near memory were skewed (Figure 1).

3.2 Limitations of State-of-the-art↩︎

Prior work, HugeScope [15], proposes changes to hypervisor to detect skewed hot huge page by splitting huge pages. This splitting solves the issue of skewed hot huge page as the host-level tiering solution can now track the access profile at base page granularity. All subsequent tiering operations such as migration from/to near memory are performed at the base page granularity. The solution comes at a cost of increased TLB-related activities at the host level. Moreover, HugeScope is built on top of vTMM [16] for memory tiering at host which needs additional guest support. Note that only splitting the guest huge page while retaining the corresponding huge page in the host only shifts the type of skewness from a skewed hot huge page in the guest to scattered hot base pages in the guest.

GPAC, a novel approach, addresses the problem of skewness in the guest without splitting the huge pages at the host level and is compatible with any memory tiering solution at the host. GPAC outperforms HugeScope by \(\approx\)4.2% in throughput at scale without any hypervisor modifications (§ 5.4.1).

4 GPAC Design and Implementation↩︎

In this section, we discuss the GPAC’s scope, design goals and the key idea. We also discuss the design components and their implementation. GPAC is applicable for any sized huge page. However, for ease of discussion we discuss assuming a 2 MB page composed of 512 4 KB base pages.

Huge page region: The design of GPAC is independent of whether a guest enables huge pages or not. We use the term huge page region to keep the discussion applicable to both the scenarios. Inside a guest, if huge pages are enabled, a huge page region is the same as a guest huge page. Otherwise, it is a contiguous, GPA-aligned region of huge-page size with no guest-side TLB benefits. A huge page region is always backed by a same-size huge page at the host.

4.1 Design goals↩︎

We design GPAC to achieve the following goals:

Host-agnostic: Develop a solution that does not need any modification in the host or hypervisor.
Tiering technique agnostic: Independent of any memory tiering solution at the host without requiring modifications.
Memory tier agnostic: Independent of memory tiers, such as DRAM, Intel Optane, CXL-attached memory, High Bandwidth Memory (HBM).
Telemetry agnostic: Any supported telemetry technique (mentioned in §2.2) can be used to identify skewed hot huge page in the guest..
Leverage huge pages: No splitting of huge pages at the host to get huge page benefits.

4.2 Scope↩︎

GPAC aims to mitigate skewed hot huge pages at the host-level while operating completely in the guest. The input to GPAC is a set of telemetry data (access counts) at base page granularity inside the guest. GPAC is responsible for identifying skewed hot huge pages based on the telemetry data and mitigating the issue. GPAC is not responsible for generating the telemetry data. Any existing supported state-of-the-art technique can be used with GPAC [4], [16], [26]. We assume that hosts always use huge pages to benefit from the higher TLB reach [39]–[41] and are free to use any supported memory tiering technique.

GPAC placement in Cloud: In a typical setup, the guest is not aware of the available memory tiers and the memory tiering at the host [42]. However, since GPAC is a guest-side technique, the cost of GPAC in terms of CPU cycles is paid by the guest. A natural question arises: what incentive does a guest have to bear this cost? We envision a cooperative model where CSPs offer GPAC-enabled VMs at a lower cost. GPAC enables better utilization of near memory at the host-level. This allows CSPs to host additional VMs at similar performance while also reduces rental costs, creating a “win-win” for both guest and host.

4.3 Key idea - Consolidation↩︎

A skewed hot huge page occurs at the host level when only a portion of a guest-mapped host huge-page is actively accessed. GPAC exploits the additional level of address translation inside a guest (GVA \(\xrightarrow{GPT}\) GPA) to “consolidate” scattered hot regions at the granularity of 4 KB base pages in the guest into a contiguous huge page size region inside the guest. Consolidation consists of data copy of scattered hot base pages and updating the \(GVA\xrightarrow{GPT}GPA\) mapping accordingly. By relocating the hot data out of the huge page region, the corresponding host huge page appears as cold. Thus, reducing the number of skewed hot huge pages at the host. This operates entirely within the guest and requires no modification at the host (hypervisor) level.

Figure 4: Hot base page consolidation inside guests to reduce the number of skewed hot huge pages at the host. All skewed hot huge pages are converted into dense hot huge pages. — Figure 4: Hot base page consolidation inside guests to reduce the number of *skewed* hot huge pages at the host. All *skewed* hot huge pages are converted into dense hot huge pages.

Example: Using the same example from earlier (§3.1), as shown in Figure 4, consolidation moves the content of hot 4 KB base pages \(B\) and \(C\) together in a contiguous huge-page sized region inside the guest (backed by an actual huge page at host) and modifies \(GVA \xrightarrow{GPT} GPA\) mappings inside Guest-1. Similarly, pages \(D\) and \(E\) inside Guest-2 are also “consolidated" to a single huge page region. As shown in Figure 4, after consolidation, all skewed hot huge pages at the host level are eliminated, resulting in three densely hot huge pages instead of five hot huge pages earlier, thus, reducing the amount of data in near memory.

4.4 GPAC components↩︎

Figure 5: Design components of GPAC for memory consolidation inside guest. Scattered Page Filter selects huge page region A, B, and D skewed based on CL value as 3. Scattered Page Filter pass the addresses A_1, A_6, B_1, B_3, D_2, and D_4 to Page Consolidator for consolidation at GPA level. — Figure 5: Design components of GPAC for memory consolidation inside guest. *Scattered Page Filter* selects huge page region A, B, and D skewed based on CL value as 3. *Scattered Page Filter* pass the addresses \(A_1\), \(A_6\), \(B_1\), \(B_3\), \(D_2\), and \(D_4\) to *Page Consolidator* for consolidation at GPA level.

At a high level, GPAC’s approach consists of the following steps. The first step is to collect telemetry data, i.e., access profile of data within guest at base page granularity. The second step is to apply a “policy" and identify skewed hot huge pages inside the guest. The third step is to address the skewed hot huge pages inside the guest. GPAC operates through guest user and kernel space and consists of two major components inside the guest: Scattered Page Filter and Page Consolidator as shown in Figure 5.

4.4.1 Scattered Page Filter↩︎

Scattered Page Filter operates in the guest’s address space and is responsible for identifying skewed hot huge pages inside the guest based on an access profile at base page granularity as an input and the value of a user-tunable knob we call consolidation limit or CL. Consolidation reduces the number of skewed hot huge pages at the host but incurs performance overhead due to data copy and page table updates inside the guest. GPAC uses CL to balance memory savings and performance overhead due to consolidation.

Consolidation candidate : GPAC profiles the application for a given time window \(T\) and collects the access profile using the preferred telemetry method. Based on the profile, GPAC counts the number of hot 4 KB base pages \(h_i\) for huge-page region \(i\). All huge page regions with \(h_i < {\tt CL}\) are classified as skewed hot huge pages and are candidates for consolidation. The candidate set is: \[\mathcal{C}(CL)=\{\, i \mid h_i < CL \,\}.\]

Memory savings and Work Done: For a highly skewed hot huge region (small \(h_i\)) consolidation offers large memory savings (\(512 - h_i\) cold pages) at low cost (\(h_i\) consolidations). On the other hand, for a low skewed hot huge region (high \(h_i\)) the memory savings are low and the consolidation cost is high. Given a candidate set for a particular CL, the total memory savings and total work done are defined as: \[S(CL) = \sum_{i \in \mathcal{C}(CL)} (N - h_i), \qquad W(CL) = \sum_{i \in \mathcal{C}(CL)} h_i,\] Here, \(N\) is the number of base pages inside a huge page (\(N\)=512 for a 2 MB huge page). We define the effectiveness of consolidation as: \[E(CL) = S(CL) - \lambda\,W(CL),\] where \(\lambda>0\) is the per-page migration cost and is system- and workload- dependent. \(E(CL)\) captures the net efficiency of consolidation for a given CL. A low value of \(\lambda\) indicates the migration cost is low and favors an aggressive consolidation. A high value of \(\lambda\) indicates high migration cost and favors a conservative consolidation. estimating \(\lambda\) directly is non-trivial, as it depends on the memory tier, memory pressure, bandwidth, and workload access pattern.

4.4.2 Default CL Values↩︎

To approximate a practical default CL, we apply the “Kneedle” algorithm [43] to the per-workload CDF of \(h_i\) across its huge page regions (see Figure 3). The detected “knee” point captures the transition from highly skewed to low-skewed regions, beyond which savings diminish relative to the work required. We use this knee as the default CL, capped at 60% of \(N\) (\(N=512\) for 2 MB) based on our empirical observation, as consolidation beyond this point yields diminishing returns.

CL tunability: CL is fully tunable in the range \([1,511]\) assuming a 2 MB huge page. A user can decrease CL to limit consolidation overhead at the expense of smaller memory savings or increase it beyond the default values to increase the memory savings at higher consolidation cost. We evaluate the impact of different values of CL on memory savings and performance in §5.2.3.

4.4.3 Page Consolidator↩︎

Once the consolidation candidate set of skewed hot huge pages is identified, the Page Consolidator, running in the guest’s kernel-space, demotes each candidate to base pages and consolidates the hot base pages into densely hot huge page regions. The set of skewed hot huge pages can serve as target regions where hot and cold base pages are swapped in place, without fresh page allocation as shown in Figure 5. Each swap requires three copy operations but opens optimization opportunities to reduce the total number of swaps.

Alternatively, the consolidator processes hot base pages in batches of up to 512. For each batch, it allocates a contiguous huge-page sized memory region, copies the content of each hot base page into the new region, and updates the corresponding GVA to GPA mappings. Since the GVA remains unchanged from the application’s perspective, the consolidation process is transparent to workloads. The newly allocated regions are not guest huge pages; they are huge-page sized regions at base page granularity. However, they are aligned and backed by actual huge pages at the host (§4.3). To ensure that the region allocation succeeds under memory pressure, a small pool of huge-page-sized regions can be pre-reserved.

4.5 Implementation↩︎

Figure 6: Page Consolidation: consolidate_pages() — Figure 6: Page Consolidation: `consolidate_pages()`

4.5.1 Scattered Page Filter↩︎

Scattered Page Filter requires an access profile at base page granularity to identify skewed hot huge pages.

4.5.2 Page Consolidator↩︎

Page Consolidator exposes an application-kernel interface to get necessary information from the guest user space, such as process pid and the candidate set of skewed hot huge pages to be consolidated (see Algorithm 6). The interface is implemented using a custom system call “consolidate_pages()” which takes the candidate set and their corresponding hot base pages. It then consolidates the hot base pages by using the allocation mechanism discussed in Section 4.4.3.

5 Evaluation↩︎

5.1 Experimental setup↩︎

5.1.1 Host configuration↩︎

Table 1 summarizes our testbed configurations. We use two hosts, Host1 and Host2, each equipped with different memory tiers and host-based memory tiering to evaluate the effectiveness of GPAC.

Tiering at Host and comparison: We evaluate GPAC with three different host-level tiering solutions (enabled one at a time): Memtierd [29], TPP [4], and vTMM [16]. We also compare against HugeScope [15], the current state-of-the-art for addressing page skewness in a virtualized setup. HugeScope is built on top of vTMM [15] .

Memtierd performs memory tiering using Idle Page Tracking (IPT) [26] and runs entirely in user space to perform tiering, enabling tiering even in the absence of memory pressure. TPP is part of the Linux mainline kernel and combines the LRU mechanism to detect cold pages and NUMA faults to detect hot pages. vTMM leverages Intel’s Page-Modification Logging (PML) [44] to track memory accesses by guests at the host to perform memory tiering. Memtierd and TPP require no guest-side support, whereas vTMM requires guest support.

Guest: We use the KVM hypervisor [19] to run guest(s) with the same Linux kernel version as in the host. Each guest runs a single application instance, configured with 20 GB of memory and 12 vCPUs. we restrict guests to using only 4 KB pages for GPAC consolidation as discussed earlier (§4.5.1). To avoid network bottlenecks, both server and client execute in the same guest. We refer to guests using 4 KB base pages and 2 MB huge pages as G:4K and G:2M, respectively, in the rest of the paper. The host always uses 2 MB huge pages.

Table 1: Experimental platforms used in our evaluation.

CPU	Xeon Gold 6554S	Xeon Gold 6252N
	(Emerald Rapids)	(Cascade Lake)
Memory	2 TB DRAM	64 GB DRAM
	1 TB CXL	512 GB NVMM
Kernel	Linux 6.12	Linux 5.18 / 5.4
Huge Page	always	always
Tiering	Memtierd	TPP, vTMM, HugeScope

Table 2: Benchmarks’ description with memory footprint.
Workload	Description	Guest RSS
Redis [38]	In-memory key-value store	12.5 GB
Memcached [34]	In-memory key-value store	11 GB
Hash [45]	Hash data structure	8.8 GB
Ocean_ncp [46]	Ocean simulation	5.5 GB
Masim [47]	Memory access simulator	9.8 GB

5.1.2 Workloads↩︎

We evaluate the effectiveness of GPAC using a microbenchmark and a set of real-world workloads. Table 2 lists the workloads along with their guest resident set size (RSS). For Redis [38] and Memcached [34], we populate data with a key size of 1 KB and use the Memtier [35] to generate requests as per a Gaussian distribution. For the Hash workload, we use Hash_bkt_rcu [45], a hash table protected by a per-bucket lock for updates and RCU for lookups. We also use Ocean_ncp, an ocean simulation with a W-cycle multigrid solver from the SPLASH2x application in PARSEC 3.0 [46]. We configure the microbenchmark Masim [47] to access only one 4 KB page out of 512 pages in a huge page boundary.

Workload		Hot 4 KB pages	Skewed hot huge pages	Consol. Time (ms)
Redis	48	91,959	3,638	257
Memcached	78	140,977	2,706	376
Hash	12	307,480	3,389	790
Ocean_ncp	290	360,641	1,402	802
Masim	2	4,106	4,102	15

Consolidation time of selected hot 4 KB pages using for different workloads along with the share of each operation (see Algorithm [algo:page95consol]) during consolidation.
alloc	unmap	flush	copy	set_pte	free
19.9%	3.2%	24.9%	13.2%	2.9%	17.9%

Default CL values: As discussed earlier, we use the consolidation limit – CL to classify a hot huge page inside the guest as skewed hot huge page or not. We profile workloads to analyze their memory access pattern and set the application-specific CL based on the access pattern. Table ¿tbl:tab:consol95time? shows the CL, selected hot base pages, identified skewed hot huge pages, and the time taken to perform consolidation. Figure 3 shows the knee point for different workloads. GPAC does not perform consolidation for workloads with left-skewed CDF like Liblinear and SPEC 654.Roms, in which most of the huge pages are densely hot. Redis is right-skewed, where the bulk of the data is concentrated towards low values, resulting in better consolidation opportunity. In contrast, SPEC 654.Roms_s is left-skewed, with the bulk of the data concentrated towards high values.

5.2 Evaluation with Single Guest Instance↩︎

We start evaluation of GPAC with a single guest under no near memory pressure to quantify the extent of DRAM savings using Host1. All consolidation processes succeed due to sufficient available contiguous memory regions. We analyze memory pressure scenarios in §5.3 and §5.4. Here, we use Memtierd as the representative tiering mechanism at host to migrate identified hot pages to the near memory.

Figure 7 summarizes the total near memory savings and performance impact across all workloads. GPAC achieves an average near memory reduction of \(70\%\) (excluding the microbenchmark Masim) while incurring a negligible average performance overhead of \(\approx\)0.86%. GPAC completely operates within the guest while the host-based tiering migrates data pages purely based on its own hot/cold classification method and policies.

Figure 7: Reduction in DRAM usage with GPAC+ Memtierd compared with standalone Memtierd. GPAC reduces near memory usage with minimal performance impact. Normalized to all-DRAM setting without memory tiering.

We now perform a deep dive on memory tiering, and performance impact using Redis as the representative workload as it exhibits significant access skewness (see Figure 3).

5.2.1 Memory Tiering impact↩︎

On Host1, we start the Redis with far memory(CXL) as the preferred memory tier. When tiering begins around 180 seconds, Memtierd without GPAC identifies 85% of data as hot and migrates it to near memory(Figure 8 (a)). In contrast, with GPAC consolidating skewed hot huge regions inside the guest, Memtierd+ GPAC identifies only 33% (a reduction of \(\approx\) 52%) as hot and migrates to the near memory. Figure 9 captures the impact of consolidation using DAMON [27] to capture the detected hot regions at the host. Figure 9 (a) shows the identified hot regions without GPAC. The figure shows a hot region around the 11 GB offset in the address space, which decreases in intensity (no. of accesses) as we move away from this hot region. Figure 9 (b) shows the detected hot regions after the consolidation with GPAC. The low intensity scattered hot regions get consolidated into smaller regions. This conclusively shows the impact of GPAC on the detected hot region, and eventually on the underlying memory tiering.

5.2.2 Performance impact↩︎

The overall impact of the tiering on an application in terms of two critical metrics – loads from far memory and LLC miss latency remains unaffected. In an ideal tiering solution, all the LLC misses will be served from the near memory causing a reduction in LLC miss latency (compared to a miss served from a far memory). We compare the number of demand data loads from CXL and LLC miss latency (captured using perf¹ ) for non-tiered (DRAM-only) and tiered (Memtierd and Memtierd+ GPAC) setups.

As shown in Figure 10 (a), the number of demand loads from CXL before tiering begins is identical for the two tiering setups. After tiering activates, the number of CXL accesses drops sharply and approaches DRAM-only levels. Notably, Memtierd+ GPAC places only 33% of data in DRAM compared to 85% for just Memtierd, confirming that GPAC enables efficient detection of actual hot data (as the drop in CXL loads is identical in both cases). Figure 10 (b), capturing LLC miss latency, also shows a similar pattern. We observe similar consistent behavior across other workloads.

5.2.3 Impact of Consolidation Limit (CL)↩︎

The consolidation limit (CL) balances the trade-off between near memory savings and the consolidation overhead. A higher CL may classify more huge page regions as skewed (depending on the access pattern), triggering aggressive consolidation. Conversely, a lower CL only classifies highly skewed regions, resulting in conservative consolidation.

Figure 11 shows near memory(DRAM) savings for the Hash workload across different consolidation limits (CL). A CL value of 50 results in a memory saving of \(\approx58\%\). As CL increases beyond 150, near memory usage drops substantially, but we also observe a throughput degradation of \(\approx14\%\) relative to no-consolidation performance. The performance overhead is due to aggressive consolidation process. A CL value of more than 250 results in no additional near memory savings and even no additional performance overheads, indicating only a few skewed huge page regions with more than 250 hot base pages (as confirmed by the CDF in Figure 3).

Figure 11: Impact of CL on DRAM usage for Hash workload.

5.3 At-scale evaluation using Memtierd↩︎

In this section, we measure the efficacy of GPAC in a multi-tenant scenario in a near memory pressure setting. In such a setting, near memory used by one guest can impact the performance of other guests. We configure multiple guests (each hosting one Redis workload with Memtier) on the same host to simulate DRAM pressure. We configure guests to prefer near memory first and use far memory if additional memory is required.

We use Host1 with Memtierd tiering as the host-level tiering solution to perform memory tiering across six guests, each running the same workload. Guests are allocated a shared pool of 30 GB DRAM and 70 GB CXL. For performance assessment, each guest runs Redis and Hash workloads independently with the same setting as the single VM scenario.

5.3.1 Near memory distribution impact↩︎

As shown in Figure 12, tiering with Memtierd leads to an uneven distribution of DRAM across guests, with \(VM_1\) consuming \(\approx\)40% of DRAM, while the remaining 60% of DRAM is shared by the other 5 VMs. This uneven distribution is due to skewed hot huge page, where the majority of data pages in guests are incorrectly identified as hot, and hence, they continue to occupy space in the already constrained near memory(DRAM). Guests using either base pages (G:4K) or huge pages (G:2M) suffer from the same issue.

With GPAC enabled in all the guests, the number of skewed hot huge pages is reduced at the host, resulting in correct identification of hot and cold huge pages. Cold huge pages are subsequently demoted to far memory, freeing up DRAM for actual hot huge pages. Since the DRAM capacity is sufficient to hold all the actual hot data from the guests, and as all guests run the same workload, we see an even distribution of DRAM usage.

Figure 12: DRAM distribution across guests using G:4K and G:4K+GPAC for Redis workload (Memtierd at host).

5.3.2 Impact on Promotion-Demotion↩︎

We monitor memory pages being promoted from CXL to DRAM and demoted from DRAM to CXL. As shown in Figure 13 (a), a number of promotions fail in G:4K due to access skewness and DRAM scarcity, leading to incorrect placement of hot huge pages. We observe similar behavior in the case of G:2M. In contrast, with G:4K+ GPAC(Figure 13 (b)), the total number of failed promotions drops significantly, as consolidation reduces the number of skewed hot huge pages and frees DRAM for actual hot pages, enabling correct hot-cold placement.

5.3.3 Impact on application performance↩︎

Figure 14: Performance improvement in Redis workload while using Memtierd for memory tiering at host.

Due to improved DRAM distribution, G:4K+ GPAC delivers an average performance gain of \(\approx\)7.25% and \(\approx\)4.72% over G:4K and G:2M, respectively, as shown in Figure 14 for Redis. All guests show a performance improvement of 5% – 12% with G:4K+ GPAC, compared to G:4K, with the exception of \(VM_1\). For \(VM_1\), we see a performance degradation of only 0.34% even though its DRAM usage is down from \(\approx\)40% to just \(\approx\)18% with G:4K, since only the actual hot huge pages from \(VM_1\) are occupying DRAM. Overall, GPAC provides better system performance due to a reduction in the number of skewed hot huge pages at the host and a better utilization and distribution of near memory. For the Hash workload, G:4K+ GPAC improves performance by \(\approx\)26% compared to G:4K.

Performance deep dive: We observe significant performance gain while using G:4K+ GPAC, compared to G:4K and G:2M. We use hardware performance counters (Table 3) to capture key statistics related to execution stalls, CXL memory accesses, and LLC miss latency. We observed that G:4K+ GPAC reduces execution stalls by 23.4% and 10.6% on average as shown in Figure 15 (a), compared to G:4K and G:2M, respectively. Figure 15 (b) shows a significant drop in loads served from CXL for the entire host (by 99%) with G:4K+ GPAC, demonstrating efficient DRAM utilization. As a result, host-wide LLC miss latency for demand loads is reduced by \(\approx\)20.2% with G:4K+ GPAC as shown in Figure 15 (c).

Table 3: Hardware performance counters used in the analysis.
Counter	Description
`memory_activity.` `stalls_l3_miss`	Execution stalls due to outstanding LLC miss demand loads
`unc_cha_tor_inserts.` `ia_miss_drd_cxl_acc`	Demand loads served from CXL-attached memory
`llc_demand_data_read_` `miss_latency`	LLC miss latency for demand data reads

5.3.4 Effectiveness under varying memory pressure↩︎

We analyze the impact of GPAC under varying near memory pressure using 6 guests, each running Redis. We keep the total memory fixed at 100 GB and adjust the DRAM-to-CXL ratio. At 20:80 and 30:70 DRAM-to-CXL ratio, Memtierd+GPAC outperforms standalone Memtierd by \(\approx6\%--8\%\) in terms of average throughput across the 6 guests, demonstrating its effectiveness under high near memory pressure. However, with 40:60 and 70:30 ratio, as DRAM availability increases, the performance gap narrows as the workloads can fit comfortably in DRAM and the advantage comes down to \(\approx\)4% and \(\approx\)2%, respectively. In summary, Memtierd+ GPAC offers clear benefits in DRAM-constrained environments, though its advantage decreases as more near memory becomes available and skewed hot pages can be accommodated without tiering.

5.3.5 Huge pages with tiering and skewness↩︎

Figure 16: Performance and DRAM memory consumption of Redis for different combination of base (4K) and huge page (2M) in host and guest. Encoding: [Guest]-[Host]. Normalized to 4K-4K setting. Tiering performed based on Idle Page Tracking [26].

In a virtualized setup, in the absence of tiering, both guest and host using huge pages (G``2M``:H``2M) outperforms all other combinations by up to \(\approx\)20% (Figure 16) [22]. However, as discussed earlier, using 2 MB pages in the guest allows for only coarse-grained access tracking and does not allow for skewness detection. Therefore, GPAC uses base pages in the guest and huge pages at the host (G``4K``:H``2M). We show that, with host-level tiering enabled in the presence of skewed hot huge pages, G``4K``:H``2M outperforms G``2M``:H``2M in terms of performance and fast memory savings due to better utilization of fast memory

5.4 At-scale with Various Tiering Solutions↩︎

In this section, we show the performance of GPAC with different state-of-the-art tiering solutions at the host: TPP [4], vTMM [16], and HugeScope [15].

5.4.1 GPAC vs. vTMM and HugeScope↩︎

Figure 17: Performance improvement in Redis workload while using vTMM for memory tiering at host.

Figure 18: Stalls due to memory subsystems for Redis workload while using vTMM for memory tiering at host.

vTMM leverages Intel PML to perform memory tiering using guest support. HugeScope [15] proposes a page splitting/coalescing policy to identify skewed hot huge pages by modifying the hypervisor. HugeScope is built on top of vTMM to perform memory tiering. We use Host2 with Linux kernel 5.4 in guests and host with NVMM support to compare GPAC with HugeScope to maintain a similar test environment². To show effectiveness using vTMM and HugeScope as the memory tiering technique at host, we hosted six guests running Redis workload (each with 12.5 GB RSS as shown in Table 2) to create pressure on DRAM. Guests are allocated a shared pool of 20 GB DRAM and 80 GB NVMM.

Performance comparison: We compare the performance of G:4K+ GPAC with vTMM against both G:2M with vTMM and HugeScope. HugeScope improves performance by an average of \(\approx\)4.1% over G:2M with vTMM. However, G:4K+ GPAC with vTMM improves performance by an average of \(\approx\)8.5% and \(\approx\)4.2% over G:2M with vTMM and HugeScope, respectively, as shown in Figure 17.

Promotion-demotion and stall analysis: We are unable to collect promotion-demotion stats for HugeScope and vTMM as they do not integrate the memory tiering solution with the existing NUMA interface in the Linux kernel. However, we performed analysis of execution stalls due to memory subsystems at the host level using a hardware performance counter (cycle_activity.stalls_mem_any). We observe that HugeScope and G:4K+ GPAC with vTMM reduce stalls due to memory subsystems, compared to G:2M with vTMM, by an average of \(\approx\)16.6% and \(\approx\)28.4%, respectively, as shown in Figure 18.

5.4.2 GPAC vs. TPP↩︎

Figure 19: Performance improvement in Redis workload while using TPP for memory tiering at host.

TPP [4] is a state-of-the-art memory tiering solution available in the Linux kernel. TPP performs memory tiering only under memory pressure by demoting cold pages to far memory tiers and promoting hot pages to the near memory tier. We use the same machine (Host2) to show the effectiveness of GPAC using TPP as the memory tiering technique at host. We hosted eight guests running Redis workload to create pressure on DRAM. We need additional guests as TPP only tiers in extreme memory pressure scenarios.

Performance comparison: G:4K+ GPAC with TPP improves the performance by an average of \(\approx 11\%\) for all eight guests over G:4K with TPP as shown in Figure 19. The performance of VM7 and VM8 improves by 20% and 18%, respectively.

Figure 21: Stalls due to memory subsystems for Redis workload while using TPP for memory tiering at host

Promotion-demotion and stall analysis: Figure 20 (a) and Figure 20 (b) show the data promoted from NVMM to DRAM and data demoted from DRAM to NVMM, respectively, with G:4K and G:4K+ GPAC with TPP. We observe the G:4K+ GPAC with TPP reduces the total data promoted and demoted by an average of \(\approx64\%\) and \(\approx87\%\), respectively, due to the consolidation of hot pages inside the guest to a few sets of hot huge pages at the host level. In case of TPP, we observed significant reduction in stalls by up to 42% as shown in Figure 21.

6 Conclusion↩︎

Efficient memory tiering reduces Total Cost of Ownership by minimizing expensive near memory usage. We introduced GPAC, a guest physical address space consolidation mechanism that reduces the number of skewed hot huge pages at the host by consolidating scattered hot base pages within the guest. GPAC is host-agnostic, requires no changes to host-side tiering mechanisms, the hypervisor, or hardware, and is compatible with any telemetry technique and memory tier.

GPAC introduces a tunable consolidation limit (CL) that allows users to control the trade-off between near memory savings and performance overhead. For a single VM, GPAC reduces near memory consumption by 50–70% with minimal performance overhead. At scale, across different memory technologies, physical machines, and host-level tiering solutions such as Memtierd (IPT), vTMM (with and without HugeScope), and TPP, GPAC improves application performance by 4–11% while significantly lowering near memory usage. Notably, GPAC with base pages inside the guest outperforms huge pages inside the guest under near memory pressure, due to correct placement of densely hot huge pages in near memory.

References↩︎

[1]

A. Lagar-Cavilla et al., “Software-defined far memory in warehouse-scale computers,” in International conference on architectural support for programming languages and operating systems, 2019, [Online]. Available: http://doi.acm.org/10.1145/3297858.3304053.

[2]

J. Weiner et al., “TMO: Transparent memory offloading in datacenters,” in Proceedings of the 27th ACM international conference on architectural support for programming languages and operating systems, 2022, pp. 609–621, doi: 10.1145/3503222.3507731.

[3]

J. Kim, W. Choe, and J. Ahn, “Exploring the design space of page management for Multi-Tiered memory systems,” in 2021 USENIX annual technical conference (USENIX ATC 21), Jul. 2021, pp. 715–728, [Online]. Available: https://www.usenix.org/conference/atc21/presentation/kim-jonghyeon.

[4]

H. A. Maruf et al., “TPP: Transparent page placement for CXL-enabled tiered-memory,” in Proceedings of the 28th ACM international conference on architectural support for programming languages and operating systems, volume 3, 2023, pp. 742–755, doi: 10.1145/3582016.3582063.

[5]

A. Raybuck, T. Stamler, W. Zhang, M. Erez, and S. Peter, “HeMem: Scalable tiered memory management for big data applications and real NVM,” in Proceedings of the ACM SIGOPS 28th symposium on operating systems principles, 2021, pp. 392–407, doi: 10.1145/3477132.3483550.

[6]

T. Lee, S. K. Monga, C. Min, and Y. I. Eom, “MEMTIS: Efficient memory tiering with dynamic page classification and page size determination,” in Proceedings of the 29th symposium on operating systems principles, 2023, pp. 17–34.

[7]

J. Ren, D. Xu, J. Ryu, K. Shin, D. Kim, and D. Li, “MTM: Rethinking memory profiling and migration for multi-tiered large memory,” in Proceedings of the nineteenth european conference on computer systems, 2024, pp. 803–817, doi: 10.1145/3627703.3650075.

[8]

P. Duraisamy et al., “Towards an adaptable systems architecture for memory tiering at warehouse-scale,” in Proceedings of the 28th ACM international conference on architectural support for programming languages and operating systems, volume 3, 2023, pp. 727–741, doi: 10.1145/3582016.3582031.

[9]

VMware, Accessed: 2025-05-14“Memory tiering: Power your server infrastructure with memory innovations.” https://www.vmware.com/explore/video-library/video/6360757998112, 2024.

[10]

“Decadal plan for semiconductors.” Semiconductor Research Corporation; https://www.src.org/about/decadal-plan/decadal-plan-full-report.pdf, 2020.

[11]

J. W. Rae et al., “Scaling language models: Methods, analysis & insights from training gopher,” ArXiv, vol. abs/2112.11446, 2021.

[12]

S. Smith et al., “Using DeepSpeed and megatron to train megatron-turing NLG 530B, a large-scale generative language model,” ArXiv, vol. abs/2201.11990, 2022.

[13]

T. Brown et al., “Language models are few-shot learners,” in Advances in neural information processing systems, 2020, vol. 33, pp. 1877–1901, [Online]. Available: https://proceedings.neurips.cc/paper_files/paper/2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf.

[14]

G. Cox and A. Bhattacharjee, “Efficient address translation for architectures with multiple page sizes,” ACM SIGPLAN Notices, vol. 52, no. 4, pp. 435–448, 2017.

[15]

C. Li et al., “Taming hot bloat under virtualization with HUGESCOPE,” in 2024 USENIX annual technical conference (USENIX ATC 24), Jul. 2024, pp. 999–1012, [Online]. Available: https://www.usenix.org/conference/atc24/presentation/li-chuandong.

[16]

S. Sha, C. Li, Y. Luo, X. Wang, and Z. Wang, “vTMM: Tiered memory management for virtual machines,” in Proceedings of the eighteenth european conference on computer systems, 2023, pp. 283–297.

[17]

A. R. Nonell, B. Gerofi, L. Bautista-Gomez, D. Martinet, V. B. Querol, and Y. Ishikawa, “On the applicability of PEBS based online memory access tracking for heterogeneous memory management at scale,” in Proceedings of the workshop on memory centric high performance computing, 2018, pp. 50–57.

[18]

“Avoid unnecessary work in guest filtering.” https://lists.ubuntu.com/archives/kernel-team/2019-November/105645.html, 2019.

[19]

A. Kivity, Y. Kamay, D. Laor, U. Lublin, and A. Liguori, “Kvm: The linux virtual machine monitor,” in Proceedings of the linux symposium, 2007, vol. 1, pp. 225–230.

[20]

“Performance Evaluation of Intel EPT Hardware Assist.” https://www.vmware.com/pdf/Perf_ESX_Intel-EPT-eval.pdf, 2009.

[21]

VMware, “Performance evaluation of AMD RVI hardware assist,” VMware, 2008.

[22]

T. Merrifield and H. R. Taheri, “Performance implications of extended page tables on virtualized x86 processors,” in Proceedings of The12th ACM SIGPLAN/SIGOPS international conference on virtual execution environments, 2016, pp. 25–35, doi: 10.1145/2892242.2892258.

[23]

S. Bergman, M. Silberstein, T. Shinagawa, P. Pietzuch, and L. Vilanova, “Translation Pass-Through for Near-Native paging performance in VMs,” in 2023 USENIX annual technical conference (USENIX ATC 23), Jul. 2023, pp. 753–768, [Online]. Available: https://www.usenix.org/conference/atc23/presentation/bergman.

[24]

M. Gorman, [Online; accessed 2025-03-16]“Huge pages part 1 (introduction) [LWN.net].” 2010, [Online]. Available: https://lwn.net/Articles/374424/.

[25]

A. Nair, S. Kumar, A. Prasad, Y. Huang, A. Rudoff, and S. Subramoney, “Telescope: Telemetry for gargantuan memory footprint applications,” in 2024 USENIX annual technical conference (USENIX ATC 24), Jul. 2024, pp. 409–424, [Online]. Available: https://www.usenix.org/conference/atc24/presentation/nair.

[26]

C. Hansen, “Linux idle page tracking.” 2018, [Online]. Available: https://www.kernel.org/doc/Documentation/vm/idle_page_tracking.txt.

[27]

S. Park, Y. Lee, and H. Y. Yeom, “Profiling dynamic data access patterns with controlled overhead and quality,” in Proceedings of the 20th international middleware conference industrial track, 2019, pp. 1–7, doi: 10.1145/3366626.3368125.

[28]

“AutoNuma: NUMA BALANCING MEMORY TIERING.” https://docs.kernel.org/admin-guide/sysctl/kernel.html, 2024.

[29]

A. Kervinen, (Accessed on 12/09/2024)“Memtierd.” https://github.com/intel/memtierd, 2024.

[30]

Y. Sun et al., “Demystifying CXL memory with genuine CXL-ready systems and devices,” in Proceedings of the 56th annual IEEE/ACM international symposium on microarchitecture, 2023, pp. 105–121, doi: 10.1145/3613424.3614256.

[31]

Inc. Micron Technology, [Online; accessed 2025-08-20]“Micron launches memory expansion module portfolio to accelerate CXL 2.0 adoption.” https://investors.micron.com/news-releases/news-release-details/micron-launches-memory-expansion-module-portfolio-accelerate-cxl, 2022.

[32]

S. Garg, A. Prasad, D. Mishra, and S. Subramoney, “Motivating next-generation OS physical memory management for terabyte-scale NVMMs.” 2023, [Online]. Available: https://arxiv.org/abs/2310.03370.

[33]

W. Jia, J. Zhang, J. Shan, and X. Ding, “Making dynamic page coalescing effective on virtualized clouds,” in Proceedings of the eighteenth european conference on computer systems, 2023, pp. 298–313, doi: 10.1145/3552326.3567487.

[34]

(Accessed on 11/18/2019)“Memcached - a distributed memory object caching system.” https://memcached.org/, 2019.

[35]

“Memtier Benchmark.” https://github.com/RedisLabs/memtier_benchmark, 2024.

[36]

R.-E. Fan, K.-W. Chang, C.-J. Hsieh, X.-R. Wang, and C.-J. Lin, “LIBLINEAR: A library for large linear classification,” the Journal of machine Learning research, vol. 9, pp. 1871–1874, 2008.

[37]

S. C. 2017, “Standard performance evaluation corporation.” https://www.spec.org/cpu2017/, 2024.

[38]

(Accessed on 11/18/2019)“Redis.” https://redis.io/, 2019.

[39]

B. N. Robert Pang, [Online; accessed 2025-03-16]“Linux huge pages | google cloud blog.” Aug. 2021, [Online]. Available: https://cloud.google.com/blog/products/databases/cloud-sql-postgresql-now-supports-linux-huge-pages.

[40]

tejasaks, [Online; accessed 2025-03-16]“Performance best practices for SQL server on linux - SQL server | microsoft learn.” Nov. 2024, [Online]. Available: https://learn.microsoft.com/en-us/sql/linux/sql-server-linux-performance-best-practices?view=sql-server-ver16.

[41]

Y. Kwon, H. Yu, S. Peter, C. J. Rossbach, and E. Witchel, “Coordinated and efficient huge page management with ingens,” in 12th USENIX symposium on operating systems design and implementation (OSDI 16), Nov. 2016, pp. 705–721, [Online]. Available: https://www.usenix.org/conference/osdi16/technical-sessions/presentation/kwon.

[42]

Lenovo Press, Whitepaper on VMware vSphere 9.0 host-based memory tiering with NVMe“Implementing memory tiering over NVMe using VMware ESXi 9.0,” Lenovo Press, LP2288, 2025. [Online]. Available: https://lenovopress.lenovo.com/lp2288.

[43]

V. Satopaa, J. Albrecht, D. Irwin, and B. Raghavan, “Finding a" kneedle" in a haystack: Detecting knee points in system behavior,” in 2011 31st international conference on distributed computing systems workshops, 2011, pp. 166–171.

[44]

S. Bitchebe, D. Mvondo, A. Tchana, L. Réveillère, and N. De Palma, “Intel page modification logging, a hardware virtualization feature: Study and improvement for virtual machine working set estimation,” arXiv preprint arXiv:2001.09991, 2020.

[45]

P. E. McKenney, (Accessed on 10/28/2024)“Hash datastructure.” https://github.com/paulmckrcu/perfbook/blob/master/CodeSamples/datastruct/hash/hash_bkt_rcu.c, 2024.

[46]

X. Zhan, Y. Bao, C. Bienia, and K. Li, “PARSEC3. 0: A multicore benchmark suite with network stacks and SPLASH-2X,” ACM SIGARCH Computer Architecture News, vol. 44, no. 5, pp. 1–16, 2017.

[47]

S. Park, (Accessed on 10/28/2024)“Memory access workload simulator.” https://github.com/sjp38/masim, 2024.

unc_cha_tor_inserts.ia_miss_drd_cxl_acc, llc_demand_data_read_miss_latency↩︎
HugeScope failed to boot on Host1↩︎

Efficient Memory Tiering in a Virtual Machine