PVLDB Reference Format:
. . PVLDB, 19(10): XXX-XXX, 2026.
doi:XX.XX/XXX.XX

¹

1 Introduction↩︎

2.8pt

Durable cloud logging is remote. On-premise database systems ensure durability through write-ahead logging (WAL) to a local disk or SSD. In the cloud, instance-local storage is ephemeral and can be lost on instance failure or deprovisioning [1]. Cloud database systems must therefore store their log on durable remote storage. An ideal cloud logging backend combines strong durability and availability with low-latency appends, low per-append cost, and low storage cost. 1 qualitatively compares logging backends along these dimensions.

Remote block storage: easy, but slow and expensive. A straightforward approach is remote block storage, such as Amazon EBS, which provides a network-attached volume that can be re-attached to a different VM after failure. EBS makes it straightforward to lift on-premise architectures to the cloud and is used in many deployments, including Amazon RDS [2]. However, even high-end EBS variants (e.g., io2) incur substantially higher write latency than instance-local SSDs and are expensive, with costs driven by both provisioned capacity and provisioned IOPS.

Object storage: great archive, poor WAL backend. Cloud object storage, such as S3, provides very high availability and durability, low cost per GB, and high throughput for scans. These properties make it attractive as a log archive, and we use it for this purpose in our design. However, placing a transactional WAL directly onto object storage (even its lower-latency variant, S3 Express) is impractical due to high latency and high per-append cost.

Log systems exist, but are not reusable. The limitations of general-purpose storage services for logging are well known, and many cloud-native database systems therefore rely on purpose-built logging backends. Examples include Microsoft Socrates’ XLOG [3], Huawei TaurusDB’s PLog [4], and Neon’s Safekeeper [5]. AWS’s proprietary internal “Journal” service is used by Amazon Aurora DSQL, S3, DynamoDB, Lambda, and Kinesis [6]. In contrast, the open-source log system Apache BookKeeper [7] was designed for a pre-cloud world and does not exploit modern low-latency networks and object storage for WAL archival.

BtrLog: fast quorum, cheap archive. We argue that logging is a fundamental primitive and should be available as reusable infrastructure. BtrLog (pronounced “better log”) is a cloud logging service that combines low-latency appends with low-cost archival. It replicates log records across multiple SSD-backed log nodes; for the common single-writer architecture, each append sends one request per log node and becomes durable once a quorum (e.g., 2 of 3 nodes) has persisted it, limiting the impact of stragglers on commit latency. Once a log segment (e.g., 16 MB) fills, it is asynchronously written to S3, exploiting S3’s low storage cost without suffering from its high latency. BtrLog can be deployed within a single data center (Availability Zone or AZ in AWS) for the lowest latency and cost, or across AZs to protect the unarchived log tail against datacenter-wide failures. As 1 shows, this design achieves high availability, low latency, and low per-append and storage cost.

Novelty. We build on established distributed systems ideas, combining them in a way specifically tailored to cloud database logging. To the best of our knowledge, BtrLog is the first cloud logging system that simultaneously

• achieves single network round-trip append latency for the common single-writer architecture,

• provides a reusable service interface instead of coupling logging to a specific engine,

• exploits cloud object storage for low-cost, high-durability archival without placing it on the critical path, and

• is engineered for high-speed datacenter networks and NVMe SSDs, with explicit attention to tail latency and throughput.

Microbenchmarks in realistic AWS setups show that BtrLog reduces append latency by up to 4 compared to the high-end io2 variant of EBS, and end-to-end experiments with LeanStore [8] show how these gains translate into higher transaction throughput.

2 Background: WAL in the Cloud↩︎

In this section, we briefly summarize WAL semantics and the resulting requirements on a logging backend, then review existing log systems and cloud storage services. The goal is to clarify the tradeoffs that motivate our design choices.

2.1 WAL Semantics and Requirements↩︎

Log interface. Write-ahead logging (WAL) in database systems requires only a small interface. During transaction execution, the DBMS appends log records via append [9]. Each record is assigned a monotonically increasing log sequence number (LSN). At commit, the engine calls sync to ensure all records up to the commit LSN are persisted – the ARIES paper [9] refers to this as forcing the log. For restart recovery, the engine issues a scan operation to retrieve log records starting from a given LSN. During recovery or rollback, it calls read to fetch individual records by LSN for undo processing.

Key requirements. Because the log is the source of truth for recovery, ARIES-style logging requires durable storage that survives process, system, and device failures [9]. Accordingly, the logging backend must provide durability and high availability. It must also support low-latency appends, since WAL lies on the critical path of transaction processing. Cloud database systems may also operate at extreme transaction rates (e.g., 70 M/s [10]), producing massive log volumes. This makes cost-efficiency, both per append and per GB, another important requirement.

Single-writer semantics. A key observation underlying our work is that many cloud and distributed databases achieve durability via per-partition (or per-replica) log streams. Each log stream has an exclusive writer at any given time. In primary/secondary architectures such as Amazon Aurora [11], Microsoft Azure SQL HyperScale [3], Alibaba PolarDB [10], and Google AlloyDB [12], only the writer instance issues updates and generates the corresponding log records. Similarly, in strongly consistent distributed database systems such as Google Spanner [13], each replicated partition (Paxos group) elects a leader. All writes are ordered through that leader, yielding a single total order of writes per group [13]. Even systems that expose multi-region write APIs (e.g., Cosmos DB [14]) implement durability via locally replicated physical partitions. Ordering guarantees are defined by a leader within the associated replication scopes [14]. As we discuss next, exploiting the monotonic, single-writer append order (via sequence numbers) is key to reducing WAL latency.

2.2 Log Service Architectures↩︎

Durability and ordering guarantees. General-purpose distributed log systems, and some distributed streaming systems (e.g., Apache Kafka), provide an append-only abstraction and use replication to achieve durability and availability. In contrast to the single-writer WAL streams common in cloud database systems, they are designed for multiple concurrent writers. They therefore include a sequencer that orders writes across clients, linearizing appends. The need to coordinate ordering and replication adds substantial latency on the append path, and architectures differ in how these responsibilities are implemented, as 1 visualizes.

Consensus-based systems (Paxos, Raft, VSR). As the left-hand side of 1 shows, leader-based consensus protocols [15]–[17] replicate a log across a set of nodes coordinated by a leader. The leader serializes appends and replicates them to a quorum of followers, combining ordering and replication on the write path. Each append requires four network hops (or two network round trips): client \(\to\) leader and leader \(\to\) follower. The leader is both a throughput bottleneck and a source of latency variance.

Throughput-optimized systems (Corfu, Scalog). Corfu [18], Scalog [19], and similar systems [20], [21] decouple sequencing from storage. This avoids the leader bottleneck by scaling the sequencer layer across nodes. However, these systems still use eager ordering [22]: clients receive an LSN for an append once both ordering and replication are complete (append(rec)\(\rightarrow\)LSN). As 1 shows, this results in additional network communication across layers. The replication strategy further impacts latency: Corfu’s chain replication requires six network hops; Scalog’s primary-backup strategy reduces it to four, which is still well above the single-round-trip latency achievable with single-writer WAL semantics.

Lazy ordering (LazyLog). LazyLog [22] reduces append latency by separating ordering from durability. Clients send metadata to sequencers while also sending data to storage nodes in parallel. Appends are acknowledged without an LSN (append(rec)\(\rightarrow\)bool) since ordering is resolved lazily during reads. This yields single round-trip appends, but leaving LSNs unspecified at append time is incompatible with ARIES-style WAL, which requires a well-defined log order (and commit LSNs) on the write path.

Single-writer systems (Apache BookKeeper). In contrast to multi-writer log systems, Apache BookKeeper [7] targets single-writer logs, as in primary/replica database architectures. On writer failure (e.g., a crash), safe failover requires additional mechanisms. BookKeeper therefore implements fencing and recovery to prevent split-brain scenarios during failover. With a single writer, the order of records is implicitly defined by the writer’s append sequence. Thus, sequencing shifts to the client, eliminating the need for dedicated ordering nodes and their associated latency overheads. BookKeeper uses client-driven quorum replication; in the common case, an append completes in a single client round trip.

3 BtrLog Design and Deployment↩︎

Design space. One possible solution for WAL in the cloud is writing log records directly to cheap and durable object storage. However, object storage writes take tens of milliseconds [23] and are expensive for workloads with many small writes – such as WAL. At the other extreme, one can envision a DRAM-based design similar to RAMCloud [24], where the client sends data to a quorum of nodes (“log nodes”) that keep data in DRAM and provide durability through replication. This approach provides low latency but is expensive due to high DRAM cost [25]. Furthermore, it is prone to correlated failures such as data center-wide outages or software bugs.

3.1 System Overview↩︎

BtrLog’s high-level design. BtrLog overcomes durability and cost tradeoffs by combining a staging layer for low-latency appends with object storage for durable, cost-efficient persistence. BtrLog stages appends on a cluster of log nodes and flushes them asynchronously to object storage in large segments. Log nodes retain only a bounded unflushed tail locally: the tail resides in RAM for fast reads and is backed by local NVMe SSDs for recovery after crashes or power loss. By flushing large segments asynchronously, BtrLog amortizes object store PUT cost and eliminates both object store latency and long-term storage capacity management. BtrLog thus consists of four components, visualized in 2:

1. a client library used by the DBMS to append log records,

2. a cluster of log nodes that form the staging layer,

3. a cloud object store for durable, low-cost storage, and

4. metadata storage for log metadata and cluster configuration.

BtrLog requires few metadata operations, which can be implemented using conditional writes (If-Match in S3). Using S3 or a compatible service for metadata simplifies deployment and avoids a separate coordination system such as ZooKeeper or etcd.

Log abstraction and API. BtrLog exposes a single-writer, append-only log abstraction. The client library hides the internal storage tiering and provides a WAL-compatible API: append, sync, scan, and read. Failover, metadata updates, and other control operations are handled transparently when the client opens a log for writing.

Write path. The append (write) path is BtrLog’s hot path, optimized for single network round-trip commits. As 2 illustrates, each client maintains a local LSN counter and replicates appends in parallel to all log nodes . Upon receiving an append, a node adds the record to an in-memory segment and writes it out-of-place [26] to a local NVMe SSD . Only after the SSD write and fsync complete, the node acknowledges the append to the client. The client library returns success to the DBMS after receiving acknowledgments from a quorum of nodes. If multiple appends are in flight, the client library ensures that appends are acknowledged in LSN order to preserve ARIES semantics. This client-driven sequencing eliminates a separate sequencer, enables single round-trip latency, and masks network and SSD latency jitter [27] via quorum replication.

Read path. BtrLog distinguishes between hot and cold reads. The hot, unflushed tail (e.g., for transaction rollback) is read from in-memory segments on log nodes . Cold entries are read from flushed segments on object storage  (e.g., for point-in-time recovery), leveraging its aggregate read bandwidth while reducing load on log nodes. The client routes requests transparently via read and scan based on the requested LSN range and the replicated commit watermark, and may cache recently read tail records.

3.2 Guarantees and Usage in Database Systems↩︎

Durability and correlated failures. BtrLog replicates each append to a quorum of \(Q_w\) log nodes before acknowledging it and thus tolerates up to \(Q_w - 1\) node failures without data loss. For example, with 3 log nodes, \(Q_w=2\), and the system tolerates 1 node failure. Log segments are flushed asynchronously to object storage for long-term durability . Single writer, multiple readers. BtrLog enforces a single-writer, multi-reader abstraction, supporting use cases like log shipping. The writer maintains a commit watermark that is replicated across log nodes, allowing readers to distinguish committed from uncommitted data. Enforcing a single writer prevents inconsistent appends during DBMS failover (e.g., when replacing a failed primary).

Failure handling in the cloud. Quorum replication masks transient node failures and network jitter, providing predictable tail latency for appends. The client-side failover protocol  prevents split-brain during DBMS failover. BtrLog also handles compound and AZ+1 failure scenarios, as discussed in 4.

Database integration. As 2 illustrates, the DBMS integrates the BtrLog client into its WAL subsystem rather than writing log records to a storage device. When the primary starts, or when explicitly requested, the client creates a new log on the log nodes and registers the log in the metadata management layer (c.f., 4), which establishes the primary as the log’s single writer. It is then free to append log records as described in 3.1.

Transaction rollback. The client may cache recently appended records in memory to enable low-latency reads for abort and rollback. In addition, log nodes keep the hot log tail in memory, which accelerates hot reads. The hot log tail is asynchronously flushed to object storage when it fills up or becomes cold.

Database recovery. The BtrLog client supports high-throughput reads of cold log data for database recovery. For example, when a database node must replay a large log prefix to reconstruct database state, the client transparently fetches the hot tail from log nodes and streams older segments from high-bandwidth object storage [28].

Database failover. For planned maintenance or primary failure, a backup node can take over as the writer. When a BtrLog client opens an existing log for writing, it acquires ownership and fences off the previous writer to prevent conflicting appends. 4 details the protocol that implements these mechanisms.

Ease of use. Integrating BtrLog into an existing DBMS that relies on file system storage requires minor source code changes due to interface differences (append vs. write). Note, however, that file-system-based logging necessitates protecting against torn writes and identifying the last successful log record write during recovery. BtrLog prevents both issues due to its record-level atomicity and failover support: its interface is purpose-built for database WAL.

Parallel logging. To avoid contention on a single log, some high-performance DBMSs support parallel logging through multiple (e.g., per-thread) log streams [29]–[32]. Such designs already manage transaction order across multiple log streams to ensure correct recovery and can thus simply use multiple BtrLog logs.

3.3 Deployment Options↩︎

Configurable availability. BtrLog supports two deployment options to adapt to different latency and durability/availability requirements: single-AZ and multi-AZ. Although we use AWS terminology, the same concepts apply to other cloud providers as well.

Single-AZ deployment. To minimize append latency, BtrLog can be deployed in the same availability zone as the client application. An AZ consists of one or more data centers within an AWS region, which are isolated from failures such as power, network outages, or flooding [33]. Services such as EBS and S3 Express operate within a single AZ. Intra-AZ network communication has latency of about 100 μs and is free of charge. BtrLog log nodes reside in partitioned placement groups [34] to reduce correlated failures like rack-level power or network faults. If the entire AZ becomes unavailable (e.g., due to a major power outage), all log nodes in that AZ and BtrLog become unavailable as well. Because BtrLog writes log records to instance-local SSDs, which persist data across power-loss reboots [1], data is still durable and can be recovered upon restart. Records that were already flushed to object store remain accessible during the outage, since S3 replicates data across multiple AZs.

Multi-AZ deployment. To improve availability, BtrLog also supports deployments that span multiple AZs. A straightforward approach would place three log nodes across three AZs, ensuring that AZ failures affect only one replica. However, an additional independent failure (e.g., an SSD failure in a second AZ) can reduce a three-node deployment below the required 2/3 quorum, making the log unavailable for reads. To tolerate such AZ\(+1\) failures [11], BtrLog uses six nodes (two per AZ). This configuration remains write-available after losing any two nodes or an entire AZ (4/6 append quorum), and remains read-available after losing up to three nodes (3/6 read quorum). Even with six nodes, BtrLog persists records to local SSDs to guard against correlated software failures.

Improving cost-efficiency in multi-AZ deployments. Besides increasing latency, multi-AZ deployments also increase cost due to cross-AZ data transfer. For example, replicating 100 million 1 KB log records from one AZ to another AZ of the same AWS region transfers 100 GB and costs $2. The overhead is amplified by quorum replication: if a client sends each append to all four remote nodes, it pays the transfer charge per remote replica (e.g., \(4 \cdot \$2 = \$8\)). BtrLog reduces this overhead by 2 via hierarchical replication: the client sends each record to one node per AZ, which forwards it to its peer in the same AZ. This significantly reduces cost but only slightly increases latency because intra-AZ latency (about 100 μs) is significantly lower than cross-AZ latency (about 500 μs).

4 Protocols and Fault Tolerance↩︎

Our design is simple: quorum-replicated appends to a staging layer, with asynchronous, segment-based persistence to object storage. The challenge is to preserve BtrLog’s guarantees under realistic cloud failure modes – transient faults, correlated node/AZ outages, and DBMS failovers – without sacrificing one-round-trip commits. To this end, we employ a leader-based quorum protocol with view changes, similar to Apache BookKeeper, customized for BtrLog’s object storage-oriented persistence path. The protocol is grounded in two invariants: (i) log records commit in order, and (ii) no committed entries are lost, even in the presence of failures within our failure model. We formalize these safety properties and an abstract model of the protocol in TLA+ and model-check them (see artifact URL). The remainder of this section gives a high-level overview of the protocol design.

Failure model. BtrLog adopts the standard non-Byzantine crash-stop failure model in asynchronous networks [35]. Accordingly, our protocol tolerates message loss, reordering, duplication, delay, network partitions, log node failures, and client failures. In the following, we discuss client and log node failures separately. Client failures coincide with database failover. Log node failures must not compromise the availability or durability of committed data in the log tail. Because cloud object storage provides high durability and availability (e.g., S3 advertises 99.99% availability and 11 nines of durability [36]), we treat object storage and the metadata store as reliable and focus on failures in the staging and replication layers.

Protocol overview. BtrLog targets the same failure modes as Paxos-like protocols, but shifts responsibilities to reduce latency while preserving fault tolerance. It assigns the leader/proposer role to the single writer (the client) to minimize commit latency, and leverages WAL monotonicity to avoid per-record ordering at log nodes. BtrLog integrates object storage to simplify node recovery and avoid re-replication across log nodes. Cloud object storage also serves as a highly available metadata store with atomic compare-and-swap (CAS) semantics for leader election (client failover). In the following, we discuss how BtrLog uses these concepts to achieve fault tolerance. We assume a single-AZ deployment for simplicity, but our explanations generalize to BtrLog’s multi-AZ deployment.

4.1 Failure-Free Operation↩︎

Log creation. Before appending, clients have to create a new log by initializing its metadata (such as the segment size (bSize)) in the metadata store, as shown in 3 . To open a log for writing, the client acquires a write token (wtoken) from the metadata store (c.f. 4.2) and sends an open request to all log nodes to install that token . Before issuing append requests, the client waits for acknowledgments from a quorum of log nodes.

Append operations. The BtrLog client sends append requests to all log nodes and commits after replies from a write quorum . In addition to the payload, each append request includes three fields: (1) the record’s log sequence number (nLSN), (2) the last committed LSN (cLSN, a commit watermark for readers), and (3) the record’s byte offset within the current segment, computed as the prefix sum of payload sizes of preceding records in the segment. The prefix sum allows log nodes to place each record at a deterministic offset within a segment, enabling optimized flushing to object storage.

Idempotent segment flushes. On the failure-free fast path, log nodes flush full segments to object storage asynchronously . If all nodes wrote the same segment, this would incur 3 the PUT cost in a 3-node deployment. To avoid redundant PUTs, BtrLog uses the prefix sum of record sizes to construct byte-by-byte identical segments across all nodes. As a result, each node derives the same deterministic object name for the segment, making the flush idempotent. Nodes create the object using a conditional PUT (with If-None-Match in S3), so at most one concurrent uploader succeeds. To optimize costs, a node checks whether the object already exists before issuing a PUT and skips the write if it does.

4.2 Client Failover↩︎

Given the client’s leader role, client failures require explicit failover handling. The key challenge is to preserve the single-writer invariant and prevent split-brain, in which two clients concurrently believe they hold leadership for the same log. Without fencing, concurrent append operations can create divergent tails across log nodes and conflicting persisted segments in object storage.

Running example. We use the example depicted in 4 to show how BtrLog prevents split-brain scenarios and enforces single-writer semantics through its failover protocol. In this example, client C1 is isolated in a network partition and retains connectivity only to log node L1. To replace the faulty client, a new client, C2, opens the existing log as the writer.

Writer fencing. Concurrent writes to the same log violate BtrLog’s single-writer invariant. We therefore use a monotonically increasing write token (wtoken) to implement writer fencing. In 4, client C2 atomically increments the log’s write token from 1 to 2 via an atomic metadata operation. It then installs the token on log nodes by sending open and waiting for acknowledgments from a write quorum . A log node accepts requests only for the highest token it has observed; if it has already seen a higher token, it rejects open and returns the higher value. If the client cannot obtain a quorum of acknowledgments, it aborts the takeover and reports failure to the application (which may retry).

Avoiding data loss. Before taking over, a new writer must establish a correct log tail to preserve monotonicity and avoid losing committed data. 4 highlights a subtlety introduced by object storage: Even if client C2 cannot reach log node L1, it may still observe segments that node L1 has already flushed to object storage. Naively consulting object storage during failover would add substantial latency (tens of milliseconds per access), so the protocol avoids putting object storage reads on the critical path.

Finding the log tail. Fortunately, failover does not require using object storage. To locate the tail of the log without losing committed data, the new writer determines the largest contiguous recoverable LSN prefix supported by a read quorum of log nodes. To accelerate this step, log nodes piggyback their last committed LSN watermark when acknowledging the installation of the client’s write token. The client then starts reading log records sequentially , starting from the highest watermark it observed (cLSN = 2 in the example).

Determining committed data. The retrieved log record’s LSN might be below the last committed watermark observed by a node, and only one copy of the record might exist. A stricter setting that requires the writer to observe a quorum of copies might cause data loss, as illustrated in 4. In the example, C2 observes only one copy of LSN 3 even though the previous writer replicated it to 2/3 nodes and reported success to the application (cLSN = 3 on C1). Such cases occur because quorum replication can introduce gaps, e.g., due to message loss, and the last committed watermark can lag, especially when multiple appends are in flight. To avoid data loss, new writers are conservative when inferring the log tail: they include every consecutive LSN present on at least one node in the quorum. Only LSNs that a quorum of log nodes confirms as absent could not have been committed. In the example, although LSN 4 is durably stored in object storage, C2 can infer it was not committed because a read quorum confirms it absent.

Log repair and appending. Under-replicated records (e.g., LSNs 2 and 3 in 4) are more susceptible to data loss during network partitions. For example, the only accessible copy of LSN 2 is lost when node L3 fails. To improve durability of under-replicated records, the new client re-replicates such records to all nodes using the new write token . These repair writes may fail if the new client is also fenced, in which case it stops the failover process and informs the application to handle retries. Writer fencing can happen at any stage, but the protocol guarantees that concurrent repairs will recover the same or a higher log tail. Once fencing and repair are complete, the new client continues appending to the same log.

Reconstructing the client’s log state. As described earlier, the failover protocol determines all required metadata for the new writer’s state, including the last committed LSN and the next LSN (nLSN). During tail recovery, the writer also retrieves each record’s byte offset from log nodes. This information is used to reconstruct the segment prefix sum for idempotent object storage flushes.

4.3 Node Failures↩︎

Detecting node failures. Although BtrLog uses a client-driven replication protocol, it does not depend on clients to detect or handle node failures. If failure detection were delegated to clients, node outages could go unnoticed when a client idles or crashes, leaving the tail unreplicated and increasing the risk of data loss. Instead, BtrLog detects node failures via peer-to-peer heartbeats between log nodes. If a node misses heartbeats from a peer for a configurable interval, it marks that peer as failed.

Handling node failures. Traditional protocols, like VSR [15], Raft [16], Corfu [37], or Scalog [19], use re-replication from surviving nodes to recover and rebuild failed nodes. Healthy nodes serve read requests to recovering nodes, increasing load. To avoid this overhead, BtrLog flushes a snapshot of the log segments to object storage rather than re-replicating the data to other nodes. The snapshot flush persists all LSNs up to the point at which the failure was detected. Entries committed after detection are still replicated to a quorum by design. Besides reducing load on healthy nodes, this strategy also enables new or recovering nodes to accept new writes immediately after receiving the current write token.

Cost-efficient failure handling. Flushing log segment snapshots to object storage on every node failure might, however, increase costs due to the number of PUT requests. To reduce the flush costs, BtrLog introduces two optimizations: First, log nodes trigger a flush only when they detect \(N-Q_w\) node failures. For example, in a cluster with 3 nodes and \(Q_w=2\), a single failure would already trigger a flush. In a cluster of 6 nodes and \(Q_w=4\), snapshot flushes are only triggered after two failures. Second, we deterministically assign a per-log flush leader: the leader flushes immediately, while other nodes first check whether the object already exists before writing. This optimization avoids unnecessary flushes.

Node replacement and reconfiguration. BtrLog does not automatically redeploy failed nodes and assumes that a higher-level monitoring component handles such logic. Currently, faulty nodes can be replaced by removing a node and replacing it with a new node, reusing the same DNS name or IP address. More automated node replacement strategies and cluster resizing through reconfiguration are interesting avenues of future work.

4.4 Unreliable Networks↩︎

Impact of network issues. BtrLog assumes unreliable networks where append requests may get reordered or dropped entirely. Network issues can introduce gaps in node-local segments and delay the propagation of the committed watermark from the client to the log nodes. Nodes may also flush incomplete or overlapping segments to object storage, potentially leading to uncommitted or duplicated records stored there. For instance, a node may flush while client failover is in progress. 4 illustrates how this failure case can cause data to appear on object storage even without being replicated and committed by quorum.

Handling uncommitted data. To keep the write path fast, BtrLog postpones handling such cases to the read path. A key invariant for the read path is that data in object storage is only durable, not necessarily committed. Readers identify committed data through the last committed watermark (cLSN) that the client propagates to log nodes, which in turn include it as object metadata on each flush. For example, when clients retrieve data with the read API, nodes return both the requested data and the committed LSN they observed. This mechanism enables readers to distinguish committed data from merely durable data. Similarly, when retrieving data from object storage, clients evaluate the committed watermark in the object’s metadata to filter out uncommitted data if required.

Handling overlapping data. Due to quorum replication, log segments may have gaps and overlapping ranges. For instance, some nodes might observe LSN1 and LSN3, while others receive LSN2 and LSN4. When such overlapping log segments are flushed to object storage, e.g., due to a node failure, clients may observe overlapping copies of those segments. The BtrLog client transparently merges log segments while reading from object storage.

Handling duplicate data. A more subtle issue can occur during client failover. As 4 illustrates, pending requests from an old writer (client C1) can fill up log segments and trigger a flush to object storage. When a new writer (client C2) takes over the log, it might append new data with the same LSN, for instance, LSN4 in 4. Once the new log segment is flushed to object storage, readers would observe multiple different copies of LSN4. To resolve such conflicts in object storage, BtrLog utilizes the write token as an epoch. Each flushed object encodes the epoch in its key and includes it in its metadata. Similar to the last committed watermark, BtrLog clients use this information to ignore outdated data from prior epochs while streaming from object storage.

5 Engineering BtrLog for Low Latency↩︎

Implementation overview. Achieving high throughput and low latency requires efficient protocols and an optimized implementation that can effectively utilize modern networks, NVMe SSDs, and multi-core CPUs [38]. Our prototype implements the latency-critical, failure-free append path; failure cases are modeled in TLA+.

Lower latency bound. Microbenchmarks using sockperf [39] and fio [40] in AWS show that a cluster of c6id.metal instances can achieve approximately 40 μs network round-trip and 30 μs SSD write latency without load in a partitioned placement group. Our implementation thus targets 70 μs latency without load. Achieving this requires careful optimization of cache misses, allocations, and system calls: For example, system calls and large allocations can take hundreds of nanoseconds or even microseconds, potentially deteriorating response latency by several percent. Thus, our technology stack needs to allow for fine-grained control over such operations.

Technology stack: Rust, io_uring, UDP. BtrLog is implemented in Rust. Since no tested off-the-shelf asynchronous Rust runtime offered consistently low latency and observability, we implemented our own. Our custom asynchronous runtime is built on io_uring [41], which allows batching system calls for common operations such as send, recv, and write [42]. Finally, since the quorum-based BtrLog protocol obviates the need for many of TCP’s guarantees, we use UDP by default, avoiding stream and connection overhead.

Multi-Core architecture. BtrLog is designed as a multi-tenant cloud service, so it expects to manage many independent open log streams simultaneously. Conversely, each log requires sequential semantics internally, and the most common append operation needs few CPU cycles: As 5 illustrates, each thread checks the message and copies it to the in-memory log segment if valid, asynchronously writes it to the local SSD, ensuring fsync semantics, replies to the client once data is durable, and then asynchronously flushes full segments to S3 if necessary. This model naturally yields an architecture where each log is owned by a single thread that sequentially processes requests for that log.

Per-Thread runtime. I/O dominates the end-to-end latency of requests, so I/O wait times should be utilized to process other requests. One could achieve this by using synchronous I/O requests and letting the OS schedule other threads. However, the resulting thread-to-core oversubscription and excessive OS scheduling can lead to latency spikes. BtrLog thus spawns one thread per CPU core and uses cooperative scheduling within a custom asynchronous Rust runtime to interleave I/O and CPU work on each thread.

Symmetric networking. Many systems use dedicated networking threads to simplify the architecture and guarantee fast packet acceptance. However, this asymmetric approach requires cross-thread synchronization for every packet and is detrimental on small nodes with few CPUs, since one CPU thread is designated to networking only. In BtrLog, every thread accepts network requests on its own port using the io_uring-based runtime and processes them directly.

Internal load balancing. The symmetric networking approach has a significant drawback: If clients send requests directly to individual threads on BtrLog nodes, the node cannot balance log segments across threads. Load balancing is important since we expect multi-tenant workloads to serve tenants with both very high and very low append frequency. Having multiple high-frequency logs handled by the same thread would negatively impact peak throughput. To enable load balancing while still allowing threads to respond to clients directly and reduce cross-thread synchronization, our network protocol includes a reply_to field that specifies a port number. Clients can use any previously used port, including a well-known port for initial requests, and the BtrLog node may internally forward the message to a different thread, handing over the connection. The receiving thread responds to the client directly, setting its own port in the reply_to field, which the client uses for future requests – until the log moves to a different thread again. 5 illustrates the resulting architecture.

S3 integration. BtrLog uses AWS’s Rust S3 SDK to flush full log segments to S3. As 4 describes, BtrLog reduces network utilization and costs when flushing to S3: Log segments flushed to S3 by some node will not be flushed again by another. The Rust S3 SDK integrates with Rust’s asynchronous ecosystem but exhibits long-running blocking operations (single-digit milliseconds) that deteriorate latency when run on regular log node worker threads. The implementation thus executes S3 operations on dedicated threads.

6 Evaluation↩︎

Outline. After detailing the experimental setup, this section evaluates BtrLog across a number of dimensions, including append latency (Sec. 6.1), single-AZ and multi-AZ deployment (Sec. 6.3), impact of node failure and quorum (Sec. 6.2), cost and availability (Sec. [sec:sec:eval-cost]), and end-to-end OLTP performance (Sec. 6.4).

Comparison systems. As discussed, Apache BookKeeper is the only alternative log system that eagerly assigns LSNs and executes appends with a single network round trip. Since other log systems have higher latency by design, our comparison focuses on Apache BookKeeper. We also compare BtrLog with Amazon EBS [43] in its low-latency, high-durability variant io2, which is a common choice for attaining durable writes on ephemeral cloud instances [44].

Experimental setup. We evaluate all systems on a cluster of three c6id.metal log nodes with local SSDs in the AWS eu-central-1 region, plus a c6in.metal client node, which has enough network bandwidth to serve all log nodes simultaneously. The cluster uses “partitioned” placement groups, as recommended by AWS [45] for high-availability services. All measurements for an experiment are executed on the same cluster to ensure comparability. All nodes run Linux 6.14, and BtrLog uses instance-local SSDs as block devices.

BookKeeper setup. For BookKeeper, we measured both the default configuration and an optimized configuration for low-latency writes on SSDs. The default BookKeeper configuration buffers appends for up to 2 ms before group-flushing them to SSD. This buffering time is a fraction of typical HDD latency, but 60 times the write latency of a modern SSD. Our SSD-optimized BookKeeper configuration removes buffering entirely, aligns writes to 4  , and disables the page cache. We combine SSDs in a RAID0 using the XFS file system, which has little overhead [46], [47].

Amazon EBS. EBS experiments directly attach an io2 volume to the client node to achieve IOPS comparable with the BtrLog cluster. This volume is used as a block device, without a file system, similar to how BtrLog log nodes write to their local SSD.

6.1 Append Latency and Throughput↩︎

Open-loop latency under load. Response latency depends on both system configuration and load. For example, systems may batch disk writes from multiple requests to improve peak throughput at the cost of higher latency. Therefore, we vary the system load (appends per second), and record the end-to-end latency of each append. For comparability, we configure all systems to achieve the best possible latency by deactivating write batching. Our open-loop benchmark schedules requests with exponentially distributed inter-arrival times per log stream, i.e., as a Poisson point process. The client node adds more independent log streams over time,

Latency with increasing load. 6 shows the median and 99th-percentile response latency for BookKeeper, EBS io2, and BtrLog with 128-byte appends, Each point corresponds to a 500 ms interval over which throughput and latency percentiles are measured. BookKeeper does not scale beyond 240,000 appends per second in either configuration (, ), and its throughput and latency fluctuate strongly under load, producing the chaotic pattern in 6. The default BookKeeper configuration is not visible in the figure because its latency never falls below 2 ms. EBS io2 () shows steadily increasing write latency as load rises and consistently exhibits 4–5 higher median latency than the BtrLog prototype. At approximately 1 M appends/s, which is the IOPS limit of the instance-local SSD, both EBS and BtrLog latency deteriorate.

Best-case latency. Let us now discuss interesting latency and system load results for each system using the data points shown in 6. Across a large set of BookKeeper configurations, the best median latency we observed was 262 μs at 6,800 appends per second; latency deteriorates quickly as load increases. The best median latency on EBS was 318 μs, which increased steadily with higher load. BtrLog’s best median latency is 70 μs (79 μs p99 latency) at a load of 35,500 appends per second. At half of the maximum system load, approximately 500 k appends per second, BtrLog achieves a median latency of 111 μs.

Full system load. At the maximum load of 1 M appends/s (128-byte), EBS reaches 503 μs median latency and 651 μs p99 latency. Once the provisioned-IOPS volume is saturated, EBS latency increases beyond the range shown in 6. BtrLog reaches 188 μs median latency at full load, limited by the 1.1 M IOPS of the underlying SSDs. To enable overload detection and to bound the latency of admitted requests, BtrLog limits its internal queue lengths and drops excess requests once the system is saturated [48]. As a result, it does not exhibit the typical “hockey stick” latency curve near full load.

Impact of SSD latency. Applications with weaker durability requirements may run BtrLog with SSD writes disabled on log nodes: Each log node appends to log segments in memory and flushes them to S3 when full, but does not write its own WAL to SSD. This setting improves latency, since SSD write latency is similar to the network round-trip time in a single availability zone (35 μs vs 40 μs). We find that disabling SSDs reduces median latency by approximately 50% and improves tail latency across most throughput settings.

6.2 Impact of Node Failure and Quorum↩︎

Impact of node failure. Our experimental setup with three log nodes can tolerate one node failure, since a quorum of two nodes suffices for appends. The following experiment tests the append availability and latency of the BtrLog prototype under node failure by manually stopping a random log node while the benchmark is running. The load is kept constant at approximately 400,000 appends per second. As 7 shows, the client keeps successfully appending log entries on the BtrLog cluster after a node is killed at \(t = 64s\). The median latency deteriorates from 95 μs to 115 μs (21%), and the p99 latency from 192 μs to 221 μs (15%).

Discussion: quorum latency. BtrLog’s quorum-based algorithm does not need to wait for all responses to arrive, thereby hedging against network latency jitter. Killing one node effectively stops request hedging and causes the latency increase visible in 7.

Impact of quorum size. BtrLog can be configured with different quorum sizes. The default number of responses required for a successful append operation is a majority, e.g., two out of three nodes. In the following experiment, we investigate the effect of disabling quorum for append requests, so that they require responses from all nodes. As 8 shows, the quorum-based protocol significantly improves end-to-end latency: At half system load (500,000 appends/s), the quorum improves median latency by approximately 30 μs and p99 latency by approximately 50 μs.

6.3 Cross-AZ Latency↩︎

Network latency dominates. Previous experiments used a log cluster inside a single availability zone (AZ). This yields better latency and availability comparable to EBS, but may not provide enough availability for some use cases. To validate that BtrLog, like in a single AZ, achieves latency close to the hardware limit, we now spread the three-node cluster across multiple AZs of the same region (eu-central-1). In this configuration, the client sees idle-load median network latency of 270 μs, 440 μs, and 540 μs to the three log nodes as measured using sockperf [39]. Note that these numbers may vary by region. As the following figure shows, the overall append request latency in BtrLog is indeed dominated by cross-AZ network latency (400,000 appends per second):

As the previous single-AZ experiments showed, BtrLog achieves 4 better latency than EBS io2. In a multi-AZ configuration with three nodes, BtrLog achieves comparable latency (12% worse) to a single-AZ provisioned EBS volume at the same throughput while providing multi-AZ availability and a WAL-optimized interface.

6.4 End-to-End Database Performance↩︎

Integration in LeanStore. Experiments so far have used a custom open-loop benchmarking client designed to measure response latency as accurately as possible under different load settings. We now turn towards evaluating the performance of these systems in a transactional database, LeanStore [8]. LeanStore’s autonomous commit protocol [32] uses multiple log streams (one per worker thread) that are flushed independently. This design was conceived for local SSDs, which require many parallel writes to saturate bandwidth, but it also fits the BookKeeper and BtrLog APIs, since their multi-tenant architecture supports multiple independent log streams. We modify LeanStore by swapping its WAL implementation with different backends: Custom-built backends for BookKeeper and BtrLog, and the default block-device backend for EBS.

Transaction Throughput. 9 shows the measured YCSB-A (50% reads, 50% writes) throughput using different WAL backends in LeanStore. LeanStore using the BtrLog WAL backend achieves 2 higher throughput than LeanStore using the BookKeeper backend, 3 higher throughput than EBS gp3, and 1.25 higher throughput than EBS io2. These results show that systems previously using EBS io2 for their WAL can replace EBS with a non-provisioned, multi-tenant service while simultaneously increasing transaction throughput using a simple append interface.

7 Related Work↩︎

In addition to prior discussions, we review related work on cloud-native databases, shared-log designs, and cloud storage backends.

Disaggregated and cloud databases. Decoupling and optimizing logging functionality is a common pattern in modern cloud database systems. Amazon Aurora [11] uses a log-centric design with a highly coupled log and storage component to reduce network traffic and improve fault tolerance. Microsoft Socrates [3] implements a dedicated XLOG service with quorum-based commit, and Huawei TaurusDB [4] uses distinct log nodes (PLog) to reduce commit latency. OceanBase PALF [49] and FoundationDB [50] similarly utilize a dedicated log component in their design. Neon separates PostgreSQL compute from a replicated WAL service (“safekeepers”) that provides durable WAL ingestion and supports failover [51]. Many of these systems, e.g., Aurora, Socrates, OceanBase, and Neon, also integrate object storage for low-cost storage. Although these systems highlight the need for a specialized logging component, their components are tightly coupled with the database engine and are not described sufficiently in the literature. In contrast, BtrLog aims to provide a reusable write-ahead logging system that can be integrated with database engines, supporting the development of modular and composable cloud database architectures [52], [53].

Shared logs and distributed log systems. Besides industrial systems, log abstractions have also been studied in academic research. Hyder proposed using a single totally ordered log on shared flash as the backend for OLTP [54], motivating systems such as Corfu [55]. However, shared-log systems including Corfu, Delos, and Scalog emphasize multi-writer semantics and throughput rather than low latency [19], [20], [55]. Milliscale [56] recently proposed using S3 Express for OLTP to achieve scalable multi-millisecond latency, while BtrLog targets scalable multi-microsecond latency. LazyLog [22] and SpecLog [57] explore lazy or speculative binding of records to log positions to reduce append latency, and FuzzyLog [58] relaxes ordering to partial orders to increase concurrency. These works, however, introduce new append semantics in which LSNs are unknown at commit time, making them incompatible with common recovery protocols such as ARIES.

Log-centric streaming systems. There has also been work on distributed log systems for streaming. LogDevice is a deprecated distributed log system developed by Facebook that was designed to support generic record-oriented and append-only use cases such as write-ahead logging for durability and stream processing [59]. Kafka [60] popularized the replicated commit-log model for high-throughput streaming and introduced tiered storage [61] to archive old log segments to object storage. Streaming-oriented systems such as Apache Pulsar [62] and Pravega [63], built on top of Apache BookKeeper [7] as a backend for durability, also provide data offloading to object storage. BtrLog complements these works by optimizing for databases instead of streaming use cases: it targets the single-writer, low-latency append pattern of database WAL and leverages ARIES-style semantics to minimize commit overhead, while still supporting asynchronous archival to object storage. Still, the prevalence of the log abstraction in other systems motivates us to explore using BtrLog for other latency-sensitive use cases.

Low-latency durable commits. Related work has studied how fast networks and memory can reduce commit latency. FaRM [64] and RAMCloud [24] replicate logs to remote DRAM to make commits durable with microsecond-scale overhead. QueryFresh [65] uses an RDMA-accessible NVRAM log as primary storage to enable fresh reads on standbys. In contrast to these works, BtrLog aims to achieve low-latency durability on commodity cloud infrastructure.

Using cloud storage for databases. Cloud storage characteristics shape database design beyond WAL. Durner et al. study how to use object storage for analytics despite higher latency and different access patterns [28]. Other benchmarking efforts characterize cloud storage services and database I/O behavior in cloud-native systems [66]. Early work evaluated alternative database architectures on cloud primitives such as EC2/EBS/S3 and quantified their performance implications [67]. More recent studies characterize cloud database system architecture tradeoffs [68], [69] and the implications of object vs.block storage latency/variance [70]. Instead of adapting database systems to different storage backends, BtrLog provides a specialized WAL storage backend that requires minimal database adaptation and transparently tiers storage for cold data.

8 Summary & Future Work↩︎

Summary. We present BtrLog, a reusable single-writer logging service purpose-built for cloud-native database systems. It combines low-latency durable appends with low-cost, highly durable archival on cloud object storage. Our evaluation shows that BtrLog improves latency by up to 4 relative to the commonly used EBS io2, translating to higher end-to-end transaction throughput in OLTP systems. Unlike EBS, BtrLog does not require provisioned IOPS or capacity when run as a multi-tenant service.

Beyond database systems. Logging is a primitive with a multitude of applications, and although BtrLog is designed as a WAL backend for database systems, we believe its low latency and cost can also be advantageous in other use cases. Like BookKeeper, BtrLog can be used as a storage backend for streaming systems such as Apache Pulsar. Using an additional ordering layer, it can also support multi-writer semantics, similar to Corfu and Scalog.

Towards extensibility. BtrLog provides a reusable interface and is not coupled to a specific database engine. One can conceive of further BtrLog extension points in an open ecosystem. For example, custom log segment filtering and transformation functions on log nodes enable writing to S3 in application-specific formats [71], or verification metadata [72] for long-term storage. This can be achieved securely using WebAssembly functions, which have recently been used for future-proof storage formats [73], [74]. Custom archival backends, such as a page materialization service, enable optimizations for cloud-native OLTP systems, such as Socrates [3].

References↩︎

---

1. This work is licensed under the Creative Commons BY-NC-ND 4.0 International License. Visit https://creativecommons.org/licenses/by-nc-nd/4.0/ to view a copy of this license. For any use beyond those covered by this license, obtain permission by emailing info@vldb.org. Copyright is held by the owner/author(s). Publication rights licensed to the VLDB Endowment.
   

   Proceedings of the VLDB Endowment, Vol. 19, No. 10ISSN 2150-8097.
   doi:XX.XX/XXX.XX
   ↩︎