1 Introduction↩︎

Video over the Internet has effected societal and economic transformations. A technology with potentially similar transformative power, 3D video, is on the horizon. In 3D video, each frame depicts a representation of a scene in three dimensions, and viewers can view the scene from any perspective. This technology will enable entirely new ways of viewing sporting events, and new modes of instruction delivery, as well as new visual story-telling paradigms.

1.1 Gaussian Splatting↩︎

At the core of any 3D video technology is a technique to represent a 3-D scene. While many representations exist, 3-D Gaussian Splats [1] (or 3DGS) have been the subject of recent research interest. 3DGS represents a scene as a collection of 3-D Gaussians. Each Gaussian represents a small, ellipsoidal volume, and is defined by its position in 3-D and a set of attributes. The Gaussians and their attributes are learned by training on a set of camera images capturing a scene from multiple perspectives. For this reason, we refer to the set of Gaussians and their attributes as a 3DGS model.

More precisely, a 3DGS model is a collection of \(N\) 3D Gaussians, each with the following attributes:

• Mean (\(\in \mathbb{R}^3\)) represents the position of the Gaussian in 3D space (i.e., the center of the ellipsoid).

• Scales (\(\in \mathbb{R}^3\)) define the size of the ellipsoid along its three axes.

• Rotation (\(\in \mathbb{R}^4\)) defines the orientation of the ellipsoid in 3D space, typically represented as a quaternion.

• Opacity (\(\in \mathbb{R}\)) represents the transparency of the Gaussian.

• Colors: In 3DGS, spherical harmonics (SH), which are functions defined on a surface, represent view-dependent color, capturing realistic lighting effects, material properties, and reflections. Each SH function is represented by three coefficients corresponding to three colors (RGB). The coefficients of a degree \(d = 0\) SH—the DC coefficients—represent the overall color of the Gaussian. Higher degree (\(d > 0\)) AC coefficients capture more complex lighting and shading effects. A common choice of SH degree is 3, which results in \(48\) SH coefficients per Gaussian.

Given a 3DGS model and a viewpoint, a rendering algorithm [1] produces an image of the view of the scene from the given perspective. While training a 3DGS model can take minutes, rendering only requires milliseconds.

Dynamic 3DGS. A sequence of 3DGS frames constitutes a dynamic 3DGS video, often referred to as 4DGS (time being the fourth dimension). In this paper, we focus on delivery of 4DGS content. This delivery is challenging, because each 3DGS frame can be large. Each Gaussian requires about 236 bytes to represent its attributes. 3DGS frames with a million Gaussians are not uncommon, so a single frame can be 236 MB in size. By contrast, a single frame of 4K UHD video requires 24-30 MB, an order of magnitude less.

1.2 4DGS Compression↩︎

To reduce the size of 4DGS, recent work has explored compression techniques. In addition to reducing size, compression can be used to encode 4DGS at different bitrate levels for streaming, as for 2D video. Two complementary approaches have emerged (6 discusses related work in detail).

During-training compression techniques reduce information in a variety of ways. QUEEN [2], 4DGC-Pro [3], and CompGS++ [4] reduce the number of Gaussians by using temporal prediction between key frames and their immediately following frames. A frame following a key frame is represented using residuals—differences with respect to the key frame. Other approaches exploit human tolerance for small color distortions. Vega [5] compresses SH (color) coefficients aggressively by first grouping semantically-related Gaussians, then training an MLP for each Gaussian group. QUEEN [2] and 4DGC-Pro [3] also train a quantizer that can learn optimal quantization levels for SH coefficients. Quantization reduces the number of bits required to represent SH coefficients at the expense of information loss.

Post-training compression techniques take a sequence of 3DGS frames as input and encode Gaussian attributes using 2-D or 3-D video codecs. For example, V³ [6] encodes Gaussian attributes in images and compresses the image sequence using a H.264 codec. MesonGS [7] and LTS [8] extend point-cloud compression techniques, such as G-PCC [9] or Draco [10], to compress 4DGS. Post-training compression has two advantages: (a) it can be faster than during-training compression ([2] reports about a 20% increase in training time with a during-training quantizer), and (b) it can help encode 4DGS videos at different bitrates without re-training.

1.3 Approach, Challenges, and Contributions↩︎

Goal. We present GS-NFS (or GS-NFS)¹, a post-training compression method for 4DGS which can encode and decode at full frame-rate, 30 frames per second (fps). It is also mobile-friendly; on a mobile GPU, it can decode some 4DGS sequences at 25 fps. No prior work on post-training compression has achieved full frame rate encode and decode.

Full frame-rate decoding is obviously useful; without that, it will not be possible to play 4DGS videos at frame rate. We argue that full frame-rate encoding is useful for three important reasons. First, for on-demand streaming, it will be necessary to encode 4DGS videos at different bitrates. At scale, an efficient encoder can reduce dollar costs significantly for encoding in the cloud. Second, today, 2D video providers use per-title encode optimization [11]. This approach seeks to find the best quality to encode a video, by repeatedly encoding and decoding frames at different qualities. A fast encoder/decoder like GS-NFS’s can enable similar optimizations for 4DGS. Third, recent work in feed-forward 3DGS networks [12] can generate Gaussian frames within tens of milliseconds. This can generate Gaussian frames at nearly full frame rate, and GS-NFS can help encode and transmit these frames in real-time.

Approach. To achieve this goal, GS-NFS uses a compression approach similar to a point-cloud compression technique, G-PCC [9]. This is a more natural starting point than the 2D compression used in V³ [6], since a 3DGS frame’s representation resembles that of a point-cloud. The latter contains a set of points, each with one or more attributes; 3DGS is similar, except that each Gaussian has more attributes (1.1). Next, we describe the G-PCC pipeline for encoding 3DGS frames; MesonGS [7] has used such an approach.

G-PCC [9]’s encoder takes a set of positions and associated attributes as input, then serializes (produces a bitstream) this set for transmission. Its decoder reconstructs the positions and attributes from the bitstream. To do this, G-PCC (a) encodes positions and associated attributes in such a manner that they can be decoded efficiently, and (b) compresses these encodings (either with or without information loss) to ensure bandwidth and storage efficiency.

G-PCC encodes position and attributes using qualitatively different approaches. It represents positions of Gaussians using a spatial data structure, an octree. Usually, 3DGS Gaussians are spatially sparse, so not all parts of the octree are occupied. G-PCC serializes the octree such that the bitstream only contains information for occupied octree parts.

For attributes, G-PCC provides several alternatives. We use Region-Adaptive Hierarchical Transform (RAHT [13], [14]), which organizes the Gaussians into a hierarchy of levels and predicts attributes at finer levels from attributes at coarser levels. The output of RAHT is a set of coefficients, which can be quantized (this introduces loss) and then entropy-coded (a lossless step that removes redundancy).

G-PCC’s decode pipeline inverts these operations, de-serializing Gaussians from the encoded bitstream. Unfortunately, G-PCC and MesonGS have high encode and decode times (5.2). GS-NFS achieves frame-rate encode/decode by running the entire encoding and decoding pipeline on a GPU. Beyond accelerating encoding, executing the encoder on a GPU is efficient when the encoder takes input from a training method that emits 3DGS frames. These frames are already in GPU memory, since 3DGS training requires a GPU, thereby avoiding memory copy costs.

Challenges and Contributions. Unfortunately, accelerating G-PCC’s algorithms on a GPU is non-trivial. Unlike 2D video frames, where pixels exhibit regularity, 3DGS training can produce irregularly-spaced Gaussians. Mapping this to a single-instruction, multi-threaded (SIMT) GPU is difficult.

CPU-resident algorithms for octree encoding and RAHT manage this irregularity by imposing a spatial hierarchy on the data. Unfortunately, GPUs are a poor fit for computations on hierarchical data. Traversing the hierarchy requires following pointers; this pointer chasing is known to be highly inefficient on a GPU, since if the referenced memory is not in the cache, all warps in a GPU thread must wait for the data to be loaded (sometimes from relatively slow DRAM), resulting in poor performance. This problem is well-known for GPU-based computations on irregular data in general [15], [16], and for parallelizing point-cloud algorithms in particular [17], [18].

Against this backdrop, GS-NFS is, to our knowledge, the first post-compression encoder/decoder for 4DGS in which all components (1) execute on a GPU and can encode and decode at full frame rate. To achieve this, the paper makes the following novel contributions:

• A GPU-based parallelization of octree encoding (2).

• A GPU-based parallelization of RAHT (3).

• A technique to improve the compression of 4DGS (4).

Our evaluations (5) on two different popular 4DGS datasets demonstrate that GS-NFS: (a) can encode an order of magnitude faster than the state-of-the-art; (b) dominates MesonGS [7], a GPCC [19] variant modified for 4DGS, and LTS [8] both in quality and compression performance, and performs comparably or better than V³ [6] on large scenes; (c) can decode 4DGS at up to 25 fps on a Jetson Orin; and (d) can encode and decode point clouds and live 4DGS videos at full frame rate, a capability that, to our knowledge, has not been demonstrated in the literature.

2 GPU-Accelerated Octree Encoding/Decoding↩︎

We first describe our GPU-accelerated octree encoding algorithm, which encodes Gaussian position attributes efficiently.

2.1 Background↩︎

Voxelization. Gaussians can be positioned irregularly in space, and all 3-D compression algorithms first voxelize Gaussian positions to regularize the data. Voxelization discretizes the cuboid bounding all the Gaussians into a regular grid of \(2^J \times 2^J \times 2^J\) cubes, where \(J\) is a voxelization parameter which controls the resolution of the 3DGS frame. Then, it maps each Gaussian’s 3-D position to one of the cubes, by quantizing the position coordinates to use \(J\) bits. Multiple Gaussians might fall into a single voxel; GS-NFS replaces those Gaussians with a single Gaussian whose position is at the center of the voxel and other attributes are the average of the original Gaussians.²

The Octree Data Structure.

After voxelization, many voxels are often empty and only a small fraction of voxels are occupied by Gaussians. An octree [20] efficiently encodes the positions of occupied voxels to exploit sparsity and spatial locality. It recursively partitions the 3D space into octants (8 equally-sized child nodes), creating a tree structure where each node represents a cubic region of occupied space.

Consider the octree in 2. The root node represents the entire 3D space, and its 8 child nodes represent the 8 equally-sized children. The child nodes that are occupied (i.e., contain at least one Gaussian) can be further subdivided into their own child nodes. The process continues until we reach a leaf node that contains a single occupied voxel.

This structure can efficiently encode occupancy. Each internal tree node uses a byte, where each bit indicates whether the corresponding child node is occupied (1) or empty (0). For example, if the fourth and fifth bits are set to 1, it means that the fourth and fifth child nodes are occupied, while the rest are empty (2).

Recursive Octree Encoding. 3-D compression must construct the octree (i.e., determine octants and their occupancy), then encode (i.e., serialize) the octree into a bitstream. Both of these can be accomplished with a single sequential level-order traversal from the root to the leaves; a reference implementation of G-PCC [21] uses this approach. This algorithm starts from the root, determines the occupancy of its child octants, then writes out the occupancy byte of the root. It then recursively repeats the process for each child in a fixed sequence (i.e., if children are numbered \(0..7\), it visits the children in that order), and stops when all occupied voxels form the leaves.

The output of this algorithm is a sequence of bytes representing occupancy at various nodes of the octree. The total number of bytes in this sequence is proportional to the number of occupied voxels \(\hat{N}_v\), which is often more compact than recording the occupancy of all \(2^{3J}\) voxels. In 2, for example, only two children of the root node (the 4th and the 5th) contain at least one occupied voxel, so its occupancy byte is \(00011000\). This forms the first byte of the output bitstream. The second byte corresponds to the root’s 4th child, all of whose children are occupied, so its occupancy byte is \(11111111\), and so on.

2.2 GPU-accelerated Octree Encoding↩︎

GS-NFS parallelizes octree construction and encoding on the GPU. Before describing this algorithm, we explain why this is a challenge, and how prior work has attempted to accelerate octree construction and encoding. The next subsection describes GPU-accelerated octree decoding.

Challenges. Recursive octree construction is inherently sequential. State-of-the-art point cloud compression algorithms [9], [10], [22] implement sequential CPU-based octree construction and encoding. Constructing the octree requires traversing the tree top-down and creating nodes and pointers. This is difficult to efficiently parallelize on a GPU since the memory layout of the tree can be irregular, and tree-traversal might require pointer-chasing, resulting in GPU thread stalls while waiting for memory accesses to complete [17], [18], [23]. Generating the occupancy bitstream also requires a tree traversal in level-order and pointer-chasing to determine the occupancy byte of a node depending on existing children, which can be inefficient on a GPU. 1 includes quotes from prior work [17], [18] that illustrate the difficult of parallelizing octree encoding.

Prior Work on Parallelization. One line of work has explored CPU parallelism for octree construction and encoding. ViVo [24], MeshReduce [25], and MetaStream [26] use coarse-grained CPU parallelism by partitioning the 3D points into separate blocks, generating the octree encoding independently for each block and concatenating these encodings. Others [17], [23] employ slightly more sophisticated hybrid CPU-GPU strategies. For example, Groot [17] sequentially constructs the octree top-down up to a certain depth \(d\), then generates encodings in parallel for each occupied voxel (on a GPU) by traversing the tree bottom-up up to level \(d\).

Another line of work [18], [27], [28] has achieved GPU-accelerated parallelization of octree construction. They linearize the positions in 3D to a 1D list using a space-filling curve (the Morton code, described below) while preserving spatial locality [29]. Thus, the generated octree is not explicitly represented as a tree structure, but rather as arrays of indices in the space-filling curve for each node at a certain level. In these approaches, however, encoding is still sequential, since that requires an in-order tree traversal. GS-NFS uses a similar space-filling curve, but parallelizes both construction and encoding on a GPU.

Morton Codes. The key to our GPU-accelerated octree encoder is to never build explicit tree pointers. Rather, we represent each occupied voxel by its Morton code [29], which is a bit-interleaving of its integer voxel coordinates. For a voxelized coordinate \(\hat{v}=(x,y,z)\in\{0,\ldots,2^J-1\}^3\), the Morton code is a bit string \(z_{J-1}y_{J-1}x_{J-1}\ldots z_0y_0x_0\) where \(x_b,y_b,z_b\) are the \(b\)-th least-significant bits of \(x,y,z\).

For example, consider the voxelization shown in 3, which depicts a \(4\times4\times4\) voxel grid (\(J=2\)) with 11 occupied voxels. The occupied voxel with ID \(33\) has coordinates \((1,0,2)\) so its Morton code is \(100001\). The Morton code is obtained by first taking the most significant bits of the 3 coordinates (\(100\)) in the order from \(z-x\), followed by the next most significant bits (0 for the z-coordinate value of 2, 0 for the y-coordinate value of 0, and 1 for the x-coordinate value of 1), and so on.

The Morton code has three properties crucial for GPU-acceleration of octree construction, encoding, and decoding. First, it preserves spatial locality: when we sort the voxels by their Morton code, voxels near each other in space will be near each other in the sorted order [29]. For example, if in 3, the voxel with ID \(33\) and coordinate \((1,1,2)\) is also occupied, it would have the Morton code \(100011\). This differs from the Morton code of its neighboring voxel \((1,0,2)\) only in the last 3 digits. This property ensures memory locality if voxels are laid out in GPU memory in Morton order, and avoids non-local memory access overheads.

Second, the Morton code for the parent of a voxel is a prefix of the voxel’s Morton code. In 3, the parent of \((1,0,2)\) (ID \(33\)) has the code \(100\), which can be obtained by shifting the voxel’s Morton code three bits to the right. Conversely, to obtain the Morton code of, say, the 5th child of an octree node, we simply append \(101\) to the node’s Morton code.

Third, given the Morton code for a voxel, we can obtain its voxelized coordinates using the definition of the Morton code. In our example, given the code \(100011\), the first coordinate is obtained by concatenating every third bit starting from position 3, the second coordinate by doing so from position 2, and so on, resulting in the coordinate \((1,1,2)\).

Thus, if given the Morton code for an occupied voxel, it is an \(O(1)\) operation to determine its parent. Conversely, if all octree nodes are sorted in Morton order, then a parent can in an \(O(1)\) operation determine its child’s Morton code, and look up its occupancy status. Finally, in \(O(1)\), given a node’s Morton code, we can obtain its voxel coordinates.

Before explaining how we use these properties to construct, encode and decode octrees, we describe how we parallelize voxelization (2.1). This is important since voxelization precedes octree construction. We require all steps in GS-NFS to be GPU-resident to avoid data movement costs (1.3).

Voxelization. Voxelization (a) discretizes Gaussian positions and (b) merges Gaussian attributes (by averaging them) of Gaussians at the same discretized positions. The input to this step is an \(N \times C\) tensor for a 3DGS frame with \(N\) Gaussians, with each Gaussian having \(C\) attributes. Step (a) is simple to parallelize on a GPU by launching one thread per Gaussian, which computes the voxel coordinates (2.1) of the Gaussian. This step also computes the voxel’s Morton code, and creates an array sorted by the Morton code that contains a pointer to the Gaussian. We use a 64-bit Morton code to support octree depths up to \(J=21\), which is sufficient for all scenes of interest [7]. If \(J\leq 21\), we set to zero the unused most-significant bits in the Morton code.

In this array, Gaussians belonging to the same voxel are contiguous; we identify these clusters using standard PyTorch operations. Then, we launch one GPU thread for each cluster to average the attributes of Gaussians in that cluster. The output of this process is an \(N_v \times C\) tensor, where \(N_v\) is the number of occupied voxels. This is sorted by Morton order, and forms the input to the next step.

Octree construction. In contrast to the recursive top-down approach (2.1), our construction identifies, bottom-up, occupied parents of occupied voxels, and occupied internal nodes up the octree (2). Let \(L_J\) be an array of occupied voxels, sorted by Morton order, obtained directly from the input tensor. We launch one GPU thread per occupied voxel; this thread computes the parent’s Morton code and writes this to a new list \(L_{J-1}\). This array contains parents of occupied voxels in Morton order, but can have duplicates, since multiple occupied voxels can have the same Morton code. We use CUDA’s unique primitive to compact duplicates in parallel, so \(L_{J-1}\) has exactly one instance of an occupied parent.

We can now repeat this process to find the next-level occupied parents (i.e., parents of occupied parents) up to the root level. The output of this process is a list of Morton codes for the occupied nodes at each level, i.e.,\(L_0,\ldots,L_J\), where \(L_0\) has the root node, \(L_J\) has the leaf nodes, and \(L_d\) has the occupied nodes at depth \(d\).

Bitstream generation. From these lists, the encoder must emit the octree occupancy bitstream. Each internal node contributes exactly one occupancy byte, whose \(i\)-th bit is set if child \(i\) exists. This step requires us to compute the occupancy byte for each internal node. To compute this, a node must: (a) determine the Morton codes of its children and (b) identify which children are occupied.

Given a parent code \(p_m\in L_d\), the Morton prefix of its children can be obtained by \(c_m = p_m \ll 3\). The eight possible children are therefore the contiguous code range \([c_m,c_m+7]\). To compute the occupancy byte for \(p_m\), we need to check which of these eight candidate child codes exist in the child list \(L_{d+1}\). This requires searching for the child codes in \(L_{d+1}\).

We make 2 observations that make this search efficient. First, all children with Morton codes in the range \([c_m,c_m+7]\) must appear contiguously in the sorted list \(L_{d+1}\). Second, we can perform a binary search for the first occupied child code \(\geq c_m\) in \(L_{d+1}\), since that listed is sorted.

To compute the occupancy bytes for all octree nodes, we use a top-down approach. We compute occupancy bytes for nodes in \(L_d\) before doing so for nodes in \(L_{d+1}\). At level \(L_d\), our octree encoder launches a custom CUDA kernel³ with one thread per parent node \(p_m\) in \(L_d\). It initializes \(p_m\)’s occupancy byte to 0. Then, the thread for \(p_m\) performs the following steps:. (i) compute \(c_m = p_m \ll 3\); (ii) binary-search \(L_{d+1}\) to find the first occupied child code \(\geq c_m\); (iii) scan forward over at most 8 entries until the child code exceeds \(c_m+7\); (iv) set bit for every child code \(c_m\) found. The thread then writes its occupancy byte to a known location in the GPU buffer. We present the detailed algorithm in 7.

At the end, all the occupancy bytes are written in order, similar to 2. To improve compression efficiency, GS-NFS then runs a GPU-based entropy encoder (1), ANS [30].

2.3 GPU-Accelerated Octree Decoding↩︎

Inputs and Output. The input to this algorithm is a sequence of occupancy bytes in the octree. We obtain this from the output of the ANS entropy decoder. The output of the algorithm is a list of voxelized positions of occupied voxels. Each such position corresponds to a Gaussian, whose attributes we obtain as described in 3.

This algorithm also needs to know \(J\), the depth at which the 3DGS frame was encoded, and the number of occupied nodes \(n_d\) at each level \(d\) in the tree. Such information is encoded in the bitstream as metadata.

Prior work. GROOT and [23] try to decode lower levels of the octree in parallel on a CPU, but most tree levels are still decoded sequentially on the CPU top-down. We know of no other work that has GPU-accelerated decoding.

Octree decoding. Our octree decoding algorithm relies on properties of the Morton code. We note that the first byte is the occupancy byte of the root. From this, we can determine (a) which children are occupied, and (b) the Morton code of the children (both \(O(1)\) operations). We can repeat this process to obtain the Morton codes of all nodes in the tree, and eventually, the Morton codes of all occupied voxels. From these, we can obtain the voxelized positions of the occupied voxels using the definition of the Morton code.

To parallelize this, we use a top-down level-synchronous approach (similar to the encoder), where, at each level \(d\), we launch one thread per octree internal node. For level \(d\), we can determine from the metadata where the occupancy bytes for that level start and end. But each node at that level can have a variable number of occupied children. To determine the offset of the \(k\)-th occupied node, we first create an array per internal node that counts the number of occupied children (using the population count CUDA primitive), then do a CUDA prefix sum on that array to find the offset of this node’s children in the occupancy bytes at level \(d+1\).

The \(k\)-th occupied node writes the Morton codes of its occupied children into a separate array starting at the corresponding offset. Then, the decoder launches one thread for each occupied thread at level \(d+1\), and the process repeats. After processing all levels, the decoder’s output is a list of Morton codes for the leaf nodes, which correspond to the occupied voxels. The decoder launches a final CUDA kernel to decode the Morton codes back into integer voxel coordinates by reversing the bit interleaving. 10 contains a detailed listing of this algorithm.

3 GPU-Accelerated Attribute Encoding/Decoding↩︎

In this section, we describe how we GPU-accelerate the encoding and decoding of Gaussian attributes, such as opacity, scale, rotation, and spherical harmonics.

3.1 Background: RAHT↩︎

Region-Adaptive Hierarchical Transform (RAHT) is an algorithm for encoding and decoding the attributes of sparse, spatially non-uniform data (such as point clouds [14] and 3DGS frames). The algorithm is complex to explain in its entirety, so we use an example to explain the algorithm; details can be found in [13], [14]. To further simplify the description, we describe how it encodes a single attribute, say opacity.

RAHT exploits the octree structure derived from voxel positions (4). It operates on a binary tree derived from the octree; in this binary tree, each octree level is represented by 3 levels in the binary tree, each of which corresponds to one of the directional axes. The leaves of the tree correspond to voxels, and in 4 we only show branches in the binary tree leading to occupied voxels, for simplicity.

If a node has a sibling (e.g., nodes 35 and 36 in 4), RAHT decomposes the attribute into:

1. Low-frequency Coefficient: A weighted average representing coarse signal information.

2. High-frequency Coefficient: A weighted difference representing the signal details.

More precisely, if two siblings \(l\) and \(r\) have attribute values \(a_l\) and \(a_r\), then computing these coefficients requires a \(2 \times 2\) matrix-vector multiplication: \[\begin{bmatrix} a^0\\ a^1 \end{bmatrix} = \frac{1}{\sqrt{w_l+w_r}} \begin{bmatrix} \sqrt{w_l} & \sqrt{w_r}\\ -\sqrt{w_r} & \sqrt{w_l} \end{bmatrix} \begin{bmatrix} a_l\\ a_r \end{bmatrix}, \label{eq:raht-butterfly}\tag{1}\] In this equation, \(a^0\) represents the low-frequency (coarse) coefficient, \(a^1\) the high-frequency coefficient, and \(w_l\) and \(w_r\) represents weights associated with \(a_l\) and \(a_r\) respectively. The weight at any node is the number of occupied voxels in the octant below that node; attributes at leaf nodes have unit weights. 4 shows node weights next to the node. RAHT propagates the low-frequency coefficient to the parent, while storing the high-frequency coefficients at the right sibling. (If a node does not have a sibling such as node 33, it passes its attribute value up to its parent). RAHT then repeats this procedure level by level, bottom-up, on this tree.

At the end of this computation, the octree root stores the low-frequency component, and internal nodes and leaves store high-frequency coefficients.⁴ Smoothly varying spatial signals (e.g., point clouds can have smoothly varying color, just like images) have near-zero high-frequency coefficients, especially close to the leaves, which can be quantized and entropy-coded, resulting in significant compression. In GS-NFS, we use RAHT to encode all 3DGS attributes (1.1) other than positions. RAHT decoding inverts this operation; for brevity, we discuss only the RAHT encoder and leave the decoder (inverse-RAHT) to 11.

3.2 Background: A Re-formulated RAHT Algorithm↩︎

Pavez et al. [13] describe a re-formulation (henceforth P-RAHT) of the original RAHT algorithm [14] that exploits properties of the Morton code. P-RAHT takes as input occupied voxels and their associated Morton codes. It pre-computes, in a step called the RAHT prelude, the constructs needed for the hierarchical RAHT computation, the sibling relationships and the node weights, since these are a function of voxel occupancy alone, not of the attribute. Then, it re-uses these when encoding each attribute.

The RAHT prelude. This step computes three lists at every level \(\ell \in \{1,\dots,3J\}\): (i) an index list \(I_\ell\) indicating those nodes in the binary tree at level \(\ell\) that are either left siblings, or singletons (those with no occupied siblings); (the binary tree in 4 shows only the nodes in these lists) (ii) a weight list \(W_\ell\) for those nodes (indicated by numbers inside nodes in 4, and (iii) a flag list \(F_\ell\) indicating which nodes are left siblings (not shown in the figure for brevity).

P-RAHT computes these lists bottom-up. Consider a node \(n\) at level \(\ell\). Its weight is the sum of the weights of its children at level \(\ell+1\). If either of its children is in \(I_{\ell+1}\), node \(n\) must be in \(I_{\ell}\). Finally, if \(n\)’s sibling at level \(\ell\) is also in \(I_{\ell}\), it is the left sibling if it has a lower Morton code. For example, the rightmost node at the second level from the bottom has a weight of 2, because its children are both occupied. For that reason as well, it belongs in the \(I_\ell\) list for that level. But, it does not belong in the corresponding \(F_\ell\) list, since it is not the left sibling at that level.

Computing RAHT Coefficients. Using these lists, P-RAHT computes the RAHT coefficients using another bottom-up pass for each attribute. If a node \(n\) at level \(\ell\) is a left sibling (i.e., its flag is set in \(F_\ell\)), then it computes the low-frequency and high-frequency coefficients using 1 . Nodes 24 and 35 in 4 are examples of left siblings. It then passes the low-frequency coefficient to its parent and stores the high-frequency attribute in its right sibling. For this, it uses the pre-computed weights in \(W_\ell\). If \(n\) is a singleton, such as node 33 in 4, it simply passes its attribute to its parent.

3.3 GPU-Accelerated RAHT↩︎

Inputs and Outputs. GS-NFS’s GPU-accelerated RAHT takes as input occupied voxels, their Morton codes, and the attributes of the merged Gaussian in the voxel. It outputs RAHT coefficients for all attributes.

Key Ideas. Our GPU-accelerated RAHT relies on three observations. First, we observe that the basic RAHT algorithm is similar to the octree construction, encoding and decoding algorithms (2). RAHT proceeds bottom-up on a tree, performing one or more computations at some but not all nodes (e.g., only siblings), at each level in the tree. The same is true for octree construction (2.2), for example, which performs computations only occupied nodes at each level.

This suggests that we can compute RAHT coefficients using the following design elements from those algorithms. (a) Those algorithms use properties of Morton codes to mitigate compute stalls due to pointer chasing. (b) They proceed sequentially level-by-level bottom-up or top-down, but within a level, they launch one GPU thread for every node in the tree involved in the computation. (c) The set of nodes at a level involved in the computation is of variable length, so these algorithms employ PyTorch [31] operations to convert fixed length vectors to variable length ones. For example, the octree construction algorithm first creates a list of all occupied parents, then uses the unique operation to identify exactly the set of nodes at the next higher level involved in determining occupied nodes.

Our second observation is that P-RAHT is an ideal starting point for parallelization. In particular, the pre-computed lists from the RAHT prelude greatly simplify applying the design elements described above.

Our third observation is that GPUs allow us to trivially parallelize RAHT coefficient computation across attributes. We can simply vectorize the attributes, and compute the coefficients in one bottom-up pass.

Computing RAHT coefficients on the GPU. GS-NFS computes RAHT coefficients bottom-up, exactly like P-RAHT. Unlike P-RAHT, at each level, it launches multiple GPU threads. Specifically, at level \(\ell\), it launches a GPU kernel with one thread per entry in the list \(I_\ell\). For example, at the lowest level in 4, GS-NFS will launch 11 threads, and at the next level up, 7. Only threads whose flag \(F_\ell\) is valid execute the \(2\times2\) transform; each such thread reads indices of a sibling pair (current and next element in \(I_\ell\)), looks up their weights in \(W_\ell\), and then applies Eq. 1 to the attributes.

Parallelizing the RAHT Prelude. At each step \(\ell\), GS-NFS maintains the current active list \(I_\ell\) as a flat GPU tensor of indices. We launch one GPU thread per element of \(I_\ell\) to compute \(W_\ell\) and \(F_\ell\). Each thread compares the Morton code of its current element with the next element (adjacent in Morton order) to decide whether they form a sibling pair.

After \(F_\ell\) is computed, GS-NFS obtains the next active list \(I_{\ell+1}\) by removing right siblings. It implements this efficiently using boolean shift and masked select of tensors which are available in PyTorch (Alg. 11). The output of this is a compacted list containing exactly the set of nodes at \(\ell+1\) that should run the RAHT coefficient computation at that level.

Especially for accelerating RAHT on a mobile GPU, we have found it essential to implement some of these steps using CUDA ( (5),

3.4 Quantization and Entropy Coding↩︎

After obtaining the RAHT coefficients for each attribute, GS-NFS quantizes these coefficients. Quantization trades off visual quality by reducing the number of bits required to represent a coefficient. We quantize coefficients of different 3DGS attributes to different levels, similar to [7], [19]. GS-NFS uses dead-zone quantization [32], which is simple to implement on a GPU, using one thread per coefficient, and vectorizing the attributes.

GS-NFS uses an entropy coder to efficiently serialize quantized coefficients into a bitstream. We use the Run-Length Golomb-Rice (RLGR) [33] entropy coder, which uses run-length coding for consecutive zeros, and Golomb-Rice coding for non-zero magnitudes using an adaptive parameter. RLGR is inherently sequential, but GS-NFS parallelizes this by dividing the vector of coefficients, and entropy coding blocks (e.g., 2K or 4K successive coefficients) using one GPU thread per block. This ensures fast entropy coding at the expense of a slight loss in compression efficiency.

We have omitted a discussion of de-quantization and entropy decoding for brevity; GS-NFS also parallelizes this. Thus, in GS-NFS, all steps in octree and attribute encoding are GPU-accelerated.

4 Improving 4DGS Compression↩︎

In many 4DGS videos, especially those that generate SH coefficients of degree 3, SH coefficients dominate the size of each 3DGS frame. Some post-training compression techniques have realized this, and have developed specialized techniques to compress SH coefficients. For example, MesonGS [7] devises a vector quantization codebook for each frame, in order to compress SH coefficients. This is extremely compute-intensive, so GS-NFS uses a different approach.

In 3DGS, each SH function has 3 RGB color channels. GS-NFS exploits the fact that RGB colors can be correlated [34]. It first transforms RGB to YUV for each SH function. In YUV, since Luma (Y) has more energy and it is more perceptually important than Chroma (UV), this transformation improves compression. GS-NFS applies this RGB to YUV transformation, a simple matrix multiplication, which is fast on a GPU, to all SH coefficients (both DC and AC).

While RGB to YUV conversion is common in image and video compression, because 3DGS represents color with high dimensional SH coefficients, there are additional opportunities to remove redundancy. After YUV conversion, GS-NFS applies a decorrelating transform, Karhunen-Loève transform (KLT [35]), only to the AC coefficients (i.e., to SH coefficients of degree greater than 0). More specifically, we apply three different KLTs, one to each channel in YUV color space. The KLT effectively concentrates signal energy on the first few coefficients. [fig:rate-dist-example] illustrates this. On the left is a heatmap of the Pearson correlation coefficients for the original RGB SH coefficients. On the right is the corresponding heatmap after applying RGB to YUV conversion followed by KLT; notice how the figure on the right has significant white-space (white corresponds to zero correlation).

By decorrelating color across AC SH coefficients, KLT reduces redundancy, concentrates energy, and thereby improves compression. We apply KLT before RAHT (1). KLT requires a matrix multiplication, which we implement on the GPU with minimal overhead.

5 Evaluation↩︎

In this section, we compare GS-NFS against the state of the art, using two popular 4DGS data sets.

5.1 Methodology↩︎

Implementation. We have implemented GS-NFS in Python. It uses custom CUDA kernels for the octree encoder/decoder (2), RAHT (3), and RLGR (3.4). It compresses position using the ANS entropy coder from NVIDIA’s nvCOMP library [30] and implements KLT decorrelation and quantization in PyTorch [31]; these run on the GPU. Most of the data and intermediate tensors use PyTorch.

Hardware. Most experiments run on a PC equipped with an NVIDIA RTX 4500 Ada Generation GPU (24 GB VRAM), 128 GB system RAM, and Intel(R) Xeon(R) w7-2475X with 20 physical cores (40 threads with hyperthreading). Mobile decoding runs on an NVIDIA Jetson.

Baselines. We compare GS-NFS against four post-training compression baselines:

V³-2D [6]

maps Gaussian attributes to 2-D image stacks, one per attribute channel, and compresses the resulting images with a H.264 codec using x264 from FFmpeg [36]. It compresses each channel of attributes into a single MP4 stream, where frames for that attribute form a group of pictures (GOP), and a single quantization parameter (QP) controls the quality-size tradeoff. It splits the position of Gaussians into MSB and LSB, where QP for MSB is 0 (lossless) and QP for LSB is the QP passed to the encoder (default: 25). V³-2D caps QPs for rotation and scale attributes to 22, and quantizes all other attributes (position LSB, color, opacity) with the same QP (default: 25). It compresses each attribute channel using one call of the H.264 encoder, resulting in 62 MP4 files per GOP (default: 20).

MesonGS [7]

is a partly GPU-accelerated codec that uses an octree for geometry, RAHT and block quantization for attributes, vector quantization for higher-degree SH coefficients, and LZ77 for entropy coding. We use the authors’ default parameters.

G-PCC [9]

Wang et al. [19] use G-PCC to compress static 3DGS, and adapt octree depth for voxelization to preserve quality. This requires fine-tuning of Gaussians after voxelization to reach high quality. To obtain a baseline that directly extends G-PCC to 4DGS videos, we implement a simplified version of [19] (without adaptive voxelization and fine-tuning). The approach uses RAHT for color and opacity (lossy, depending on QP) and hierarchical neighborhood prediction for rotation and scale (lossless).

LTS-Draco [8], [37]

uses modified Draco to independently compress each 3DGS frame on the CPU. The number of quantization bits per attribute is configurable (default: 16 bits for all attributes, compression level 10).

In each case, we use the implementations provided by the authors. Moreover, these baselines use qualitatively different compression techniques for 4DGS: 2D codecs [6], Draco [8], and G-PCC [7], [19].

Datasets. We evaluate on two widely-used 4DGS video datasets spanning diverse content types:

HiFi4G (7 sequences)

contains person-centric captures of actors performing various actions, with \(200\) frames per sequence. We use 3DGS models trained using V³ [6] with SH coefficient degrees \(0..3\) (\({\sim}120\)K–\(300\)K Gaussians per frame).

Neural 3D Video (N3DV) (6 sequences)

contains full-scene captures of indoor table-top activities with day and night lighting, each with 300 frames. We use models trained with QUEEN [2] (without its in-training compression) with SH degrees \(0..2\) (\({\sim}150\)K–\(400\)K Gaussians per frame).

Metrics. For each approach, we report measures of: (1) visual quality based on PSNR computed on rendered test views of the decompressed frames against the uncompressed ground truth (we use the test views used by V³ and QUEEN); (2) per-frame encode and decode latency in milliseconds (ms); and (3) compression efficiency in terms of 3DGS frame size after compression. We describe the precise metrics below.

5.2 Baseline Comparison↩︎

Methodology. This section compares GS-NFS against all baselines at a single operating point. For competing approaches, we select the default parameters for compression, such as QP, octree depth, bit-depth, etc., that we either found in the corresponding paper or source code. For GS-NFS, we use parameters that give us the same quality (PSNR) on the first frame of a sequence as the default setting for each approach. For instance, if, on the first frame of sequence \(S\), V³-2D has a PSNR of \(p\), we find a parameter setting for GS-NFS (using the technique described in 5.3) whose PSNR for the first frame on \(S\) is close to \(p\). This results in a pair-wise comparison between GS-NFS and each baseline.

We use these settings to encode, decode, and estimate PSNR and compression sizes for every 10th frame⁵ of every sequence in the two datasets listed above. We do this primarily because some of the baselines [7], [19] have exceedingly high encode/decode latencies, rendering an exhaustive evaluation intractable. Then, we compute, for each sequence: (a) the average per-frame encode latency, (b) the average per-frame decode latency, (c) the average per-frame PSNR difference between GS-NFS and the baseline (the \(\Delta\)PSNR), (d) average per-frame size ratio between GS-NFS and the baseline (the relative compression ratio, or RCR).

5.2.1 Encode-Decode Latency↩︎

2 summarizes per-frame encode and decode times aggregated across all sequences within each dataset. We do this because, within a dataset (e.g., HiFi4G), there is little variability in encode and decode times across sequences, since all sequences are person-centric. 8 contains a more detailed breakdown.

GS-NFS achieves \({\sim}23\) ms encode and \({\sim}21\) ms decode on N3DV, and \({\sim}18\) ms encode and \({\sim}14\) ms decode on HiFi4G—well under the 33 ms budget for 30 frames-per-second (fps).

LTS-Draco, the next fastest baseline, is 9–17\(\boldsymbol{\times}\) slower than GS-NFS for encoding and 7–8\(\boldsymbol{\times}\) slower for decoding. Draco encoding and decoding could potentially be made faster using coarse-grained CPU parallelism [24]–[26], but this trades off compressibility.

Though 2D video codecs are fast, V³-2D’s several hundred milliseconds to encode decode, due to the overhead of calling video encoder sequentially for each channel⁶; It is 16–61\(\boldsymbol{\times}\) slower than GS-NFS for encoding and 14–24\(\boldsymbol{\times}\) slower for decoding. V³-2D could offload H.264 encoding to an NVIDIA GPU using nvenc/nvdec [38]; however, nvenc allows only 8 parallel encoders on a desktop-class GPU [39], and V³-2D needs to encode 62 videos, so it is unlikely to achieve full-frame rate performance with offloading.

G-PCC has the next-highest encode and decode latencies; on both data sets, its encode and decode latencies are at least 260\(\times\) larger than GS-NFS’s. Although GS-NFS’s design borrows heavily from G-PCC elements, its GPU-acceleration results in a significant performance difference.

MesonGS is the slowest baseline. It also borrows heavily from G-PCC, and uses some GPU acceleration, but takes several seconds or minutes to encode and decode each 3DGS frame, mostly because of the time to learn the vector quantization codebook for AC coefficients [7]. Its decoding time is dominated by RAHT decoding which takes up to \(200\)–\(600\) ms.

Overall, GS-NFS can encode and decode one or two orders of magnitude faster than the baselines on our datasets.

5.2.2 Quality and Compression Performance↩︎

3 shows the \(\Delta\)PSNR and RCR performance for each sequence from both datasets, across all baselines. To understand the results, recall that: (a) \(\Delta\)PSNR captures the PSNR difference between GS-NFS and a baseline, and RCR the relative frame sizes after compression between the baseline and GS-NFS; (b) when we do this experiment, we find a setting for GS-NFS which has approximately the same PSNR on the first frame as the baseline’s default setting; and (c) the values in the table average these quantities across measured frames. Finally, a positive value of \(\Delta\)PSNR indicates that GS-NFS has better quality, and a value \(> 1\) for RCR indicates that GS-NFS has better compression performance. Given this, we expect \(\Delta\)PSNR to be close to zero for all sequences, for all approaches. This is because the sequences do not have significant scene changes, so a parameter setting from the first frame is likely to give mostly similar PSNR values for all other frames, for all baselines.

V³-2D. For this approach, \(\Delta\)PSNR is near zero for all sequences in HiFi4G. However, even though we roughly equalized PSNR, V³-2D has nearly 0.4 dB lower PSNR in the coffee_martini sequence, and a 0.74 db lower PSNR in the flame_salmon sequence in the N3DV data set. V³-2D has evaluated their approach on person-centric datasets, not on the larger scenes in this data set. We discuss this in greater detail in 5.3, where we show that this approach fails to achieve high quality on some N3DV sequences.

However, V³-2D has better compression performance than GS-NFS. On HiFi4G, for example, GS-NFS’s frame sizes can be 1.2–3\(\times\) larger than V³-2D. On N3DV, GS-NFS’s frame sizes can be 1.2–2.6\(\times\) larger than V³-2D. On average, for both datasets, V³-2D’s frame size is about half that of GS-NFS. By using a 2D codec, V³-2D can exploit inter-frame compression techniques in those codecs. In contrast, all other techniques, including GS-NFS only use intra-frame compression, so have lower compression performance.

MesonGS. For this approach, \(\Delta\)PSNR is uniformly near zero for all sequences, as expected. However, it is less compression-efficient than GS-NFS. On the HiFi4G, data set, MesonGS’s frame size is nearly 2\(\times\) that of GS-NFS, and on N3DV it is 1.46\(\times\). MesonGS uses similar algorithms as ours, but with two important differences. They use vector quantization for SH coefficients⁷, but we apply KLT (4) and then quantize RAHT coefficients. They also use LZ77 for encoding attributes and positions, we use ANS and RLGR entropy encoders. We conjecture that these differences contribute to MesonGS’s lower compression performance and to its higher encode-decode latencies (2).

LTS-Draco. For this approach, \(\Delta\)PSNR is near zero, except for a few sequences in N3DV, where LTS-Draco exhibits a 0.16–0.19 dB PSNR drop. In conducting this experiment, we sought settings for GS-NFS whose PNSR matched that of LTS-Draco, but this is not always possible, so we selected the configuration that provided the closest match. This is likely the reason for the discrepancy. LTS-Draco, however, has much worse compression performance; its frame sizes are nearly 3\(\times\) GS-NFS’s for HiFi4G and 5\(\times\) for N3DV.

G-PCC. Of all the approaches, G-PCC has the highest RCR; its compression performance is 3.4\(\times\) to 7.35\(\times\) worse than GS-NFS. Although GS-NFS uses similar components, the adaptation [19] of G-PCC to 4DGS that we use compresses SH AC coefficients far less aggressively than GS-NFS. That adaptation also applies lossless compression to scales and rotations (that is, it does not quantize coefficients before entropy coding), while GS-NFS applies lossy compression by quantizing those attributes. These two factors contribute to the significant differences.

5.3 Rate-Distortion Curves↩︎

To understand quality and compression trade-offs of different designs, we can also use a rate-distortion curve or R-D curve. Such a curve plots rate on the x-axis, quality or distortion on the y-axis, and the curve represents the best quality one can obtain for a given rate. To obtain this curve, practitioners explore a large space of codec parameters, obtain the rate and quality for each parameter, and plot these as points on the R-D plot. The R-D curve is the convex hull of those points.

R-D curves can also help encode videos at a given bitrate \(B\). To do this, we find the point on the convex hull whose rate is closest to \(B\), and use its configuration to encode at the target bitrate. We also used R-D curves to find the GS-NFS configuration with the nearest PSNR in 5.2.2.

Methodology. We computed R-D curves for the first frame of all sequences, for all our baselines. To do this, we sweep the space of parameters of each baseline and GS-NFS, plot rate and distortion and compute the convex hull as discussed above. 13 discusses the parameters we used for each baseline and for GS-NFS. For some approaches like MesonGS and G-PCC, we could not explore all parameter combinations because of their high encoding latency, and exploring the entire space would have taken hours or days. In contrast, for GS-NFS, we were able to evaluate 1000 combinations in about 12 minutes.

R–D curves. 6 shows the R–D curves for one HiFi4G sequence (Actor1) and two N3DV sequences (flame_salmon_1 and sear_steak). 13 and 14 depict the R-D curves for the remaining sequences for the two datasets respectively. If the R-D curve for approach \(A\) is entirely to the left of the curve for \(B\), for a given sequence, we say that \(A\) dominates \(B\). If the two curves intersect, then one is better in some bitrate regimes, and worse in others.

Consider the R-D curves for Actor1. Clearly, GS-NFS dominates MesonGS, Draco-LTS, and G-PCC. However, V³-2D dominates GS-NFS: it can, for a given quality, encode at a lower rate than GS-NFS. This explains why its compression performance was better in our baseline comparisons (5.2.2), and is likely due to its use of 2D codecs’ ability to do inter-frame compression.

The same is true for frame_salmon, except for this full-scene sequence, V³-2D’s R-D curve is closer to, but still dominates GS-NFS’s. For this sequence, both these approaches dominate MesonGS, Draco-LTS, and G-PCC.

On the other hand, for sear_steak, V³-2D does not dominate GS-NFS. In fact, for this sequence, and for three other sequences in N3DV out of a total of six (cook_spinach, cut_roasted_beef, and flame_steak, 14), V³-2D’s R-D curve shows that there is no parameter setting for which its quality reaches close to the ground-truth PSNR (we explored QP values from 1 to 40, where 1 is the best quality).

In each case, GS-NFS has a parameter setting that can provide 0.5–2 dB higher PSNR for the same rate. Moreover, for all of these sequences, GS-NFS has parameter settings that can reach ground-truth (GT) quality. Put another way, if GS-NFS were used to encode videos for DASH-based 4DGS streaming, for these sequences, it would provide much better quality at higher bandwidth-availability.

To our knowledge, V³-2D has only been evaluated on person-centric datasets and not on full-scene ones. Now, V³-2D must map 3D Gaussian positions to a 2D image. There are many possible 3D-to-2D mappings; V³-2D applies a fixed criterion to determine where an attribute corresponding to a Gaussian in 3D should be located in the 2D. Consider two consecutive frames in 3D; the 3D-to-2D mapping algorithm is applied independently to each frame, which means that even if motion in the 3D scene is smooth, Gaussian attributes close to each other on the first 2D frame may not be close to each other in the second 2D frame. As a sequence, because the location of the same 3D attributes may change from one 2D frame to another, motion-compensated prediction in the 2D frames may not work well, potentially resulting in a higher rate than an intra-only approach. We conjecture that, for this reason, V³-2D does not perform well in some N3DV sequences, where inter-frame coding becomes inefficient. On the other hand, GS-NFS does not exhibit this degradation because it uses a 3D codec.

5.4 Mobile Performance↩︎

GS-NFS supports 4DGS playback on resource-constrained devices. To demonstrate this, we evaluate GS-NFS’s decode performance on an NVIDIA Jetson Orin, a mobile GPU with compute comparable to those in in AR/VR headsets and edge devices [40]. 4 shows end-to-end decode latency on Jetson Orin. We observed that many 3DGS mobile implementations can support 30 fps 3DGS rendering for Gaussians with only SH DC coefficients (SH0), but not at higher SH degrees [6]. At SH0, GS-NFS achieves 40–59 ms across all sequences, allowing about \(17\)–\(25\) fps decoding. Latency scales sub-linearly with channel count: going from SH0 to SH3 (5.1\(\times\) more channels) increases average decode latency across all sequences by \({\sim2}\times\). At higher SH degrees, scene complexity dominates for full-scene content (more Gaussians): flame_salmon at SH2 (35 ch) is \(2\times\) slower than Actor1 at SH3 (56 ch), despite having fewer channels.

5.5 Novel GS-NFS Use Cases↩︎

So far, we have discussed using GS-NFS to compress stored 4DGS videos. In this section, we demonstrate that GS-NFS can compress, on-the-fly, live 4DGS videos generated by feed-forward Gaussian generators and point-clouds as well, both at full frame-rate. 14 also demonstrates GS-NFS’s ability to quickly compress large static 3DGS scenes.

Live 4DGS Compression. GPS-Gaussian [41] generates Gaussian at \({\sim}25\) fps (\({\sim}270\)K Gaussians per frame) using captures from 2 views of a person. We adapted GS-NFS to compress these frames on-the-fly. Specifically, our approach repeatedly takes 2 views of a person per frame (pre-loaded from disk), generates a 3DGS frame using GPS-Gaussian, compresses the frame using GS-NFS, and finally decompresses the frame to reconstruct the 3DGS frame. This pipeline can achieve 22 fps and an end-to-end latency (from capture to after decode) of 75 ms. While GS-NFS’s encoder and decoder takes only \(15\) ms each, the GPS-Gaussian generator takes \({\sim}43\) ms, and is the bottleneck in this pipeline.

Point cloud compression. Since GS-NFS builds upon G-PCC, it can compress point clouds (3DGS with only DC attribute). So, we evaluate its performance on popular point cloud-based volumetric video datasets. We use six sequences from two datasets: four single person sequences from 8i [42] (\({\sim}700\)K–1M points/frame; \({\sim}17\)–\(26\) MB) and 2 multi-person full-scene sequences from CMU Panoptic [43] (\({\sim}700\)K–850K points/frame; \({\sim}17\)–\(21\) MB). Point clouds are voxelized at octree depth \(J\!=\!10\) and GS-NFS’s codec uses a quantization step \(0.1\) for color attributes. GS-NFS can encode each frame in \(12\)–\(14\) ms and decode in \(9\)–\(12\) ms while achieving lossless geometry and color Y-PSNR of \(31\)–\(37\) dB. It can compress 8i sequences to \(270\)–\(370\) KB and CMU Panoptic sequences to \(660\)–\(780\) KB, 26–64\(\times\) smaller than the original size.

5.6 Ablations↩︎

KLT color decorrelation. GS-NFS decorrelates SH coefficients using a YUV transform followed by a per-channel KLT (4). 11 shows the impact on compressed size. RGB inflates the attribute bitstream by 43% compared to raw RGB, while YUV is larger than KLT by 36% for flame_salmon. For others, the reductions are smaller, but still significant. The additional latency for computing and applying the KLT matrix is less than 1 ms per frame, a negligible overhead.

CUDA vs.PyTorch RAHT decode on Jetson. RAHT takes about \({\sim}50\)% of the per frame decode latency, particularly on Jetson Orin. An efficient implementation of RAHT is crucial for mobile decode. GS-NFS’s custom CUDA RAHT kernels achieve \(2.0\)–\(3.9\times\) speedup over a PyTorch baseline on Jetson, with larger gains for lower SH degrees (10). We see a reduction of about \(4\)–\(5\times\) for SH0 and \(2\)–\(4\times\) for higher SH degrees for RAHT decode, which is the main bottleneck for mobile decode; prelude has \({\sim}2\times\) of speedup, since it doesn’t depend on the number of channels. These gains also reduce encode latency, since both forward and inverse RAHT perform the same work.

Other Ablations. 15 describes the cost of CPU octree coding (2), and explores RLGR parameter sensitivity of (3.4).

6 Related Work↩︎

During-training 3DGS compression. Recent research has focused on integrating compression directly into an optimization loop within training to enhance model sparsity. Compact3DGS [44] identifies prunes redundant Gaussians (as do [45], [46]) and replaces spherical harmonics with hash-based grids. CompGS [47] utilizes quantization-aware training and opacity regularization. 4D-GS [48] trains an MLP to predict motion across frames, while QUEEN [2] learns quantized attribute residuals and uses a gating module to sparsify positions. LapisGS [49] trains 3DGS in a layered progressive representation. Relative to GS-NFS, these methods incur significantly higher overhead for compression.

Post-training 3DGS compression. MesonGS [7] prunes insignificant Gaussians and then applies octree and RAHT. Some schemes [50], gauspcgc? use learning based methods to find the compression parameters for position and attributes. Others [8], [19] use point cloud compression techniques to compress trained 3DGS. To address this, V³ [6] encodes 4DGS in multiple 2D videos. GS-NFS encodes and decodes much faster than these techniques.

7 Conclusions↩︎

GS-NFS is a frame-rate GPU-accelerated encoder and decoder for 4DGS videos. To our knowledge, it is the first to demonstrate frame-rate encoding and decoding performance not just for 4DGS, but also for point clouds. It achieves this performance using novel parallelization algorithms for position and attributes. Its compression performance is also better than many state-of-the-art approaches due to a novel de-correlation approach. Future work can develop rate-adaptive codecs that can dynamically adapt compression levels in response to bandwidth changes, using GS-NFS’s ability to quickly encode and decode frames.

8 Coding Latency↩︎

In 5, we report the actual encoding and decoding latencies for all sequences in the HiFi4G and N3DV datasets.

9 GS-NFS Octree Encoding↩︎

Algorithm 7 summarizes our GPU octree encoder. Starting from voxelized integer coordinates, we first compute Morton codes for all occupied leaf voxels, then radix-sort and deduplicate them to obtain the leaf set \(L_J\). The upper octree levels are constructed bottom-up by repeatedly mapping child codes to their parents and removing duplicates, yielding level-wise node sets \(\{L_d\}_{d=0}^J\).

After the hierarchy is built, we serialize a compact occupancy bitstream. The header stores the number of levels and the node count of each level. The payload is then generated in breadth-first order: for each parent in level \(L_d\), we produce one occupancy byte indicating which of its eight children are present in \(L_{d+1}\). As shown in Algorithm 7, this is done by locating the base child Morton code and setting the corresponding bits for all valid children in the next level.

This construction avoids explicit pointer-based octree data structures and operates directly on sorted Morton codes, which is well suited to parallel primitives such as radix sort, deduplication, and batched searches. The resulting bitstream is a level-synchronous octree representation that can be decoded with the same breadth-first traversal order.

10 GS-NFS’s Octree Decoding↩︎

The GPU decoder reconstructs voxelized geometry by replaying the serialized occupancy stream in a level-synchronous manner, without building any explicit tree pointers. Starting from the implicit root Morton code (\(L_0=\{0\}\)), for each depth \(d\) it reads the \(|L_d|\) occupancy bytes, counts the number of occupied children per parent (population count), and performs an exclusive prefix-sum to compute disjoint write offsets for the next-level array. It then launches expand_children_kernel (Alg. 9), which generates each occupied child Morton code in \(O(1)\) via \((parent \ll 3)+i\) and writes all children contiguously into \(L_{d+1}\), avoiding atomics despite variable fan-out. Repeating this from \(d=0\) to \(J{-}1\) yields the leaf Morton codes \(L_J\), which are finally decoded into integer voxel coordinates (and optionally interleaved) in a parallel post-pass. Overall, decoding preserves the same breadth-first structure as the encoder while turning the key sequential dependency (unknown next-level size) into a standard GPU pattern: count \(\rightarrow\) prefix-sum \(\rightarrow\) parallel scatter. Full algorithm can be found here 8.

11 GS-NFS’s RAHT Implementation↩︎

We implement RAHT as a two-stage GPU pipeline: a prelude that builds the merge schedule from Morton-sorted occupied voxels, and a transform stage that applies the weighted orthonormal rotations using that schedule. This separation isolates the octree-dependent control flow from the per-attribute transform, allowing the latter to be executed with simple parallel kernels.

Algorithm 10 gives the reference non-parallel prelude. For each of the \(3J\) binary merge steps induced by a depth-\(J\) octree, it constructs the active index list \(I_\ell\), the corresponding run-length weights \(W_\ell\), and the sibling flags \(F_\ell\). Here, \(W_\ell\) records the number of original voxels represented by each active entry, and \(F_\ell\) marks whether an entry is the left element of a valid sibling pair. The sibling relation is tested directly from Morton codes using XOR and a level-dependent mask, without explicitly materializing the octree.

Algorithm 11 shows the GPU version of this schedule construction. At each level, one thread computes the weight and sibling flag for one active entry. The next active list is then obtained by removing right siblings while keeping left siblings and singletons. This produces the schedule \([(I_\ell, W_\ell, F_\ell)]_{\ell=1}^{3J}\) used by the transform.

Algorithm 12 applies the forward or inverse RAHT on the GPU. In the forward pass, levels are processed bottom-up; in the inverse pass, they are processed in reverse order. For each sibling pair, the transform uses the standard RAHT weights \[a=\sqrt{\frac{w_0}{w_0+w_1}}, \qquad b=\sqrt{\frac{w_1}{w_0+w_1}},\] to perform the corresponding orthonormal rotation independently for each attribute channel. Since pairs within a level are disjoint, the computation can be parallelized across active entries.

12 R-D Curves↩︎

This section provides the per-sequence R-D curves for all evaluated methods. For completeness, we include one representative frame from each sequence and plot the corresponding bitrate–quality trade-off. Figures 13 and 14 report the results for the HiFi4G and N3DV datasets, respectively.

13 Generating R-D Curves↩︎

To generate the rate-distortion (R-D) curves, we perform an exhaustive parameter sweep for each baseline codec, per frame.

For each parameter configuration, we compress and decompress the 3DGS representation of a single frame, render the decompressed Gaussians, and compute PSNR against the ground-truth renders (rendered from the uncompressed model).

All renders use a resolution-downscale factor of  2.

For V³-2D, which operates on groups of frames, we compress a group of 20 consecutive frames starting from the target frame and report the per-frame average compressed size; PSNR and SSIM are measured only on the target frame. All other baselines compress and evaluate a single frame independently.

The resulting (compressed size, quality) operating points are reduced to their upper convex hull to produce the final R-D curve.

Table [tab:rd-params] lists the swept parameters and their ranges for each baseline. Dataset-specific settings are noted where applicable.

\(^\ast\) HiFi4G / N3DV. G-PCC total = octree depths \(\times\) 45 QP triples.
\(^\dagger\) \(q_r\): QP for remaining attributes, \(q_d\): QP for DC coefficients, \(q_o\): QP for opacity.

14 Compressing Static Scenes↩︎

Static 3DGS scene compression. We evaluate GS-NFS on two large static outdoor scenes from the Deep Blending dataset (drjohnson,  3M Gaussians, 748 MB) and Tanks & Temples (truck,  2M Gaussians, 485 MB). 6 shows the results. GS-NFS achieves encode times of \(91\)–\(165\) ms, which is \(\boldsymbol{31\times}\) faster than LTS-Draco and \(\boldsymbol{1{,}200}\)–\(\boldsymbol{3{,}400\times}\) faster than MesonGS. At quality comparable to LTS-Draco (\(<\)​1 dB PSNR difference), GS-NFS produces compressed files that are \(5\)–\(10\times\) smaller.

15 Other Ablations↩︎

GPU vs.CPU octree encoding. We compare GS-NFS’s GPU octree encoder against Draco [10] and G-PCC [9] for position-only compression. 9 shows that GS-NFS’s GPU octree encoding time is 16-34\(\boldsymbol{\times}\) faster (\(8\)–\(14\times\) for decoding) than Draco and 70–100\(\boldsymbol{\times}\) faster than G-PCC for encoding (\(50\)–\(65\times\) for decoding), while producing bitstreams of comparable size. ANS entropy coding of the octree occupancy bytes adds only \({\sim}0.3\) ms but roughly halves the compressed size.

RLGR parallelization.

GS-NFS parallelizes RLGR entropy coding by partitioning each attribute channel into fixed-size blocks and encoding them independently on GPU threads (3.4). 8 compares the CPU baseline (sequential per-channel RLGR) against GPU variants with different block sizes. Parallel RLGR a block size of \(512\) achieves a \(\boldsymbol{58\times}\) encode and \(\boldsymbol{109\times}\) decode speedup on a desktop GPU over CPU, with only \({\sim}1.5\)% increase in compressed size. On Jetson Orin, a block size of \(512\) achieves \({\sim}6\) ms decode time per frame, compared to \({\sim}180\) ms for the CPU implementation–a \(30\times\) speedup. Smaller block sizes below 512 starts having diminishing returns. Block size 2048–8192 provides a balanced tradeoff: \(18\)–\(51\times\) speedup in encode/decode latency with negligible size overhead.

References↩︎

---

1. NFS, or Need for Speed, is a popular car racing game.↩︎

2. This may not be ideal for small \(J\), but suffices for the range of \(J\) we consider in this paper.↩︎

3. For efficiency, when necessary, we replace PyTorch operations with custom CUDA kernels to improve encoding speed.↩︎

4. This computation can be intuitively thought of as an extension of the Haar wavelet transform to a sparse 3D grid [14].↩︎

5. For V³-2D, we compute the size of a frame as the average size of the GOP that it belongs to.↩︎

6. V³-2D encodes and decode the full scene N3DV dataset faster than the HiFi4G person-centric dataset, because, on the former dataset, QUEEN does not produce degree-3 SHs.↩︎

7. MesonGS [7] was designed to encode static scenes and we have used it to encode 4DGS. Most video codecs would only retrain the vector quantization codebook every few frames, so our approach slightly over-estimates MesonGS frame sizes↩︎