1 Introduction↩︎

The current generation of Large Language Models (LLMs) feature on the order of \(10^{12}\) learnable parameters trained on \(10^{13}\) tokens of text data [1], [2]. Training models of this scale is not feasible on a single Graphical Processing Unit (GPU) due to memory limitations and prohibitively long training times. As a result, distributed computation across multiple GPUs has become essential in artificial intelligence (AI) model engineering. Moreover, power delivery constraints and physical infrastructure limitations increasingly prevent the concentration of sufficiently large GPU clusters within a single datacenter [3], while building facilities in remote locations with abundant and affordable power further motivates geographic distribution. Together, these factors are driving the adoption of multi-datacenter training architectures where computation is distributed across geographically separated sites. In such distributed systems, efficient communication between GPUs is critical for achieving high training throughput.

Data parallelism is one strategy for distributed training, where the model is replicated across multiple GPUs and each GPU processes a different subset of the training data. After computing local gradients, GPUs must synchronize through collective communication operations such as all-reduce to average gradients before updating model parameters [4]. Modern deep learning frameworks employ compute-communication overlap techniques, where gradient synchronization is performed concurrently with computation to hide communication latency [5], [6]. Discrete-event simulation (DES) methodology enables the analysis of transient compute and communication events during LLM training across various network configurations.

However, as network interface card (NIC) bandwidths continue to increase—from 400 Gbps to 800 Gbps and beyond 1.6 Tbps—the serialization time for data transmission decreases, causing fiber propagation latency to become an increasingly dominant factor in total communication delay. For geo-distributed training across datacenters (DC) separated by tens to hundreds of kilometers, this latency can significantly degrade compute-communication overlap and reduce effective GPU utilization. Despite the growing interest in scale-across multi-DC training architectures, the impact of fiber latency on training efficiency has not been systematically characterized.

In this paper, we use DES to simulate data-parallel training of GPT-3 models (13B and 175B parameters) across dual-DC GPU clusters of up to 8192 units, comparing hollow core fiber (HCF) and standard single-mode fiber (SMF) over distances from 0.3 km to 1000 km. To the best of our knowledge, this is the first systematic study that: (1) quantifies how inter-DC fiber propagation latency degrades compute-communication overlap as a function of distance and GPU generation; and (2) demonstrates that HCF enables up to 25% higher compute-communication overlap than SMF at inter-DC distances of 10–100 km.

2 Methods↩︎

Communication Time Model. For point-to-point communication between GPUs, the total time to transmit a message can be decomposed into serialization time (bandwidth-dependent) and propagation time (latency-dependent): \[\label{eq:comm95time} T_{\mathrm{comm}} = T_{\mathrm{serialization}} + T_{\mathrm{propagation}} = \frac{M}{B} + \frac{D}{v}\tag{1}\] where \(M\) is the message size, \(B\) is the NIC bandwidth, \(D\) is the distance between GPUs, and \(v\) is the signal propagation speed in the fiber. For SMF, \(v \approx 2 \times 10^{8}\) m/s, while HCF achieves \(v \approx 3 \times 10^{8}\) m/s (close to the speed of light in vacuum), resulting in \(\sim33\%\) lower propagation latency. We assume negligible switch latency and dedicated uncongested LLM training network.

As NIC bandwidths increase, the serialization term decreases, causing the propagation term to dominate for messages transmitted over longer distances. Gradient bucket sizes in distributed training typically range from 1–100 MB, with most collective communication frameworks providing bucket sizes in this range [5]. Pure data-parallel workloads operate at the upper end, while hybrid parallelism strategies that combine model and data parallelism [7], [8] reduce the per-rank gradient volume and push effective message sizes toward the lower end. This can cause the communication to enter a latency-dominated regime at longer distances.

Compute-Communication Overlap. Distributed training efficiency depends on the ability to overlap communication with computation. During backpropagation, gradients are computed layer-by-layer and can be communicated incrementally or combined through bucketing. While one chunk undergoes all-reduce synchronization, the GPU continues computing gradients for subsequent layers. The degree of overlap is quantified as: \[\label{eq:overlap} \eta_{\mathrm{overlap}} = \frac{T_{\mathrm{compute}}}{T_{\mathrm{total}}}\tag{2}\] where \(T_{\mathrm{compute}}\) is the time spent on computation and \(T_{\mathrm{total}}\) is the total training iteration time. When \(\eta_{\mathrm{overlap}} = 1\), all communication is hidden behind compute [6]; when \(\eta_{\mathrm{overlap}} < 1\), GPUs experience idle time waiting for communication to complete.

ASTRA-sim. For our simulations we use ASTRA-sim, an open-source DES for distributed machine learning systems [9], [10]. It models the interplay between training workload (model architecture, compute requirements), system configuration (collective communication algorithms), network properties (topology, bandwidth, latency), and schedules events to enforce compute-communication overlap.

Lumped DC Representation. To focus on inter-DC communication effects and simplify the intra-DC complexities, we adopt a lumped DC abstraction where each DC is represented as a single high-capacity GPU node with symmetric internal connectivity. This simplification avoids modeling detailed leaf-spine network topologies, which means our results isolate the impact of inter-DC propagation latency. Figure 1 illustrates the multi-DC architecture with two lumped DC nodes connected via long-haul fiber.

Simulation Parameters. Table 1 summarizes the simulation parameter space. We simulate two DCs with data-parallel training of GPT-3 models (13B and 175B parameters) [1]. We estimate compute times from theoretical peak GPU performance: 312 TFLOPS for A100 and 989 TFLOPS for H100 (tensor cores). Intra-DC communication uses a switch topology with 6 parallel links, representing effective attributes for an aggregated representation of NVLink (scale-up) and InfiniBand/Ethernet (scale-out) fabric within the DC. Inter-DC bandwidth is set to 100 GB/s or 200 GB/s, with latency determined by distance and fiber type.

3 Results↩︎

Figure 2 presents the compute-communication overlap (\(\eta_{\mathrm{overlap}}\)) as a function of inter-DC distance for the 8192 GPU configuration.

Several key trends emerge from the simulations. First, compute-communication overlap decreases monotonically with increasing inter-DC distance for all configurations, as longer propagation delays reduce the ability to mask communication behind computation. At distances below 10 km, near-complete overlap (\(\eta_{\mathrm{overlap}} \approx 1\)) is achieved regardless of fiber type. As the distance increases, overlap drops significantly for both fiber types, though HCF maintains a consistent advantage.

Second, HCF outperforms SMF across all tested configurations. Figure 3 shows the absolute improvement in overlap (\(\Delta\eta = \eta_{\mathrm{HCF}} - \eta_{\mathrm{SMF}}\)) with the benefit peaking at intermediate distances where the system transitions from bandwidth-limited to latency-limited operation. In the latency-dominated regime, HCF can reach up to 50% greater inter-DC separation than SMF for the same propagation delay, extending the feasible radius for multi-DC deployments.

However, at very long distances both fiber types yield low absolute overlap, limiting practical utility (figure 2). This low overlap is affecting the total training time, as captured in figure 4. Training time increases with distance, with H100 being impacted more at longer distances, due to the less overlap when compared to A100. This further validates that all-reduce operations with communication message size and frequency to maximize compute-communincation overlap at longer distances become very costly.

Third, larger models exhibit higher baseline overlap than smaller models. The GPT-3 175B model, with its greater computational requirements per layer, provides more time for communication to complete in the background compared to the 13B variant. This suggests that the largest frontier models may be more amenable to geo-distributed training than smaller models.

Fourth, faster GPUs reduce overlap by completing computation more quickly, thereby exposing communication latency. The H100, which is approximately 3\(\times\) faster than the A100 in FP16 tensor operations, shows lower overlap values across all distances. Note that lower overlap does not imply slower training—H100 clusters still complete iterations faster in absolute time—but it does indicate lower hardware efficiency for these GPT-3 transformer models. As GPU compute capabilities continue to advance, communication latency will become an increasingly critical bottleneck.

Simulations with 256 and 2048 GPUs exhibit the same trends in overlap efficiency and are omitted for brevity. To isolate the role of bandwidth versus latency, we also doubled the inter-DC link rate from 800 Gbps to 1.6 Tbps for the 256-GPU configuration. When bandwidth is doubled, a maximum of 0.66% improvement is observed in \(\eta_{\mathrm{overlap}}\), confirming that propagation latency—not bandwidth—is the dominant bottleneck at inter-DC distances for communication message sizes aimed at maximizing compute-communication overlap.

4 Conclusions↩︎

ASTRA-sim simulations of data-parallel GPT-3 training show that compute-communication overlap degrades significantly with inter-DC distance. HCF consistently outperforms SMF when performing synchronous per-layer collective communications and it enables up to 50% greater separation and 25% higher compute-communication overlap. Larger models and slower GPUs are more amenable to geo-distribution due to higher compute-to-communication ratios. This benefit is more pronounced between 10-100km distances where the overlap benefits are high but the training time degredation due to latency-sensitive communication is low. Although not studied in this work, we expect that the use of hybrid (data and model) parallelization strategies [7], [8] will reduce per-message sizes and enter the latency-dominated regime at shorter distances. This, will expand the range of distances where the benefit of low-latency fiber is prevalent. Future work will extend to hybrid parallelism, mixture-of-experts architectures, distributed inference workloads, bucketing of gradient updates, and multi-tier network hierarchies.

References↩︎