1 Introduction↩︎

The performance of large machine-learning systems increasingly depends on their networks. High-bandwidth domains (HBDs) such as NVLink provide abundant bandwidth, but are expensive and difficult to scale. Once a job extends beyond the HBD, communication over the scale-out network often becomes a bottleneck.

Optical networks offer a way to expand HBDs with significantly lower power and cost than a conventional packet switch at every tier. Direct-connect and optical circuit-switched (OCS) networks are already used inside production training and inference systems [1], [2], and recent proposals demonstrate the benefits of reconfiguring the scale-out fabric for individual collectives [3] or interconnecting multiple smaller HBDs [4]. Because optical networks directly connect nodes to each other, they are particularly attractive when the communication pattern is static and regular, as is the case for data-, tensor-, and pipeline-parallel training. A direct-connect fabric can be structured to match this stable collective structure, providing strong performance at production scale in industry today [1], [2], [5]–[7].

Mixture-of-Experts (MoE) models break this regularity. In every MoE layer, a gate selects, independently for each token, its top \(K\) experts out of \(E\). When experts are distributed across accelerators, the resulting expert-parallel (EP) communications are sparse AllToAll-V: each accelerator dispatches tokens to a data-dependent subset of peers and later collects the expert results. The demand changes with every batch, can span the full EP group, and is skewed because some experts receive more tokens than others. Unlike a ring or an AllReduce, this traffic does not lend itself to one stable, structured topology.

Expert domains are rapidly outgrowing commodity HBDs: Mixtral has eight experts per layer [8], Qwen3 and LLaMA 4 have 128 [9], [10], DeepSeek-V3 and Pangu use 256 [11], [12], Qwen3.5 reaches 512 [13], and Kimi K3 uses 896 [14]. The number of active experts (top-k) is also increasing, with DeepSeek-V3 using \(k=8\), Qwen3.5 using \(k=10\), and Kimi K3 using \(k=16\) [11], [13], [14], as are EP group sizes, with DeepSeek-V3 already being trained using 64-way EP [9], [11]. A low-degree direct-connect fabric cannot provide single-hop connectivity to all such destinations. At the same time, OCSes reconfigure too slowly to be operated during a collective [3], [4]. While MixNet [4] has combined runtime-optimized direct optical links with packet switches for traffic lacking a direct circuit, this strategy greatly limits the utility of the optical network when system scale exceeds the degree at each node. Efficient optical MoE therefore requires indirect routing, even with topology specialization.

Indirect routing creates two problems. First, every relay hop consumes bandwidth without delivering data locally, resulting in bandwidth tax. Second, relay traffic may be distributed unevenly across links, resulting in load imbalance. This occurs in irregular topologies such as random expanders and in nominally regular topologies whose physical connectivity is asymmetric, such as TPU 8i Boardfly. Skewed MoE demand further amplifies these structural hotspots. Their effect is significant because collective completion time is determined by the most loaded link: in our evaluation on a DeepSeek-V3 MoE traffic, the busiest link carries up to \(1.35\times\) the average link load within Boardfly’s network.

These two problems expose a topology tradeoff. Common structured topologies, such as 3D-torus, use symmetry to distribute relay traffic evenly, but at fixed node degree their relatively high radius requires more relay hops and therefore a larger bandwidth tax. Low-radius irregular topologies remedy the first problem. Random expanders connect distant endpoints in few hops at low degree and can scale to large systems [15]; however, their irregular paths make some links more likely to relay traffic, worsening the second problem. We use expanders to develop this intuition, but our techniques apply to any topology.

We present MoX¹, a static, demand-oblivious MoE routing system that uses EP semantics and precomputed top-K-specific per-link weights instead of predicting traffic matrices or reconfiguring the network. Its two techniques address indirect routing’s costs:

Token-aware multicast and reduction trees. MoE dispatch is a collection of per-token multicasts: the same token must reach the \(K\) accelerators hosting its selected experts. MoX performs indirect routing of a token via nodes which also receive that token, forming a multicast tree. During combine, it reverses these trees and partially reduces expert outputs at intermediate accelerators. Similar techniques were applied in prior work on efficient collectives in direct-connect topologies [16], [17], but MoX is the first to show that applying them to MoE in this way is highly effective for reducing the bandwidth tax.

Precomputed skew-oblivious load-balanced routing. MoX formulates relay selection as a compact surrogate for the fractional multicast-tree packing problem studied in traffic engineering [18], [19]. Each source and top-\(K\) expert selection defines a multicast demand class; the selected experts are mapped to their hosting accelerators, inducing a set of candidate multicast trees. Rather than predict the deployed traffic matrix, MoX assigns equal likelihood to all top-\(K\) expert combinations and computes link routing weights, one weight per link, that minimize the maximum expected physical-link load over the resulting tree distribution. Enumerating all destination sets and trees is combinatorial, so MoX computes the weights offline from a small sample of multicast workloads. The resulting policy remains effective on real, non-uniform MoE traffic for the given choice of \(K\), without runtime demand prediction or per-iteration topology adaptation.

Expert popularity is often skewed [4], [20], and both load and its variation differ across MoE layers [21]. Nevertheless, modern MoE training encourages balanced marginal utilization through auxiliary losses [22] or load-dependent routing biases [11], [23]. Moreover, experts from different layers are randomly mapped to accelerators, so a hot expert in one layer need not create a hotspot at the same physical endpoint in another. Thus, after aggregating layers under a randomized mapping, we expect a uniform prior to provide a useful topology-level center for offline optimization.

We evaluate MoX using token-level Chakra traces in ASTRA-sim 2 [24], [25], with DeepSeek-V3 and Qwen-3 MoE layers and traffic distributions recorded on a real GPU cluster.

We make three observations:

For recorded traces, on 16-, 32-, and 64-node degree-8 expanders, top-12 training is within 0.5%, 0.6%, and 7% of an ideal switch, while min-hop routing is 42%, 69%, and 96% slower. With 256 inference tokens per GPU, MoX is within 2.1%, 3.1%, and 13.6%, while min-hop is 42–98% slower.

On recorded traffic, static MoX routing achieves 27% lower completion time than a demand-optimized topology with min-hop routing and nearly attains the perfect switch performance across evaluated skews.

On a regular 1,024-TPU hierarchy constructed from Google’s published TPU 8i Boardfly parameters, where connectivity constraints introduce endpoint-level asymmetries, MoX reduces bottleneck cable load by up to 47%.

Bottom line. MoX shows that high-performance MoE communication is achievable without continuously adapting the physical network to runtime demand: a static direct-connect topology can approach switched-network performance when routing is designed around MoE’s multicast semantics and load-balanced offline.

2 Routing Algorithm↩︎

MoX routes MoE token embeddings over a fixed, known network topology. For every token, it first constructs a multicast tree that minimizes redundant forwarding and then selects among equivalent relays using offline-computed link weights. During combine, it reverses the tree and partially reduces expert outputs at intermediate nodes. 1 shows a simple example of the routing process. Throughout this section, dispatching a token means transmitting its MoE-layer embedding (activation vector).

2.1 Building the Multicast Tree↩︎

For a token generated at source \(s\), let \(D\) denote the accelerators hosting its top-\(K\) experts. MoX constructs a directed multicast tree rooted at \(s\) and spanning \(D\). It maintains a holder set \(H\), containing the nodes that already have the embedding, and initializes \(H=\{s\}\).

At each step, MoX selects an unreached destination \(d\in D\setminus H\) and a path from a node in \(H\) to \(d\). It prefers a path whose intermediate nodes are themselves members of \(D\): those nodes require the embedding and can subsequently relay it to another destination. More generally, it selects paths that minimize the number of intermediate nodes outside \(D\). The selected path is added to the multicast tree and its nodes are added to \(H\). This process continues until all destinations have been reached.

The two-hop case illustrates the rule. If a selected expert \(r\) lies on a path from \(s\) to another selected expert \(d\), MoX sends one copy along \(s\rightarrow r\rightarrow d\). Because \(r\) consumes the embedding as well as forwarding it, both hops perform useful delivery. If no such destination relay exists, MoX uses a non-destination relay and incurs one bandwidth-tax hop. Ties among paths with the same tax are resolved by the load-balancing policy in 2.2.

The source controls the tree and encodes the remaining destinations as a bitmap in the packet header, similarly to stateless multicast mechanisms such as Xcast [17] and BIER [16]. A relay forwards one copy along each outgoing tree edge. At 64 endpoints, the bitmap occupies 8 bytes, compared with a 14 KB token embedding.

2.2 Load Balancing with Per-Link Weights↩︎

Tree construction often exposes several relay choices with the same bandwidth tax. Consider an unreached destination \(d\) and let \(C=H\cap N(d)\) be the current holders adjacent to \(d\). Every \(r\in C\) can deliver the embedding to \(d\) in one hop. MoX associates a positive weight \(w_{(r,d)}\) with every directed physical link and assigns the relay share \[p(r\mid C,d) = \frac{w_{(r,d)}}{\sum_{r'\in C}w_{(r',d)}}. \label{eq:relay-prob}\tag{1}\] The weights are shared across all tokens and candidate sets; MoX does not store a separate policy for each \(C\).

At runtime, MoX realizes these shares with weighted round robin (WRR). Each source maintains per-link counters for the duration of one MoE collective. For candidate set \(C\), it selects the relay minimizing \[\frac{\operatorname{counter}(r,d)}{w_{(r,d)}} \qquad r\in C, \label{eq:wrr}\tag{2}\] and then increments the selected link’s counter. The counters are reset at the start of the next collective. Over many tokens, this deterministic policy approaches the shares in 1 . A single weight per directed link keeps the policy state proportional to the physical topology; for example, a 64-node degree-8 topology requires 512 weights.

When no destination relay is available, MoX selects among equal-tax first hops using round robin. We treat the load produced by these unavoidable paths as background load when computing the WRR weights. Paths with multiple non-destination relays are handled analogously.

2.3 Fractional Multicast-Tree Packing↩︎

The ideal load-balancing problem is a fractional multicast-tree packing problem [18], [19]. Let \(Q\) denote the set of multicast demand classes. A class \(q\in Q\) specifies a source, a top-\(K\) destination set, and expected demand \(a_q\). Let \(\mathcal{T}_q\) be the set of minimum-tax multicast trees admitted by the forwarding algorithm for \(q\). For each \(T\in\mathcal{T}_q\), variable \(x_{qT}\geq0\) is the fraction of class-\(q\) demand routed on tree \(T\). With directed-link capacity \(c_e\), the minimum-congestion packing is the linear program \[\begin{align} \min_{x,L_{\max}}\quad & L_{\max} \tag{3} \\ \text{s.t.}\quad & \sum_{T\in\mathcal{T}_q}x_{qT}=a_q && \forall q\in Q, \tag{4}\\ & \sum_{q\in Q}\sum_{T\in\mathcal{T}_q:e\in T}x_{qT} \leq c_e L_{\max} && \forall e\in E. \tag{5} \end{align}\] The first constraint routes all expected demand; the second bounds every physical link’s utilization. The objective minimizes the utilization of the busiest link. This formulation is closely related to classical multicast packing and minimum multicast-congestion formulations, which likewise assign concurrent multicast demands to Steiner trees while minimizing their maximum edge congestion [18], [19].

The LP is not practical to materialize: \(\mathcal{T}_q\) is exponential, and the number of top-\(K\) destination sets is combinatorial. It would also require candidate-set- or tree-specific routing state at runtime. MoX therefore optimizes a compact surrogate: the tree shares are not independent variables, but are induced by the per-link weights through 1 . The surrogate retains the LP’s min-max link-load objective while reducing the runtime policy to one value per directed link.

2.4 Iterative Weight Computation↩︎

MoX computes the weights offline using Monte Carlo workloads. It starts with uniform weights, \(w_e=1\), routes every sampled token using the complete tree-construction and WRR policy, and measures the resulting directed-link loads \(L_e\). This simulation includes the fixed round-robin traffic from paths that require non-destination relays, so the measured load combines useful forwarding and bandwidth-tax traffic.

After a replay, MoX updates every weight using \[\log w_e \leftarrow \log w_e - \eta\left(\frac{L_e}{\bar L}-1\right), \qquad \eta=0.05, \label{eq:weight-update}\tag{6}\] where \(\bar L\) is the mean directed-link load. An overloaded link loses weight and is selected less frequently in subsequent replays; an underloaded link gains weight. The update is a multiplicative min-max controller for the compact weight policy: it seeks the same low-congestion outcome as 3 5 , but does not enumerate or explicitly solve for the LP’s tree variables. We retain the weight vector from the replay with the lowest observed maximum link load.

2.4.0.1 Sampling configuration.

MoX samples \(1{,}000\) tokens per source accelerator; for each, it draws a top-\(K\) expert combination from the assumed uniform workload distribution and applies the expert-to-accelerator mapping to obtain its destination set. This same sample is replayed across all six iterations—one with uniform weights followed by three weight updates using 6 . We validate that at small sizes with \(n=16\), where full enumeration is feasible, the sampled solution matches the enumerated solution. In terms of the runtime, at \(n=64\), the offline computation completes in seconds on one CPU core.

2.5 Partial Reduction on Combine↩︎

After expert computation, the combine phase returns the expert outputs to the token’s source. Semantically, this operation is a reduction: the outputs of the top-\(K\) experts are combined into one activation vector for the next layer. MoX traverses the dispatch multicast tree in reverse. Whenever an intermediate node has results from multiple child branches, it partially reduces them before forwarding one vector toward the source. This avoids sending every expert output independently across shared links.

3 Evaluation↩︎

We evaluate MoX with ASTRA-sim 2 [25] using token-level EP Chakra traces [24], representing each routing decision as per-link transfers. We record all layers in three late-training DeepSeek-V3 runs [11] on 16, 32, and 64 GPUs with 256 experts. In addition, we record the expert popularity for the Qwen-3 225B with 128 experts. Expert popularity is stable over the sampled iterations and is similar across layers, consistent with prior observations [4], [21]; we evaluate a randomly selected layer. In all the experiments the experts are randomly and evenly placed across GPUs.

Training dispatches 5,120 BF16 embeddings of width 7,168 per GPU; inference uses the same expert distribution with between 64 and 256 tokens. We compare MoX and pairwise min-hop routing on bandwidth-equivalent direct-connect fabrics against an ideal full-connectivity packet switch. Unless noted, each endpoint’s 800 Gbps is split into eight 100 Gbps links on a randomly generated degree-8 expander selected for low radius. We also evaluate the performance of MoX on the TPUv8i Boardfly topology.

3.1 MoE end-to-end performance↩︎

2 reports full MoE-block (dispatch–compute–combine) time for DeepSeek MoE. MoX improves over min-hop by up to \(1.8\times\). Training overhead grows with topology size as paths lengthen, but decreases with higher top-\(k\) because more destinations can serve as useful relays. MoX is within \(0.6\%\), \(3.0\%\), and \(15.1\%\) of an ideal switch for 16, 32, and 64 GPUs at top-\(8\), shrinking to \(0.5\%\), \(0.6\%\), and \(7.3\%\) at top-\(12\).

Inference follows the same trend but small batches expose fixed transfer costs and offer less traffic to balance. At 256 tokens per GPU, MoX is within 2.1–3.6% of the switch on 16 GPUs and 3.1–6.4% on 32, whereas min-hop is 42–70% slower (2 (b)).

3.1.0.1 Analytical proxy.

We divide the expander’s analytically computed maximum normalized link load (bytes/capacity) by the switch’s one obtained in simulation, and compare it with the ASTRA-sim-reported end-to-end slowdown which simulates the collective on the topology. The close match in 3 validates maximum link load as a proxy for these bandwidth-bound configurations, which is expected since the maximum loaded link determines the total collective completion time.

3.2 Sensitivity to load skew and topology optimizations↩︎

We compare MoX against the demand-adaptive topology tuning approach inspired by MixNet [4] on a 16-node expander and top-8 experts. This approach assumes full knowledge of the traffic matrix: it reconfigures the topology prior to the execution, maximizing the amount of direct traffic thus reducing the bandwidth tax. The tokens are routed via min-hop routing. Note, however, that the number of experts is larger than the node degree, therefore some traffic remains indirect. Last, we combine the topology tuning with the MoX routing approach.

4 presents the results. Across all skews MoX stays within \(1.8\%\) of the switch, significantly outperforming both min-hop routing (\(34\)–\(42\%\) slower) and topology optimization (\(24\)–\(37\%\) slower). MoX’s routing also rescues the optimized topology, cutting its overhead to \(2\)–\(5\%\); even so, combining the two does not improve on MoX over the plain expander.

MoX’s overhead over the switch stays roughly flat across the skew range. The real MoE traffic we recorded for two large-scale models stays well balanced, so MoX remains within \(0.8\%\) of the switch.

3.3 Google TPUv8i Boardfly↩︎

MoX’s applicability goes beyond irregular asymmetric topologies. Here we show that topologies based on more regular graphs are often structured in a way that can create load imbalances when using min-hop routing. As a representative example, Google’s Boardfly topology used in TPUv8i [2] is regular, but doubles the capacity of certain links in order to fully utilize the available bandwidth at each node. Min-hop routing may lead to underutilization of the capacity of these parallel links, resulting in bottlenecks on the remaining links. Boardfly uses 400GB/s links [26].

We resample the DeepSeek trace to a projected 4K-expert model, placing four experts on each of 1K TPUs. Because we do not simulate this full topology end to end, we report the validated maximum-link-load proxy for dispatch.

5 shows that MoX reduces the bottleneck link by up to \(47\%\) (\(33\%\) by relaying tokens and \(14\%\) by weighted routing) for training batch sizes and \(41.7\%\) (\(32.9\%\) by relaying and \(9\%\) by weighted routing) for inference batches.

3.4 Analysis↩︎

We analyze MoX across expander sizes and top-\(k\) values. 6 (a) shows that the multicast tree alone reduces most of the bandwidth overhead of the min-hop routing down to \(8\%\), \(11\%\), and \(31\%\) for 16, 32, and 64 nodes respectively, but MoX improves further by mitigating the load imbalance.

With larger topologies, a higher degree expander and higher values of top-\(k\) achieve much better results (6 (b)). This suggests that one should maintain degree \(d\) that is high enough for the expander topology size \(N\), ensuring that the fraction of nodes reachable with a single relay remains constant, e.g. \(\frac{d^2}{N} = 2\).

4 Discussion↩︎

4.0.0.1 Implementation

MoX requires per-token forwarding rather than today’s aggregated sends. We believe that this can be implemented with low overhead due to the large size of each individual token. Per-token multicast forwarding is accomplished using a destination matrix stored in each packet, allowing it to be handled by either the GPU or NIC. During combine, reduction must be handled by the GPU, which already stores one of the inputs to the reduction. Line rate AllReduce has been demonstrated in GPUs [27], [28].

4.0.0.2 Practicality of high-degree random expander topologies

When scaling up to high node counts, it is essential for each node to have a high degree as shown in 6 (a). Cabling has commonly been considered the main barrier to high-degree random topologies. However, Amazon has recently shown that it has solved this problem in its RNG infrastructure using shuffle boxes [15]. Further, we expect that the effective number of links per NIC will grow as well, due to increasing adoption of multi-plane topologies [29].

Another approach is to consider each NVLink domain as a single routing endpoint, as suggested in [4], using NVLink to share all the NICs in each server. Splitting each of the server’s 8 scale-out links into 8 lanes results in a degree of 64, allowing scaling up to 2,048 servers (16k total GPUs) while maintaining \(\frac{d^2}{N} \ge 2\).

5 Related Work↩︎

5.0.0.1 Multicast Tree and Traffic Engineering

Multicast routing combines NP-hard tree construction  [30] with traffic engineering that minimizes maximum link utilization [31], [32]. Protocols build shortest-path trees  [33], stateless BIER-TE steers static replication paths [16], and controllers adapt routes to group demand [34]. MoE changes too quickly for per-iteration weight optimization.

5.0.0.2 Collectives on Direct-Connect Topologies

Topology-specific collective algorithms provide near-optimal schedules on symmetric direct-connect fabrics [35]–[38], but assume regular, predictable traffic rather than dynamic, skewed MoE demand.

5.0.0.3 Direct-Connect Topologies for ML Training

TopoOpt [39] co-optimizes topology and parallelization. Google’s optically reconfigurable TPU v4 [1] targets structured traffic, whereas TPUv8i adopts the more general Boardfly topology, better suited to dynamic MoE inference [2]. Photonic Rails [3] time-multiplexes links across training phases; MixNet [4] predicts demand to reconfigure a hybrid optical-electrical fabric at runtime.

6 Conclusion↩︎

MoX demonstrates that routing optimizations which are lightweight, computed offline, and load-oblivious provide a significant performance boost for MoE communication on direct-connect topologies, without the need for costly dynamic topology adoption or continuous adjustment of routing weights. We show that MoX significantly reduces the overheads of the MoE phase for both inference and training, achieving performance close to an ideal packet switch, and demonstrates significant potential for optimizing Google’s recent Boardfly direct-connect network.

References↩︎