Contents↩︎

1 Introduction↩︎

AI agents are moving beyond one-shot answers toward using tools, inspecting code and documents, and acting over long trajectories. ReAct formalizes this interaction as a sequence of reasoning, actions, and observations  [1], while LongCat-Flash-Thinking trains agents over long, multi-turn tool trajectories [2]. For these agents, context carries evidence, environment observations, tool outputs, and earlier decisions into the next action.

Long-context inference and post-training use memory differently. An inference server can prefill a prompt, cache the state used for decoding, and discard the forward graph [3]. Post-training must score several responses and backpropagate through them. Group Relative Policy Optimization (GRPO) compares responses that share a prompt through group-relative advantages  [4]. Each response may be short, but its score still depends on the full prompt and its cached state.

Existing techniques reduce important parts of this cost, but they do not by themselves make a fixed-GPU GRPO run fit. Memory-efficient attention reduces the workspace of an attention layer [5]. FlashAttention improves the data movement of exact attention [6]. LoRA reduces the number of trainable parameters [7], while QLoRA also reduces the storage cost of the base model [8]. The prompt graph, response graphs, cached state, and distributed communication still compete for the same device memory.

Large accelerator fabrics can extend sequence length by distributing this work more widely. Ring Attention reports 4.096M-position training for a 7B model on 32 A100 GPUs [9]. DeepSpeed-Ulysses studies one-million-token training while scaling to 256 A100 GPUs [10]. ByteScale reports a 2M LLaMA-7B case on 1,024 GPUs [11], and USP combines ring and all-to-all sequence parallelism [12]. These systems establish the scale-out route. This report asks a complementary question: how far can a GRPO execution path go when the GPU count stays fixed?

LongStraw answers this question by separating the long prompt from the short response computation. It evaluates the shared prompt once without automatic differentiation, stores the information needed to condition later tokens, and then processes one response at a time. Gradients from the response members are accumulated before the optimizer is called. Serial replay increases elapsed time, but it avoids keeping the prompt graph and every response graph live at the same time.

The stored information follows the model architecture. Qwen3.6-27B combines recurrent layers with full-attention layers [13]; LongStraw keeps the recurrent state and context-sharded key/value pages needed by later tokens. GLM-5.2 uses compressed attention state, sparse attention indices, and routed experts [14]. Its implementation moves the prompt state to CPU memory and stages one decoder layer at a time. Figure 1 shows the shared execution schedule and where the two implementations differ.

LongStraw runs inside the training side of MinT [15]. MinT manages model workers, adapter revisions, and the outer policy transaction; LongStraw manages the prompt state and response work inside one long-context transaction. The two systems therefore operate at different levels of the same training stack.

This execution boundary changes what is differentiated. Both implementations detach the stored prompt state from the response backward pass. The Qwen path does not synchronize every key/value-related gradient across context-parallel ranks. The GLM path uses attention keys from each rank’s local prompt shard and skips the usual cross-rank gradient reduction. We therefore report completed execution and local optimizer calls separately from claims about a correct distributed update or learned policy quality.

1.0.0.1 Contributions.

This report makes three contributions:

• It presents a shared-prompt execution design that stores prompt state once and serializes response work under a fixed GPU budget.

• It implements that design for two different model structures: a Qwen hybrid recurrent/attention stack and a GLM compressed-attention/MoE stack.

• It reports the completed paths, timing, and memory measurements together with the distributed operations that remain unverified.

1.1 Report Roadmap↩︎

The report first defines the training dependency and model structures, then describes the Qwen and GLM implementations. The remaining sections present the fixed-budget measurements, explain the main memory and communication costs, and state the limits of the current execution paths.

2 GRPO Training Dependency Graph↩︎

The systems problem begins with the update graph, not with a particular attention kernel. Let a prompt contain \(P\) tokens, group member \(i\) contain \(R_i\) scored response tokens, and the group contain \(G\) responses. For old policy \(\pi_{\mathrm{old}}\), current policy \(\pi_\theta\), and normalized advantage \(A_i\), define \[\rho_{i,t}(\theta)= \exp\!\left( \log\pi_\theta(y_{i,t}\mid x_{1:P},y_{i,<t})- \log\pi_{\mathrm{old}}(y_{i,t}\mid x_{1:P},y_{i,<t}) \right). \label{eq:importance-ratio}\tag{1}\] The clipped policy term is \[\mathcal{L}_{\mathrm{policy}} = -\frac{1}{G}\sum_{i=1}^{G}\frac{1}{R_i}\sum_{t=1}^{R_i} \min\!\left( \rho_{i,t}A_i, \operatorname{clip}(\rho_{i,t},1-\epsilon,1+\epsilon)A_i \right), \label{eq:grpo}\tag{2}\] with a tokenwise reference-policy KL term weighted by \(\beta\). The clipped ratio surrogate follows PPO [16]; group-relative advantages, member normalization, and the reference-policy penalty follow the GRPO objective [4]. This is the member-normalized GRPO term implemented by the audited paths. The contribution of this report is how its conditional log-probabilities are executed when the context length exceeds two million positions under a fixed accelerator allocation without adding devices.

2.1 Branches and Ordering Constraints↩︎

A grouped update has five logically different phases.

1. Prompt capture. Evaluate the shared prompt without autograd and retain the architecture-specific conditional state.

2. Pre-step scoring. Evaluate old-policy and reference-policy response log-probabilities before the corresponding policy backward. Parameters remain fixed through the group. GLM materializes both old score sets first; Qwen performs old/reference scoring and policy replay member by member.

3. Advantage construction. Convert group rewards into the advantages used by Equation 2 .

4. Policy replay. Rebuild one short response graph at a time, backpropagate its loss, and accumulate gradients into the same adapter.

5. Optimizer transaction. Synchronize accumulated gradients, step once after all \(G\) members, and clear gradients.

The ordering is load-bearing. Stepping between group members changes both the importance ratio and the prompt state. Holding all response graphs makes activation memory scale with \(G\), whereas serial replay makes group cardinality primarily a scheduling and time dimension. This is a systems statement, not a claim that \(G\) is statistically irrelevant: \(G\) still defines reward normalization and the GRPO advantage set. Figure 2 contrasts the full-sequence and captured-state graphs.

The acceptance runs use supplied deterministic responses and rewards. They exercise prompt capture, frozen scoring, live policy backward, group accumulation, and optimizer execution, but exclude online generation and reward-model execution. The report therefore covers only the executed training graph.

2.2 Conditional Replay and the Missing Gradient Term↩︎

Let \(z_P(\theta)\) denote the complete model state after processing the prompt. A conventional full-sequence loss differentiates both its explicit response computation and the parameter dependence of the prompt state: \[\nabla_\theta \ell(\theta,z_P(\theta)) = \left.\frac{\partial\ell}{\partial\theta}\right|_{z_P}+ \frac{\partial\ell}{\partial z_P} \frac{\partial z_P}{\partial\theta}. \label{eq:gradient-boundary}\tag{3}\] The implementations in this report store \(\bar z_P=\operatorname{stopgrad}(z_P(\theta))\) and compute only the first term. An exact attention reduction inside response replay can preserve the conditional response operator, but it cannot reconstruct the second term in Equation 3 . “Exact attention” and “full-sequence gradient equivalence” are therefore different claims.

The distinction also defines state validity. A state captured from parameters \(\theta_k\) may condition every response in update \(k\), provided the parameters do not change between branches. After the optimizer produces \(\theta_{k+1}\), that state is stale. A repeated training loop must recapture the prompt or use an explicitly analyzed stale-state approximation. The receipts stop after worker-local optimizer calls and provide no repeated-loop evidence.

2.3 Four Levels of Evidence↩︎

We use four progressively stronger tests. Distributed-update consistency remains separate from full-sequence parity because a correct distributed forward may still leave replicated parameter gradients rank-local.

2.3.0.1 Execution capacity.

Did every requested score, backward, collective, and optimizer event complete on every rank with finite values? This is established by terminal logs and rank-local traces.

2.3.0.2 Response-operator fidelity.

Does distributed replay compute the model-defined conditional response operator? Qwen reaches this level for full-attention layers through a global CP8 merge with BF16 numerator reduction. The GLM fallback does not: sparse selection remains local to each CP shard.

2.3.0.3 Distributed-update consistency.

Are all sharded gradient contributions reduced to the correct parameter owner, and do replicated adapters remain identical after the optimizer call? The Qwen probe all-reduces \(dQ\) but not the local \(dK/dV\) contributions to replicated projection adapters. Its eight AdamW instances step independently. The current GLM resident path calls backward outside the normal Megatron schedule and skips finalize_model_grads; CP-replicated non-expert adapters therefore step from unreduced gradients. Neither path reaches this level.

2.3.0.4 Full-gradient parity.

Do all trainable gradient shards and optimizer deltas match a conventional full-sequence reference? Neither path has completed this test. It requires a shorter context at which both the conventional and replay implementations fit, followed by parameter-by-parameter comparison.

An execution receipt passes when old scores are frozen, every group member produces a live backward, local gradients accumulate, each worker issues exactly one optimizer call, values remain finite, and all ranks terminate. This is a systems result, not evidence for stronger semantic levels.

3 Architecture Anatomy and Bottleneck Sources↩︎

The two models differ along two independent axes. The feed-forward axis is dense versus MoE. It determines parameter residency, token routing, and the shape of activation buffers. The token-mixing axis is GDN/full attention versus MLA/DSA; it determines retained prompt state and response-time collectives. Model-level labels such as “dense”, “MoE”, or “sparse” hide this separation.

3pt

Figure 3 opens both decoder stacks at the level needed by the training runtime. The diagram shows counts and ownership rather than implying that the two models share one layer template.

3.1 Qwen: Dense FFNs with Hybrid Token Mixing↩︎

The inspected Qwen3.6-27B configuration has hidden width 5,120, 64 decoder layers, 48 linear_attention entries, and 16 full_attention entries [13]. The audited runtime instantiates the former with its recurrent GDN module. Gated DeltaNet supplies that module’s gated delta rule [17], while grouped-query attention shares fewer KV heads across a larger set of query heads [18]. Every layer ends in a dense gated feed-forward network with intermediate width 17,408. A gated dense FFN applies the same three matrices to every token in the SwiGLU form [19], \[F_{\mathrm{dense}}(h)= W_2\!\left(\operatorname{SiLU}(W_1h)\odot W_3h\right). \label{eq:dense-ffn}\tag{4}\] There is no token router and no expert all-to-all. Once the long-prompt FFN graph is detached, a response replay only invokes the same dense matrices for the short suffix.

The two token mixers expose different prompt-state scaling. A GDN layer carries a fixed-shape recurrent state across the prompt boundary. A full-attention layer carries key and value pages whose storage grows with \(P\). Full attention also has a global semantic requirement: every response query must attend to pages owned by all context-parallel ranks. Qwen thus places most of the long-context storage problem in 16 layers, while the other 48 layers contribute compact recurrent state.

The reported Qwen implementation uses NF4 QLoRA with 116,727,808 trainable parameters. Quantized base weights reduce persistent model storage, but they do not by themselves remove prompt pages or the response activation graph. The working design is therefore built around physical page compaction and conditional replay, not only parameter quantization [8].

3.2 GLM Attention: MLA, DSA, and IndexShare↩︎

Multi-head latent attention (MLA) compresses per-token KV content into a latent representation [20]. The inspected GLM-5.2 configuration has hidden width 6,144 and 78 decoder layers. Its KV and query latent widths are 512 and 2,048, respectively [14]. Its sparse indexer has 32 heads of dimension 128 and selects at most 2,048 preceding positions for each query [14]. Following the DSA definition [21], an index score can be written as \[I_{t,s}=\sum_{j=1}^{H_I}w^I_{t,j} \operatorname{ReLU}\!\left(\left\langle q^I_{t,j},k^I_s\right\rangle\right), \qquad H_I=32, \label{eq:dsa-index}\tag{5}\] and the sparse response output is \[u_t=\operatorname{Attn}\!\left( q_t,\{c_s:s\in\operatorname{TopK}(I_{t,:},2048)\} \right). \label{eq:dsa-attention}\tag{6}\] The 2,048 selected positions define the operator, not merely its memory layout. Faithful distributed replay requires global selection over the logical context and correct composition of the selected values.

Computing a new sparse index in every layer would repeat similar work. The inspected GLM configuration instead yields 21 index-computing layers and 57 IndexShare layers that consume a selection published by a nearby source layer [14]. This cross-layer reuse resembles the mechanism analyzed by IndexCache [22]. It changes the runtime contract in two ways. First, only the 21 compute layers need durable prompt indexer-key pages. Second, the response forward must maintain a per-forward producer/consumer holder whose index schedule cannot span response branches or parameter versions.

3.3 GLM Feed-Forward Path: Dense then MoE↩︎

The inspected configuration assigns dense FFNs to the first three GLM decoder layers and 256 routed experts with top-8 routing plus one shared expert to the remaining 75 layers [14]. For token \(t\), \[F_{\mathrm{MoE}}(h_t)=F_{\mathrm{shared}}(h_t)+ \sum_{e\in\operatorname{Top8}(g(h_t))}p_{t,e}F_e(h_t). \label{eq:moe-ffn}\tag{7}\] MoE sparsity reduces the number of experts evaluated for one token, but it creates a distributed parameter and data-movement problem. EP ranks hold different experts. The router expands one token into eight expert assignments, dispatches those rows to their owners, evaluates the selected expert paths, and combines the outputs  [23]–[25].

The parameter and activation scales explain why full-sequence GLM autograd is not made cheap by sparsity. An expert has intermediate width 2,048, so its three gated-FFN matrices contain \[3\times6144\times2048=37{,}748{,}736\] weights. The 256 routed experts in one sparse layer contain approximately 9.66 billion weights, even though a token activates only eight of them. At a balanced 65,536-token CP shard, top-8 routing produces \(65{,}536\times8=524{,}288\) expert-token rows. One BF16 buffer with shape \([524{,}288,6144]\) occupies exactly 6 GiB before outputs, permutations, LoRA intermediates, or routing skew. A conventional beyond-2M-context graph can hold several such tensors across layers and backward phases.

Expert parallelism distributes the parameter working set, but it does not change the number of routed token copies. Context parallelism distributes attention state, but it does not place experts. Prior MoE folding work studies heterogeneous mappings across these parallel dimensions  [26]. In our run, folding CP32 and EP32 over the same ranks reduces separate process groups and matches the available 32 GPUs; the two dimensions still solve different ownership and communication problems.

3.4 What the Long Prompt Must and Need Not Retain↩︎

Both models still execute every prompt token through every layer. The next layer needs the output even when the current layer is sparse or recurrent. The capacity gain comes from tensor lifetime: dense FFN intermediates, MoE routes, expert-token permutations, attention scratch, and adapter activations from the prompt are allowed to die immediately. Only the conditional state required by future response tokens survives. For Qwen this is compact GDN and KV state. For GLM it is MLA latent pages, selected indexer-key pages, position/page metadata, and the rules needed to reconstruct IndexShare during each short replay.

4 LongStraw: Long-Context Execution Design↩︎

LongStraw is the execution stack developed in this report. Its design follows one rule: retain a tensor across the prompt boundary only when a later response token depends on it. The rule applies to physical allocations, not just logical tensor views. It also separates state required by the model operator from scratch and activations required only while producing that state.

Throughout this report, budget-constrained means a fixed accelerator count together with each device’s H20 memory limit: eight H20 GPUs for the Qwen path and 32 H20 GPUs for the GLM path. We report elapsed time and allocated GPU memory where the artifacts support them; a whole-transaction GLM peak is not available. Host-memory capacity, network traffic, energy, utilization, and monetary cost are not fully accounted. The resource claim is therefore fixed-device feasibility for the stated transaction, not a lowest-cost or state-of-the-art efficiency claim; those require matched baselines and full-system resource accounting.

4.1 Capture Once, Replay the Suffix↩︎

For update \(k\), the runtime performs the following transaction.

1. Run \(x_{1:P}\) under \(\theta_k\) with autograd disabled. At each layer, save the model-specific prompt state and release transient hidden tensors, attention scratch, FFN activations, and MoE routing buffers.

2. Treat the completed prompt state as read-only. Materialize each response’s old/reference log-probabilities before its policy backward, without updates between group members. GLM freezes both old score sets before replay; Qwen uses a member-serial old/reference/policy schedule.

3. For each group member, rebuild the short current-policy response path under autograd. Reuse the same read-only prompt state, backpropagate the member loss, and immediately release that member’s graph.

4. After all \(G\) backwards, retain the accumulated local gradients and issue one optimizer call per worker. The captured state is now stale with respect to the worker’s updated parameters.

This schedule changes the dominant activation scale from \(P+R\) to \(R\), but prompt compute remains. Phase 1 still sends the full prompt through every decoder layer and retains architecture-specific state: full-attention pages, recurrent state, and DSA latent values plus index keys.

Nothing in the transaction is specialized to \(G=2\) or \(G=8\). For a member-serial group, the leading resource accounting is \[\begin{align} M_{\mathrm{live}} &\approx M_{\mathrm{fixed}}+M_{\mathrm{prompt}}(P) +M_{\mathrm{grad}}+\max_i M_{\mathrm{branch}}(R_i) +M_{\mathrm{score}}\!\left(\sum_i R_i\right), \\ T_{\mathrm{update}} &= T_{\mathrm{prompt}}(P) +\sum_{i=1}^{G}T_{\mathrm{score+replay}}(R_i). \end{align} \label{eq:serial-group-scaling}\tag{8}\] Equation 8 bounds the live policy graph by the largest member rather than by the number of members. It does not make total memory constant in \(G\): input/label objects, rewards, reports, and frozen old/reference scores remain \(O(G)\) or \(O(\sum_i R_i)\). The runners accept a configured list of group members rather than hard-coding \(G=2\) or \(G=8\). Only those two settings have Qwen receipts; larger groups remain unmeasured and must fit the growing buffers and wall-time budget. The \(G=1\) run is only an execution canary because it cannot form a nondegenerate GRPO advantage group.

4.2 State Inventory and Ownership↩︎

Table 2 distinguishes durable prompt state from transient prompt work. The distinction is the basis for both memory accounting and correctness during layerwise response replay.

3pt

Logical sharding is insufficient when a retained tensor is a view into a larger allocation. The allocator cannot release the parent while any view is alive. The Qwen implementation therefore copies every retained page shard into a right-sized physical allocation. GLM applies the same rule on CPU: a restored layer’s page table, local positions, latent pages, and indexer pages must all describe one local shard.

Stored state is immutable within one grouped update. Response branches may append suffix pages or build temporary IndexShare selections, but cannot mutate the shared prefix. Branch-local state is released after scoring or backward so every response observes identical conditioning state.

4.3 Device Placement and Layer Staging↩︎

Qwen retains compact GDN and KV state on GPU because eight-way context parallelism makes the per-rank state fit and the response attention merge reads all shards repeatedly. GLM retains its CP-local MLA and indexer-key pages on CPU. During response replay, it stages the pages needed by one layer, executes the short response layer, and releases or returns the staged copy before advancing. A shared RoPE cache avoids rebuilding position tensors for every layer and branch.

CPU placement trades transfer time for bounded residency. The trade is useful only because replay is layerwise. Copying the complete 78-layer prefix back to GPU would recreate the storage peak. Conversely, staging tensors without preserving their page order and global positions would change the attention operator. Device placement and logical ownership are therefore one contract.

4.4 Whole-Layer Checkpointing↩︎

Activation checkpointing trades retained tensors for recomputation, either at a whole block or at selected operations within it  [27], [28]. For our native MoE replay, however, the required boundary is the complete layer. Attention-only checkpointing does not bound a GLM response graph if the MoE tail retains router outputs, dispatch permutations, expert inputs, LoRA intermediates, and concatenation buffers. The working path checkpoints the complete decoder layer. Forward saves the short layer input and minimal metadata; backward recomputes attention projection, sparse selection, IndexShare publication or consumption, output projection, router decisions, expert dispatch/combine, and the selected dense or expert LoRA paths without retaining the complete layer graph across backward or any prompt activation. Checkpointing is applied to the short response graph, not the full 2,097,152-position prompt. The prompt was already evaluated without autograd. This distinction explains why checkpointing succeeds here while conventional full-sequence checkpointing still exposes large routed-token and attention workspaces during backward recomputation through the complete decoder stack.

4.5 Parallel Layout↩︎

The parallel dimensions distribute different objects. Table 3 lists the layouts used by the acceptance runs.

3pt

Qwen uses CP8 to distribute full-attention pages and recurrent state over eight GPUs. Its dense FFNs do not require expert dispatch. GLM uses TP1/CP32/EP32/ ETP1/PP1. CP32 distributes long attention state. EP32 distributes 256 routed experts, nominally eight per rank. The same 32 ranks participate in both groups, but CP collectives cannot replace EP collectives: response attention needs a cross-context selection or reduction, while MoE needs token dispatch and combine across expert owners.

4.6 Adapter and Optimizer Scope↩︎

The final GLM run uses rank-8 LoRA over the configured attention projections, dense FFNs, routed and shared expert FFNs, and the output head. Base weights, embeddings, normalization parameters, router parameters, and the DSA indexer remain frozen. The Qwen path uses NF4 QLoRA over its configured dense model targets. These are parameter-efficient adapter updates; neither receipt provides evidence of full-parameter model training [7], [8]. Rank allocation, adapter sharing, mixed-precision fine-tuning, and pruning are complementary design axes  [29]–[34]; LongStraw keeps those choices fixed and targets the execution boundary and distributed-state contract examined throughout the audited replay path.

All old-policy values are frozen before policy replay. In the fresh GLM acceptance run, the old snapshot is also the reference for the first update. With \(\epsilon=0.2\) and \(\beta=0.01\), the initial importance ratios are one and measured KL is zero. The run exercises the recorded branch and optimizer-call ordering, but it does not exercise an active clip boundary or a nonzero first-step KL penalty.

5 Qwen: Dense Hybrid Replay within an Eight-H20 Budget↩︎

The Qwen path is the cleaner of the two architecture cases because it has no expert router, no expert-parallel token exchange, and no sparse index whose meaning changes under context sharding. It is nevertheless not a conventional data-parallel training job. The implementation replicates the dense model and response query computation across eight ranks while distributing the long key/value sequence of each full-attention layer. This section opens that contract at the tensor level. It documents a verified global conditional forward operator inside the fixed eight-H20 execution envelope; model-parallel gradient composition remains open.

5.1 Hybrid Layer Anatomy and the Prompt Boundary↩︎

The audited model snapshot contains 64 decoder layers at hidden width 5,120, with a repeating pattern of three linear_attention entries and one full_attention entry [13]. The audited runtime maps the former to its gated-delta-network (GDN) module, giving 48 GDN layers and 16 full-attention layers. The recurrent mechanism follows Gated DeltaNet  [17]. In the pinned snapshot, each full-attention layer has 24 query heads, four KV heads, head dimension 256, an output gate, and a partial rotary factor of 0.25 [13]. The 24-to-4 query/KV arrangement follows grouped-query attention [18], while its rotary position mechanism follows RoPE [35]. Every layer uses the same dense gated FFN shape, with intermediate width 17,408. Ignoring biases, one FFN therefore contains \[3\times 5{,}120\times 17{,}408 = 267{,}386{,}880 \label{eq:qwen-dense-ffn-weights}\tag{9}\] base weights. Every token activates all three projections; there is no conditional expert capacity to distribute.

The two token mixers leave different state at the end of a no-grad prompt. For a GDN layer, the boundary consists of a recurrent matrix and the final three-token convolution tail. Its shape is independent of prompt length. For a full-attention layer, the boundary is every prompt key and value, so storage is linear in prompt length. Prompt capture evaluates the complete dense stack, including every FFN, but releases prompt hidden states, FFN intermediates, and temporary mixer work after each chunk. Only 48 compact GDN boundaries and 16 full-attention KV page sets survive. Thus pages serve the 16 quadratic-history layers, while recurrence removes length-dependent state from the remaining 48 GDN layers.

The NF4 QLoRA path uses rank 16, scaling 32, learning rate \(2\times10^{-4}\), and zero weight decay. LoRA targets full-attention, GDN, and dense-FFN projections, giving 116,727,808 trainable parameters. NF4 reduces persistent base-weight storage, but it does not reduce KV length or response activation lifetime [8]. Those two terms are handled by physical context parallelism and blockwise replay.

5.2 From Logical Shards to Physical Pages↩︎

Let page size be \(S=64\), global page index be \(p\), and the CP world size be \(C=8\). Page ownership is block-cyclic, \[\operatorname{owner}(p)=p\bmod C. \label{eq:qwen-page-owner}\tag{10}\] Each rank retains only its owned pages but keeps the global logical KV length. Thus RoPE positions, query start indices, and causal masks use original sequence coordinates, not compacted local ones.

This distinction was initially only logical. A page obtained by slicing a 4,096-token prefix chunk could be contiguous as a tensor while still retaining the storage of the complete parent chunk. Discarding the unowned slice did not release that parent allocation. The manager instead allocates right-sized page tensors and copies owned slices. PagedAttention manages serving KV through logical-to-physical block tables over fixed-size physical blocks  [3]. At our training boundary, copying owned slices makes allocator ownership agree with page-table ownership, so releasing a logical page also releases its physical storage.

The arithmetic beyond two million context positions is explicit. The reported sequence has 2,088,960 prompt tokens followed by 8,192 response-input tokens, for an exact context length of 2,097,152. The prompt contains \[N_{\mathrm{page}}=\frac{2{,}088{,}960}{64}=32{,}640, \qquad N_{\mathrm{page/rank}}=\frac{32{,}640}{8}=4{,}080. \label{eq:qwen-page-count}\tag{11}\] For BF16 K and V with four KV heads and head dimension 256, the raw prompt KV payload retained by one rank across the 16 full-attention layers is \[\begin{align} B_{\mathrm{KV/rank}} &=16\times4{,}080\times 2_{\{K,V\}} \times64\times4\times256\times2\;\text{bytes} \\ &=17{,}112{,}760{,}320\;\text{bytes} \approx 15.94\;\text{GiB}. \label{eq:qwen-kv-bytes} \end{align}\tag{12}\] The measured allocator footprint is larger because it includes quantized weights, adapters, recurrent state, page metadata, response pages, and temporary kernels. Equation 12 exposes the ownership slope: each additional prompt page resides on one CP rank per full-attention layer, not on all eight ranks.

5.3 Global Full-Attention Forward Composition↩︎

For response query \(t\), let \(\mathcal{K}_r\) be the keys owned by rank \(r\) and let \(s_{t,j}=q_t^\top k_j/\sqrt{d}\). A local paged-attention kernel returns a normalized local output and its row log-normalizer, \[\ell_{r,t}=\log\!\sum_{j\in\mathcal{K}_r}e^{s_{t,j}}, \qquad o_{r,t}=\frac{\sum_{j\in\mathcal{K}_r}e^{s_{t,j}}v_j}{e^{\ell_{r,t}}}. \label{eq:qwen-local-attn}\tag{13}\] The global output is reconstructed without moving the KV pages.

\[\begin{align} m_t &= \max_r \ell_{r,t}, \\ a_t &= \sum_r e^{\ell_{r,t}-m_t}, \\ n_t &= \sum_r e^{\ell_{r,t}-m_t}o_{r,t}. \label{eq:qwen-lse-parts} \end{align}\tag{14}\] Then \[o_t=\frac{n_t}{a_t}, \qquad \ell_t=m_t+\log a_t. \label{eq:qwen-global-attn}\tag{15}\] This requires one MAX all-reduce and two SUM all-reduces per composed output. It is the same stable log-sum-exp identity used by exact tiled attention  [5], [6]. It reconstructs the dense conditional response-attention operator from disjoint KV partitions.

Production reduces \(n_t\) in BF16 to lower communication payload. The maximum, denominator, and global LSE remain FP32. Here “exact” means exact partition composition of the specified dense operator under that finite-precision reduction, not bitwise equality with unsharded FP32 execution.

5.4 Four-Block Response Replay↩︎

The shared prompt is captured once in 510 chunks of 4,096 tokens. A response branch is then split into four 2,048-token blocks. The branch first runs a no-grad suffix pass. At each block boundary it clones the input GDN states and appends that block’s KV pages. The policy pass traverses the four blocks in reverse. For one block it restores the corresponding GDN input state with gradient tracking, recomputes all 64 decoder layers under whole-layer checkpointing, forms causal selected-token log-probabilities, runs backward, propagates the recurrent/conv-state gradient to the preceding block, and pops the temporary KV update. Only one 2,048-token response graph is live at a time, so block count increases replay work without retaining all block graphs together across the complete suffix on each device.

For each group member the observed ordering is old scoring, reference scoring, policy suffix forward, and policy reverse. Members are serialized after the shared prompt. Their parameter gradients accumulate until one optimizer call is issued after the last member. This schedule explains why, for this workload, increasing group size mainly increases time rather than peak memory.

There are three workload qualifications. First, the inputs are synthetic random tokens and rewards are deterministic functions of group index. Second, the old and reference scores are produced by the same current model before the step; the run uses \(\beta=0\), so it does not exercise an independent reference policy or an active KL term. The live policy expression is the unclipped ratio term. Because old and current scores coincide at the first step, the ratio is one and clipping would be inactive even if present. Third, 8,192 is the number of response-input tokens. The causal selector scores labels from response index one onward, yielding at most 8,191 suffix targets per branch under the audited implementation. The terminal logs record the response length but do not serialize this scored-token count.

5.5 The Fixed Eight-H20 Execution Envelope↩︎

Both runs use the fixed eight-NVIDIA-H20 budget, CP8, page size 64, a 4,096-token prefix chunk, a 2,048-token response block, and 512-token FFN microblocks during backward. CUDA peak counters are reset after model, adapter, optimizer, and input construction. Reported memory is decimal GB from max_memory_allocated, not reserved memory or process memory from nvidia-smi. Timings exclude model loading.

5.5.0.1 Group size two.

The reported rank records 5,198.780 seconds end to end and a 97.503 GB peak. The shared prefix consumes 4,656.225 seconds, about 89.6% of the reported time. Across the eight terminal records, total time ranges from 5,196.750 to 5,200.684 seconds; local optimizer calls take 0.155–0.200 seconds.

5.5.0.2 Group size eight.

The reported run completes in 6,785.225 seconds at a 97.711 GB peak. Prefix capture takes 4,653.420 seconds. Group zero completes at 4,931.993 seconds; each of the seven additional serialized members adds a median 264.739 seconds. After the first member, a typical branch spends about 32.0 seconds in old scoring, 32.0 seconds in reference scoring, 31.2 seconds in suffix forward, and 169.1–169.6 seconds in reverse replay. Terminal times across ranks range from 6,783.943 to 6,785.225 seconds, while the corresponding local optimizer-call timers range from 0.163 to 0.186 seconds.

The increase from \(G=2\) to \(G=8\) is therefore 1,586.445 seconds but only 0.208 GB (0.213%) at the reported peak. Removing the shared prefix, the post-prefix cost per member is 271.278 versus 266.476 seconds. Amortizing the prefix reduces mean wall time per supplied response from 2,599.390 to 848.153 seconds. These derived values come from two single runs, not a fitted scaling law. They establish that the requested scoring, four-block policy backwards, gradient materialization, and terminal optimizer calls fit at a context length of 2,097,152. They do not establish throughput-optimal group parallelism, online rollout, policy improvement, or repeated fresh-prefix training.

5.6 Why the Receipt Is Not Yet a Coherent CP8 Update↩︎

The forward collective is complete, but model-parallel backward is not. Each attention rank computes a local query-gradient contribution \(dq_r\) and local gradients for its K/V tokens. The correct query gradient is \[dq=\sum_{r=0}^{7}dq_r, \label{eq:qwen-dq-reduce}\tag{16}\] and the custom backward performs this all-reduce. In contrast, its \(dK_r\) and \(dV_r\) remain rank local. That is valid for sharded KV storage, but the K/V projection LoRA weights are replicated. Their correct parameter contributions require composition across the disjoint token owners, \[\nabla W_K=\sum_r \nabla W_K^{(r)}, \qquad \nabla W_V=\sum_r \nabla W_V^{(r)}, \label{eq:qwen-kv-param-reduce}\tag{17}\] with K/V hidden-state contributions composed before differentiating earlier replicated layers.

The audited runner initializes only NCCL: it has no DDP wrapper, parameter-gradient reducer, or selective model-parallel composition required by Equation 17 . Each rank owns an AdamW instance  [36] and steps locally. All-reducing every completed gradient would still be wrong: query contributions are already global but K/V contributions are not, so the two paths require reductions at their distinct parameter-ownership boundaries for replicated adapters.

No terminal receipt records a gradient norm, parameter delta, post-step adapter hash, or next-forward replica comparison. The supported statement is therefore narrower than “a coherent distributed update”: the implementation executes global full-attention response forwards, response-shaped backward graphs, and eight terminal optimizer calls within the fixed eight-H20 envelope at 2,097,152 context positions. We call this an execution/update-shaped capacity receipt. Closing the Qwen path requires model-parallel K/V and hidden-gradient composition, followed by per-parameter gradient parity and post-step replica-consistency tests. The detached prompt-state term in Equation 3 remains a separate limitation even after that synchronization is repaired and numerically validated against a tractable reference.

6 GLM: Paged MLA/DSA Replay within a 32-H20 MoE Budget↩︎

The GLM path is architecturally more demanding. Its long-range token mixer is neither dense attention nor a prompt-length-independent recurrence. Each decoder layer combines multi-head latent attention (MLA) with a dynamic sparse-attention (DSA) index, while most feed-forward blocks are routed MoE layers. Consequently, a useful prompt boundary must preserve two attention representations, reproduce the cross-layer IndexShare schedule, and re-enter the expert-parallel tail under autograd. Traversing 78 layers is therefore necessary but does not prove a faithful global full-context operator or coherent distributed update; we state the implemented path and missing tensor-level collectives separately.

6.1 Layer Anatomy: MLA/DSA Before Dense or Routed FFNs↩︎

The audited configuration has hidden width \(H=6{,}144\) and 78 decoder layers. Attention uses 64 MLA query heads, query and KV latent ranks 2,048 and 512, and a 64-dimensional positional channel. The absorbed key retained by the runtime therefore has width \(512+64=576\). The DSA indexer uses 32 heads of dimension 128 and selects 2,048 key positions per response query. These dimensions follow the released GLM architecture and the inspected runtime configuration artifact [14].

Not every layer recomputes the index. With index frequency four and skip offset three, the zero-based index-computing set is \[\mathcal{I}_{\mathrm{compute}} =\{0,1,2\}\cup\{6,10,14,\ldots,74\}, \qquad |\mathcal{I}_{\mathrm{compute}}|=21. \label{eq:glm-index-schedule}\tag{18}\] The other 57 layers are IndexShare consumers. A compute layer publishes its top-\(k\) tensor to a carrier scoped to one decoder forward; consumers reuse it instead of retaining prompt-length index state. This implements cross-layer index reuse, not a per-layer cache [22].

The FFN schedule is similarly nonuniform. Layers 0–2 are dense gated FFNs. The remaining 75 layers have 256 routed experts, top-8 routing, and one shared expert. With intermediate width 2,048, one routed expert contains, ignoring biases, \[B_{\mathrm{expert}} =3\times6{,}144\times2{,}048 =37{,}748{,}736 \label{eq:glm-expert-weights}\tag{19}\] base weights. A sparse layer contains approximately 9.66 billion routed expert weights, but one token evaluates eight experts, or about 302 million routed weights, plus the shared branch. MoE reduces activated parameter count; it does not remove router, permutation, all-to-all, expert-input, and combine tensors from the layer execution [20], [24], [25], [37].

The final run uses rank-8 LoRA on eight target categories: the query and KV down/up projections, the attention output projection, both FFN projections, and the output head. The FFN patterns match dense, routed-expert, and shared- expert modules. Base weights, embeddings, normalization parameters, DSA indexer projections, and router parameters remain frozen. The router’s expert_bias is a non-parameter state buffer; its transition is not recorded in the final receipt.

6.2 TP1/CP32/EP32 Assigns Two Different Kinds of Ownership↩︎

The acceptance topology is TP1/CP32/EP32/ETP1/PP1 on 32 H20 GPUs. TP1 leaves each attention and dense projection structurally intact. CP32 distributes the prompt token axis and therefore the retained MLA/DSA state. EP32 distributes the 256 routed experts, nominally eight experts per rank. CP and EP contain the same workers in this run, but they are not interchangeable dimensions: CP answers which rank owns a prompt position, whereas EP answers which rank owns an expert parameter shard [26].

The distinction appears directly in the data movement. During prefix capture, one rank processes 65,536 prompt tokens. Top-8 routing expands that shard to \(65{,}536\times8=524{,}288\) routed rows before load skew. A single balanced BF16 hidden buffer of shape \([524{,}288,6{,}144]\) occupies \[524{,}288\times6{,}144\times2 =6{,}442{,}450{,}944\;\text{bytes}=6\;\text{GiB}. \label{eq:glm-routed-buffer}\tag{20}\] This is a derived lower-level buffer size, not a measured whole-layer peak. The live MoE path also needs dispatch metadata, output storage, inverse permutations, shared-expert work, and LoRA intermediates. They are released during capture because autograd is disabled; a conventional beyond-2M graph cannot. Successive full-sequence runs therefore moved from DSA scratch OOMs to expert-LoRA and expert-output OOMs.

6.3 Zigzag Context Pages within the 32-H20 Route↩︎

Let the prompt length be \(P=2{,}097{,}152\), the physical page size be \(S=64\), and the CP size be \(C=32\). Megatron’s context-parallel layout first partitions the global sequence into \(2C=64\) contiguous chunks. Each chunk contains \[N_{\mathrm{page}}=\frac{P}{S}=32{,}768\;\text{pages}, \qquad N_{\mathrm{chunk}}=\frac{N_{\mathrm{page}}}{2C}=512\;\text{pages}. \label{eq:glm-page-count}\tag{21}\] Rank \(r\in\{0,\ldots,31\}\) owns chunk \(r\) and its mirrored chunk \(63-r\). Its ordered global page set is therefore \[\mathcal{P}_r= \{512r,\ldots,512r+511\} \mathbin{\Vert} \{512(63-r),\ldots,512(63-r)+511\}, \label{eq:glm-tr-zigzag-pages}\tag{22}\] where \(\Vert\) denotes concatenation in the local tensor order. Every rank stores 1,024 pages, or 65,536 tokens. For example, rank 0 stores pages 0–511 followed by 32,256–32,767; rank 31 stores 15,872–16,383 followed by 16,384–16,895. The final log records all 32 page samples and agrees with this formula.

The page manager copies into right-sized CPU pages, not views of larger GPU capture tensors. PagedAttention manages serving KV through fixed-size physical blocks and a logical-to-physical block table [3]. In our training page manager, memory is bounded only when allocator ownership matches page-table ownership. Global page IDs stay attached to local tensor order, so DSA masks and selections retain their original global coordinates throughout capture and replay.

6.4 Stored Tensor Contract and Derived Residency↩︎

Every layer stores one absorbed MLA page component. Its runtime key is mla_latent_kv_pages. The 21 index-computing layers additionally store DSA index-key pages under dsa_indexer_key_pages; an IndexShare consumer stores no duplicate index key. Earlier live shape diagnostics record both components as BF16 and the final page contract follows from those inspected tensors: \[\begin{align} \text{MLA page} &: [1,64,1,576], &\text{local materialization} &: [1,65{,}536,1,576],\\ \text{DSA key page} &: [1,64,128], &\text{local materialization} &: [1,65{,}536,128]. \label{eq:glm-page-shapes} \end{align}\tag{23}\] The final manifest records component names, page counts, and CPU placement, but omits component dtype and shape. The shapes above are contract-derived and cross-checked against the earlier live trace rather than measurements serialized directly in the rank-complete final manifest.

The corresponding retained-state arithmetic is exact once that contract is assumed. Per rank, one layer’s MLA pages occupy \[1{,}024\times64\times576\times2=72\;\text{MiB},\] and one compute layer’s index pages occupy \[1{,}024\times64\times128\times2=16\;\text{MiB}.\] Thus the CPU-resident prompt state is \[\begin{align} B_{\mathrm{CPU/rank}} &=78\times72\;\text{MiB}+21\times16\;\text{MiB}\\ &=5{,}952\;\text{MiB}=5.8125\;\text{GiB}, \label{eq:glm-prefix-bytes} \end{align}\tag{24}\] or 186 GiB across 32 ranks. Layerwise staging uses 72 MiB for an IndexShare layer and 88 MiB for an index-computing layer. These payloads exclude response activations and kernel workspaces, so they are not whole-step peaks. The final log reads the retained CUDA peak after prefix capture, before policy replay.

6.5 One Replayed Layer: State, Index, Attention, and MoE↩︎

The response path consumes two scored positions; its decoder input has shape \([2,1,6{,}144]\). The runner materializes that rank’s 65,536 prompt positions, recomputes the response-side projections, and forms a 65,538-position local attention problem. An index-computing layer evaluates the response queries against its local DSA key and publishes indices with shape \([1,2,2{,}048]\). An IndexShare layer consumes the previously published indices. Both paths run the runtime’s unfused absorbed sparse attention, the live output projection and bias/dropout/add path, and the dense-or-MoE _forward_mlp tail.

For a routed layer, each local response row is expanded to eight assignments, permuted by expert owner, exchanged through the EP all-to-all, evaluated by the owner’s expert FC1/FC2 LoRA path, and inverse-combined with router probabilities. The shared expert branch is evaluated in parallel and added to the routed result. Thus the final trace is not an attention-only or synthetic FFN surrogate: all 75 MoE tails execute under the native EP autograd graph. What the trace does not retain is a final-run router histogram, per-expert row count, all-to-all payload, or communication timing.

6.6 Whole-Layer Checkpointing and the Recorded Backward↩︎

Checkpoint placement controls which intermediates survive and which operators are recomputed [27], [28]. Attention-only checkpointing left the routed tail live, so final policy checkpoints whole decoder layers reentrantly with RNG preservation. Forward retains only the short \([2,1,6{,}144]\) input and metadata. Reverse re-enters layers 77 through 0, restages CPU pages, recomputes sparse attention plus the dense/MoE tail, and releases workspace; saved sparse-attention tensors may be offloaded to CPU.

The trace shapes make this lifecycle observable. A representative policy trace records:

• decoder input and every layer output: \([2,1,6{,}144]\) BF16;

• local top-\(k\): \([1,2,2{,}048]\), with 65,536 prefix positions;

• logits: \([1,2,154{,}880]\), followed by a BF16 logits-gradient event;

• layer and attention-projection gradients: \([2,1,6{,}144]\) BF16;

• sparse-attention output gradients: \([2,1,16{,}384]\) BF16.

Policy and old-policy traces contain 396 and 160 events, respectively. Two members, two phases, and 32 ranks produce 128 files with 35,584 JSONL events. Each policy rank records 78 checkpointed layer ends and 78 each of layer, attention-projection, and sparse-attention backwards. This strongly shows the architecture-shaped path completed twice, but contains no parameter-gradient tensors or cross-rank reductions.

6.7 Why the DSA Operator in the Accepted Receipt Is Local, Not Full Context↩︎

For response query \(q_t\), rank \(r\) in the accepted receipt computes index scores only against its page set \(\mathcal{P}_r\), then selects \[I_{r,t}=\operatorname{TopK}_{j\in\mathcal{P}_r} \left[\sum_{h=1}^{32}w_{t,h}\, \phi\!\left(q_{t,h}^{\top}k_j\right)\right], \qquad |I_{r,t}|=2{,}048, \label{eq:glm-local-dsa}\tag{25}\] with causal masking in global page coordinates. Selected values yield rank-local hidden states; CP ranks merge neither candidates nor outputs. Disjoint prompt ownership makes \(I_{r,t}\) rank-specific. Each layer therefore realizes local top-2,048 over 65,536 tokens, not model-global top-2,048 over the full 2,097,152-position prompt.

A faithful distributed operator first needs sufficient local candidates with global scores, positions, and owner IDs; a deterministic cross-rank merge must then compute \[I_t=\operatorname{TopK}_{j\in\bigcup_r\mathcal{P}_r}s_{t,j}, \qquad |I_t|=2{,}048. \label{eq:glm-global-dsa}\tag{26}\] The selected K/V rows must be exchanged from their owners, or their attention numerators and normalizers must be composed with an equivalent distributed operator. Finally, all ranks that continue the replicated decoder must agree on the composed response hidden state. Candidate merge, selected-value exchange, and output composition are all absent from the final run. This is the primary forward-fidelity gap.

6.8 Gradient Finalization Audit: A Terminal Call Is Not an Update↩︎

The optimizer audit exposes a second, independent gap. LoRA transformation is applied before Megatron DDP construction, so trainable adapter parameters are registered in DDP gradient buffers. The inspected DDP configuration has overlap_grad_reduce=false. Its parameter hooks therefore add each local autograd gradient into param.main_grad, but they do not launch an all-reduce or reduce-scatter during backward. In the conventional Megatron schedule, the post-backward finalize_model_grads call invokes finish_grad_sync on every model chunk. This is the required DP\(\times\)CP gradient reduction for non-expert replicas.

The resident path does not call that schedule. It runs two local loss.backward() calls and then invokes engine.optimizer_step() directly. The concrete engine method calls the distributed optimizer’s step() without first finalizing model gradients. The distributed optimizer then copies the shard it owns from the unreduced full gradient buffer, updates that parameter shard, and all-gathers the updated parameter shards. In symbols, the required non-expert gradient is \[g^{\star}=\sum_{r=0}^{31}g_r, \qquad \text{but the observed path supplies shard owner }r(s) \text{ with }\left.g_{r(s)}\right|_s. \label{eq:glm-missing-grad-finalize}\tag{27}\] The subsequent parameter all-gather can make complete parameter replicas look mutually consistent even though different parameter shards were updated from different local objectives. Parameter equality after all-gather would not repair the missing gradient sum. In this run the local gradients are expected to differ because each rank’s DSA forward sees a different 65,536-token shard.

The affected class is the CP-replicated, non-expert LoRA state: attention projections, the three dense FFNs, the output head, and any other adapter placed in the non-expert DP\(\times\)CP buffer. Routed experts require a different interpretation. With world size 32, EP32, ETP1, and PP1, expert data-parallel size is one. Routed-expert adapter weights are marked allreduce=false, are uniquely owned, and receive token contributions through the differentiable EP dispatch/combine path; no replica average is required for those unique expert shards. Shared experts have an additional PEFT EP gradient hook in the inspected runtime, but the final trace does not record per-module hook coverage or hashes, so their synchronization is not promoted to a run-level claim. The router expert_bias state is also unrecorded.

The final manifest contains no CP finalization event, per-rank parameter-gradient count or norm, optimizer-state hash, pre/post adapter hash, parameter delta, post-all-gather checksum, or next-forward equality test. Its terminal evidence must therefore be read literally: all 32 workers completed two architecture- shaped local backwards and invoked the distributed optimizer once per worker. It does not establish a correct CP32-reduced GRPO gradient or a coherent full-context GLM update.

6.8.0.1 Receipt classification.

The strongest supported description is a fixed-budget architecture-shaped, rank-complete resident replay/backward capacity receipt at 2,097,152 positions. Converting it into a model-faithful distributed training result requires, in order, global DSA candidate/output composition, restoration of finalize_model_grads, per-parameter gradient and optimizer-delta parity against a short-context conventional reference, and post-step replica checks. Only then can online rollout, reward execution, repeated updates, and quality evaluation answer the separate question of whether the method trains a useful policy under the stated budget and context.

7 Making the GLM GRPO Path Fit a Fixed 32-H20 Budget↩︎

The final grouped run is the endpoint of a budget-constrained progression. We held the 32-H20 allocation fixed at TP1, CP32, EP32, ETP1, and PP1 and moved one limiting resource at a time. Each gate isolated one dependency class: full-graph memory, prefix-state capacity, differentiable replay, cross-layer architecture state, device placement, or group ordering. This section reconstructs the progression as a systems argument. Table 4 summarizes the gates; Figure 9 tracks the limiting resource across graph boundaries.

3pt

7.1 Stage I: Full-Graph Bottleneck Localization↩︎

A conventional full-sequence LoRA path passed at 32K. Extending the same graph to a 2,097,152-position prompt did not reveal one dominant allocation that could be optimized in isolation. Reducing or bypassing one peak moved the OOM to the next part of the layer. The observed sequence was DSA attention scratch, expert-LoRA scale/add work, and finally MoE output concatenation near the H20 memory limit.

This movement is consistent with the GLM layer anatomy. DSA avoids dense core attention over every key, but its indexer must still process the long context. MoE evaluates a subset of experts per token, but full-sequence backward retains routing, permutations, selected expert inputs, and adapter intermediates. At a 65,536-token shard, one fully expanded routed hidden buffer can already occupy 6 GiB, as derived in Section 3. Optimizing sparse attention alone therefore cannot bring the complete beyond-2M autograd graph within the per-rank H20 memory budget for this workload.

The conclusion from this stage was structural: the long prompt could not remain inside the differentiable graph. The next tests separated prompt-state capacity from response replay.

7.2 Stage II: Prefix-State Scaling↩︎

The prefix-only path evaluated all 78 layers without autograd and retained MLA latent pages plus DSA indexer-key pages. It was scaled through 128K, 256K, 512K, 1M, and 2,097,152. The final pass established that the full decoder could process the prompt, CP-sharded pages could cover it, and prompt MoE work could be released per layer without retaining a backward graph.

Prefix capture did not establish trainability: it omitted state restoration, response logits, IndexShare lifetime, expert routing under autograd, and an optimizer event. A scale-only prefill is not a training result.

7.3 Stage III: Single-Layer Differentiable Replay↩︎

Layer-0 replay provided the first small differentiable slice. The runtime restored one layer’s prompt pages, appended a short response, produced a live loss, and ran backward. Early 32K and 128K gates checked the tensor contract. At 1M, GPU-resident state and MoE work became costly. CPU-resident pages and chunked staging then enabled layer-0 backward plus optimizer-call canaries at 1M and 2,097,152.

This stage separated two questions that are often conflated: whether the prompt state fits, and whether a response query can consume it under autograd. The layer-0 result answered both for one layer, but it could not exercise cross-layer IndexShare or accumulate activation lifetime through the 78-layer stack.

7.4 Stage IV: All-Layer Architecture Closure↩︎

Short 32K and 64K all-layer runs exposed bugs invisible to prefix-only and layer-0 tests.

7.4.0.1 IndexShare lifetime.

An index-computing layer must publish a top-k selection to a per-forward holder, and dependent layers must consume the matching selection. The holder cannot be global across response branches, and checkpoint recomputation must reproduce the same producer/consumer order.

7.4.0.2 Resident DSA call contract.

The stock non-packed DSA path expected query and key layouts compatible with its CP all-gather guard. A short resident query over long saved pages did not match that contract. The acceptance runner used the runtime’s unfused absorbed MLA sparse fallback over materialized local pages. This fixed execution but introduced the CP-local semantic boundary described in Section 6.

7.4.0.3 Views and in-place operations.

Restored-page views tolerated in no-grad capture became invalid under autograd. Explicit ownership and mutation removal closed the layer tail.

7.4.0.4 Activation lifetime.

Checkpointing only attention retained MoE routing and expert activations. A complete-layer boundary made saved activations scale with response length.

The resulting 32K and 64K gates executed all 78 attention and FFN tails, including 21 index producers, 57 IndexShare consumers, three dense FFNs, 75 MoE FFNs, and their backward paths.

7.5 Stage V: Parallel Ownership and CPU Pages↩︎

The acceptance topology uses TP1/CP32/EP32/ETP1/PP1. CP32 assigns a portion of the long prompt pages to every rank. EP32 assigns the 256 routed experts, nominally eight experts per rank. CPU page storage and one-layer staging bound attention-state residency; complete-layer checkpointing bounds response activations. A shared RoPE cache avoids repeated layer-local position allocation.

This topology is a budget choice, not a generic recipe for every MoE model. It reuses the same 32 ranks for CP and EP rather than scaling a second device group; attention state uses all ranks for CP, and the routed expert set is large enough to use those ranks for EP. Both communication patterns still execute.

7.6 Stage VI: A 2.097M Single-Member Canary under the Fixed Budget↩︎

Before introducing group accumulation, a \(G=1\) canary ran the 2,097,152-position prompt, all 78 response layers, one backward, and one optimizer call per worker under the same 32-H20 allocation. This gate checked that state restoration, checkpoint recomputation, local gradient materialization, and the terminal optimizer path could coexist within the target per-rank H20 storage envelope.

The canary was not a valid GRPO group and did not test group ordering: one member needs neither frozen old scores nor cross-response gradient accumulation. The grouped acceptance run started in a fresh process.

7.7 Stage VII: Fresh Grouped Execution↩︎

The final run uses one 2,097,152-position prompt and two deterministic responses, each with three input and two scored tokens. Rewards \([0,1]\) produce advantages \([-1,1]\). The run captures the prompt, materializes both old-policy score sets, then executes two serial 78-layer policy backwards into rank-8 LoRA adapters. After the second backward, workers make one optimizer call and one gradient clear each; all 32 ranks terminate.

PyTorch allocation during prefix capture is not uniform across ranks. The capture-window max_memory_allocated ranges from 112.571 to 145.148 GB per rank. A direct read of dsw-6601 reports NVIDIA H20-3e devices with 143,771 MiB (140.401 GiB, or 150.755 GB in decimal) each. The 32-GPU GLM log does not include its own device query; if those workers expose the same per-device total, the capture readings correspond to 74.7–96.3% of device total. This 32.577 GB spread reveals substantial rank nonuniformity and motivates testing better placement and load balance. A historical Qwen log also reports a PyTorch allocator capacity of 139.73 GiB (about 150.0 GB decimal); that process-level limit is not the H20 hardware specification. Because the GLM counter is read before response replay, it does not quantify whole-transaction headroom or establish that any specific context beyond 2M fits. The completed 2M run is an operating point, not an OOM-derived memory frontier; it establishes grouped execution within the fixed budget.

This stage adds no semantic guarantees beyond earlier stages: the DSA response operator is still CP-local, Megatron gradient finalization is skipped for CP-replicated non-expert adapters, the prompt state remains detached, and the supplied responses do not exercise online rollout or reward computation. Those properties are reported beside the terminal evidence in the execution audit that follows on every participating rank.

8 Execution Receipts and Trace Evidence↩︎

The acceptance evidence is organized around terminal events and semantic scope. A row enters the main table only when every participating worker reaches the requested scoring, backward, and optimizer-call boundary. The table does not promote those events to a coherent distributed update.

2.7pt

\(^*\)The Qwen scorer drops the first response label, so 8,192 response input tokens yield at most 8,191 scored positions; the run log does not record the realized count. Peaks are max_memory_allocated/\(10^9\) over the measured run. \(^\dagger\)GLM prompt capture plus supplied-group execution; model creation adds 170.211 s. No valid GLM whole-run peak was recorded. The recorded capture-window peak allocation spans 112.571–145.148 GB per rank but is read before response replay, so it remains a diagnostic rather than the GLM Peak entry. All rows condition on a detached prompt state; distributed-update consistency is unverified. The observed \(G\)

values are validation anchors, not loop limits: both paths are driven by configured member lists rather than hard-coded to those values, while payload and score storage still grows with group size. Only the listed settings have terminal receipts.

8.1 Runtime Ownership and Pinned Rerun Stack↩︎

In the current rerun protocol, MinT Runtime supplies the model/session control plane and managed Megatron trainer groups [15]. The pinned local stack implements its asynchronous request lifecycle with Ray-resident Megatron workers. LongStraw is an opt-in long-context execution extension on that substrate. It adds GLM prefix capture, CPU-resident MLA/DSA state, CP/EP ownership, response-only replay, serial GRPO accumulation, and source-bound validation and receipt tooling. The 2M runner creates the model through MinT and then invokes a LongStraw-installed method on the resident Megatron actor rather than the stock MinT forward/backward path. Thus MinT owns the managed transaction and worker lifecycle; LongStraw owns the rerun’s long-context execution path.

The accepted historical 2M receipt predates this pinned stack. The current tree implements global cross-CP DSA composition and restores Megatron gradient finalization. Only a 32K forward canary covers those paths; no fresh source-bound 2M rerun or full gradient-parity receipt upgrades the historical evidence.

Table 6 pins the source identities declared by the current rerun protocol. These declarations are prospective: they do not retroactively establish that the historical 2M receipt was produced from this exact stack.

3pt

8.2 Qwen Receipts↩︎

The Qwen \(G=2\) and \(G=8\) probes share one 2,088,960-position prompt and use 8,192 response-input tokens per member, producing a context length of exactly 2,097,152. Both finish on eight H20 workers. The reported whole-run allocated-memory peaks are 97.503 and 97.711 decimal GB per rank, and wall times are 5198.780 and 6785.225 seconds. The near-flat peak from \(G=2\) to \(G=8\), together with the recorded member ordering, validates the serial response-graph lifetime at these endpoints. Post-prefix work is 271.278 versus 266.476 seconds per member, while mean total wall time per response falls from 2,599.390 to 848.153 seconds because the same prefix is amortized over four times as many members. These are two validation anchors, not a group-size limit or a broadly applicable throughput scaling law.

Every worker records old/reference scoring, policy backward, and a local AdamW call. Forward response attention composes all CP8 KV partitions. The backward audit narrows the terminal claim: \(dQ\) is all-reduced, while page-owner \(dK/dV\) contributions to replicated adapters are not. No post-step parameter hash, delta, or replica comparison is stored; these are execution, not distributed-update, receipts.

The workload is synthetic. Old and reference scores come from the same current runner, \(\beta=0\), and the implemented live policy term is unclipped. At the first step, old and current scores coincide and the importance ratio is one. The probes do not exercise an independent reference model, nonzero KL, or the clipped-min branch of the GRPO objective in Equation 2 .

8.3 GLM Terminal Manifest↩︎

The GLM manifest records 32/32 rank reports, two old-policy score phases, two live policy backwards, one optimizer call and one zero-grad call per worker, and no execution errors. The supplied rewards are \([0,1]\), giving normalized advantages \([-1,1]\). The per-member losses are in \([-0.5,0.5]\). The old snapshot is also the reference for the first update, so measured KL is zero and the importance ratio is one. The code contains a clipped surrogate with \(\epsilon=0.2\), but that branch remains inactive in the live beyond-2M-context transaction.

Model creation takes 170.211 seconds. Prompt capture plus both old phases, both policy phases, and terminal optimizer calls take 2975.138 seconds. These are single-run wall times. There is no variance estimate.

8.4 Trace Inventory↩︎

The final GLM run emits separate rank/phase JSONL traces. Table 7 summarizes the inventory.

4pt

The 64 old-policy traces cover all 78 forward layers on every rank. The 64 policy traces cover all 78 forward and 78 backward layer events on every rank. All policy layers report activation checkpointing. The traces include attention-projection and sparse-attention backward events. They show that the architecture-shaped layer graph executed twice and that every worker reached the terminal path.

Trace presence is not numerical correctness. A layer-end event does not record the complete parameter-gradient tensor, and the final run stores no numeric gradient norm. Trace completion also cannot reveal a skipped cross-rank reduction. The historical accepted GLM runner bypasses Megatron gradient finalization for CP-replicated non-expert adapters. Thus the trace remains an execution-level receipt.

8.5 Memory Accounting↩︎

After model, adapter, optimizer, and input construction, Qwen resets the CUDA peak counter for the measured probe. It reports allocated bytes in decimal GB, excluding reserved blocks and host memory.

The GLM artifact records a capture-window max_memory_allocated range of 112.571–145.148 GB per rank. The counter is reset immediately before no-grad prefix capture and read before full-GRPO response replay. The prompt-state contract stores 5.8125 GiB on CPU per rank and stages only 72 MiB for an IndexShare layer or 88 MiB for an index-computing layer, so the length-dependent state is not persistently GPU-resident. The rank spread shows nonuniformity and suggests room for placement and load-balance improvements. Thus 2M is an achieved operating point rather than a measured capacity ceiling. Because the artifact records neither reserved memory nor non-PyTorch usage and omits later replay and checkpoint-recomputation peaks, we retain “n/r” in Table 5 and infer from this capture-only diagnostic neither a whole-transaction limit nor a specific \(>2\)M capacity ceiling for the complete GLM transaction.

8.6 Semantic Claim Matrix↩︎

Table 8 maps the receipts to four evidence levels and separates completed from missing checks.

3pt

The ordering of the remaining work follows the matrix. Qwen first needs selective model-parallel composition of shard-local K/V and hidden-gradient contributions. The current GLM tree implements global DSA candidate/output composition and restores Megatron gradient finalization, but it still needs short-context full adapter-gradient and optimizer-delta parity followed by a fresh source-bound 2M rerun. Only after these checks does it make sense to build a repeated online rollout, reward, update, checkpoint, and reload loop.

9 Fixed-Budget Systems Lessons↩︎

The two implementations share a prompt-state graph boundary but expose different bottleneck chains, determined by what each architecture must retain and communicate afterward.

9.1 Fixed-Budget Capacity Comes from Lifetime, Not Sparsity Alone↩︎

Neither path avoids the long prompt forward, and neither path adds accelerators as context grows within its reported envelope. Every prompt token still passes through every decoder layer. The capacity gain comes from allowing prompt attention scratch, dense FFN intermediates, MoE routes, expert-token permutations, and adapter activations to die before response backward. Once the prompt has been captured, the live autograd working set follows response length rather than prompt length.

This explains why several plausible optimizations are insufficient on their own. DSA reduces attention arithmetic but retains long-context indexing  [21], [22]. MoE evaluates only selected experts but expands tokens into routed rows and distributes a very large parameter set  [23]–[25]. QLoRA reduces persistent trainable state, not activations [8]. Activation checkpointing reduces saved tensors by trading storage for recomputation [27], [28]. In the traced runs reported here, recomputation still re-created peak workspace at the guarded boundary. The working design composes all of these tools around the prompt-state graph boundary shared by both execution paths under review.

9.2 Physical Ownership Is Part of the Algorithm↩︎

A context shard is useful only when its physical allocation is also sharded. The first Qwen page implementation retained small slices whose parent chunks remained allocated. The logical page table looked distributed while allocator memory did not change. Copying selected pages into right-sized tensors made the ownership statement true and moved the 2,088,960-position prefix peak into the feasible range.

The same principle applies to GLM CPU state. A page tensor, its position order, and the consuming layer form one object. Restoring correct bytes in the wrong CP order changes causal positions and sparse selection. Offload transfers operator state and must preserve identity and order.

9.3 Dense and MoE Move the Peak to Different Places↩︎

For Qwen, every token evaluates the same dense FFN, and no expert routing state crosses devices. Context-growing storage is concentrated in 16 full-attention KV sets; 48 GDN boundaries remain fixed-size with prompt length. Page compaction and context partitioning therefore target the dominant stored state.

For GLM, expert sparsity separates total parameters from activated parameters, but full-sequence training still handles eight expert assignments per token. At the final CP shard size, one expanded BF16 hidden buffer can be 6 GiB before output, permutation, or LoRA work. Removing DSA scratch simply revealed the next MoE allocation. Response-only replay succeeds because only a few suffix rows are routed under autograd; the no-grad prompt routes remain transient.

The broader lesson is that sparsity transfers cost. DSA turns dense attention into an index-selection and selected-value movement problem. MoE turns dense FFN compute into expert residency and token communication. A training system must implement the transferred problem, not only count fewer FLOPs.

9.4 Context and Expert Parallelism Are Orthogonal↩︎

Context parallelism partitions token history. Expert parallelism partitions FFN parameters [26]. Folding CP32 and EP32 onto the same 32 ranks is a useful placement, but their collectives have different meanings. EP all-to-all sends response rows to expert owners and combines expert outputs. It cannot turn 32 local sparse candidate sets into one global top-2048 set. A CP attention merge cannot balance routed expert load. This distinction also appears in backward. A globally composed attention forward does not guarantee a coherent adapter update. Replicated projections need all shard-local gradient contributions. In the inspected Qwen path, \(dQ\) is all-reduced, while \(dK/dV\) and the corresponding adapter gradients remain local to page owners; eight independent AdamW instances then step. A complete distributed implementation must specify parameter ownership and gradient reduction as carefully as forward state ownership.

9.5 Forward Fidelity, Update Consistency, and Gradient Parity Differ↩︎

The Qwen and GLM paths occupy different positions on the four-level evidence ladder.

• Both produce fixed-budget 2.097M execution receipts: the requested scoring, response backward, and terminal optimizer calls complete on every worker.

• Qwen composes a global full-attention response forward over CP8. Its production numerator reduction uses BF16, so the claim is partition-correct forward semantics, not bitwise FP32 equality. GLM response DSA remains local to one CP shard.

• Qwen distributed-update consistency is incomplete because shard-local K/V adapter gradients are not synchronized. The GLM receipt also requires an explicit reduction: its custom resident path bypasses finalize_model_grads, so CP-replicated attention, dense/shared FFN, and output-head adapter gradients remain local before the optimizer call.

• Neither path matches a conventional full-sequence gradient because the prompt state is detached. GLM must first close its forward mismatch; both then require a parameter-by-parameter gradient reference at a shorter context.

Loss values and layer-end backward events cannot substitute for these tests. A single global gradient norm could also hide target-family errors. The parity suite must compare each LoRA shard, report per-family outliers, compute global cosine and relative \(L_2\), and verify the optimizer delta from identical initial state.

9.6 Group Scaling Is a Scheduling Result↩︎

Qwen \(G=8\) consumes nearly the same peak allocated memory as \(G=2\), consistent with response branches being serialized after one prompt capture. A fourfold group increase raises measured peak allocation by 0.208 GB, or 0.213%, while post-prefix work grows by a factor of 3.93. The marginal time across the six additional members is 264.408 seconds per member. Meanwhile, sharing the 4,655-second prefix reduces mean total wall time per supplied response by 67.4%, from 2,599.390 to 848.153 seconds.

The implementation is therefore parameterized by group cardinality rather than capped at the observed \(G=2\) or \(G=8\). The serial loop consumes a configured member list and is not specialized to either value, but only these two settings have Qwen receipts. This does not make memory strictly constant: stored inputs, labels, rewards, reports, and frozen scores grow with \(G\) or total response length, and larger groups remain subject to the wall-time budget. Nor does it make \(G\) algorithmically irrelevant; changing the group changes reward normalization and the GRPO estimator. The measured claim is narrower: group size is not the dominant live-autograd capacity axis in this design. The GLM acceptance run uses two extremely short responses and has no whole-run peak, so it provides no independent group-scaling curve.

These single-run timings span different models, GPU counts, suffixes, numerical paths, and semantic gaps. They establish termination within a specified resource envelope, not a model or system throughput ranking.

9.7 Eight H20s Carry Qwen to a 4.25M Context Envelope↩︎

Under the same eight-H20 budget used for the 2,097,152-position receipts, Qwen reaches 4.25M context, where 4.25M means exactly \(4.25\times2^{20}=4,456,448\) positions: 4,448,256 prompt positions and an 8,192-position response. A resident \(G=8\) run completes one 1,086-chunk prefix capture, all eight serial old/reference/policy branches, and all four 2,048-position response-backward blocks per member. The measured path takes 21,750.133 seconds from prefix start through the group and peaks at 82.960 GB per rank. This is a complete 4.25M response-replay receipt, not a prefix-only or stage-1-forward result.

The stronger prefix-frozen response-only run converts that receipt into an 8-step curve. Each step accumulates all eight group members before applying an optimizer update. Across 64 member replays, every rank records eight applied optimizer steps, prefix_stale=false, and a peak of 83.894 GB. Freezing adapter deltas on prompt positions makes the captured prefix invariant across those updates, so this is complete continuous training for the stated prefix-frozen objective. It is not the original prompt-adapted QLoRA objective, whose state must be recaptured after every optimizer step before reuse.

The remaining frontier numbers are capacity diagnostics rather than the main 4.25M result. A clean prompt-adapted run before detached-prefix gradient-page pruning OOMed in policy backward. A train-block proxy reaches 4,538,368 before the next 4,096-position chunk OOMs at 4,542,464. None of these labels erases the separate CP8 reduction gap described above: local optimizer application is recorded, but coherent replicated-adapter updates still require the missing cross-rank gradient composition. See Appendix 18 for full details.

9.8 The Claim Is a Fixed-Budget Envelope, Not a Context Record↩︎

Million-token execution predates LongStraw, and this report makes no first, longest, or world-record claim. Its comparison axis is the fixed accelerator envelope: 4,456,448 Qwen positions on eight H20 GPUs and 2,097,152 GLM positions on 32 H20 GPUs. Adding devices is a valid scale-out strategy, but it is outside these experiments. LongStraw instead trades GPU residency for compact physical pages, CPU state, recomputation, and serial replay time within the stated allocations without adding ranks.

The stored Qwen configuration has a native maximum position setting of 262,144; its receipt at a context length of 2,097,152 is exactly \(8\times\) that setting. The published GLM-5.2 configuration uses 1,048,576; the captured 2,097,152-position prompt is \(2\times\) that setting. No long-context task evaluation, loss curve, or repeated learning result accompanies the receipts. The result is therefore an accelerator-bounded systems feasibility envelope, not evidence of useful learned behavior or a context-length leaderboard.

10 Related Work↩︎

10.1 Scale-Out Long Context and the Fixed-Budget Axis↩︎

Prior work already establishes that million-token sequence processing is possible. Ring Attention reports exact-attention training at 4.096M positions for a 7B model on 32 A100 GPUs [9]. DeepSpeed-Ulysses studies a scale-out regime in which sequence length and device count grow together; its experiments scale to 256 A100 GPUs and include a one-million-token sequence for a 1.2B GPT model [10]. ByteScale reports a 2M LLaMA-7B case on 1,024 GPUs within a production cluster exceeding 12,000 GPUs  [11]. USP then combines ring-style and all-to-all sequence parallelism and analyzes its interaction with tensor parallelism, ZeRO, recomputation, and offload [12]. DistFlashAttn adds load-balanced exact-attention scheduling and overlaps peer-to-peer KV transfer with attention compute, while LoongTrain combines head and context parallelism through 2D-Attention and a double-ring schedule  [38], [39]. Both remain scale-out methods that widen device-level parallelism.

Ring Attention and ByteScale provide stronger full-sequence training semantics; LongStraw uses fewer devices for a different detached-prefix GRPO replay problem. The comparison therefore motivates the budget and evidence axes rather than an efficiency ratio across unmatched workloads.

Large-scale technical reports operate at a different industrial scale. DeepSeek-V3 reports a 2,048-H800 training cluster and 2.788 million H800 GPU-hours for its full 671B-MoE training program [37]. LongCat-Flash reports a 560B MoE trained with infrastructure spanning tens of thousands of accelerators, while GLM-5 reports a 744B MoE trained over 28.5 trillion tokens [40], [41]. These results are not apples-to-apples baselines for LongStraw: model size, objective, hardware, sequence semantics, and evidence level all differ. They instead establish why our claim is not “first long context.” LongStraw fixes the accelerator envelope at eight H20 GPUs for Qwen and 32 H20 GPUs for GLM, then asks which state-lifetime and ownership decisions make a GRPO-shaped execution path fit. We report a budget-conditioned feasibility envelope, not a context-length or throughput leaderboard. The relevant comparison is accelerator-bounded GRPO execution, not absolute long-context capability.

10.2 Relationship to MinT↩︎

MinT manages LoRA adapter revisions across rollout, update, export, evaluation, and serving over resident base-model deployments; its training plane includes distributed Megatron execution for dense and MoE models, parallelism-aware adapter state, and MLA/DSA support [15]. The current LongStraw rerun interface directly reuses that managed control plane and resident Megatron LoRA substrate, but changes the state boundary inside one update: it captures architecture-specific prompt state without autograd, stages that state under CP/EP ownership, and serially rebuilds response graphs. This architectural lineage does not retroactively bind historical LongStraw receipts to the current MinT revision. The one-million-adapter result of MinT measures addressable policy-catalog scale with bounded serving working sets, not long-context execution. The 2M and 4.25M quantities in LongStraw count positions in execution receipts, not addressable adapter revisions or simultaneously served policies.

10.3 Memory-Efficient and Distributed Attention↩︎

Exact attention does not require materializing the full quadratic score matrix. Memory-efficient algorithms stream score blocks or improve IO locality while preserving the softmax result [5], [6]. Ring Attention distributes sequence blocks over devices and composes attention as the blocks circulate [9]. DeepSpeed-Ulysses exchanges sequence and attention-head partitions with all-to-all collectives, while USP combines Ulysses-style and ring-style sequence parallelism  [10], [12]. These methods address the core attention computation. Our setting adds a training-specific boundary: prompt state is retained across old, reference, and policy branches, while the suffix is replayed under autograd. The central questions become physical page ownership, state validity, and which distributed reductions are required in both forward and backward.

PagedAttention makes KV allocation and page tables first-class serving-system objects [3]. In this report, pages cross a training boundary. A logical page shard must own a compact physical allocation; a view into a larger parent chunk does not release memory. Prompt pages are read-only during grouped response replay, and their validity ends when an optimizer step changes the adapted parameters.

10.4 MLA, Sparse Attention, and Index Reuse↩︎

MLA compresses KV state into a latent [20]. DeepSeek-V3.2 forms DeepSeek Sparse Attention (DSA) by adding a lightweight top-k indexer to MLA [21]. The GLM-5 report provides MLA/DSA background [41]; the pinned GLM-5.2 configuration artifact instantiates index producers and IndexShare consumers [14]. IndexCache formalizes the generic cross-layer sparse-index reuse mechanism  [22]; the pinned configuration supplies this model’s concrete pattern.

Our focus is the resulting training-state contract. A saved MLA latent page is not sufficient when the sparse indexer also needs long-context keys. Index reuse saves work but creates producer/consumer lifetime inside each response forward. Local sparse selection is not global selection over a context-parallel prompt. Distributed training must define candidate merge, selected-value movement, and output composition.

10.5 MoE Training and Multidimensional Parallelism↩︎

Sparse MoE models distribute the full parameter set but activate only selected experts per token [24], [25]. Expert parallelism reduces per-rank parameter residency but introduces token permutation, all-to-all dispatch, expert computation, and combine. DeepSeek-V3 links fine-grained MoE to cross-node overlap and memory-efficient training [37]. LongCat-Flash similarly co-designs MoE layer structure, communication overlap, deterministic kernels, and its EP/CP/PP layout [40]. MoE Parallel Folding analyzes heterogeneous tensor, context, expert, data, and pipeline mappings [26]. Tutel selects MoE all-to-all implementations and pipeline degree by workload and cluster scale; MegaBlocks maps irregular expert-token work to block-sparse operations  [42], [43]. These systems optimize sparse kernels, communication, and parallel mappings; LongStraw instead isolates long-lived prompt state from each short-lived native MoE replay and makes the remaining distributed-gradient obligations explicit before optimizer consumption.

The GLM path in this report uses CP32 and EP32 over the same ranks. This placement is economical, but the parallel dimensions remain semantically different. CP owns context and must preserve attention; EP owns experts and routed-token computation. Neither collective can replace the other.

10.6 Distributed Training State and Optimizer Sharding↩︎

Megatron-LM established tensor model parallelism for large Transformers [44]; subsequent Megatron training systems compose tensor, pipeline, and data parallelism at cluster scale  [45]. ZeRO and PyTorch FSDP instead shard parameters, gradients, and optimizer state across data-parallel workers  [46], [47]. These systems reduce persistent state and define ownership for standard layer graphs. LongStraw composes CP, EP, and sharded optimizer machinery with custom prompt-state replay. That boundary must still call the appropriate gradient finalization or selective reduction; optimizer sharding cannot recover contributions that never reach a parameter owner.

Heterogeneous-memory variants widen the placement space. ZeRO-Offload moves selected model states from GPU to CPU memory, while ZeRO-Infinity can place partitioned model states in CPU or NVMe memory  [48], [49]. LongStraw’s CPU prompt pages are different state with a different lifetime, but they inherit the same requirement that ownership and transfer timing be explicit.

10.7 Activation Checkpointing and Parameter-Efficient Adaptation↩︎

Activation checkpointing trades recomputation for lower retained activation memory [27]. For the GLM response path, the useful boundary is the complete decoder layer. Checkpointing attention alone leaves MoE router, dispatch, selected-expert, and adapter intermediates alive. The long prompt is not checkpointed for backward; it is evaluated without autograd and represented by stored conditional state. Selective activation recomputation instead retains the layer boundary while recomputing only memory-heavy, relatively inexpensive attention operations  [28]. That complementary design reduces redundant recompute; it does not replace the full-layer boundary required by our native MoE replay.

LoRA and QLoRA reduce trainable parameter, gradient, and optimizer-state storage [7], [8]. They do not remove response activations or the need to synchronize replicated adapter gradients. In Qwen, full-attention forward composition is global, but coherent CP8 adapter updates still require the missing cross-rank gradient synchronization in Section 5. AdaLoRA allocates adapter rank under a parameter budget, while DoRA separates weight magnitude from low-rank directional updates [50], [51]. Related work explores hierarchical rank allocation, intra/inter-layer adapter sharing, mixed-precision fidelity, and joint quantization with low-rank adapters  [29]–[32]. GPTQ and SparseGPT are established post-training quantization and pruning methods [52], [53]; global rank/sparsity optimization and probabilistic layer quantization further change the base model’s storage and sensitivity profile [33], [34]. Qwen uses NF4 QLoRA and GLM rank-8 LoRA; adapter design is not a LongStraw contribution. LongLoRA combines LoRA with shifted sparse attention for efficient long-context adaptation [54]; LongStraw instead preserves the model’s native prompt semantics and changes graph lifetime and placement.

10.8 Adapter Serving and Inference Infrastructure↩︎

Punica and S-LoRA establish multi-tenant batching, memory management, and kernel paths for serving many adapters concurrently  [55], [56]. Dynamic operator selection likewise treats adapter placement and operator reuse as serving-time systems problems  [57]. Budget-driven depth routing and dynamic low-rank substitution address adaptive inference, where the objective is to reduce latency or compute for a fixed request [58], [59]. LongStraw is adjacent but distinct: it targets the training-time prompt/response boundary, preserves architecture-specific state across a grouped GRPO update, and exposes the gradient-ownership conditions that serving systems do not need to satisfy.

10.9 GRPO Systems↩︎

PPO alternates policy sampling with multiple optimization epochs over a clipped surrogate objective [16]. GRPO normalizes outcome rewards within a response group and removes the learned critic used by PPO-style training [4]. DeepSeek-R1-Zero applies GRPO in a reasoning-training pipeline that begins without supervised fine-tuning  [60]. DAPO then extends the GRPO family with decoupled clipping, dynamic sampling, token-level policy-gradient loss, and overlong reward shaping [61]. These methods change optimization behavior, not prompt-state lifetime.

DeepSeek-V3.2 describes additional stabilization mechanisms for scaled GRPO, including off-policy sequence masking and preservation of MoE routing between inference and training [21]. LongCat-Flash-Thinking-2601 develops an asynchronous system for long-tailed environment interaction and large-scale agentic RL [2]. DeepSpeed-Chat, HybridFlow, and OpenRLHF address the broader RLHF execution problem, including model-role placement and transitions among generation, scoring, and training; OpenRLHF assigns rollout and actor/training engines distinct roles under Ray  [62]–[64]. AReaL goes further by decoupling rollout and training asynchronously and explicitly managing data staleness [65].

Our acceptance workload is narrower. Responses and rewards are supplied, and the runs terminate after one update-shaped transaction. The report isolates the long-context policy-backward machinery from rollout scheduling, environment execution, and repeated learning. This separation permits precise systems receipts, but it also prevents claims about online RL throughput or downstream policy improvement.

11 Conclusion↩︎

LongStraw shows that long-context GRPO under a fixed GPU budget is a tensor-lifetime and ownership problem, not a context-length race or a single-kernel change. Long-context processing itself is established when a sufficiently large accelerator fabric is available. The systems question here is how much GRPO-shaped execution fits without adding accelerators. LongStraw fixes the inventory at eight H20 GPUs for Qwen and 32 H20 GPUs for GLM, then makes the contract among model structure, prompt-state ownership, suffix replay, parallel communication, gradient composition, and optimizer ordering explicit. The shared mechanism is a no-grad prompt boundary followed by serial short-response replay; the state on that boundary is architecture specific at every layer and ownership boundary in both model families.

For the dense-hybrid Qwen model, compact physical KV pages and recurrent GDN state make the prompt fit across CP8, and a global LSE/output reduction composes the full-attention response forward. For GLM, CPU-resident MLA and indexer-key pages, one-layer staging, complete-layer checkpointing, IndexShare reconstruction, and CP32/EP32 placement carry two 78-layer response backwards to terminal distributed-optimizer calls. The GLM progression from 32K full-graph failures through prefix capture, layer-0 replay, all-layer closure, and grouped execution shows why capacity must be built one dependency class at a time.

The same audit also identifies what did not run correctly. Qwen lacks composition of shard-local K/V adapter gradients. The historical GLM receipt uses CP-local DSA and bypasses Megatron gradient finalization for CP-replicated non-expert adapters. Both detach the prompt state. The durable result is therefore a budget-conditioned, architecture-aware execution envelope rather than a first-to-length or longest-context claim. Within the same eight-H20 envelope, Qwen completes a 4.25M \(G=8\) response replay and, under the explicit prefix-frozen response-only parameterization, eight consecutive \(G=8\) optimizer steps comprising 64 member replays at a peak of 83.894 GB per rank. The 4,538,368/4,542,464 train-block bracket exposes further capacity room; the 32-H20 GLM path carries a 2,097,152-position prompt through two complete 78-layer backwards and terminal optimizer calls. That GLM point is not an OOM-derived frontier: recorded capture-window peak allocation ranges from 112.571 to 145.148 GB per rank, and no GLM probe above 2M was attempted. The rank spread suggests room for better placement and load balance, but only a direct \(>2\)M probe with whole-transaction memory accounting can turn that opportunity into a capacity result. These are accelerator-bounded feasibility receipts, not minimum-cost or distributed-gradient-parity claims.

12 Limitations and Validation Roadmap↩︎

These rank-complete execution receipts lack support for stronger correctness or training claims.

12.1 Distributed Gradient Composition Is Incomplete↩︎

The Qwen forward path composes global CP8 attention statistics, and its custom backward all-reduces \(dQ\). It leaves page-owner \(dK/dV\) contributions local even though the K/V projection adapters are replicated. No DDP or selective model-parallel gradient reducer runs before eight independent AdamW calls. Consequently the adapter updates can diverge and are not the gradient of the global conditional response computation.

The historical GLM custom resident path calls loss.backward() outside the normal Megatron schedule and then invokes the distributed optimizer. It bypasses finalize_model_grads and the DDP finish_grad_sync it normally calls, leaving CP-replicated attention, dense/shared FFN, and output-head adapter gradients unreduced. Routed-expert adapters differ: EP32 with expert data-parallel size one makes them unique shards that need no averaging; an optimizer all-gather can still align replicas after updates from incorrect local CP gradients. The current tree restores Megatron gradient finalization, but it has no fresh source-bound 2M receipt or full adapter-gradient and optimizer-delta parity.

Before a stronger fixed-budget training claim, both paths must record the expected gradient collective, compare post-step parameter hashes across ranks, and verify the optimizer delta against a reference.

12.2 The Historical GLM Receipt Uses CP-Local Response Attention↩︎

In the historical receipt, every GLM rank selects top-2048 positions from its own 65,536-token prompt shard. There is no cross-rank candidate merge, global top-2048 selection, selected-value exchange, or output composition. IndexShare reuses this local schedule. It is an architecture-shaped surrogate, not distributed DSA over a global prompt of 2,097,152 context positions as seen by each response query.

The current tree implements local candidate production, global top-2048 selection, selected-value movement, and sparse-output composition. It still needs per-layer response-logit parity against an unsharded reference at a context that fits on one process, followed by a fresh source-bound 2M rerun.

12.3 The Prompt-State Gradient Is Detached↩︎

Even after the distributed forward and gradient collectives are repaired, both paths compute the response gradient conditional on a stored prompt state. They omit the second term in Equation 3 . Full-sequence parity must be tested at 32K or 64K, where a conventional reference fits. The comparison must cover every LoRA target family, not only loss, one gradient norm, or a subset of attention projections.

12.4 The Workload Is an Execution Probe↩︎

The runs use supplied synthetic responses and deterministic rewards. They do not include policy sampling, environment or reward-model execution, data filtering, checkpoint publication, reload, or repeated updates. The Qwen probe uses the same runner for old and reference scores, \(\beta=0\), and an unclipped live ratio term. The GLM path implements clipping and \(\beta=0.01\), but the first-step ratio is one and old equals reference, so clipping is inactive and KL is zero. Neither run measures policy improvement.

The stored model configurations also have native context settings below the exercised scale: 262,144 for Qwen and 1,048,576 for GLM. No task evaluation establishes useful behavior at this systems scale.

12.5 Resource and Reproducibility Gaps↩︎

The fixed-budget claim covers accelerator count and reported device-memory measurements, not total cost. We lack a matched baseline on the same models, objective, suffixes, precision, and hardware, and the artifacts do not fully account for host memory, network traffic, utilization, energy, or monetary cost. LongStraw is therefore accelerator-bounded or resource-constrained, not a claim of universal lowest-cost execution.

All timings are single runs. Qwen samples only \(G=2\) and \(G=8\); GLM has one grouped \(G=2\) receipt, while its \(G=1\) point is only a control-flow canary. There is no broad group-size sweep, evidence of strictly linear time, or proof that total memory is independent of \(G\). Qwen records whole-run allocated CUDA peaks, while the recorded GLM capture-window peak allocation is capture-only and cannot be used as a whole-transaction number. The final GLM artifact does not bind a numeric learning rate, final model revision, token checksum, LoRA seed, software/NCCL version set, whole-step finite scan, final gradient norm, router histogram, or all-to-all byte/timing profile. These omissions do not invalidate terminal execution, but they limit independent reproduction and end-to-end performance analysis across runs.

12.6 Validation Order↩︎

The next work should proceed in dependency order.

1. Restore the missing Qwen K/V reductions and validate the current GLM gradient-finalization path at 32K; verify post-step adapter equality across ranks.

2. Validate the current global cross-CP DSA candidate selection, selected-value movement, and output composition with layerwise forward parity.

3. At 32K–64K, compare every LoRA gradient shard and optimizer delta with a conventional full-sequence execution under identical tokens and rewards.

4. Run a real rollout group through sampling, reward computation, frozen old log-probabilities, update, checkpoint, and reload.

5. Repeat the 2M capacity run from the pinned stack, then run a direct \(>2\)M GLM sweep with whole-transaction memory accounting, variance, and phase-separated time.

References↩︎