1 Introduction↩︎

Recent investigations aiming at representative disassembly of a language model, commonly termed under the broad umbrella of mechanistic interpretability [1], have identified striking phenomena emerging within these models — inference-time self-repair or the Hydra effect [2], induction heads [3], function vectors [4]. Each of these phenomena is universal to autoregressive language models trained on large natural language corpora; yet, we lack any framework to explain why they appear or an explanation for their coincidental emergence.

Such an understanding would require embedding the representative disassembly of the model within the trifecta of training data generation process, optimization dynamics, and model architecture. All three, due to the sheer scale, are intractable without simplifying assumptions. Multiple prior attempts at understanding training dynamics do not consider the data generation process (e.g., [5]), or do so in a very simplified structure, non-reflective of the topological and statistical properties of real text corpora (e.g., [6], [7]). A few promising works connected certain properties in isolation (e.g., in-context learning) to macro-scale properties of the training data [8], [9]. However, they fall short of explaining the co-temporary emergence of different mechanistic phenomena. We note that not all mechanistic phenomena act on the same scale/scope. For example, induction circuits are pertinent to local structures of the form \({\tt [A][B]}\cdots {\tt [A]}\rightarrow{\tt [B]}\); whereas the Hydra effect is a more global property, affecting the model architecture. Most prior work in this regime assumes a flat, sequential data generation process via fixed or variable-order Markov chains [10]–[12]. The limitations of sequential models in explaining global linguistic structures are well-established results in computational linguistics, often alleviated by richer assumptions of hierarchical structures [13], [14]. Very recently, [15] found that warm-starting pretraining with formal language corpora helps better generalization.

To this end, we ask the following central research question: Can suitably parametrized, hierarchical data-generation models explain the co-emergence of mechanistic phenomena across different scales? We utilize probabilistic context-free grammars (PCFGs) as the underlying data generation process to analyze the emergence of three well-known mechanisms: induction heads, function vectors, and the Hydra effect. Our choice is motivated by the apparent heterogeneity in the scope of these phenomena: induction heads are localized circuits formed by 2 attention heads that perform prefix-matching and copying from repeated patterns in the context. Function vectors, on the other hand, are representational digests that summarize input-output mappings from context, disentangling shared semantics from lexical irregularities. Finally, Hydra effect is the phenomenon when ablation of a model component (attention, MLP, etc.) leads to increased compensation from a subsequent component; unlike the prior two, Hydra effect is less task-specific and more tied with the global scope of the architecture. We demonstrate that a PCFG-controlled corpus captures the emergence of mechanistic phenomena more faithfully than a sequential baseline. We validate our findings by comparing them to a real-world model training run. We find that the emergence and development of all three phenomena across training coincide with the stage when the internal representation starts reflecting the latent hierarchical geometry. Finally, we provide a theoretical linkage between hierarchical structures in the data generation process and the mechanistic phenomena under investigation. To the best of our knowledge, our work is the first of its kind to propose a unified tooling and theoretical framework to explain different, seemingly unrelated mechanistic phenomena emergent in language models.

2 Related Work↩︎

2.0.0.1 Training dynamics

offer insight into emergent behaviors in neural networks, where gradual increases in scale or training progress lead to sudden qualitative changes in capability [16], [17]. Such phase transitions have been hypothesized to arise from the rapid formation of specialized circuits [5]. Transformer architecture has been shown to possess implicit biases of gradient flow that the near-optimal model is equipped with induction heads [9]. [18] study how factual recall is acquired over training. Despite isolated understandings of different such phenomena, a unified picture remains missing. Moreover, phenomena such as Hydra effect [2] remain unexplored from the training dynamics and data distribution lens.

2.0.0.2 Synthetic data for interpretability

Prior work has shown that model performance follows robust scaling laws determined not only by model size and compute but also by the structure of the training data [19], [20], highlighting a gap between expressibility and learnability that depends on data structure and optimization dynamics [21], [22]. To study how data properties give rise to emergent behaviors, researchers have increasingly turned to synthetic datasets, which allow precise control over statistical and structural features while preserving key learning dynamics observed in natural language models [6], [23]. However, many synthetic setups rely on flat or sequential data and fail to capture the hierarchical and recursive structure central to language. Probabilistic context-free grammars (PCFGs) offer a principled middle ground, exposing a transparent data generation process while retaining core linguistic properties, such as compositionality, ambiguity, and recursion. Previous work has used PCFGs to probe inductive biases [24], derive exact comparisons between learned and true distributions [25], and study the acquisition of hierarchical syntax [26]. Most closely, [27] show that transformers reduce loss across grammatical components in parallel yet struggle to learn deep recursion, suggesting optimization limits rather than architectural constraints. While these studies establish PCFGs as effective tools for analyzing linguistic competence, they largely stop short of mechanistically explaining how models internalize structure.

[15] demonstrate a strong precedence to our case; they coin the term pre-pretraining (training on formal language before natural languages) and found that hierarchical formal language data elicits linguistic generalization in natural language models.

Our work takes the next step forward to use hierarchical formal language, realized via PCFGs, as empirical and theoretical tools to explain quirks of training dynamics in natural language. Instead of isolated studies of different mechanistic phenomena, we propose a unified lens of interpretation.

3 Experimental Setup↩︎

We design an experiment that isolates the effect of structure in training data while closely matching the surface statistics of natural language. We construct two data generation processes that produce text with similar token distributions, vocabulary usage, and short-range dependencies. One process contains an explicit underlying structure, while the other intentionally lacks such structure. We train identical language models on the resulting corpora using the same architecture, optimization method, and hyperparameters. At fixed intervals of \(x\) training steps, we evaluate a predefined set of model properties. We then compare these measurements to those obtained from a model trained on real-world natural language data (OLMo-1B) to validate faithfulness. An overview of this procedure is depicted in Figure 1, and the pertaining configuration can be found in Appendix 8.

3.1 Data generation process↩︎

Each process draws from a fixed vocabulary of size \(V\), denoted by the token set \(\mathcal{V} = \{0, 1, \ldots, V-1\}\), where eos is a special end-of sequence-symbol. See Appendix 8 for full details.

3.1.0.1 N-gram

The N-gram data generation process serves as a baseline because it represents the simplest form of generative structure, one that lacks hierarchy and recursion. It captures only local and sequential dependencies between tokens defined by a fixed history window. Sampling from a Zipf distribution preserves the statistical patterns characteristic of natural language. This simplicity makes it an ideal reference model. Any additional emergent behaviors observed in more complex generative systems, in our case, a PCFG, can therefore be attributed to structural complexity rather than basic statistical or optimization effects. For an N-gram model of order \(n\), the context \(h\) consists of the sequence of the previous \(n-1\) tokens. For each unique history \(h = (w_1, \dots, w_{n-1}) \in \mathcal{V}^{n-1}\), we define a transition distribution over the next token \(t\): \[P(t | h) \propto \frac{1}{(t + 1)^{\alpha_h}}\] where the Zipf exponent \(\alpha_h\) is sampled as \(\alpha_h \sim \max\left(\mathcal{N}(\mu, \sigma^2), \alpha_{\min}\right)\) for each distinct history context. To generate a sentence \(\mathbf{x} = (x_1, x_2, \ldots, x_L)\), we first determine the sentence length \(L = \ell_{\min} + Z, \quad \text{where } Z \sim \text{Zipf}(s)\). We initialize the history \(h_0\) with a set of padding tokens. Then for \(i = 1, \ldots, L\): \[\begin{align} x_i &\sim P(\cdot | h_{i-1}) \;h_i &= \text{shift}(h_{i-1}, x_i) \end{align}\] where the update step \(\text{shift}\) discards the oldest token in the history and appends \(x_i\) to form the new context for step \(i+1\). Finally, we append the eos token: \(x_{L+1} = V - 1\).

3.1.0.2 PCFG

The probabilistic context-free grammar represents an intermediate level of generative complexity. Unlike regular grammars, which capture only local token dependencies, a PCFG introduces a hierarchical structure through nested production rules. This structure enables the generation of sentences with grammatical relationships between constituents, specifically subjects, verbs, and objects. As shown in Appendix 7.2, the model constructs sequences using recursive nonterminal expansions that correspond to different levels.

The data generation process begins with a document symbol \(S\), which expands into multiple document segments \(D_i\). Each segment represents a hierarchical unit, such as a section or paragraph, that further decomposes into sentences. The shuffle operator applied at the document level randomizes the order of these units, preventing the model from overfitting to a fixed sequence order. Within each sentence, the production rules distinguish between two types: assertions and questions. Assertions combine subjects, verbs, and objects into simple or compound statements, while questions invert the order of these components to form interrogative patterns. This hierarchical design captures the syntactic relationships found in natural language, even though the tokens themselves remain abstract.

Terminal symbols (\(\text{S}_i\), \(\text{V}_i\), and \(\text{O}_i\)) correspond to vocabulary items. Each set of terminals is sampled from a Zipf distribution to reflect the frequency imbalance typical of real linguistic data. Nonterminal symbols like \(\text{Subject}\), \(\text{Verb}\), and \(\text{Object}\) expand into these terminal choices, while optional permutation functions \(\mathrm{PermuteOrder}_{D_i}\) introduce local variability across document segments. The resulting sentences maintain syntactic coherence while allowing statistical diversity.

3.2 Mechanistic Phenomena↩︎

3.2.0.1 Hydra effect

in our context follows the definition of Hydra effect introduced by [2]: whether ablation of a layer causes its following layers to compensate by increasing the predictive influence. To assess the dependency of layer \(\ell\) on a preceding layer \(\ell-m\), we measure the drop in the confidence of the true prediction when the precursor layer is removed. Let \(f^{(\ell)}(\mathbf{x})_y\) denote the intermediate logit for the ground-truth token \(y\), computed by applying the unembedding matrix \(\mathbf{W}_{\text{U}}\) to the hidden state at layer \(\ell\). We define an ablated state \(\tilde{f}^{(\ell)}_{\ell-m}(\mathbf{x})_y\) where the output of layer \(\ell-m\) is zeroed before computing the features at layer \(\ell\). The impact is quantified by the difference between the base and ablated logits for the ground-truth token. Formally, calculated as \(\bar{\Delta}^{(\ell)}_m = \frac{1}{|\mathcal{D}|} \sum_{(\mathbf{x}, y) \in \mathcal{D}} \left( f^{(\ell)}(\mathbf{x})_y - \tilde{f}^{(\ell)}_{\ell-m}(\mathbf{x})_y \right)\). A high positive \(\bar{\Delta}^{(\ell)}_m\) indicates that the information processed at layer \(\ell-m\) is critical for the correct prediction formed at layer \(\ell\).

3.2.0.2 \(k\)-Order Induction Heads

are attention mechanisms that enable in-context learning by implementing a copy-and-paste behavior: if a specific context appears again, the head attends to its previous occurrence to predict the token that followed it. While standard induction heads rely on a single token context (\(k=1\)), we generalize this to \(k\)-order induction, where the head activates only upon matching a specific sequence of length \(k\). Following [3], we quantify this capability using a generalized prefix score. Given a random sequence \(\mathbf{x}\) repeated twice, the \(k\)-order prefix score of a head \(h\) at layer \(l\) is the average attention from the current token \(x_i\) to the end of the identical context in the first occurrence: \[\frac{1}{|\mathbf{x}|-k}\sum_{i=|\mathbf{x}|+k+1}^{2|\mathbf{x}|}\alpha^{(h,l)}(x_i, x_{i-|\mathbf{x}|}).\]

3.2.0.3 Function vectors

encode task-specific mappings within attention heads. We quantify their strength by extracting an activation \(\mathbf{h}\) from a few-shot context and patching it into a zero-shot query. The score is the mean logit increase for the correct target token \(o_q\) induced by this intervention: \(\bar{\Delta}^{(\ell)} = \mathbb{E}_{\mathcal{D}}[f_{\text{patch}}(\mathbf{x}_q)[o_q] - f_{\text{base}}(\mathbf{x}_q)[o_q]]\).

3.2.0.4 Parse-Tree Geometry

To investigate whether the model’s internal representation space encodes the hierarchical structure of the data, we employ a structural probe [28]. We learn a linear transformation \(B \in \mathbb{R}^{d_\text{probe} \times d_{\text{model}}}\) such that the squared Euclidean distance between transformed vector representations \(h_i, h_j\) approximates the distance between words \(w_i, w_j\) in the ground-truth parse tree. Formally, the probe minimizes the deviation between the tree distance \(d_{\text{tree}}(w_i, w_j)\) and the probe distance \(d_B(h_i, h_j)^2 = (h_i - h_j)^T (B^T B) (h_i - h_j)\). The geometric alignment is quantified using the unlabeled undirected attachment score (UUAS), which measures the percentage of edges shared between the true parse tree and the minimum spanning tree constructed from the predicted distances \(d_B\).

4 Results↩︎

4.1 Hierarchy Allows for \(k\)-th-Order Induction Heads to Emerge↩︎

We follow this behavior by analyzing diagonal attention patterns in duplicated random sequences and by varying the induction order (\(k = 1 \dots 10\)). At each training checkpoint, we compute the maximum attention score across all layers and heads for each value of \(k\) for 100 samples. Figure 2 shows that for the N-gram, the model does not develop induction heads at any point during training. In contrast, the PCFG shows a sharp increase in induction-related attention at approximately 6k training steps. It is to be noted that, despite analyzing ten \(k\) values, they are identical and overlap in accuracy exactly on this scale. Furthermore, its trajectory is comparable to that of the reference model, although it flattens out slightly earlier and is less unstable towards the end of training.

4.2 Emergence of Function Vectors↩︎

Figure 3 shows the function vector trajectory, where improvement substantially increases after around 6k training steps. This is also the point at which the \(k\)-th-order induction heads emerge. N-gram models are not shown due to no observed existence of function vector formation.

4.3 Hydra Effect Emerges with Hierarchy↩︎

Figure 4 (for \(m = 1\)) summarizes a comparative demonstration of Hydra effect in PCFG model and OLMo-1B. Here again, for the model trained using N-grams, layer ablation is not compensated by a successor layer. PCFG-trained model shows substantial signs of Hydra effect. In fact, the degree of compensation is higher than OLMo-1B. Notice that in the very early stage of training, both models show compensation in the deeper layers. After training, the effect localizes in the middle layers.

4.4 Internal Geometry Mirrors Hierarchical Structure↩︎

To validate that the model internalizes the generative topology, we first analyze the probability mass assigned to all syntax-valid tokens (Figure 5 (a)) and find that this shallow syntax is already learned early in training, at around 4k steps. Next, to get an understanding of the more abstract syntax, we view the geometric alignment between the representation space and the ground-truth parse trees. Figure 5 (b) displays the layerwise development of the unlabeled undirected attachment score. We observe that the model progressively learns to map the Euclidean distance of its internal representations to the true tree distance of the generative grammar. This emergence is highly layer-dependent: intermediate layers (layers 5–10) exhibit the strongest structural alignment, reaching a UUAS of approximately 0.9, while earlier and later layers remain less structured. This localization suggests a dedicated syntactic processing stage within the network depth. These two observed shifts in hierarchy learning are also reflected in the loss trajectory (see Figure 6): two sharp dips in loss correspond to the emergence of shallow and deep hierarchy representations.

5 Theoretical Results↩︎

So far, we have empirically demonstrated the effects of structures in the data generation process on the mechanistic phenomena observed in the model. Next, we show that under realistic assumptions, hierarchically persistent features are guaranteed to enforce such emergence in an autoregressive model trained via gradient descent.

Let \(X = x_1x_2\cdots x_n\) be a sequence of tokens generated by a latent hierarchical process \[Z\rightarrow Z_1\cdots Z_k \rightarrow X\] where local emissions depend on the parent latent (denoted by \(\operatorname{Pa}()\)) as \[p\left( X \right)= \sum_Z\sum_{Z_{1:k}} p\left( Z \right) \prod_{i=1}^{k} p\left( Z_i|Z \right) \prod_{j=1}^{n} p\left( x_j|\operatorname{Pa}\left(x_j\right) \right)\]

We make the following assumptions about the data generation process:

Let \(f_\theta\) be an autoregressive model with internal representation \[h_\theta = \left(C_1\left( X\right), \cdots, C_m\left( X\right)\right)\] where each \(C_i\) denotes identifiable architectural components (e.g., attention heads or layers). The final prediction is \[f_\theta\left( X \right)=g_\theta \left(C_1\left( X\right), \cdots, C_m\left( X\right)\right)\] The model is optimized using the expected negative log-likelihood objective to a near-global minima \(\theta^*\): \[\theta^* = \mathop{\mathrm{arg\,min}}_\theta \mathbb{E}_{X\sim {\cal D}} \left[- \log f_\theta\left(x_t|X_{<t}\right)\right]\] We make the following assumptions about the model architecture and optimization bias:

Proof: The Bayes optimal predictor satisfies: \[P^*(x_t|X_{<t})=\sum_Z P(Z|X_{<t}) P(x_t|Z)\] Then, minimizing the loss requires correctly estimating the posterior belief over \(Z\). Inability to retrieve \(Z\) from past context \(X_{<t}\) induces irrecoverable loss.

Due to Assumption 1, such a retrieval needs to be distance invariant.

Due to Assumption 3, the retrieval needs requires implementing some similarity metric \(\phi\) such that, \[\phi(X_{S_i})\approx \phi(X_{S_j})\]0◻

The need for correct estimation of the posterior belief over the latent trivially leads us to the following corollary:

That is, a language model cannot magically learn to generalize to arbitrarily complex distributions from few-shot examples. This result coincides with prior findings characterizing the nature of in-context learning [29].

Proof: It suffices to construct a measurable function \(T_i(X_{S_i})\) which is sufficient statistic of \(Z\). Let \(T_i\) be defined as \[T_i(x):= P(Z|X_{S_i}=x)\]

Formally, \(T_i(X_{S_i})\) is a random probability measure taking values in \({\cal T}:= {\cal P}({\cal Z})\), the space of all distributions defined over \(Z\).

Since \(T_i(X_{S_i})\) is a function of \(X_{S_i}\), by data processing inequality, \[I(Z;X_{S_i})\geq I(Z;T_i(X_{S_i}))\] To show the sufficiency of the statistic \(T_i\), we need to show that equality holds.

By definition of \(T_i\), for \(A\in {\cal B}_{\cal Z}\), \[P(Z\in A|X_{S_i}=x)=T(x)(A)\] By regularity, for almost every \(x\), \[P(Z\in A|X_{S_i}=x, T_i(X_{S_i})=T_i(x)) = T_i(x)(A)\] Since \(T(x)\) is a deterministic map, we can write \[P(Z\in A| T_i(X_{S_i})) = T_i(X_{S_i})(A)\] Therefore, \[P(Z\in A|X_{S_i}, T_i(X_{S_i}))=P(Z\in A|T_i(X_{S_i}))\] that is, \(T_i(X_{S_i})\) is as informative as \(X_{S_i}\), proving its sufficiency.0◻

Proof: By Lemma 1, there exists multiple sufficient statistics \(T_i(X_{S_i})\), each corresponding evidence stream \(X_{S_i}\). Following parallel additivity from Assumption 4, there exists a parametrization of \(f_\theta\) such that, \[C_i=\phi (T(X_{S_i}));\; C_j=\psi(T(X_{S_j}))\] for suitable functions \(\phi, \psi\), with equal predictive performance.

Therefore, the loss function admits a non-singleton manifold of global minimizers corresponding to different allocations of predictive load across components \(C_i\). Following Assumption 5, the implicit biases of gradient descent then favor balanced load-sharing of predictive power among components at convergence, i.e., multiple components bear non-zero information predictive of \(Z\).0◻

Implications. Theorem 1 generalizes the in-context information processing observed in induction and function vectors. A distance-invariant retrieval ensures the generalizability of induction as a mechanism. The latent similarity-based matching is an immediate precursor of function vectors: the model constructs neural digests of similar input-output mappings by estimating the relevant latent, then uses this latent for future predictions. Theorem 2 implies Hydra effect: if there exist two parallel representation components equally predictive of latent \(Z\), then ablating one of them makes the other act as a substitute, avoiding a drastic increase in loss due to ablation. Note that the validity of Theorem 2 depends strongly on the assumption that gradient descent has an implicit bias towards symmetric solutions, favoring shared predictive power across parallel components over concentration. While this is yet to be proved as a general truth for Transformers, there is supporting evidence in the context of simpler neural architectures [30], [31].

One important caution against misinterpreting these results is that we do not claim hierarchy to be a necessary property of the data generation process, but a sufficient one to elicit the phenomena under investigation. In this light, Theorem 1 does not contradict prior findings [9] demonstrating induction as a gradient flow bias of Transformers trained on Markov processes. However, it is empirically evident that hierarchical latent exerts a stronger bias towards induction head formation (see Section 4.1).

6 Discussion and Conclusion↩︎

In this work, we seek to unify the development of three mechanistic phenomena, namely induction heads, function vectors, and Hydra effect, under the lens of data generation process and optimization dynamics. We identify that hierarchical latent structure in the data generation process can be indicated as a sufficient condition for all three phenomena. Such unification of phenomena that on the surface look very different from each other can be a stepping stone towards building a more robust understanding of language models. To facilitate a forward-moving discourse, we now highlight the overarching implications of our work.

Compositionality and emergence. All three phenomena investigated in this work demonstrate emergent optimization [32], [33] — a drastic jump from a near-zero to a near-maximum value. Very often this behavior is associated with phase shift. Such drastic dynamics have been linked to the compositionality of the associated computation [32], [34]. This can be linked to learning hierarchical structures as well, both conceptually and empirically. It is natural to assume that the model’s estimation of the posterior belief over shallower latents needs to be composed for the subsequent estimation of deeper latents. This is evident in the internalized hierarchy as we show in Section 4.4.

Geometry of hierarchy. How hierarchical latents are organized within the representation space of the model is an immediate consequence of our findings. A natural candidate would be negative-curvature or hyperbolic manifolds [35]. In the context of BERT-like masked language models, the existence of an implicit hyperbolic manifold has been indicated [36]. However, there is no widely accepted proof that standard pretrained Transformer-based LMs internally learn a hyperbolic manifold without any externally imposed geometric modeling. The foundational role of hierarchical structure that we show, along with [15], begs further attention to this question: can introducing geometric priors, such as [37]–[39], lead to more efficient language models?

Redundancy, interpretability, alignment. Our theoretical results establish a strong connection between hierarchical structures in the data generation process and redundant distribution of predictive power in the model. Scaling model size and complex reasoning over increasing hierarchical structures, both being the stairway to powerful model, positively reinforce computation in redundancy. This makes intervention-based interpretability (ablate a variable and record its effect) extremely challenging. This extends to safety alignment as well: unless an alignment method can replace all possible realizations of a harmful capability, the model will remain misaligned. This calls for a newer paradigm of mechanistic interpretability.

Limitations↩︎

6.0.0.1 Expressivity of PCFGs.

A methodological limitation lies in the expressive power of PCFGs. We chose hierarchy as the single point of focus in this study, ignoring many local structural complexities native to natural language. It is possible they play a crucial role in eliciting local and/or global representational structures that remain unobserved in our setup.

6.0.0.2 Scope and Level of Analysis.

Our findings rely on existing metrics for identifying phenomena, which, although widely used, may not fully capture their underlying complexity. Alternative metrics could emphasize different structural or functional properties, leading to different interpretations. Moreover, we only briefly examine how different phenomena are interrelated or correlated, which represents an important avenue for future work. Future research should therefore compare and validate multiple metrics and explore the connections between distinct phenomena to ensure that findings are not artifacts of measurement choices. This study focuses exclusively on the pre-training phase of LLMs, ignoring the effects that arise during fine-tuning or reinforcement learning. Excluding later stages limits our understanding of how these phenomena evolve when models are adapted for real-world tasks. In principle, at least for fine-tuning, adaptation would be possible. Our validation compares training dynamics on OLMo-1B. While these parallels are informative, it remains unclear whether the same mechanistic phenomena scale identically in frontier models with trillions of parameters. Large-scale models may exhibit emergent behaviors that depend on data diversity and architectural scale not captured in our framework. Lastly, our tokenization approach diverged from that employed in large-scale language models. Specifically, we utilized a word-level tokenization scheme, whereas contemporary models typically rely on subword methods such as byte pair encoding or its variants. This distinction is nontrivial: tokenization plays a crucial role in shaping how models represent and interpret language, influencing everything from vocabulary coverage to syntactic coherence. Moreover, differences in tokenization can introduce subtle yet significant effects on model interpretability, often yielding counterintuitive behaviors in how linguistic structures and meanings are internally represented.

Acknowledgments↩︎

This work was funded by the LOEWE Distinguished Chair “Ubiquitous Knowledge Processing”, LOEWE initiative, Hesse, Germany (Grant Number: LOEWE/4a//519/05/00.002(0002)/81), as well as by the German Federal Ministry of Education and Research and the Hessian Ministry of Higher Education, Research, Science and the Arts within their joint support of the National Research Center for Applied Cybersecurity ATHENE. Computational resources were provided by the German AI Service Center WestAI.

7 Generative Processes Details↩︎

7.1 N-gram↩︎

Table 1 shows the parameterization for the N-gram data generation process.

7.2 PCFG↩︎

The PCFG production rules are defined in Figure 7 and parameterized as per Table 2.

8 Model & Training Specifications↩︎

This appendix details the specifications of our custom model based on the LLaMA 2 architecture [40]. It follows the same architectural design but uses adjusted configuration values derived from an informed, proportional downscaling relative to allenai/OLMo-1B-hf to maintain a comparable performance trajectory. We used four H100 GPUs; the total amount required for training was one day. Analysis was done using the nnsight library [41].

8.1 Loss Trajectories↩︎

Figure 6 shows that training loss for the model based on the N-gram data generation, the PCFG data generation, and the reference model all follow a power-law trajectory. Interestingly, the PCFG shows a sharp decrease after an initial flat line. Overall, the Figure shows that the loss trajectories across all three settings are comparable, thereby establishing a baseline for subsequent analyses. Without this quantitative alignment, any proceeding comparison, such as generalization behavior, would be limited, as significant deviations in loss dynamics would indicate that the models had not reached comparable representational regimes. Thus, these results serve as a sanity check confirming that all models exhibit similar convergence behavior.

References↩︎

---

1. We make our code available at: https://github.com/UKPLab/arxiv2026-hierarchical-latent-structures.↩︎