1 Introduction↩︎

The integration of deep learning into the creative arts has produced state-of-the-art autoregressive models capable of respecting long-term structural dependencies [1]. However, these models often function as “black boxes” [2] where the relationship between high-dimensional internal activations and discrete output attributes is obscured. For applications requiring precise intervention such as algorithmic composition or signal synthesis, global conditioning often lacks the granularity needed for specific manipulation of internal states. There is a pressing need for precise, inference-time intervention methods that do not require computationally expensive retraining without disrupting the entire generative process [3].

The field of mechanistic interpretability [4] seeks to reverse-engineer neural networks to understand the algorithms they implement. A central tenet is the Linear Representation Hypothesis, which posits that high-level concepts are represented as vectors within the activation space [5], [6]. While validated in Large Language Models (LLMs), its application to symbolic signal domains remains nascent. Identifying these vectors enables Activation Steering, the process of intervening on the model’s internal activations during the forward pass to bias the generation toward a desired attribute [7]. While recent advancements have led to unified frameworks capable of high-quality generation via multi-modal conditioning, such as Seed-Music [8], XMusic [9], and the flow-matching based JASCO [10], these systems often require extensive retraining or complex architectural changes. This work addresses the problem of inference-time steerability. Models such as the Multitrack Music Transformer (MMT) [11] utilize specialized data representations to encode discrete musical events into token sequences, learning the high-dimensional probability distributions of musical corpora like the Symbolic Orchestral Database (SOD) [12]. Rather than focusing on computationally expensive fine-tuning or prompt engineering, we apply mechanistic interpretability to reverse-engineer the latent representations of a pre-trained MMT, focusing on Average Note Pitch and Average Note Duration.

Our contribution provides a layer-wise sensitivity analysis to map attribute encoding topology and investigate linear steerability within the MMT and evaluates steering dynamics across the network depth. We implement orthogonalization methods to disentangle correlated features, ensuring independent modulation and precise dual-concept control. Audio examples and code on https://giannisprokopiouorfium.github.io/music-transformer-sae/.

2 Related Work↩︎

Symbolic music generation has transitioned from rule-based systems to sophisticated deep learning architectures, with the Transformer [13] architecture revolutionising the modeling of long-term structural dependencies. The MMT introduced a compact 6-tuple event representation, incorporating type, beat, position, pitch, duration, and instrument, to significantly reduce sequence lengths compared to prior formats like REMI+ [14]. This efficiency makes MMT an ideal backbone for real-time steering research.

Activation engineering [15] facilitates this shift by optimizing internal model states rather than input prompts grounded in the Linear Representation Hypothesis [16]. Research on LLMs has demonstrated that injecting these vectors during the forward pass can deterministically bias outputs without weight optimization [17]. The Difference-in-Means (DiffMean) [7], [18] methodology serves as a robust baseline for extracting these vectors by capturing the centroid difference between contrasting concept clusters. Recent benchmarks like AxBench [19] confirm that DiffMean often outperforms complex alternatives like Sparse Autoencoders (SAEs) for steering tasks.

In the musical domain, mechanistic interpretability [4] is an emerging frontier. Early attempts at controllable synthesis utilized latent regularization in architectures like Music FaderNets [20], or compositional augmentation of Transformer architectures [21]. Panda et al. [22] demonstrated that steering residual streams and attention heads can enable fine-grained style transfer and genre fusion. Modern techniques such as SMITIN [23] employ self-monitored interventions with classifier probes to dynamically adjust steering strength, while MusicRFM [24] analyzes internal gradients to identify concept directions for autoregressive models. Most relevant to our study is the work of Facchiano et al. [25], which applies activation patching to interpretable latent directions, confirming that localized interventions can effectively modulate high-level musical attributes. In their work they applied activation patching to MusicGen [26] to control tempo and timbre, identifying mid-range layers as the most effective locus for intervention. Our research distinguishes itself by applying these concepts to the symbolic domain, where the discrete nature of MIDI-based attributes allows for mathematically precise evaluation and manipulation.

A persistent challenge in multi-attribute steering is the conceptual interference between correlated features [27], such as the natural entanglement of pitch and duration. While techniques involving geometric operations, such as angular steering [28], have been proposed to define stable steering planes in latent space, and the Composer Vector method [29] that utilizes latent space arithmetic to fuse styles, we build upon these mathematical foundations by applying Gram-Schmidt Orthogonalization [30]. By explicitly decoupling correlated features, this approach prevents interference during generation and addresses the entanglements typically learned by the model’s residual stream, ensuring independent control over discrete musical attributes.

3 Methodology↩︎

3.1 Signal Representation and Model Infrastructure↩︎

We utilize the pre-trained MMT, a decoder-only Transformer architecture trained on the SOD dataset. The model processes musical events as discrete 6-dimensional tuples, which are embedded into a 512-dimensional continuous vector space. Each event tuple consists of:

• Type (4 tokens): Categorizes the event (note, time-shift, end-of-song, instrument-change).

• Beat (257 tokens): Integer beat position (0 to 256).

• Position (384 tokens): Sub-beat resolution at intervals.

• Pitch (128 tokens): MIDI values ranging from 0–127.

• Duration (768 tokens): Note length in ticks, from staccato (1 tick) to sustained (768).

• Instrument (128 tokens): MIDI program numbers for orchestral instrumentation (0 to 127).

Our steering interventions target Pitch (dimension 4) and Duration (dimension 5) for interpretability by modifying the residual stream activations that influence the probability distribution over these discrete vocabularies at generation time.

3.2 Latent Vector Extraction and Concept Definition↩︎

To compute steering vectors via DiffMean, we defined “High” and “Low” concept thresholds by analyzing the statistical distribution of the SOD corpus. Thresholds (Table 1) were empirically set at the 20th and 80th quantiles, yielding robust, representative, non-overlapping clusters and stable latent directions across alternative extreme quantile choices.

We intercept the residual stream at the output of every transformer decoder block using forward hooks as it contains the model’s semantic musical representations, rather than the output layer that only projects to vocabulary space for next-token prediction. For a given layer \(l\), we extract the summary activation \(h\) corresponding to the last valid token that has full sequence context. The steering vector \(v^{(l)}\) is calculated as shown in 1 as the difference between the means of the high-attribute and low-attribute set. Here, \(N_{pos}\) and \(N_{neg}\) denote the number of samples in the high and low concept clusters balanced at \(1,280\) samples each based on the previously defined quintile thresholds, while \(h(x)^{(l)}\) represents the activation vector for a specific input segment \(x\) at layer \(l\).

\[\label{eq:steering95vector} v^{(l)} = \frac{1}{N_\text{pos}} \sum_{i=1}^{N_\text{pos}} h(x_\text{pos, i})^{(l)} - \frac{1}{N_\text{neg}} \sum_{j=1}^{N_\text{neg}} h(x_{\text{neg}, j})^{(l)}\tag{1}\] Our sensitivity analysis reveals that Note Pitch is most linearly separable at Layer 11, while Note Duration shows peak separation at Layer 2 shown in Fig. 1, suggesting that rhythmic features are encoded earlier in the network than melodic ones.

3.3 Inference-Time Generation Steering↩︎

To steer the model’s behavior during inference, we modify the hidden states \(h^{(l)}\) by injecting the steering vector \(v^{(l)}\) scaled by a coefficient \(\alpha\) as shown in 2 : \[\label{eq:steering} h_\text{steer}^{(l)} \leftarrow h^{(l)} + \alpha v^{(l)}\tag{2}\] We evaluated several injection strategies: All-to-All (layer-specific vectors injected at every layer), One-to-All (broadcasting a single optimal direction), and Some-to-Some (targeting sensitive sub-layers). Empirical results indicate that the All-to-All strategy yields the most robust trade-off between steering potency and generation quality for the symbolic MMT.

3.4 Dual Steering and Orthogonal Disentanglement↩︎

A critical challenge in multi-attribute steering is conceptual interference; for example, high-pitch passages in the training corpus often statistically correlate with shorter durations. We measured an average absolute cosine similarity of \(0.49\) between Pitch and Duration vectors across the network, peaking at \(0.81\) in Layer 3. To enable independent manipulation and mitigate interference between features, we evaluated four strategies to construct the combined steering vector \(v_\text{combined}\):

1. Simple Addition: A baseline linear combination where vectors are added without modification in 3 : \[\label{eq:simple} v_\text{combined} = \alpha_{p} v_{p} + \alpha_{d} v_{d}\tag{3}\]

2. Gram-Schmidt (Pitch Priority): To preserve pitch purity, the duration vector is orthogonalized against the pitch vector (\(v_{d}^{\perp}\)) shown in 4 : \[\label{eq:grahm} v_{d}^{\perp} = v_{d} - \frac{v_{d} \cdot v_{p}}{\|v_{p}\|^2} v_{p} \quad \Rightarrow \quad v_\text{combined} = \alpha_{p} v_{p} + \alpha_{d} v_{d}^{\perp}\tag{4}\]

3. Gram-Schmidt (Duration Priority): Conversely, we prioritize rhythmic stability by orthogonalizing the pitch vector against duration (\(v_{p}^{\perp} = v_{p} - \text{proj}_{v_{d}}(v_{p})\)).

4. Symmetric Orthogonalization: We utilize Singular Value Decomposition (SVD)  [31] to define a subspace where both vectors are mutually orthogonal, maximizing simultaneous independence.

4 Experimental Evaluation↩︎

To rigorously assess the steerability of symbolic musical attributes, we conducted experiments across two paradigms:

• Unconditional Generation: Sequences are generated from an empty context to isolate the steering vector’s influence on the model’s intrinsic priors. This measures the coherence and structural integrity when governed solely by pretrained weights and the injected vector.

• Conditional Generation: To evaluate the steering vector’s capacity to override strong local contexts, we rank SOD training tracks by average pitch or duration in the first 16 beats and select extreme cases. These 16 beats are used as a conditioning prefix to establish a statistically strong autoregressive prior, after which we attempt to steer the continuation in the opposite direction (e.g., imposing low pitch following a high-pitch context).

Performance is quantified via Steering Success, percentage of generations successfully moving in the intended direction, and Quality Degradation (\(\delta\)), the cumulative absolute deviation from SOD baselines: Pitch Class Entropy (2.974), Scale Consistency (92.26%), and Groove Consistency (93.05%). These metrics are inherently weighted toward pitch-related factors and thus serves as a proxy for human perception, \(\delta\) correlates directly with melodic integrity; a \(\delta \approx 10\) indicates severe dissonance akin to random note generation, whereas \(\delta \le 3\) remains perceptually coherent to listeners. Generations used temperature \(1.0\) and Top-K filtering (filter_thres \(= 0.9\)).

4.1 Single-Attribute Unconditioned Dynamics↩︎

4.1.1 Pitch Steering Dynamics↩︎

Unconditioned pitch steering trials revealed a robust and predictable linear response with \(R^2 = 0.8154\). Statistical analysis of 50 generations across the alpha grid between \(-2\) and \(2\) produced a Pearson correlation coefficient of \(r = 0.9030\) (\(p = 0.0009\)), indicating that roughly \(82\%\) of the variance in output pitch is directly attributable to steering strength. The linear slope was estimated at \(+13.35\) semitones per unit of \(\alpha\). As shown in Table 2, the effect demonstrated asymmetry regarding the baseline mean of \(65.73\): positive steering (\(\alpha = +2.0\)) shifted the mean pitch by \(+15.50\) semitones (\(+23.6\%\)), whereas negative steering (\(\alpha = -2.0\)) induced a shift of \(-29.00\) semitones (\(-44.1\%\)).

4.1.2 Duration Steering↩︎

Duration steering in the same configuration exhibited a monotonic relationship with a Pearson correlation of \(r = 0.9263\) (\(p = 0.0003\)) and linear response with \(R^2 = 0.8580\). The linear model suggests a rate of change of approximately \(+10.77\) ticks per \(\alpha\). Response was asymmetric; negative \(\alpha\) values hit a physical lower at \(\sim 3.08\) ticks (a \(59.0\%\) reduction), positive \(\alpha\) values produced expansions of up to \(+406.9\%\) (\(\alpha = +2.0\)) as shown in Table 2.

3pt

4.2 Single Context Override Contrasting Scenarios↩︎

To rigorously evaluate the robustness of our steering vectors, we designed experimental contrasting scenarios to test whether internal activations can override strong autoregressive context. We utilize a 16-beat conditioning prefix from training samples exhibiting extreme attribute values and generate 512-token continuations while injecting steering vectors across a comprehensive grid of strengths (\(\alpha \in \{\pm 0.5, \pm 0.75, \pm 1.0, \pm 1.25, \pm 1.5\}\)). Alphas were filtered per scenario to test only directionally-relevant combinations (e.g., negative \(\alpha\) for High\(\to\)Low transitions).

4.2.1 Pitch Context Override↩︎

We evaluated Low\(\to\)High (conditioning pitch 32.9–47.8 MIDI) and High\(\to\)Low (conditioning pitch 68.4–82.7 MIDI) scenarios. Vectors achieved a \(93.4\%\) overall success rate with an average magnitude shift of \(31.7 \pm 4.3\) semitones (\(\sim\)​2.6 octaves). We observed a Directional Asymmetry where upward steering was more effective (\(96.1\%\) success) than downward (\(85.6\%\)), suggesting a model bias toward lower pitch ranges. Optimal success was achieved at \(\alpha_{pitch} = +0.75\) for upward success with \(0.02\) degradation and \(-1.25\) for downward with \(0.05\) degradation.

4.2.2 Duration Context Override↩︎

Duration tests evaluated Short\(\to\)Long (conditioning duration 1.87–6.40 ticks) and Long\(\to\)Short (conditioning duration 14.89–17.0 ticks) transitions, an overall success rate of \(91.7\%\) with an average magnitude shift of \(12.1 \pm 1.2\) ticks. For positive \(\alpha\), Short\(\to\)Long reached \(97.8\%\) success at \(\alpha_{dur} = +0.75\). For negative \(\alpha\), Long\(\to\)Short achieved \(99.4\%\) success at \(\alpha_{dur} = -1.0\), bounded by the 3-tick floor. Degradation for expansion (\(\delta\): \(1.8\)–\(3.4\)) scaled sub-linearly, while compression degradation scaled linearly (\(2.1\)–\(4.3\)), suggesting the model’s internal representations are optimized for moderate-to-long durations.

4.3 Layer Sensitivity and Injection Strategy↩︎

To identify the optimal locus of intervention, we conducted systematic grid searches across the transformer’s depth, comparing All-to-All (systemic bias), One-to-All (single broadcast), and Some-to-Some (targeted groups). For Pitch, Layer 10 was most potent for reduction, while Layer 11 was optimal for increase. For Duration, later layers (Layers 8–11) exhibited the highest sensitivity. Empirical knowledge across both attributes shows that while specific layers provide targeted control for either decrease or increase, the All-to-All strategy remains the most robust intervention paradigm. By introducing a systemic bias in the residual stream, it achieved the superior trade-off, providing balanced control in either increase or decrease nudging the global context for both attributes with minimal degradation compared to the localized strategies.

4.4 Dual Steering and Disentanglement Results↩︎

4.4.1 Unconditioned Strategy Validation↩︎

Using a validation grid of \(1,600\) generations, we compared four composition strategies to address conceptual interference. Success is measured via the Dual Steering Success and Quality Degradation (\(\delta\)). Results in Table 3 show that Gram-Schmidt (Pitch Priority) outperformed other methods, achieving 88.5% success with the lowest degradation (\(\delta=2.14\)). In contrast, Symmetric Orthogonalization via SVD, implemented by extracting the scaled orthonormal basis from the \(U\) and \(S\) matrices of the stacked vector matrix \(V=USV^T\), underperformed at 79.3%. This suggests that symmetric rotation dilutes both concept directions, whereas Gram-Schmidt preserves the primary pitch vector’s integrity. This supports an inherent hierarchy where pitch serves as a fundamental anchor. As Fig. 2 shows, each \(alpha\) steers the expected corresponding features while showing an optimal configuration at moderate positive alphas (0.75–1.25 range) where dual steering effectiveness is maximized with minimal conceptual interference.

4.4.2 Conditioned Dual Steering↩︎

For conditioned dual steering, we forced the model to counteract contradictory 16-beat conditioning context from training samples that were fell into both categories calibrated thresholds using a 10\(\times\)​10 alpha grid (\(\alpha \in \{\pm 0.5, \pm 0.75, \pm 1.0, \pm 1.25, \pm 1.5\}\)) that was filtered per scenario to test only directionally relevant combinations under Gram-Schmidt Orthogonalization (Pitch Priority) with All-to-All layer injection. The results are shown in Table 4 for the different contrasting scenarios.

Emprirically, the best configuration for dual steering contains \(\alpha_{pitch} \in [\pm 0.5, \pm 1.0]\) and \(\alpha_{dur} \in [\pm 0.5, \pm 0.75]\). Beyond \(|\alpha| > 1.25\), degradation increases exponentially. Key insights include:

1. Directionality Asymmetry: Upward steering is significantly easier (\(96.1\% success\)) than downward (\(82.2\%\)), reflecting the training distribution’s melodic bias.

2. Quality Cost: The High/Short \(\to\) Low/Long scenario exhibited \(2.8\times\) higher degradation than its inverse (5.80 vs 1.33), indicating high costs for fighting extreme high-pitch contexts.

3. Context Dominance: Steering successfully overrode local context in \(88.6\%\) of trials on average, with effects (2–3 octaves) far exceeding typical transposition ranges.

5 Conclusions↩︎

This study explores the Linear Representation Hypothesis in symbolic music, confirming that the MMT encodes attributes such as pitch and duration as linear latent directions, which can be used for activation steering as a training-free method for precise musical control. The systemic All-to-All injection strategy provides the most robust steering, successfully overriding extreme autoregressive contexts. By applying Gram-Schmidt Orthogonalization, we decoupled entangled features to enable novel combinations. Future work will utilize SAEs, adaptive or feedback-based control and reverse-prompting to isolate monosemantic features for abstract concepts.

Acknowledgment↩︎

This research was funded by the European Union’s Horizon Europe research and innovation programme under the AIXPERT project (Grant Agreement No. 101214389), which aims to develop an agentic, multi-layered, GenAI-powered framework for creating explainable and transparent AI systems.

References↩︎