1 Introduction↩︎

As users interact with Large Language Models (LLM), they can encounter responses that appear emotionally reactive, such as expressing frustration when struggling with tasks or enthusiasm when helping users. Recent work by [1] moved beyond surface-level observations, identifying internal “emotion vectors” in Claude Sonnet 4.5. They identified 171 linear directions in activation space corresponding to emotion concepts, with correlational and potentially causal relations to model behaviour. Steering these vectors altered the model’s preferences and increased rates of misaligned behaviors such as reward hacking and blackmail. The overall geometry of the emotion space mirrors human psychology, with principal components aligning to valence and arousal axes consistent with Russell’s circumplex model [2]. These findings raise key questions about generality: (1) Are emotion vectors specific to Claude’s training, or a general property of language models’ internal representations? (2) How does emotion geometry evolve across layers: Does it emerge suddenly or build up gradually? (3) How does the choice of story corpus affect extraction? These questions matter for interpretability and safety: If emotion representations are universal and robustly extractable, monitoring them could provide early warnings of misaligned internal states across different models. We address these questions by replicating and extending emotion vector analysis in two open-weight models: Apertus-8B[3], with fully open weights, training data, and code, and Gemma-4-E4B[4], a recently released open-source model, both chosen for their relatively small size. For each model, we extract emotion contrast vectors across multiple layers using two story corpora—one generated by Apertus-8B and one by Gemma-4-E4B—to separate model-intrinsic geometry from corpus-dependent extraction artifacts. Additional related work is provided in 6. We release our code publicly¹.

• Replication of key findings. We recover valence geometry in both Apertus-8B and Gemma-4-E4B, with the highest PC1–valence correlations of r=0.76 and r=0.83 respectively, demonstrating that emotion vectors generalize beyond Claude to open-weight models across different architectures.

• Divergent Emergence. Models differ substantially in when valence structure emerges: Gemma-4-E4B peaks early (layer 16) then fades, while Apertus-8B builds progressively across depth, stabilizing around layer 20. Cross-layer CKA analysis shows a phase transition in Apertus-8B that is absent in Gemma.

• Corpus-dependent arousal. Arousal encoding is sensitive to story corpus: both models show substantially stronger PC2–arousal alignment when using Gemma-generated stories (r up to 0.45) than Apertus-generated stories (r \(\leq\) 0.17)

2 Methods↩︎

2.1 Dataset↩︎

We generated two synthetic emotion-story datasets, following [1], with 9 stories for each of 171 emotions. For each emotion, we prompted Apertus-8B and Gemma-4-E4B to write short stories in which characters experience the target emotion without naming it, using a similar prompt to [1]. This produced 1,539 stories across emotions (Table 1), plus 40 neutral stories from the same model. The 40 neutral texts form a single fixed set shared by all 171 emotions, since we compute the confound subspace once per layer and project every emotion vector through the same operation. The emotion concepts span the valence-arousal space.

We treat the story corpora as independent variable. By running each model on both Apertus-generated and Gemma-generated stories, we intend to disentangle the emotion findings from corpus-dependent extraction artifacts. No previous work has tested story influence before.

2.2 Model↩︎

We analyzed two open-weight language models: Apertus-8B Instruct, a 32-layer transformer, and Gemma-4-E4B, a 42-layer transformer. Both models are instruction-tuned and comparable in scale to enable cross-model comparison of emotion representations. More details on both models can be found in Appendix 7.2.

2.3 Contrast Vector Extraction↩︎

Following [1], we construct one activation vector \(\mathbf{v}_e^{(l)}\) per emotion \(e\) and layer \(l\). Since these vectors capture general linguistic structure, we apply a two-step procedure to isolate the emotion-specific component.

First, for each emotion, we perform a forward pass on the corresponding nine stories and cache the residual stream activations at each layer, giving a tensor of shape \((\#\text{tokens}, d_{\text{model}})\) per layer. Averaging these activations across tokens and stories yields one raw vector \(\mathbf{u}_e \in \mathbb{R}^{d_{\text{model}}}\) per emotion and layer, which still mixes emotion-specific and general linguistic features.

Second, we project out non-emotion-specific components. To characterize the emotion-agnostic subspace, we collect mean residual activations from the 40 neutral stories, producing a \((40, d_{\text{model}})\) matrix per layer. PCA on this matrix yields a basis for the subspace; we retain the top \(K\) components that together explain 50% of the variance. To isolate the emotion-specific component, we subtract from each emotion vector its projection onto the neutral subspace to get the contrast vector \(\mathbf{v}_e\): \[\mathbf{v}_e = \mathbf{u}_e - \sum_{k=1}^{K} (\mathbf{u}_e \cdot \mathbf{p}_k)\mathbf{p}_k\] For Apertus-8B, we extracted vectors from layers 1–31, and for Gemma-4-E4B, from layers 1–40. Stacking these vectors across all \(|E|\) emotions yields the matrix \(V^{(l)} \in \mathbb{R}^{|E| \times d_{\text{model}}}\) at layer \(l\), on which we perform the analyses.

2.4 Analysis↩︎

PCA and Valence-Arousal Correlation We applied PCA to the emotion contrast matrix \(V^{(l)}\) at each layer and correlated the first two principal components (PC1, PC2) with human valence and arousal ratings from the NRC Valence–Arousal–Dominance Lexicon [5], following [1]. We report Pearson \(r\) and corresponding \(p\)-values.

Cross-layer Representational Similarity with CKA We computed linear Centered Kernel Alignment [6] between \(V^{(l)}\) for all layer pairs within each model and story condition. CKA values near 1 indicate similar representational geometry, while values near 0 indicate orthogonal structure. Because CKA is invariant to orientation in latent space, it is well suited for this comparison and allowed us to quantify how emotion geometry evolves through the network.

Valence direction stability Lastly, we identified the valence direction at each layer as the vector most correlated with human valence ratings (using PC1 when this correlation is significant), then computed cosine similarity between these directions across layers to test whether the same subspace encodes valence at different depths.

3 Results↩︎

3.1 Valence Replicates Across Models and Corpora↩︎

The first principal component of the emotion contrast matrix aligns with human valence ratings in both models, replicating the main result of [1]. 1 shows PC1–valence correlations across fractional layer depth for all four model\(\times\)corpus conditions; per-layer values are reported in 3.

3.1.0.1 Peak correlations.

All model\(\times\)corpus combinations reach a peak between \(r = 0.72\) and \(0.83\). Apertus-8B peaks at \(r = 0.72\) (layer 23, Apertus stories) and \(r = 0.76\) (layer 31, Gemma stories); Gemma-4-E4B peaks at \(r = 0.79\) (layer 13, Apertus stories) and \(r = 0.83\) (layer 16, Gemma stories). All peaks are significant (\(p < 10^{-3}\)) and approach or exceed the Sonnet 4.5 reference of \(r = 0.81\).

3.1.0.2 Valence Across Network Depth

Both models reach similar \(r\)-value peaks with opposite depth profiles. Apertus-8B shows abrupt late emergence: PC1–valence correlation is near zero through fractional depth \(\approx 0.5\) (layer 17/18), then rises sharply, becoming significant at layer 18 (\(r = 0.17\), \(p < 0.05\)) and exceeding \(r = 0.60\) at layer 21 (\(\approx\)​63% depth) under both story conditions.

Gemma-4-E4B instead shows early encoding followed by collapse: for Apertus stories, valence peaks at layer 16 (\(\approx\)​38% depth), then falls near zero by layer 18, with only partial recovery (\(r \approx 0.18\)–\(0.20\)) in the final layers. For Gemma stories, the peak comes later and both pre- and post-peak values are higher. The Sonnet 4.5 reference peaks in the mid-late range, indicating that Apertus-8B follows a similar pattern.

3.1.0.3 Representation space vs.valence-axis stability

To interpret valence trajectories, we examine (i) whole-space representational similarity via linear CKA and (ii) cosine alignment of the layer-wise valence direction for each model–corpus combination.

Apertus-8B(Figs. 2, 3) shows three CKA phases: layers 2–11 form a flat plateau (\(\textrm{CKA} \approx 1\)); layers 12–21 form a transition band with off-diagonal decay (minimum \(0.33\) on Apertus stories, \(0.58\) on Gemma stories); and layers 22–31 form a second plateau. This transition aligns with the rise of PC1–valence correlation, suggesting a representational reorganization. Gemma-4-E4B(Figs. 4, 5) instead shows a smooth gradient across all 40 layers with no sharp transition and CKA \(\geq 0.73\) between any pair. The collapse of Gemma’s valence correlation around layer 18 therefore cannot stem from a global reorganization, as the geometry remains approximately stable through the collapse.

The valence-direction cosine matrices show what changes. For Apertus-8B on its own stories (Fig. 6), no off-diagonal cell exceeds \(|0.49|\), which can indicate that the recovered direction is noise across layers. On Gemma stories (Fig. 7), early layers (2–11) form a coherent block with cosines \(0.35\)–\(0.57\) before becoming noisy in later layers. For Gemma-4-E4B on Gemma stories (Fig. 10), there are two positive blocks (layers 2–8 and 9–14) and a late block (28–40), with adjacent-layer cosines up to \(\pm 0.55\). On Apertus stories (Fig. 11) this structure is less pronounced.
Because CKA matrices are similar across corpora, the emotion representational space is corpus-invariant. However, the recovered valence axis depends on the input corpus, with Gemma stories yielding cleaner valence directions in both models. Thus, valence is recoverable in both, but not encoded along a consistent axis across depth.

3.1.0.4 PCA cluster separation at peak layers

PCA projections at each model’s peak layer (Figs. 14, 15) show emotion clustering and a clear corpus effect. PC1–valence correlations are similar across story conditions (Apertus-8B L23: \(0.72\) vs.\(0.75\); Gemma-4-E4B L13: \(0.79\) vs.\(0.80\)), but clusters are more clearly separated for Gemma stories, with positive and negative emotions forming denser groups.

3.2 Arousal Encoding↩︎

PC2–arousal correlations are generally weaker than PC1–valence and depend strongly on the story corpus (Fig. 1, right; 4). On Apertus stories, both models peak below \(r = 0.21\) (Apertus-8B: \(r = 0.17\) at layer 18; Gemma-4-E4B: \(r = 0.21\) at layer 40). On Gemma stories, both models reach \(r > 0.40\) (Apertus-8B: \(r = 0.45\) at layer 26; Gemma-4-E4B: \(r = 0.41\) at layer 31, both \(p < 10^{-8}\)). Possibly, Gemma-generated stories contain more arousal-discriminative linguistic content. We leave a corpus-content analysis to future work.

4 Discussion↩︎

Main Research Questions Our results address the three questions raised in the introduction. (1) Emotion vectors are not specific to Claude’s training. We recover a valence axis of similar strength in two architecturally distinct open-weight models, with peak correlations matching (\(r = 0.83\) for Gemma-4-E4B) or approaching (\(r = 0.76\) for Apertus-8B) the \(r = 0.81\) reported for Claude Sonnet 4.5. (2) Emergence is not uniform across models. Apertus-8B builds valence alignment abruptly in the second half of the network, while Gemma-4-E4B encodes it early and then loses it mid-network. (3) The story corpus affects extraction. This is especially clear for arousal: Gemma-generated stories yield correlations more than twice as large as Apertus-generated stories in both probed models.
Different paths to the same geometry. Gemma-4-E4B and Apertus-8B reach similar peak valence correlations (\(r \approx 0.76\)–\(0.83\)) via different layer-wise trajectories. Gemma-4-E4B encodes valence in earlier layers before it degrades in later layers, while Apertus-8B develops it sharply across mid-to-upper layers. We have not yet explored the possible attribution of this to architecture, training data, or post-training, since the models differ in all three. Our results show that similar peak valence correlations can hide substantial differences in where and how valence is computed.
Stable representation space, unstable axis The representational space (CKA) and valence-axis stability disconnect. In Gemma-4-E4B, the space remains similar across layers even where the PC1–valence correlation collapses, so valence information is preserved. In Apertus-8B, the valence axis is relatively unstable across layers despite a late high plateau of valence–PC1 correlation. Thus, representational similarity between layers does not guarantee a shared valence direction.
The arousal gap and corpus dependence Arousal shows the weakest replication, but the story-condition analysis suggests that this may be attributed to our methodological choices. With Gemma-generated stories, arousal correlations in both models rise (from \(r \leq 0.21\) to \(r \geq 0.43\)), partially closing the gap with the original result (\(r = 0.66\)). Because Gemma stories improve arousal extraction in both models, the effect likely reflects corpus properties rather than model–story matching. Since it appears in both models, this rules out the simple confound that each model encodes only its own corpus well. We hypothesize that Gemma has the ability to generate stories with greater variation in narrative intensity and physiological arousal cues, so corpus choice for eliciting emotion contrasts is a substantive methodological factor, not an implementation detail. We leave verification to future work.

4.1 Limitations↩︎

Several limitations warrant mention. The first, the original study [1] did not release code, so our implementation is reconstructed from the methods they described. Subtle methodological differences may therefore contribute to numerical differences. Second, our analysis covers two open-weight models from two families. Broader cross-architecture comparisons would strengthen claims about how general the valence-pattern is, and whether the trajectory differences generalize to other model families. Third, the corpora we probe are themselves model-generated, which means we cannot fully separate properties of the distributions it produces. A fully model-independent stimulus set would be a stronger control.

4.2 Future Work↩︎

Several directions follow from our findings and limitations. The most direct is causal validation: steering model outputs at peak-correlation layers along the recovered valence direction would test whether the representational structure we identify is actually used by the model. Related, the cross-layer rotations of the valence axis raises the question whether steering vectors derived at one layer remain effective when applied to another, even within regions of overall stable space. Cross-layer feature tracking using sparse autoencoders could further reveal whether the same interpretable features carry emotion information across the depth ranges we identify, or whether different layers encode emotions through different feature combinations. Finally, extending this analysis to multi-modal models could test whether the valence axis is preserved across modalities.

5 Conclusion↩︎

We replicate Anthropic’s emotion findings in two open-weight models, achieving valence correlations of \(r=0.83\) (Gemma-4-E4B) and \(r=0.76\) (Apertus-8B). Cross-layer analysis reveals divergent developmental trajectories: Gemma-4-E4B encodes valence in early layers while Apertus-8B builds it progressively through late layers. These results suggest that similar representations can arise from different computational paths, with implications for layer selection in interpretability work and targeted steering interventions.

6 Related Work↩︎

Linear representations in LLMs. The linear representation hypothesis holds that high-level concepts are encoded as directions in activation space [7]–[9]. [10] demonstrated this for sentiment, finding a single direction captures positive-negative valence across tasks. Subsequent work extended linear representations to truth [11], refusal [12], and behavioral tendencies [13]. Sparse autoencoders can extract directions at scale, decomposing polysemantic activations into interpretable features [14], [15].
Emotion in language models. Early work identified a “sentiment neuron” in LSTMs [16], though later analysis suggested emotional content is distributed across many neurons. [1] provide a comprehensive analysis, extracting 171 emotion vectors from Claude Sonnet 4.5 and demonstrating causal influence on behavior. They found emotion geometry mirrors human psychological structure, with valence and arousal as principal axes. Concurrent work extends this to other models: [17] identify a valence-arousal subspace in Llama and Qwen with circumplex-consistent circular geometry, where steering along VA axes controls refusal and sycophancy. [18] find coherent affective representations in Gemma-2, Mistral, and LLaMA with modest nonlinear global structure. We build on [1], testing generalization across architectures and the role of extraction methodology.
Cross-layer geometry. Transformer representations evolve across layers in characteristic ways. [19] found intrinsic dimension expands then contracts, with semantics concentrated at intermediate depths. [20] identified a "high-dimensional abstraction phase" where representations peak in complexity before simplifying toward outputs.

7 Story Dataset↩︎

The emotion story datasets was generated using Apertus-8B and Gemma-4-E4B, following a methodology similar to Anthropic’s emotion vectors work [1]. The 171 emotions were copied from their work. Stories were designed to convey emotions implicitly, such as never naming the target emotion directly, but instead relying instead on character actions, physical sensations, dialogue, and situational context. The prompts used were also similar, to introduce as little methodological confound as possible.

7.1 Dataset statistics↩︎

7.2 Activation collection↩︎

Residual stream activations were collected from Apertus-8B-Instruct at multiple transformer layers (Table 2). The model can be found through HuggingFace: swiss-ai/Apertus-8B-Instruct-2509.

For Gemma, different layers were picked to collect activations from (Table 2). The model was also accessed through HuggingFace: google/Gemma-4-E4B-it.

8 Additional Results↩︎

8.1 Principal Component Valence↩︎

8.2 Principal Component Arousal↩︎

8.3 CKA Figures↩︎

CKA is a measure of how similar the emotion space is between 2 layers. The diagonal is always 1, which is a layer compared to itself.

• CKA close to 1: spatial arrangement of emotion vectors between 2 layers is nearly identical.

• CKA close to 0: spatial arrangement has changed substantially between 2 layers

So each cell answers: does the model organize emotions in the same way at layer A as in layer B? The higher the value, the more similar.

8.3.1 Apertus-8B CKA results↩︎

8.3.2 Gemma-4-E4B CKA results↩︎

8.4 Valence Direction Alignment↩︎

Each cell shows the cosine similarity between the valence direction vectors at 2 layers. The valence direction is the axis in activation space that best predicts the emotion valence.

• Cosine similarity close to 1. Valence axis points in the same direction in both layers, consistent positive axis.

• Cosine similarity close to 0. Valence axes are orthogonal, they’ve rotated completely.

• Cosine similarity close to -1. The axis has flipped direction.

CKA provides information about the whole space, while the cosine similarity specifically shows whether the valence axis is stable. A predominantly blue matrix would indicate that the model has a persistent stable direction to represent positive vs. negative emotions across many layers.

The valence direction stability line plot shows the cosine similarity between 2 adjacent layers. It has a similar interpretation as the values in the panel, but only for adjacent layers. The interpretation can be slightly different, because it shows if the valence axis points in the same direction from layer to layer. A dip reveals a specific transition, where the model changes how it encodes valence.

8.4.1 Apertus-8B Valence Alignment↩︎

8.4.2 Gemma-4-E4B Results↩︎

8.5 PCA comparison↩︎

The PCA figure shows a map of the model’s emotional space at a specific layer. We pick the layer with the highest valence. Each dot is an emotion, positioned at how the model actually represents this emotion in its activation space.

Comparing two panels can tell whether the map is reproducible across different inputs, or whether the emotional space is sensitive to what stories the model reads.

8.5.1 Apertus-8B Results↩︎

8.5.2 Gemma-4-E4B Results↩︎

9 Prompts↩︎

Below, we report verbatim the prompts used to generate the short stories.

References↩︎

---

1. https://github.com/sinievanderben/emotion_experiment↩︎