1 Introduction↩︎

Sequential recommendation (SR) [1]–[3] focuses on predicting a user’s next preferred item by modeling their interaction history. Recently, Large language models (LLMs) have been integrated into sequential recommender (SR) models to enrich user and item representations with rich semantic knowledge. Notably, the high inference cost of LLMs has motivated the LLM-as-Extractor [4]–[7] paradigm, which substitutes randomly initialized item embeddings of conventional SR models with LLM-generated representations for downstream fine-tuning.

As an early study, LLM2X [4] transfers LLM-generated embeddings into SR models by applying a simple principal component analysis (PCA) followed by fine-tuning. However, this strategy provides limited control over how LLM semantic representations adapt to collaborative signals. Recent studies [5], [6], [8] have thus focused on improving the adaptation of LLM embeddings to SR backbones. AlphaRec [6], LLMEmb [5], and WhitenRec [9] employ MLP projection layers to align the LLM semantic space with collaborative signals. More recently, AlphaFuse [7] freezes the semantically meaningful subspace while reinitializing the remaining dimensions to encourage collaborative learning. While these methods enhance adaptation to SR backbones, they largely overlook a more fundamental factor: the importance of properly initializing pretrained embeddings before fine-tuning.

In this paper, we investigate the geometry of LLM-generated embeddings with respect to anisotropy [10]. This property fundamentally influences how effectively LLM semantics adapt to collaborative signals during fine-tuning. Prior studies [9] have shown that Sentence-BERT embeddings [11] suffer from strong anisotropy, collapsing into a few dominant components. In Figure 1 (a), we observe that LLM-generated embeddings exhibit the same phenomenon. Most vectors are concentrated within a narrow cone, causing them to cluster tightly and lose representational diversity. To alleviate this, WhitenRec [9] applies the whitening transformation that enforces perfect isotropy. However, this rigid operation collapses the eigenvalue hierarchy that reflects semantic importance, leading to substantial semantic distortion (Figure 1 (b)). These observations highlight the need for a more flexible and semantically faithful approach to addressing anisotropy in LLM-generated embeddings.

To this end, we introduce Anisotropy-Controllable Embedding (ACE), which provides continuous and semantically consistent control over the anisotropy of LLM-generated embeddings. ACE is built upon a linear autoencoder (LAE), which regulates the intrinsic spectral behavior of the embedding space. Through the spectral characterization of the LAE solution, we observe that its two objective components play distinct geometric roles: (i) the reconstruction objective that preserves the semantic directions encoded in the original LLM embeddings, and (ii) the \(\lambda\)-weighted regularization term, which directly controls the magnitude of singular values, suppressing overly dominant semantic directions and modulating anisotropy in a continuous manner. Together, ACE yields a well-conditioned embedding space that mitigates geometric imbalance while maintaining the underlying semantic hierarchy for stable learning. As shown in Figure 1 (c), it empirically reduces directional over-concentration without causing semantic distortion. Our extensive experiments further demonstrate that ACE consistently improves LLM-enhanced SR models, achieving performance gains of up to 12.4% and 11.8% in Recall@20 and NDCG@20, respectively.

2 Preliminaries↩︎

Problem Definition. Let \(\mathcal{U}=\{u_1, \dots, u_m\}\) and \(\mathcal{I}=\{i_1, \dots, i_n\}\) denote the sets of \(m\) users and \(n\) items. For an arbitrary user \(u\in\mathcal{U}\), the sequence \(S_u=[i_1, i_2, \text{...}, i_{|S_u|}]\) represents the items that the user has interacted with in chronological order. Given \(S_u\), the goal of SR is to predict the next item \(i_{|S_u|+1}\) that the user is likely to prefer.

LLM-as-Extractor Paradigm. Recent advances in LLMs enable the usage of semantically rich item embeddings for SR. Given an item and its textual attributes, an LLM encoder produces an embedding \(e \in \mathbb{R}^d\) for each item, and stacking these embeddings yields the item embedding matrix \(\mathbf{E}= [e_1, e_2, …, e_n]^\top \in \mathbb{R}^{n \times d}\), which forms the LLM-generated representation space [7]. These embeddings are typically high-dimensional (e.g., \(d = 3072\)) and therefore need to be projected into a lower-dimensional space (e.g., \(k = 128\)) using PCA [4] or MLP projection [5], [6] for compatibility with SR models.

Embedding Anisotropy Problem. Despite the rich semantics, LLM-generated embeddings are known to be highly anisotropic [10], meaning that most embedding vectors lie in a narrow region of the space and align along a few dominant directions. Given the covariance matrix \(\text{Cov}(\mathbf{E}) = \frac{1}{n}(\mathbf{E} - \mu)^\top (\mathbf{E} - \mu)\), where \(\mu \in \mathbb{R}^{d}\) is the mean embedding vector, the distribution of its eigenvalues provides a direct measure of the anisotropy of embedding space. As shown in the original eigenvalue spectrum of Figure 2 (i.e., red line), LLM embeddings reveal extreme variance imbalance. Such collapsed distributions distort embedding similarity and reduce semantic separability, which can harm downstream task performance [9].

Whitening-based Anisotropy Mitigation. To mitigate the embedding anisotropy problem, WhitenRec [9] adopts the whitening transformation [12]. Although whitening is commonly introduced through the eigendecomposition of a covariance matrix, it can be equivalently understood through the singular value decomposition (SVD), offering a clearer interpretation for LLM embeddings. Given the mean-centered embedding matrix \(\tilde{\mathbf{E}}\), it can be decomposed as: \[\tilde{\mathbf{E}}=\tilde{\mathbf{U}} \tilde{\mathbf{S}} \tilde{\mathbf{V}}^\top,\] where \(\tilde{\mathbf{V}}\) contains the principal components and \(\tilde{\mathbf{S}}\) is the diagonal matrix of singular values. A strong anisotropy manifests as a highly skewed singular value spectrum, indicating that most variance is confined to a few directions. From the SVD perspective, whitening rescales each principal component by the inverse singular value [12]: \[\mathbf{E}_\text{whiten} = \tilde{\mathbf{E}} \tilde{\mathbf{V}} {\tilde{\mathbf{S}}^{-1}} = \tilde{\mathbf{U}},\] resulting in whitened features with an identity covariance matrix, i.e., \(\text{Cov}(\mathbf{E}_{\text{whiten}}) = \mathbf{I}\). This transformation equalizes variance across all directions, thereby enforcing perfect isotropy. However, this isotropic property of whitening collapses the representational spectrum. WhitenRec+ [9] applies group-wise whitening to preserve some semantic coherence, but its discrete formulation still lacks continuous control and preserves the internal semantic hierarchy.

3 Proposed Method: ACE↩︎

In this section, we present Anisotropy-Controllable Embedding (ACE), which mitigates the geometric imbalance of LLM-generated embeddings. ACE employs a linear autoencoder (LAE), whose spectral decomposition enables continuous control of anisotropy while preserving the semantic hierarchy of LLM embeddings.

LAE-based Embedding Reshaping. ACE applies a Linear Autoencoder [13] (LAE) to reshape LLM-generated item embeddings prior to training the SR backbones. We directly use the LLM embedding matrix \(\mathbf{E} \in \mathbb{R}^{n\times d}\) as the input and define an LAE objective that adjusts its spectral structure. To this end, we introduce an item–item similarity matrix \(\mathbf{B}_{\text{ACE}} \in \mathbb{R}^{n\times n}\), and the ACE objective: \[\mathcal{L}(\mathbf{B}_\text{ACE}) = \min_{\mathbf{B}_\text{ACE}} \| \mathbf{E}^\top - \mathbf{E}^\top\mathbf{B}_\text{ACE} \|_F^2 + \lambda \| \mathbf{B}_\text{ACE} \|_F^2, \label{eq:lae95llm}\tag{1}\] where the first term reconstructs the original embeddings, and the second term regularizes the \(\mathbf{B}_\text{ACE}\) matrix to avoid trivial solutions [13], [14]. Here, \(\lambda\) is a hyperparameter to control the strength of the second term (i.e., regularization). This objective yields a closed-form solution: \[\hat{\mathbf{B}}_{\text{ACE}} = (\mathbf{E}\mathbf{E}^{\top} + \lambda \boldsymbol{I})^{-1}\mathbf{E}\mathbf{E}^{\top}. \label{eq:lae95closed}\tag{2}\]

1.6pt

ACE aims to directly construct a geometrically adjusted embedding matrix \(\mathbf{E}_\text{ACE}\) that mitigates embedding anisotropy. Since anisotropy arises from an imbalanced singular value spectrum, we start from the singular value decomposition of the original embedding matrix \(\mathbf{E}=\mathbf{U}\mathbf{S}\mathbf{V}^{\top}\). We define the ACE-adjusted embedding \(\mathbf{E}_\text{ACE}\) by applying a shrinkage function to each singular value: \[\mathbf{E}_\text{ACE} = \mathbf{U} \, g_\lambda(\mathbf{S}), \label{eq:lae95embed95s}\tag{3}\] where \(\mathbf{U}\) contains the principal directions of the item-item geometry induced by \(\mathbf{E}\mathbf{E}^{\top}\) and the shrinkage function is defined as: \[g_\lambda(\mathbf{S}) = \sqrt{\frac{\mathbf{S}^2}{\mathbf{S}^2 + \lambda \mathbf{I}}}=\mathrm{diag}\!\left( \sqrt{\frac{\sigma_1^2}{\sigma_1^2 + \lambda}}, \ldots, \sqrt{\frac{\sigma_n^2}{\sigma_n^2 + \lambda}} \right). \label{eq:singular95value}\tag{4}\]

Note that \(\mathbf{V}\) is omitted since ACE focuses on reshaping the item–item similarity geometry rather than reconstructing the original feature space. To mitigate anisotropy, ACE attenuates overly dominant singular directions while preserving relative semantic structure, achieved via magnitude-dependent spectral shrinkage.

The corresponding linear operator induced by \(\mathbf{E}_\text{ACE}\) is then \(\hat{\mathbf{B}}_{\text{ACE}}=\mathbf{E}_\text{ACE}\mathbf{E}_\text{ACE}^\top\), which yields the following spectral form: \[\hat{\mathbf{B}}_{\text{ACE}} = \mathbf{U}\, \frac{\mathbf{S}^2}{\mathbf{S}^2+\lambda \mathbf{I}}\, \mathbf{U}^{\top} =\mathbf{U}\,\mathrm{diag}\!\left( \frac{\sigma_1^2}{\sigma_1^2 + \lambda}, \ldots, \frac{\sigma_n^2}{\sigma_n^2 + \lambda} \right)\,\mathbf{U}^{\top}, \label{eq:lae95svd2}\tag{5}\] which coincides with the spectral solution derived in prior work [14].

Interpretation of ACE Objective. Based on this SVD characterization, we now discuss the roles of the two components in the ACE objective and how each contributes to controlling anisotropy.

(i) Reconstruction loss (i.e., \(\|\mathbf{E}^\top - \mathbf{E}^\top\mathbf{B}\|_F^2\)) is responsible for preserving the semantic directions encoded by the original LLM embeddings. When \(\lambda=0\), the second term (i.e., \(L_2\)-regularization) in Eq. 1 disappears, and ACE reduces to optimizing only the reconstruction loss. Substituting \(\lambda=0\) into the shrinkage function in Eq. 4 yields \(g_{0}(\mathbf{\sigma_i})=1\) for all \(i\), meaning that all singular components are retained with equal weight. Consequently, the reconstructed embedding matrix \(\mathbf{E}_\text{ACE}=\mathbf{U}\) is expressed entirely in the left singular basis \(\mathbf{U}\), which represents the semantic directions of the original LLM embedding space. Thus, the reconstruction loss of ACE preserves semantic directions exactly, an effect analogous to whitening, i.e., every principal component is normalized to equal variance 1.

(ii) \(L_2\)-regularization term (i.e., \(\lambda\|\mathbf{B}\|_F^2\)) governs how strongly each principal component is weighted by regulating the shrinkage function \(g_{\lambda}(\mathbf{S})\) in Eq. 4 . Figure  2 illustrates how the shrinkage function progressively smooths the singular value spectrum as \(\lambda\) increases, demonstrating the continuous anisotropy control enabled by ACE. Unlike the \(\lambda=0\) case, where all directions are treated equally and leading to an isotropic embedding space, increasing \(\lambda\) gradually downweights dominant singular values and smooths the spectral imbalance. When \(\lambda\) is very large, \(g_{\lambda}(\mathbf{S})\approx \mathbf{S}/\sqrt{\lambda}\), which follows the original eigenvalue distribution of the LLM embeddings. This makes the basis vectors in \(\mathbf{U}\) weighted according to their original variance, thus maintaining an anisotropic representation. In this way, the regularization term acts as a generalized form of whitening, modulating the spectral balance of the embedding space without collapsing its semantic hierarchy.

Applying ACE to LLM-as-Extractor. After the embedding reshaping, ACE integrates seamlessly into the LLM-as-Extractor paradigm by producing a geometry-adjusted, \(k\)-dimensional embedding matrix used to initialize the SR backbones.

Given the spectral formulation in Eq. 3 , ACE constructs the reduced embedding matrix by retaining the top-\(k\) singular directions: \[\mathbf{E}_\text{ACE}^{(k)} = \mathbf{U}^{(k)}\, g_\lambda(\mathbf{S}^{(k)}) \cdot \gamma,\] where \(\mathbf{U^{(k)}}\) and \(\mathbf{S^{(k)}}\) denote the top-\(k\) singular vectors and values, respectively. Since \(g_\lambda(\cdot)\) compresses the embedding magnitudes, ACE introduces a scaling factor \(\gamma\) to recover the embedding scale.

The resulting normalized embeddings serve as the initialization of the item embedding table in any SR backbone, e.g., SASRec [1], GRU4Rec [2], or BERT4Rec [3]. The SR model is then trained as usual, benefiting from an embedding space that is both semantically coherent and geometrically balanced.

4 Experimental Setup↩︎

Datasets. We evaluate ACE on four benchmark datasets: two Amazon Review 2014 subcategories (Beauty and Toys)  [15], the Yelp 2018 dataset¹, and the ML-20M dataset². Following existing works [5], [7], we filter out users and items with fewer than five interactions. For LLM-based item initialization, we encode each item’s textual metadata (e.g., title, category, brand) using text-embedding-3-large from OpenAI³. In addition, we experiment with multiple LLM encoders for item initialization, including F2LLM-4B [16], Qwen3-Embedding-8B [17], and KaLM-Embedding-Gemma3-12B-2511 [18], [19].

Evaluation Protocols. Following [7], we adopt the leave-one-out protocol to construct training, validation, and test sequences for each user. For evaluation, we rank all items in a full-sort manner without additional candidate filtering, and report Recall@\(K\) (R@\(K\)) and NDCG@\(K\) (N@\(K\)), where \(K \in \{10,20\}\).

Baseline Models. We compare ACE with five competitive LLM-as-Extractor approaches. PCA-based method (LLM2X [4]), whitening-based methods (WhitenRec+ [9]), and semantic adaptation approaches (LLMEmb [5], AlphaRec [6], and AlphaFuse [7]). For integrating LLM-as-Extractor approaches into SR backbones, we employ three widely used SASRec [1], GRU4Rec [2], and BERT4Rec [3].

Implementation Details. All models are implemented using the open-source RecBole framework [20]. We set the item embedding dimension to 128, the maximum sequence length to 50, and use a batch size of 256. We adopt early stopping with a patience of 10 epochs, based on NDCG@10 on the validation set. All models are trained with Adam optimizer and the learning rate is tuned on a logarithmic scale from \(10^{-3}\) to \(10^{-5}\). For ACE, the \(L_2\)-regularization coefficient \(\lambda\) is selected from \(\{0,1,5, \cdots, 5\times10^3\}\), and the scaling parameter \(\gamma\) is used to rescale the embedding standard deviation to \(\{0.1, 0.5, 1\}\). All results are averaged over four runs. The source code is available at https://github.com/DCheol/ACE.

5 Experimental Results↩︎

Overall Performance. Table 1 shows that ACE consistently outperforms all existing LLM-as-Extractor models across all benchmark datasets and backbone architectures. Notably, ACE achieves absolute improvements of up to 4.9% in Recall@10 and 5.9% in NDCG@10 over the best competing baseline. Notably, whitening-based method (i.e., WhitenRec+ [9]) sometimes underperforms even the simple PCA-based LLM2X [4]. This degradation aligns with our geometric analysis: aggressive whitening uniformly flattens the embedding spectrum, distorting the semantic hierarchy essential for item differentiation. In contrast, ACE provides a more balanced embedding geometry, achieving substantial gains by simultaneously reducing anisotropy and preserving semantic hierarchy.

Various Backbone Analysis. To verify that ACE generalizes beyond a specific SR architecture, we further evaluate it on two representative SR backbones, GRU4Rec [2] and BERT4Rec [3]. As shown in Table 2, ACE consistently delivers performance gains across both backbones, improving Recall@20 by up to 12.4% and NDCG@20 by up to 11.8% over the strongest LLM-as-Extractor baseline. These improvements are observed across all datasets, indicating that ACE’s spectral reshaping mechanism remains effective regardless of sequence modeling architecture. These results highlight that ACE provides a robust and architecture-agnostic embedding reshaping.

Various LLM Encoders. Figure 3 reports the results on the Beauty dataset for LLM-as-Extractor baselines using SASRec backbone under different LLM encoders: F2LLM-4B [16], Qwen3-Embedding-8B [17], and KaLM-Embedding-Gemma3-12B-2511 [18], [19]. Across all encoder settings, ACE consistently achieves the best performance on Recall@20, outperforming all LLM-as-Extractor baselines. Specifically, ACE improves Recall@20 by up to 2.94% over the strongest baseline (i.e., AlphaFuse) under each LLM encoder. ACE maintains stable and consistent gains across all encoders, indicating that its performance gains are not tied to a specific LLM embedding.

Hyperparameter Analysis. Figure 4 illustrates how the performance of ACE with the SASRec backbone varies with the regularization coefficient \(\lambda\). When \(\lambda\) is too small, the shrinkage function over-flattens the spectrum, erasing semantic hierarchy and degrading accuracy. Conversely, very large \(\lambda\) values provide insufficient shrinkage, leaving the original anisotropy largely unchanged and limiting representation quality. The best results consistently emerge at moderate \(\lambda\) values, which balance anisotropy reduction with semantic hierarchy preservation. This demonstrates the effectiveness of ACE’s balanced anisotropy-control mechanism.

6 Conclusion↩︎

In this work, we investigate the geometric limitations of LLM-generated embeddings, highlighting that their strong anisotropy hinders effective adaptation to collaborative signals. To address this issue, we introduced Anisotropy-Controllable Embedding (ACE), which reshapes LLM embeddings through the spectral behavior of a linear autoencoder (LAE). By preserving semantic directions through reconstruction loss and modulating spectral variance via an \(L_2\)-regularization term, ACE provides continuous and controllable anisotropy mitigation. This yields geometrically balanced embeddings that maintain semantic integrity while effectively reducing anisotropy. Our experimental results confirm that ACE delivers consistent improvements, achieving gains of up to 12.4% in Recall@20 and 11.8% in NDCG@20, respectively.

References↩︎