1 Introduction↩︎

Recently, large language models (LLMs), known for their strong semantic understanding capabilities, have been increasingly integrated into recommender systems, demonstrating substantial potential in user modeling [1], item representation [2], and reasoning tasks [3]. Among the various LLM-based approaches, two paradigms have received the most attention: generation-based recommenders (Fig. 1 (a)) that place LLMs at the core, and representation-based methods that leverage LLMs as feature extractors to enhance traditional models (Fig. 1 (b)). The latter offers key advantages: semantic embeddings can be precomputed offline to eliminate real-time inference latency and can be seamlessly integrated into existing architectures, making it more practical for real-world deployment [4]. Yet, how best to design and deploy this paradigm remains an open question, motivating a closer look at its internal mechanisms.

The LLM-as-feature-extractor paradigm typically follows three stages: first, transforming structured or unstructured item attributes (e.g., title, category, brand) into natural language prompts; second, feeding the prompt into an LLM to extract semantic embeddings through a designated aggregation strategy; and third, adapting the high-dimensional LLM outputs (e.g., 4096-dim vectors from LLaMA [5]) to a compact representation compatible with downstream recommender models. Finally, the adapted representations are fed into sequential recommenders such as SASRec [6] for training. Existing studies have explored a variety of techniques for each stage. For prompt construction, [7], [8] adopt simple attribute concatenation, while others incorporate keyword extraction [9], summarization [10], or knowledge-enhanced prompting [11], [12]. In the text encoding stage, aggregation strategies range from mean pooling to last-token or special-token representations [13], [14]. For parameter tuning, earlier works [7], [15] mostly froze the LLM, whereas recent studies [16] show that lightweight fine-tuning methods, such as supervised contrastive learning, can further boost performance. Feature adaptation designs include linear projection [17], multilayer perceptrons (MLPs) [18], and mixture-of-expert (MoE) [7] networks.

Despite these efforts, existing studies often investigate isolated components or propose single-model designs, resulting in limited comparability across methods. The absence of a unified framework makes it difficult to disentangle the effects of individual design choices, thereby hindering a holistic understanding and principled advancement of this paradigm. This raises a natural question: How do design choices within the LLM-as-feature-extractor pipeline impact recommendation performance? Moreover, can we attain stronger results by simply combining well-established techniques, without resorting to overly complex architectures?

To address this question, we propose RecXplore—a modular and reproducible analytical framework for fair and systematic evaluation of the LLM-as-feature-extractor pipeline. We decompose the pipeline into four core modules: Data Processing, Feature Extraction, Feature Adaptation, and Sequential Modeling. Through controlled experiments on four widely-used public datasets, we derive several key insights. First, simple attribute concatenation serves as a robust prompting strategy, while excessive prompt engineering often introduces noise and degrades performance. Second, a two-stage fine-tuning pipeline—continued pretraining (CPT) followed by supervised fine-tuning (SFT)—yields superior semantic representations, with mean pooling outperforming other aggregation methods. Third, a hybrid adapter that combines principal component analysis (PCA) with a mixture-of-experts (MoE) architecture proves most effective for feature adaptation. Finally, when LLM-derived semantic embeddings are sufficiently rich, traditional ID embeddings provide marginal benefit, and direct replacement emerges as the most efficient integration strategy.

Building on these insights, we instantiate the most effective design choices into RecXplore, which consistently outperforms strong baselines across all datasets and evaluation metrics, achieving up to a 12.7% and 18.7% relative improvement in HR@5 and NDCG@5, respectively. These results underscore the effectiveness and practical value of systematic modular analysis. Our main contributions are as follows:

• We introduce RecXplore, the first modular framework for systematic analysis of the "LLM-as-feature-extractor" paradigm, explicitly decoupling key components within the recommendation pipeline.

• We conduct comprehensive evaluations of representative design choices for each module across multiple datasets, establishing effective and reusable practices to guide future development.

• We demonstrate that, without introducing complex architectures, a simple combination of the best-performing designs from each module in RecXplore is sufficient to achieve consistently strong performance across diverse datasets and evaluation metrics.

2 Related Work↩︎

2.1 Sequential Recommendation↩︎

Sequential recommendation aims to predict the next item a user will interact with based on their behavioral history. Early methods used Markov chains [19] to model short-term dependencies. With deep learning, RNN-based models such as GRU4Rec [20] and CNN-based models like NextItNet [21] and Caser [22] were proposed to better capture sequential patterns. Transformer-based models, including SASRec [6] and BERT4Rec [23], further improved performance by modeling long-range dependencies. Recent advances introduce Mamba-based architectures [24]–[26] that offer linear inference complexity for long sequences, and diffusion-based models [27]–[29] that enhance recommendation by modeling uncertainty through generative denoising processes. However, despite their success, these models typically ignore the rich semantic information contained in item attributes such as titles, descriptions, and categories, which can provide valuable complementary signals beyond interaction IDs.

2.2 LLM for Recommendation↩︎

Recent research has explored two primary paradigms for integrating LLMs into recommendation pipelines. The first paradigm is LLM-centric, in which user behaviors and item attributes are converted into natural language descriptions and processed by a pre-trained LLM to directly generate recommendation outputs [30]–[32]. This approach treats recommendation as a language modeling task and leverages the generative strengths of LLMs. The second paradigm explores the use of LLMs as feature extractors to enhance recommender systems, focusing on different components such as prompt design [7], [10], [12], aggregation strategies [13], [14], or adaptation modules [8], [16], [18]. While these studies demonstrate the potential of LLM-enhanced representations, they typically investigate specific techniques in isolation or under inconsistent setups, making it difficult to draw general conclusions about what design choices matter and why.

In contrast, our work introduces RecXplore, a unified and modular framework that systematically decomposes the LLM-as-feature-extractor pipeline into four core modules. This design enables controlled comparisons across design dimensions and datasets, allowing us to derive actionable principles for building effective LLM-enhanced recommender systems.

3 RecXplore FRAMEWORK↩︎

To systematically analyze the LLM-as-a-feature-extractor paradigm, we use the following analysis framework (Fig. 2).

• Data Processing Module: Converts raw item attributes into a textual input format suitable for the LLM.

• Feature Extraction Module: Encodes the textual input into semantically rich feature embeddings using LLM.

• Feature Adaptation Module: Performs transformation, dimensionality reduction, and adaptation on the high-dimensional semantic embeddings.

• Sequential Modeling Module: Models user behavior sequences with the adapted features within the downstream recommendation model.

The framework consists of four core modules, with data flowing sequentially from data processing to the final recommendation. This modular structure facilitates the testing and analysis of how different module designs and implementations affect overall performance. Next, we delve into the detailed design and exploration setup for each core module.

3.1 Data Processing Module↩︎

The core objective of this module is to construct high-quality input representations for items by converting raw item attributes (e.g., price, category, brand) into a format suitable for the LLM. Our investigation unfolds along two primary dimensions: Template-based Methods and LLM-based Semantic Enhancement.

3.1.1 Template-based Method.↩︎

This method aims to consolidate an item’s structured and unstructured information into a coherent natural language input using a predefined template. We test the Simple Attribute Concatenation approach, which combines all available attributes (e.g., brand, category, description) into a comprehensive descriptive sentence to serve as the LLM’s input.

3.1.2 LLM-based Semantic Enhancement.↩︎

This approach leverages the capabilities of an additional LLM to refine or enrich the input text. We evaluate three techniques: (1) Keyword Extraction, which uses the LLM to extract keyword information from item attributes; (2) Summarization, which uses the LLM to summarize item attributes into one or two sentences, reducing redundancy; (3) Knowledge Expansion, which utilizes the LLM’s world knowledge to augment the original item attributes.

3.2 Feature Extraction Module↩︎

This module aims to utilize an LLM to transform the processed textual input into semantic embeddings. We focus on two core strategies in this module: the LLM fine-tuning strategy and the feature aggregation strategy.

3.2.0.1 LLM Fine-tuning Strategy.

This strategy investigates whether the LLM’s parameters should be optimized for the recommendation task. We compare two primary approaches: (1) Frozen LLM, which keeps the LLM’s parameters fixed during training, directly leveraging its zero-shot capabilities; and (2) Fine-tuned LLM, which applies Parameter-Efficient Fine-Tuning (PEFT) techniques, such as LoRA, to optimize the LLM on recommendation data. Within the fine-tuning approach, we further evaluate several methods, including Continued Pre-training (CPT), Supervised Fine-tuning (SFT), Supervised Contrastive Fine-Tuning (SCFT) [16], and a combined CPT+SFT strategy, to identify the optimal implementation for using LLM as a feature extractor in recommendation tasks.

3.2.0.2 Feature Aggregation Strategy.

This strategy defines how to aggregate the sequence of token-level vectors from the LLM’s output to generate a final, item-level embedding. We systematically compare four mainstream methods: (1) Mean Pooling, which averages the hidden states of all tokens in the final layer; (2) Max Pooling, which takes the maximum value across the hidden states; (3) Last Token, which directly uses the hidden state of the sequence’s final token as the representation; and (4) Explicit One-word Limitation (EOL) [33], which prompts the model to distill item information into a single word and uses this word’s embedding as the final representation.

3.3 Feature Adaptation Module↩︎

This module adapts raw semantic embeddings for the downstream recommender backbone models, addressing potential issues like high dimensionality and feature space mismatch. To this end, we explore two primary approaches: the design of different Adaptation Architectures and various strategies for Fusion with item ID Embeddings.

3.3.0.1 Adaptation Architectures.

This approach aims to identify optimal adaptation methods for semantic features, preserving or even enhancing key semantic information while reducing computational complexity. We systematically evaluate five distinct architectures: (1) Principal Component Analysis (PCA); (2) Linear Projection; (3) Multilayer Perceptron (MLP) Adapter; (4) Product Quantization (PQ) [34]; and (5) Mixture-of-Experts (MoE) [35], which uses a network of specialized experts for dynamic, input-dependent adaptation.

3.3.0.2 Fusion with ID Embeddings.

To investigate whether item ID information can serve as an effective supplement to semantic embeddings, we compare three mainstream methods for fusing semantic embeddings and ID embeddings: (1) Replacement, in which the pre-trained ID embeddings is directly replaced by the semantic embeddings, enabling the system to fully rely on semantic embeddings; (2) Concatenation, which directly concatenates the semantic embeddings and pre-trained ID embeddings; and (3) Alignment, which uses an additional alignment loss [16] to constrain the semantic embedding space to align with the pre-trained ID embedding space.

3.4 Sequential Modeling Module.↩︎

This module serves as the backbone of the recommendation system, responsible for modeling user behavior sequences. In our study, we adopt the powerful and widely-used sequential recommendation model, SASRec [6]. SASRec takes the item embeddings from the feature adaptation module as input to capture users’ dynamic preferences and generate the final recommendation list.

3.5 Training and Inference↩︎

3.5.0.1 Training Stage.

We investigate two training modes, distinguished by whether the LLM’s parameters are updated:

• Frozen LLM (Single-Stage Training): In this mode, the LLM’s parameters remain unchanged throughout training. The training process focuses solely on optimizing the parameters of the Adaptation Module and the downstream SASRec, driven entirely by the recommendation task’s loss function (e.g., cross-entropy loss).

• Fine-tuned LLM (Two-Stage Training): This mode consists of two stages. In the first stage, the LLM is fine-tuned on recommendation data using Parameter-Efficient Fine-Tuning (PEFT) techniques like LoRA to instill domain knowledge. In the second stage, the fine-tuned LLM is frozen, and its superior output features are used to train the Adaptation Module and SASRec.

3.5.0.2 Inference Stage.

To ensure the low-latency responses required for industrial applications, all item semantic embeddings are pre-computed and cached offline. During online inference, the system only needs to perform a forward pass through the lightweight Adaptation Module and the Sequential Module. This completely avoids real-time calls to the LLM, thus guaranteeing efficient recommendation services.

4 Experiments↩︎

We conduct systematic experiments based on the proposed RecXplore  framework to analyze how design choices across different modules affect recommendation performance. Our study is guided by the following research questions:

• RQ1 What are the specific impacts of different data processing strategies on overall performance?

• RQ2 What is the impact of different feature aggregation strategies on the final recommendation performance?

• RQ3 What is the impact of different LLM fine-tuning strategies on recommendation quality?

• RQ4 What is the impact of different feature adaptation strategies on the final recommendation performance?

• RQ5 Can pre-trained item ID embeddings bring significant additional performance gains on top of the semantic embeddings?

• RQ6 Can the optimal combination of components, distilled from our systematic decoupled analysis, outperform the state-of-the-art methods, and by what margin?

4.1 Experimental Settings↩︎

4.1.0.1 Dataset.

We conduct experiments on four real-world datasets, i.e., the Steam dataset [36] and three datasets from Amazon product reviews [37]: Beauty, Fashion, and Games. These datasets span different product domains and user behavior patterns, serving as common benchmarks for evaluating sequential recommendation models. To ensure consistency, we adopt the data pre-processing method proposed in LLMEmb [16] for all datasets.

4.1.0.2 Base Configuration.

This base configuration integrates the most straightforward and representative design choices from each module, serving as the reference benchmark for all subsequent decoupled analyses. The base configuration is composed as follows:

• Data Processing: Uses the ‘Simple Attribute Concatenation’ strategy, which combines the values of item attributes to form the input text. This strategy serves as the default benchmark for our experiments.

• Feature Extraction: Utilizes a frozen-parameter LLaMA model combined with Mean Pooling to efficiently convert the input text into a single item embedding.

• Feature Adaptation: Reduces feature dimensionality using a single-layer Linear Projection adapter, without processing via Principal Component Analysis (PCA).

• Sequential Modeling: Adopts SASRec as the downstream backbone to model user behavior sequences.

4.1.0.3 Implementation Details.

Our experiments were conducted on a server with an Intel Xeon Platinum 8457C CPU and 8 NVIDIA L20 GPUs. We chose LLaMA-7B [38] as the base model for its superior performance over alternatives (e.g., BERT, RoBERTa) in preliminary tests. We employ LoRA (r=8,alpha=32) and an 8-expert MoE adapter for parameter-efficient fine-tuning. Full details on the model comparison and hyperparameters are available in the Appendix.

4.1.0.4 Evaluation Metrics.

Following previous work [6], we use Hit Rate (HR@K) and Normalized Discounted Cumulative Gain (NDCG @K) as our evaluation metrics, with K set to 5, 10. For all metrics, a higher value indicates better performance. In the evaluation, each expected recommended item in the test set is paired with 100 randomly sampled uninteracted items to calculate the metrics. To reduce the randomness of the results, all experiments are conducted three times with different random seeds (42, 43, 44), and we report the average metrics. As the standard deviation across all runs was consistently \(\leq\) 0.002, we omit it from the tables for brevity.

4pt

4pt

5pt

4.2 Analysis of Data Processing Strategies (RQ1)↩︎

4.2.0.1 Experimental Setup.

To answer RQ1, we investigate the impact of different data processing strategies on recommendation performance. The experiment is benchmarked against the Base Configuration, which uses the most straightforward Simple Attribute Concatenation (SAC) strategy. Building upon this base, we fix all other modules (e.g., Feature Extraction, Adaptation) and vary only the data processing method. We also evaluate three LLM-based Attribute Enhancement methods: (1) Keyword Extraction, (2) Summarization, and (3) Knowledge Expansion. Detailed results are presented in Table 1.

4.2.0.2 Analysis.

Our experiments on data processing highlight the following key takeaways:

• Complex Enhancement is Suboptimal. Additional attribute enhancements (e.g., keyword extraction, summarization) provide minimal or even negative performance improvements. This can be attributed to the powerful downstream encoder’s intrinsic ability to understand semantics, which makes further processing on structured data redundant and potentially harmful.

• Simplicity is Optimal. With a powerful downstream encoder, simple attribute concatenation proves to be the most effective and efficient strategy, eliminating the need for complex attribute enhancements.

4.3 Analysis of Feature Aggregation (RQ2)↩︎

4.3.0.1 Experimental Setup.

Building on the optimal finding from RQ1 (SAC), we evaluate different feature aggregation strategies by comparing the default Mean Pooling against three other methods. The results are shown in Table 2.

4.3.0.2 Analysis.

Our experiments yield the following takeaways:

• Mean Pooling is Optimal. It consistently outperforms all other strategies, as integrating all token information provides a more comprehensive semantic representation.

• Max Pooling Performs Poorly. Max Pooling is clearly the worst-performing strategy on all datasets. This is likely because it over-focuses on isolated features, which harms the representation quality. The EOL and Last Token strategies provide middling, but better results.

4.4 Analysis of LLM Fine-tuning (RQ3)↩︎

4.4.0.1 Experimental Setup.

Building upon the optimal conclusions from prior research (RQ1 and RQ2), this experiment fixes the data processing to Simple Attribute Concatenation and feature aggregation to Mean Pooling, aiming to evaluate the impact of different LLM fine-tuning paradigms. The results are shown in Table 3.

4.4.0.2 Analysis.

Our experiments highlight the following key takeaways:

• Fine-tuning helps. A clear conclusion is that any form of fine-tuning significantly outperforms keeping the LLM frozen, confirming that domain adaptation is essential for improving recommendation quality.

• CPT+SFT is optimal. The two-stage CPT+SFT combination demonstrates the highest overall performance. This result strongly supports the effectiveness of a combined domain-first (CPT) and task-specific (SFT) training path.

2pt

4.5 Analysis of Feature Adaptation (RQ4)↩︎

4.5.0.1 Experimental Setup.

Building upon the optimal strategies from RQ1-RQ3 (SAC, Mean Pooling, and CPT+SFT), this experiment evaluates the impact of different adaptation architectures. We compare four architectures—Linear, MLP, PQ, and MoE—each with and without PCA processing, with results presented in Table 4.

4.5.0.2 Analysis.

Our experimental results reveal several key takeaways regarding adapter design:

• MoE architecture is superior. Among all evaluated architectures, MoE consistently demonstrates the best performance. This highlights the superiority of its dynamic, input-dependent adaptation capabilities in capturing complex semantics.

• PCA helps MoE perform better. PCA is not always helpful and can sometimes hurt performance. However, we found that when PCA is used with MoE, it achieves the best results on three out of our four datasets.

4.6 Analysis of Item ID Embeddings Fusion (RQ5)↩︎

4.6.0.1 Experimental Setup.

To answer RQ5, we investigate whether fusing traditional ID embeddings is always beneficial, and if this effect depends on the downstream adapter’s architecture. The experiments are based on two adapters from RQ4 with different expressive capabilities: the top-performing MoE adapter and the efficient Linear adapter. On top of these two architectures, we compare three integration strategies: (1) Direct Replacement (our baseline), (2) Concatenation, and (3) Alignment. The results are presented in Table 6.

4.6.0.2 Analysis.

Our experimental results reveal several key takeaways:

• Powerful Adapters Render ID Embeddings Redundant. For the powerful MoE adapter, the rich semantic information it generates renders ID embeddings redundant, thus the Direct Replacement strategy performs best.

• ID Embeddings Effectively Supplement Simpler Adapters. Conversely, for the simpler Linear adapter, fusing ID information effectively supplements its limited semantic capabilities, leading to superior performance from the Concatenation or Alignment in most cases.

4.7 Putting It All Together: RecXplore vs Baselines (RQ6)↩︎

Through systematic and decoupled analysis in prior sections, we have identified a set of optimal practices for the LLM-as-a-feature-extractor paradigm. Unlike the baseline configuration, this optimized setup integrates: (1) Simple Attribute Concatenation for processing item attributes; (2) Mean Pooling for feature aggregation; (3) a two-stage CPT+SFT strategy for LLM fine-tuning; (4) an MoE Adapter with PCA for feature adaptation; and (5) a Direct Replacement strategy, omitting item ID information, for backbone recommender systems.

We evaluate RecXplore against two categories of baselines: traditional sequential models (GRU4Rec [20], Bert4Rec [23], SASRec [6]) and up-to-date LLM-enhanced baselines (SAID [39], LLMESR [40], LLMEMB [16]).

As shown in Table 5, our optimal configuration, RecXplore, comprehensively outperforms all baselines, achieving improvements across the metrics on all datasets. This result demonstrates that our approach of systematically decoupling and optimizing individual components can yield a superior configuration.

5 Conclusion and Future Work↩︎

We proposed RecXplore, a modular framework for optimizing the LLM-as-a-feature-extractor paradigm in sequential recommendation. By decoupling the pipeline into four components—data processing, feature extraction, feature adaptation, and sequential modeling—and analyzing each in isolation, we identified effective design choices that, when combined into RecXplore, consistently outperformed both traditional and LLM-based baselines across four datasets. While effective, our approach uses a greedy optimization strategy that may miss globally optimal configurations, and focuses solely on the SASRec backbone; future work can explore joint optimization and extend the analysis to other architectures.

References↩︎