1 Introduction↩︎

In the digital age, with the explosion of information and data, recommender systems have emerged as an essential tool for dealing with information overload [1]. These systems are designed to provide users with personalized and relevant content, which can help cater to their preferences and needs, stream their decision-making process, and improve service efficiency [2]. However, users may not be satisfied with the results they receive without clear and meaningful explanations, as shown in Figure 1. Providing effective explanations for the recommendation results can significantly enhance the users’ experience, foster their engagement and trust, and boost their loyalty and stickiness to the service and products. Meanwhile, an effective explanation could also be useful for the users to improve their decision-making accuracy and efficiency [3] or increase users’ satisfaction with the explanations [4].

Early explainable methods, such as collaborative filtering (CF), generated simple justifications like “users who liked X also liked Y" [5], [6]. While intuitive, these explanations often lack the granularity and persuasive power to be truly compelling. A significant step towards building more interpretable models came from leveraging the rich, explicit information contained in user reviews. Reviews offer direct evidence of user preferences and item characteristics at a fine-grained level. As illustrated in Figure 1, users’ preferences can be discerned through the reviews they compose. By examining the customer’s previous reviews, we can deduce that this particular customer places a higher priority on”price" and “quality", followed by the camera’s attributes. Similarly, we can infer the quality of items by examining the reviews they have received. For instance, the foremost considerations for the first phone revolve around”price" and “quality". Consequently, we can recommend items to users by aligning their preferences with the features of the items, thereby offering more insightful and relevant explanations. In contrast, the explanation that one of your friends has purchased is not sufficiently convincing, thus failing to provide effective explanations. Building on this insight, researchers have developed sophisticated models that incorporate textual data. Attention-based methods [7]–[9] were introduced to dynamically weigh the importance of different words, sentences, or reviews, allowing the model to focus on the most salient information. Similarly, path-based models [10]–[12], often leveraging knowledge graphs, aimed to find explicit reasoning paths between users and items. More recently, the advent of Large Language Models (LLMs) has introduced powerful capabilities for natural language understanding and generation, opening new frontiers for creating fluent, human-like explanations.

Despite this progress, generating effective and personalized explanations in recommender systems remains a challenging task due to several critical limitations in existing methods. First, (C1. Heavy reliance on human effort and expertise) Designing interpretable components, such as explainable paths or attention mechanisms, often requires significant manual effort and domain expertise. This process can be labor-intensive and prone to human bias, which may result in sub-optimal or inconsistent explanations. Second, (C2. Poor generalization across dynamic datasets) Recommendation tasks often span diverse datasets with varying characteristics. A single, fixed explainability architecture may fail to generalize effectively across these datasets, limiting its adaptability to different recommendation contexts. Third, (C3. Risk of hallucination in LLM-based methods) While LLMs have shown impressive capabilities in natural language tasks, directly applying them to explanation generation can introduce hallucinations, factual errors, and generic outputs. Without careful adaptation to user-specific contexts, LLMs often fail to capture nuanced preferences, resulting in unreliable and impersonal explanations that undermine user trust.

To address the aforementioned challenges, we propose a Bi-level Neural Architecture Search (Bi-NAS) to jointly search for a recommender system design with optimal, effective, and explainable architecture. In particular, to address C1 and C2, we introduce NAS, which aims to automatically find a neural architecture that achieves the best possible performance with limited computing resources while requiring minimal human intervention [13]. NAS strategically explores possible neural architectures within a predefined search space and adapts to different datasets, thus generalizing to diverse datasets by producing tailored architectures for varying data characteristics. Additionally, to address C3, we employ LLMs to enhance explanation generation using aligned features between users and items discovered by NAS, while incorporating user history to enable more personalized and contextually relevant explanations. Our approach learns user preference weights and item quality weights by extracting meaningful features from user reviews. By aligning users’ feature preferences with item quality scores on corresponding attributes, we leverage LLMs to generate effective and personalized explanations. Specifically, our method first learns the embedding of users, items, and their associated reviews. Next, we employ Bi-NAS to automatically search for the optimal cross-attention network architecture and corresponding interaction functions. Finally, we integrate LLMs with the aligned user-item features to generate personalized explanations that reflect individual user preferences and item characteristics. Extensive experiments on real-world datasets demonstrate the effectiveness of our proposed approach.

We summarize our contributions as follows:

• Problem Formulation: We formulate the problem of effective explanation generation in recommender systems and identify the key challenges inspired by the real applications.

• Model: We propose an end-to-end Bi-level Neural Architecture Search (NAS) framework that automatically searches for the optimal explainable architecture for generating effective explanations while enhancing personalization through zero-shot prompting strategies.

• Comprehensive Evaluation: We conduct extensive experiments on four real-world datasets, evaluating both recommendation performance and explanation quality. The results demonstrate the superior performance of our approach across multiple evaluation metrics.

2 Problem Definition↩︎

In this section, we introduce the preliminaries of our problem setting. Then, we give the formal definition of explainable recommendation with NAS. We use upper case calligraphic font letters to denote sets (e.g., \(\mathcal{U}\)), bold upper case letters to denote matrices (e.g., \(\boldsymbol{A}\)), bold lower case letters to denote vectors (e.g., \(\boldsymbol{x}\)), and regular lower case letters to denote scalars (e.g., \(\alpha\)).

Preliminaries and Notations. Assume we have a user set with \(M\) users \(\mathcal{U}=\{u_1,u_2,\cdots,u_M\}\) , an item set with \(N\) items \(\mathcal{I}=\{i_1,i_2,\cdots,i_N\}\), and a feature set with \(P\) features \(\mathcal{F}=\{f_1, f_2, \cdots,f_P\}\). We apply the sentiment analysis toolkit  [14] to extract (Aspect, Opinion, Sentiment) triplets from the textual reviews. For example, in the domain of clothing, the extracted features (aspects) include price, quality, color, size, etc. The corresponding opinions include high, good, great, fit, etc., and the sentiment could be either +1 or -1. Following the similar pre-processing steps adopted by previous work  [15]–[19], we compute the user-feature preference vector \(f_{u} \in \mathbb{R}^{p}\) and \(u \in \mathcal{U}\), and the item-feature quality vector \(f_{i} \in \mathbb{R}^{p}\) and \(i \in \mathcal{I}\). \(f_{u}^k\) indicates to what extent the user \(u\) cares about the feature \(f_k\). Considering that user tends to comment on the features they care more about and may express their opinions more frequently on those features, it is a frequency-based vector. \(f_{i}^k\) indicates how well the item \(i\) performs on the feature \(f_k\). Considering that the quality of items can be reflected in the review sentiments from users, it is formulated by considering both review feature frequency and average sentiment. Specifically, \[\label{eqn:32review32feature32score} \begin{align} f_{u}^k =&\begin{cases} 0, \text{if user u did not mention feature f_k} \\ 1+(T-1)(\frac{2}{1+\mathrm{exp}(-c_{u}^k)}-1), \text{otherwise} \end{cases} \\ f_{i}^k =& \begin{cases} 0, \text{if item i was not mentioned with feature f_k} \\ 1+(\frac{T-1}{1+\mathrm{exp}(-c_{i}^k \cdot s_{i}^k)}), \text{otherwise} \end{cases} \end{align}\tag{1}\] where \(T\) is the maximum review score, \(c_{u}^k\) is the frequency that user \(u\) mentioned feature \(f_k\), \(c_{i}^k\) is the frequency that item \(i\) is mentioned with feature \(f_k\), while \(s_{i}^k\) is the average sentiment of item \(i\) regarding feature \(f_k\). After this transformation, all entries of these feature vectors are mapped into the range of \([0, T]\).

Cross-Attention. Different from self-attention, which operates on a single embedding vector, cross-attention typically involves two distinct vectors of the same dimension. It calculates attention scores for each dimension in one vector based on its relevance to another vector and integrates the two vectors. In this paper, we utilize cross-attention between user/item embeddings and their corresponding feature embeddings to explore their interconnections.

Neural Architecture Search. Neural Architecture Search refers to the systematic process of automating the design of neural network architectures. It includes three main parts: search space, search strategy, and performance evaluation strategy. Designing a search space is a crucial aspect of NAS as it directly impacts the efficiency and effectiveness of the entire process [13]. The search space determines the realm of architectural possibilities, allowing NAS to explore diverse neural network configurations. On one hand, the space needs to be broad enough to capture a diverse range of architectural possibilities, allowing for innovation and inclusion of human knowledge. On the other hand, the space cannot be too general, as that would lead to computational inefficiency and high exploration costs [20]. The search strategy (optimization methods) details how to explore the search space. It involves the classical trade-off between exploration and exploitation. On the one hand, there is a desire to rapidly discover high-performing architectures. On the other hand, it’s crucial to avoid early convergence with suboptimal architectures  [21]. The evaluation process assesses the generalization performance of each architecture on unseen data, offering insights into its effectiveness and aiding in the identification of optimal architectures [22].

In this paper, we delve into the realm of recommendation systems, focusing on the scenario of providing effective explanations. For each user-item pair \((u, v)\), we deduce the preferences of user \(u\) and features of item \(v\) from reviews. An effective explainable recommendation system should recommend items with features that closely align with users’ preferences and provide the corresponding common features for the explanation. Here, we give the formal definition of our recommendation problem:

3 Methodology↩︎

In this section, we go over the details of our proposed framework, including the recommendation component, the NAS-driven architecture search space, and our optimization strategy. We begin by using NAS to automatically explore a predefined architecture space, optimizing performance under resource constraints with minimal human intervention and tailoring architectures to each dataset. Next, we leverage LLMs to generate personalized, context-aware explanations from the NAS-aligned user–item features, enriched by user history. Finally, we present illustrative examples of our results.

3.1 Recommendation Component↩︎

Existing deep recommender system frameworks usually share a similar structure, as shown in Figure 2. They have three modules: embedding layer, interaction layer, and output layer [23], [24]. We provide a high-level overview in this subsection.

Embedding Layer. The input of the embedding layer includes both user and item IDs as well as various user and item features, and the output is the embeddings for each of these fields. In this paper, we focus on using features extracted from reviews for both users and items. We embed each word within these features into a lower-dimensional representation using pre-trained word embeddings (e.g., GloVe). Given the high dimensionality and sparsity of one-hot encoding for user and item IDs, as well as the difference in their corresponding embedding size, we also introduced an embedding layer to transform the user and item IDs into a low-dimensional continuous vector, the final output of the embedding layer is: \[\label{eqn:embedding32layer} \begin{align} E=[e_u, e_i, f_u, f_i] \end{align}\tag{2}\] where \(e_u\) and \(e_i\) are the embeddings for user and item IDs, while \(f_u\) and \(f_i\) are the embeddings of their corresponding features.

Interaction Layer. The interaction layer receives embeddings from the embedding layer and aims to capture the interactions between features explicitly. This layer can be easily extended to capture the higher order of feature interaction. The most commonly used way is to adopt element-wise operations to capture second-order feature interaction. For simplicity, we adopt the element-wise operations without the loss of generality to capture pairwise interactions. \[\label{eqn:user95item95interaction} \begin{align} I=g(f_u, f_i) \end{align}\tag{3}\] where \(g(\cdot)\) represents the interaction function between features, and \(f_u\) and \(f_i\) are embeddings of features from user \(u\) and item \(i\). The function \(g\) may take various forms, such as inner product, plus/minus, max/min, and so forth.

Output Layer. The output layer learns the transformation and combination of features, i.e., \(E\) and \(I\), and outputs the probability score \(\hat{y}\) for the interaction between specific users and items. Typically, it consists of several fully connected layers with non-linear activation functions. The selection of activation function \(\mathrm{\sigma(\cdot)}\) depends on the specific recommendation task. In this paper, sigmoid has been chosen as the activation function.

3.2 Architecture Search Space↩︎

In this subsection, we define a search space that considers intra-layer design and inter-layer design. We propose first searching for a way of constructing cross-attention and then integrating these vectors. Specifically,

• Intra-layer: A possible way to construct cross-attention.

• Inter-layer: Interaction functions that operate on users/items and their corresponding features.

Intra-layer Design. Intuitively, in a specific application domain, users tend to concentrate on particular features that matter most to them. Similarly, within this context, each item is expected to have a distinct set of features that are particularly attractive to its potential customers. This selective focus on relevant features by both users and items is a fundamental aspect of explainable recommender systems, as it allows for the delivery of highly customized and meaningful recommendations. Inspired by this observation, we highlight the significance of these essential features by designing the attention network as: \[\label{eqn:intra-layer32design} \begin{align} \alpha_k=\frac{\mathrm{exp}(e_uM_aw_k)}{\sum_{k=1}^{p}\mathrm{exp}(e_uM_aw_k)} \end{align}\tag{4}\] where \(e_u \in \mathbb{R}^{m}\) and \(w_k \in \mathbb{R}^{p}\) are the embedding of user \(u\) and \(k\)-th feature of user review, respectively. \(M_a \in \mathbb{R}^{m\times p}\) denotes the attention matrix between user embedding and feature embedding. The attention weight for each user towards \(k\)-th feature is represented as \(\alpha_k \in (0,1)\). Likewise, each item can also obtain its own attention weight denoted as \(\beta_k \in (0,1)\) with a different attention mapping matrix \(M_b \in \mathbb{R}^{m\times p}\). Then, the attention-aggregated representations of users/items features can be obtained as follows: \[\label{eqn:tilda} \begin{align} \tilde{f_u} = [\alpha_1,\cdots,\alpha_p] \odot f_u \quad \text{and} \quad \tilde{f_i} = [\beta_1,\cdots,\beta_p] \odot f_i \end{align}\tag{5}\]

As shown in Equation 4 , the attention matrix \(M_a\) for user features is learned from user embedding, and the attention matrix \(M_b\) for item features is learned from item embedding, respectively, \(\mathrm{att}_{0}\) denotes this way of cross-attention construction. However, user preference and item characteristics often share a complex interplay: user preference may exert influence on both user and item features (denoted as \(\mathrm{att}_{1}\)), while item characteristics can also, in turn, shape user preferences and item features (denoted as \(\mathrm{att}_{2}\)). This dynamic interaction can also manifest in a reciprocal manner, where users and items cross-influence each other (denoted as \(\mathrm{att}_{3}\)). To encapsulate these multifaceted relationships, we introduce four interconnected constructions of cross-attention (\(\mathrm{att}_{0-3}\)), which is depicted in the lower right corner of Figure 2.

Inter-layer Design. Having defined the intra-layer for constructing cross-attention, the next crucial aspect of design involves the fusion of these results. As illustrated in Figure 2, the input of the feature interaction layer includes embeddings of users and items, as well as the output obtained from the cross-attention. In this context, the operation set is composed of three commonly employed element-wise interaction functions: \(O = \{\mathrm{Plus}, \mathrm{Multiply}, \mathrm{Concat}\}\). Each of these interaction functions carries its unique characteristics. The ‘Plus’ operator encourages a cooperative interaction, emphasizing the additive combination of its inputs. The ‘Multiply’ operator fosters a more selective and focused interaction, highlighting the mutual influence between its inputs, while the ‘Concat’ operator expands the feature space, allowing a more complex and expressive representation of their interaction. More specifically, \[\label{eqn:inter32layer32design} \begin{align} I=[g_1(\tilde{f_u}, \tilde{f_i}), g_2(e_u,e_i)] \end{align}\tag{6}\] where the operators \(g_1, g_2 \in O\).

It’s worth noting that the operators need not necessarily be the same, enabling us the flexibility to employ diverse interaction strategies. Moreover, to address the potential discrepancy in the embedding dimensions between users/items and the features associated with users/items, we explicitly utilize the ‘Concat’ operator, which seamlessly merges these distinct components, addressing dimension-related issues. Additionally, the second part \(g_2\) may not always be a mandatory element in our design. We consider the option of keeping one of them (\(e_i\) or \(e_u\)) or removing both of them.

As shown in Table 1, the scale of the search space is determined by both the intra-layer design and inter-layer design. Precisely, the size of the search space for the intra-layer is \(|\mathrm{att}_{0-3}|=4\), and the size of the search space for the inter-layer can be computed as \(3+2*3+3*3=18\). Thus, the possible number of architectures in the search space is \(18*4=72\).

The intra-layer and inter-layer design equip our recommender system with the versatility needed to adapt and optimize its performance across diverse recommendation scenarios. Through a careful selection and combination of these functions, we can finetune the system to provide personalized and effective recommendations that cater to the preferences and interests of every user.

3.3 Optimization Strategy↩︎

Generally speaking, the major challenge in NAS lies in its substantial computational cost of training and evaluating numerous neural network architectures. To address this issue, we introduce Bi-level NAS (Bi-NAS), a method that efficiently conducts architecture searches within the predefined search space by jointly optimizing intra-layer and inter-layer configurations in an end-to-end manner.

As depicted in Figure 3, our proposed architecture can be viewed as a directed acyclic graph (DAG) comprising sequential nodes. Each node denotes the latent representation to be learned (e.g., the embedding of users), and each directed edge denotes an operation (e.g., plus) that transforms the corresponding node. The value of each intermediate node is calculated based on all of its preceding nodes. Additionally, we introduce a special “Not connected" operation to indicate when there’s no connection between two nodes.

It is worth noting that the search space includes both discrete elements(e.g., selection of operation \(\boldsymbol{\lambda}\)) and continuous variables (e.g., hyper-parameters \(\boldsymbol{w}\)). Conducting an exhaustive search without considering the diverse search space is inherently inefficient. Inspired by recent advances in the differentiable search of NAS, we propose to relax the choices among operations as a sparse vector in a continuous space (shown in Figure 3 (a)). The main idea is to parameterize the architecture using continuous variables, making it differentiable with respect to these parameters. Thus, the gradient descent could be utilized to optimize the architecture directly. Specifically, let \(\boldsymbol{\lambda} = [\lambda_t] \in \mathbb{R}^{|O|}\) where \(\lambda_t\) denotes the weight of the \(t\)-th operation. The selection of a specific operation from a categorical choice can be relaxed to a mixture of all possible operations: \[\label{eqn:32operation32selection} \begin{align} \hat{g}=\sum_{t=1}^{|O|}\lambda_t(g_t(\tilde{f_u},\tilde{f_i}))), \: s.t. \: \boldsymbol{\lambda} \in \;\mathcal{C} \end{align}\tag{7}\] where \(g_t \in O\) is the \(t\)-th operation of \(O\), and the constraint set \(\mathcal{C}=\{\boldsymbol{\lambda}, ||\boldsymbol{\lambda}||_0=1\) and \(0 \leq \lambda_t \leq 1\}\).

At the end of searching, a discrete operation can be obtained by replacing mixed operations with the most likely operation with the largest \(\lambda\), i.e., \(g=\mathop{\mathrm{arg\,max}}_{g_i \in O} \lambda_i\) (shown in Figure 3 (c)). This constraint guarantees that only one operation can be chosen, and the model’s architecture can be reflected by \(\boldsymbol{\lambda}\). The task of operation search then reduces to learning a set of continuous variables \(\boldsymbol{\lambda}\). Our goal is to jointly learn the architecture \(\boldsymbol{\lambda}\) and the associated weights \(w\), striving for a unified system that integrates architectural design and parameter selection to improve performance and efficiency.

We optimize the following bi-level objective function:

\[\label{eqn:loss} \begin{align} \mathcal{G} =\mathop{\mathrm{arg\,min}}_{\boldsymbol{\lambda}} \mathcal{L}_{\text{val}}(\hat{w}(\lambda), \lambda) \\ \text{s.t.} \quad \hat{w} =\mathop{\mathrm{arg\,min}}_{\boldsymbol{w}} \mathcal{L}_{\text{train}}(w, \lambda) \\ \end{align}\tag{8}\]

where \(\mathcal{L}_{\text{train}}\) and \(\mathcal{L}_{\text{val}}\) are the loss on the training set and validation set. Specifically, \(\mathcal{L}=\mathcal{L}(y, \text{MLP}_{w}(\mathrm{att}_i, \hat{g}))\) is the binary cross-entropy loss, \(y\) is the ground truth label. And \(y=1\) indicates that user \(u\) interacted with item \(i\), otherwise \(y=0\). \(\mathrm{att}_i \in \mathrm{att}_{0-3}\), \(\mathrm{MLP}(\cdot)\) is the output of output layer.

3.4 Effective and Personalized Explanation Generation↩︎

To illustrate the explanations generated by our framework, we present some examples through an integrated user interface. For this demonstration, we utilize data from the Amazon-Video dataset [25] and employ Llama-3.1-8B-Instruct [26] as our primary LLM. First, we generate a general user profile based on the features they prioritize. Then, we recommended the five most relevant items based on the user’s history. For each recommended item, we leveraged the LLM to generate an explanation for why it was suggested. Figure 4 illustrates the interactive user interface used in our demonstration. The interface consists of two main sections: the top section displays multiple user accounts logged into the device, each with their respective preferences represented. For example, a dad’s profile may prioritize video games featuring combat, plot twists, realism, and cinematic excellence, while a mom’s profile may favor games centered on character development, storytelling, drama, and engaging scripts. Once an account is selected, the bottom section presents the personalized recommendations. Users can hover over each recommended item to view its detailed explanation. Unlike traditional methods that generate explanations directly using LLMs, our approach primarily derives explanations from the aligned features shared between the user and the recommended item and user history, as outlined in Figure 5. For example, the explanation why Halo is recommended for the role of dad is because: Halo delivers intense combat and stunning cinematography, with unexpected plot twists driving both its epic battle scenes and the emotional story of Master Chief and Cortana.

4 Experiments↩︎

In this section, we assess our method’s performance and explanation quality, conduct an ablation study on cross-attention effectiveness, and analyze the obtained architecture.

4.1 Experimental Settings↩︎

Dataset. We utilize the Amazon dataset [25], where the raw data is collected as a user-item-review triplet format. We conduct experiments on its top-level product categories¹: Instrument, Video, Beauty, and Clothing.

The statistics of these datasets are summarized in Table 3. In preprocessing, we keep the user-item interactions that have text reviews. We treat the 4-star and 5-star reviews as positive feedback, and the rest are unlabeled feedback where we performed negative sampling within it during training. Following this preprocessing protocol, all user and item interactions are eventually stored in the interaction matrix with binary values \(0\) and \(1\). Additionally, we utilize Sentire² to get phrase-level aspect words and sentiment.

Evaluation protocols. We randomly split the dataset into 70%, 20%, and 10% for training, validation, and testing, respectively, considering only users with more than five positive samples. We evaluate the performance of all methods by three standard measurements for ranking tasks: Hit Rate (Hit@10), Normalized Discounted Cumulative Gain (NDCG@10), and Mean Reciprocal Rank (MRR).

4.2 Baselines↩︎

We conduct comparison experiments between our method and the following state-of-the-art approaches. We follow the official implementation and use the default setting on hyperparameters for these baselines. NCF [27]: One of the most popular embedding-based neural network recommendation methods. It replaces the inner product from matrix factorization with a neural architecture that can learn an arbitrary function from data. VBPR [28]: A modified version of the original VBPR by using the neural network-based embeddings and the contextual features. CER [18]: A counterfactual explainable recommendation model. This model leverages counterfactual reasoning from causal inference to deliver explainable recommendations, aiming to provide simple and effective explanations for the model decision. NAR [29]: The attention-based intrinsic recommendation model, has intrinsically explainable designs with two major components: a representation network for users and items, and a prediction network to predict the relevance scores. Additionally, its model architecture can be searched using our Bi-NAS framework. MANAS [30]: MANAS automatically assembles the three basic logical operation modules into a network architecture tailored to the given user by neural architecture search.

4.3 Performance Evaluation↩︎

We first evaluate the overall performance of the recommendation task of our proposed method. The results are reported in Table 2. In general, we have the following observations: (1) Our proposed method can outperform all other baselines in most of the cases in terms of various evaluation metrics, regardless of the dataset size. (2) Furthermore, other models like CER and NAR also exhibit competitive performance. Specifically, CER shows competitive performance when it comes to large-scale datasets; we believe this is due to the factor that CER is more reliant on reviews, which tend to be richer and more informative in larger datasets. (3) In contrast, MANAS does not perform as well as other baseline methods. We believe this disparity in performance is partly due to this method’s reliance on sequential information without leveraging review data. (4) Our model consistently outperforms NAR, an attention-based model with a fixed cross-attention and interaction configuration. The fixed architecture does not generalize well across different dataset, which could further demonstrate the effectiveness of our method.

4.4 Explanation Evaluation↩︎

In this subsection, we want to evaluate the explainability based on the user’s review of the item, which has been widely used in previous work  [18], [31], [32]. Specifically, we utilize the review as the ground truth to explain why the item is recommended to the user. We measure the performance of the explanation by metrics utilized in other work [18], [19]. Here, \(R_{u,i}^{k}\) denotes the interactions between user \(u\) and item \(i\) regrading feature \(f_k\). If the review between user \(u\) and item \(i\) has positive sentiment on the feature \(f_k\), then \(R_{u,i}^{k}=1\); otherwise, \(R_{u,i}^{k}=0\). \[\label{eqn:32pre95recall} \begin{align} \mathrm{Precision} = \frac{\sum_{k=1}^{p}R_{u,i}^{k}\cdot I_{u,i}^{k}}{\sum_{k=1}^{p} I_{u,i}^{k}}, \mathrm{Recall} = \frac{\sum_{k=1}^{p}R_{u,i}^{k}\cdot I_{u,i}^{k}}{\sum_{k=1}^{p}R_{u,i}^{k}} \end{align}\tag{9}\] where \(I_{i,j}^{k}\) is an identity function to identify the feature explanation obtained by our model. If the obtained explanation identify item \(i\) is recommended to user \(u\) because of feature \(f_k\), then \(I_{u,i}^{k}=1\); otherwise \(I_{u,i}^{k}=0\). Precision is defined as the ratio of features that users really like in the obtained feature explanation, Recall is defined as the ratio of features explanation to the number of features that users like, and F1 is represented as the harmonic mean of Precision and Recall. NDCG can be computed similarly.

Furthermore, we randomly choose features from the feature space to generate explanations. The results of this random baseline can provide some insight into the difficulty of this task. The results are reported in Table 4. In general, we have the following observations: (1) Our method consistently performs the best in terms of all metrics, demonstrating that our method can provide effective explanations. (2) CER does not perform as well as other methods. We believe this is because CER employs item-oriented perturbations for explanations. Consequently, when explaining a user’s ranking decision, the perturbations from all the items they’ve reviewed are averaged, leading to a reduction in granularity for explanation. (3) Although NAR has an explicitly designed architecture for learning the attention weight, it remains outperformed by our method. This implies that human prior knowledge can occasionally constrain the potential of models.

4.5 Ablation Study↩︎

In this subsection, we investigate the effectiveness of the cross-attention mechanism employed in our proposed method. Figure 6 presents the results of Amazon-Beauty dataset w/o employing the cross-attention mechanism. It can be seen that the cross-attention mechanism is essential to enhance the performance of both the recommendation evaluation and the explanation evaluation. Similar trends are observed in other datasets. For brevity, we only present the key results.

4.6 Computational Cost Analysis↩︎

In this subsection, we analyze the computational cost of our proposed method. The primary expense of our method lies in the bi-level architecture search. Specifically, the intra-layer search space comprises 4 choices, and the inter-layer search space consists of 18 choices, resulting in a total of 4 × 18 = 72 potential architectures. However, rather than performing an exhaustive discrete search, we relax the discrete space into a continuous, differentiable one. This relaxation allows us to employ gradient-based optimization methods, which are markedly more efficient than traditional techniques such as reinforcement learning or evolutionary algorithms.

Table 5 summarizes the running times of each method. While the bi-level search introduces some computational overhead during the search phase, the continuous relaxation substantially reduces this burden. It leads to faster overall running times compared to conventional NAS techniques, such as MANAS.

4.7 Cross Attention and Interaction Functions Obtained↩︎

In this subsection, we show the cross-attention and interaction functions that are obtained by our proposed method in Table  6. We observed that utilizing three inputs yields better performance on the small-scale datasets. Additionally, the construction of the cross-attention mechanism exhibits slight variations across different datasets. Furthermore, there is no definitive winner among all operations, and the optimal operation may depend on the dataset. These observations demonstrate that no single architecture consistently outperforms the others, and there is no optimal architecture that works for all scenarios, supporting the notion that a universally optimal architecture is not feasible. These observations further underscore the necessity for our proposed method, which dynamically adjusts the architecture to match the specific needs and challenges of various datasets. This adaptability ensures that the recommendation model remains robust and effective across diverse contexts.

5 Related Work↩︎

In this section, we briefly introduce relevant studies, focusing on explainable recommender systems and neural architecture search for recommendations.

Explainable Recommender Systems. Recent work has highlighted the importance of building reliable and trustworthy systems [33], [34]. Explanations for recommendations are essential to enhance transparency, trust, and user engagement. While traditional models like collaborative filtering [5] and matrix factorization [6] offer generic explanations, they often fail to capture individual user preferences. To address this, more advanced methods have emerged, including sentiment-aware models that analyze user reviews [15], [35], deep learning models like DeepCoNN that encode reviews for explainability [36], [37], and aspect-based methods that select informative sentences using attention mechanisms. Some approaches employ path-based techniques [11], [12], while others utilize attention mechanisms [7], [8]. Despite these advancements, designing attention architectures and path-based explanations requires extensive human effort. LLMs have recently demonstrated remarkable versatility in various tasks [38], [39]. However, directly relying on LLMs for explanation generation can lead to issues such as hallucination, factual inaccuracies, and inconsistencies [40]. To address these challenges, we leverage NAS to automate architecture search, optimizing cross-attention structures and interaction functions. We then integrate LLMs to enhance explanation generation by the aligned features shared between users and items.

Neural Architecture Search. Designing effective neural architectures remains highly dependent on expert knowledge, making it challenging for non-experts and limiting the exploration of novel architectures. To mitigate this, NAS automates model design by optimizing network structures while minimizing human intervention [13]. The success of Zoph’s work [41] validated the feasibility of automated architecture search, leading to research focused on improving efficiency through parameter sharing [42], modular search spaces [43], and differentiable search strategies [44]. With advancements in NAS, researchers have applied these techniques to recommendation systems. AutoField [45] automatically selects essential feature fields, while AutoCross [46] discovers high-order feature interactions in a structured search space. AutoFIS [47] identifies important feature interactions through an exhaustive enumeration approach, and AutoLoss [23] adapts loss functions for optimal model performance. More recently, NASRec [48] introduced weight-sharing supernets to generate diverse recommendation architectures, while Meta’s latest studies emphasize the potential of NAS in ranking and recommendation tasks [49], [50]. Despite the growing adoption of NAS in recommendations, little research has focused on improving model explainability. Most existing NAS approaches prioritize predictive accuracy, often overlooking the need for transparent and interpretable recommendations. To bridge this gap, we propose Bi-NAS, a novel approach that searches for optimal cross-attention mechanisms and interaction functions, enhancing both recommendation accuracy and explainability.

6 Conclusion↩︎

The significance of explanations in recommender systems cannot be overstated. They serve as a critical link between users and recommended items, enhancing decision-making, building trust, and fostering user loyalty. Effective explanations can clarify the rationale behind algorithmic decisions and make the system more transparent and user-friendly. However, traditional explainable models aimed at providing explanations usually fail to achieve the desired level of interpretability that the users seek, and few studies are dedicated to quantifying the effectiveness of interpretable recommendation systems. Additionally, most current methods require substantial time and expert knowledge for the intricate construction of explainable pathways, attention mechanisms, etc. While LLMs demonstrate notable versatility in generating natural language text, directly using them for explanation generation can introduce issues such as hallucination, factual inaccuracies, a lack of true personalization, and inconsistencies. To address these challenges, this paper introduces a Bi-NAS framework. By jointly optimizing cross-attention structures and feature interaction functions, Bi-NAS promotes the generation of effective explanations and minimizes the dependency on expert knowledge. Furthermore, LLMs have been employed to enhance explanation generation from the aligned features shared between users and items and user history, enabling more personalized justifications. Our extensive experimental results demonstrate the efficacy of our method in enhancing recommender systems. This approach yields superior recommendations for the users while generating more effective explanations.

Acknowledgements↩︎

We thank the anonymous reviewers for their constructive comments. This work is supported by the National Science Foundation under Award No. IIS-2339989 and No. 2406439, DARPA under contract No. HR00112490370 and No. HR001124S0013, U.S. Department of Homeland Security under Grant Award No. 17STCIN00001-08-00, Amazon-Virginia Tech Initiative for Efficient and Robust Machine Learning, Amazon AWS, Google, Cisco, 4-VA, Commonwealth Cyber Initiative, National Surface Transportation Safety Center for Excellence, and Virginia Tech. The views and conclusions are those of the authors and should not be interpreted as representing the official policies of the funding agencies or the government.

References↩︎

---

1. https://jmcauley.ucsd.edu/data/amazon/↩︎

2. https://github.com/lileipisces/Sentires-Guide↩︎