1 INTRODUCTION↩︎

Recommender systems play a crucial role in modern information society. An effective recommender system not only provides users with higher quality and more personalized content in a world overloaded with information, but also creates more commercial value for enterprises. In recommender systems, increasing the model size has always faced various challenges [1]–[6]. However, the recently demonstrated scaling law in autoregressive generative recommendation models [7]–[10] offers the possibility of creating large-scale recommender systems with better performance. Although larger models improve performance, they also introduce several cost and efficiency issues: (1) Training and serving a large-scale recommendation model requires substantial GPU computational resources. If the computing cost significantly exceeds the commercial value it can generate, then the model loses its value. (2) Most recommendation applications have stringent latency constraints (usually less than 100ms) [8], [11]–[16] , which poses significant challenges for the deployment of large-scale recommendation models.

Recent autoregressive generative recommendation models, such as HSTU [8] and FuXi-\(\alpha\) [17], are based on the Transformer [18] architecture. To address the efficiency challenges posed by increasing model sizes, we revisited recent explorations aimed at optimizing the efficiency of Transformer-like structures. Notable strategies include low-precision quantization [19]–[21], distillation of smaller models using larger teacher models [22], [23], and pruning of model weights [24]–[26]. Additionally, there are numerous methods for directly designing more efficient models, such as simplifying the model [27], accelerating through sparse attention weights [28], [29], utilizing kernel methods for attention acceleration [30], employing mixture of experts models [31], and leveraging RNN characteristics for acceleration [32], [33]. However, the training processes for pruning and distillation are relatively complex, as there are additional steps conducted after the base model training, and thus, they cannot directly expedite the training process. Moreover, quantized models necessitate hardware support, and deploying them on different GPUs may result in model failure. Some quantization methods [19], [20] still do not accelerate the training process. Although numerous efficient Transformer-like structures exist, they are tailored for NLP tasks and may not be suitable for recommendation tasks. Currently, the structural design for efficient generative recommendation models remains underdeveloped.

To design a more efficient structure for recommendation tasks, we analyzed the execution times of various operators during training FuXi-\(\alpha\) [17] and HSTU [8]. Among the top three operators with the highest execution times, the one with the highest execution time is associated with computing the query-key attention map. The second and third highest execution times both pertain to the indexing operator in the Relative Attention Bias (RAB) module for timestamps. On the one hand, the index operators used in RAB require noncontiguous memory access, which are not hardware-friendly, and the number of these operations is related to the square of the sequence length. These factors result in the actual runtime of the RAB being slower than the matrix multiplication, even though the latter may have a higher computational complexity. On the other hand, calculating the query-key attention map aims to facilitate feature interaction among items, but this process incurs significant computational and memory costs. Previous research suggests that the attention map used in self-attention does not necessarily need to be generated in this manner. For example, using a fixed attention map [34] or a learnable constant attention map [35] can still achieve comparable results. In FuXi-\(\alpha\) and HSTU, aside from the query-key attention map, the bias term produced by RAB can itself be regarded as an attention map. This naturally raises the question: Are some of these attention maps redundant? To answer this question, we conducted empirical experiments on multiple datasets and found that removing the temporal or spatial attention maps results in varying degrees of performance degradation. However, removing the query-key attention map, conversely, improved performance. This suggests that the query-key attention map might be redundant in recommendation tasks.

To address the aforementioned issues, we propose a new framework that can be utilized in various Transformer-like models. On the one hand, we introduced the Functional Relative Attention Bias (FRAB) module, which addresses the inefficiency of RAB by avoiding various hardware-unfriendly operations inherent in RAB. On the other hand, we removed the query-key attention map from the original self-attention layer and designed a new module called Attention-Free Token Mixer (AFTM). Subsequently, we apply this framework to the prior state-of-the-art generative recommendation model and introduce a new model, FuXi-\(\beta\). And FuXi-\(\beta\) achieves performance comparable to the prior state-of-the-art models on public datasets while significantly reducing training time. On large-scale industrial datasets, it achieves both effective acceleration and notable performance improvements. Additionally, we have validated on industrial datasets that our model exhibits the properties of scaling laws, demonstrating its potential for application in large-scale recommendation systems. Our contributions are as follows:

• We propose a new framework that can simplify Transformer-like generative recommendation models and introduce a new efficient and lightweight generative recommendation model, FuXi-\(\beta\).

• We empirically find that the query-key attention map can have a negative impact on recommendation tasks. Additionally, we designed a new module called Attention-Free Token Mixer.

• We propose a novel and efficient method for modeling relative temporal information, termed Functional Relative Attention Bias.

• Experiments on four real-world datasets demonstrate that FuXi-\(\beta\) outperforms several state-of-the-art models in performance and efficiency. Furthermore, its scaling law properties suggest potential for large-scale recommendation systems.

2 RELATED WORK↩︎

2.1 Positional and Temporal Embedding Methods↩︎

Due to the inability of self-attention to capture the sequential order, it is necessary to incorporate positional encodings as an additional component. Positional encodings can be categorized into absolute and relative positional encodings. Absolute positional encodings involve directly adding a position embedding to each token’s hidden representation. Common approaches include Sinusoidal positional encodings [18] and learnable positional encodings [36]. In contrast, relative positional encodings consider the relative distances between tokens when computing attention weights. In 2018, Shaw et al. [37] introduced the concept of relative positional encodings for the first time. Subsequent works [38]–[41] have proposed various forms of relative position encoding by considering the interaction terms between positions and tokens, as well as between positions themselves. T5 [42] incorporates a learnable bias based solely on relative position information directly into the attention weights. Huang et al. [43] propose more sophisticated relative positional encodings, asserting that existing methods do not fully leverage the potential of relative positions.

In the domain of sequential recommendation, it is crucial to consider both positional and temporal information [8], [44]–[47]. TiSASRec [44] and STAN [48] model time intervals by segmenting them into discrete bins, each associated with an embedding vector added to the key vector. MEANTIME [49] applies various functions to the relative time differences to obtain temporal embeddings, which are then incorporated into the key vector. HSTU [8] employs a logarithmic bucketing approach for time intervals, followed by a T5-style relative position encoding to integrate temporal information.

In our approach to modeling time intervals, we adopt the T5-style relative positional encoding but replace bucketing and learnable vector parameters with learnable functions, thereby enhancing computational efficiency.

2.2 Efficient Transformer-based Models↩︎

Transformer-based models have been widely applied across various domains [18], [50]–[56]. However, the quadratic time and space complexity of self-attention with respect to sequence length imposes significant limitations. Consequently, numerous algorithmic strategies have been proposed to reduce these complexities. Techniques such as model pruning [24]–[26], which removes less important weights to reduce parameter count and improve computational efficiency, and model quantization [19], [20], which compresses models by converting high-precision floating-point parameters to lower-precision floats or integers, are commonly used. Knowledge distillation [22], [23], [57], [58] further accelerates models by training a smaller student model under the guidance of a larger teacher model, effectively transferring learned knowledge.

Additionally, designing more efficient model architectures is also an option to address these challenges [27], [32], [59]–[61]. ALBERT [62] reduces parameters through low-rank factorization of embedding layers and cross-layer parameter sharing. Approaches such as MQA [63] and GQA [64] decrease memory usage by reducing the number of query and value heads. MLA [65] builds on GQA by incorporating additional projections to enhance performance while further compressing the KV cache. However, these efforts can only accelerate the inference process. He et al. [27] simplified the Transformer architecture by removing components such as the value projection matrix and normalization layers. FLASH [59] achieves simplification by integrating Self-attention layers with Gated Linear Units (GLU), while reducing the complexity and memory overhead of self-attention through chunking and linear attention mechanisms. RWKVs [33], [66] and RetNet [32] modified the attention mechanism to achieve the low-cost inference characteristic of RNNs, while simultaneously maintaining the capability for parallel training.

3 PROBLEM STATEMENT↩︎

In the paradigm of generative recommendation, the primary objective is to predict the next item a user is likely to interact with, based on their historical interaction sequence. Besides the item and position information, we need to pay additional attention to the timestamp of each interaction. Formally, consider a set of users \(\mathcal{U} = \{u_1, u_2, \ldots, u_{|\mathcal{U}|}\}\) and a set of items \(\mathcal{I} = \{i_1, i_2, \ldots, i_{|\mathcal{I}|}\}\). For each user \(u \in \mathcal{U}\), we define an interaction sequence

\[\begin{align} \mathcal{S}_u = \left [\left (i_1^{(u)}, t_1^{(u)}, 1\right), \left (i_2^{(u)}, t_2^{(u)}, 2\right), \ldots, \left (i_{n_u}^{(u)}, t_{n_u}^{(u)}, n_u\right) \right ] \end{align}\] which is a chronologically ordered list of items, their corresponding timestamps and positions.

The task is to predict the next item \(i_{n_u+1}^{(u)}\) that user \(u\) will interact with, based on their interaction sequence \(\mathcal{S}_u\). This prediction can be formulated as estimating the probability distribution over the item set \(\mathcal{I}\) for the next interaction, conditioned on the historical interactions: \(P(i_{n_u+1}^{(u)} = i \mid \mathcal{S}_u)\) for all \(i \in \mathcal{I}\). During training, our objective is to predict the subsequent item \(i^{(u)}_{j + 1}\) for every prefix \(j\) of the given sequence \(\mathcal{S}_u\). For an input sequence \(\mathcal{S}_u\), the desired output sequence is \([i_2^{(u)}, i_3^{(u)}, \ldots, i_{n_u +1}^{(u)}]\) [51].

4 PRELIMINARY↩︎

4.1 Decoder-only Transformer↩︎

The standard Transformer [18] comprises an encoder and a decoder. The encoder employs a bidirectional attention mechanism, allowing each token to attend to all other tokens. In contrast, the decoder uses a unidirectional attention mechanism, where each token can only attend to its preceding tokens. Consequently, decoders are often used independently for various autoregressive tasks [67], [68].

A Transformer Decoder is constructed by stacking multiple Decoder Blocks, each primarily comprising one multi-head attention layer and one feed-forward neural network. Let the input to a Decoder Block be \(\boldsymbol{X} \in \mathbb{R}^{n\times d}\), where \(n\) is the sequence length and \(d\) is the embedding dimension. The \(i\)-th row of \(\boldsymbol{X}\) represents the latent vector of the \(i\)-th token. To ensure training stability, pre-norm is typically employed in the decoder [69]. Denote the multi-head attention as \(f_{\text{MHSA}}\) and the feedforward neural network as \(f_{\text{FFN}}\). Thus, the final output of the decoder, \(\boldsymbol{Y}\), is given by: \[\begin{align} \boldsymbol{M} &= \boldsymbol{X} + f_{\text{MHSA}}(\text{norm}(\boldsymbol{X})) \\ \boldsymbol{Y} &= \boldsymbol{M} + f_{\text{FFN}}(\text{norm}(\boldsymbol{M})) \end{align}\]

4.1.1 Multi-Head Self-Attention↩︎

The multi-head self-attention consists of \(h\) heads, each with a size of \(d_h = d/h\). In each head, query, key, and value vectors are computed through projection. The attention score between any two tokens is calculated by the dot product of a token’s query vector and another’s key vector, followed by normalization and multiplication with the value vector to yield the output of this head, \(\boldsymbol{h}_i\): \[\begin{align} \boldsymbol{h}_i = \text{mask}\left (\varphi((\boldsymbol{X}\boldsymbol{W}_q^{(i)}) (\boldsymbol{X}\boldsymbol{W}_k^{(i)})^T) \right )(\boldsymbol{X}\boldsymbol{W}_v^{(i)}) \end{align}\] where \(\boldsymbol{W}_q^{(i)}, \boldsymbol{W}_k^{(i)}, \boldsymbol{W}_v^{(i)} \in \mathbb{R}^{d\times d_h}\) are learnable parameters, and \(\varphi\) is the normalization function, typically softmax in NLP, while SiLU is found to be more effective in recommendation tasks [8]. The function \(\text{mask}(\boldsymbol{X})\) applies a causal mask to the matrix \(\boldsymbol{X}\) by setting its upper triangular part (excluding the diagonal) to zero. The final output of the multi-head self-attention is: \[\begin{align} f_{\text{MHSA}}(\boldsymbol{X}) = \text{concat}(\boldsymbol{h}_1, \dots, \boldsymbol{h}_h)\boldsymbol{W}_o \end{align}\] where \(\boldsymbol{W}_o \in \mathbb{R}^{d\times d}\) is a learnable parameter.

4.1.2 Feed-Forward Neural Network↩︎

In a feed-forward neural network, the input vector is first projected into a higher-dimensional space and then passed through a nonlinear activation function. This process can be represented mathematically as follows: \[\begin{align} f_{\text{FFN}}(\boldsymbol{X}) = \phi(\boldsymbol{X} \boldsymbol{W}_1) \boldsymbol{W}_2 \end{align}\] where \(\boldsymbol{W}_1 \in \mathbb{R}^{d\times d_{\text{FFN}}}, \boldsymbol{W}_2 \in \mathbb{R}^{d_{\text{FFN}}\times d}\) are learnable parameters, and \(\phi\) denotes a non-linear activation function.

4.2 Relative Attention Bias↩︎

To address the inability to perceive the position of tokens when calculating attention weights, absolute or relative positional encodings are incorporated to add positional information. Relative Attention Bias (RAB), proposed in T5 [42], is a method of relative positional encoding that adds a trainable bias term directly to the attention map, denoted as \(\boldsymbol{B} = (b_{i, j})_{n\times n} \in \mathbb{R}^{n\times n}\): \[\begin{align} \boldsymbol{h}_i = \text{mask}(\varphi((\boldsymbol{X}\boldsymbol{W}_q^{(i)}) (\boldsymbol{X}\boldsymbol{W}_k^{(i)})^T + \boldsymbol{B}))(\boldsymbol{X}\boldsymbol{W}_v^{(i)}) \end{align}\] The bias term is calculated using \(b_{i,j} = \boldsymbol{\beta}_{f(i - j)}\), where \(\boldsymbol{\beta} \in \mathbb{R}^{d_{\text{rab}}}\) represents learnable parameters, and \(f\) is a bucketing function that maps the relative position to one value of \(\boldsymbol{\beta}\). Similarly, we can express relative temporal information as: \[\begin{align} b_{i,j}^t = \boldsymbol{\beta}^t_{f_t(t_i - t_j)} \end{align}\] where, \(t_i\) denotes the timestamp of the \(i\)-th item, and \(f_t\) represents the bucketing function for relative time. In HSTU [8], the attention map is computed as follows: \[\begin{align} \boldsymbol{A} = (\boldsymbol{X}\boldsymbol{W}_q) (\boldsymbol{X}\boldsymbol{W}_k)^T + \boldsymbol{B} + \boldsymbol{B}^t \end{align}\] In FuXi-\(\alpha\) [17], the relative time and position bias terms, \(B\) and \(B^t\), are directly applied to individual heads as the attention map. In subsequent sections, we will refer to this method as the "bucketed relative attention bias" for differentiation purposes.

5 METHODOLOGY↩︎

We introduce our newly proposed framework based on the current state-of-the-art generative recommendation model, FuXi-\(\alpha\), and we refer to the new model as FuXi-\(\beta\). Although we present this framework based on FuXi-\(\alpha\), it can be applied to other Transformer-like generative recommendation models. FuXi-\(\beta\) is also a Transformer-like model, mainly consisting of \(L\) layers of stacked FuXi-\(\beta\) Blocks, as shown in Figure 1. In the following subsections, we will introduce FuXi-\(\beta\) in the following order: (1) an embedding layer that provides item embeddings and absolute positional information; (2) a new and fast module in the framework to model relative temporal information: the FRAB module; (3) a method within the framework to simplify the self-attention layer; (4) a discussion of other details within the FuXi-\(\beta\) Block; (5) prediction and training loss.

5.1 Embedding Layer↩︎

To address the variability in user interaction sequence lengths, we standardize these sequences into a fixed length \(n\) through truncation or padding with a special "padding item" prior to the embedding layer. In the embedding layer, each unique item \(i \in \mathcal{I}\) is mapped to a \(d\)-dimensional vector using a learnable embedding matrix \(\mathbf{E} \in \mathbb{R}^{|\mathcal{I}| \times d}\). Additionally, we incorporate absolute positional information into each token in the embedding layer by using learnable positional encodings [36]. Let the positional embedding vector for the \(i\)-th position be \(\boldsymbol{p}_i\), and the embedding of the \(i\)-th item be \(\boldsymbol{e}_i\). For a user \(u\) with sequence \(\mathcal{S}_u = \left[i_{1}^{(u)}, \ldots, i_{n_u}^{(u)}\right]\), the output of the embedding layer is expressed as \[\begin{align} \mathcal{S}_u = \left[\boldsymbol{e}_{1}^{(u)} + \boldsymbol{p}_1, \ldots, \boldsymbol{e}_{n_u}^{(u)} + \boldsymbol{p}_{n_u}, \boldsymbol{0}, \cdots, \boldsymbol{0}\right] \end{align}\] where \(\boldsymbol{0}\) denotes the positions that are padded.

5.2 Functional Relative Attention Bias↩︎

The bucketed relative attention bias used for modeling relative time was inefficient due to numerous time-consuming non-continuous indexing operations. To address this inefficiency, the Functional Relative Attention Bias is proposed, which only utilizes special function and arithmetic operations to model relative temporal information. Let the timestamp of the \(i\)-th item be denoted as \(t_i\). The attention bias for modeling the relative temporal information are computed via the following equation: \[b^t_{i,j} = f(t_j - t_i)\] where \(f\) is a monotonically function of the following form: \[f(x) = a(1 + x)^{-b}\] here, \(a\) and \(b\) are learnable parameters. Because the given sequence is arranged in a chronological order, we assume \(x\) is non-negative. To prevent numerical issues with identical timestamps, we add 1 to \(x\) before applying the power function.

5.3 Attention-Free Token Mixer↩︎

In the self-attention layer, beyond deriving the attention map from the query-key multiplication, the bias term from the relative attention bias module can also act as an attention map. Our ablation experiments, detailed in Section 6.4.3, showed that the query-key derived attention map is unnecessary for feature interaction among items. Consequently, we removed the query and key matrices in a Transformer-like architecture and utilized the attention map from the relative attention bias module to facilitate inter-item feature interaction. We refer to this new inter-item interaction layer as the Attention-free Token Mixer (AFTM). Let \(\boldsymbol{X} \in \mathbb{R}^{n \times d}\) denote the latent vector representation of the input items, and let \(\boldsymbol{t} \in \mathbb{R}^{n}\) represent the sequence of timestamps corresponding to these items. Initially, the matrices \(\boldsymbol{U}\) and \(\boldsymbol{V}\) are computed through projection as follows: \[\begin{align} \boldsymbol{U} = \phi(\boldsymbol{X} \boldsymbol{W}_u), \boldsymbol{V} = \phi(\boldsymbol{X} \boldsymbol{W}_v) \end{align}\] where \(\phi\) represents the SiLU activation function, and \(\boldsymbol{W}_u \in \mathbb{R}^{d \times 2d}\) and \(\boldsymbol{W}_v \in \mathbb{R}^{d \times 2d}\) are learnable parameters. Subsequently, the Relative Attention Bias (RAB) is used to transform positional information into an attention map \(\boldsymbol{B}\), and the Functional Relative Attention Bias (FRAB) is used to convert temporal information into an attention map \(\boldsymbol{B}^t\). The output of the AFTM is given by: \[\begin{align} f_{\text{AFTM}}(\boldsymbol{X}, \boldsymbol{t}) = \boldsymbol{U} \odot \text{concat}(\boldsymbol{B} \boldsymbol{V}, \boldsymbol{B}^t \boldsymbol{V}) \end{align}\]

5.4 FuXi-\(\beta\) Block↩︎

Each FuXi-\(\beta\) Block consists of an Attention-Free Token Mixer (AFTM) and an Multistage FFN (MFFN) utilizing SwiGLU [70] as the activation function, and employs RMSNorm [71] as the normalization function. The MFFN is a variant of FFN introduced in FuXi-\(\alpha\) [17]. The specific computational process is as follows: \[\begin{align} \boldsymbol{M} &= f_{\text{AFTM}}(\text{RMSNorm}(\boldsymbol{X}), \boldsymbol{t}) \\ \boldsymbol{f}_{\text{block}}(\boldsymbol{X}, \boldsymbol{t}) &= f_{\text{MFFN}}(\boldsymbol{M}, \boldsymbol{X}) \end{align}\]

5.5 Prediction Layer & Optimization objective↩︎

Following the passage through \(L\) layers of FuXi-\(\beta\) blocks, each position has garnered ample information related to the items that previously interacted. We implement a multiplication with the transpose of the input embedding matrix, which is subsequently subjected to a softmax function, thereby generating a probability distribution across the predicted items. This transformation can be mathematically articulated as follows: \[\begin{align} P\left(i_{j}^{(u)} = i \mid (i_1^{(u)}, t_1^{(u)},1), \dots, (i_{j-1}^{(u)}, t_{j-1}^{(u)},j-1) \right) = \text{softmax}\left (\mathbf{x}^{B} \mathbf{E}^T \right)_i \end{align}\] To expedite the training process, we incorporate the sampled softmax loss utilizing \(N\) randomly selected negative samples [72].

5.6 Complexity Analysis↩︎

-0.1cm

In this section, we theoretically analyze the complexity variations of several Transformer-like models, namely LLaMa [73], HSTU [8], and FuXi-\(\alpha\) [17], before and after applying our framework. We assume that in the self-attention layer, the condition \(d_h \times h = d\) holds, where \(d_h\) represents the projection dimension of each attention head, and \(h\) denotes the number of attention heads. Definitions of other variables can be found in the previous sections. Since the discontinuous index operations involved in the bucketed relative attention bias modules are significantly more time-consuming than arithmetic operations, like multiplication and addition, we list them separately in our analysis. The computed complexities for different models are presented in Table 1.

• LLaMa’s self-attention requires one computation of QKV projections and output projection, with a computational cost of \(4nd^2\), and one computation of attention weights, followed by multiplication with the V matrix, which incurs an additional cost of \(2n^2d\). HSTU incurs an extra computational cost of \(nd^2\) for calculating the U matrix. In FuXi-\(\alpha\), the dimension of the concatenated vector of all the channels is three times the embedding dimension. Therefore, this adds an extra \(4nd^2\) to the complexity compared to HSTU, along with an additional \(2n^2d\) due to the calculation of weights from two additional attention maps.

• In LLaMa-\(\beta\) and HSTU-\(\beta\), there is no need to compute queries and keys, nor their dot products, thereby reducing the complexity by \(2nd^2 + n^2d\) compared to their respective base models. In FuXi-\(\beta\), the dimension of the concatenated vector is reduced, resulting in a computational reduction of \(4nd^2 + 2n^2d\) compared to FuXi-\(\alpha\).

• In both HSTU and FuXi-\(\alpha\), the application of FRAB can reduce time consumption by avoiding discontinuous index operations. However, LLaMa uses RoPE [74] to utilize positional information, and replacing it with FRAB would increase computational complexity. Nonetheless, this substitution addresses the issue that RoPE encoding cannot simultaneously utilize both temporal and positional information.

In a word, our method can reduce approximately half of the computational load in the self-attention layer and replace the high time-consuming indexing operations in the bucketed relative attention bias with more efficient calculations.

6 EXPERIMENTS↩︎

-0cm -0.1cm

0cm 0cm

-0.0cm -0.2cm

-0.0cm -0.2cm

-0.0cm -0.2cm

-0.0cm -0.2cm

-0.0cm -0.2cm

6.1 Experiment Setup↩︎

6.1.1 Datasets↩︎

We conducted experiments on two public datasets (MovieLens-1M and MovieLens-20M) and two private large-scale industrial datasets (Daily Recommendations and All Scenarios). The details of these datasets are outlined below:

• MovieLens¹. MovieLens is a popular benchmark dataset for movie recommendation. The dataset comprises various subsets of different sizes. In our work, we use the MovieLens-1M and MovieLens-20M subsets for our experiment.

• Daily Recommendations. In the daily recommendation scenario, 30 exclusive songs are selected for each user every day. The displayed list is updated in a daily manner. We sampled a portion of the data to form an offline dataset, which includes tens of millions of users and billions of interactions.

• All Scenarios. The all scenarios dataset includes user behaviors from all music recommendation scenarios, with a more complex and challenging data distribution, which can better reflect the effectiveness and robustness of the model. We also sampled tens of millions of users and billions of interactions to build an offline dataset.

The statistical information about the datasets is presented in Table 2. For the public datasets, we employed the same preprocessing techniques and the train/validation/test set partitioning method as used in HSTU [8] and FuXi-\(\alpha\) [17]. For the industrial datasets, we applied similar preprocessing methods.

6.1.2 Baselines↩︎

We conducted a validation of FuXi-\(\beta\) in two aspects: performance and efficiency. For performance, we employed traditional models (BPRMF [75], GRU4Rec [76], and NARM [77]) as well as autoregressive generative models (SASRec [51], LLaMa [73], HSTU [8], and FuXi-\(\alpha\) [17]) as baselines. Regarding efficiency, we also introduced some work on efficient transformers in NLP, including FLASH [59], and RetNet [32].

6.1.3 Evaluation Metrics↩︎

We employ three widely used metrics (NDCG@\(K\), HR@\(K\), and MRR) to evaluate the recall quality of the model. Higher values of these metrics indicate better performance. We rank all items based on the model’s results and report the outcomes for \(K = 10\) and \(K = 50\) by default. Additionally, we measure the efficiency of the model using the total training time; a shorter training time implies a faster model.

6.1.4 Parameter Settings↩︎

We implemented our model, FuXi-\(\beta\), using PyTorch [78], and employed the Accelerate library [79] to achieve multi-NPU and multi-machine parallel training. In public datasets, we adopted the same parameters as HSTU [8], except for the number of layers. For models with a Feedforward Neural Network (FFN), the hidden layer width is the same as the embedding dimension in MovieLens-1M, while it is four times the embedding dimension in MovieLens-20M. In experiments on public datasets, we set the number of layers for the base model to 2, whereas the Large model refers to the corresponding model with 8 layers. For industrial datasets, we set the embedding size to 256 and the number of layers to 4 for all models to maintain consistency with the online version.

6.2 Performance Comparison↩︎

6.2.1 Public Dataset Performance↩︎

The results of FuXi-\(\beta\) and the baseline models on the public datasets are presented in Table 2. We observe the following:

• Generative recommendation models generally outperform traditional models, even when the model has only 2 layers.

• As an early model, SASRec’s structural design may lead to training difficulties. Under our training parameters, increasing the layers to 8 actually resulted in a decline in performance. However, other generative models have successfully increased their layers to 8.

• Although FLASH outperforms the Transformer in NLP, it performs poorly in recommendation tasks. RetNet, despite having equally performance compared to models like LLaMa and HSTU, still lags behind the strongest baseline, FuXi-\(\alpha\), due to its lack of adaptation to recommendation tasks.

• FuXi-\(\beta\) generally exhibits performance similar to FuXi-\(\alpha\). When using a 2-layer model, its performance on MovieLens-20M slightly decreases, while in other cases, it shows a modest improvement.

6.2.2 Industrial Dataset Performance↩︎

Table 4 shows the performance comparison of our proposed FuXi-\(\beta\) against several baseline models using two large-scale private industrial datasets. Our FuXi-\(\beta\) model outperforms both FuXi-\(\alpha\) and HSTU across two datasets. In the first dataset, the NDCG@10 metric has improved by 47.4% and 73.5% respectively, compared to FuXi-\(\alpha\) and HSTU. In the second dataset, the improvements are 27.2% and 38.2% respectively. This may be attributed to the fact that industrial datasets are more complex than public datasets. The incorporation of queries and keys in the self-attention layer, along with bucketed relative attention bias, increases the model’s complexity, making it prone to overfitting. However, Our framework mitigates the issue of overfitting, resulting in an exceptional performance on the industrial datasets.

6.2.3 Scaling on Industrial Dataset↩︎

We investigated the performance changes of FuXi-\(\beta\) as the model size increases on both industrial datasets, with the results shown in Figure 2. Due to memory constraints, we only increase the number of layers up to 32 layers. Additionally, the performance of FuXi-\(\beta\) continuously improves as the number of layers increases. This illustrates that FuXi-\(\beta\) exhibits excellent performance on large-scale datasets while still adhering to the scaling law, which states that the model’s performance continues to improve with the increase in data, model size, and computational resource.

6.3 Efficiency Comparison↩︎

6.3.1 Public Datasets.↩︎

Table 3 presents the total training time for different models over the same number of epochs. Due to the significant structural and performance differences between traditional models and generative recommendation models, we do not consider the efficiency of these traditional models. Although FLASH have a lower complexity [59], it do not exhibit a noticeable speed advantage when the sequence length is short. RetNet exhibits relatively fast training speeds. However, due to its lack of adaptation for recommendation problems, its performance still falls short compared to some recommendation models. The training times for LLaMa, HSTU, and FuXi-\(\alpha\) are gradually increasing as the models become more complex. FuXi-\(\beta\), when using 2 layers, reduces the training time by 5.51% and 10.6% on two datasets, respectively, compared to FuXi-\(\alpha\). When using 8 layers, the reductions in training time are 13.1% and 27.0%, respectively.

6.3.2 Industrial Datasets.↩︎

Table 4 presents the training times of different models on industrial datasets. For LLaMa, we employed the same relative attention bias as HSTU instead of RoPE [74], which results in a longer training time on industrial datasets compared to HSTU. FuXi-\(\alpha\), due to its more complex architecture, lagged behind LLaMa in terms of speed. In contrast, FuXi-\(\beta\), by applying our new framework to simplify its structure, reduced training time by 15.8% and 20.8% on the two datasets compared to FuXi-\(\alpha\), respectively.

6.4 Ablation Study of FuXi-\(\beta\)↩︎

6.4.1 Component Ablation Study↩︎

We first investigated the impact of different components on the model. We denote FuXi-\(\beta\)-FRAB as the model that replaces the RAB module in FuXi-\(\alpha\) with the FRAB module to utilize relative temporal information, and the model that removes all query and key matrices from FuXi-\(\alpha\) as FuXi-\(\beta\)-AFTM. The experimental results on public datasets are shown in Table 5. Although using FRAB alone and removing queries and keys led to a certain degree of performance degradation on MovieLens-1M, the combination of these two modules still achieved performance close to that of FuXi-\(\alpha\). On MovieLens-20M, these modifications resulted in a slight performance enhancement. On MovieLens-1M, they reduced the training time for the Large model by 8.68% and 1.78%, respectively, while on MovieLens-20M, they reduced the training time by 13.3% and 14.2%, respectively. This indicates that in smaller models, the acceleration is mainly due to FRAB, whereas in larger models, both contribute significantly to the acceleration.

6.4.2 Function Variations on Functional Relative Attention Bias↩︎

To validate the effectiveness of the function employed in our FRAB module, we conducted experiments by replacing it with various alternative functions. The functions considered for this purpose are as follows:

1. Linear function: \(f_{\text{lin}}(x) = a \cdot x + b\)

2. Logarithmic function: \(f_{\text{log}}(x) = a \cdot \log(1 + \exp(b) \cdot x) + c\)

3. Exponential function: \(f_{\text{exp}}(x) = a \cdot \exp(-\exp(b)\cdot x)\)

4. Sine function: \(f_{\text{sin}}(x) = c \cdot \sin(a \cdot x + b) + d\)

5. Power function: \(f_{\text{pow}}(x) = a \cdot (1 + x)^{-b}\), which is used in our module

6. Mixed function: \(f_{\text{mix}}(x) = \frac{f_\text{lin}(x) + f_\text{log}(x) + f_\text{exp}(x) + f_\text{sin}(x) + f_\text{pow}(x)}{5}\)

7. A small 3-layer Multi-Layer Perceptron (MLP) \(f_{\text{nn}}\), utilizing sinusoidal functions and SiLU as activation functions.

8. Zero function: \(f_{\text{zero}}(x) = 0\)

9. Bucket function: \(f_{\text{bucket}}\) represents the method used in relative attention bias [8].

In the above functions, \(a\), \(b\), \(c\), and \(d\) represent learnable parameters unique to each function. Our experimental results on MovieLens-1M are presented in Table 6. From the results, replacing the power function we used with other functions results in varying degrees of performance degradation.

To further investigate the effects of different functions, we visualize the attention weights in the temporal channel with the time difference as the horizontal axis and the attention weight as the vertical axis. The results of the plot are shown in Figure 3. We have the following observations:

• The function \(f_{\sin}\) varies periodically at a fixed frequency, but it is not effective in distinguishing between items that are far apart and those that are close together.

• The functions \(f_{\text{mixed}}\) and \(f_{\text{NN}}\) are difficult to train due to their complex models, resulting in suboptimal performance.

• The functions \(f_{\text{lin}}\) and \(f_{\text{log}}\) fail to adequately control the numerical range, leading to unreasonable weights assigned to items that are far apart, thus resulting in poor performance.

• The functions \(f_{\text{pow}}\) and \(f_{\text{exp}}\) have absolute values that monotonically decrease, which assigns higher attention weights to recent items. Additionally, the rate of decrease slows down, making it easier to distinguish recent features, leading to good performance. Since \(f_{\text{pow}}\) can be expressed as \(ae^{-b\ln(1+x)}\), it has a slower decay rate compared to \(f_{\text{exp}}\). Therefore, \(f_{\text{pow}}\) is more effective at assigning weights to items over longer distances, which is why it performs slightly better than \(f_{\text{exp}}\).

6.4.3 Ablation Study of Simplifying the Self-Attention Layer↩︎

To investigate whether there is redundancy among the three types of attention maps (query-key, temporal, and positional) in the self-attention layer for recommendation tasks, we empirically conducted ablation experiments based on FuXi-\(\beta\)-FRAB-Large. The results are shown in Table 7. Removing the temporal attention map significantly degrades the model’s performance. Removing the positional attention map leads to a performance drop on MovieLens-1M. In contrast, removing the query-key attention map not only results in the fastest execution time but also achieves the best performance on both datasets.

6.4.4 Compatibility Analysis↩︎

We validated our proposed framework across multiple models. Since RoPE used by LLaMa requires multiplying the query and key vectors to encode relative positional information, which is incompatible with the AFTM module, we replaced it with relative attention bias in LLaMa-\(\beta\). In addition, we applied the FRAB module to process timestamps and removed the query and key matrices. In HSTU-\(\beta\), we made modifications in a manner similar to FuXi-\(\beta\). The results are shown in Table 8. For LLaMa, due to the application of fusion operators for acceleration and the low time complexity of RoPE, the training time for LLaMa-\(\beta\) on the MovieLens-1M dataset increased. However, on the MovieLens-20M dataset, the size of the query and key matrices increased so their removal can significantly reduce the training time, resulting in an 11.8% decrease for LLaMa-\(\beta\). The performance of LLaMa-\(\beta\) was significantly improved due to the addition of temporal information. For HSTU-\(\beta\), the training time was reduced by 15.5% and 28.3% on the two datasets, respectively, but its performance significantly declined. This decline may be attributed to HSTU’s weakened implicit feature interaction, which hinders the effective utilization of information obtained from explicit feature interactions. For FuXi-\(\alpha\), the training time was reduced by 13.1% and 27.0% on the two datasets, respectively, with a slight improvement in accuracy.

7 CONCLUSION↩︎

In our work, we focus on simplifying generative recommendation models, and we propose a new framework. On the one hand, we introduce the Functional Relative Attention Bias module, which employs special functions and arithmetic operations to avoid the time-consuming indexing operations in the bucketed relative attention bias, thereby accelerating the process. On the other hand, we found that the query-key attention map in self-attention might yield negative effects in recommendation tasks. Therefore, we removed the query-key attention map from the original self-attention layer and designed a new module called the Attention-free Token Mixer. Based on this framework, we propose a new model, FuXi-\(\beta\). Comprehensive experiments demonstrate the remarkable efficiency of FuXi-\(\beta\) and its outstanding performance. In the future, we plan to further reduce the complexity of the model and aim to address more complex recommendation tasks.

References↩︎