1 Introduction↩︎

Long-term time series forecasting (LTSF) is a key task in machine learning, with wide applications in areas like energy management [@energy_1], financial markets [@finance_1], and traffic prediction [@traffic_1; @traffic_2]. Recent Transformer-based models have drawn inspiration from computer vision [@VIT], increasingly using patch-based strategies [@PatchTST; @Crossformer; @Pathformer; @timexer] to better capture long-range dependencies. Most of these approaches treat variables independently (VI) [@TimeBase; @SparseTSF], while variable-dependent (VD) methods [@iTransformer; @ModernTCN] that model interactions across variables have not yet shown clear gains over VI baselines.

Unlike 2D images with a fixed spatial structure, time series are one-dimensional curve [@shape_3; @shape_4], with the primary focus being on capturing shape similarities between patches or variables [@shape_1; @shape_2] as well as modeling long-range trend [@RLinear]. For instance, two patches may share similar trends, such as gradual increases with comparable rates of change. Identifying such shape correspondence helps the model extract temporally consistent patterns and improves forecast accuracy. However, time series data is inherently non-stationary [@DishTS; @nonstationary_1], where patch scales fluctuate wildly across time. As illustrated in the top panels of Figure 1, The attention weight of \((P_1, P_3)\) is higher than that of \((P_1, P_2)\), despite the more similar shape between \(P_1\) and \(P_2\). This occurs due to the different scales among \(P_1\), \(P_2\) and \(P_3\), which influence the attention weights, thereby failing to reflect true shape similarity. Consequently, the model may learn incorrect similarity relationships, leading to performance degradation. Furthermore, this scale bias is even more pronounced when modeling dependencies between variables, further hindering the effectiveness of VD models.

To balance the scale differences of patches, recent methods have employed Patch Normalization [@SAN], which Z-score normalizes each patch by subtracting the mean and dividing by the standard deviation. However, the removal of the standard deviation inadvertently distorts the original shape of the patch. As a result, it hampers the model’s ability to identify shape similarities across patches or variables. In this paper, we propose a simple yet effective alternative method called patch-mean decoupling (PMD). We subtract the mean of each patch, which recenters each patch to zero mean and explicitly separates the long-range trend component which is encoded in the means of patches from the residual shape information. Unlike Patch Normalization, our method preserves the original amplitude variations and maintains the intrinsic shape structure, ensuring that the model better captures true shape similarities across patches. As shown in Figure 1, through our method attention favors shape-aligned pairs (\(P_1, P_2\)) over shape-unaligned (\(P_1, P_3\)).

PMD thus enables more shape-focused attention across patches and variables, revealing true similarities obscured by scales. For cross-variable shape modeling, existing methods [@ModernTCN; @Crossformer] often compute interactions over the entire historical window. However, cross-variable relationships are often non-stationary and evolve over time, so recent interactions are the most predictive of future dynamics. For example, in financial markets asset correlations often spike sharply during crises. Relying on the entire historical dependencies introduces substantial noise and redundancy, degrading performance. To address this, we introduce proximal variable attention (PVA), which confines self-attention to the most recent patch—the segment most proximal to the prediction horizon. By capturing shape similarities among variables in this temporally relevant window, PVA minimizes noise from historical drifts and risk overfitting.

Complementarily, recentering via PMD inherently attenuates the long-term trend signal, potentially overlooking global dependencies. To restore this without disrupting shape matching between temporal patches, we propose trend restoration attention (TRA), which explicitly injects the decoupled means (long-range trend information) into the value pathway of the attention mechanism. This seamless integration allows the model to jointly encode local shape patterns and global trend yielding more stable forecasts.

Building on above, we propose PMDformer, which combines patch-mean decoupling (PMD) module, Proximal variable attention (PVA), trend reinsertion attention (TRA) and a projection layer for final forecasting. The comparison of predictive accuracy of our PMDformer and other state of the art models refer to Figure [fig:two-images]. Our contributions are:

• We introduce a novel mechanism to decouple trend and residual shape within the attention module via residual mean deduction, enabling more effectively capture shape similarity among temporal patches and varibles.

• We introduce proximal variable attention, which focuses on the most recent patch to capture the most relevant shape similarities, mitigating overfitting.

• We demonstrate the effectiveness of our approach through extensive experiments on a variety of LTSF benchmarks, showing that PMDformer provides more stable and accurate forecasts than current state-of-the-art methods.

2 Related Work↩︎

Deep learning models have demonstrated remarkable performance in long-term time series forecasting. These models can be broadly divided into Transformer-based models [@transformer; @autoformer; @Pyraformer; @FEDformer], MLP-based models [@DLinear; @RLinear; @timemixer; @FDNet], GNN-based models [@CrossGNN; @fouriergnn] and CNN-based models [@micn; @TSLANet; @TimeCNN].

2.0.0.1 Transformer-based time series models.

The success of Transformers [@transformer] in NLP has inspired their adaptation for LTSF to capture long-range dependencies. Early models treat series as token sequences with efficient attention: Informer [@Informer] uses ProbSparse for complexity reduction; Pyraformer [@Pyraformer] employs pyramidal attention; Autoformer [@autoformer] adds decomposition; and FEDformer [@FEDformer] incorporates frequency blocks. Yet, their efficacy is challenged by simple linear models [@DLinear], underscoring needs for better temporal modeling.

2.0.0.2 Patch-based time series models.

Inspired by vision transformers [@VIT], recent works segment time series into overlapping or non-overlapping patches to bolster local semantic capture. Transformer-based examples include PatchTST [@PatchTST], which uses variable-independent shared encoders for temporal patch semantics (SOTA in LTSF), and Pathformer [@Pathformer] with multi-scale patches and adaptive path selection for intra/inter-dependencies. MLP variants like TSMixer [@TSMixer] and PatchMixer [@PatchMixer] model patch relations via MLPs, while foundation models such as Moirai [@Moirai], Timer [@timer], TimesFM [@TimesFM], and LLM-based [@S2LLM; @time-llm]leverage patches for pretraining and cross-modal alignment. Recent TimeBase [@TimeBase] employs orthogonalized patches to reduce redundancy for SOTA efficiency, which further underscores patches’ success in LTSF modeling.

2.0.0.3 Patch-Normalization.

Due to the non-stationary nature of time series, some works [@DishTS; @RevIN] apply normalization to mitigate scale discrepancies and stabilize distributions. Among them, Patch-level normalization works include SAN [@SAN], a model-agnostic framework that adaptively normalizes slices by removing non-stationarity for flexible forecasting, and SIN [@SIN], which selectively learns normalization parameters to maximize local invariance and global variability, enabling interpretable long-term predictions. However, these normalization methods distort intrinsic patch shapes by scaling with standard deviation, hindering true shape similarity capture. In contrast, our PMD overcomes through mean subtraction to preserve amplitudes.

3 Proposed Method↩︎

We consider the task of long-term time series forecasting, where the goal is to predict the future evolution of multiple correlated variables given their historical observations. Formally, let \(\mathbf{X} = \{x_{t} \in \mathbb{R}^{C} \mid t = 1, 2, \ldots, L\}\) denote an input sequence of length \(L\), where \(C\) is the number of variables. Each \(x_t = (x_t^1, x_t^2, \ldots, x_t^C)\) contains the values of all variables at time \(t\). Given \(\mathbf{X}\), the objective is to forecast the subsequent \(T\) time steps \(\hat{\mathbf{Y}} = \{\hat{x}_{t} \in \mathbb{R}^{C} \mid t = L+1, \ldots, L+T\}\).

3.1 The General Structure↩︎

Our proposed PMDformer architecture is a unified framework composed of four synergistic modules designed to explicitly decouple the long-term trend from the shape structure, selectively focus on the most relevant inter-variable dependencies, and ensure the accurate restoration of global dynamics for stable forecasting, as illustrated in Figure 2. (a) Patch-Mean Decoupling (PMD): This module partitions the input time series into non-overlapping patches and explicitly separates each patch into its long-term trend component and its residual shape component. (b) Proximal Variable Attention (PVA): To capture the most relevant cross-variable dependencies, the PVA module focuses its self-attention mechanism only on the \(C\) tokens of the last (proximal) patch, modeling interactions across all variables. (c) Trend Restoration Attention (TRA): This module is designed to model the shape similarities across patches. Crucially, it then restores the long-range trend information into the value pathway, enabling to accurately capture and utilize the overall long-term trend. (d) Projection Layer: This final layer combines the learned temporal representations with the reincorporated trend information through a fully connected projection to produce the final predictions.

3.2 Model Architecture↩︎

3.2.0.1 Patch-Mean Decoupling (PMD) & Embedding.

We first divide the input sequence \(\mathbf{X}=\{x_t\in\mathbb{R}^{C}\}_{t=1}^{L}\) into \(N\) non-overlapping patches of length \(S\), where \(N=\lfloor L/S \rfloor\). For variable \(i\!\in\![C]\) and patch index \(j\!\in\![N]\), the raw patch vector is \[\begin{align} \mathbf{P}_{j}^{i}=\big(x_{(j-1)S+1}^{i},\,x_{(j-1)S+2}^{i},\,\ldots,\,x_{jS}^{i}\big)\in\mathbb{R}^{S}. \end{align}\] We then compute its temporal mean and the corresponding mean-decoupled residual: \[\begin{align} \label{eqa:mu} \mu_{j}^{i}=\tfrac{1}{S}\sum_{k=1}^{S}x_{(j-1)S+k}^{i}, \qquad \mathbf{r}_{j}^{i}=\mathbf{P}_{j}^{i}-\mu_{j}^{i}\,\mathbf{1}_{S}, \end{align}\tag{1}\] where \(\mathbf{1}_{S}\) is the \(S\)-dimensional all-ones vector. Each residual patch is then embedded into a \(d\)-dimensional representation through a shared linear projection. To encode location, we add learned positional embeddings to form the Transformer token: \[\begin{align} \mathbf{P}_{j}^{i}:=\mathbf{r}_{j}^{i}\,\mathbf{W}_E+\mathbf{b}_{E} + \mathbf{z}_{p_j} \end{align}\] where \(\mathbf{W}_E\in\mathbb{R}^{S\times d}\), \(\mathbf{b}_E\in\mathbb{R}^{d}\), and \(\mathbf{z}_{p_j}\in\mathbb{R}^{d}\) denotes the positional embedding of patch \(j\). By removing patch means before embedding, each patch is centered, which alleviates local inconsistencies across patches and variables so that attention mechanism can focus on shape similarities.

3.2.0.2 Proximal Variable Attention (PVA).

Intuitively, accurate time series forecasting hinges on the immediate interactions between variables at the most recent time steps, as these dependencies are most indicative of near-term changes. Therefore, the PVA module is designed to concentrate its attention mechanism on the most proximal (i.e., most recent) tokens to model these critical cross-variable relationships.

Let \(N\) be the index of the last (most recent) patch. We collect the most recent tokens of all \(C\) variables, denoted as \(\mathcal{P}_N=\{\mathbf{P}_{N}^{1},\ldots,\mathbf{P}_{N}^{C}\}\), where each token \(\mathbf{P}_{N}^{i}\in\mathbb{R}^{d}\) is derived from the Patch-Mean Decoupling (PMD) embedding. The PVA then applies Multi-Head Self-Attention (MHSA) exclusively within the set \(\mathcal{P}_N\) to effectively capture the cross-variable shape dependencies that are most relevant for forecasting. Following the attention mechanism, a Feed-Forward Network (FFN) is employed to enhance the non-linear feature representation: \[\begin{align} &\hat{\mathcal{P}}_N = \text{LayerNorm}(\text{MHSA}(\mathcal{P}_N) + \mathcal{P}_N), \\ &\mathcal{P}_N = \text{LayerNorm}(\text{FFN}(\hat{\mathcal{P}}_N) + \hat{\mathcal{P}}_N). \end{align}\] Tokens from the earlier historical patches, specifically those indexed \(\{1,\ldots,N-1\}\), maintain their original representation derived from the PMD module. Following the PVA operation, the refined token set \(\mathcal{P}_N\) is concatenated with these remaining historical tokens along the patch dimension to form the full sequence of shape embeddings, denoted as \(\mathcal{P}\in\mathbb{R}^{C\times N \times d}\). This deliberate strategy of restricting cross-variable attention solely to the most proximal patch offers dual advantages: it enhances model robustness by avoiding spurious long-range couplings from historical noise, and it improves computational efficiency by reducing the complexity from \(O(C^2 N)\) to \(O(C^2)\).

3.2.0.3 Trend Restoration Attention (TRA).

Following the refinement of the most proximal tokens by the PVA module, the TRA module aims to capture temporal shape similarities across all historical patches while preserving long-range trend information. This is achieved by applying a parameter-shared Transformer encoder (MHSA + FFN) along the patch axis for each variable independently.

In this design, the Query(\(\mathbf{Q}\)) and Key(\(\mathbf{K}\)) projections operate solely on the shape embeddings, ensuring that the resulting attention scores \(\mathcal{A}\) emphasize precise inter-patch shape similarity. To counteract the potential loss of global dynamics inherent in shape-focused modeling, we explicitly incorporate the per-patch mean (\(\mu^i\)) into the Value (\(\mathbf{V}\)) pathway. The additive reintegration is inspired by residual connections in ResNet [@ResNet]. Concretely, for the \(i\)-th variable’s patch sequence \(\mathbf{P}^i\in\mathbb{R}^{N\times d}\), the computation is defined as: \[\begin{align} &\mathbf{Q}^i = \mathbf{P}^i \mathbf{W}_Q, \qquad \mathbf{K}^i = \mathbf{P}^i \mathbf{W}_K, \\ &\mathcal{A} = \text{Softmax}\Big(\frac{\mathbf{Q}^i (\mathbf{K}^i)^\top}{\sqrt{d}}\Big), \\ &\mathbf{V}^i = \mathbf{P}^i \mathbf{W}_V + \mu^i, \end{align}\] where \(\mathbf{W}_Q, \mathbf{W}_K, \mathbf{W}_V\) are the projection matrices, and \(\mu^i\) is the per-patch mean (Eq. 1 ), broadcast to match the dimensions of \(\mathbf{P}^i \mathbf{W}_V\). This architectural separation allows the \(\mathbf{Q}/\mathbf{K}\) pathway to model fine-grained local shape dependencies, while the \(\mathbf{V}\) pathway ensures the preservation of the essential global trend dynamics. The resulting trend-integrated tokens are then refined through a Feed-Forward Network (FFN) to enhance the temporal representation learning.

3.2.0.4 Projection Layer.

The temporal tokens produced by the TRA module are rich in shape dependencies but still require the final restoration of the global trend information for stable and accurate multi-step forecasting. This final step is essential to fully recover the original scale and long-term dynamics that were decoupled earlier. To achieve this, before generating the multi-step forecasts, we re-incorporate the per-patch trend means (\(\mu^i\)) into the refined shape embeddings: \[\begin{align} \hat{\mathbf{Y}}^i = (\mathbf{P}^i + \mu^i)\,\mathbf{W}_o + \mathbf{b}_o,\quad \hat{\mathbf{Y}}^i \in \mathbb{R}^{T}. \end{align}\] Here, \(\mathbf{W}_o \in \mathbb{R}^{(N \times d) \times T}\) and \(\mathbf{b}_o \in \mathbb{R}^{T}\) are the weight matrix and bias vector, respectively. The mean \(\mu^i\) is implicitly broadcast to align with the dimensions of \(\mathbf{P}^i\). This final step ensures the model’s predictions are well-calibrated with the long-range trend observed in the input series.

3.3 Theoretical Analysis↩︎

Scale Bias Without Patch-Mean Decoupling (PMD) Consider embedding raw patches \(\tilde{\mathbf{x}} = \mathbf{r} + \mu \mathbf{1}\), where \(\mathbf{r}\) is the residual and \(\mu\) is the patch mean. The attention logit between tokens \((i, j)\) is given by: \[\begin{align} \tilde{z}_{ij} = \mathbf{q}_i^\top \mathbf{k}_j = \tilde{\mathbf{x}}_i^\top \mathbf{M} \tilde{\mathbf{x}}_j = \underbrace{\mu_i \mu_j \mathbf{1}^\top \mathbf{M} \mathbf{1}}_{\text{mean--mean}} + \underbrace{\mu_i \mathbf{1}^\top \mathbf{M} \mathbf{r}_j + \mu_j \mathbf{r}_i^\top \mathbf{M} \mathbf{1}}_{\text{cross}} + \underbrace{\mathbf{r}_i^\top \mathbf{M} \mathbf{r}_j}_{\text{residual similarity}}, \end{align}\] where \(\mathbf{M} := \mathbf{W}_E^\top \mathbf{W}_Q^\top \mathbf{W}_K \mathbf{W}_E\) and \(\mathbf{1}\) is the all-ones vector. The first three terms depend on the means and can affect even dominate the residual similarity, inducing scale bias.

3.3.0.1 Proposition 1: Sufficient Condition for Level-Dominated Logits

Let \(i\) be a fixed query. A sufficient condition for the mean-dependent part of \(\tilde{z}_{ij}\) to dominate the residual similarity uniformly over all keys \(j\) is: \[\begin{align} |\mu_i| |\mu_j| |\mathbf{1}^\top \mathbf{M} \mathbf{1}| \ge \|\mathbf{M}\|_2 \|\mathbf{r}_i\| \|\mathbf{r}_j\| + |\mu_i| \|\mathbf{M} \mathbf{1}\| \|\mathbf{r}_j\| + |\mu_j| \|\mathbf{M} \mathbf{1}\| \|\mathbf{r}_i\|, \end{align}\] where \(\|\cdot\|_2\) represents the spectral norm. This condition guarantees that the mean-dependent terms outweigh the residual term and cross terms, leading to scale-induced bias in attention. This confirms that attention is biased toward scale when the means are large, which motivates the need for patch-mean decoupling in our method.

4 Experiment↩︎

4.1 Experiment Setup↩︎

4.1.0.1 Datasets

We conduct experiments on 8 widely-used and publicly available real-world datasets. These include: ECL⁴, Traffic⁵, Weather⁶, Solar⁷, ETTh1, ETTh2, ETTm1, and ETTm2⁸. Following the experimental protocol established in prior work [@TSLib; @TFB; @LargeTS], we partition the datasets into training, validation, and test sets with the following ratios: 6:2:2 for the four ETT datasets and 7:1:2 for the remaining datasets. The detailed statistics of each dataset are summarized in Table ¿tbl:tab:datasets?.

4.1.0.2 Baselines

We compare PMDformer against 9 baselines, including state-of-the-art (SOTA) long-term forecasting models: TQNet [@TQNet], TimeBase [@TimeBase], SOFTS [@SOFTS], SparseTSF [@SparseTSF], ModernTCN [@ModernTCN], iTransformer [@iTransformer], TimeMixer [@timemixer], and PatchTST [@PatchTST].

4.1.0.3 Setups

Consistent with prior research [@TimeBase], we use an input length \(L\) of 720 and evaluate prediction lengths \(T\) of {96, 192, 336, 720}. Results for TimeBase, SparseTSF, iTransformer, TimeMixer, and PatchTST are derived from the TimeBase study, while other outcomes are from our own experiments. All experiments are conducted using PyTorch [@pytorch] on an NVIDIA A100 80GB GPU. The Adam optimizer [@Adam] is employed, with learning rates chosen from {2e-4, 5e-4, 1e-3, 1e-2}. The number of patches \(N\) is adjusted based on the requirements of each dataset.

4.2 Main Results↩︎

Table ¿tbl:tab:main95results95paper? summarizes the quantitative results for long-term time series forecasting across multiple prediction horizons and datasets. As shown, our proposed PMDformer achieves the lowest Mean Squared Error (MSE) and Mean Absolute Error (MAE) on 7 out of 8 real-world datasets, outperforming all baselines in the majority of cases. This success is directly tied to PMDformer’s ability to overcome fundamental limitations in existing architectures.

Specifically, compared to the patch-based model TimeBase, PMDformer yields an average MSE reduction of 5.68% and MAE reduction of 6.61%. This improvement stems from our method’s capacity to indentify meaningful shape similarities across patches, a capability that TimeBase’s orthogonal patch selection inherently sacrifices to reduce redundancy. Moreover, against TQNet, PMDformer achieves an average MSE reduction of 8.62% and MAE reduction of 9.96%. TQNet’s fixed periodic queries constrain its ability to handle diverse cycles, whereas PMDformer’s adaptive proximal variable attention offers greater flexibility in modeling variables’ shape similarities. Compared to the Transformer-based iTransformer, PMDformer delivers an average MSE reduction of 11.44% and MAE reduction of 12.38%. iTransformer captures dependencies among variable tokens embedded from the entire historical sequence, which can lead to overfitting on early, weakly relevant variable relationships that degrade future predictions. In contrast, our PVA module succeed to avoid this by focusing on the shape similarities of variables within the most nearest patch.

max width=,center

max width=,center

4.2.0.1 PMD Module Analysis.

We assessed the effectiveness of the PMD module through extensive ablations conducted on five non-stationary benchmarks: ETTh2, ETTm1, Weather, Traffic, and Solar [@ab_PMD_1; @ab_PMD_2]. Using a fixed input length of 720, we tested the model’s performance across various prediction horizons (96, 192, 336, and 720) against several patch-wise normalization variants: (i) mean–variance standardization (‘w/ stdev’), (ii) utilizing the Scale-Adaptive Normalization (SAN) [@SAN] method, and (iii) removing the PMD module entirely. As presented in Table ¿tbl:tab:ablation95pmd?, the PMDformer consistently achieves superior accuracy across all datasets. We attribute this advantage to the PMD module’s per-patch centering mechanism, which effectively preserves crucial intra-patch shape information. This preservation allows the Transformer architecture to specifically concentrate its attention on modeling shape similarity. Furthermore, by explicitly injecting the patch mean as a separated trend component into the Transformer pathway, PMDformer is uniquely positioned to accurately capture and model long-term trends. In stark contrast, SAN explicitly decouples the scale and residual components for independent prediction. Since global scale estimation is inherently unstable in highly non-stationary series, this rigid decoupling undermines the essential joint modeling of scale–shape interactions, consequently leading to overfitting and weaker generalization capabilities.

4.2.0.2 TRA & PVA Analysis.

To assess the effectiveness of the TRA and PVA modules, we conducted ablation studies on the ETTh2, ETTm1, Traffic, and Solar datasets. For the TRA module, we tested two alternatives: replacing it with standard self-attention or removing it entirely. For the PVA module, we either modified it to compute variable-wise shape similarity across all patches or removed the module completely. Additionally, we investigated a structural variant that swaps the sequential order of the two modules. The experimental outcomes are summarized in Table ¿tbl:tab:ablation95TVA95TRA?.

The results unequivocally show that \(\text{PMDformer}\) consistently outperforms all ablated variants across every dataset and configuration. When TRA is replaced with standard self-attention, performance degrades significantly because the crucial long-term trend information is neglected. Similarly, when PVA is forced to compute variable-wise shape similarity across all historical patches, performance decreases. This confirms our hypothesis that early variable relationships are often only weakly or spuriously correlated with the predictive sequences, justifying PVA’s proximal focus. Furthermore, removing both TRA and PVA results in the largest performance drop observed, emphatically highlighting the dual importance of TRA in modeling temporal patch shapes and long-range trends, and PVA in capturing relevant variable-wise shape similarity. Finally, swapping the original order of TRA and PVA also causes notable performance degradation. When TRA is applied first, it compresses patch information too early, making it harder for the subsequent variable modeling to identify meaningful cross-variable dependencies.

4.3 Parameter Sensitivity Analysis↩︎

4.3.0.1 Patch Count for Cross-Variable Modeling.

We evaluate the impact of capturing variable patterns within different numbers of patches, where \(k \in \{1,2,3,5,7,10\}\). For each setting, the \(k\) nearest patches to future sequences are selected to capture the shape similarity of variable, thereby further validating the effectiveness of PVA. Experiments are conducted on the ETTh1 and Solar datasets. As shown in Figure 8 (a), the mean squared error (MSE) exhibits an overall upward trend as \(k\) gradually increases on the ETTh1 dataset. On the Solar dataset, this increase is more pronounced when predicting 192, 336, or 720 steps ahead, because future sequences are more weakly correlated with early variable relationships. Moreover, the MSE curves show some fluctuations, indicating that different values of \(k\) may lead to more significant differences in prediction performance. In contrast, across all four prediction horizons, using \(k=1\) yields more stable performance compared with larger \(k\). This is because the nearest patch is typically more closely aligned with the target sequence to be predicted, making it more beneficial for accurate modeling.

4.3.0.2 Patch Size.

Different patch sizes lead to varying degrees of distinction among patches. To investigate this, we evaluate multiple patch sizes \(\{8, 16, 24, 48, 72, 120\}\) on the ETTh2 and ECL datasets. As shown in Figure 8 (b), both overly small and overly large patch sizes fail to deliver optimal performance. This is because excessively small patches provide insufficient shape information to distinguish similarity, making it difficult for the attention mechanism to capture underlying temporal dependencies or genuine variable correlations. Conversely, overly large patches reduce the number of tokens, thereby limiting the model’s ability to capture long-range dependencies. Based on these observations, we find that moderate patch sizes, particularly within \(\{24, 48, 72\}\), achieve a better trade-off and yield more robust performance.

5 Conclusion↩︎

In this paper, we tackle challenges in long-term time series forecasting by emphasizing true shape similarities hidden by scale variations in non-stationary data. Our patch-mean decoupling (PMD) separates trends from residual shapes while preserving amplitudes, enabling shape-focused attention across patches and variables. Integrated with proximal variable attention (PVA) for recent inter-variable dependencies and trend restoration attention (TRA) for global trend reintegration. Experiments on LTSF benchmarks show PMDformer surpasses state-of-the-art baselines in accuracy and stability, underscoring the value of shape-centric Transformer designs. Future directions include scaling to higher-dimensional multivariate data and multimodal integrations for applications in energy, finance, and traffic.

6 Acknowledgements↩︎

This work was supported by the Major Science and Technology Special Project of the Sichuan Provincial Department of Science and Technology (Grant No. 2024ZDZX0002), the Sichuan Provincial Innovation Group Project (Grant No. 2024NSFTD0054), Fundamental Research Funds for the Central Universities (JBK202511081), the Blockchain Research Center of China, the Natural Science Foundation of China (Grant No. 62502397), the National Natural Science Foundation of China (Grant No. 72471197), and the Sichuan Provincial Philosophy and Social Science Fund (Grant No. SCJJ25ND091).

7 Appendix↩︎

7.1 Efficiency Analysis↩︎

To evaluate the efficiency of our model in handling complex tasks, we conduct experiments under two settings: varying the number of variables and varying the input length. In the first setting, we fix the input length at 720 and change the number of variables; in the second setting, we fix the number of variables at 100 and test PMDformer with different input lengths. The batch size is set to 1 in all experiments. The results are shown in Figure 9. Under both settings, compared with recent popular models such as PatchTST [@PatchTST], iTransformer [@iTransformer], and ModernTCN [@ModernTCN], PMDformer requires significantly less GPU memory, thereby reducing the overall computational cost.

7.2 Comparison on Synthetic Data↩︎

To further validate the effectiveness of our PMD module, we conduct an experiment on a synthetic dataset. This dataset consists of patches alternating between two different shapes: a sharp pulse wave with large amplitude and a smooth sine wave with small amplitude. To simulate non-stationary time series, the patches exhibit varying scales and are augmented with moderate noise. We compare a standard patch-based Transformer (w/o PMD) against our model incorporating the patch-mean decoupling module (w/ PMD). As illustrated in Figure 10, the ‘w/o PMD’ model struggles to recognize true shape similarities due to scale differences between patches, leading to predictions that resemble mostly smooth curves with inadequate trend fitting. In contrast, our ‘w/ PMD’ model, by removing scale factors, enables attention to focus more effectively on intrinsic shapes, resulting in predictions that better capture both the underlying patterns and long-range trends.

8 ON THE USE OF LARGE LANGUAGE MODELS↩︎

The authors used large language models (LLMs) exclusively for language polishing and minor rephrasing during the final writing stage. All scientific content, ideas, and initial drafts were created entirely by the authors without any text improved by LLMs was carefully checked and edited by the authors. LLMs played no role in developing research questions, designing experiments, analyzing results, or any other aspect of the research itself.

---

1. Equal contribution.↩︎

2. Corresponding author.↩︎

3. Project leader.↩︎

4. ↩︎

5. ↩︎

6. ↩︎

7. ↩︎

8. ↩︎