1 Introduction↩︎

The Transformer architecture [1] has become a cornerstone of modern deep learning, driving breakthroughs in natural language processing [2]–[5], computer vision [6]–[8], and large-scale foundation models [9]. Motivated by this success, recent works have applied Transformers to multivariate time-series forecasting, leveraging their ability to model pairwise dependencies and extract multi-level sequence representations [10], [11]. However, their effectiveness in this domain remains contested. Notably, simple linear models rooted in classical statistics [12] have been shown to outperform Transformers in both accuracy and efficiency [13], [14]. At the same time, emerging architectures that explicitly model multivariate correlations [15], [16] underscore the limitations of vanilla self-attention for complex time-series dynamics.

We identify three fundamental shortcomings of existing Transformer-based approaches for multivariate forecasting: (1) Permutation-invariant self-attention fails to capture temporal order, leading to weak modeling of sequential dynamics. (2) Uniform attention across tokens not only overlooks the varying significance of climate variables across spatiotemporal scales, but also leads to computational inefficiencies when scaling to high-dimensional meteorological data. (3) The architecture lacks an explicit mechanism to model hierarchical spatiotemporal correlations, which are crucial for long-horizon forecasting.

Unlike FocalNet [8], which was designed for spatial representation learning in vision tasks, FATE introduces key innovations tailored for multivariate time-series forecasting:

• Tensorized Attention Design: FATE preserves the full 3D tensor structure (\(X \in \mathbb{R}^{T \times S \times P}\)), maintaining temporal and variable axes explicitly. This enables more effective modeling of long-range dependencies through grouped attention across both time and features.

• Focal Grouping for Temporal Blocks: Instead of spatial grids, FATE dynamically defines temporal focal groups that adapt to prediction horizons, allowing the model to capture hierarchical temporal dependencies unique to time-series data.

• Cross-axis Modulation: Focal modulation is extended beyond temporal steps to the variable dimension, enabling rich cross-feature interactions that are absent in FocalNet.

In this way, FATE is not a simple adaptation of FocalNet, but a principled redesign that leverages the structural properties and forecasting demands of multivariate time-series data.

Long-term variations in temperature, precipitation, wind, and other environmental factors define climate change [17]. These shifts have profound global impacts, threatening sustainability in domains such as food security, public health, and energy systems. For instance, a projected increase of up to \(2^\circ\)C in global mean temperature this century could severely reduce crop yields. Unlike short-term fluctuations, climate change evolves over decades, driven primarily by greenhouse gas emissions, deforestation, and limited adoption of renewable energy [18]. Accurate long-horizon forecasting of such multivariate processes is therefore critical. It enables policymakers and practitioners to assess risks, monitor climate drivers, and design mitigation strategies [19]. However, the multidimensional and highly correlated nature of climate data poses significant challenges for existing forecasting models.

To address these challenges, we propose FATE, a novel Transformer that (1) introduces tensorized focal modulation for explicit spatiotemporal correlation modeling, (2) employs dual modulation scores to enhance interpretability, and (3) adaptively emphasizes relevant tokens via selective attention. We evaluate FATE across seven diverse real-world datasets and demonstrate that it consistently outperforms state-of-the-art methods, particularly on long-horizon and high-dimensional climate datasets. Extensive ablation studies further confirm that FATE generalizes effectively across broader multivariate forecasting tasks.

Contributions. The main contributions of this work are threefold:

• We introduce FATE, a Transformer architecture with a novel focal-modulation mechanism that preserves 3D tensor structure (\(T \times S \times P\)) for multivariate time-series forecasting.

• We design dual modulation scores that improve both predictive performance and interpretability by identifying critical temporal and variable dependencies.

• We achieve new state-of-the-art results on seven benchmark datasets, including accuracy gains of \(13.3\%\), \(9.1\%\), and \(10.1\%\) on ETTm2 [20], Weather5k [21], and LargeST [22], respectively, with strong improvements across all other datasets.

2 Related Work↩︎

Transformers for Time Series Forecasting. Transformer architectures [1] have achieved remarkable success across NLP [2], [3], [23], computer vision [24]–[26], and speech [27], [28] due to their scalability and effective sequence modeling. Vision Transformers (ViTs) divide images into patches to preserve local semantic information [24], [29], [30], while NLP models like BERT [4] leverage subword tokenization for contextual dependencies. Inspired by these successes, Transformer variants have been widely adapted for time-series forecasting [11], [31]. Early models, such as LogTrans [32] and Informer informer?, addressed computational inefficiencies via sparse attention. Autoformer [10] introduced decomposition-based inductive biases, FEDformer [33] employed Fourier-enhanced blocks for seasonal modeling, Pyraformer pyraformer? added pyramidal attention for multi-scale dependencies, and Triformer [34] proposed pseudo-timestamp-based patch attention. Despite these advances, many Transformer forecasters still rely on point-wise or handcrafted attention, limiting their ability to capture semantic relationships across patches or dimensions [35]–[37]. For example, Autoformer’s fixed auto-correlation modules may fail to generalize, and Triformer does not treat patches as first-class units nor model internal semantics. TimeMixer++ [38] advances multi-scale, multi-resolution forecasting by converting time series into 2D time images (via Multi-Resolution Time Imaging, MRTI) and separating seasonal/trend components in latent space using dual-axis attention, followed by hierarchical Multi-Scale Mixing (MCM) and Multi-Resolution Mixing (MRM). This allows parallel modeling of concurrent temporal contexts (daily, weekly, seasonal), improving forecasting, classification, and anomaly detection. TimeTensor [39] generalizes linear attention to 3D tensor inputs via Kronecker decomposition, improving efficiency while retaining the standard attention paradigm. In contrast, FATE introduces tensorized focal modulation, explicitly preserving 3D spatiotemporal structure, enabling hierarchical and localized context aggregation, and jointly modeling long- and short-range dependencies. This represents a novel architectural strategy distinct from previous tensorized attention mechanisms.

Self-supervised and Representation Learning. Transformer adaptations for time series can be categorized into four directions [40]: (i) attention-level modifications for efficiency, (ii) adaptations for stationarity and signal processing, (iii) architectural changes capturing cross-variate and temporal dependencies, and (iv) novel tensor-based designs. Most methods focus on the first three, while few explore fundamental tensor-based redesigns. Self-supervised learning (SSL) has also gained traction for time-series representation learning. Methods such as TNC [41], TS2Vec [42], and BTSF [43] learn rich representations without supervision, whereas Transformer-based SSL models like TST [44] and TS-TCC [45] remain underexplored for capturing complex temporal and cross-variate dependencies. FATE’s tensorized focal modulation inherently supports richer hierarchical representations, bridging this gap by jointly modeling time, feature, and spatial dimensions. Focal Modulated Tensorized Encoder introduces a novel tensorized focal modulation mechanism tailored for multivariate time-series forecasting. It preserves the input’s 3D tensor structure (\(T \times S \times P\)), enables hierarchical spatiotemporal correlation modeling, and applies tensorized attention design, temporal focal grouping, and cross-axis modulation. Unlike prior work, FATE balances efficiency with semantic richness and provides a principled framework for long- and short-range dependency modeling in high-dimensional time series.

3 Proposed Methodology↩︎

In this section, we present FATE, a Focal Modulated Tensorized Encoder Transformer designed for multivariate time-series forecasting. The architecture preserves the full 3D structure of the input tensor to jointly model temporal, spatial (station-wise), and feature dimensions. Central to FATE are tensorized focal modulation mechanisms that efficiently capture hierarchical temporal patterns, cross-station interactions, and feature dependencies, while providing interpretable modulation scores that highlight the contribution of each station and attention head. The following subsections detail the encoder design, the tensorial focal modulation computations, and the aggregation strategy for interpretable predictions.

3.1 Multi-Dimensional Tensored FocalNet Encoder↩︎

We extend the FocalNet Transformer [8] to propose the Tensorized Focal Encoder Transformer, specifically designed to capture complex patterns in multi-dimensional time-series data. Our model operates on climate parameters organized as a 3D tensor \(X \in \mathbb{R}^{T \times S \times P}\), where \(T\) denotes the temporal dimension, \(S\) indexes different stations, and \(P\) represents diverse climate parameters (e.g., temperature, humidity, wind speed). The full 3D structure preserves variable–time step relationships and supports parallel yet separate attention across temporal and feature dimensions.

The architecture is encoder-only, as illustrated in Figure 1, and comprises: (i) a positional encoding layer, (ii) a tensorial focal modulation encoder layer, and (iii) a linearly activated fully-connected layer. Each encoder layer integrates tensorial modulation (Sections 3.2 and 3.3) followed by a residual connection and normalization. A densely connected FFN, consisting of two linear transformations with ReLU activation, follows the modulation layer, and is again succeeded by residual connection and normalization, consistent with [8].

3.2 Tensorial Focal Modulation↩︎

To encode temporal hierarchies, we apply a constant positional encoding [8] along the time axis \(T\) and parameter axis \(P\):

\[\text{PE}(\text{pos}, 2i) = \sin\left(\frac{\text{pos}}{10000 \cdot 2^i / P}\right), \label{eq1}\tag{1}\]

where \(\text{pos}\) indexes time and \(i\) indexes parameters; the station axis \(S\) transmits the encoded values.

Focal modulation replaces pairwise attention with hierarchical context aggregation [8], offering three key benefits: (i) improved computational efficiency, (ii) preservation of locality biases, and (iii) non-quadratic long-range dependency modeling. For multivariate time series, FATE leverages this through: (1) nested focal windows that hierarchically aggregate temporal information, and (2) dynamic contextual gating that adapts to input distributions, outperforming fixed receptive fields or conventional attention kernels.

We formalize tensor slices as follows: for a tensor \(N \in \mathbb{R}^{X \times Y \times Z}\), \((N_{y,z})_x \in \mathbb{R}^{Y \times Z}\) denotes the \(x\)-slice, and \((N_z)_{x,y} \in \mathbb{R}^{Z}\) denotes the \(x, y\)-slice. Lowercase letters indicate slice sizes.

Tensorial focal modulation operates on \(X \in \mathbb{R}^{T \times S \times P}\). We first compute 3D Query (\(Q\)), Key (\(K\)), and Value (\(V\)) tensors, \(Q,K,V \in \mathbb{R}^{T \times S \times H}\), via element-wise multiplication with learnable weight tensors \(W^Q,W^K,W^V \in \mathbb{R}^{S \times F \times H}\):

\[\begin{align} (Q_h)_{t,s} &= (X_p)_{t,s} \times (W^Q)_{p,h,s},\\ (K_h)_{t,s} &= (X_p)_{t,s} \times (W^K)_{p,h,s},\\ (V_h)_{t,s} &= (X_p)_{t,s} \times (W^V)_{p,h,s}, \quad \forall t = 1..T, s = 1..S. \end{align} \label{eq2}\tag{2}\]

Next, we compute the multiplicative interaction across time steps:

\[(\widetilde{R}_{s,s^l})_{t,t^l} = (Q_{s,h})_t \times ((K_{s^l,h})_{t^l})^T, \quad R = \frac{1}{\sqrt{H}} \sum_{s^l=1}^{S} (\widetilde{R}_{t,t^l,s})_{s^l}, \label{eq3}\tag{3}\]

followed by a softmax across the station dimension to obtain attention weights \(\widetilde{A} \in \mathbb{R}^{T \times T^l \times S}\):

\[(\widetilde{A}_s)_{t,t^l} = \text{Softmax}\left((R_{t,t^l,s})_s\right), \quad \forall t, t^l = 1..T. \label{eq4}\tag{4}\]

Finally, the output \(Z \in \mathbb{R}^{T \times C \times D}\) is computed by broadcasting \((\widetilde{A}_s)_{t,t'}\) to match the shape of \((V_{s,d})_{t'}\) and summing over the temporal dimension:

\[(Z_{s,d})_{t} = \sum_{t'=1}^{T} \text{broadcast}((\widetilde{A}_s)_{t,t'}) \circ (V_{s,d})_{t'}, \quad \forall t = 1..T. \label{eq5}\tag{5}\]

3.3 Focal Modulation Aggregation↩︎

Modulation weights have been widely used for feature selection and interpretability [46]. In FATE, the focal modulation tensors \(\widetilde{A}\) (Eq. 4 ) serve to provide interpretable insights into model predictions.

To quantify the relationship between attention heads and cities (stations), we compute head-wise focal modulation scores:

\[\label{eq:s95hc} N\widetilde{A}_{s}^{h} = \sum_{t=1}^{T} \sum_{t'=1}^{T'} A_{t,t',c}^h, \quad \forall h = 1..H, \; c = 1..C.\tag{6}\]

We then aggregate across all heads to obtain city-wise modulation scores, reflecting the overall contribution of each city to the prediction:

\[\label{eq:s95c} N\widetilde{A}_{s} = \sum_{h=1}^{H} N\widetilde{A}_{s}^{h}, \quad \forall c = 1..C.\tag{7}\]

This aggregation completes the tensorial focal modulation process, explicitly linking attention heads to cities and highlighting the importance of each city in driving the model’s forecasts.

4 Experiments↩︎

To rigorously evaluate FATE, we conduct extensive experiments on seven diverse real-world datasets spanning environmental and infrastructural domains, comparing against 17 state-of-the-art baselines including Transformer-, RNN/CNN-, Linear-, and spatial-temporal models. We analyze predictive performance across short- and long-horizon forecasts using standard metrics (MAE, MSE), benchmark computational and memory efficiency, and provide interpretability through focal modulation visualization. These studies demonstrate FATE’s superior accuracy, robustness, and capacity to model multi-scale temporal and spatiotemporal dependencies.

4.1 Datasets↩︎

We evaluate FATE on seven diverse real-world multivariate time-series datasets, encompassing both environmental (Weather5k, USA-Canada, Europe) and infrastructural (ETTh1, ETTm2, Traffic, LargeST) domains.

ETTh1 [20] and ETTm2 [20] are electricity transformer datasets at hourly and minute resolutions, respectively, capturing seasonal and trend-driven consumption patterns. Traffic [47] consists of road occupancy rates from multiple sensors, serving as a standard benchmark for traffic flow prediction. Weather5k [21] is a large-scale dataset with 10 years of hourly measurements from 5,672 weather stations worldwide, including temperature, humidity, wind speed, and other climate parameters. USA-Canada [48] contains hourly meteorological data from 30 cities (Oct 2012–Nov 2017), enriched with spatial coordinates and temporal features such as hour and day-of-year. The Europe dataset [49] spans 18 European cities (May 2005–Apr 2020), with normalized temporal and meteorological features; the test split covers 2017–2020, and the training/validation span 2005–2017. Finally, LargeST [22] provides traffic data from 8,600 sensors in California over 5 years, including rich sensor metadata for enhanced interpretability.

Across all datasets, FATE consistently outperforms baselines—including Transformer [8], [50], 3D-CNN [51], LSTM [52], and ConvLSTM [53]—achieving the lowest Mean Absolute Error (MAE) and Mean Squared Error (MSE), particularly on long-horizon and high-dimensional climate datasets.

4.2 Additional Implementation Details↩︎

Computational and Memory Requirements. We use a fixed 30-day input window; for climate datasets, we consider 7 meteorological features, while feature selection for other datasets follows the original data schema. Experiments were conducted on an NVIDIA A100 GPU with 40GB VRAM. Optimizers were selected per architecture following prior best practices.

We analyze FATE’s computational complexity and provide empirical runtime benchmarks against Transformer and CNN-based baselines. While tensorized focal modulation introduces moderate overhead compared to standard Transformers, the performance gains in long-horizon forecasting justify this cost. Preserving the 3D tensor increases memory complexity due to grouped modulation, but efficient projections keep runtime and GPU usage comparable to baseline Transformers.

r0.5

6pt max width=

Hyperparameters. Table [tab:hyperparams] details all training hyperparameters. Multi-head attention is used in both FATE and Transformer models, with FATE employing four focal levels and eight attention heads to capture hierarchical temporal dependencies. 3D-CNN [51] and ConvLSTM [53] models use convolutional layers with kernel sizes tuned for spatiotemporal patterns. LSTM [52] and ConvLSTM models employ recurrent units with hidden dimensions optimized for sequential modeling. Scheduled learning rate decay is applied in FATE and Transformer models, while 3D-CNN, LSTM, and ConvLSTM use fixed rates. Batch sizes are scaled for memory efficiency and stable training.

4.3 Forecasting Results↩︎

r0.5

max width=

We evaluate FATE across diverse real-world datasets and benchmark it against 17 state-of-the-art models spanning four categories: (1) Transformer-based: iTransformer [54], Autoformer [55], etc.; (2) RNN/CNN-based: LSTM [52], ConvLSTM [53], 3D-CNN [51]; (3) Linear-based: DLinear [56], TiDE [57]; (4) Spatial-temporal (LargeST dataset): DGCRN [58], D2STGNN [59].

Continental-scale Forecasting. On USA-Canada and Europe datasets, we evaluate 4–16 hour forecasts using MAE and MSE(Table [tab4]). FATE consistently outperforms all baselines, including robust Transformers and linear models. For example, in Vancouver, FATE reduces MAE and MSE by up to 15.9% and 24.9%, respectively, over the best baseline. The Europe dataset exhibits similar trends, highlighting FATE’s robustness and ability to model long-horizon temporal dynamics effectively.

Large-Scale Spatiotemporal Forecasting. On the LargeST dataset (Table [tab:lagest]), FATE achieves the lowest MAE and MSE(0.160 and 0.255), surpassing D2STGNN by 10.1% and 13.6%, respectively. These results demonstrate FATE’s capacity to capture intricate spatiotemporal dependencies in large-scale traffic data, making it highly suitable for real-world forecasting applications.

max width=

Benchmark Dataset Evaluation. Across four standard multivariate time series datasets (ETTh1, ETTm2, Traffic, Weather5k; Table [tab:combined954]), FATE consistently achieves state-of-the-art or competitive performance. Notably: - ETTh1: 4.3% lower MAE, capturing fine-grained temporal patterns. - Traffic: compared to PatchTST, MAE is slightly higher, but MSE decreases by 3%. - Weather5k: 9.1% MAE and 12.3% MSE improvements over CI-TSMixer, demonstrating robustness to high-dimensional noise. - ETTm2: 13.3% MAE and 7.9% MSE improvements, confirming generalizability across diverse datasets. These results collectively validate FATE’s ability to model multi-scale temporal and spatial dependencies, yielding accurate and stable forecasts across both regional and large-scale datasets.

6pt max width=

5 Outlook and Future Directions↩︎

The strong empirical performance of FATE opens multiple avenues for advancing spatio-temporal forecasting.

Scaling to global and ultra-long horizons. While FATE performs strongly on regional datasets (Table [tab4]), scaling to continental or global domains requires optimized training and inference. Future work may explore hierarchical or distributed focal-modulation architectures to retain interpretability while handling millions of spatial points over decades of data. Richer variables and cross-domain fusion. Current experiments emphasize temperature and standard meteorological features (Table [tab:combined954]). Adding variables such as precipitation, aerosols, oceanic indices, or soil moisture and fusing satellite imagery, reanalysis products, and socio-economic data could enhance predictive power and policy relevance. Self-supervised pretraining. Unlabeled climate data motivates self-supervised learning tailored to the focal-tensor setup. Objectives like contrastive or masked prediction can enrich spatio-temporal representations, improve robustness, and reduce dependence on labeled data. Physics-informed inductive biases. Incorporating physical constraints e.g., conservation laws or dynamical couplings into focal-modulation blocks may improve physical plausibility and reduce extrapolation error (Appendix §[app:corr]). Hybrid integration with NWP ensembles is a promising future direction. Efficiency and real-time inference. Though efficient, FATE remains costlier than linear baselines. Techniques such as tensor compression, sparse kernels, or adaptive focal levels could enable lightweight, real-time variants for edge or on-device use. Decision-support and societal impact. Translating forecasts into actionable insights for agriculture, energy, and disaster response remains a key challenge. Interpretable modulation maps (Figure [fig:NewYork]) and tailored visualizations can foster trust and support decision-making.

Summary. The tensorized focal-modulation design of FATE offers a scalable, extensible foundation for climate forecasting. Future extensions across scale, modality, physics, and application position it as a comprehensive tool for sustainable development.

6 Conclusion↩︎

In this study, we introduced the Focal-Modulated Tensorized Encoder (FATE), a framework designed to capture complex spatiotemporal dependencies in climate data. By leveraging tensorized focal modulation, FATE effectively models multi-scale interactions across time, space, and climate parameters. We evaluated FATE on seven diverse real-world multivariate time series datasets, consistently achieving state-of-the-art performance. Additionally, we proposed head-wise and city-wise modulation scores to enhance interpretability and conducted ablation studies to quantify their impact. This work provides a foundation for informed climate policy decisions and broader applications that exploit 3D tensor-structured data.

Limitations↩︎

Our current evaluation focuses on temperature and related climate variables within mid-scale regional datasets. Extending FATE to additional meteorological variables and global-scale grids is a direction for future work. While FATE introduces modest computational overhead (trainable on a single A100 GPU), it remains practical for deployment and can be further optimized for edge or real-time applications. These limitations are operational rather than conceptual.

References↩︎