1 Introduction↩︎

The dominant approach to time series classification (TSC) decomposes the problem into two stages: first, a feature encoder maps each time series to a fixed-dimensional representation; second, a task-specific classifier is trained on these representations to produce predictions. Despite the surface-level differences of existing methods, this two-stage design is universal. Classical methods such as MiniRocket [1] and Hydra [2], per-dataset deep learning methods such as TS2Vec [3], and recent foundation models such as Chronos-2 [4], TiRex [5], and MantisV2 [6] all instantiate it.
We argue that this is not merely an implementation convention, but a structural bottleneck. The encoder is optimized independently of the classification objective, and a new classifier must be fitted for every new dataset. Most critically, at inference time, the encoder cannot directly access the labeled training samples and therefore cannot produce representations tailored to the dataset’s class structure.

In-context learning (ICL) eliminates all three of these downsides simultaneously. Rather than fitting per-dataset weights, an ICL model observes the labeled training set at inference and directly outputs a predicted class distribution in a single forward pass. No weight updates are required. This paradigm has proven transformative for tabular classification: TabPFN [7]–[9] and TabICL [10], [11] achieve state-of-the-art performance by meta-training over synthetic labeled tasks drawn from a structured prior, with no per-dataset training at test time.

Extending this to TSC, however, requires solving a non-trivial data generation problem. The tabular ICL literature has largely resolved synthetic pre-training through structured Bayesian priors over causal graphs. Existing synthetic generation approaches for time series either produce unlabeled corpora [12] or assign class membership arbitrarily [13]. Neither approach provides the structured, label-meaningful datasets that ICL meta-training requires to learn a classification prior. To address this problem, we introduce a novel synthetic data generation pipeline.

We introduce TimEE, a 4.5M-parameter model for end-to-end TSC via ICL. TimEE is meta-trained exclusively on synthetic, labeled time series classification tasks and requires no per-dataset training at test time. Despite seeing no real time series during pre-training, TimEE achieves the lowest mean rank on ROC AUC across all 128 UCR benchmark datasets, outperforming all compared methods, including both foundation models and supervised baselines. TimEE lies on the Pareto front of inference speed versus predictive performance.

Our contributions are:

2 Background↩︎

We introduce the time series classification problem and review key concepts and related work.

2.1 Time Series Classification↩︎

Consider a time series classification problem with labeled training data \(\mathcal{D}_{train} = {(\mathbf{x}_i, y_i)}_{i=1}^n\), where \(\mathbf{x}_i \in \mathbb{R}^{T \times d}\) is a \(d\)-dimensional time series of length \(T\) and \(y_i \in \mathcal{Y}\) is the corresponding class label. The goal is to estimate the predictive distribution \(p(\cdot \mid \mathbf{x}_{test}, \mathcal{D}_{train})\) for a held-out test sample \(\mathbf{x}_{test}\).

2.2 The Two-Stage Paradigm↩︎

The dominant approach to TSC decomposes the problem into two stages. First, a feature encoder \(F_\theta: \mathbb{R}^{T \times d} \rightarrow \mathbb{R}^m\) maps each time series to a fixed-dimensional representation. Second, a task-specific classifier \(G_\phi\) is trained on these representations to produce predictions \(\hat{y} = G_\phi(F_\theta(x))\). We describe three general classes of methods below. A detailed comparison is provided in Appendix Table 1.

Classical and Time Series Features Engineering Methods define \(F_\theta(x)\) without gradient-based learning, either through hand-crafted time series features or random convolutional projections. DTW-1NN uses dynamic time warping distance, while Catch22 [14] extracts a curated set of 22 time series features. Rocket [15] and MiniRocket [1] transforms each series using a large bank of random convolutional kernels, aggregating features via global pooling. Hydra [2] extends this by organizes kernels into competing groups to bridge random projections and dictionary-based pattern counting. Despite requiring no representation learning, these methods remain surprisingly competitive on standard benchmarks [16].

Per-Dataset Deep Learning Methods learns \(F_{\theta}\) on each target dataset. InceptionTime [17] trains a convolutional encoder jointly with \(G_\phi\). TS2Vec [3] instead learns representations via hierarchical contrastive objectives over time and instance dimensions on the unlabeled target dataset, before fitting a downstream classifier head. These methods can achieve strong per-dataset performance but require training from scratch on every new dataset with no cross-task transfer.

Time Series Foundation Models amortize representation learning through large-scale pre-training. TSC-focused models, namely NuTime [18], MOMENT [19], Mantis [20], and MantisV2 [6], pre-train encoders via contrastive or reconstruction objectives on large corpora of real and/or synthetic time series data, producing general-purpose embeddings that can be passed to lightweight downstream classifiers. As a complementary line of work, [21] studied transferring forecasting foundation models (notably Chronos-2 [4] and TiRex [5]) to classification using the extracted representations. In practice, however, competitive performance across all these models requires non-trivial layer selection: intermediate representations systematically outperform final-layer embeddings [21].

Shared Structural Limitation Despite their differences, all of the above share a fundamental constraint: \(F_\theta\) is optimized independently of the target classification task, and a new \(G_\phi\) must be fitted per dataset. This decoupling has also been identified as a key source of sub-optimality in [22].² Furthermore, \(F_\theta\) encodes each series in isolation, without any awareness of the other labeled samples in the dataset, and therefore cannot exploit discriminative structure across classes during encoding. Most critically, at inference time, \(F_\theta\) does not exploit the labeled training samples, and therefore cannot produce representations tailored to the class structure of the dataset at hand. TimEE address these three constraints simultaneously via end-to-end in-context learning.

2.3 In-Context Learning for Supervised Prediction↩︎

In-context learning (ICL) offers an alternative to task-specific classifiers: rather than updating model weights, the model observes \(\mathcal{D}_{train}\) at inference and approximates the conditional predictive distribution \(p(\cdot \mid x_{test}, \mathcal{D}_{train})\) in a single forward pass.

This idea was first introduced in Prior-data Fitted Networks (PFN) [23]. TabPFN [7] extended this paradigm for tabular classification by defining a synthetic Bayesian prior over tasks, while TabPFNv2 [8] scaled this substantially by introducing advancement in the synthetic priors and architecture. Parallel works that adopt the same paradigm include TabICL [11] and TabICLv2 [10] which introduce different architectures and synthetic priors.

For TSC, [24] introduce ICL by pre-training an ICL head atop a fixed feature encoder. While this yields strong results, it remains structurally two-stage: the encoder is frozen and cannot be shaped by the classification objective. TiCT [13] addresses the problem more directly by pre-training an end-to-end model on binary tasks, using threshold-based mixing of KernelSynth examples as synthetic prior. TimEE extends this line of work to multi-class tasks and grounds the synthetic prior in a structured data-generating process that reflects meaningful class differences in the underlying dynamics. For a more in-depth discussion on the differences in the two approaches, we refer to Appendix 6.2.

2.4 Synthetic Data Generation for Time Series Foundation Models↩︎

Due to the scarcity of high-quality labeled real-world time series data, synthetic generate has become a key enabler of zero-shot generalization in many time series foundation models. KernelSynth, originally introduced in Chronos [25], generates diverse synthetic time series for forecasting through compositions of Gaussian process kernels. Chronos-2 further introduces using temporal causal models in order to generate synthetic univariate time series.

In a separate direction, several works have proposed data mixing strategies to further diversify synthetic pre-training data. TiRex [5] applies augmentation techniques including spike injection and convex mixing of base sequences, as well as an extended version of KernelSynth. CauKer [12] uses Structural Causal Models to produce clustering structure suitable for contrastive pre-training of time series encoders.

ICL pre-training for classification imposes a strictly harder requirement: synthetic data must consist of labeled classification tasks, not merely diverse time series. No prior work generates classification tasks in which class identity reflects differences in the data-generating process itself.

3 Method↩︎

We formulate the training objective of TimEE and detail the key components of our approach, beginning with the synthetic data generation procedure and concluding with the model architecture and pre-training setup.

3.1 Problem Formulation and Training Objective↩︎

While existing approaches solve each instance of \(p(\cdot \mid \mathbf{x}_{test}, \mathcal{D}_{train})\) independently via fine-tuning or two-stage pipelines, TimEE learns an amortized predictor \(f_\theta\) that approximates it in a single forward pass with no per-dataset weight updates. Rather than assuming a fixed dataset or label space, \(f_\theta\) accepts tasks of varying support size \(n_{train}\) and number of classes \(K\). Following the PFN [23] training objective, we train \(f_\theta\) by meta-learning over synthetic tasks drawn from a prior \(p_{synth}\). At each iteration, a synthetic dataset \(\mathcal{D}_k = \{(x_i, y_i)\}_{i=1}^N\) is sampled and partitioned into: \[\mathcal{D}_{k,train} = \{(x_j, y_j)\}_{j=1}^{n_{train}}, \quad \mathcal{D}_{k,test} = \{(x_l, y_l)\}_{l=1}^{n_{test}},\] and the model minimizes the negative log-likelihood over query positions: \[\mathcal{L}(\theta) = -\mathbb{E}_{\mathcal{D} \sim p_{synth}} \left[ \sum_{(x, y) \in \mathcal{D}_{k,test}} \log p_\theta(y \mid x, {D}_{k,train}) \right]. \label{eq:loss}\tag{1}\]

3.2 Synthetic Data Generation↩︎

The design of the prior \(p_{synth}\) is central to the performance of TimEE. While prior work (Section 2.4) has established synthetic data generation as a promising data source for pre-training time series foundation models, these approaches target either forecasting or produce classification tasks in which class identity does not reflect meaningful differences in the underlying generative process. We address this gap by pre-training TimEE on synthetic labeled TSC datasets in which class structure arises from controlled variation in the data-generating process itself.

Data-Generating Process. The foundation of our data-generating process is a temporal structural model over \(d\) endogenous variables. Let \(\mathbf{y}_t \in \mathbb{R}^d\) denote a multivariate³ random variable at time \(t\). We model its evolution as a Vector Autoregressive process with exogenous inputs (VARX) [26]: \[\begin{align} \mathbf{y}_t = \sum_{k=1}^{p}\mathbf{A}_k\mathbf{y}_{t-k} + \sum_{j=0}^{s}\mathbf{B}_j{\mathbf{x}_{t-j}} + \boldsymbol{\epsilon}_t, \qquad \boldsymbol{\epsilon}_t \sim \mathcal{N}(\mathbf{0}, \mathbf{\Sigma}) \end{align}\] where \(\mathbf{A}_k \in \mathbb{R}^{d \times d}\) are the lag-\(k\) autoregressive coefficient matrices, encoding inter-variable temporal relationships, \(\mathbf{x}_t \in \mathbb{R}^q\) is a \(q\)-dimensional exogenous input signal and \(\mathbf{B}_j \in \mathbb{R}^{d \times q}\) maps lag-\(j\) exogenous inputs to endogenous variables.

3.2.0.1 Class Identity Mechanism.

Classification structure is introduced through two complementary and independently controllable mechanisms:

• Structural Variation: Classes are defined by structural variation in the dependency structure encoded in \(\mathbf{A}_k\) and \(\mathbf{B}_j\) as shown in Figure 2 (left). Concretely, class identity is determined by which the set of edges that are present (or absent) in the dependency graph. This reflects well-established observations that many real-world classification tasks correspond to qualitatively different regimes of inter-variable interaction [27].

• Signal Variation. Classes are defined by the statistical properties of the exogenous input \(\mathbf{x}_t\) while the underlying dependency structure \(\mathbf{A}_k\), \(\mathbf{B}_k\) remains fixed. In particular, we consider warping of \(\mathbf{x}_t\) as class-discriminative feature, as shown in Figure 2 (right). The same underlying signal is hereby expressed at different speeds across classes, inducing systematic distributional differences in the endogenous \(\mathbf{y}_t\) without any change to the inter-variable structure [28].

Augmentations. To further enhance the diversity of our synthetic pre-training corpora, we apply a set of augmentations at the individual time series level. Concretely, we consider three transformation types:

Additionally, we apply a mixup scheme [5], [25], to generate convex combinations of synthetic time series across the dataset level. Full details about the augmentations are provided in Appendix 8.2 and 8.3.

3.3 Model Architecture↩︎

The architecture of TimEE is designed to jointly reason over all time series in the context, enabling in-context comparison between support and query examples in a single forward pass. It consists of five stages: (i) Tokenization to represent each series as a sequence of patch embeddings; (ii) Label Conditioning to inject class information to the patch embeddings; (ii) Encoder Stack to build class-aware contextual representations for each series (iv) In-Context Reasoning to perform cross-sample comparison; (v) Decoder to map query representations into class probabilities. The complete pipeline is illustrated in Figure 3, with an in-depth evaluation of each component provided in Appendix 7.4.

Tokenization. Like many time series foundation models (mentioned in Section 2), we first normalize each series to remove the individual scale, in order to encourage the model to focus on temporal shape rather than magnitude. Like Chronos-2 [4], we apply \(sinh^{-1}\) transformation to better handle the outliers. A subsequent cross-normalization step, computed using only the training-series statistics, further standardizes the representation across the context. Each series is then divided into non-overlapping patches of size \(p = 16\) and projected to \(d = 128\) dimensions via a shared linear layer.

Label Conditioning. A key advantage of the end-to-end paradigm is that class labels are available throughout the forward pass. We exploit this by conditioning the encoder directly on label identity. Concretely, following TabICLv2 [10], a learned class embedding is added to each training series’ patch representations, while query series receive no such signal. This allows subsequent attention layers to build class-aware contextual representations that are more discriminative and separable across classes, while preserving the label ambiguity of query patches that the model must resolve through in-context reasoning.

Encoder Stack. The goal of the encoder is to produce a compact, class-aware summary of each series that captures both its internal temporal structure and its relationship to other series in the context. To this end, \(K\) learnable CLS tokens are prepended to each series’ patch sequence, yielding representations \(\mathbf{Z} \in \mathbb{R}^{N \times (P+K) \times d}\). The encoder then operates in two sequential phases: \(L_t\) temporal attention layers first process each series independently across its own patches using Rotary Positional Embeddings (RoPE) [29], capturing within-series temporal structure; \(L_c\) cross-series attention layers then apply attention across all \(N\) series at each patch position, allowing each series to condition on the full labeled context. Queries attend to support examples but not to one another (see Appendix 7.2 for details). Finally, a compression layer collapses each series into its \(K\) CLS vectors by attending only over patch positions, producing a compact series summary \(\mathbf{r}_i \in \mathbb{R}^{K \times d}\).

In-Context Reasoning. Given compressed summaries of all series, the in-context reasoning layers perform the classification decision by comparing each query against the labeled support examples in the context. The \(N\) series summaries \(\{\mathbf{r}_i\}\) are processed by \(L_\mathrm{icl} = 2\) transformer layers operating in a 512-dimensional space. Class embeddings are again added to training summaries to keep support representations class-aware at this stage. The same masking strategy applies (Appendix 7.2), ensuring queries attend over labeled support examples without conditioning on one another.

Decoder. To produce a final classification decision, query representations are passed through layer norm and a two-layer MLP to produce logits over \(C\) classes. The model is trained with cross-entropy loss over query positions (Equation 1 ).

Many-Class Extension. TimEE is trained with at most \(C_\mathrm{max} = 10\) classes, which covers the majority of UCR datasets. For datasets with \(C > C_\mathrm{max}\), we adopt a one-vs-rest (OvR) decomposition strategy, reducing any \(C\)-class problem to \(C\) binary classification contexts. While more sophisticated strategies exist, we find OvR to be a simple and effective solution; we leave richer many-class handling as future work.

3.4 Pretraining and Inference↩︎

Pretraining Setup. TimEE is pretrained on 7M synthetic TSC tasks, derived from 1.5M unique VARX-generated datasets via augmentation (Section 3.2). Each task is constructed by uniformly sampling a support-query split and cropping all series in the context to a shared length sampled uniformly from \([16, 1024]\). We train for \(30K\) steps on a single NVIDIA H200 GPU with bf16 automatic mixed precision, taking approximately \(40\) hours. Following recent advances in optimizer design, we use Muon [30] for matrix parameters and AdamW [31] for all remaining weights, which we find leads to faster and more stable convergence than AdamW alone. Details of pre-training setup are included in Appendix 9.1.

Inference. To optimize inference, TimEE ensembles predictions across a small set of differently preprocessed inputs, combining two elementary transformations. We apply interpolation of short time series and first-order differencing, both individually and in composition. Final predictions are obtained by averaging predicted distributions across 4 ensemble members. We provide a more detailed description of our inference pipeline in Appendix 9.2.

4 Experiments↩︎

We evaluate TimEE against a broad set of state-of-the-art baselines for time series classification, assessing predictive performance, calibration, and inference efficiency.

4.1 Evaluation setup↩︎

Datasets. We evaluate TimEE on the UCR archive [16], a standard benchmark comprising 128 univariate TSC datasets, using the data splits and loading pipeline provided by aeon [32].

Baselines. We compare against a broad set of baselines spanning three categories (Section 2): classical and feature-based methods (DTW-1NN, Catch22, MiniRocket, Hydra), deep learning methods (InceptionTime, TSVec), and time series foundation models (NuTime, MOMENT, MantisV2, TiCT, Chronos-2, and TiRex).

As discussed in Section 2.2, most foundation model baselines require tuning—such as selecting the optimal intermediate layer representation—to achieve competitive performance; we therefore report both default and tuned results for these models. Full tuning details are provided in Appendix 10.

Regarding downstream classifier, for Catch22, MiniRocket, and Hydra, we use the aeon implementations with their canonical classifier pairings. For the remaining representation and foundation models, we select between Random Forest and SVM based on validation performance, with Random Forest generally preferred except for TS2Vec where SVM performs better. See Appendix 10 for more details.

4.2 Main Results↩︎

Figure 1 illustrates the mean rank across 128 UCR datasets. TimEE achieves the best mean rank on ROC AUC, outperforming all baselines in class discrimination. While TimEE ranks 3rd in Accuracy, its superior ROC AUC suggests it is the most robust choice across varying deployment thresholds and class distributions [33]. Notably, ridge-regression-based models like Hydra and MiniRocket rank substantially lower on ROC AUC despite competitive accuracy, a discrepancy indicative of overconfident predictions that degrade outside of a fixed 0.5 decision threshold. We present full per-dataset results in Appendix 11.3 Probabilistic Calibration. To evaluate whether these scores represent well-calibrated probabilities, we report mean rank over log-loss for datasets with \(\leq10\) classes⁴ (\(n=102\)) in Figure 5. TimEE achieves the top rank by a significant margin. As Log Loss is a strictly proper scoring rule, this result confirms that TimEE provides better-calibrated probabilistic estimates. The gap in Accuracy suggests that while ridge-based methods may force correct hard-label decisions on boundary samples via their objective function, TimEE maintains a more reliable representation of uncertainty, yielding more robust inputs for downstream decision-making.

Inference Speed. TimEE’s end-to-end ICL mechanism enables fast inference without any per-dataset training. While many other methods require a costly separate training phase, TimEE combines the traditional fit+predict stages in a single forward pass that directly produces predictions. As a result of this, as shown in Figure [fig:pareto], TimEE lies on the Pareto front of inference speed vs. predictive performance. Note that this comparison is restricted to datasets with at most 10 classes, as OvR is required for larger class counts and this introduces additional inference overhead (see Appendix Figure 17 (a)). This remains as a limitation of TimEE and is discussed in Section 5.

4.3 Ablations↩︎

We present ablation studies on:

.

4.3.0.1 Multivariate Extension.

We consider two strategies for adapting TimEE to multivariate settings:

The variate pooling variant is fine-tuned on synthetic multivariate data generated with our prior (Section 3.2). We evaluate both strategies against the baselines from Section 4 on the UEA archive. We restrict to 24 out of 30 UEA datasets⁵ as baselines’ prediction failure. As shown in Figure 6, the per-variate variant achieve competitive performance despite treating variates independently. Fine-tuning the variate-pooling variant, however, only yields a marginal gain. We attribute this to a limitation of our synthetic data generation: although the prior is natively extensible to multivariate inputs, the per-variate discriminative signal appears sufficient in practice, leaving little room for cross-variate aggregation to help. We discuss this further in Appendix 8.1.3.

4.3.0.2 How the Encoder Stack Builds Class-Discriminative Representations.

To understand how the encoder progressively builds class-discriminative representations, Figure 7 traces pairwise cosine similarities across its successive stages. Initially, patch embeddings are largely undifferentiated – similarities are uniformly high across all series regardless of class membership. Temporal attention begins to differentiate series by refining within-series structure, and same-class representations grow slightly more similar to one another. Cross-series attention then sharply accentuates this emerging structure akin to a contrastive separation in representation space. Crucially, query series — which receive no class-label signal during the forward pass — nonetheless align to their correct class cluster, confirming that the encoder builds class-discriminative representations that prime the subsequent in-context reasoning stage.

5 Conclusion↩︎

We introduced TimEE, a foundation model for time series classification that departs from the two-stage paradigm by framing TSC as an in-context learning problem. The key enabler is a synthetic prior grounded in a structured data-generating process, which produces labeled classification tasks where class identity reflects meaningful differences in the underlying dynamics. To our knowledge, no prior work addresses this for TSC. Our strong empirical results on the 128 UCR datasets suggest that the gap between the two-stage paradigm and end-to-end ICL is not merely from architecture, but also from the right synthetic prior and the ability to leverage class structure at inference time. We hope this work encourages further exploration of prior design, model scale, and richer class generation mechanisms, which is likely to lead to further substantial performance improvements.

Limitation 1: Many-class handling. TimEE is pre-trained with at most 10 classes and relies on a one-vs-rest decomposition for larger label spaces, which introduces significant inference overhead and diminishes the speed advantage of ICL-based classification (Figure 17). Extending TimEE to natively support larger class counts is an important direction for future work.

Limitation 2: Multivariate classification. Extending TimEE to multivariate inputs via variate pooling yields only marginal gains over the simpler per-variate baseline, which we attribute to a limitation of the VARX prior: class-discriminative information tends to concentrate in individual variates, leaving little signal for cross-variate aggregation to exploit. Designing priors that better leverage inter-variate structure remains an open problem.

Future Direction: Prior design and scale. TimEE is a small model trained on a single family of synthetic priors. The strong UCR results suggest that prior quality is a primary lever for performance in this paradigm — richer class generation mechanisms and larger model scale are therefore natural and promising avenues for future work.

Acknowledgments and Disclosure of Funding↩︎

J.K. is supported by the Konrad Zuse School of Excellence in Learning and Intelligent Systems (ELIZA) through the DAAD program Konrad Zuse Schools of Excellence in Artificial Intelligence, sponsored by the Federal Ministry of Education and Research. L.P. acknowledges funding by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) under SFB 1597 (SmallData), grant number 499552394; F.H. acknowledges the financial support of the Hector Foundation. The authors gratefully acknowledge the computing time made available to them on the high-performance computer at the NHR Center of TU Dresden. This center is jointly supported by the Federal Ministry of Research, Technology and Space of Germany and the state governments participating in the NHR (www.nhr-verein.de/unsere-partner). We thank the maintainers of the UCR [16] and UEA [34] archives for providing and maintaining the datasets used for evaluation. We thank the developers of open-source libraries used in the development of TimEE, including but not limited to: torch [35], transformers [36], scikit-learn [37], numpy [38] and aeon [32].

Author Contributions↩︎

J.K. and S.B.H. co-led the project, code development and experiments and writing of the paper. J.K. led the synthetic data generation and contributed to the model development, training infrastructure, and evaluation. S.B.H. led the model development and contributed to the synthetic data generation, training infrastructure, and evaluation. M.M. contributed to the evaluation and writing of the paper. F.H. supervised L.P. during the project. L.P. supervised J.K., S.B.H., and M.M. during the project, contributed ideas, and technical advice.

Competing Interests↩︎

L.P., and F.H. are affiliated with Prior Labs, a company focused on developing tabular foundation models. The authors declare no further competing interests.

6 Related Work Details↩︎

6.1 Comparison to related methods for Time Series Classification↩︎

6pt

6.2 Comparison TimEE vs TiCT [13]↩︎

Concurrent with this work, TiCT proposes end-to-end ICL for TSC with synthetic pre-training — the paradigm most directly related to ours. TiCT pre-trains a ResNet-based encoder on binary synthetic tasks via KernelSynth mixup, uses bit-based label encoding for arbitrary class counts, and selects context via 64-nearest-neighbor retrieval at inference. TimEE departs from TiCT in three key respects: our class generation mechanism — structural breaks in a VARX process – produces multi-class tasks where class identity reflects differences in the underlying dynamics; our transformer architecture explicitly models both temporal within-series structure and cross-series ICL; and we use the full training set as context without a retrieval step. We evaluate against substantially stronger baselines on both UCR and UEA. That TiCT and TimEE independently converge on the same paradigm strengthens the case that end-to-end ICL with synthetic pre-training is a promising direction for TSC.

7 Model Architecture↩︎

7.1 Model Hyperparameters↩︎

We report the model hyperparameters for TimEE in Table 2

7.2 Attention Masking Strategy↩︎

Let \(\mathcal{S} = \{1, \ldots, n\}\) and \(\mathcal{Q} = \{n+1, \ldots, n+m\}\) denote the indices of training and query series respectively, with \(N = n + m\). The binary attention mask \(\mathbf{M} \in \{0, 1\}^{N \times N}\), applied consistently across all cross-series attention layers, is defined as:

\[\mathbf{M}_{ij} = \begin{cases} 1 & \text{if } i \in \mathcal{S} \\ 1 & \text{if } i \in \mathcal{Q} \text{ and } j \in \mathcal{S} \cup \{i\} \\ 0 & \text{otherwise} \end{cases}\]

where \(\mathbf{M}_{ij} = 1\) indicates that series \(i\) is permitted to attend to series \(j\), and masked positions are set to \(-\infty\) in the pre-softmax attention logits. The resulting block structure is:

\[\mathbf{M} = \bordermatrix{ & \mathcal{S} & \mathcal{Q} \cr \mathcal{S} & \textcolor{cmarkcolor}{✔}& \textcolor{cmarkcolor}{✔}\cr \mathcal{Q} & \textcolor{cmarkcolor}{✔}& \mathbf{I} \cr }\]

Training series attend to all series; query series attend to all training series and themselves, but not to other query series. Temporal attention layers and the compression layer operate within each series independently and require no masking.

7.3 Extension to Multivariate↩︎

Architecture. TimEE is designed and trained to operate on univariate series only. To handle multivariate inputs \(\mathbf{X} \in \mathbb{R}^{N \times S \times M}\), we encode each of the \(M\) variates independently through the shared encoder, producing \(M\) sets of CLS token vectors per series. These are aggregated via slot-wise attention pooling: each of the \(K\) CLS slots has a learnable query that attends over the \(M\) variate embeddings via a softmax, allowing each slot to specialize on the most informative variates. The resulting pooled summary \(\mathbf{r}_i \in \mathbb{R}^{K \cdot d}\) is then passed to the ICL phase.

Fine-tuning Details. Queries are zero-initialized so that pooling starts as a uniform average, providing a stable initialization from the pretrained univariate checkpoint. The encoder weights are frozen and only the pooling queries, ICL block, and decoder are updated during multivariate fine-tuning.

7.4 Ablation of Model Components.↩︎

We conduct the following ablation studies in the following training setup while we modify the architecture configuration by:

• scaling down the model configuration to \(d_{kv}=16\), \(n_{heads}=2\);

• fixing the sequence length at 512 with batch size 64;

• training on our synthetic univariate data for 25K steps;

• evaluating on the train split of 86 UCR datasets (defined by: at least 100 samples, the least frequent classes must at least have 2 examples) with stratified 3-fold cross-validation.

Results reported are averaged over 3 seeds.

Contribution of Temporal and Cross-Series Attention. We ablate the contribution of cross-series (vertical) attention within the encoder by varying the allocation of encoder layers between temporal (\(L_t\)) and cross-series (\(L_c\)) attention, keeping the total fixed at \(L_t + L_c = 8\) with \(L_\mathrm{icl} = 2\).

Removing cross-series attention entirely (\(L_c = 0\)) causes a clear 2pp drop, confirming that encoder-level cross-series context provides complementary signal to the ICL phase and cannot be recovered from CLS-level comparison alone.

Removing temporal attention (\(L_t = 0\)) causes a smaller but still notable drop of approximately 1pp; local patch features provide a partial substitute for temporal structure, while cross-series comparison at the patch level is harder to replicate from compressed representations.

Performance is stable across all mixed configurations, indicating that the model is insensitive to the exact allocation as long as both components are present.

We note that all configurations retain a single compression attention layer (Section 3.3), so the \(L_t = 0\) condition is not strictly temporal-attention-free.

Contribution of conditioning on Class Label. We ablate the two class embedding injection sites: patch-level conditioning (added to training patches before the encoder) and ICL-level conditioning (added to CLS tokens before the ICL block).

Without any class conditioning, accuracy collapses to below 30%: the model has no access to label information during the forward pass and cannot associate support series with their classes. Patch-level conditioning is the dominant contributor—and in isolation sufficient: removing ICL-level conditioning while retaining patch-level conditioning leaves accuracy essentially unchanged (\(80.32 \to 80.35\), within run-to-run variance), whereas retaining only ICL-level conditioning drops accuracy by \(\sim\)​1.8pp (\(80.32 \to 78.55\)), because the encoder’s cross-series attention operates on class-unaware patch representations and produces less discriminative CLS summaries.

Although ICL-level conditioning provides no measurable benefit over patch-level conditioning alone, we retain it in our final model: it was included in the large-scale pretraining run, and the negligible overhead does not justify re-training without it. Together, the two sites interact non-linearly: their individual contributions do not account for the full \({\sim}\)​50pp collapse when both are removed, confirming that the model requires at least one explicit class signal to function.

8 Synthetic Data↩︎

8.1 Details on the TimEE Prior↩︎

8.1.1 Sampling Procedure↩︎

Synthetic datasets under the structural variation mechanism are sampled from a VARX-based prior [26], in which class identity is determined by class-specific edge dropout applied to a shared base dependency graph. The full sampling procedure is given in Algorithm 8. Synthetic datasets under the signal variation mechanism are sampled from a VARX-based prior in which the dependency structure \(\mathbf{A}_k, \mathbf{B}_j\) remains fixed across all classes, and class identity is instead determined by statistical indicators of the exogenous input signal. The full sampling procedure is given in Algorithm 9.

8.1.2 Relation to ARX↩︎

The VARX process described in Section 3.2 naturally subsumes the univariate setting as a special case. For \(d = 1\), the system reduces to a scalar AutoRegressive process with Exogenous inputs (ARX): \[\begin{align} y_t = \sum_{k=1}^{p} a_k y_{t-k} + \sum_{j=0}^{s} \mathbf{b}_j^\top \mathbf{x}_{t-j} + \varepsilon_t, \qquad \varepsilon_t \sim \mathcal{N}(0, \sigma^2) \end{align}\] where \(a_k \in \mathbb{R}\) and \(\mathbf{b}_j \in \mathbb{R}^q\) are the scalar and exogenous coefficient vectors respectively. All structural properties of our original VARX prior, the class-discriminative structural variations as well as the signal variations, carry over directly to this special case without modification. This ensures that the same DGP underlies both the univariate and multivariate synthetic data, and that results on univariate benchmarks such as UCR are fully consistent with the general VARX formulation. We note, however, that the limited transfer from multivariate synthetic pre-training data to real-world multivariate classification tasks remains a limitation of TimEE and identify it as an active avenue for future research.

8.1.3 Multivariate Synthetic Data↩︎

Our VARX-based prior natively supports multivariate data generation for arbitrary \(d>1\), and using the extension discussed in Section 4.3.0.1 TimEE can in principle be applied to multivariate time series classification tasks directly. However, we observe that in synthetically generated VARX datasets, the information necessary for class discrimination is typically well-concentrated in individual variates already. As a consequence, averaging predictions across all \(d\) variates rarely improves over the best single-variate prediction, and often even degrades performance due to the inclusion of uninformative channels. This behavior is consistent with observations in the forecasting literature [4] and finds motivation in Takens’ embedding theorem [39], which states that under mild conditions, an observation function of a dynamical system suffices to reconstruct the full state space of that system. While Takens’ theorem applies strictly to deterministic smooth dynamical systems, the underlying intuition is suggestive in our setting: in the VARX process, each observed \(y_t^{(i)}\) is a linear combination of the full history of all endogenous and exogenous variables, meaning inter-variable information naturally propagates into individual channels through the autoregressive dynamics. We highlight this through an empirical study on 20 synthetic multivariate datasets generated by the TimEE prior, shown in Figure 10. For each dataset, we compare the predictive performance of the single best univariate prediction - that is, the best performance achieved by any individual variate, against predictions obtained via variate pooling as described in Section 4.3.0.1. In nearly all datasets, variate pooling fails to improve over the single best variate prediction, and in several cases actively degrades performance. This confirms that under our VARX prior, class-discriminative information is already captured by individual variates, and aggregating across channels provides little to no benefit.

8.2 Augmentations↩︎

Smoothing. We apply a Gaussian filter with bandwidth \(\sigma \sim \mathcal{U}(\sigma_{\min}, \sigma_{\max})\), independently sampled per time series.

Spike Injection. The number of spikes \(n_s \sim \mathcal{U}\{n_{\min}, n_{\max}\}\) is sampled per series. Spike positions are drawn uniformly from \(\{1, \ldots, T\}\), and each spike amplitude is sampled independently as \(s \sim \mathcal{U}(s_{\min}, s_{\max})\), giving \(\mathbf{y}_t' = \mathbf{y}_t + s \cdot \mathbf{1}\) for a randomly drawn position \(t\).

Nonlinear Transformation. A nonlinearity is sampled uniformly from \(\{\tanh, \sin, \text{ReLU}, |\cdot|\}\) and applied pointwise as \(\mathbf{y}_t' = g(\mathbf{y}_t)\).

8.3 Mixup↩︎

To further diversify our synthetic pre-training datasets, we follow the mixup scheme used to augment forecasting data [4], [5]. We adapt this mechanism to the classification setting by forming convex combinations of samples drawn from within the same dataset, thereby preserving label structure while increasing intra-class diversity. Concretely, given \(k\) time series samples \(\tilde{\mathbf{y}}^{(1)}, \ldots, \tilde{\mathbf{y}}^{(k)}\) drawn from arbitrary classes, the mixed sample is: \[\begin{align} \mathbf{y}^{\text{mix}} = \sum_{i=1}^{k} \lambda_i \tilde{\mathbf{y}}^{(i)}, \qquad \lambda \sim \text{Dir}(\alpha) \end{align}\] where \(\sim \mathcal{U}\{1, K_{\max}\}\) and mixing weights \(\lambda\) are drawn from a Dirichlet distribution. The full procedure is described in Algorithm 11.

8.4 Examples of synthetic data↩︎

We present example datasets generated from our synthetic prior, for different class generation mechanisms. Figures 12, 13 present samples generated by structural variation. Figures 14 15 present datasets generated by signal level variation. Further we showcase the impact of the mixup scheme described in Section 8.3 on a generated synthetic dataset, where Figure 16 (a) shows the originally sampled dataset before applying mixup while Figure 16 shows the same dataset after applying mixup.

9 Pretraining and Inference↩︎

9.1 Pretraining Configurations↩︎

Table 5 summarizes the full set of hyperparameters used during pretraining.

9.2 Inference Pipeline↩︎

At inference, we found it beneficial to apply lightweight preprocessing transformations and ensemble predictions across differently preprocessed inputs. Concretely, we consider two elementary transformations: interpolation of short time series to a target length in {128,256,512}, and first-order differencing. Each ensemble member is formed from one of the individual transformation (or compositions of multiple individuals). Final predictions are obtained by averaging the predicted distributions across all members. By default, TimEE ensembles 4 preprocessing variants.

10 Implementation of Baselines↩︎

10.1 Statistical Methods↩︎

MiniRocket [1]: We apply MiniRocket feature extraction followed by a StandardScaler and RidgeClassifierCV, using the aeon implementation.

Hydra [2]: We use the aeon implementation with default hyperparameters.

Catch22 [14]: We extract 22 canonical time-series summary statistics via pycatch22 and classify with a RandomForestClassifier (n_estimators=300).

DTW-1NN [40]: 1-nearest-neighbour classifier with Dynamic Time Warping distance, using the aeon implementation.

10.2 Deep Learning Methods↩︎

TS2Vec [3]: A contrastive self-supervised encoder trained independently per dataset with output dimension 320. We extract full-series embeddings and classify with an SVM whose regularisation parameter \(C\) is selected via 5-fold cross-validation on the training set.

InceptionTime [17]: An ensemble of 5 InceptionTime networks trained for 1500 epochs per dataset, using the aeon implementation.

10.3 Foundation Models↩︎

For all foundation models we use a RandomForestClassifier from scikit-learn [37] with n_estimators=300 consistent with [6]. All remaining hyperparameters take their default values. Where a tuned variant is reported, we select the encoder layer whose embeddings yield the highest mean ROCAUC across the evaluation suite and apply it uniformly across all datasets.

10.3.1 TSC Foundation Models↩︎

Mantis-V2 [6]: We use the official implementation. ⁶ As Mantis-V2 requires input sequences of length divisible by num_patches, we interpolate to next higher num patches. For the tuned version, embeddings from the 3rd layer yielded the strongest downstream performance. We also interpolate each time series to a minimum length of 512.

Moment [19]: We use the large size model for all experiments with Moment. We follow all defaults from the official implementation. ⁷ Using later encoder layers performs comparably; we use the 13th layer for the univariate setting and the 11th for the multivariate setting.

NuTime [18]: A window-based ViT pretrained via BYOL on UCR datasets. We extract the frozen CLS-token embedding and classify with the above classifier. Input series are resampled to length 512. For the tuned variant, embeddings from the 6th layer yielded the strongest downstream performance.

TiCT [13]: A 47M-parameter ResNet pretrained for in-context time series classification. For each test sample, we retrieve its 64 nearest neighbours from the training set as context and perform a single forward pass; no separate classifier is required.

10.3.2 Forecasting Models↩︎

For both forecasting models, we extract patch-level embeddings per time series, apply mean pooling over patches to obtain a fixed-size representation, and classify using the above mentioned classifier.

Chronos-2 [4]: We use the base model throughout. By default, embeddings are extracted from the final encoder layer. For the tuned variant, we take embeddings from the 5th layer, which yielded the strongest downstream performance.

TiRex [5]: We apply the same extraction and classification strategy. For the tuned variant, we likewise use the 6th layer, following the same selection procedure.

10.4 Multivariate Setting↩︎

For models where we implement the multivariate adaptation ourselves, we reuse the best layer settings identified in the univariate search. Per-channel embeddings are concatenated before classification; mean pooling consistently degraded performance across models. We use a RandomForestClassifier throughout, as SVMs struggled with the extended embedding dimension produced by concatenation. TiCT classifies each channel independently and averages the per-channel softmax probabilities. DTW-1NN, MiniRocket, Hydra, and InceptionTime use their native multivariate implementations via aeon without modification. For TiRex, reusing the univariate tuned settings did not transfer to the multivariate setting and the default outperforms the tuned version.

11 Experiment Results↩︎

11.1 Inference Time vs. Performance on datasets↩︎

To evaluate the trade-off between the inference efficiency and the predictive performance, we measure the inference time of each method.

For GPU baselines, we evaluate all GPU-accelerated candidates on a NVIDIA H100 GPU; for CPU baselines, we evaluate them on 32-cores AMD EPYC 9655.

11.2 Critical Difference Diagram↩︎

11.3 Per-dataset results↩︎

References↩︎