1 Introduction↩︎

Unsupervised TSAD is a core component in industrial systems, as the ability to generate early alarms on abnormal timestamps without annotated data can support safe operation and reduce maintenance and downtime cost [1], [2]. In real deployments, anomalies are heterogeneous: some are short transients, some are gradual drifts, and some mainly appear as changes in periodic structure [3] and spectral energy [4], [5]. This heterogeneity renders a detector focusing on a single perspective unreliable, as illustrated in Fig. 1, and motivates the extraction of multi-view information for unsupervised TSAD [6].

From a signal processing perspective, the time and frequency domains are linked by the Fourier transform, which provides a principled connection between these two views [7]. This connection motivates unsupervised TSAD methods to jointly use both views [8]: time domain features capture local dynamics and abrupt deviations [9], while frequency domain features highlight periodic irregularities and band-specific shifts that can be subtle or suppressed by noise in the raw data [10]. As such, by measuring the disagreement between the representations learned from these two views, an anomaly score can be computed for each timestamp in an unsupervised fashion. However, most methods based on time and frequency views still operate on a single sampling resolution, where the model observes only one temporal granularity. Apart from restricting a model’s receptive field when learning time series representations, it also prevents the detector from explicitly verifying whether coarse structure (e.g., seasonality) and fine-grained dynamics (e.g., daily change) remain consistent. For example, a gradual drift can reshape the coarse trend while slightly detuning short-term rhythms [10]. In reality, abnormal traces can exhibit different patterns at different sampling granularities, and such anomalies can hardly be detected without inspecting such nuanced coarse-to-fine associations [4], [11]. In this regard, introducing multi-scale features at different sampling resolutions as an additional view is a natural enhancement to using only time and frequency features. Intuitively, by jointly modeling time-frequency alignment and cross-resolution consistency, a mutually complementary set of signals can be considered for TSAD. However, this brings an additional layer of complexity when learning useful anomaly detection signals from these views with varying semantics, and two significant challenges are to be addressed to maximize the utility of all views for unsupervised TSAD. We outline the challenges below.

Challenge 1: How to learn reliable view-specific and scale-specific representations without supervision? For time and frequency views, most TSAD methods fuse features or scores across both views, or align their feature spaces [12]. These designs rarely enforce Fourier-based consistency, meaning the predicted spectrum is not required to reconstruct the signal via an inverse transform, and the reconstructed signal is not required to match the spectrum under a forward transform [13]. This leaves the training process prone to learning a biased shortcut that only fits its own view, making cross-domain disagreement a less reliable indicator of true anomalies [12]. For the third view involving multi-scale feature encoding, its performance also hinges on the choice of sampling scales/granularities that are usually tuned via heuristics. However, in different time series data, regimes differ in dominant periods and noise, so the same setting can place the key periodic band at the wrong scale or mix it with noise and miss anomaly-related changes [5]. For example, in some time series with frequent fluctuations, performing straightforward downsampling on time series can fold fast oscillations into low frequencies, so a coarse-scale sequence may show a false flat trend created by aliasing rather than real long-range structure, distorting the learned representations [8].

Challenge 2: How to coordinate cross-view information while preserving useful discrepancies for timestamp-level anomaly scoring? Information fusion across three distinct views is expected to distil complementary information from time, frequency, and multi-scale structure, which enables coherence-based detection on anomalies. However, when generating anomaly scores for each timestamp, existing TSAD methods typically focus on either: (1) the disagreement between features extracted from time and frequency views; or (2) the disagreement across all data resolutions in the multi-scale view. Some works exploit time and frequency views through short-time Fourier transform (STFT) assisted by fusion or reconstruction objectives [8], [12], while others emphasize multi-scale features by learning representations at diverse temporal resolutions and aggregating scale-wise outputs [4]. While often developed in parallel, few pipelines jointly model time, frequency, and explicit multi-scale characteristics of the same time series in a unified framework to facilitate effective anomaly scoring on timestamps [14].

Motivated by these challenges, we propose Learnable Fusion of Tri-view Tokens (LEFT) for unsupervised TSAD, treating anomalies as violations of agreement among time dynamics, time-frequency structure, and multi-scale structure. LEFT builds three complementary views with measurable agreement. It enforces a bidirectional time–frequency cycle, so the time prediction must agree with the input spectrum after analysis and the spectrum prediction reconstructs the input signal after synthesis. This analysis-synthesis coupling goes beyond feature or score fusion, reduces view-specific shortcuts, and makes time–frequency disagreement more reliable anomaly evidence. For multi-scale structure, LEFT uses a Nyquist-constrained learnable filterbank with residual spectral coverage, learning dataset-specific bands without fixed partitions while band-limiting each scale before downsampling to control aliasing and avoid heuristic scale choices. The three token streams interact through selected lightweight fusion rather than dense all-pairs fusion, aligning shared information without washing out view-specific evidence. Training includes a cross-path consistency constraint that aligns the time–frequency cycle with the multi-scale reconstruction, and inference combines multi-scale residuals with cycle-based discrepancy signals to form the final anomaly score. We summarize our contributions as follows:

• We formalize a unified tri-view setting that combines time dynamics, time-frequency structure, and multi-scale structure, where anomalies are exposed as violations of cross-view and cross-resolution consistency.

• We propose a Nyquist-constrained learnable filterbank with residual spectral coverage for multi-scale structural tokenization. We introduce bidirectional time-frequency cycle consistency, enforcing analysis-synthesis agreement, and cross-path consistency, aligning the time-frequency pathway with multi-scale reconstruction. We design lightweight token interactions to promote alignment without over-mixing, preserving view-specific evidence for detection.

• Extensive experiments show consistent improvements over strong baselines, and demonstrate LEFT improves about 3% in VUS-ROC and 6% in VUS-PR, while reducing FLOPs by over 80% and speeding up training by about \(8\times\).

2 Related Work↩︎

Unsupervised TSAD     Unsupervised TSAD has been studied for decades, from classical pipelines to modern deep models [4], [15]. Classical methods such as LOF [16], DAGMM [17], and SVDD [18] are simple and cheap, but they can degrade on industrial data with non-stationary behavior, long-range dependence, and strong variable coupling [19]. Deep unsupervised TSAD is often grouped into forecasting, reconstruction, and self-supervised or contrastive methods [20]. Attention-based models are widely used to capture long-range patterns and variable interactions [4]. Anomaly Transformer detects anomalies via association inconsistency [21], while DCdetector uses discrepancy-aware contrastive learning [20].

Time–frequency modeling for unsupervised TSAD     Because anomaly evidence can be weak in a single domain, many works incorporate time–frequency information to capture spectral changes such as periodic shifts and band-wise energy redistribution [22]. Common designs extract Fourier or wavelet features, or adopt frequency-aware architectures [11]. In many cases, time and frequency are encoded in parallel and coupled through feature alignment or late fusion, which can underuse abnormal time–frequency coupling as a detection signal [8]. Empirical studies also suggest that single-scale time domain sampling can miss anomaly signatures in time–frequency analysis [4].

Multi-scale modeling for unsupervised TSAD     Multi-scale modeling addresses this issue by representing a series at multiple temporal resolutions. Recent work uses multi-resolution tokenization, patching, or scale mixing [23], and TimesNet models multiple period-based views [11]. In TSAD, multi-scale methods often use per-scale experts with late fusion or shared backbones with implicit scale mixing [24]. Yet anomalies may remain weak at any single resolution and become clearer when coarse structure and fine dynamics stop aligning, which motivates more explicit cross-scale modeling [4], [10]. At the same time, many multi-scale pipelines focus on time domain representations and do not explicitly integrate frequency domain evidence, even though spectral information can be critical for anomaly detection [5], [8].

To the best of our knowledge, in unsupervised TSAD, many methods emphasize either time–frequency modeling (e.g., [8]) or explicit multi-scale structure (e.g., [4]), while comparatively fewer works integrate both in a unified framework.

3 Method↩︎

Given a multivariate time-series window \(\boldsymbol{X}\in\mathbb{R}^{T\times C}\), the objective is to learn an unsupervised anomaly scoring function that outputs a timestamp-level anomaly map \(\mathcal{A}(t)\).

As illustrated in Fig. 2, LEFT constructs three token streams: \(\boldsymbol{H}^{\mathrm{time}}\), \(\boldsymbol{H}^{\mathrm{freq}}\), and \(\boldsymbol{H}^{\mathrm{ms}}\). LEFT is based on the premise that normal patterns tend to maintain agreement across domains and resolutions, whereas anomalous patterns are more likely to disturb such agreement. Training is guided by multi-scale reconstruction, cycle consistency, and cross-path consistency, which aligns the cycle pathway with the multi-scale pathway.

3.1 Tri-view Tokenization↩︎

3.1.1 Time Tokens↩︎

Time tokens provide a per-timestamp representation, which preserves local dynamics and abrupt deviations for timestamp-level scoring. We obtain them with a lightweight 1D convolutional stack \(\mathcal{C}^{(\mathrm{1d})}_t\) to provide a stable local inductive bias without resampling: \[\boldsymbol{H}^{\mathrm{time}} = \mathcal{C}^{(\mathrm{1d})}_t(\boldsymbol{X}), \label{eq95time95tokens}\tag{1}\] where \(\boldsymbol{H}^{\mathrm{time}}\in\mathbb{R}^{T\times D}\) and the kernel sizes of \(\mathcal{C}^{(\mathrm{1d})}_t\) uses 1D convolutions with kernel sizes \((5,3,1)\) and corresponding paddings to preserve temporal resolution \(T\).

3.1.2 Frequency Tokens↩︎

Frequency tokens capture short-time spectral structure, including changes in periodicity and shifts in band-wise energy that may be hard to see in raw samples. We obtain \(\boldsymbol{S}\) using a differentiable short-time Fourier transform. This design makes the frequency pathway directly trainable under the cycle consistency objective in Sec. 3.4, which encourages spectral consistency with the reconstructed signal. \(\boldsymbol{S}\) is obtained from \(\boldsymbol{X}\) as: \[\boldsymbol{S}=W_\theta(\boldsymbol{X}), \label{eq95stft95def}\tag{2}\] where \(\boldsymbol{S} \in\mathbb{R}^{C\times 2\times F\times T_F}\). \(W_\theta(\cdot)\) denotes the differentiable STFT used in the time-frequency branch. In the rest, we use \(\mathcal{F}(\cdot)\) and \(\mathcal{F}^{-1}(\cdot)\) for the FFT/iFFT used in the multi-scale decomposition. \(F\) is the number of frequency bins, and \(T_F\) is the number of STFT frames. Then a frequency encoder \(\mathcal{C}^{(\mathrm{2d})}_f\) produces frame-wise tokens: \[\boldsymbol{H}^{\mathrm{freq}} = \mathcal{C}^{(\mathrm{2d})}_f(\boldsymbol{S}), \label{eq95freq95tokens}\tag{3}\] where \(\boldsymbol{H}^{\mathrm{freq}}\in\mathbb{R}^{T_F\times D}\) and \(\mathcal{C}^{(\mathrm{2d})}_f\) treats \((2C)\) as channels, applies two 2D convolution layers with kernel size \((3,3)\), pools over \(F\) to obtain frame-wise features, and projects to dimension \(D\).

3.1.3 Multi-scale Structural Tokens via Nyquist-Constrained Learnable Filterbank↩︎

To capture multi-scale structure without fixed band partitions, we construct \(K\) band-limited downsampled sequences \(\{\boldsymbol{X}^{(k)}\}_{k=1}^{K}\) using a Nyquist-constrained learnable filterbank. Unlike enlarging the receptive field, this construction explicitly produces cross-resolution structure, so coarse-to-fine agreement can be checked during training and inference. Multi-scale tokens are only useful if they remain stable after downsampling.

Lemma 1 (proof in Appendix ¿sec:sec95proof95lemma1?) shows that a Nyquist-feasible learnable filterbank provides explicit control over aliasing. After downsampling, the aliasing energy is bounded by a term that scales with the out-of-cutoff leakage \(\epsilon_k\), so keeping \(\epsilon_k\) small makes the downsampled component \(\boldsymbol{X}^{(k)}\) a stable approximation of a band-limited signal. This property is useful for uncovering normal and abnormal patterns in the multi-scale view. Under normal operation, most energy stays within the learned cutoff, so coarse tokens preserve consistent long-term structure and support reliable coarse-to-fine reconstruction. When abnormal behavior shifts energy across bands or introduces atypical high-frequency content, \(\epsilon_k\) can increase and cross-scale agreement becomes harder to satisfy, which makes inconsistencies more detectable through reconstruction.

Given a boundary-extended signal \(\tilde{\boldsymbol{X}}\), let \(\tilde{\boldsymbol{S}}=\mathcal{F}(\tilde{\boldsymbol{X}})\). We then apply \(K\) learnable soft masks \(\{\tilde{m}_k(f)\}_{k=1}^K\) to \(\tilde{\boldsymbol{S}}\): \[\tilde{\boldsymbol{S}}^{(k)}(f)=\tilde{m}_k(f)\tilde{\boldsymbol{S}}(f), \quad k=1,\ldots,K. \label{eq95band95decomp}\tag{4}\] Let downsampling factors \(\{r_k\}_{k=1}^{K}\) be ordered coarse to fine (\(r_1\ge\cdots\ge r_K\)), and define normalized Nyquist cutoffs \(c_k=\tfrac{1}{2r_k}\). Band edges are parameterized monotonically with learnable parameters \(\{u_k\}\): \[e_k=e_{k-1}+\bigl(c_k-e_{k-1}\bigr)\sigma(u_k),\quad k=1,\ldots,K, \label{eq95learned95edges}\tag{5}\] where \(e_0=0\) and \(\sigma(\cdot)\) is the logistic sigmoid. Given \(s_\tau(x)=\sigma\big(x/(\tau+\varepsilon)\big)\) with temperature \(\tau>0\), unnormalized band masks are constructed by: \[m_k(f)=s_\tau(f-e_{k-1})-s_\tau(f-e_k),\quad k=1,\ldots,K, \label{eq95soft95masks95raw}\tag{6}\] and normalized as: \[\tilde{m}_k(f)=\frac{m_k(f)}{\sum_{j=1}^{K}m_j(f)+\varepsilon}. \label{eq95mask95norm}\tag{7}\] The residual spectral coverage is implemented with a residual mask above the last learned edge. Within the learned structural bands, the normalized band masks form an approximate partition, i.e., \[\sum_{k=1}^{K}\tilde{m}_k(f)\approx 1 \quad \forall f \;\text{s.t.}\quad \sum_{k=1}^{K} m_k(f)>0. \label{eq95mask95partition}\tag{8}\] Each component is transformed back by \(\mathcal{F}^{-1}(\cdot)\) and downsampled: \[\boldsymbol{X}^{(k)}=\mathcal{D}_{r_k}\!\left(\mathcal{F}^{-1}\!\left(\tilde{\boldsymbol{S}}^{(k)}\right)\right), \quad T_k=\left\lceil \frac{T}{r_k}\right\rceil, \label{eq95downsample}\tag{9}\] where \(\boldsymbol{X}^{(k)}\in\mathbb{R}^{T_k\times C}\). Each scale is patch-tokenized and projected into \(D\) dimensions: \[\boldsymbol{H}^{(k)}=\mathcal{P}\bigl(\boldsymbol{X}^{(k)}\bigr), \label{eq95ms95patch}\tag{10}\] where \(\boldsymbol{H}^{(k)}\in\mathbb{R}^{L_k\times D}\). \(\mathcal{P}(\cdot)\) denotes the patch tokenization and projection operator that maps each downsampled sequence into token representations. To preserve scale-specific structural patterns within the multi-scale encoder, we use a scale independent attention mask \(\boldsymbol{M}_{\mathrm{blk}}\in\mathbb{R}^{L\times L}\) that is block-diagonal over scales and shared across layers. This prevents mixing across scales inside \(\phi^{\mathrm{ms}}\). Cross-view and cross-resolution exchange is handled later by the tri-view interaction module: \[\boldsymbol{H}^{\mathrm{ms}} = \phi^{\mathrm{ms}}\!\left(\mathrm{Concat}\bigl(\boldsymbol{H}^{(1)},\ldots,\boldsymbol{H}^{(K)}\bigr);\boldsymbol{M}_{\mathrm{blk}}\right), \label{eq95ms95encoder}\tag{11}\] where \(\phi^{\mathrm{ms}}(\cdot;\boldsymbol{M}_{\mathrm{blk}})\) denotes a masked self-attention encoder.

3.2 Tri-view Interaction↩︎

Given \((\boldsymbol{H}^{\mathrm{time}},\boldsymbol{H}^{\mathrm{freq}},\boldsymbol{H}^{\mathrm{ms}})\), LEFT uses lightweight token-level interaction to share evidence while preserving view-specific information. LEFT avoids dense all-pairs fusion because some view pairs are scale-mismatched. \(\boldsymbol{H}^{\mathrm{freq}}\) is organized by STFT frames, while \(\boldsymbol{H}^{\mathrm{ms}}\) comes from downsampled sequences. Direct attention between them would force implicit alignment across scales, which can over-mix features and amplify noise.

LEFT therefore uses selected links. Time and frequency views are connected by the Fourier transform, so \(\boldsymbol{H}^{\mathrm{time}}\) attends to \(\boldsymbol{H}^{\mathrm{freq}}\) to inject spectral information into time-aligned representations, and \(\boldsymbol{H}^{\mathrm{freq}}\) attends back to \(\boldsymbol{H}^{\mathrm{time}}\) to stay tied to the raw signal. Multi-scale tokens are also derived from the time domain, so \(\boldsymbol{H}^{\mathrm{time}}\) updates \(\boldsymbol{H}^{\mathrm{ms}}\). This design also allows frequency evidence to influence the multi-scale pathway indirectly through the updated time tokens \(\boldsymbol{H}^{\mathrm{time}}\). The Fusion Strategy Analysis further shows that all-pairs fusion or denser fusion does not improve performance in our setting.

The interaction module is a stack of \(L_f\) shared fusion blocks: \[(\boldsymbol{H}^{\mathrm{time}+},\boldsymbol{H}^{\mathrm{freq}+},\boldsymbol{H}^{\mathrm{ms}+}) = \Phi^{(L_f)}\!\left(\boldsymbol{H}^{\mathrm{time}},\boldsymbol{H}^{\mathrm{freq}},\boldsymbol{H}^{\mathrm{ms}}\right), \label{eq95triview95fusion95compact}\tag{12}\] where each \(\Phi^{(L_f)}\) layer updates the three views via directed cross-attention: \(\boldsymbol{H}^{\mathrm{time}}\!=\!\phi^{\text{tf}(L_f)}(\boldsymbol{H}^{\mathrm{time}},\boldsymbol{H}^{\mathrm{freq}})\), \(\boldsymbol{H}^{\mathrm{freq}}\!=\!\phi^{\text{ft}(L_f)}(\boldsymbol{H}^{\mathrm{freq}},\boldsymbol{H}^{\mathrm{time}})\), and \(\boldsymbol{H}^{\mathrm{ms}}\!=\!\phi^{\text{mt}(L_f)}(\boldsymbol{H}^{\mathrm{ms}},\boldsymbol{H}^{\mathrm{time}})\), followed by a token-wise residual with normalization. Finally, this selected fusion is closed by the cross-path constraint \(\mathcal{L}_{\mathrm{cons}}\) in Sec. 3.4, which ties the multi-scale reconstruction \(\hat{\boldsymbol{X}}_{\mathrm{ms}}\) to the cycle reconstruction \(\hat{\boldsymbol{X}}_{\leftarrow f}\), so alignment learned through interaction is made verifiable at the reconstruction level rather than remaining a purely embedding-level agreement.

3.3 Prototype-based Time–Frequency Cycle Calibration and Memory Mixing↩︎

To provide a shared reference for matching time and frequency representations and reduce noisy or drifting alignment, we introduce two prototype banks \(\boldsymbol{P}_t,\boldsymbol{P}_f\in\mathbb{R}^{M\times D}\) and map fused tokens to soft prototype assignments: \[\boldsymbol{p}_t=\eta\!\left(\gamma\,\mathrm{cos}\!\left(\boldsymbol{H}^{\mathrm{time}+},\boldsymbol{P}_t\right)\right),\quad \boldsymbol{p}_f=\eta\!\left(\gamma\,\mathrm{cos}\!\left(\boldsymbol{H}^{\mathrm{freq}+},\boldsymbol{P}_f\right)\right), \label{eq95proto95assign}\tag{13}\] where \(\eta\) is Softmax. \(\mathrm{cos}(\cdot,\cdot)\) denotes cosine similarity and \(\gamma\) is a temperature. We align \(\boldsymbol{p}_f\) to length \(T\) by interpolation and compute a prototype-assignment discrepancy between the time and frequency views: \[d_{\mathrm{JS}}(t)=\mathrm{JS}\!\left(\boldsymbol{p}_t(t),\,\tilde{\boldsymbol{p}}_f(t)\right). \label{eq95js}\tag{14}\] The discrepancy \(d_{\mathrm{JS}}(t)\) is used for prototype calibration during training. Specifically, LEFT uses the averaged JS discrepancy, together with the reconstruction error and assignment confidence, to select reliable samples for EMA-based prototype updates. This design avoids directly optimizing JS divergence while still using cross-view assignment disagreement to maintain stable prototype memories. We also derive an uncertainty gate \(g(t)\) from assignment sharpness and entropy. Different from \(d_{\mathrm{JS}}(t)\), this gate is directly used as auxiliary evidence in the inference score.

The prototypes further act as a memory of stable patterns, read by time-averaged assignments: \[\boldsymbol{z}_t^{\mathrm{mem}}=\Bigl(\mathrm{Mean}_t(\boldsymbol{p}_t)\Bigr)\boldsymbol{P}_t, \quad \boldsymbol{z}_f^{\mathrm{mem}}=\Bigl(\mathrm{Mean}_t(\boldsymbol{p}_f)\Bigr)\boldsymbol{P}_f. \label{eq95mem95read}\tag{15}\]

Let \(\boldsymbol{z}_t=\mathrm{Mean}(\boldsymbol{H}^{\mathrm{time}+})\) and \(\boldsymbol{z}_f=\mathrm{Mean}(\boldsymbol{H}^{\mathrm{freq}+})\). We use curriculum memory mixing to stabilize decoding, gradually increasing the memory contribution with \(\lambda\in[0,1]\): \[\boldsymbol{z}_t^{+}=(1-\lambda)\boldsymbol{z}_t+\lambda\boldsymbol{z}_t^{\mathrm{mem}}, \quad \boldsymbol{z}_f^{+}=(1-\lambda)\boldsymbol{z}_f+\lambda\boldsymbol{z}_f^{\mathrm{mem}}. \label{eq95mem95mix}\tag{16}\]

3.4 Time-Frequency Cycle Decoding↩︎

The time-frequency pathway is trained as a bidirectional closed loop through \(W_\theta\) and \(W_\theta^{-1}\). The key purpose is to make the frequency prediction physically checkable: a valid \(\hat{\boldsymbol{S}}\) should synthesize to a consistent signal, and a reconstructed signal should induce a consistent time-frequency representation. Without this loop, the frequency branch can be under-constrained and drift to spectra that match coarse energy statistics but are not synthesis-consistent, which weakens cross-domain discrepancy as anomaly evidence.

We fuse time/frequency latents into a shared \(\boldsymbol{z}=\phi\!\left([\boldsymbol{z}_t^{+};\boldsymbol{z}_f^{+}]\right)\in\mathbb{R}^{D}\). Then decode into both time and frequency outputs, \(\hat{\boldsymbol{X}} = D_t(\boldsymbol{z})\) and \(\hat{\boldsymbol{S}} = D_f(\boldsymbol{z})\). Define cycle reconstructions: \[\hat{\boldsymbol{X}}_{\leftarrow f}=W_\theta^{-1}(\hat{\boldsymbol{S}}), \quad \hat{\boldsymbol{S}}_{\rightarrow f}=W_\theta(\hat{\boldsymbol{X}}). \label{eq95cycle95decode95aux}\tag{17}\]

Enforcing both directions makes the two outputs mutually verifiable and discourages one-branch shortcut fitting: a spectrum is penalized if it cannot synthesize back to a consistent \(\hat{\boldsymbol{X}}_{\leftarrow f}\), and a waveform is penalized if it induces an inconsistent \(\hat{\boldsymbol{S}}_{\rightarrow f}\).

From Lemma 2 (proof is given in Appendix ¿sec:sec95proof95lemma2?), when the time-frequency transform \(W_\theta\) satisfies frame bounds, the reconstruction errors in the time domain and the time-frequency domain bound each other up to constants. Therefore, beyond feature or score level fusion, we introduce the following cycle consistency constraint: \[\mathcal{L}_{\mathrm{cyc}} = l\!\left(\hat{\boldsymbol{S}}_{\rightarrow f},\,\boldsymbol{S}\right) + l\!\left(\hat{\boldsymbol{X}}_{\leftarrow f},\,\boldsymbol{X}\right). \label{eq95l95cyc}\tag{18}\]

We add a cross-path constraint to explicitly align the multi-scale reconstruction pathway with the time-frequency cycle pathway. This alignment is required for tri-view learning, since LEFT aims to enforce agreement among time, frequency, and multi-scale structure within a single model. The alignment target can be either \(\hat{\boldsymbol{X}}_{\leftarrow f}\) or \(\hat{\boldsymbol{X}}\). The cycle loss in Eq. 18 aligns both reconstructions to the same reference \(\boldsymbol{X}\), and it also implies that \(\hat{\boldsymbol{X}}_{\leftarrow f}\) and \(\hat{\boldsymbol{X}}\) become close because \(\|\hat{\boldsymbol{X}}-\hat{\boldsymbol{X}}_{\leftarrow f}\|_2 \le \|\hat{\boldsymbol{X}}-\boldsymbol{X}\|_2 + \|\hat{\boldsymbol{X}}_{\leftarrow f}-\boldsymbol{X}\|_2\). Therefore, it is sufficient to constrain \(\hat{\boldsymbol{X}}_{\mathrm{ms}}\) to one of them. We choose \(\hat{\boldsymbol{X}}_{\leftarrow f}\) and enforce their agreement as follows: \[\mathcal{L}_{\mathrm{cons}} = l\!\left(\hat{\boldsymbol{X}}_{\mathrm{ms}},\,\hat{\boldsymbol{X}}_{\leftarrow f}\right). \label{eq95l95cons}\tag{19}\] This ties the multi-scale reconstruction pathway to the cycle pathway, so cross-resolution structure learned by \(\hat{\boldsymbol{X}}_{\mathrm{ms}}\) is required to agree with the synthesis-consistent reconstruction \(\hat{\boldsymbol{X}}_{\leftarrow f}\). The tri-view alignment is enforced at the reconstruction level, and cross-path inconsistencies are unlikely to be caused solely by representation drift, making them useful evidence for anomaly scoring.

3.5 Multi-scale Reconstruction Head↩︎

The multi-scale pathway aims to reconstruct Nyquist-feasible structural content across resolutions, so that coarse-to-fine agreement can be explicitly checked. Starting from fused multi-scale tokens \(\boldsymbol{H}^{\mathrm{ms}+}\), we perform reconstruction in a coarse-to-fine manner: coarser tokens provide structural context while finer targets enforce detailed consistency. To retain scale-specific structure while injecting cross-view evidence, we combine the original multi-scale tokens and the interaction-enhanced tokens: \[\tilde{\boldsymbol{H}}^{\mathrm{ms}}=\beta_{\mathrm{res}}\boldsymbol{H}^{\mathrm{ms}}+\beta_{\mathrm{int}}\boldsymbol{H}^{\mathrm{ms}+}, \label{eq95ms95fuse}\tag{20}\] where \(\boldsymbol{H}^{\mathrm{ms}}\) preserves scale-pure structural evidence produced by the block-diagonal encoder in Eq. 11 , while \(\boldsymbol{H}^{\mathrm{ms}+}\) carries alignment signals injected through tri-view interaction.

We then reconstruct multi-scale targets by splitting tokens by scale, upsampling in token space, and decoding patches: \[\hat{\boldsymbol{Y}} = D_{\mathrm{ms}}\!\left(\mathrm{Up}\bigl(\mathrm{Split}(\tilde{\boldsymbol{H}}^{\mathrm{ms}})\bigr)\right), \label{eq95ms95decode}\tag{21}\] which yields reconstructions for scales \(k=2,\ldots,K\) and the full-rate reconstruction \(\hat{\boldsymbol{X}}_{\mathrm{ms}}\in\mathbb{R}^{T\times C}\). We treat the coarsest scale \(k=1\) as structural context and do not supervise it, since directly fitting the coarsest component can encourage overly smooth solutions while contributing limited timestamp-level anomaly localization.

Define the multi-scale target: \[\boldsymbol{Y}=\mathrm{Concat}\bigl(\boldsymbol{X}^{(2)},\ldots,\boldsymbol{X}^{(K)},\boldsymbol{X}\bigr). \label{eq95ms95target}\tag{22}\]

We optimize a segment-aware objective so each scale contributes comparably despite different lengths: \[\begin{align} \mathcal{L}_{\mathrm{ms}} &= \sum_{k=2}^{K}\omega_k\,l\!\left(\hat{\boldsymbol{X}}^{(k)}_{\mathrm{ms}},\boldsymbol{X}^{(k)}\right) +\omega_{\mathrm{full}}\,l\!\left(\hat{\boldsymbol{X}}_{\mathrm{ms}},\boldsymbol{X}\right), \\ &\text{s.t.} \quad \sum_{k=2}^{K}\omega_k+\omega_{\mathrm{full}}=1, \end{align} \label{eq95l95ms}\tag{23}\] where \(l(\cdot,\cdot)\) denotes SmoothL1.

3.6 Training Objective↩︎

LEFT is trained to make agreement an inherent property: the multi-scale pathway should reconstruct Nyquist-feasible structure across resolutions, the time-frequency pathway should satisfy bidirectional analysis-synthesis consistency, and the two pathways should agree on the full-rate reconstruction. Accordingly, the overall training loss combines (i) multi-scale reconstruction, (ii) cycle consistency, and (iii) cross-path consistency on the time domain: \[\mathcal{L} = \lambda_{\mathrm{ms}}\mathcal{L}_{\mathrm{ms}} + \lambda_{\mathrm{cyc}}\mathcal{L}_{\mathrm{cyc}} + \lambda_{\mathrm{cons}}\mathcal{L}_{\mathrm{cons}}, \label{eq95total95loss}\tag{24}\] where \(\mathcal{L}_{\mathrm{ms}}\) teaches scale-specific structure to be reconstructable, \(\mathcal{L}_{\mathrm{cyc}}\) restricts the time-frequency outputs to be mutually reconstructable through \(W_\theta\) and \(W_\theta^{-1}\), and \(\mathcal{L}_{\mathrm{cons}}\) locks the two pathways to a shared full-rate prediction, so discrepancies at inference are less likely to arise only from pathway drift and can provide useful evidence for anomaly scoring.

3.7 Tri-consistency Anomaly Scoring↩︎

At inference, LEFT scores anomalies as violations of agreement in two complementary forms: inconsistency within the time-frequency cycle, and reconstruction residuals across resolutions.

From Lemma 3 (proof is given in Appendix ¿sec:sec95proof95lemma3?), we propose a timestamp-level anomaly map: \[\mathcal{A}(t)=\alpha_{\mathrm{cyc}}\mathcal{A}_{\mathrm{cyc}}(t)+\alpha_{\mathrm{ms}}\mathcal{A}_{\mathrm{ms}}(t), \label{eq95score95total}\tag{28}\] where \(\mathcal{A}_{\mathrm{cyc}}(t)\) is the cycle-based discrepancy score and \(\mathcal{A}_{\mathrm{ms}}(t)\) is the multi-scale reconstruction score. We compute cycle residuals and apply a moving-average \(\mathrm{MA}_\kappa\) with smoothing window size \(\kappa\) to suppress local noise and stabilize timestamp-level decisions: \[\mathcal{A}_{\mathrm{cyc}}(t) = \Big(\mathrm{MA}_\kappa\!\big( \alpha_f\lvert\hat{\boldsymbol{X}}_{\leftarrow f}-\boldsymbol{X}\rvert +\alpha_t\lvert\hat{\boldsymbol{X}}-\boldsymbol{X}\rvert +\alpha_g\,g(t) +\alpha_c\,c(t) \big)\Big)(t), \label{eq95score95cyc}\tag{29}\] where \(c(t)\) measures cross-path discrepancy between the multi-scale pathway and the cycle pathway on the time domain, so disagreements penalized by \(\mathcal{L}_{\mathrm{cons}}\) during training can be directly reflected in the inference score. \(g(t)\) is computed from the sharpness and entropy of the time- and frequency-view prototype assignments. Multi-scale score aggregates errors from all supervised scales by aligning them to the full resolution, so evidence that is salient at a coarse scale can still contribute to the fine-grained anomaly map: \[\mathcal{A}_{\mathrm{ms}}(t) = \lvert\hat{\boldsymbol{X}}_{\mathrm{ms}}(t)-\boldsymbol{X}(t)\rvert + \sum_{k=2}^{K}\operatorname{Up}_{T}\!\left(\lvert\hat{\boldsymbol{X}}^{(k)}_{\mathrm{ms}}-\boldsymbol{X}^{(k)}\rvert\right)(t), \label{eq95score95ms}\tag{30}\] where \(\operatorname{Up}_{T}(\cdot)\) upsamples a sequence to length \(T\). This design avoids relying on a single-view dominant residual: any-view irregularity can increase \(\mathcal{A}(t)\) either through cycle disagreement, cross-path mismatch, or cross-resolution residual accumulation.

4 Experiments↩︎

4.1 Experimental Settings↩︎

Datasets.

We evaluate LEFT on public TSAD benchmarks covering server monitoring, space telemetry, and industrial control. SMD and PSM contain server resource-utilization or performance metrics [25], [26]; MSL and SMAP are NASA telemetry datasets with multivariate sensor and actuator signals [27]; and SWaT records water-treatment sensor traces under normal operation and attacks [28]. We also use NeurIPS-TS and report results on its GECCO and SWAN subsets [29]. For MSL and SMAP, we follow prior work [4], [30] and keep only the first continuous channel, since discrete variables are less suitable for reconstruction-based scoring. Dataset details are given in Table 1, and the data split and sliding-window setup follow [4].

Baselines. Baselines include linear transformation-based methods, such as OCSVM[31] and PCA[32], and outlier-oriented detectors, including IForest[33] and LODA[34]. We consider density-based methods (HBOS[35], LOF[16]), which remain competitive in low-dimensional regimes but can be sensitive to complex temporal drift. Neural baselines span reconstruction and forecasting, including AutoEncoder (AE)[36], DAGMM[17], LSTM[27], CAE-Ensemble (CAE)[37], and Omni-Anomaly (Omni)[25]. We include recent TSAD models from transformer, contrastive, frequency-aware, and pre-trained lines, such as Anomaly Transformer (AT)[21], DCdetector (DC)[20], ModernTCN[38], GPT4TS[39], MtsCID[40], TimeMixer[23], TimesNet[11], CrossAD[4], DADA[41], and CATCH[42]. Among them, CATCH is a recent frequency-aware baseline, DADA is a recent pre-trained general baseline, and MtsCID, TimeMixer, TimesNet, and CrossAD are closely related multi-scale SOTA methods.

Metrics. Evaluation in TSAD can be misleading under point adjustment, since even random predictors may obtain inflated scores [20], [21]. TSB-AD [43] further argues that VUS-PR [44] provides a more robust and less lag-sensitive evaluation, while several common metrics can be biased across scenarios. We therefore report VUS-PR and VUS-ROC as the main metrics  [44].

Implementation Details. In our setup, the time-view encoder is a lightweight 1D convolutional stack with kernel sizes \((5,3,1)\), while the frequency-view encoder is a lightweight 2D convolutional stack with two \((3,3)\) layers. The multi-scale branch uses a fixed set of downsampling factors \(\{r_k\}_{k=1}^K\) and the number of scales \(K\) is given by the number of downsampling factors. All decoders are two-layer MLPs. We use sliding windows for both training and inference, and report results under a non-overlapping window protocol. Optimization uses Adam with learning rate \(10^{-4}\) and batch size 128. Following [45], we keep all test windows at inference time and do not apply the drop-last trick. We set decision thresholds with SPOT [46], consistent with prior TSAD practice [4], [30]. All experiments are implemented in PyTorch and run on a single NVIDIA L40 GPU. More details and code are available at https://github.com/DezhengWang/Left.git.

4.2 Detection Results↩︎

We evaluate LEFT on seven real-world TSAD benchmarks and compare it with 21 competitive baselines, as reported in Table 2. Across datasets with varied dynamics and anomaly types, LEFT achieves stronger ranking-based detection quality under both VUS-PR and VUS-ROC. As shown in Table 2, LEFT improves both metrics on every benchmark over the strongest baseline (underlined), with average gains of about 0.023 in both VUS-ROC and VUS-PR. Relative to the strongest baseline on each dataset, LEFT improves VUS-ROC by about 3.45% and VUS-PR by about 6.45% on average. The largest relative gains appear on SMAP, where LEFT improves VUS-ROC by 13.55% and VUS-PR by 16.70% relative to the strongest baseline.

For a comprehensive comparison, we report additional evaluation metrics in Table 3. We include AUC-ROC (AUC-R), AUC-PR (AUC-P), Range-AUC-ROC (R-A-R), Range-AUC-PR (R-A-P), Accuracy (Acc), and Standard F1 (F1). Here we compare only with CrossAD [4], since it is the best-performing baseline in our main results (see Table 2). The results show that LEFT performs strongly across these metrics, which further supports its effectiveness for time series anomaly detection.

4.3 Ablation Studies↩︎

As shown in Table 4, to evaluate the components of LEFT, we conduct detailed ablations. The results show that LEFT achieves the best performance, which indicates that the gains come from the combined effect of the proposed modules rather than any single component. Removing tri-view interaction reduces the average from 0.7518 / 0.3590 to 0.7256 / 0.3455, showing that cross-view alignment contributes to detection quality. Replacing the learnable filterbank with a fixed one causes a similar drop to 0.7261 / 0.3456 on average, suggesting that adaptive multi-scale structure captures evidence that fixed band splits miss in this setting. The largest effect comes from cycle consistency, since removing it collapses average performance to 0.5422 / 0.2411, which supports the view that many anomalies are better exposed by agreement breaks between the time and frequency paths than by a dominant single-branch residual. Removing cross-path consistency lowers the average to 0.7266 / 0.3416, which is consistent with its role in aligning the multi-scale pathway with the time–frequency pathway. In Appendix 7.1, we provide a more comprehensive ablation analysis.

4.4 Model Efficiency↩︎

We compare the efficiency of LEFT with representative TSAD methods on PSM. The baselines cover several common design lines in recent work. AnomalyTransformer [21] uses a transformer and detects anomalies through association discrepancy. CrossAD [4] uses a transformer for multi-scale modeling. TimeMixer [23] uses MLP-based multi-scale mixing. TimesNet [11] uses frequency-aware convolutional modeling. Table 5 shows that LEFT achieves the lowest FLOPs and the fastest measured runtime among the compared methods, while maintaining a moderate model size. LEFT uses 188.849M FLOPs and 4.256M parameters. LEFT runs in 25.66 seconds per training epoch and 0.01527 seconds per 128 samples at inference. LEFT also shows clear efficiency advantages over strong baselines. The results suggest that LEFT stays small enough for typical deployment budgets, while its compute profile leads to clear wall-clock gains during both training and inference.

4.5 Model Analysis↩︎

4.5.1 Fusion Strategy Analysis↩︎

We compare all-pairs fusion and two-view variants. All-pairs fusion, denoted as \(M\leftrightarrow F\leftrightarrow T\), updates each view by aggregating information from the other two views, followed by residual addition and layer normalization. The two-view variants \(M\leftrightarrow F\), \(M\leftrightarrow T\), and \(T\leftrightarrow F\) enable interaction only within the selected pair of views, while keeping the third view unchanged. \(T\leftrightarrow F\) exchanges information only between time and frequency tokens, \(M\leftrightarrow T\) exchanges information only between multi-scale and time tokens, and \(M\leftrightarrow F\) exchanges information only between multi-scale and frequency tokens. The impact of fusion strategies on detection performance is reported in Table 6. Across PSM, MSL, and SMAP, the proposed tri-view fusion yields competitive overall performance. It consistently improves over the best two-view variant in both VUS-ROC and VUS-PR, showing that using evidence from all three views jointly is more effective than relying on any pair of views. All-pairs fusion is not always better. It adds many cross-view links, which can mix the views too strongly and spread noise across branches. This can weaken discrepancy signals and hurt performance on some datasets. In contrast, the proposed fusion uses a selected set of interactions, which is enough to align the views, but it keeps view-specific information separated so discrepancies stay meaningful for anomaly scoring. In Appendix 7.2, we provide a more comprehensive analysis.

4.5.2 Fusion Hyper-parameters Analysis↩︎

Figs. 3 (a, b) show that increasing the depth or width of the fusion layer does not produce a meaningful performance gain, which aligns with the observation in Sec. 4.5.1 that stronger fusion is not always better. In this setting, a heavier fusion tends to over-mix the views, so distinctive evidence becomes less separable in the shared representation.

4.5.3 \(\lambda_{\mathrm{cyc}}\) and \(\lambda_{\mathrm{cons}}\) Sensitivity↩︎

Figs. 3 (c, d) suggest that detection quality depends on the choice of cycle consistency and cross-path consistency. LEFT remains competitive when cycle consistency is strong and cross-path consistency is relatively weak. Pushing cross-path consistency to large values does not yield stable gains. The results suggest that an overly strong cross-path consistency weight can over-align the pathways and weaken fault evidence. A stronger cycle consistency weight is often beneficial because it better exploits the analysis-synthesis link between time and frequency views.

4.5.4 \(\alpha_{\mathrm{cyc}}\) and \(\alpha_{\mathrm{ms}}\) Sensitivity↩︎

Figs. 3 (e, f) show that both VUS-ROC and VUS-PR depend on the choice of \(\alpha_{\mathrm{cyc}}\) and \(\alpha_{\mathrm{ms}}\), and neither metric improves monotonically when only one weight is increased. High-score regions concentrate in a middle range of the grid, while several extreme settings lead to lower values. This pattern echoes our main contribution that LEFT gains from combining cycle disagreement and reconstruction disagreement with multi-scale residual evidence, where reliable detection comes from their complementary use rather than amplifying a single term.

4.5.5 Effect of Window Size↩︎

As shown in Fig. 4, LEFT shows stable performance under different window sizes, with dataset-dependent optima. PSM changes only slightly and peaks at 192, MSL favors shorter windows, while SMAP improves up to 192 but drops at 224. These results suggest that LEFT is robust within a practical window-size range, though moderate dataset-specific tuning remains useful.

4.6 Visualization↩︎

Fig. 5 presents qualitative results on PSM, MSL, and SMAP. In these examples, CrossAD assigns elevated scores to many normal timestamps, leading to frequent off-interval alarms and more false positives. LEFT produces more concentrated predictions and aligns more closely with the annotated anomalous intervals on PSM and SMAP, although false alarms still remain in some cases.

5 Conclusion↩︎

We propose LEFT, a unified tri-view framework for unsupervised TSAD. LEFT constructs time tokens, frequency tokens from a differentiable STFT, and multi-scale structural tokens from a Nyquist-constrained learnable filterbank whose residual mask covers the spectrum above the learned structural bands. LEFT uses lightweight token interactions to fuse the three views and expose cross-view inconsistency. It also applies cross-scale reconstruction consistency and time–frequency cycle consistency, which makes the inconsistency signal more reliable. Experiments on real-world benchmarks show that LEFT improves detection quality and efficiency. Compared with strong baselines, LEFT improves VUS-ROC and VUS-PR by about 3% and 6%, respectively, while reducing FLOPs by over 80%, speeding up training by about \(8\times\). Future work will study when cross-view inconsistency is most informative and will improve interpretability by linking detected anomalies to specific time–frequency or cross-scale conflicts.

6 Acknowledgments↩︎

This work is supported by the National Natural Science Foundation of China (No. 12433011), the Guangdong Basic and Applied Basic Research Foundation (No. 2024A1515011962), the Australian Research Council (Nos. FT210100624, DP260100326, DE230101033, DP240101814, LP230200892, and LP240200546).

7 Appendices↩︎

7.1 Ablation Study Details↩︎

This appendix reports the ablation results in Table 7. LEFT achieves the best performance on each dataset and on average, reaching 0.7518 in VUS-ROC and 0.3590 in VUS-PR. Among single-component variants, cycle consistency brings the largest gain: the cycle-only setting in Row 3 raises the average to 0.6880 / 0.3320, while the other settings stay near the low baseline range. Row 13 confirms its importance, as removing cycle consistency with the other components retained sharply drops performance to 0.5422 / 0.2411. With cycle consistency, adding tri-view interaction or the learnable filterbank brings further gains, as Rows 9 and 7 show against Row 3, while Rows 11 and 12 remain below the full model. Cross-path consistency contributes beyond the cycle constraint, since removing it in Row 14 reduces the average from 0.7518 / 0.3590 to 0.7266 / 0.3416. Overall, time-frequency agreement is central to reliable learning, while the other components provide additional gains.

7.2 Fusion Strategy Ablation↩︎

Table 8 compares fusion strategies on PSM, MSL, and SMAP using VUS-ROC, VUS-PR, and measured runtime. The proposed strategy gives the most consistent detection quality, achieving the best VUS-PR on PSM and the best VUS-ROC and VUS-PR on MSL and SMAP. Although all-pairs fusion \(M \leftrightarrow F \leftrightarrow T\) obtains the highest VUS-ROC on PSM, it has higher compute cost and drops markedly on SMAP, indicating that denser interaction does not provide stable gains. The \(T \leftrightarrow F\) variant remains competitive on PSM and MSL and clearly outperforms \(M \leftrightarrow F\) and \(M \leftrightarrow T\) on SMAP, suggesting the importance of direct time-frequency exchange. In contrast, strategies mixing \(M\) with only one branch show weaker ranking quality, despite runtimes similar to the proposed strategy.

7.3 Fusion Hyper-parameters Sensitivity↩︎

Figs. 6 (a.1, a.2, b.1, b.2) show that increasing fusion depth or width brings no clear gain, and several settings slightly reduce VUS-ROC or VUS-PR, consistent with the observation that stronger fusion is not necessarily beneficial. Figs. 6 (a.3, a.4, b.3, b.4) further show that performance depends on the joint setting of cycle consistency and cross-path consistency. Higher scores appear when cross-path consistency remains low and cycle consistency is moderate, while large cross-path consistency does not yield reliable improvements. Figs. 6 (a.5, a.6, b.5, b.6) show that VUS-ROC and VUS-PR vary with the joint choice of \(\alpha_{\mathrm{cyc}}\) and \(\alpha_{\mathrm{ms}}\). The trend is not monotonic along either axis, since increasing one weight does not reliably improve the score when the other is fixed. Higher values concentrate in a middle region of the grid, where \(\alpha_{\mathrm{cyc}}\) is moderate to large and \(\alpha_{\mathrm{ms}}\) stays below its maximum, while several boundary settings yield lower scores. This pattern appears in both metrics, which supports the scoring design that balances cycle-based disagreement and multi-scale residual evidence rather than letting one term dominate.

7.4 Sensitivity of Downsampling Factors↩︎

Fig. 7 shows that downsampling factors \(\{r_k\}_{k=1}^K\) affect performance. The VUS-ROC and VUS-PR results in Table 2 use fixed factors, such as (16, 8, 4) for SMAP, which may not be optimal. A more thorough hyper-parameter search may further improve performance.

7.5 Effect of JS vs. Wasserstein↩︎

In LEFT, JS divergence serves as a prototype calibration criterion for reliable EMA updates. To examine this choice, we replace the JS discrepancy with a Wasserstein-based variant (WD), keeping the other components unchanged. As shown in Table 9, WD does not improve performance on PSM, MSL, or SMAP.

7.6 Limitations↩︎

This work develops a unified tri-view framework for unsupervised TSAD and shows competitive performance with favorable efficiency on the evaluated benchmarks. Nevertheless, this study is validated on a limited set of datasets and has not been tested on broader benchmarks such as UCR [47] or in practical deployment settings. In addition, the method may still produce false alarms in some cases, which remains a direction for future work.

7.7 Details of the Proof↩︎

Proof of Lemma 1↩︎

Proof of Lemma 2↩︎

Proof of Lemma 3↩︎

References↩︎