1 Introduction↩︎

Periodic structure in time series data holds significant importance in monitoring individuals’ behaviors and health [1], [2]. A wide range of tasks related to time series, including step counting [3] and heart rate monitoring [4], can reveal crucial information about one’s health situation. Moreover, on a daily basis, people generate substantial amounts of time series data from smartphones or wearable devices [5], which can be used to provide insights into their behaviors and health [6], [7]. However, labeling these huge amounts of temporal data is challenging, expensive, and resource-intensive. For instance, if the desired task is to monitor the respiration or heart rate, collecting gold-standard synchronized signals for obtaining ground truth information from a medical sensor or video is not only time-consuming but also expensive [8], [9]. Moreover, obtaining ground truth is not feasible in daily life, i.e., non-controlled environments outside of the lab, due to the potential discomfort it may impose on individuals [10], which can be caused by privacy concerns, such as the use of video recording [11], or the use of bulky electrodes with gels, as in the case for medical sensors [12].

The self-supervised learning (SSL) paradigm provides a solution to overcome this problem by exploiting unlabelled data to formulate pretext tasks such as predicting the rotation of images [13], or contexts [14] to learn the invariant representations of samples [15] to the applied transformation. The effectiveness of representations on downstream tasks directly depends on the inter-sample semantic similarity relations [16], [17] that are created through tailored data augmentation techniques to preserve semantics depending on the task [18], [19] and input characteristics [20]. However, even when the downstream task is known, creating samples that keep the task-relevant information intact is more challenging in noisy quasi-periodic time series due to the complexity of the dynamical data generation mechanisms [21], where strong augmentations can cause model collapse [15] while weak augmentations can hinder model convergence [18].

Considering these limitations, in this work, we propose a novel method that regularizes the parametric learners to detect periodic components in time series data by punishing the randomness in a sequence while capturing as much information as possible about the periodicity without requiring any specialized data augmentation technique. Since our proposed method bypasses the intricate data augmentation process required for generating positive or negative samples to learn the relations or invariant representations between samples, we prevent the model collapse issue by relating the learned representations with original samples from which they originated, rather than decorrelating the representations with each other or imposing random variance constraints on the batch. Our approach argues that every learned representation should retain a remnant of the sample from which it originates. The contributions can be summarized as follows:

• We propose a novel set of regularizers that consider the generation mechanism of time series data to detect periodic patterns of interest in the presence of noise.

• We introduce a new approach to prevent collapse in unsupervised periodicity detection by relating learned representations with samples from which they originated, resulting in relaxed assumptions for batch statistics.

• We demonstrate that the proposed regularizers help the learners extract useful representations without requiring any supervision or specialized time series augmentations for generating positive/negative samples.

2 Method↩︎

2.1 Notations↩︎

We use lowercase symbols \((x)\) to denote scalar quantities, bold lowercase symbols \((\boldsymbol{\mathrm{x}})\) for vectors, e.g., time series, and capital letters \((X)\) for random variables. The parametric functions is represented as \(f_{\theta}(.)\) where \(\theta\) is the parameters. The discrete Fourier transformation is denoted as \(\mathcal{F}(.)\), yielding a complex variable as \(\mathcal{F}(\boldsymbol{\mathbf{x}}) \in \mathbb{C}\). The detailed calculations for each operation are given in the Appendix 7.1.

2.2 Objective↩︎

Given a dataset \(\mathcal{D} = \{ (\boldsymbol{\mathrm{x}}_i) \}_{i=1}^K\) where each \(\boldsymbol{\mathrm{x}}_i\) consists of real-valued sequences with length L, \((x_1, x_2, \dots, x_L)\), which are sampled uniformly. The objective is to train a learner \(f_{\theta}: X \rightarrow \mathbb{R}^l\) which seeks to learn periodic representations of interest in the data such that when it is evaluated on another set \(\mathcal{D}_l = \{ (\boldsymbol{\mathrm{x}}_i, y_i) \}_{i=1}^M\), the representations \(\mathbf{z}_i=f_{\theta}(\mathbf{x}_i)\) will mainly contain desired periodic source, which can be detected i.e., \(\mathop{\mathrm{argmax}}\limits_k |\mathcal{F}(\mathbf{z}_i)_k| = {y_i} \in \mathbb{R}^+\).

Since our objective is to detect the periodic source of interest in the uniformly sampled time series data without using any labels, we proposed a set of regularizers designed to help the learners extract useful representations that capture the desired periodicity. In the following sections, we describe the proposed regularizers and outline their primary objectives. Our first proposed regularizer decreases the randomness of the overall time series sequence while maximizing the periodicity by minimizing the entropy of the spectra as follows.

2.2.1 Maximizing the Periodicity↩︎

The last inequality completes the proof by showing that minimizing the spectral entropy of sequential data is equivalent to maximizing its periodicity. A detailed proof, with the bandlimited discrete case, can be found in the Appendix 7. We also provide an intuitive demonstration of samples from data generation processes and corresponding spectra in Figure 1 where it can be observed that the periodic data has lower entropy compared to the sample from Gaussian distribution.

Although this regularizer can induce periodicity in sequences, it will cause the model to collapse to a single point and produce a constant output. The existing solutions for preventing collapse generally incorporate the notion of class balance across the dataset [22] or batch [23], [24]. Such approaches, however, are overly optimistic since during SSL training, where there is no label information, this regularizer may punish the models even for learning correct representations. Alternatively, the models can learn task-unrelated information to introduce diversity in representations. Therefore, to prevent the degenerate solution, we have incorporated a secondary regularizer that ensures the extracted representations retain information from the original samples.

2.2.2 Preventing the Collapse↩︎

We provide an intuitive illustration of the proposition with a sample in Figure 2 and the detailed proof in Appendix 7.

The proposition 2 suggests that degenerate solutions can be prevented if the learner seeks to minimize the relative entropy between the spectra of samples and extracted representations. In other words, this proposed regularization technique helps the learner to generate diverse representations according to the input distribution, preventing the collapse where inputs are mapped to the same vector, while avoiding the imposition of random variances within a batch.

The usage of these two regularizers together also eliminates the requirement for generating positive or negative samples to construct embedding space with specialized augmentations which is known to be particularly challenging for non-stationary time series data. Moreover, most of the unsupervised learning methods prevent model collapse by repelling the negative pairs which are randomly chosen from the batch [19], selected from memory banks according to the learned relevance [25], or enforcing distinct embeddings within a batch [23]. However, since there is no label information during training, these methods can repel/decorrelate representations of samples from the same category which can lead to suboptimal embedding space. Therefore, to mitigate this problem, we take a different approach and govern the representation space through two opposing regularizers, seeking to find the periodicity of interest in the noisy time sequence data without causing significant distortions to the features.

When we calculate these two loss functions, we specify a bandlimited frequency range of interest \(f^*\) instead of considering the entire continuous spectrum while minimizing the overall power in frequencies outside of this defined range \(f^{'}\). Minimizing frequencies outside of the specified range, essentially applying a bandpass filter, is a common approach in the literature [24], [26]. We observe marginal improvements with this addition while the improvement is diminished with better-designed filters. The overall loss for training models is given in Equation 1 . \[\begin{gather} \label{eq:overall95loss} \mathcal{L} = - \sum_{f^*}^{} S(\omega) \log S(\omega) + \sum_{f^*}^{} S_X(\omega) \log \frac{S_X(\omega)}{S(\omega)} \\ + \sum_{f^{'}}^{} S(\omega),\omega = 2 \pi f\text{and}f^{'} = \mathbb{U} \setminus f^* \end{gather}\tag{1}\] The learner which is regularized by the proposed loss function is similar to an adaptive sinusoidal selector based on the Fourier transform. In uniformly sampled time sequences, periodicity is typically identified by the highest value in the Fourier transform [27]. However, this approach fails when there exist multiple periodicities within the frequency range of interest, or when the desired periodic pattern is obscured by noise. The proposed training regularizers help the model learn the desired pattern in the time domain such that it can detect the signal even under the noise. The characteristic of this loss is provided by the minimization of the spectral entropy. This is evident when considering that, if the spectral entropy term is excluded from the overall loss function, the optimization converges to a bandpass-filtered Fourier transform of the original signal.

3 Experiments↩︎

We compare our method with several techniques, from traditional heuristic-based methods such as autocorrelation and Fourier transform-based approaches, as well as learning-based methods including supervised and self-supervised.

3.1 Datasets↩︎

We conducted experiments on eight datasets across three tasks, including heart rate (HR) estimation from electrocardiogram (ECG) and photoplethysmography (PPG) signals, step counting using inertial measurements (IMUs), and respiratory rate (breathing) estimation from PPG signals. We provided short descriptions of each dataset below, and further detailed information can be found in Appendix 8.

3.1.0.1 Heart rate

For ECG, we used PTB-XL [28] to investigate performance in a dataset with cardiovascular diseases, WESAD [29], and DaLiA [30] to study the real-life settings, where the signals are corrupted with different level of noise as they are collected during activities like cycling. We used the IEEE SPC12 with 22 [31], and DaLiA [30] for PPG-based HR prediction.

3.1.0.2 Respiratory rate

We used CapnoBase [32] with two different window sizes and BIDMC [33] for estimating respiratory rate (RR) from the PPG, where both datasets provide true RR using gold standard respiratory signals and capnography.

3.1.0.3 Step counting

We used the Clemson dataset [34], which is proposed to improve pedometer evaluation. We conducted experiments using inertial measurements from the wrist with both regular and semi-regular walking settings where labels are available through videos.

3.2 Baselines↩︎

3.2.0.1 Self-supervised

We compared our method with self-supervised learning techniques within the linear evaluation setting [19], including SimCLR [19], NNCLR [35], BYOL [36], TS-TCC [37], TS2Vec [38], VICReg [23], Barlow Twins [39], and SimPer [8].

3.2.0.2 No supervision

We conducted experiments with a learning-based method that requires no labels for fine-tuning. Specifically, we compared our work with SiNC [24] which is a non-contrastive unsupervised learning framework for periodicity detection without using any labels while imposing random variance constraints on the batch.

3.2.0.3 Supervised

We included two supervised models: a fully convolutional network and a convolutional network with LSTM [40], both commonly used in time series tasks previously [41].

3.2.0.4 Traditional

We also included traditional heuristic-based methods, i.e., the Fourier transformation and autocorrelation function, where both are widely used before [42] to detect periodicity in time series. Additionally, we compared our method with RobustPeriod [43] which is a rule-based periodicity detection approach by integrating information from both time and frequency domains. More details for each baseline are given in Appendix 9.

3.3 Implementation↩︎

3.3.0.1 Self-supervised

We follow the same implementation setup with previous works [20], [41] for self-supervised learning in time series. Specifically, we use a combination of convolutional with LSTM-based network, which shows superior performance in many time series tasks [41], [44], as backbones for the encoder \(f_{\theta}(.)\) where the projector is two fully-connected layers. During pre-training, we use InfoNCE (for contrastive learning-based methods) as the loss function, which is optimized using Adam [45] with a learning rate of \(0.003\). We train the models with a batch size of 256 for 120 epochs and decay the learning rate using the cosine decay. After pre-training, we train a single linear layer classifier on features extracted from the frozen pre-trained network, i.e., linear probing. Although we aim to enhance performance by experimenting with various architectures and hyperparameters, we consistently observed the best average performance across datasets with the specified implementation details, similar to previous studies. Detailed hyperparameters and augmentations for each SSL technique can be found in Appendix 9.3.

3.3.0.2 No supervision

Unlike the self-supervised methods, we employ the backbone as a representation learner without having a non-linear projection layer. In our initial experiments, we observed that LSTM or temporal dilations cause loss of frequency information similar to [24], we, therefore, use U-Net [46] as the encoder architecture with single dimensional layers for time series. We use the proposed loss function in Equation 1 , which is optimized using Adam with a learning rate of \(0.001\). We train the model with a batch size of 512 and reduce the learning rate by half when the training loss stops decreasing for 15 consecutive epochs. We used the same implementation for all tasks and datasets while giving the same importance to each loss term in Equation 1 without tuning, i.e., no specific task or dataset optimization is performed.

For both learning paradigms, model selection is performed on the training sets with the lowest loss value similar to [20], [26]. The reported results are mean and standard deviation values across three independent runs with different random seeds.

3.3.0.3 Supervised

We follow the same implementation as previous works [20], [41] for supervised learning in time series. Specifically, we use two different network architectures: the first is a combination of convolutional and LSTM-based networks [41], [44], similar to the self-supervised implementation of the encoder architecture, and the second is a fully convolutional neural network that includes a final linear layer in the end instead of a projection block. For the training of both architectures, we use categorical cross entropy as the loss function, which is optimized using Adam with a learning rate of \(0.001\). We train the models with a batch size of 64 for 120 epochs. The model selection is performed on the validation sets with the lowest loss, where the validation set is created by randomly splitting 10% of remaining data after excluding the test subjects. Details of the architectures and parameter comparisons are provided in Appendix 10.1. The performance of various architectures with different sizes is discussed in Appendix 11.3.

3.3.0.4 Traditional

We also compare our method with the rule-based periodicity detection methods in addition to the learning-based techniques. In our evaluation, we consider the frequency domain approach, which detects underlying periodic patterns using Fourier transforms, and the time domain, where signals are correlated with themselves through autocorrelation function (ACF) [47]. Additionally, we compare our method with an approach that combines the time and frequency domains using the periodogram and the ACF, integrating the Fisher method for periodicity detection [43]. Since these methods are not learning-based, they fail to detect periodicities when evaluated on datasets containing periodic noise in the same band, limiting generalization across various conditions. Since traditional methods lack randomness, we conducted the experiments only once and reported the results, i.e., no standard deviation is calculated for rule-based methods.

4 Results and Discussion↩︎

We present the main results of our proposed approach compared to state-of-the-art methods across the three time series tasks in eight datasets in Tables ¿tbl:tab:performance95ecg? to [tbl:tab:performance95ppg]. Overall, the proposed set of regularizers has demonstrated substantial performance improvements, reaching up to 45–50% in some tasks, compared to other supervised and self-supervised learning methods with rule-based techniques. More importantly, our method outperforms other supervised learning-based techniques, which require label information, while eliminating the necessity for implementing specialized data augmentation methods to learn periodic representations from data.

Tables [tbl:tab:performance95resp] and [tbl:tab:performance95ppg] show that the rule-based methods fail to detect periodicity when applied to datasets containing multiple periodic components with noise. For example, autocorrelation, one of the most common methods for periodicity detection [48], [49], performs poorly in various datasets that contain multiple periodicities with a high noise where the periodicity of interest is hidden.

Another interesting observation from the results is that the SimPer, designed for extracting representations related to periodic information from data similar to this work, performs worse when applied to noisy time series compared to other self-supervised techniques, despite sharing the same downstream task. This might be attributed to the fact that SimPer designed specialized augmentations that alter the periodicity of the sample within and contrast through that for representation learning. However, when dealing with time series containing multiple periodicities under noisy conditions, the specific augmentation can lead to spectral overlapping, resulting in suboptimal performance. Moreover, since there is no label information during self-supervised training, this method can also learn the features related to periodic noise, which is impossible to filter out as it lies on the same frequency band with the interested periodicity.

From the results, we can see that our proposed method significantly outperforms SiNC and VICReg, which impose a hinge loss on the variance over a batch of representations to enforce diverse outputs to prevent collapse, by a large margin (up to 73.9% with a 10.1% on average in tasks).

width=1,center

These empirical findings suggest that the assumption of having diverse samples in a batch is quite arguable, especially depending on the downstream task, where the pattern of interest can be severely hidden by noise such that the model can learn random features from the samples to have diversity in a batch of representations. The empirical results align with our initial proposition and the motivation behind introducing a novel regularizer term, which prevents the model collapse by ensuring that the extracted representations retain traces from the samples from which they originated rather than relying on random variance constraints in the batch.

width=1,center

width=2,center

Another interesting outcome of the results is when our method is outperformed by supervised and self-supervised methods while having a minor performance gap with unsupervised techniques, which is the case for DaLiA\(_{PPG}\) and Clemson semi-regular. We conjecture that the performance difference between supervised and unsupervised methods in these datasets is due to the average level of noise in the samples, making the periodic pattern of interest completely hidden and hard to extract without supervision. Similarly, in the case of semi-regular walking, the periodic patterns in the training decrease significantly compared to regular, leading to poor performance by unsupervised methods due to their inability to effectively learn desired patterns. Since our approach learns the function \(f_{\theta}\) by minimizing the expectation of the proposed loss, which is known as the empirical risk minimization (ERM), the model converges to a point where the risk is minimal in the training set. When the process is unsupervised, the model might inadvertently learn the periodic noise or random features from the data, which leads to failures during the evaluation. Contrarily, in self-supervised and supervised scenarios, models may learn dataset statistics with label information, such as mean values. The model can then use this during evaluation, contributing to improved performance for imbalanced datasets [50].

These results suggest that the methods should be evaluated with several tasks in multiple datasets to investigate if they generalize and perform well under different characteristics and conditions, i.e., noise types and levels, narrow/wide bandwidth for the periodicity of interest. Otherwise, the hyperparameters (specific augmentations with thresholds) can be optimized for a single task while performing poorly for the rest. It is important to emphasize that our implementation (the model architecture and weights for loss terms) remained consistent across all experiments, ensuring a standardized evaluation. We conduct detailed ablation experiments to investigate further the impact of components of our method on the performance across the datasets.

4.1 Ablation Study↩︎

In this section, we present a comprehensive examination of our proposed method and the effect of its components on performance of the models for detecting the periodicity of interest. Mainly, we investigate the effect of each proposed loss term on performance across datasets while establishing connections between dataset characteristics in relation to the average noise levels and learning paradigms. Therefore, first, we extend our investigation for the case when the supervised algorithms significantly outperform our approach. Figure 3 illustrates the distribution of relative entropies computed between the spectra of original samples and optimal waveforms using a continuous probability density curve [51], where the optimal samples only contain a pure sinusoidal at the frequency of the interest.

width=2,center

Figure 3 gives an explanation for the performance variation in supervised and unsupervised learning across datasets by demonstrating and comparing the level to which the pattern of interest is hidden by noise. Heuristic-based approaches typically require the SNR of a sample to exceed a certain threshold to detect the pattern, which can vary depending on the specific method, as these methods are not learning-based. In contrast, our method is a learning-based approach designed to train a model that seeks to learn the patterns of interest using the training set and detect those patterns even when they are hidden by noise or other periodicities in unseen samples. One significant drawback of relying on complete unsupervision for a learning-based method is that if the training set lacks sufficient diverse samples with a relatively high SNR for the model to learn the desired periodic patterns, the model may end up learning noise or random features from samples instead of the desired periodic pattern. We, therefore, believe that the average SNR of a dataset is particularly important during the training of unsupervised and self-supervised learning methods (when there is no label information), as a higher average SNR indicates that relevant patterns, those related to downstream tasks, are distinguishable from noise.

Second, we analyze the contribution and impact of each proposed regularizer on model performance across various datasets under different levels and types of noise. We, therefore, trained the neural network models with the same original implementation settings, i.e., the same architecture and training hyperparameters, using the different combinations of the loss components to analyze their contributions separately.

Specifically, we first train the models without maximizing periodicity and while ensuring the extracted representations retain information to the original samples in the desired frequency band, i.e., \(\mathcal{L}_1 = \mathcal{L}_{ds} + \mathcal{L}_{bw}\), without maximizing the periodicity (w/o Proposition 1, the second rows in Tables [tbl:tab:performance95ppg95ablation] and ¿tbl:tab:performance95ecg95ablation?). Then, we train the models to maximize the periodicity without forcing the extracted representations to retain information from the original samples, i.e., \(\mathcal{L}_2 = \mathcal{L}_{se} + \mathcal{L}_{bw}\), without preventing the collapse (w/o Proposition 2, the third rows in Tables [tbl:tab:performance95ppg95ablation] and ¿tbl:tab:performance95ecg95ablation?). We also add the bandwidth loss, a common practice in the literature, to the loss variants for each case. The experimental results for each dataset are presented in Tables [tbl:tab:performance95ppg95ablation] and ¿tbl:tab:performance95ecg95ablation? for heart rate detection task from PPG and ECG signals, respectively, and in Figure 4 for respiration and step counting tasks.

When the models are trained without maximizing the periodicity in the extracted representations (w/o Proposition 1), the performance of the models is similar to periodicity detection using the Fourier transformation where the strongest harmonic is detected from the frequency spectrum of samples. This behavior is expected since the applied regularizers transform the model into a non-adaptive bandpass filter with cut-off frequencies set according to the task. The combination of the proposed loss terms consistently performs well in all tasks, especially when the models have sufficient examples with a high SNR to learn the underlying periodic patterns of interest in the training set. However, when the training data is noisy, i.e., the majority of the training set includes samples with low SNR, where the desired patterns are completely hidden, the Fourier transformation method with bandpass performs well as can be seen in Table [tbl:tab:performance95ppg95ablation].

The results obtained from the ablation studies also support the previous theoretical propositions and our motivation for presenting a novel regularizer for preventing the degenerate solution, i.e., the model collapse. For example, a closer inspection of these results shows that when we train the models without relating representations with samples from which they originated, i.e., models that are trained only using \(\mathcal{L}_1\) (w/o Proposition 2), the models generally performed the worst among the three variants of the combinations, except for the ECG signals where the noise is less significant compared to other signal types, i.e., the periodic pattern of interest is more obvious. This observation constitutes an empirical evidence for our proposed proposition, where a learner, which is only trained to maximize periodicity in uniformly sampled time series within a specific frequency band, can converge to a degenerate solution or learn the random waveforms in the signal. Notably, this degenerated model even fails to identify obvious and trivial periodic patterns that can be detected by traditional rule based Fourier transformation or autocorrelation methods.

More ablations about the sensitivity of loss terms can be found in Appendix 11.2. Investigations regarding the effect of architectures in supervised learning methods are given in Appendix 11.3. Comparative results regarding the performance of our approach with traditional methods that are specifically tailored for the applications are given in Appendix 11.4. Examples that visually demonstrate the learned representations from samples can be found in Appendix 12. Appendix 13 provides a detailed evaluation and comparison of the computational efficiency of our proposed method.

5 Related Work↩︎

5.0.0.1 Periodicity Detection

Numerous studies have investigated periodicity detection for several time series applications [42], [52]. Detecting the periodicity of physiological signals, such as heart and respiratory rate, plays a pivotal role in understanding individuals’ health conditions [53]. Periodicity detection also finds application in workload forecasting [54], anomaly detection [43], and voice/speech analysis [42]. For instance, it helps identify features like pitch and intonation, which convey essential aspects of human communication, including recognizing and understanding speech [55]. Although current methods are valid in addressing specific problems, a common characteristic is the development of specialized tools tailored to particular issues with optimized hyperparameters, i.e., several thresholds [43], often lacking generalized solutions under noisy data [42]. Moreover, if there are two periodicities in the interested frequency bandwidth (i.e., noise and pattern of interest), these methods will fail to detect the desired one as they lack the adaptability to learn to distinguish between multiple periodic patterns [47]. In contrast, our proposed method, which is adaptive and learns the desired pattern from the data in an unsupervised way, can detect periodicity across various conditions in three time series tasks without requiring labels for training or task-specific data augmentations with parameter optimizations.

5.0.0.2 Unsupervised Learning

Self-supervised learning methods have gained significant attention as they enable the discovery of useful representations from data without labels by designing pretext tasks that change the unsupervised learning problem into a supervised one such as predicting the rotation of images [13], or contexts [14], [56]. Recently, SSL techniques, particularly in the vision domain, have been explored for periodicity detection from the sequence of frames [8], [57]. Most of these methods [8], [26] generate similar and dissimilar (positive/negative) pairs by applying tailored augmentations and learn periodic representations with slight modifications to the InfoNCE loss [58]. However, these approaches do not consider the challenges posed by noisy non-stationary time series data where signals may exhibit multiple periodicities in the interested bandwidth under significant noise [59]. Moreover, the usage of augmentations, in such cases, can be prone to issues like spectrum overlap, i.e., aliasing, resulting in the loss of information [60].

Recent approaches have used regularizers to prevent collapse issue by penalizing the model when it produces similar embeddings within a batch [24] instead of using similar/dissimilar samples as in contrastive learning. For example, VICReg applies the hinge loss on the variance over a batch of predictions to enforce diverse representations/outputs [23]. However, having diverse samples in a batch is an overly optimistic claim, because during self-supervised training, where there is no label information available to the learners, this regularizer may punish the models for learning correct representations. Even worse, the model might learn the noisy patterns to satisfy the arbitrarily set variance constraint in a batch. Therefore, we take a different approach and introduce a new method to prevent the model collapse by ensuring the representations retain information from the original samples. In other words, our approach hypothesizes that each representation should maintain a trace of the sample from which it is derived.

6 Conclusion↩︎

In this paper, we proposed a set of regularizers for detecting desired periodic patterns in time series sequences without requiring any label information or specialized data augmentation techniques. Theoretically, we proved that an optimized learner trained with the proposed loss will detect the periodic pattern in the sequence while preventing degenerate solutions. In contrast to previous methods that rely on optimistic assumptions on batch statistics to prevent the collapse of unsupervised models, we presented a novel approach that guarantees representations to preserve information from the original samples, thus relaxing assumptions on the batch. Empirically, we showed that our method outperforms the prior works, achieving significant performance gains of up to 40–45%, in three real-world tasks that involve diverse noisy conditions. We believe that the methods introduced in this work have the potential to significantly improve learning solutions for a wide variety of time series tasks.

Impact Statement↩︎

This paper aims to advance machine learning by introducing a new method for detecting periodic patterns in noisy time series data without relying on labeled data. Since our method operates in an unsupervised manner, it can leverage vast amounts of unlabeled data commonly generated in everyday life. This characteristic underscores its potential for real-world deployment, tapping into the rich resource of naturally occurring time series data for applications such as health monitoring, behavior analysis, and beyond.

7 Proof↩︎

In this section, we present complete proofs of our theoretical study, starting with notations.

7.1 Notations and Preliminaries↩︎

Fourier transform of a real-valued sample with a finite duration is obtained as in Equation 2 .

\[\label{eq:appendix95fourier} \mathcal{F}(\boldsymbol{\mathbf{x}}) = \sum_{n=-\infty}^{\infty} \boldsymbol{\mathrm{x}}_n e^{-j2\pi k n}\tag{2}\]

The absolute frequency spectra of the samples are calculated as below.

\[S_{X}(w) = \left| \sum_{n}^{} \mathbf{x}_n e^{-j\omega n} \right|^2,\text{where}\omega = 2\pi f\text{and}f \in [0, f_s/2]\]

We applied probability normalization to the obtained spectra for both original samples and the extracted representations in the desired frequency as in below before calculating the entropy and divergence. \[\label{appendix95eq:normalization} \begin{gather} S_X(\omega^*) = \frac{S_X(\omega^*)}{\sum_{\omega^*} S_{X}(\omega^*)},\text{where}\omega^* \in [0, 2 \pi f_s/2] \\[2pt] H(S_{X}) = \sum_{\omega} S_X(\omega^*) \log \frac{S_X(\omega^*)}{S_X(\omega^*)} \end{gather}\tag{3}\]

The autocorrelation of a real-valued signal with time shift \(\tau\) is calculated as in Equation 4 , where the denominator is the sample variance of the time series in that segment.

\[\label{appendix:eq95autocorrelation} R_{XX}(\tau) = \frac{\sum_{n=1}^{N-\tau} (x_n - \mu_x)(x_{n+\tau} - \mu_x)}{\sum_{n=1}^{N} (x_n - \mu_x)^2}\tag{4}\]

We also used the Wiener-Khintchine [61], [62] theorem for calculating the spectral density of the sample \(\mathbf{x}\) from the autocorrelation as in Equation 5 .

\[\label{appendix:eq95wiener95khinchin} S(\omega) = \sum_{\tau}^{} R_{XX}(\tau) e^{-j\omega \tau}\tag{5}\]

Similarly, as the time series samples are absolutely summable in our case, the autocorrelation can be obtained as in below.

\[R_{XX}(\tau) = \frac{1}{2\pi} \int_{-\pi}^{\pi}S(\omega)e^{j \tau \omega}d(\omega)\]

Since the spectral density is periodic in the frequency domain, the summation is performed over one period.

7.2 Proof for Proposition 1↩︎

The last inequality concludes the proof by showing that minimizing the spectral entropy of uniformly sampled time series data is equivalent to maximizing its periodicity.

7.3 Proof for Proposition 2↩︎

The proof suggests that degenerate solutions can be prevented if the learner seeks to minimize the relative entropy between the spectra of samples and extracted representations.

8 Datasets↩︎

In this section, we give details about the datasets that are used during our experiments.

8.1 Heart rate↩︎

8.1.1 ECG based↩︎

8.1.1.1 PTB-XL

The PTB-XL ECG is a large dataset of 21799 clinical 12-lead ECGs, where lead-II was used during experiments, from 18869 patients of 10-second length segments. The dataset itself provides recommended splits into training and test sets. We, therefore, follow the exact recommendation. ten percent of the training training set is used as fine-tuning for self-supervised learning techniques.

8.1.1.2 DaLia\(_{ECG}\)

PPG dataset for motion compensation and heart rate estimation in Daily Life Activities (DaLia) was recorded from 15 subjects (8 females, 7 males, mean age of \(30.6\)), where each recording was approximately two hours long. The signals were recorded while subjects went through different daily life activities, for instance sitting, walking, driving, cycling, working, and so on. ECG signals were recorded at a sampling rate of 700 Hz. We follow leave-one-out-cross-validation for each subject. For the self-supervised setting, we used the first five subjects for fine-tuning.

8.1.1.3 WESAD

The multimodal WESAD data set includes physiological and mobility data from wrist-worn (Empatica E4) and chest-worn (RespiBAN) devices. Data were acquired from 15 subjects and contain multiple features, pulse rate, ECG, and body temperature, as extracted from a wrist-worn device, and blood volume pulse (BVP) and respiration extracted by the chest-worn devices [29]. We evaluate the dataset using leave-one-subject-out.

ECG datasets are standardized as follows. Initially, a fourth-order Butterworth bandpass filter with a frequency range of 0.7–40 Hz is applied to the signals. In the case of DaLia and WESAD, we use an 8-second sliding window with 2-second shifts for segmentation. Differently, for PTB-XL, the ECG segments are pre-provided. After segmentation, we calculated the square of the first differentiation before feeding samples to the models, which helps to emphasize beats [64]. Lastly, the signals are normalized to zero to one range.

8.1.2 PPG based↩︎

8.1.2.1 IEEE SPC

This competition provided a training dataset of 12 subjects (SPC12) and a test dataset of 10 subjects [30]. The IEEE SPC dataset overall has 22 recordings of 22 subjects, ages ranging from 18 to 58 performing three different activities [65]. Each recording has sampled data from three accelerometer signals and two PPG signals along with a sampling frequency of 125 Hz. All these recordings were recorded from the wearable device placed on the wrist of each individual. All recordings were captured with a 2-channel pulse oximeter with green LEDs, a tri-axial accelerometer, and a chest ECG for the ground-truth HR estimation. During our experiments, we used PPG channels. We use leave-one-out-cross-validation for the SPC12 and SPC22 as source domains similar to the previous setup while the last six subjects of SPC22 are used for source domains to prevent overlapping subjects with SPC12, similar to [20].

8.1.2.2 DaLia\(_{PPG}\)

PPG signals from the DaLia dataset were recorded at a sampling rate of 64 Hz. The first five subjects are used as target domains with leave-one-out-cross-validation, following the [20].

All PPG datasets are standardized as follows. Initially, a fourth-order Butterworth bandpass filter with a frequency range of 0.5–4 Hz is applied to PPG signals. Subsequently, a sliding window of 8 seconds with 2-second shifts is employed for segmentation, followed by z-score normalization of each segment. Lastly, the signal is resampled to a frequency of 25 Hz for each segment.

8.2 Respiratory rate↩︎

8.2.0.1 CapnoBase

The CapnoBase dataset includes recordings of ECG and PPG along with capnometry signals from 42 subjects (13 adults, 29 children, and neonates). The dataset also includes the inhaled and exhaled carbon dioxide (CO2) signal labeled by the research assistance [32] and used as the reference breathing rate. The dataset is split into 10 folds where each fold contains four subjects. The last five fold is used as the test set for all baselines. For the self-supervised learning, we used seven fold for pre-training and ten percent of the pre-training for fine-tuning.

8.2.0.2 BIDMC

The BIDMC dataset consists of ECG, pulse oximetry (PPG), and impedance pneumography respiratory signals. The data was acquired by randomly selecting critically ill patients undergoing hospital care at the Beth Israel Deaconess Medical Centre (Boston, MA, USA). Two trained annotators manually annotated individual breaths in each recording using the respiratory impedance signal. The data set contains 8-minute recordings of ECG, PPG, and impedance pneumography signals from 52 adult patients aged from 19 to older than 90, including 32 females [33]. For each set of annotations, the RR value was determined based on the average time between consecutive breaths within a given window; only those windows of data for which the agreement between both estimates was within 2 breaths per minute were retained. We used 10-fold cross-validation for the evaluation where each fold contains 5 subjects. For the self-supervised learning, we used seven fold for pre-training and ten percent of the pre-training for fine-tuning.

8.3 Step counting↩︎

8.3.0.1 Clemson

The Clemson dataset has 30 participants, including 15 males and 15 females. Each participant wore three Shimmer3 sensors. We use the data from the device which is positioned on the non-dominant wrist. The inertial measurement data is recorded at 15 Hz. The accelerometers were set to record from -2 to 2 gravities. We computed the overall magnitude of the accelerometer and used it as input to the models. We used the regular gait experiment, where the participant was instructed to walk two laps around a designated path at their normal walking pace. In the semi-regular gait experiment, participants were instructed to perform a scavenger hunt, locating four objects in four different rooms throughout a building [34]. We used 10-fold cross-validation for the evaluation where each fold contains 3 subjects. For the self-supervised learning, we used seven fold for pre-training and two fold for fine-tuning.

When we split the datasets for training and testing folds, we ensure that each person’s recordings appear in only one set. We followed this procedure across all datasets to ensure that the trained model had not seen the testing data during training.

8.4 Metrics↩︎

We used the common evaluation metrics in the literature for each task. Specifically, we used mean absolute error (MAE), root mean square error (RMSE), and Pearson correlation coefficient for heart rate prediction [24], and we used the mean absolute percentage error (MAPE) for step counting [66], [67].

Here, we explain how to calculate each metric for different time series tasks. The MAE, RMSE, and correlation coefficient (\(\rho\)) are calculated using the following equations:

\[\label{eq:MAE} \text{MAE} = \frac{1}{n} \sum_{i=1}^{n} |\hat{y}_i - y_i|\tag{8}\]

\[\label{eq:RMSE} \text{RMSE} = \sqrt{\frac{\sum_{n=1}^{n} (\hat{y}_i - y_i)^2}{K}}\tag{9}\]

\[\label{eq:corr} \rho = \frac{\sum_{i=1}^n (\hat{y}_i - \mu_{\hat{y}})(y_i - \mu_y)}{\sqrt{\sum_{i=1}^n (\hat{y}_i - \mu_{\hat{y}})^2}\sqrt{\sum_{i=1}^n (y_i - \mu_y)^2}}\tag{10}\]

\[\label{eq:MAPE} \text{MAPE} = \frac{100}{n} \sum_{i=1}^{n} \left|\frac{y_i - \hat{y}_i}{y_i}\right|\%\tag{11}\]

where n represents the total number of segments. The variables \(\hat{y}_i\) and \(y_i\) denote the output of the model and ground truth values in beats (respiration)-per-minute or number of steps for the \(n^{th}\) segment, respectively. We reported the percentage for the Pearson correlation coefficient to save precision in the tables.

9 Baselines↩︎

9.1 Traditional-Heuristic based methods↩︎

9.1.0.1 Fourier transform

We calculated the Fourier transform of signals and found the maximum amplitude sinusoidal from it to detect the periodicity. The length of transformation is set to 2048 for ECG signals and 512 for the rest. The frequency range of interest is defined according to common physiological limits for each task: [5, 40] rpm for respiration rate, [40, 140] for step counts per minute, and [30, 210] bpm for heart rate. We increased the length of the Fourier transform for ECG signals as their sampling frequency is approximately four times higher than the other signals. During the loss calculation of our proposed method, we use the same hyperparameters, frequency range of interests with length, of Fourier transformation.

9.1.0.2 Autocorrelation

Similar to the Fourier-based method, we calculate the autocorrelation, as in Equation 12 , of the signals and find the maximum value of \(y_k\) in the range of frequency of interest.

\[\label{eq:autocorrelation} y_k = \frac{\sum_{t=k+1}^{n}(x_t - \mu_x)(x_{t-k} - \mu_x)}{\sum_{t=1}^{n}(x_t - \mu_x)^2}\tag{12}\]

9.1.0.3 RobustPeriod

RobustPeriod adopts maximal overlap discrete wavelet transform (MODWT) to decompose the input time series into multiple time series at different levels, ranks by wavelet variance, and then performs Huber periodogram with autocorrelation. We used Daubechies 10 wavelet with 4 levels, the lambda in the Hodrick–Prescott (HP) filter is set to 1e6, the Huber function hyperparameter is set to 2, and the M-Periodogram is set to 1.345. Although we searched for the optimum hyperparameters, the search space was limited due to the significant time consumption arising from the wavelet computation and optimization for each sample, given the numerous hyperparameters involved in the technique. Since this method includes autocorrelation to detect periodicity at each scale after the wavelet decomposition, the performance decreases severely when the desired period of interest is hidden by other periodic patterns in the same frequency band.

9.2 Supervised↩︎

9.2.0.1 DCL

We used a similar implementation of DeepConvLSTM architecture which is a 4-layer convolutional neural network with a kernel size of five, and ReLU activation after each layer, followed by a Dropout [68] and a two-layer LSTM with a hidden size being 128. We chose this architecture for the supervised and self-supervised learning paradigms as it was widely used before in the literature [20], [41], especially for detecting the periodicities in the supervised paradigm from signals [44].

9.2.0.2 CNN

We also implemented a fully convolutional neural network with a 3-layer followed by ReLU activation and MaxPooling after each convolutional layer. Dropout with 0.5 is applied after the first convolutional layer. We set the kernel and padding size to 8 and 4, respectively for each convolutional layer. The number of kernels for each convolutional layer is set to 32 for the first one and 64 for the rest.

We also performed a grid search for the hyperparameters of the architectures where we mainly investigated the number of convolutional layers with the kernel size. However, we did not observe a performance improvement with the increased number of parameters for the architectures as shown with the additional results in Appendix 11.3.

9.3 Self-Supervised↩︎

9.3.0.1 NNCLR

We follow a similar setup to SimCLR by applying two separate data augmentations, then we use nearest neighbors in the learned representation space as the positive in contrastive losses [35]. The maximum size of the support set equals 1024.

9.3.0.2 BYOL

For the BYOL implementation, the exponential moving average parameter is set to 0.996 where the projector size is set to 128. We set the learning rate to 0.03 similar to other SSL techniques. Following the original implementation, we use a weight decay parameter of \(1.5e-6\).

9.3.0.3 TS-TCC

We follow the same architecture implementation with the losses, i.e., contextual and temporal contrasting. TS-TCC proposed applying two separate augmentations, one augmentation is weak (jitter-and-scale) and the other is strong (permutation-and-jitter). The authors also change the strength of the permutation window from dataset to dataset. In our experiments, we first used the original augmentations for each time series task, however, we observed performance decreases depending on the signal type. We, therefore, applied the specific augmentations for each time series, where we observed a general performance improvement in other SSL techniques as well.

9.3.0.4 TS2Vec

TS2Vec [38] is a SSL method specifically designed for time series based on contrastive (instance and temporal wise) learning in a hierarchical way over augmented context views where the context is generated by applying timestamp masking and random cropping on the input time series. Following the original framework, we use a dilated CNN architecture with a depth of 10 and hidden size of 64, which has a similar number of parameters with the architectures used by other SSL methods. The batch size is set to 256, and the number of epochs to 120, consistent with other SSL techniques.

9.3.0.5 VICReg

We follow the original implementation and set the coefficients for each loss term to 25 (\(\lambda\)), 25 (\(\mu\)), and 1 (\(\nu\)), corresponding to the invariance, variance, and covariance terms, respectively. Although we conducted a search for these loss term values, no performance enhancements were detected across the tasks.

\[\ell = \lambda \left[ s(z,z^{\prime})\right] + \mu \left[ v(z) + v(z^\prime) \right] + \nu \left[ c(z) + c(z^{\prime}) \right],\]

where \(s\) is the mean-squared Euclidean distance, \(v\) is a hinge function on the standard deviation of the embeddings along the batch dimension, \(c\) is the covariance regularization term as the sum of the squared off-diagonal coefficients

9.3.0.6 Barlow Twins

Barlow Twins [39], [69] presents an objective function that naturally avoids collapse for SSL by measuring the cross-correlation matrix between the outputs of two identical networks fed with augmented versions of a sample, and making it as close to the identity matrix as possible. This causes the embedding vectors of augmented versions of a sample to be similar, while minimizing the redundancy between the components of these vectors. Following the original implementation, we applied batch normalization to the extracted embeddings and set the hyperparameter \(\lambda\) coefficient (in Equation 13 ) to 0.005.

\[\label{eq:barlow95twins} \mathcal{L} = \sum_{i} (1 - C_{ii})^2 + \lambda \sum_{i} \sum_{j \neq i} C_{ij}^2,\tag{13}\]

where \(C\) is the cross-correlation matrix computed between the two sets of normalized embeddings.

9.3.0.7 SimPer

SimPer is a simple contrastive self-supervised technique for learning periodic information in data. To exploit the periodic inductive bias, SimPer introduces customized augmentations and feature similarity measures. We follow the same augmentation in the original paper, and we downsample or interpolate the signals, which the authors call speed change. The range and the number of periodicity variant frequency augmentation are searched over [0.5, 3] and [3, 20], respectively, as in the original paper. We use the MXCorr for the similarity metric. Although the objective of the SimPer is the same as our work, the performance is extremely poor in all tasks and datasets compared to our proposed method even though the SimPer requires labels for training.

9.4 No supervision↩︎

9.4.0.1 SiNC

The three loss terms (bandwidth, sparsity, variance) with equal weights are implemented for signal estimation via the non-contrastive unsupervised learning (SiNC) technique. However, we adapted these limits individually for each task, following the specific frequency boundaries outlined in the Fourier transform.

10 Implementation Details↩︎

Here, we have provided the details of the searched architectures, hyperparameters, and data augmentations to push the performance of the supervised and self-supervised techniques. To begin, we considered the model topologies that were used in previous works [20], [41], [44] and expanded upon them. We also evaluate the 1D Res-Net [70] implementation; however, we observe no significant performance improvement across datasets. This aligns with findings from previous works indicating that complex models may perform worse for time series data [71].

10.1 Architectures↩︎

Here, we present the details of architectures that are investigated for the performance of learning-based algorithms. Mainly, we consider the baselines in the literature and increase the number of parameters approximately four times to observe if the performance increases. Batch normalization [72] is applied after each convolutional block, except the first version of DCL, similar to [41]. ReLU activation is employed following batch normalization, in line with [73], with the exception of the final activation function in the U-Net, which is set to hyperbolic tangent. The number of parameters for each model is calculated for an input with 200 dimensions, i.e., a segment of the PPG signal.

3pt

3pt

3pt

10.2 Augmentations↩︎

In this section, we give details about the data augmentations that are applied to the time series tasks. During our experiments, we searched for the best traditional augmentation technique for a given task. We searched over common time series augmentation methods in literature (Table [tbl:tab:appendix95augmentations]), and applied them with self-supervised learning baselines.

Specifically, we applied scaling for ECG and IMU signals to detect heart rate and step counting, permutation with noise for HR detection from PPG signals. scaling and permutation with noise for respiratory rate from PPG. We also observed that while permutation enhances the performance of PPG signals, it significantly diminishes the performance of ECG signals.

width=0.85,center

SimPer applies interpolation and decimation to change the periodicity of the sample as augmentations. We limit the speed change—the term authors used for resampling the short and long samples—range to be within [0.5, 2], ensuring the augmented sequence is longer or shorter than a fixed length in the time dimension, following the same implementation in the paper.

11 Additonal Results↩︎

11.1 BIDMC dataset↩︎

The BIDMC dataset exhibits a significant imbalance [33], leading self-supervised and supervised learning techniques to essentially leverage the training data statistics, particularly the mean output, during inference. We, therefore, only report the heuristic and unsupervised learning-based methods in Table [tbl:appendix95tab:performance95resp95bidmc].

width=0.5,center

As seen in Table [tbl:appendix95tab:performance95resp95bidmc], our proposed method demonstrates a significant performance improvement over the baselines. Furthermore, the performance gap between our method and another deep learning solution, SiNC, exceeds 25%.

11.2 The effect of loss weights↩︎

In this section, we investigate the effect of loss weights. During our experiments, we give the same importance to each regularizer, i.e., the coefficients (\(\lambda\), \(\nu\)) in Equation 14 are set to 1.

\[\label{appendix95eq:overall95loss} \mathcal{L} = -\lambda \sum_{f^*}^{} S(\omega) \log S(\omega) + \nu \sum_{f^*}^{} S_X(\omega) \log \frac{S_X(\omega)}{S(\omega)} + \sum_{f^{'}}^{} S(\omega),\omega = 2 \pi f\text{and}f^{'} = \mathbb{U} \setminus f^*\tag{14}\]

We observed the effect of two coefficients on the performance by setting them to discrete values of \(\{0.5, 1.0, 1.5, 2.0\}\), independently. The below figures present the results for two different signals, including electrocardiogram and photoplethysmography. It is evident from these figures that the optimal performance is achieved when the weights for both losses are balanced, rather than assigning greater importance to one over the other. Moreover, a closer inspection of these figures reveals an important outcome regarding the losses. When the models are trained while giving more importance to decreasing spectral entropy (\(\lambda\)) rather than keeping the relative entropy between input and output close (\(\nu\)), they collapse. This is noticeable when examining the blue line—increasing the importance of periodicity with the same weight for the other loss—the model error increases significantly.

Moreover, this effect is more pronounced with the datasets that include more noisy samples. For example, the mean absolute error increases by twice when more weight is given to periodicity as in Figure 6.

When investigating the model performance with varying loss weights on two signals, we can see the different behavior of the model. Specifically, in less noisy signal segments such as ECG, increasing the importance of the periodicity loss does not lead to substantial performance declines when compared with PPG signals. This phenomenon can be attributed to the distinct and clear periodic patterns, such as QRS complexes, present in the ECG signal. In contrast, the periodicity in blood volume changes is less apparent, hidden beneath higher noise levels, leading the model to capture more noise rather than the desired periodicity when the weights are imbalanced.

11.3 The effect of architectures↩︎

Here, we present the performances of the baseline architectures that are explicitly given in Appendix 10 for each dataset to investigate the effect of model capacity on generalization. An interesting result from these tables is that even though the number of parameters increases for architectures, the overall performance decreases for some datasets. For example, the second version of the fully convolutional neural network has roughly twice the number of parameters when compared with DCL. However, it performs worse than the DCL in cases such as PTB-XL and DaLiA. Although we applied this architecture search for the self-supervised learning paradigm as well, we have not observed a performance change across datasets.

width=1,center

11.4 Comparison with additional traditional methods↩︎

Here, we compared our approach with a traditional wavelet-based method [74] specifically designed for detecting the QRS complex in ECG signals, as QRS complexes represent the main source of periodicity in the ECG [64], which aligns with the periodicity of interest in our experiments for the ECG signals. The comparison results are given in Table ¿tbl:appendix:tab95performance95ecg95traditionals?.

As shown in Table ¿tbl:appendix:tab95performance95ecg95traditionals?, our proposed method outperforms traditional methods by a significant margin. we believe that the main reason for this performance difference is the fact that these methods are often developed by optimizing numerous hyperparameters specific to the dataset statistics, making them less effective in a general case.

12 Visual Examples↩︎

Here, we give some examples where the heuristic rule-based methods fail to detect periodicities in signals due to various types of noise. First, we show the cases when our method performs better than the Pan-Tompkins [64] algorithm for periodicity detection in ECG signals.

One interesting outcome of these illustrations is that the model demonstrates the ability to entirely disregard the impulse noise at the beginning (See Figure 9 (a)), and effectively approximate the periodicity in the signal. Moreover, a closer inspection of these two figures shows that the proposed method also works as an unsupervised segmentation algorithm of signals. For example, the end of Figure 9 demonstrates that the output waveform matches the desired patterns. However, when the time series becomes non-stationary in the segment, the segmentation shifts as the proposed algorithm assumes the signals are periodic or quasi-periodic in the investigated segment. This assumption is widely used in algorithm development for the human-generated data [20], such as signals generated from the cardiovascular (ECGs, PPGs) or skeletal system (IMUs) of humans similar to our case. Therefore, we believe that the proposed algorithm can be applied to segmentation tasks, enabling the identification of the desired periodic events in a completely unsupervised manner.

Here, we provided an example of how the proposed method is similar to an adaptive sinusoidal selector based on the Fourier transform. In uniformly sampled time sequences, periodicity is typically identified by the highest value in the spectral density [27]. Nevertheless, this method proves ineffective in situations where multiple periodicities exist within the frequency range of interest or noise, as illustrated in Figure 10.

According to these two figures, we can infer that the model learns the periodic representations of interest from the time domain waveform such that when it is evaluated with a signal that the model has never seen before, it can detect these patterns and surpass the noise. The task becomes relatively simpler for ECG signals due to their distinct periodicity indicator, the QRS complex. This allows the model to learn and effectively ignore impulse noise or other periodic noise sources even though they are both within the frequency range of interest.

Here, we present a more challenging example involving signals generated by blood volume changes. These signals are susceptible to noise and exhibit less evident periodicity, lacking specific complexes like ECG signals.

What stands out in these two figures is that the model learns the pattern of the desired periodic waveform from the time domain during training such that it can detect these patterns during inference (even though they are under the noise level) and surpass the other periodic noise.

These illustrative examples further support the claim that the proposed method operates as an adaptive sinusoidal selector by learning the periodic source of interest during training and amplifying them during inference without the need for supervision. We believe the results can be easily improved more by selecting clean samples during training, i.e., curriculum learning, to reinforce the pattern learning, or incorporating physiological limits more effectively into the loss function.

13 Computational Efficiency↩︎

In this section, we provide details on the computational efficiency of our proposed method. We employed a U-Net architecture with 300K parameters, making it lightweight and suitable for deployment on various platforms, including mobile devices. For context, MobileNetV2, often considered a lightweight model in deep learning, has 3.4M parameters, which is approximately 10 times larger than our model.

To provide a comprehensive comparison, we conducted two types of model overhead analyses using the blood volume pulse signals as input similar to the parameter calculations in Appendix 10.1. First, we calculated the inference time on a computer equipped with an Intel Core i7-10700k CPU running at 3.80 GHz, 32 GB RAM. Second, we deployed our model on a MAX78000 AI Accelerator, which has previously implemented U-Net architectures [75]. Detailed performance metrics and comparisons with the average of five runs with the standard deviations are provided in Table 2 where the running time is given in milliseconds.

An important aspect to note is that PyTorch (or MATLAB) efficiently computes the FFT; when analyzed with the profiler, the memory usage is close to that of the input (in-memory computation). However, for the MAX78000, our implementation, instead of DSP, requires slightly more memory. Overall, as shown by these experiments, our method not only offers better solutions for periodicity detection but also emphasizes practical implementation by having a very low memory footprint with fast inference. Moreover, some traditional methods do not fit the MCU flash memory due to heavy wavelet computations.

References↩︎