1 Introduction↩︎

Heart rate (HR) is a fundamental physiological indicator for health monitoring, but conventional ECG or contact PPG sensors require physical attachment and are less convenient for unobtrusive or continuous measurement. Remote photoplethysmography (rPPG) offers a camera-based alternative by estimating cardiac activity from facial videos [1]. However, reliable rPPG estimation remains challenging in unconstrained scenarios because the pulse-related visual signal is weak and easily corrupted by head motion, facial expression, illumination changes, compression artifacts, and appearance variation [2].

Early rPPG methods recover pulse signals using hand-crafted signal processing or color-space priors, including ICA, CHROM, and POS [3]–[5]. These methods are interpretable and efficient, but their performance often degrades in complex real-world scenarios. Deep models further improve rPPG estimation by learning spatiotemporal representations from data [6]–[13]. Nevertheless, most of them are trained with synchronized physiological labels and may struggle to generalize across datasets with different cameras, subjects, motions, and illumination conditions.

Self-supervised rPPG learning has recently been explored to reduce label dependence [14]–[17]. These methods learn from unlabeled facial videos through masked pre-training or strongly self-supervised objectives. However, such paradigms may still bias the learned representations toward facial appearance rather than latent physiological dynamics. Meanwhile, recent Mamba-based rPPG models further improve long-range temporal modeling with efficient state-space sequence processing [13], [18]–[20]. Despite their effectiveness, they mainly process facial video tokens in chronological order, without explicitly organizing frames according to recurring physiological states. This motivates a representation learning framework that is both label-efficient and rhythm-aware. Since pulse dynamics are quasi-periodic, temporally distant frames may share similar physiological states, whereas adjacent frames can belong to different phases of the cardiac cycle. Therefore, modeling only chronological continuity may be insufficient to capture inter-period relationships that are important for robust rPPG estimation.

To address these issues, we propose RhythmJEPA, a rhythm-structured joint-embedding predictive learning framework for rPPG. Following I-JEPA and VideoJEPA [21], [22], RhythmJEPA predicts latent teacher representations from masked facial videos instead of reconstructing pixels. Since rPPG estimation can be viewed as a sequence modeling problem, we design a Spatial Pulse Mixer (SPM) as a lightweight alternative to Transformer-based spatial modeling, producing effective frame-level pulse tokens for subsequent temporal modeling. We further introduce a Cyclic Rhythm-State Planner (CRSP) to assign latent physiological states to frames, and a Dual-Order Mamba Encoder (DOM) to combine chronological scanning with rhythm-state-ordered scanning. The main contributions of this paper are summarized as follows:

• We introduce RhythmJEPA, a joint-embedding predictive learning framework that learns latent physiological representations from masked facial videos for rPPG estimation.

• We design SPM to capture pulse-sensitive spatial cues effectively without the complexity of Transformer-based spatial modeling.

• We propose CRSP and DOM to model cyclic physiological recurrence beyond conventional chronological temporal scanning, together with cycle-consistency and state-balance losses that regularize rhythm-state assignments.

• Extensive intra- and cross-dataset experiments on PURE [23], UBFC-rPPG [24], and MMPD [25] demonstrate the effectiveness and robustness of RhythmJEPA.

2 Related Work↩︎

2.1 Traditional rPPG Methods↩︎

Traditional rPPG approaches recover pulse signals from facial color variations using hand-crafted physiological and optical priors. ICA and PCA separate RGB observations into latent source components [3], [26], while CHROM, POS, and LGI design color-space or local-invariance constraints to suppress non-pulsatile interference [4], [5], [27]. Although efficient and interpretable, these methods rely on assumptions about skin reflection, illumination, and motion, which limits their robustness in unconstrained videos.

2.2 Deep Learning-based rPPG↩︎

Recent deep models improve rPPG estimation by learning spatiotemporal representations from facial video sequences. CNN-based and temporal-shift methods capture pulse-related appearance changes [6]–[8], [10], while Transformer and periodicity-aware models enhance temporal reasoning under challenging conditions [9], [11], [12], [28]. However, most supervised approaches require synchronized physiological labels and may generalize poorly across subjects, cameras, and acquisition settings.

To reduce label dependence, self-supervised methods such as rPPG-MAE, MaskFusionNet, and RS+rPPG learn from unlabeled facial videos through masked pretraining or strong self-supervision [14]–[16]. Nevertheless, reconstruction- or consistency-oriented objectives can still emphasize facial appearance and low-level visual variations rather than latent physiological dynamics.

2.3 State-Space Modeling for rPPG↩︎

State-space models provide an efficient alternative to attention-based sequence modeling. Mamba introduces input-dependent selective scanning with linear complexity, and VMamba extends this idea to visual representation learning [18], [29]. Recent rPPG models adopt Mamba-style modules for temporal physiological modeling [13], [19], [20], but their scan orders remain largely chronological or multi-scale. RhythmJEPA instead organizes frames by latent rhythm states, enabling Mamba scanning to capture inter-period relationships across similar pulse phases.

2.4 Joint-Embedding Predictive Learning↩︎

Joint-embedding predictive architectures learn representations by predicting latent targets instead of reconstructing input pixels. I-JEPA demonstrates this principle for images [21], while VideoJEPA extends feature prediction to video representation learning [22]. This paradigm is well suited to rPPG because the signal of interest is a latent physiological process rather than facial appearance itself. RhythmJEPA adapts joint-embedding prediction to rPPG by predicting teacher embeddings from masked facial videos and coupling this objective with rhythm-state planning and dual-order Mamba encoding.

3 Methodology↩︎

3.1 Problem Formulation↩︎

We represent a facial video clip as \[\mathbf{X}=\{X_t\}_{t=0}^{T-1}, \qquad X_t\in\mathbb{R}^{H\times W\times 3}.\] Its synchronized rPPG waveform is represented as \[\mathbf{y}=[y_0,\dots,y_{T-1}]\in\mathbb{R}^{T}.\] We learn a video-to-waveform mapping: \[f_{\theta}:\mathbb{R}^{T\times H\times W\times 3}\rightarrow\mathbb{R}^{T}, \qquad \hat{\mathbf{y}}=f_{\theta}(\mathbf{X}),\] where \(\hat{\mathbf{y}}=[\hat{y}_0,\dots,\hat{y}_{T-1}]\) denotes the predicted rPPG waveform.

As shown in Fig. 2, RhythmJEPA consists of four modules: a facial feature encoder \(E\), a Cyclic Rhythm-State Planner \(\Gamma\), a Dual-Order Mamba encoder \(D\), and a prediction head \(h\). The overall forward process follows \[\begin{align} \mathbf{Z} &= E(\mathbf{X}), & (\mathbf{Q},\mathbf{s}^{\star}) &= \Gamma(\mathbf{Z}),\\ \mathbf{Y} &= D(\mathbf{Z},\mathbf{s}^{\star}), & \hat{\mathbf{y}} &= h(\mathbf{Y}). \end{align} \label{eq:rhythmjepa95forward}\tag{1}\] Here \(\mathbf{Z},\mathbf{Y}\in\mathbb{R}^{T\times C}\) denote frame-wise physiological tokens and rhythm-aware temporal features, respectively. \(\mathbf{Q}\in\mathbb{R}^{T\times K}\) is the rhythm-state posterior, and \(\mathbf{s}^{\star}\in\{0,\ldots,K-1\}^{T}\) is the decoded cyclic state path. RhythmJEPA is first pre-trained with a rhythm-structured JEPA objective and then fine-tuned for waveform regression.

3.2 Rhythm-Structured JEPA Pre-training↩︎

RhythmJEPA learns physiological representations by predicting latent teacher features instead of reconstructing RGB pixels. We sample a temporal mask \(\mathbf{V}\in\{0,1\}^{T\times1\times1\times1}\), where \(V_t=0\) denotes a masked frame, and define \(\mathcal{M}=\{t\mid V_t=0\}\). Masked frames are replaced by a learnable RGB token \(\mathbf{X}_{\mathrm{mask}}\): \[\widetilde{\mathbf{X}} = \mathbf{V}\odot\mathbf{X} + (\mathbf{1}-\mathbf{V})\odot\mathbf{X}_{\mathrm{mask}}.\] The student branch encodes the masked clip, while the EMA teacher branch encodes the original clip: \[\mathbf{R}_{s}=D_s(E_s(\widetilde{\mathbf{X}}),\mathbf{s}^{\star}_{s}), \qquad \mathbf{R}^{+}=\operatorname{sg}\big(D_t(E_t(\mathbf{X}),\mathbf{s}^{\star}_{t})\big).\] A lightweight predictor maps the student representation to the teacher latent space: \[\widehat{\mathbf{R}}=P(\mathbf{R}_{s}), \qquad \widehat{\mathbf{R}},\mathbf{R}^{+}\in\mathbb{R}^{T\times C}.\] Here \(\widehat{\mathbf{r}}_t\) and \(\mathbf{r}^{+}_t\) denote the \(t\)-th temporal embeddings of \(\widehat{\mathbf{R}}\) and \(\mathbf{R}^{+}\), respectively. The JEPA loss is computed only at masked temporal positions: \[\mathcal{L}_{\mathrm{JEPA}} = \frac{1}{|\mathcal{M}|} \sum_{t\in\mathcal{M}} \left\| \ell_2(\widehat{\mathbf{r}}_t) - \ell_2(\mathbf{r}^{+}_t) \right\|_2^2.\] The EMA teacher update follows \[\theta_t\leftarrow \mu\theta_t+(1-\mu)\theta_s.\]

3.3 Facial Feature Encoder↩︎

The facial feature encoder maps each frame to a frame-level feature representation. It contains an adopted Facial Feature Extractor [30] and an SPM.

3.3.0.1 Facial Feature Extractor.

FFE captures local cues by combining RGB appearance with short-term temporal differences: \[\mathbf{F}_{0} = \operatorname{FFE}(\mathbf{X}), \qquad \mathbf{F}_{0}\in\mathbb{R}^{T\times C_s\times H'\times W'}.\] A \(1\times1\) convolution projects the channel dimension to \(C\), followed by positional embedding: \[\mathbf{F} = \operatorname{PE} \left( \phi_{1\times1}(\mathbf{F}_{0}) \right), \qquad \mathbf{F}\in\mathbb{R}^{T\times C\times H'\times W'}.\]

3.3.0.2 Spatial Pulse Mixer.

We propose SPM to perform lightweight spatial modeling. For a kernel size \(a\times b\), the basic depth-wise block is \[\mathcal{B}_{a\times b}(\mathbf{F}) = \operatorname{GELU} \left( \operatorname{BN} \left( \operatorname{DWConv}_{a\times b}(\mathbf{F}) \right) \right).\] SPM uses horizontal, vertical, and local branches: \[\mathbf{F}_{h}=\mathcal{B}_{1\times5}(\mathbf{F}), \quad \mathbf{F}_{v}=\mathcal{B}_{5\times1}(\mathbf{F}), \quad \mathbf{F}_{\ell}=\mathcal{B}_{3\times3}(\mathbf{F}).\] Residual point-wise fusion gives the mixed feature map: \[\mathbf{F}^{\star} = \mathbf{F} + \gamma\operatorname{BN} \left( \rho_{1\times1} \left([ \mathbf{F}_{h}; \mathbf{F}_{v}; \mathbf{F}_{\ell} ] \right) \right).\] SPM further applies attentive spatial pooling to produce frame-level features. For each frame, the spatial attention map is computed as \[\alpha_t = \operatorname{Softmax}_{H'W'} \left( \psi_{1\times1}(\mathbf{F}^{\star}_{t}) \right).\] The frame-level token is obtained through weighted pooling: \[\mathbf{z}_{t} = \sum_{i,j} \alpha_{t,i,j} \mathbf{F}^{\star}_{t,:,i,j}.\] Stacking all frame tokens yields \[\mathbf{Z} = \operatorname{LN} \left( [\mathbf{z}_0,\dots,\mathbf{z}_{T-1}] \right), \qquad \mathbf{Z}\in\mathbb{R}^{T\times C}.\]

3.4 Cyclic Rhythm-State Planner↩︎

CRSP estimates latent rhythm states and decodes a cyclic state path for rhythm-aware temporal scanning. Given \(\mathbf{Z}\), CRSP first extracts local temporal evidence: \[\mathbf{U} = \mathbf{Z} + \eta\, \operatorname{MLP} \left( \operatorname{DWConv}_{5} \left( \mathbf{Z} \right) \right).\] A Fourier Analysis Network (FAN) [31] predicts frame-wise rhythm-state logits: \[\mathbf{R}=\operatorname{FAN}(\mathbf{U}), \qquad \mathbf{R}\in\mathbb{R}^{T\times K}.\] The rhythm-state probabilities are obtained by \[\mathbf{Q}=\operatorname{Softmax}_{K}(\mathbf{R}), \qquad Q_{t,k}\in[0,1].\] Here \(Q_{t,k}\) denotes the predicted probability that frame \(t\) belongs to rhythm state \(k\). To better follow state transitions within the cardiac rhythm, we propose a cyclic consistency constraint that allows each state to either stay unchanged or move to the next state: \[\mathcal{A}(i)=\{i,(i+1)\bmod K\}.\] The decoded state path is obtained from \[\begin{align} \mathbf{s}^{\star} &= \mathop{\mathrm{arg\,max}}_{s_{0:T-1}} \sum_{t=0}^{T-1} \log Q_{t,s_t},\\ \mathrm{s.t.}\quad s_t&\in\mathcal{A}(s_{t-1}), \quad t=1,\dots,T-1. \end{align}\] This constrained path is obtained by Viterbi-style dynamic programming.

3.4.0.1 Regularization.

Let \(\mathbf{A}\in\{0,1\}^{K\times K}\) be the transition matrix induced by \(\mathcal{A}\). We define the cyclic transition loss as \[\mathcal{L}_{\mathrm{cyc}} = - \frac{1}{T-1} \sum_{t=0}^{T-2} \log \left( \mathbf{q}_{t}^{\top} \mathbf{A} \mathbf{q}_{t+1} + \epsilon \right),\] where \(\mathbf{q}_{t}=\mathbf{Q}_{t,:}\). To avoid state collapse, we add the balance loss, where \(\boldsymbol{\pi}\) denotes the desired prior distribution over rhythm states \[\mathcal{L}_{\mathrm{bal}} = \operatorname{KL} \left( \boldsymbol{\pi} \,\|\, \bar{\mathbf{q}} \right),\] where the average state usage is \[\bar{\mathbf{q}} = \frac{1}{T} \sum_{t=0}^{T-1} \mathbf{q}_t.\]

3.5 Dual-Order Mamba Encoder↩︎

Prior Mamba-based rPPG methods typically scan tokens only in chronological order, overlooking the effect of scan order on rhythm modeling. DOM addresses this by scanning the token sequence in two complementary orders. The chronological branch processes the original temporal order: \[\mathbf{Y}^{\mathrm{time}} = \operatorname{BiMamba}_{\mathrm{time}}(\mathbf{Z}).\] The rhythm branch sorts tokens by decoded state and then by time: \[\boldsymbol{o} = \operatorname{argsort} \left( s^{\star}_{t}T+t \right).\] The state-ordered sequence is formed as \[\mathbf{Z}^{\mathrm{state}}_{r,:} = \mathbf{Z}_{o_r,:}.\] A second BiMamba scan processes this sequence: \[\widetilde{\mathbf{Y}}^{\mathrm{state}} = \operatorname{BiMamba}_{\mathrm{state}} \left( \mathbf{Z}^{\mathrm{state}} \right).\] We scatter the output back to the chronological order: \[\mathbf{Y}^{\mathrm{state}}_{o_r,:} = \widetilde{\mathbf{Y}}^{\mathrm{state}}_{r,:}.\] A token-wise gate fuses the two temporal views: \[\mathbf{G} = \sigma \left( \phi_g \left( [\mathbf{Z};\mathbf{Y}^{\mathrm{time}};\mathbf{Y}^{\mathrm{state}}] \right) \right),\] \[\mathbf{Y} = \mathbf{G}\odot\mathbf{Y}^{\mathrm{time}} + (\mathbf{1}-\mathbf{G})\odot\mathbf{Y}^{\mathrm{state}}.\]

3.6 Prediction Head↩︎

The prediction head maps each fused temporal representation to a scalar pulse value: \[\hat{y}_t = \phi_o \left( \operatorname{LN}(\mathbf{Y}_t) \right).\] Collecting all predictions gives \[\hat{\mathbf{y}} = [\hat{y}_0,\dots,\hat{y}_{T-1}].\] During fine-tuning, we optimize the negative Pearson correlation loss: \[\mathcal{L}_{\mathrm{NP}} = 1 - \frac{ \sum_{t=0}^{T-1} (\hat{y}_t-\bar{\hat{y}})(y_t-\bar{y}) }{ \sqrt{ \sum_{t=0}^{T-1} (\hat{y}_t-\bar{\hat{y}})^2 } \sqrt{ \sum_{t=0}^{T-1} (y_t-\bar{y})^2 } }.\] The overall pre-training objective combines the three losses: \[\mathcal{L}_{\mathrm{pre}} = \lambda_{\mathrm{JEPA}}\mathcal{L}_{\mathrm{JEPA}} + \lambda_{\mathrm{cyc}}\mathcal{L}_{\mathrm{cyc}} + \lambda_{\mathrm{bal}}\mathcal{L}_{\mathrm{bal}}.\]

4 Experiments↩︎

This section describes the experimental protocol, including datasets, evaluation metrics, and implementation details. We then report intra-dataset and cross-dataset comparisons with representative rPPG baselines.

4.1 Datasets↩︎

We evaluate RhythmJEPA on three widely used rPPG benchmarks:

UBFC-rPPG [24] contains 42 facial RGB videos recorded at 30 fps under indoor and sunlight illumination conditions. The reference physiological signals are collected by a CMS50E pulse oximeter at 60 Hz.

PURE [23] includes 60 RGB videos from 10 subjects, captured at 30 fps under six predefined head-motion settings. The synchronized pulse signals are also recorded using CMS50E at 60 Hz.

MMPD [25] is a mobile rPPG benchmark collected from 33 subjects. It includes diverse variations in skin tone, illumination, camera setting, and activity type, making it suitable for evaluating robustness under realistic domain shifts.

2.35pt

3.5pt

4.2 Evaluation Metrics↩︎

Following mainstream evaluation settings [10]–[12], we evaluate HR estimation using mean absolute error (MAE), mean absolute percentage error (MAPE), root mean square error (RMSE), and Pearson’s correlation coefficient (\(r\)). MAE/RMSE measure absolute errors, while MAPE reflects relative error. Pearson’s \(r\) highlights the linear association between predicted and ground-truth HRs, where higher values indicate better consistency.

4.3 Implementation Details↩︎

Face regions are first detected by RetinaFace [32]. Each bounding box is enlarged by a factor of \(1.5\), cropped, and resized to \(128\times128\) pixels. Training clips contain \(180\) frames with a stride of \(90\), and random horizontal flipping is used for augmentation.

The token dimension is set to \(C=96\), and RhythmJEPA uses \(4\) latent rhythm states. The state planner employs FAN with hidden, periodic, and non-periodic dimensions of \(96\), \(24\), and \(48\), respectively. The context encoder uses dual-order Bi-Mamba scanning with \(3\) blocks per scan and Mamba state dimension \(16\). In the JEPA stage, the teacher momentum and mask ratio are set to \(0.996\) and \(0.7\), respectively. The loss weights are set to \(\lambda_{\mathrm{JEPA}}=1.0\), \(\lambda_{\mathrm{cyc}}=0.1\), and \(\lambda_{\mathrm{bal}}=0.1\).

All models are trained on an NVIDIA GeForce RTX 3090 GPU using PyTorch 2.0.0 with CUDA 11.8. We use AdamW with batch size \(4\) and an initial learning rate of \(1\times10^{-4}\). The learning rate follows a OneCycle schedule [33] with cosine annealing. Pretraining is conducted for \(30\) epochs, while finetuning is performed for \(10\) epochs.

4.4 Main Comparison↩︎

For intra-dataset evaluation, we follow the training and testing splits used in LSTS [28] to ensure a fair comparison with prior rPPG methods. For cross-dataset evaluation, the model is trained on the source dataset and directly tested on the target dataset without using target-domain training samples.

4.4.1 Intra-dataset Testing↩︎

Tab. 1 summarizes the within-dataset evaluation on PURE [23], UBFC-rPPG [24], and MMPD [25]. RhythmJEPA ranks first across all reported metrics, while the underlined entries indicate the strongest competing results. On PURE, it improves over LSTS by 20.0%/18.2%/30.0% in MAE/MAPE/RMSE and reaches a perfect correlation of 1.00, matching the best baseline correlation. On UBFC-rPPG, the error reduction is more substantial: RhythmJEPA lowers the second-best MAE/MAPE/RMSE from 0.51/0.54/0.79 to 0.21/0.24/0.34, corresponding to 58.8%/55.6%/57.0% relative improvements. On MMPD, which contains stronger appearance, illumination, and activity variations, RhythmJEPA also reduces MAE/MAPE/RMSE by 4.9%/15.9%/7.6% over the best competing error results and improves the correlation from 0.74 to 0.76. These consistent gains indicate that rhythm-structured predictive learning helps recover pulse-related cues more reliably across both clean and challenging intra-dataset settings.

4.4.2 Cross-dataset Testing↩︎

Tab. 2 further examines cross-dataset transfer, where training and testing videos come from different acquisition domains. RhythmJEPA achieves the best results in all four transfer directions, showing consistent relative error reductions over the strongest baseline. For PURE\(\rightarrow\)UBFC-rPPG, RhythmJEPA reduces MAE/MAPE/RMSE by 27.6%/7.1%/21.8% over PhysNet, while also increasing the correlation from 0.99 to 1.00. For UBFC-rPPG\(\rightarrow\)PURE, the improvement is especially clear in RMSE, decreasing from 13.80 to 8.93, with the highest correlation of 0.92. When transferring to the more challenging MMPD dataset, RhythmJEPA reduces MAE/MAPE/RMSE by 28.4%/26.7%/16.4% for PURE\(\rightarrow\)MMPD and by 10.6%/5.4%/10.5% for UBFC-rPPG\(\rightarrow\)MMPD. The correlation also improves from 0.23 to 0.27 and from 0.29 to 0.31, respectively. These results suggest that RhythmJEPA learns physiological representations that transfer more effectively across datasets, rather than relying primarily on dataset-specific appearance statistics.

4.4.3 Computational Cost and Inference Efficiency↩︎

Table 5 compares the computational cost and inference speed of RhythmJEPA against representative rPPG baselines under the same benchmarking setting. RhythmJEPA uses only 3.20M parameters and 33.50G MACs, making it substantially lighter than recent Transformer-based methods such as PhysFormer [9], LSTS [28], and Reperio-rPPG [12]. Although EfficientPhys [10] is the smallest and fastest model, its accuracy is clearly lower than RhythmJEPA in Tables 1 and 2, especially on challenging MMPD settings. In contrast, RhythmJEPA achieves the best estimation accuracy while retaining the second-best parameter count, MACs, and latency in Table 5, demonstrating a favorable accuracy–efficiency trade-off.

5 Ablation Study↩︎

This section analyzes the core design choices of RhythmJEPA from spatial, temporal, and pre-training perspectives. We first examine whether SPM attends to physiologically meaningful facial regions, then compare different Mamba scan orders to isolate the effect of rhythm-state scanning. Finally, we investigate why JEPA pre-training is useful for learning rhythm-aware representations before supervised rPPG fine-tuning.

5.1 Analysis of Spatial Pulse Modeling↩︎

Figure 3 analyzes the behavior of SPM on PURE and UBFC-rPPG using three complementary views: the recovered rPPG waveform, the corresponding power spectrum, and spatial heatmaps. The predicted waveform is highly correlated with the ground-truth BVP signal, closely matching its periodic peaks and valleys. The power spectrum further shows that the dominant frequency of the prediction aligns well with the reference rhythm, indicating accurate and temporally consistent pulse recovery.

The heatmaps further show that SPM suppresses background or weakly informative regions and instead highlights facial areas such as the cheeks, forehead, and nose. Since these regions typically exhibit stronger pulse-induced color variations, this pattern suggests that SPM provides a pulse-aware spatial prior rather than relying on irrelevant appearance or background cues.

5.2 Effect of Rhythm-State Mamba Scanning↩︎

We first examine whether the gain comes from the Mamba temporal module alone or from the proposed rhythm-structured scan order. The chronological scan processes frames in their original temporal order, whereas the rhythm-state scan reorders frames according to the latent rhythm states decoded by CRSP. As shown in Table 3, replacing chronological scanning with rhythm-state scanning reduces the error on both PURE and MMPD, indicating that frames sharing similar pulse states provide useful long-range dependencies even when they are not adjacent in time. The dual-order scan further combines local temporal continuity with rhythm-consistent dependencies and achieves the best MAE and correlation on both datasets, as well as the best RMSE on MMPD. On PURE, the rhythm-state-only scan obtains the lowest RMSE, suggesting that this cleaner setting benefits strongly from phase-consistent grouping, while dual-order scanning remains more balanced across metrics and datasets.

Table 3 also depends on whether CRSP can produce physiologically meaningful state sequences for the rhythm-state scan. We therefore ablate the CRSP constraints in Figure 4. Removing the grammar checker assigns each frame directly from frame-wise softmax probabilities, which can make the decoded state path noisy, disordered, and inconsistent with the cyclic pulse progression assumed by the scan. Removing the cyclic-transition loss weakens the phase-order prior, while removing the state-balance loss makes the learned states less evenly used across the pulse cycle. In all cases, performance drops compared with the full model. Together, Table 3 and Figure 4 show that RhythmJEPA benefits not only from scanning in rhythm-state order, but also from learning stable, balanced, and cyclically valid state assignments before performing that scan.

5.3 Analysis of JEPA Pretraining↩︎

We next isolate the contribution of joint-embedding predictive pretraining. Both variants use the same downstream architecture and supervised fine-tuning protocol; they differ only in whether the model is initialized by RhythmJEPA pretraining. As shown in Table 4, pretraining reduces MAE and RMSE on all three datasets. The gain is modest on PURE, where the baseline is already strong, but becomes clearer on UBFC-rPPG and MMPD. In particular, pretraining improves UBFC-rPPG from 0.36/0.51 to 0.21/0.34 in MAE/RMSE and improves MMPD from 5.02/10.36 to 3.92/9.15. These results suggest that predicting masked teacher embeddings provides a useful rhythm-aware initialization, especially when appearance, illumination, and motion variations make supervised fine-tuning more difficult.

3.2pt

We further analyze the mask ratio used during JEPA pretraining in Figure 5. The experiment is conducted on PURE and UBFC-rPPG while keeping the fine-tuning protocol fixed, so the curves reflect the sensitivity of the learned initialization to the pretext-task difficulty. On PURE, increasing the mask ratio from 0.5 to 0.7–0.8 steadily reduces both MAE and RMSE, but the error rises again at 0.9. UBFC-rPPG shows a similar trend, with the best performance at 0.7 and a clear degradation when the ratio is too high. This pattern indicates that a low mask ratio gives an overly easy prediction task, whereas an excessive ratio removes too much temporal evidence for stable rhythm prediction. A moderate-to-high ratio therefore provides the best balance between pretraining difficulty and physiological recoverability.

6 Conclusion↩︎

In this paper, we presented RhythmJEPA, a joint-embedding predictive learning framework for rPPG that focuses on learning physiological representations rather than reconstructing facial appearance. By predicting teacher embeddings from masked facial videos, the model is encouraged to capture information related to the underlying pulse signal while reducing dependence on pixel-level reconstruction. We also introduced SPM to extract frame-level representations with a lightweight spatial design, avoiding the computational burden of Transformer-based spatial modeling. To better exploit the periodic nature of rPPG signals, CRSP estimates latent rhythm states for video frames, and DOM uses these states to scan the sequence in both chronological and rhythm-aware orders. This design allows the model to consider not only neighboring frames but also frames that may correspond to similar phases of the pulse cycle. Experiments on PURE, UBFC-rPPG, and MMPD show that RhythmJEPA improves both intra-dataset accuracy and cross-dataset generalization over representative baselines. Despite these results, the learned rhythm states are still implicit and may not directly correspond to clinically interpretable cardiac phases. Future work will investigate more interpretable rhythm modeling and evaluate the framework under more diverse real-world conditions.

References↩︎