1 Introduction↩︎

Consider a vision-language model deployed to screen chest radiographs in a hospital. The model was validated on a large multi-centre dataset and performs well at launch. Over the following months, the imaging department replaces its X-ray machines with a newer generation that produces images with different noise characteristics and contrast curves. No label in the electronic health record changes. No engineer is notified. The model’s predictions begin to degrade, but because the model itself provides no internal alarm, it is frozen and produces a softmax distribution regardless of input quality, the degradation may go unnoticed until clinical outcomes deteriorate.

Sensor ageing, software updates, seasonal variation, and the sheer diversity of open-world environments routinely cause the input distribution \(p(x)\) to drift away from the distribution seen at training time [1]–[3]. For vision-language models such as CLIP [4], which are increasingly used as frozen feature extractors and zero-shot classifiers in downstream systems, this drift is particularly insidious: the model’s output distribution changes as a function of its input, yet the model was not designed to signal when its inputs are anomalous.

The natural remedy is monitoring: continuously inspect the model’s sequential outputs and raise an alarm when statistical evidence of shift accumulates. This reduces the problem to sequential hypothesis testing: given a stream of test statistics derived from the model’s outputs, decide at each step whether the null hypothesis of no shift is still tenable.

1.1 The Peeking Problem↩︎

Sequential monitoring violates the assumptions of classical hypothesis testing in a fundamental way. A standard two-sample test [5] fixes a sample size \(n\) in advance, computes a statistic on \(n\) observations, and rejects the null if the statistic exceeds a critical value calibrated to control type-I error at level \(\alpha\). In a monitoring context, a practitioner instead observes a running statistic and acts the first time it crosses a threshold. The sample size is no longer fixed; it is a stopping time that may depend on the data.

If the null is true and the practitioner re-examines the test statistic at every new observation, the probability of ever crossing a fixed critical value tends to \(1\) as the stream grows, regardless of the nominal \(\alpha\) [6]–[8]. This is a consequence of the optional stopping theorem: a bounded stopped martingale has the same expectation as its initial value, so any fixed threshold is eventually crossed under continuous monitoring. Practitioners who apply fixed-sample tests in a rolling or continuously-monitored fashion therefore see false-alarm rates far above their nominal \(\alpha\).

The only way to obtain rigorous type-I error control under an unknown, data-adaptive stopping time is to use a test statistic whose uniform supremal distribution is controlled.

1.2 E-Processes as the Natural Solution↩︎

The mathematical object that provides this control is an e-process: a nonnegative stochastic process that is a supermartingale under the null hypothesis [9]–[11]. Ville’s maximal inequality [6], which predates, and in fact motivated, the modern optional stopping theorem, states that for any nonnegative supermartingale \(\{M_t\}_{t \ge 0}\), \[\mathbb{P}\!\Bigl(\sup_{t \ge 1} M_t \ge \tau\Bigr) \le \frac{M_0}{\tau} = \frac{1}{\tau} \label{eq:ville}\tag{1}\] for all \(\tau > 0\). This bound holds simultaneously at every stopping time, including data-adaptive ones. Setting the threshold \(\tau = 1/\alpha\) therefore guarantees false alarms occur with probability at most \(\alpha\), no matter when the practitioner decides to stop. The e-process provides exactly the time-uniform guarantee that classical tests cannot.

1.3 Contributions↩︎

We propose E-SHIFT, a deployable sequential test for distribution shift in streaming vision-language systems. Our specific contributions are:

(1) We construct a two-term score that simultaneously measures shift in the VLM’s predictive distribution (via KL divergence to the uniform) and in its feature embedding (via Mahalanobis distance from the calibration centroid). The two terms are complementary: the KL term detects overconfident or systematically biased posteriors; the Mahalanobis term detects feature-space drift even when the predictive distribution is uninformative.

(2) With limited calibration data the population log-MGF needed to construct the e-process is unknown. We restore the supermartingale property by replacing it with a bootstrap upper-confidence bound, and show this yields an unconditional false-alarm guarantee of \(\beta + 1/\tau\).

(3) Because E-SHIFT resets after each alert to remain active for subsequent shifts, false-alarm probability can accumulate over a long stream. We derive an explicit bound showing the total false-alarm probability over \(K\) resets is at most \(K/\tau\).

(4) We prove that under persistent shift the expected detection delay is \(\mathcal{O}(\log\tau / \Gamma)\), where \(\Gamma\) equals the KL divergence between post- and pre-shift score distributions when the likelihood-ratio e-value is used, providing an information-theoretically interpretable and experimentally measurable characterization of detection speed.

2 Lay of the land of the ideas↩︎

2.1 Sequential Hypothesis Testing↩︎

Sequential analysis was formalized by [7], who introduced the sequential probability ratio test (SPRT). Unlike fixed-sample tests, the SPRT allows the analyst to stop whenever the accumulated likelihood ratio crosses either an upper or a lower boundary. Wald showed that the SPRT minimizes the expected sample size among all tests with the same type-I and type-II error rates, establishing a fundamental optimality property that fixed-sample tests cannot achieve.

The key insight is that the log-likelihood ratio accumulated over independent observations is a random walk; the stopping rules are boundaries for this walk. [12] adapted this idea to the change-point setting via the CUSUM statistic \(W_t = \max(0, W_{t-1} + \ell_t)\), which accumulates evidence of change while resetting to zero whenever the accumulated sum falls below zero. Page showed that CUSUM triggers detection as quickly as possible in expectation after a change, and [13] and [14] later established its minimax optimality under the worst-case expected detection delay criterion.

All of these methods require knowing or specifying the pre- and post-change distributions, which limits their applicability to nonparametric settings such as distribution shift in deep-learning systems.

2.2 Martingales and the Optional Stopping Theorem↩︎

A martingale is a sequence of random variables \(\{M_t\}_{t \ge 0}\) such that \(\mathbb{E}[M_{t+1} \mid M_0, \ldots, M_t] = M_t\) [8], [15]. A supermartingale satisfies \(\mathbb{E}[M_{t+1} \mid M_0, \ldots, M_t] \le M_t\): in expectation the process can only decrease. The optional stopping theorem states that for a bounded supermartingale stopped at a stopping time \(T\), \(\mathbb{E}[M_T] \le M_0\). Combined with Markov’s inequality, this gives Doob’s maximal inequality: for any nonnegative supermartingale, \(\mathbb{P}(\sup_{t \ge 0} M_t \ge \tau) \le M_0 / \tau\). When \(M_0 = 1\) and the supermartingale property holds under the null, this immediately controls the false-alarm probability uniformly in the stopping time.

[6] established this inequality independently and used it to construct what we now recognize as the first e-process. He showed that for any sequence of betting strategies against a null probability distribution, the bettor’s wealth process is a nonnegative supermartingale, and derived the \(\mathbb{P}(\sup_t M_t \ge \tau) \le 1/\tau\) bound that underpins all modern anytime-valid inference.

2.3 E-Values and E-Processes↩︎

An e-value is a one-observation test statistic. By Markov’s inequality, \(\mathbb{P}_{\mathcal{H}_0}(E \ge \tau) \le 1/\tau\) for all \(\tau > 0\), so a single e-value already controls the false-alarm rate at any fixed threshold \(\tau\). The power of the framework comes from combining e-values. If \(E_1, E_2, \ldots\) are conditionally independent e-values under \(\mathcal{H}_0\), meaning each \(E_t\) has expectation at most one given the past, then their running product \(M_t = \prod_{i=1}^{t} E_i\) is a nonnegative supermartingale.

Ville’s inequality applied to the e-process gives \(\mathbb{P}_{\mathcal{H}_0}(\sup_{t \ge 1} M_t \ge \tau) \le 1/\tau\) uniformly over all stopping times, the time-uniform guarantee that is the central property of E-SHIFT.

The game-theoretic interpretation of [16] is illuminating. Imagine a sequential game between a Skeptic and Reality. The Skeptic starts with unit wealth. At each round the Skeptic bets a fraction of their wealth against the null; Reality reveals the outcome. If the null is true, the Skeptic’s wealth is a nonnegative supermartingale and cannot grow unboundedly in expectation. If Reality deviates from the null (i.e.distribution shift occurs), a well-chosen betting strategy causes the Skeptic’s wealth to grow, eventually exceeding any threshold \(\tau\). In this framing, E-SHIFT is a recipe for designing the Skeptic’s betting strategy using non-conformity scores from a VLM.

Selecting the e-value.↩︎

A natural choice is the exponential e-value [11], [17]: given a score \(S\) and its population mean \(\mu\) under \(\mathcal{H}_0\), set \[E = \exp\!\bigl(\lambda(S - \mu) - \psi(\lambda)\bigr), \qquad \psi(\lambda) = \log\mathbb{E}_{\mathcal{H}_0} [e^{\lambda(S-\mu)}], \label{eq:expeval}\tag{2}\] where \(\lambda > 0\) is a free parameter and \(\psi(\lambda)\) is the log-moment-generating function (log-MGF) that ensures \(\mathbb{E}[E] = 1\). The product of independent such factors gives the exponential e-process used throughout this paper. Under the alternative, the expected increment per step is \(\lambda(\mathbb{E}_{P_1}[S] - \mu) - \psi(\lambda)\), which is positive when \(P_1 \ne P_0\), so the product grows exponentially.

2.4 Distribution Shift: A Taxonomy↩︎

Let \(X \in \mathcal{X}\) denote the input and \(Y \in \mathcal{Y}\) the output. Following [1] and [18], we distinguish three canonical forms of distribution shift.

Covariate shift refers to a change in the marginal \(p(x)\) while the conditional \(p(y \mid x)\) remains fixed. This arises when the data-collection conditions change, e.g.sensor upgrades, geographic transfer, or temporal drift, but the underlying relationship between inputs and labels does not. Covariate shift is the form of shift targeted by E-SHIFT.

Label shift [19] refers to a change in the marginal \(p(y)\) while \(p(x \mid y)\) remains fixed. This arises in class-imbalanced deployment, e.g.a rare-disease classifier applied to a population with a different disease prevalence. Label shift may manifest in the model’s output distribution without visible feature change.

Concept drift refers to a change in \(p(y \mid x)\) itself, meaning the learned decision boundary is no longer appropriate. This is the most severe form and may require model retraining.

E-SHIFT is designed for covariate shift: the composite non-conformity score (Section 5.1) is sensitive to drift in \(p(x)\) but does not directly measure changes in \(p(y \mid x)\). Extending the framework to label shift and concept drift is a natural direction for future work.

2.5 Vision-Language Models as Distribution Monitors↩︎

Vision-language models such as CLIP [4] jointly embed images and text descriptions into a shared metric space using a dual-encoder architecture [20]. Trained on hundreds of millions of image-text pairs via contrastive learning, CLIP learns image representations that are rich in semantic content and have been shown to generalize substantially beyond their training distribution [21].

As a consequence, CLIP feature vectors serve as a powerful surrogate for the image distribution: images drawn from the training distribution cluster tightly in CLIP embedding space, while images from a shifted distribution tend to lie farther from the training cluster [3]. The model’s softmax output distribution also changes under shift: when faced with a corrupted or out-of-distribution image, CLIP often produces either overconfident posteriors (concentrated mass on one class) or low-confidence posteriors (near-uniform mass), both of which deviate from the training-distribution calibration.

These two signals, feature-space location and predictive distribution shape, form the basis of E-SHIFT’s composite non-conformity score.

3 Related Work↩︎

3.0.0.1 Classical sequential change-point detection.

The rich literature on sequential change-point detection [12]–[14], [22] provides optimal methods under parametric assumptions. CUSUM [12] is minimax-optimal under Lorden’s worst-case expected detection delay criterion [14], and the Shiryaev–Roberts statistic [22] solves the corresponding Bayesian problem. Both require knowing the likelihood ratio of post- to pre-shift data. E-SHIFT is nonparametric: it requires only a calibration-set estimate of the score distribution under the null. When the log-likelihood ratio is used as the score, E-SHIFT’s increments coincide with those of CUSUM; however, the CUSUM recursion \(W_t = \max(0, W_{t-1} + \ell_t)\) incorporates a running-minimum reset that E-SHIFT does not replicate, so CUSUM’s Lorden–Moustakides optimality does not transfer by inspection and we treat CUSUM purely as an empirical baseline.

3.0.0.2 E-values and anytime-valid inference.

[9] establish the modern theory of e-values, including their calibration and combination properties. [10] prove minimax optimality of likelihood-ratio e-values under composite hypotheses. [11] provide a comprehensive tutorial connecting e-values to game-theoretic probability and optional stopping. [17] construct time-uniform confidence sequences from nonnegative supermartingales; their construction is dual to ours, building confidence intervals rather than tests. E-SHIFT instantiates this vocabulary specifically for covariate shift detection in VLM deployment streams.

3.0.0.3 Betting-based sequential two-sample tests.

[23] construct anytime-valid two-sample tests by framing testing as a sequential game in which a Skeptic bets on differences between two samples. Their tests have the same time-uniform type-I error guarantee as our e-process and can be applied to VLM feature vectors. We include a betting-based sequential MMD (SeqMMD-E) as a baseline that shares the anytime-valid framework but uses a simpler scalar feature score, allowing us to isolate the contribution of the composite score design.

3.0.0.4 Distribution shift detection in ML.

Kernel MMD [5] and learned deep kernels [24] detect shift in offline, fixed-sample settings with well-controlled power. [3] provide a comprehensive empirical comparison of shift-detection methods, finding that multivariate tests on learned representations outperform marginal tests. The WILDS benchmark [2] provides a diverse evaluation platform for methods that generalize across distribution shifts; E-SHIFT addresses the detection problem (knowing when shift has occurred), which is logically prior to the adaptation problem studied in WILDS.

3.0.0.5 Conformal prediction and non-conformity scores.

Conformal prediction [25], [26] constructs prediction sets with finite-sample coverage guarantees using non-conformity scores that measure how unusual a new point is relative to the calibration set. [27] extend conformal prediction to covariate-shift settings. E-SHIFT borrows the non-conformity score as its test statistic but embeds it inside an e-process, gaining time-uniform type-I error control that standard conformal tests, which fix the calibration level in advance, cannot provide.

3.0.0.6 Test-time adaptation.

Once a shift is detected, adaptation methods such as continual test-time domain adaptation [28] attempt to correct the model’s predictions without retraining. E-SHIFT is complementary to these methods: it provides the detection alarm that triggers the adaptation step in a detect-then-adapt pipeline.

4 Problem Formulation↩︎

4.0.0.1 The sequential stream.

Let \(f_\theta\) be a frozen VLM with image encoder \(f_v\) and predictive distribution \(p_\theta(y \mid x)\) over \(K\) classes. At each time step \(t = 1, 2, \ldots\) the model processes an incoming observation \(x_t\); no label \(y_t\) is observed. The observations are drawn from a distribution that may change at an unknown time \(t^\star \in \{0, 1, 2, \ldots\}\).

4.0.0.2 Hypotheses.

Let \(P_0\) denote the null (in-distribution) distribution of each observation. The null and alternative hypotheses are: \[\begin{align} \mathcal{H}_0 &: x_1, x_2, \ldots \overset{\text{iid}}{\sim} P_0 \quad (\text{no shift}), \\ \mathcal{H}_1 &: \exists\, t^\star \ge 1 \text{ such that } x_t \sim P_0 \text{ for } t \le t^\star \text{ and } x_t \sim P_1 \ne P_0 \text{ for } t > t^\star. \end{align}\]

4.0.0.3 Calibration data.

A held-out calibration set \(\mathcal{C} = \{x_1^c, \ldots, x_{n_c}^c\}\) of \(n_c\) observations drawn i.i.d.from \(P_0\) is available before the stream begins. No observation from the stream is used to construct \(\mathcal{C}\), and \(\mathcal{C}\) is consumed only once to compute summary statistics.

4.0.0.4 Decision rule and error types.

The detector maps the stream and calibration data to a sequence of binary decisions (alert / no alert). An alert issued at time \(t \le t^\star\) is a false alarm; an alert issued at \(t > t^\star\) is a true detection. The false-alarm rate (FAR) is the probability of issuing any false alarm. The detection delay is \(T(\tau) - t^\star\) where \(T(\tau) = \inf\{t \ge 1 : M_t \ge \tau\}\).

4.0.0.5 Anytime-valid requirement.

The central requirement is a time-uniform false-alarm bound: \[\mathbb{P}_{\mathcal{H}_0}\!\Bigl(\sup_{t \ge 1} M_t \ge \tau\Bigr) \le \frac{1}{\tau}, \label{eq:avreq}\tag{3}\] which must hold simultaneously at all stopping times, including those that depend on the stream. This is strictly stronger than controlling the FAR at a fixed sample size.

5 E-SHIFT: Method↩︎

5.1 Non-Conformity Scores from VLM Outputs↩︎

5.1.0.1 Motivation.

An e-process requires a scalar score \(S_t\) whose distribution under \(\mathcal{H}_0\) can be estimated from the calibration set, and that takes larger values when the input is out-of-distribution. Designing such a score for a VLM involves a tradeoff: a predictive-distribution term reacts quickly to shift that changes the model’s class probabilities, but is blind to shifts that alter the feature space without affecting the argmax; a feature-space term detects the latter but may be slow when the shift is small in embedding space. We combine both terms.

5.1.0.2 Predictive-shift term.

Recall from Section 2.5 that CLIP’s softmax distribution becomes overconfident or near-uniform under covariate shift. We measure deviation of \(p_\theta(y \mid x_t)\) from the maximum-entropy distribution (uniform over \(K\) classes) using the KL divergence: \[\mathrm{KL}\!\bigl(p_\theta(y \mid x_t) \,\|\, \mathrm{Uniform}_K\bigr) = \log K - H\!\bigl(p_\theta(y \mid x_t)\bigr), \label{eq:klentropy}\tag{4}\] where \(H\) denotes Shannon entropy [29]. This equals zero when the model is maximally uncertain and increases as the prediction becomes more peaked. Because covariate shift tends to produce either overconfident or underconfident posteriors, this term responds to shift in both directions: overconfidence increases it, while extreme underconfidence reduces it, but the complementary feature-space term covers the underconfident case.

5.1.0.3 Feature-shift term.

Under \(\mathcal{H}_0\), the image embeddings \(f_v(x_t)\) cluster near the calibration centroid \(\mu_{\mathrm{cal}} = \frac{1}{n_c}\sum_j f_v(x_j^c)\). The Mahalanobis distance [30] measures how many calibration standard deviations the embedding \(f_v(x_t)\) lies from this centroid: \[d_M(x_t) = \bigl\|f_v(x_t) - \mu_{\mathrm{cal}}\bigr\|^2_{\Sigma_{\mathrm{cal}}^{-1}}, \label{eq:mahal}\tag{5}\] where \(\Sigma_{\mathrm{cal}}\) is the empirical feature covariance on \(\mathcal{C}\). By standardizing with the calibration covariance, this distance is scale-invariant and correctly weights dimensions with high calibration variance.

5.1.0.4 Composite score.

The composite non-conformity score is \[S_t = \mathrm{KL}\!\bigl(p_\theta(y \mid x_t) \,\|\, \mathrm{Uniform}_K\bigr) + w\,\bigl\|f_v(x_t) - \mu_{\mathrm{cal}}\bigr\|^2_{\Sigma_{\mathrm{cal}}^{-1}}, \label{eq:score}\tag{6}\] where \(w > 0\) is a feature weight. The calibration-set distribution of \(S_t\) under \(\mathcal{H}_0\) can be estimated nonparametrically from \(\mathcal{C}\) without any parametric assumption on \(p_\theta\).

5.2 E-Process Construction↩︎

5.2.0.1 Required conditions.

We state the conditions under which the theoretical guarantees hold.

Assumption 1(a) requires temporally independent observations; this is reasonable for i.i.d.image streams but may fail for video or other temporally correlated data. We address the mixing case in Section 6.3. Assumption 1(c) is a mild sub-exponential condition on the score distribution; we verify it empirically in Appendix 10.

5.2.0.2 Exponential e-process.

Let the log-MGF under \(\mathcal{H}_0\) be \(\psi(\lambda) = \log \mathbb{E}_{P_0}[e^{\lambda(S - \hat{\mu})}]\). The running product \[M_t = \prod_{i=1}^{t} \exp\!\bigl(\lambda(S_i - \hat{\mu}) - \psi(\lambda)\bigr), \qquad M_0 = 1, \label{eq:eprocess}\tag{7}\] is a nonnegative supermartingale under \(\mathcal{H}_0\) [15], because each multiplicative factor has \(\mathbb{E}_{\mathcal{H}_0}[\,\cdot\,] = 1\) by construction of \(\psi\). When \(M_t \ge \tau\) we declare shift and issue an alert.

5.2.0.3 Selecting \(\lambda\).

The parameter \(\lambda > 0\) governs detection sensitivity. Larger \(\lambda\) amplifies each score increment, yielding faster detection under strong shift but poorer type-I error calibration under weak shift. In experiments we select \(\lambda\) by maximising the empirical growth rate \(\lambda(\hat{\mu}_1 - \hat{\mu}) - \hat{\psi}(\lambda)\) on a small held-out shift calibration set \(\mathcal{C}_{\mathrm{shift}}\) (\(n_s = 200\) samples from a known shifted domain). When no shifted data are available, the plug-in rule \(\lambda_{\mathrm{init}} = 1/\hat{\sigma}^2\) (where \(\hat{\sigma}^2\) is the empirical variance of \(S\) on \(\mathcal{C}\)) performs within \(4\%\) of the oracle delay on our benchmarks (Appendix 13).

5.3 Finite-Sample Bootstrap Correction↩︎

With finite calibration data, the true \(\psi(\lambda)\) is unknown. The plug-in estimate \(\hat{\psi}(\lambda) = \log\hat{m}(\lambda)\), where \(\hat{m}(\lambda) = \frac{1}{n_c}\sum_j e^{\lambda(S_j^{\mathrm{cal}} - \hat{\mu})}\), underestimates \(\psi(\lambda)\) in expectation because the sample mean is a biased estimator of the log-mean-exponential. Using \(\hat{\psi}\) in place of \(\psi\) makes each factor of the product 7 have expectation slightly greater than one, inflating the false-alarm rate.

We restore validity with a bootstrap upper-confidence bound. Draw \(B\) bootstrap resamples \(\mathcal{C}^{(1)}, \ldots, \mathcal{C}^{(B)}\) from \(\mathcal{C}\) with replacement and compute \[\hat{m}^{(b)}(\lambda) = \frac{1}{n_c}\sum_j e^{\lambda(S_j^{(b)} - \hat{\mu})}, \qquad b = 1, \ldots, B.\] Let \(\bar\psi(\lambda) = \log m_{(1-\beta)}(\lambda)\) be the empirical \((1-\beta)\)-quantile of \(\{\hat{m}^{(b)}\}\). The corrected e-process replaces \(\psi\) with \(\bar\psi\): \[M_t = \prod_{i=1}^{t} \exp\!\bigl(\lambda(S_i - \hat{\mu}) - \bar\psi(\lambda)\bigr). \label{eq:corrected}\tag{8}\] By the bootstrap consistency for the distribution of \(\hat{m}(\lambda) - m(\lambda)\) [8], [15], we have \(\mathbb{P}(\psi(\lambda) \le \bar\psi(\lambda)) \to 1 - \beta\) as \(n_c \to \infty\), justifying the approach asymptotically.

5.3.0.1 Algorithm.

After each alert, the process is reset to \(M_t = 1\), so the detector remains active for subsequent shifts. We analyse the false-alarm accumulation from multiple resets in Proposition 2.

6 Theoretical Analysis↩︎

6.1 Time-Uniform False-Alarm Control↩︎

6.2 Reset Accumulation↩︎

For \(\tau = 200\) and a stream of \(T = 10{,}000\) steps, the expected number of false alarms is at most \(10{,}000 / 40{,}000 = 0.25\), making false alarms exceedingly rare in practice.

6.3 Detection Delay under Sustained Shift↩︎

Define the post-shift e-process growth rate: \[\Gamma = \sup_{\lambda > 0}\Bigl\{ \lambda\bigl(\mathbb{E}_{P_1}[S] - \hat{\mu}\bigr) - \psi(\lambda) \Bigr\}. \label{eq:gamma}\tag{10}\] This is the Legendre transform of \(\psi\) evaluated at the mean shift.

7 Experiments↩︎

7.1 Experimental Setup↩︎

7.1.0.1 Model.

We use CLIP ViT-B/16 [4] as the frozen VLM backbone. All non-conformity scores are computed from its softmax output distribution (for the KL term) and its image embeddings (for the Mahalanobis term) without any fine-tuning.

7.1.0.2 Benchmarks.

We evaluate on three standard distribution shift benchmarks:

• ImageNet-C [32]: 15 corruption types (noise, blur, weather, digital) at 5 severity levels applied to ImageNet validation images. Each corruption type represents a distinct covariate shift.

• ImageNet-R [21]: 30-class subset of ImageNet rendered in artistic styles (art, cartoons, deviantart, etc.), representing domain shift from the photographic distribution.

• Office-Home [33]: 65-class dataset spanning four visual domains (Artistic, Clipart, Product, Real-World), commonly used for domain adaptation evaluation.

Each test stream concatenates samples from multiple corruption types or domains in a random temporal order; no domain labels are provided to the detector at test time.

7.1.0.3 Hyperparameters and calibration.

• Calibration set: \(n_c = 500\) images drawn i.i.d.from the pre-shift distribution (in-distribution ImageNet validation images for ImageNet-C/R; Artistic domain images for Office-Home).

• Shift calibration set: \(n_s = 200\) images from a held-out shifted domain for selecting \(\lambda\).

• Bootstrap: \(B = 1000\) resamples; bootstrap level \(\beta = 0.005\).

• Threshold: \(\tau = 200\), targeting \(\le 1\%\) false-alarm via Proposition 1 and Remark 1.

• Feature weight: \(w = 1.0\) selected by cross-validation on \(\mathcal{C}_{\mathrm{shift}}\).

7.1.0.4 Baselines.

• Fixed-MMD [5]: kernel two-sample test applied at a fixed window of 100 samples. This is the canonical offline method and violates anytime validity.

• CUSUM [12]: parametric sequential detector assuming Gaussian score increments.

• Shiryaev–Roberts (SR) [22]: Bayesian sequential detector with Gaussian likelihood ratio.

• SeqMMD-E [23]: betting-based sequential MMD applied to CLIP feature vectors. This baseline shares the anytime-valid framing but uses a simpler scalar score (median-kernel MMD on the features), allowing comparison of score designs within the e-process framework.

• E-SHIFT (no bootstrap): ablation using the plug-in \(\hat{\psi}(\lambda)\) instead of the corrected \(\bar\psi(\lambda)\), testing the necessity of the bootstrap correction.

7.1.0.5 Metrics.

Detection Delay: average number of samples between the true shift point \(t^\star\) and the alert time \(T(\tau)\), conditional on detection within 1000 steps. FAR: fraction of alerts issued on strictly in-distribution segments of the stream, interpreted against the theoretical target of \(\le 1\%\). We report means and standard deviations over 10 independent random shift orderings.

7.2 Main Results↩︎

Table 1 shows that E-SHIFT achieves the lowest detection delay and is the only method whose FAR falls within the theoretical \(\le 1\%\) target. The results admit the following interpretation.

Fixed-MMD is the slowest and most conservative method. Its high FAR (\(4.7\%\)) is expected: by applying a fixed critical value at a data-adaptive stopping time, it lacks type-I error control, as discussed in Section 1.

CUSUM and Shiryaev–Roberts are faster than Fixed-MMD because they exploit the sequential structure, but both exceed the \(1\%\) FAR target. This is attributable to their parametric Gaussian assumption on the score distribution: VLM non-conformity scores have heavier tails than Gaussian (see Appendix 10), causing the likelihood ratio to overstate evidence at extreme values.

SeqMMD-E shares the anytime-valid framing with E-SHIFT and correctly controls FAR (\(0.95\%\)), but is \(18\%\) slower (\(20.3\) vs.\(17.2\) samples). This gap is attributable to the score design: SeqMMD-E uses a scalar kernel-based score that does not differentiate between predictive-distribution shift and feature-space shift, whereas the composite score 6 combines complementary signals.

The no-bootstrap ablation (\(\mathrm{FAR} = 1.2\% > 1\%\)) directly validates Proposition 1: the plug-in log-MGF underestimates the true moment, inflating the false-alarm rate above the target. The bootstrap correction restores validity, with the corrected process achieving \(0.90\%\) FAR.

7.3 Score Ablation↩︎

Table 2 shows that the composite score at \(w = 1.0\) achieves the lowest detection delay. Single-term variants are both slower: KL-only (\(22.4\) samples) misses feature-space drift; Mahal-only (\(20.1\) samples) misses predictive distribution shift. The delay is flat within \(2\) samples for \(w \in [0.5, 2.0]\), indicating that the method is not sensitive to the precise value of the feature weight. Extreme weights (\(w = 0.1\) or \(w = 10.0\)) recover much of the single-term behaviour.

7.4 Per-Benchmark and Calibration Analysis↩︎

Figure 2 shows that E-SHIFT is fastest on all three benchmarks while maintaining FAR \(\le 1\%\). The delay varies across benchmarks in a pattern that is consistent with the composite score design: on ImageNet-C, where corruptions produce strong feature-space drift, the Mahalanobis term dominates and delays are shorter; on Office-Home, where domain shift is more subtle at the feature level, the KL term carries more weight and delays are longer but remain competitive.

Figure 3 validates the theoretical guarantee of Theorem 1 empirically. Sweeping \(\tau\) from \(20\) to \(500\), the empirical FAR of E-SHIFT lies uniformly at or below the theoretical bound \(\beta + 1/\tau\) across all tested values. The no-bootstrap ablation systematically exceeds the bound at every \(\tau\), confirming that the bootstrap correction is not merely a cosmetic fix but is necessary for the e-process to be a valid supermartingale on finite calibration data.

8 Discussion↩︎

8.1 Theoretical Limitations↩︎

8.1.0.1 IID assumption.

Theorem 1 requires i.i.d.scores under \(\mathcal{H}_0\). In streaming video, sensor logs, or other temporally correlated sources, consecutive frames or measurements are typically not independent. The \(\alpha\)-mixing extension of Theorem 2 provides asymptotic coverage under geometric ergodicity, but summable mixing coefficients are rarely verified in practice. A rigorous treatment of temporal dependence within the e-process framework, possibly via the dependent versions of Ville’s inequality [11], is an important open problem.

8.1.0.2 Bootstrap asymptotic validity.

Proposition 1 provides only asymptotic validity for the bootstrap construction. A tighter, finite-sample bound, for example using Bernstein-style inequalities or confidence sequences on the MGF, would strengthen the guarantee and is an open direction.

8.1.0.3 Shift type scope.

The composite score 6 is designed to detect covariate shift. Label shift [19], where \(p(y)\) changes but \(p(x \mid y)\) does not, may not produce large values of either term: the model’s outputs change in proportion to the class frequencies, but each individual prediction may be well-calibrated. Concept drift, where \(p(y \mid x)\) itself changes, requires access to labels and is outside the scope of the present work.

8.1.0.4 Domain mismatch for \(\lambda\).

The e-value parameter \(\lambda\) is tuned on a held-out shifted calibration set \(\mathcal{C}_{\mathrm{shift}}\). If the deployment-time shift domain differs substantially from this set, the realised growth rate \(\Gamma\) may be smaller, increasing detection delay. The plug-in heuristic \(\lambda_{\mathrm{init}} = 1/\hat{\sigma}^2\) mitigates this by requiring only in-distribution data.

8.2 Practical Considerations↩︎

8.2.0.1 Computational cost.

At each time step, E-SHIFT requires one forward pass through the VLM (which is already computed for the application), one matrix-vector product for the Mahalanobis distance (cost \(\mathcal{O}(d^2)\) for feature dimension \(d\)), and one scalar update of \(M_t\). The bootstrap correction is computed once at calibration time with cost \(\mathcal{O}(Bn_c)\). The per-step cost is negligible relative to the VLM inference.

8.2.0.2 Calibration set requirements.

A calibration set of \(n_c = 500\) samples from the in-distribution is sufficient in our experiments. When such data are unavailable, e.g.in purely online settings where no pre-shift baseline exists, a self-calibrating variant based on running score quantiles could be used, at the cost of the theoretical guarantees.

8.2.0.3 Detect-then-adapt integration.

E-SHIFT is designed as a detection module that integrates naturally into detect-then-adapt pipelines [28]. The alert signal triggers an adaptation step (fine-tuning, prompt tuning, or normalization update), while E-SHIFT continues monitoring after reset. The \(K/\tau\) accumulation bound (Proposition 2) ensures that repeated false alarms causing unnecessary adaptation are rare for large \(\tau\).

8.3 Future Directions↩︎

Several extensions are natural. First, extending the composite score to label shift and concept drift would require either label observations or a proxy for \(p(y \mid x)\) change; conformal prediction methods [26] may provide relevant non-conformity scores. Second, a fully self-calibrating variant that estimates the null score distribution from the running stream, without a separate calibration phase, would be applicable in purely online settings. Third, a theoretical treatment of the detection-adaptation loop, quantifying how post-adaptation score distributions affect E-SHIFT’s subsequent performance, would provide end-to-end guarantees for detect-then-adapt pipelines.

9 Conclusion↩︎

We presented E-SHIFT, a framework for anytime-valid sequential hypothesis testing for distribution shift in streaming vision-language systems. The method builds an e-process from a composite non-conformity score that tracks shift in both the predictive distribution and the feature embedding of a frozen VLM, inheriting time-uniform type-I error control via Ville’s maximal inequality.

Three theoretical contributions underpin the framework. Theorem 1 establishes the time-uniform false-alarm bound \(\mathbb{P}_{\mathcal{H}_0}(\sup_t M_t \ge \tau) \le 1/\tau\), which holds simultaneously at every stopping time. Proposition 1 extends this guarantee to finite calibration sets via a bootstrap MGF correction, asymptotically valid as calibration-set size grows. Proposition 2 bounds false-alarm accumulation over multiple resets. Theorem 2 links detection speed to information divergence via the \(\mathcal{O}(\log\tau / \Gamma)\) delay bound.

Empirically, E-SHIFT achieves the lowest detection delay on ImageNet-C, ImageNet-R, and Office-Home while being the only method to satisfy the \(1\%\) FAR target across all benchmarks. The calibration curve confirms that the theoretical bound is empirically tight, and the score ablation and SeqMMD-E comparison isolate the contributions of the composite score design and the anytime-valid framing separately.

10 Sub-Exponential Score Verification↩︎

To verify Assumption 1(c), we estimate the sub-exponential parameters of \(S - \hat{\mu}\) from the calibration scores on each benchmark. Using the empirical cumulant-generating function [31], we fit the sub-exponential parameters \((\nu^2, c)\) defined by \(\psi(\lambda) \le \nu^2\lambda^2/2\) for \(|\lambda| < 1/c\).

At \(\lambda = 0.38\), \(\lambda c < 1\) in all cases, confirming that the MGF is finite at the operating parameter and Assumption 1(c) holds. The positive \(\hat{\psi}(\lambda)\) values indicate scores with moderate sub-exponential tails, consistent with the bootstrap correction used in place of tighter parametric alternatives.

11 Calibration Curve (Numerical)↩︎

12 Per-Corruption Breakdown↩︎

Noise corruptions (Gaussian, shot, impulse) produce the strongest feature-space displacement from the calibration centroid, yielding the shortest detection delays. Digital corruptions (JPEG, pixelation) are subtler in feature space and rely more on the KL term.

13 \(\lambda\)-Free Heuristic Evaluation↩︎

The plug-in rule \(\lambda_{\mathrm{init}} = 1/\hat{\sigma}^2\) incurs only a \(4\%\) delay overhead relative to the oracle (\(17.9\) vs. \(17.2\) samples) while requiring no shifted calibration data, making it a reliable default in deployment settings where shifted samples are unavailable.

References↩︎