1 Introduction↩︎

Modern scientific instruments generate large volumes of data as a continuous stream that must be processed in near real-time to guide ongoing experiments. Examples include synchrotron radiation facilities [1]–[4], large-scale sensor deployments [5], and online scientific simulations [6], [7], where data are continuously acquired and processed using multi-stage workflows to produce timely feedback. Unlike traditional batch-oriented scientific workflows, these applications operate on unbounded data streams and must continuously produce meaningful results within strict time constraints [8], [9]. As a result, scientific streaming workflows have emerged as a critical paradigm for enabling real-time scientific discovery [10], [11].

However, achieving reliable execution for such workflows remains challenging. These workflows are typically deployed on large-scale, distributed infrastructures where failures and performance variability are unavoidable. Fail-stop events, such as process crashes or node failures, interrupt execution and can force costly recomputation if progress is not preserved. Additionally, even if processes do not abruptly end due to fail-stop events, they may be subject to performance degradation due to various other anomalies: communication delays and timeouts, transient interference due to competition for shared resources, misconfigured software stacks, etc. Even if such anomalies are localized, their effect can quickly propagate in the workflow due to dependencies, leading to quality-of-service loss and/or increase in end-to-end latency. These challenges become increasingly severe as the system scale grows, where failures and performance variability become the norm rather than exception [12], [13].

1.0.0.1 Limitations of state-of-the-art.

Traditional HPC resilience techniques emphasize global checkpointing and recovery [14], which is appropriate for tightly coupled, bulk-synchronous simulations. In this case, there is an implicit assumption that a safe point can be eventually reached (e.g., end of iteration), where a global barrier ensures there are no more in-flight messages, and simplifies the capture of a globally consistent state as a set of individual checkpoints corresponding to each process. However, stream processing workflows are typically modeled using producer-consumer patterns that are loosely coupled. Since the data are constantly in motion, identifying an opportunity to capture individual checkpoints may not be possible or may introduce excessive coordination overheads. Conversely, many stream-processing frameworks, such as Flink [15], are not designed to run on HPC machines. They provide built-in resilience techniques tightly coupled with their execution models [15], [16] and are often too conservative (e.g., based on Chandy-Lamport [17] algorithm), thereby incurring high overheads. This limits flexibility and makes them difficult to adopt in specialized scientific workflows. Furthermore, such techniques typically optimize for throughput [18] or per-partition latency [19] rather than prioritizing end-to-end latency. Thus, there is a need to develop dedicated resilience solutions that overcome these limitations.

1.0.0.2 Key insights and contributions.

In this paper, we address this gap by rethinking resilience from the perspective of scientific streaming workflows. Instead of enforcing workflow-wide coordination or embedding resilience into a specific framework, we design resilience mechanisms as modular components operating at the producer–consumer level. This design leverages the embarrassingly parallel nature of scientific streaming workloads, allowing failures and performance anomalies to be handled independently with minimal coordination. By decoupling resilience from execution frameworks and avoiding global synchronization, our approach enables resilience to be introduced without sacrificing scalability or real-time performance.

We present StreamGuard, a resilience solution for real-time scientific streaming with two complementary techniques: (i) dynamic, asynchronous, non-blocking checkpointing for low-overhead failure recovery, and (ii) progress-aware load redistribution that detects slowdown anomalies and rebalances data to satisfy real-time constraints. While each builds on established ideas, our contribution is their integration into a unified, modular design at the producer–consumer boundary, jointly mitigating failures and slowdowns without the workflow-wide coordination required by prior approaches.

We evaluate StreamGuard using a real-world scientific streaming workflow for tomographic reconstruction used at synchrotron radiation facilities [20], [21]. Experimental results demonstrate that StreamGuard reduces the impact of failures and slowdown anomalies by up to \(6\times\) while introducing less than 1% overhead in failure-free execution. Across a wide range of scales, failure rates, and anomaly conditions, StreamGuard’s design consistently maintains processing time close to ideal execution and enables workflows to meet real-time deadlines where existing approaches fail. We summarize our contributions as follows:

1. A unified, decoupled producer–consumer–centric resilience design that integrates dynamic non-blocking checkpoint-and-retry with progress-aware load balancing as composable modules, avoiding workflow-wide coordination, reducing performance degradation from failures and slowdowns by up to \(4\times\) and \(5\times\), respectively (§5).

2. A modular resilience architecture for real-time scientific streaming that allows applications to selectively enable resilience mechanisms as needed. When combined, these modules effectively prevent compounded performance degradation, with performance increase to at most 40% under extreme failure and anomaly conditions at scale (§6).

3. A systematic evaluation using a real-world scientific streaming application that demonstrates improved robustness, scalability, and performance transparency compared to state-of-the-art HPC and stream-processing resilience approaches, while consistently meeting real-time constraints in scenarios where existing solutions may fail (§7).

2 Background↩︎

We consider a large class of streaming scientific applications that follow a producer-consumer pattern as illustrated in 1. The workflow begins at a scientific instrument, such as a synchrotron radiation facility, that continuously generates large volumes of data during experiments. These data are transferred to HPC systems for processing through a sequence of stages, where each stage consists of multiple parallel workers processing different portions of the data stream (i.e., partitions).

2.1 Producer-Consumer Streaming↩︎

The communication between the stages follows a producer–consumer model. Upstream workers act as producers by emitting data records, while downstream workers act as consumers that receive and further process incoming data records. Depending on application requirements, the data records may be distributed among consumers through different patterns, such as shuffle, where records are scattered across consumers, or broadcast, where every record is delivered to all consumers. Regardless of the distribution pattern, data flowing between producers and consumers is typically divided into sequential partitions with unique identifiers (or keys). These partitions can be processed independently, enabling efficient parallel execution over large-scale data streams.

After processing, consumers may generate intermediate outputs, persist state, or provide feedback to upstream stages. The outputs are then forwarded downstream, where the same producer–consumer interaction repeats. Because most computation and data movement occur at these boundaries, the producer–consumer pair forms the primary unit where performance variability, failures, and anomalies directly affect end-to-end workflow behavior.

2.2 Key Characteristics↩︎

Building on the producer–consumer pattern described earlier, scientific streaming workflows exhibit characteristics that distinguish them from both traditional HPC and stream-processing workloads.

2.2.0.1 Unbounded data stream.

Data are continuously generated by instruments or edge devices as an unbounded sequence of records. Each record captures only a small portion of the evolving experiment, and the full outcome emerges gradually as more data arrive. This differs fundamentally from traditional HPC workloads, where the complete dataset is typically available in a parallel file system before processing begins. In scientific streaming, workers must operate on incomplete and continuously evolving data.

2.2.0.2 Iterative refinement.

Because data arrive incrementally, early outputs are necessarily approximate. As additional data are consumed, workers iteratively refine their results, steadily improving accuracy and quality. This pattern is common in applications such as iterative reconstruction [20], [22] and long-term campaigns [23]–[25]. Unlike conventional stream-processing systems that handle many independent, small, and fine-grained records, scientific streams often consist of coarse-grained, high-volume data units that collectively represent meaningful scientific states.

2.2.0.3 Embarrassingly parallel processing.

Streamed partitions can be processed independently by many parallel workers with minimal coordination. As a result, scientific streaming workflows often rely on workers deployed across multiple compute nodes, leveraging their collective power to accelerate stream processing.

2.2.0.4 Real-time quality constraints.

Although the stream is continuous, processing must produce useful results within practical time limits imposed by system policies or experimental needs. Therefore, the goal is not merely to process all incoming data, but to maximize output quality within a bounded execution window—a notion often captured as Quality of Service (QoS).

3 Problem Formulation↩︎

This paper addresses the resilience challenges of the producer-consumer pattern. We model data stream \(S\) between a producer and a consumer as a set of independent partitions \(S = \{s_{1},s_{2},...,s_{N}\}\) where a partition \(s_{i}\) is the set of records that need to be processed together in a sequential order (e.g., data collected by a single device over time) identified by a unique key \(k_{i}\). Let \(w_{1}, \dots, w_{M}\) denote the available workers (e.g., processes or compute nodes) allocated to the consumer to process data. We assume \(M \leq N\), meaning that each worker may process multiple partitions. We also assume that data partitions have similar computational demand, which can be achieved through appropriate data partitioning methods [26], [27]. In contrast, workers are not assumed to be identical, as their performance may vary due to hardware differences, resource contention, or dynamic runtime conditions. To flexibly work around this heterogeneity, we allow data partitions to be reassigned from one worker to another and let \(a(s_{i}, w_{j}, t)\) be an indicator function specifying whether the data partition \(s_{i}\) is assigned to the worker \(w_{j}\) at time \(t\):

\[a(s_{i}, w_{j}, t) = \begin{cases} 1, & \text{if } s_{i} \text{ is assigned to } w_{j} \text{ at } t\\ 0, & \text{otherwise} \end{cases}\] We assume time is discretized into time units in which partition-to-worker assignment remain unchanged. Since consumer workers process data in an iterative manner, we define partition progress \(p_{i}(t)\) as the number of records that have been processed over the data partition \(s_{i}\) by the time \(t\). Likewise, worker progress, \(q_{j}(t)\), is the total progress a worker \(w_{j}\) has made on its assigned partitions: \[q_{j}(t) = \sum_{i=1}^{N}\sum_{\tau=1}^{t}(p_{i}(\tau)-p_{i}(\tau-1))a(s_{i}, w_{j},\tau) \label{equ:worker-progress}\tag{1}\]

Compared to traditional batch processing, streaming workflows can produce usable outputs without waiting for the complete dataset. Initial results can be generated after a few processing iterations and progressively refined as additional data arrive. This property makes streaming well-suited for real-time applications, where consumers must produce meaningful outputs within a specified time window. We model this real-time requirement using a deadline \(L\) and a progress target \(P\). The real-time constraint is satisfied if all data partitions reach at least progress level \(P\) by the deadline \(L\): \[\forall i : p_{i}(L) \geq P \label{equ:real-time-constraint}\tag{2}\] The notations used in this formulation are summarized in 1 and are used throughout the remainder of the paper.

The producer-consumer pattern often requires massive parallelism to process large data streams, so the workflow can be highly sensitive to failures and performance anomalies. We consider two common classes of disruptions.

3.0.0.1 Fail-stop events

A fail-stop event occurs when a worker abruptly stops execution due to errors such as segmentation faults or out-of-memory conditions [28]. Fail-stop failures are often modeled as random processes [29], where workers fail independently with identical probabilities. These failures are characterized by the mean time to failure (MTTF), denoted as \(\mu\), which represents the average time a single worker runs before failing. Worker failure probability is typically described using an exponential distribution Exp(\(\lambda\)), where \(\lambda = \frac{1}{\mu}\) represents the failure rate [14]. The mean time between failures (MTBF) for the whole consumer is given by: \[\mu_{p} = \frac{\mu}{M}\,. \label{equ:app-mtbf}\tag{3}\]

3 shows that increasing parallelism reduces MTBF. This is intuitive, as more workers introduce more potential points of failure, causing failures to occur more frequently. This phenomenon poses a significant challenge for achieving resilient execution at large scales. For instance, if the MTTF of a single worker is one day, a consumer with 1000 workers would, on average, fail in less than 2 min—often not long enough for the consumer to provide any meaningful results [20]. Moreover, even with workers running on infrastructures designed to operate reliably for years, the sheer scale of large systems amplifies failure rates. Studies have shown that large-scale deployments, with thousands of compute nodes, can experience failures several times per day due to the aggregated likelihood of individual node failures [12], [30].

3.0.0.2 Anomalous performance degradation.

Not all performance incidents result in complete worker failures. In many cases, long-lasting conditions such as network congestion, resource contention, hardware interference, or inherent system heterogeneity reduce worker performance but do not stop execution. As a result, data streams assigned to affected workers progress more slowly than others. We refer to this phenomenon as a slowdown anomaly. Because real-time execution requires all streams to reach a minimum progress threshold (as defined in 2 ), overall consumer performance becomes dominated by the slowest stream.

Slowdown anomalies can have severe consequences if left unmitigated. Empirical studies of large-scale distributed and cloud systems have shown that worker execution times commonly exhibit heavy-tailed or power-law-like distributions under heterogeneous environments [31], [32]. In such settings, the majority of workers progress at similar speeds, while a small fraction experiences disproportionately longer execution times. This behavior is widely known as the straggler phenomenon [33], [34], where a few slow workers dominate overall completion time despite representing only a small portion of the system.

The impact of slowdown anomalies is particularly pronounced in scientific streaming workflows. Since outputs are only meaningful when progress across all partitions reaches a required level, the slowest stream effectively determines whether the real-time deadline can be satisfied. As system scale increases, the probability of encountering slow workers also increases, amplifying the likelihood that slowdown anomalies degrade overall performance. These characteristics make anomaly mitigation a critical requirement for maintaining reliable real-time execution in scientific streaming environments.

4 Related Work↩︎

4.0.0.1 Coordinated Checkpoint-Restart

HPC resilience is dominated by the checkpoint-restart (C/R) paradigm [35], necessitated by the tightly coupled nature of parallel applications. Because such applications typically use MPI for inter-process communication, the failure of a single process invalidates the global state and requires all nodes to roll back to a coordinated recovery point. Recent research has shifted from global blocking checkpoints toward multilevel hierarchies that adapt to failure severity, spanning local memory tiers (GPU HBM, host memory, SSDs), erasure coding, buddy replication, burst buffers, and external object stores such as DAOS [36]. Prominent examples in this space are VELOC [37], [38], SCR [39], FTI [40], and ADIOS [41]. A key optimization is to enable asynchronous capture and flushing of checkpoints in order to hide I/O overheads and various other transformations (data aggregation [42], compression [43], etc.). However, despite extensive work in reducing checkpoint overhead, these approaches implicitly assume a safe point at which a global barrier ensures no in-flight messages exist (e.g., end of iteration). In stream-processing workflows, such a safe point does not naturally exist and forcing one requires expensive synchronization (§2.2), making global C/R infeasible.

4.0.0.2 Resilient Stream Processing

Conversely, stream-computing resilience leverages the loosely coupled nature of producer–consumer patterns through checkpoint-replay mechanisms derived from the Chandy-Lamport algorithm [17]. Unlike HPC’s rigid synchronization, stream processors such as Google Cloud Dataflow [19] and Apache Flink [15], [44] inject barrier markers into the data stream to capture asynchronous distributed snapshots that establish a consistent recovery boundary accounting for in-flight messages, and replay input logs on failure for exactly-once semantics. Alternatively, Spark uses lineage-based micro-batch checkpointing [45], [46], while Storm and Heron use tuple acknowledgment and replay [16], [47]. Data-intensive middleware can cache compact reduction objects and redistribute unfinished work after failures [48]. These mechanisms decouple operator recovery from overall latency, supporting high throughput under transient faults. However, such approaches are often too conservative in that checkpoint capture is subject to slow serialization and synchronous I/O, while replay is done in a rigid fashion, without load-balancing or straggler mitigation strategies, which ultimately propagates delays through the entire workflow both during failure-free and faulty regimes of operation.

4.0.0.3 Mitigation of Non-Fail-Stop Anomalies and Gray Failures

A critical frontier in resilience research involves the management of non-fail-stop anomalies, often referred to as “gray failures”, where nodes remain operational but exhibit sub-optimal performance [49]. These issues stem from resource contention, network jitter, or software misconfigurations, and are particularly disruptive because they propagate delays through downstream dependencies, leading to significant tail latencies [31]. Unlike fail-stop errors, these anomalies cannot be resolved by simple restarts. Instead, modern resilience frameworks employ proactive mitigation techniques, such as speculative execution [33], [34] (launching redundant “straggler” tasks) and dynamic load balancing [50]–[52]. By utilizing observability-driven feedback loops, these systems detect performance deviations and rebalance workloads in real-time, preventing a localized slowdown from cascading into a systemic bottleneck. However, such techniques are often oblivious to end-to-end QoS objectives and latency considerations.

5 StreamGuard: Resilient HPC Streaming↩︎

We propose StreamGuard, a resilience solution that combines two complementary mechanisms to protect against failures and performance anomalies while preserving the performance characteristics required by real-time scientific streaming workflows.

5.1 Overall Design↩︎

We design resilience mechanisms that decouple resilience from workflow-specific execution frameworks while explicitly targeting real-time scientific streaming requirements. 2 illustrates the overall architecture of StreamGuard. The central design principle is modularity. Rather than embedding resilience into a specific execution framework or enforcing workflow-wide coordination, resilience is provided as independent modules that can be selectively applied to different parts of a streaming workflow. This design improves portability and allows scientific workflows to integrate resilience support without modifying their execution model.

Instead of operating at the workflow level, we focus on producer–consumer pairs, which form the fundamental building blocks of scientific streaming applications. Most computation and performance sensitivity occur at these boundaries, where data are produced, processed, and forwarded. Operating at this granularity avoids unnecessary global coordination and allows resilience mechanisms to be applied only where needed, reducing overhead while maintaining flexibility. For each producer–consumer pair, we provide two complementary resilience mechanisms that independently address failures and performance anomalies.

The failure resilience mechanism mitigates worker failures through a lightweight checkpoint and retry process. Consumer workers periodically capture snapshots of their processing state and persist them as checkpoints on the parallel file system. Upon failure, the execution module automatically restarts affected workers on available resources and restores the most recent checkpointed state, allowing execution to resume from a consistent point rather than restarting from the beginning. This significantly reduces recovery time and limits disruption to ongoing stream processing.

The anomaly resilience mechanism addresses slowdown anomalies caused by performance heterogeneity or resource contention. Using runtime feedback from consumers, the module continuously monitors progress on data partitions and estimates effective worker processing capacity. When imbalance is detected, a load balancer dynamically adjusts partition-to-worker assignments by migrating slower partitions to faster workers. This maintains balanced progress across partitions and helps ensure that end-to-end processing remains within real-time constraints.

Together, these modules provide resilience against both failures and performance anomalies while avoiding tight coupling between mechanisms or reliance on global coordination. This separation prevents failures and slowdowns from compounding and enables robust execution across a wide range of operating conditions. The detailed design and implementation of each module are described in the following subsections.

5.2 Key Design Principles↩︎

5.2.1 Decoupled Lightweight Checkpoint-Restart↩︎

We design a lightweight checkpoint-restart mechanism that leverages the embarrassingly parallel structure of scientific streaming workflows. The design is based on three key principles.

5.2.1.1 Asynchronous, Per-Partition Checkpointing.

Instead of maintaining a single global checkpoint, we construct checkpoints independently for each data partition \(s_{i}\). Each such checkpoint is self-contained and includes all information required to restore processing progress, such as intermediate state and consumed records. Because streams are processed independently, workers can checkpoint without coordinating with other workers. This eliminates global synchronization overhead and simplifies recovery, allowing failed workers to resume execution from their most recent consistent state while unaffected workers continue processing normally.

5.2.1.2 Non-blocking Execution.

Scientific stream processing is typically computation-intensive, whereas checkpoint creation is I/O-intensive. This allows us to remove checkpoint creation from the execution critical path by overlapping the two processes. When a checkpoint is triggered, the worker provides references to the memory regions containing checkpoint data and immediately resumes computation, while a separate process persists the checkpoint in the background. By overlapping computation with I/O, the system improves resource utilization and keeps checkpoint overhead negligible during failure-free execution.

5.2.1.3 Dynamic Checkpoint Period.

A central challenge in checkpoint-based resilience is determining how frequently checkpoints should be created. Frequent checkpointing reduces recomputation after failures but increases runtime overhead, whereas infrequent checkpointing minimizes overhead but increases recovery cost. Classical solutions such as the Young/Daly formula [53], [54] provide an optimal checkpoint interval under static assumptions. However, scientific streaming environments often experience fluctuating failure rates and variable I/O performance due to slowdown anomalies.

To address this, we dynamically adjust the checkpoint period for each worker based on observed runtime behavior. Specifically, each worker \(w_j\) computes its checkpoint period based on a modified Young/Daly formula: \[W(w_{j}) = \sqrt{2 \frac{\mu}{M} C(w_{j})}\,, \label{equ:dynamic-ckpt-reconfig}\tag{4}\] where \(M\) is the number of workers, \(\mu\) the MTTF, and \(C(w_{j})\) the empirically observed per-worker checkpoint overhead. Each worker computes \(W(w_j)\) independently from its own \(C(w_j)\) and reconfigures it after failures or when checkpoint overhead changes, allowing the system to adapt to changing I/O and execution conditions while minimizing checkpoint overhead and recomputation cost.

3 illustrates the mechanism with four partitions across two workers, each creating its own checkpoint asynchronously in the background. When a worker fails, only the partitions assigned to that worker stop execution, while surviving workers continue processing unaffected. The failed partitions are later recovered using their most recent checkpoints, avoiding costly recomputation. Because checkpoints are independent across streams, recovery requires no inter-worker synchronization, allowing failure handling to remain lightweight and scalable.

5.2.2 Load Balancing↩︎

To mitigate degradation anomalies, we propose a dynamic load-balancing algorithm that continuously tracks per-partition processing progress and redistributes partitions across workers according to their effective processing speed. The objective is to prevent slow workers from violating the real-time deadline. Consider a producer–consumer pair operating within a real-time deadline window \([0, L]\), where \(0\) is the time when the first record is emitted by the producer and \(L\) is the processing deadline. Let \(p_i(t)\) denote the progress of partition \(s_i\) at time \(t \in [0, L]\). We define \[p_{\min}(t) = \min_i p_i(t), \quad p_{\max}(t) = \max_i p_i(t)\] as the slowest and fastest progress on data partitions at time \(t\), respectively. From the real-time constraint (2 ), the workflow meets the deadline only if the slowest partition completes on time, i.e., \(p_{\min}(L) \geq P\). This implies that overall completion time is determined solely by the slowest partition. Consequently, improving real-time performance requires prioritizing the progress of lagging partitions, even if doing so slightly slows faster ones because the number of available workers is limited to \(M\). To quantify imbalance among partitions, we define the progress ratio, \(G(t)\), as: \[G(t) = \frac{p_{\min}(t)}{p_{\max}(t)}, \label{equ:load-balance-goal}\tag{5}\] which measures how evenly progress is distributed across partitions. When \(G(t)=1\), all partitions advance at the same rate. The goal of load balancing is therefore to maintain \(G(t)\) close to 1 by reducing progress disparity among partitions.

We achieve this objective by identifying worker performance dynamically using observed execution behavior. Let the worker active time \(v_j(t)\) denote the total time during which worker \(w_j\) is alive and assigned at least one partition. Let \(q_j(t)\) denote the total progress made by that worker (see 1 ). We define the effective worker speed as \(r_j(t) = \frac{q_j(t)}{v_j(t)}\), which captures the average processing rate of the worker during execution. Partitions associated with slower workers are candidates for reassignment to faster workers.

However, assigning too many partitions to a fast worker may overload it and degrade overall performance. To avoid this, we introduce a capacity model. Recall that \(N\) is the total number of partitions and \(M\) the number of workers. The ideal number of partitions per worker under homogeneous conditions is \(\beta = \frac{N}{M}\). We adjust this quantity for worker speed to obtain the worker’s capacity, \(c_j(t)\): \[c_j(t) = \beta \frac{r_j(t)}{\frac{1}{M}\sum_{j} r_j(t)}, \label{equ:worker-cap}\tag{6}\] which represents the number of partitions a worker \(w_j\) can sustain relative to the average worker. A slow partition is reassigned only to workers whose current load is below their capacity. If no such worker exists, the algorithm swaps the slowest partition with the fastest one, sacrificing some progress of the fastest stream in order to improve the global progress balance.

The complete procedure is summarized in 4. The algorithm operates periodically: partition progress is updated at fixed intervals, after which the progress ratio \(G(t)\) is evaluated. If \(G(t)\) falls below a predefined threshold \(G_{\text{threshold}}\), reassignment is triggered (Line #8). The algorithm selects the slowest partition (\(s_{\min}\), Line #9) and attempts to migrate it to the fastest worker with available capacity (Lines #10–#21). When no worker has spare capacity, the slowest and fastest partitions are swapped to improve balance. Through repeated adjustments, the algorithm gradually equalizes progress across streams, preventing slowdown anomalies from propagating to end-to-end processing time.

6 StreamGuard Implementation↩︎

We implement StreamGuard as a software development kit (SDK) that provides resilience capabilities as independent, composable modules. Applications enable only the mechanisms they need, while the SDK design avoids tight coupling with specific execution frameworks, allowing integration into existing workflows without modifying their execution model. The overall architecture of StreamGuard is illustrated in 5 with four resilience modules, each responsible for a distinct aspect of reliable stream processing.

6.0.0.1 Resilience Execution Module.

This module provides reliable execution by automatically restarting failed workers on available compute resources. We implement this functionality using Parsl [55] (gray boxes in 5). Each stream processing task is implemented as a function whose inputs are records from a given partition \(s_{i}\) and outputs are data to be forwarded downstream. The function is executed inside a Parsl-managed container (i.e., the worker \(w_{j}\) in our resilience model), allowing Parsl to monitor worker liveness and automatically relaunch failed workers while preserving identifiers and stream mappings. The execution module exposes runtime information, including worker status, failure events, and resource availability, to other resilience modules through a resilient communication layer (described below). This information enables checkpointing and load-balancing decisions to adapt dynamically during execution.

6.0.0.2 Resilience Communication Module.

This module provides reliable producer–consumer communication via a publish/subscribe distributed queue (blue boxes in 5), offering two benefits. First, the queue provides reliable transmission with QoS guarantees (e.g., exactly-once), letting streams tolerate network failures without application-level recovery logic. Second, data persistence in the queue is decoupled from worker execution (workers retrieve data only when ready), preventing failures and slowdowns from propagating across the workflow.

We use Mofka [56], [57], a high-performance distributed message queue designed for HPC environments, as the underlying communication layer. StreamGuard provides a wrapper that allows applications to define logical data streams using unique keys, where each key \(k_i\) corresponds to a data partition \(s_i\) mapped to a dedicated Mofka topic. Producers publish records to these topics, which are subscribed by consumers for transmission. This abstraction simplifies application logic and enables other resilience modules to observe partition progress and exchange runtime information transparently and reliably.

6.0.0.3 Failure Protection Module.

This module protects workflow state against failures through the lightweight checkpointing mechanism described in 5.2.1. The implementation is based on a customized version of VeloC [37], a multi-level checkpoint runtime specialized for HPC infrastructure and large-scale deployment. Checkpointing is organized per producer–consumer pair. Each checkpoint set represents the state of a consumer, where entries correspond to individual data partitions identified by their keys. During initialization, applications register checkpointable state and receive checkpoint handlers that can be invoked during execution.

During normal execution, each worker \(w_{j}\) issues checkpoint requests after reaching consistent processing points, typically at the end of an iteration. If the elapsed time since the previous checkpoint exceeds the current checkpoint period \(W(w_{j})\), the in-memory state is passed to a background VeloC process and written asynchronously, allowing computation to continue without interruption. Since checkpoints for different partitions are handled independently, the mechanism avoids global synchronization and introduces no inter-worker communication overhead. The module continuously monitors checkpoint latency and observed failure frequency to dynamically adjust the checkpoint period according to Equation 4 , maintaining near-optimal overhead.

A worker may also explicitly enforce checkpointing when required by execution semantics, for example, after reaching a critical processing milestone or prior to reassigning a partition to another worker during load balancing. In such cases, the SDK issues a blocking checkpoint request that forces VeloC to complete the checkpoint before control is returned to the worker. This ensures that the worker state is fully persisted before execution proceeds, preventing progress loss during reassignment or recovery.

6.0.0.4 Slowdown Protection Module.

This module mitigates slowdown anomalies using the dynamic load-balancing mechanism described in 5.2.2 (purple boxes in 5). Its objective is to maintain balanced progress across streams despite heterogeneous worker performance. The module monitors partition progress through the communication module by tracking per-partition record consumption rates at consumer workers and estimating effective processing speed.

Applications may optionally enable this module for selected producer–consumer pairs. When an imbalance is detected, the load balancer initiates partition reassignment by instructing the current consumer to checkpoint and release the affected partition. The partition is then reassigned to a faster worker, which loads the corresponding checkpoint, subscribes to the partition’s topic, and resumes processing. Because reassignment involves only the previous and new consumer, migration proceeds without global synchronization or interruption of other streams, minimizing impact on overall workflow performance.

7 Evaluation↩︎

7.1 Methodology↩︎

7.1.0.1 Objectives

The evaluation assesses whether StreamGuard improves reliability in scientific streaming workflows without compromising performance. Our methodology is designed to isolate the impact of resilience mechanisms from application behavior and system noise, allowing us to evaluate both effectiveness and efficiency under controlled conditions. Specifically, we evaluate the following aspects:

1. Resilience Capability: whether StreamGuard mitigates failures and performance anomalies sufficiently to allow workflows to complete within real-time constraints;

2. Performance Transparency: whether StreamGuard introduces minimal overhead during failure-free execution;

3. Robustness: whether StreamGuard provides consistent protection across different scales, failure rates, and anomaly scenarios.

Together, these objectives evaluate whether resilience can be achieved as a practical system property rather than a trade-off against performance.

7.1.0.2 Metrics

The above objectives are evaluated using metrics reflecting the operational requirements of scientific streaming: (1) Processing Time, measured from the first producer emission to the last consumer output, capturing both the resilience and efficiency of the solution; (2) Deadline Violation, whether processing time exceeds the maximum acceptable limit; and (3) Overhead, the additional time relative to ideal failure-free execution, quantifying performance transparency.

7.1.0.3 Workload

We evaluate StreamGuard using a real-world scientific streaming workflow based on the Trace reconstruction engine [20], [21], which reconstructs cross-sectional images from X-ray projections collected at synchrotron radiation facilities. This workload is representative of scientific streaming applications due to its iterative computation, strict timing requirements, and sensitivity to failures and performance imbalance.

As shown in 6, the workflow consists of four stages. The data acquisition component (DAQ) collects projection images of size \(2048 \times 2048\) pixels from synchrotron radiation facilities. The distributor (DIST) normalizes collected data, converts projections into sinograms, and distributes them to parallel reconstruction workers (SIRT), which perform iterative partial 3D reconstruction. The post-processing stage (PP) aggregates partial results into complete reconstructed images.

Our evaluation focuses on the DIST–SIRT producer–consumer pair because it dominates per-iteration cost, and is where reconstruction state is formed, so failures and slowdowns there most directly affect end-to-end latency. To reflect operational constraints, the pair’s outputs must be produced within a fixed deadline. Under normal execution, reconstruction requires approximately 300–350 iterations over about 1500 records, resulting in an average processing time of roughly 200 s. We therefore set the deadline to 1000 s, providing sufficient recovery margin while preserving realistic real-time requirements.

7.1.0.4 Baselines

We compare StreamGuard against baselines covering both HPC-style and stream-based resilience mechanisms:

1. Original Implementation. The unmodified workflow without resilience mechanisms. This configuration represents the performance upper bound under failure-free execution but provides no protection against failures or anomalies.

2. Original + VeloC (\(X\)). The original workflow augmented with VeloC [37] checkpointing every \(X\) iterations; a lightweight, state-of-the-art HPC baseline that enables comparison of static against dynamic checkpoint policies.

3. Flink. The workflow is implemented using Apache Flink, a widely used stream processing engine with built-in fault tolerance for stateful operators. Flink maintains resilience by inserting checkpoint barriers into data streams; workers pause processing upon receiving barriers, snapshot their state, and propagate barriers downstream until a globally consistent checkpoint is established. Since Flink does not natively restart failed workers, we implement an external monitoring mechanism to detect failures and relaunch workers when necessary. We further tune checkpointing modes, data partitioning, and parallelism to obtain Flink’s best-performing configuration for fair comparison.

4. StreamGuard. Implementing our resilient mechanisms as described in 6. Although the resilience components are designed as independent modules, they are enabled together during evaluation whenever applicable. This allows us to assess their combined impact in realistic deployment scenarios, capturing both the overall resilience benefits and the cumulative overhead introduced by the complete solution.

7.1.0.5 System Configurations

All experiments are conducted on the Polaris supercomputer at Argonne National Laboratory. To minimize external interference, the workflow is deployed on dedicated compute nodes without co-located applications. Each node contains a single AMD EPYC “Milan” processor with 64 physical cores and 512 GB of memory, interconnected via a 200 Gbps high-speed network. Checkpoints are stored on the Lustre parallel file system, enabling shared access during recovery.

Failures and anomalies are injected using independent modules that operate transparently to the workflow. For failure injection, a daemon periodically terminates active workers by sending kill signals, with inter-failure intervals drawn from an exponential distribution, which is commonly used to model real-world failure behavior [14]. The distribution is parameterized by a predefined mean time to failure (MTTF), allowing controlled evaluation under different failure intensities.

To emulate performance anomalies, each worker is assigned an artificial slowdown that adds additional computation before processing each record. The slowdown magnitude varies across workers and is sampled from an exponential distribution, commonly used to approximate the power-law behavior observed in performance variability and straggler effects [31]. The distribution is parameterized by a predefined mean at the start of each run, and the assigned slowdown remains fixed for the worker’s lifetime.

7.2 Experimental Results↩︎

7.2.0.1 Checkpoint and Retry Efficiency

First, we evaluate the effectiveness of StreamGuard’s checkpoint and retry mechanism in protecting streaming data processing against unexpected failures and compare its performance with state-of-the-art solutions from both HPC and industrial stream-processing domains, represented by VeloC and Flink, respectively.

7 reports consumer processing time as we vary the number of data partitions mapped to each worker under three conditions: failure-free execution, light failures (per-worker MTTF = 150 s), and heavy failures (MTTF = 40 s). The key observation is that StreamGuard’s asynchronous, non-blocking checkpoint mechanism introduces negligible overhead in failure-free runs. As shown in 7 (a), the overhead remains below 1%, and the completion time is nearly identical to execution without checkpointing (green bars). This demonstrates that resilience can be achieved without sacrificing performance when failures do not occur.

In contrast, the checkpoint-free configuration in the original implementation (orange bars) is highly vulnerable to failures. Each failure forces a worker to restart from the beginning, discarding previously completed computation and significantly increasing total processing time. Even under a light failure rate (MTTF = 150 s), recomputation overhead becomes substantial (7 (b)). Under heavy failures (MTTF = 40 s, 7 (c)), the accumulated recomputation cost becomes prohibitive, preventing the workflow from completing within the deadline window (1000 s).

Static checkpointing cannot fully resolve this problem. Frequent checkpointing (Original+VeloC(1), purple bars) sustains progress under heavy failures but incurs excessive overhead when failures are infrequent, exceeding 3\(\times\) at eight partitions per worker due to I/O contention from concurrent checkpoints. Reducing frequency (Original+VeloC(4), brown bars) lowers failure-free overhead but becomes ineffective at high failure rates. A fixed interval cannot adapt to changing failure rates and load. In contrast, StreamGuard adapts checkpoint frequency to observed cost and failure behavior, maintaining low overhead and resilience across all scenarios.

8 compares StreamGuard with Flink, which employs synchronized, barrier-based checkpointing. This mechanism places checkpoint cost directly on the processing critical path because workers must pause while snapshots are taken. Moreover, checkpoint progress depends on barrier propagation across all workers; under failures or performance imbalance, slower workers delay barrier completion, causing congestion and backpressure. As shown in 8 (a), Flink execution is already 25% slower than the original implementation under failure-free conditions, and processing time increases further as failure frequency grows. The degradation is magnified at scale: at 16 workers with per-worker MTTF of 640 s, the SIRT stage experiences disruptions approximately every 40 s, preventing checkpoint barriers from reliably traversing the workflow and causing deadline violations.

In contrast, StreamGuard performs checkpoints in a fully non-blocking and asynchronous manner, removing checkpoint overhead from the processing critical path. In addition, data are streamed through a dedicated queueing service (Mofka), allowing workers to fetch data on demand and decoupling resilience behavior from flow control and data movement. As a result, checkpoint progress remains unaffected by transient slowdowns or failures in other workers. Across all evaluated scales and failure settings in 8, StreamGuard consistently meets real-time deadlines. Even under the most extreme case with 16 workers and MTTF = 80 s, which is equivalent to one failure every five seconds within the SIRT stage, the processing time remains within 20% of the ideal execution.

7.2.0.2 Dynamic Load Balancing

Next, we evaluate the effectiveness of StreamGuard’s dynamic load-balancing algorithm in mitigating the impact of slowdown anomalies. To emulate heterogeneous worker performance, we deploy the workflow with injected slowdown using the anomaly model described in 7.1.0.5. 9 reports the processing time of the workflow under StreamGuard compared with Flink and the original implementation across three slowdown configurations: light (average slowdown of 500 ms), medium (1000 ms), and heavy (2000 ms), and across three deployment scales of 4, 8, and 16 workers.

Both Flink and the original implementation rely on static partition-to-worker mappings. As a result, end-to-end processing time is dictated by the slowest worker. When slowdown anomalies are present, this leads to substantial performance degradation, with processing time increasing by at least 3\(\times\). The impact becomes more pronounced as the number of workers increases, since a larger deployment increases the likelihood of encountering slower workers that delay overall progress. Under medium and heavy slowdown conditions (1000 ms and 2000 ms), the accumulated delay prevents both approaches from completing processing before the deadline, even at small scale with only four workers.

In contrast, StreamGuard continuously monitors partition progress and estimates effective worker speed during execution. Lagging partitions are proactively migrated to faster workers, allowing the system to rebalance workload dynamically as performance imbalance emerges. This dynamic rebalancing progressively equalizes progress across partitions and prevents slow workers from dominating overall execution time. As a result, with four workers, processing time increases by at most 3\(\times\) while still meeting the real-time deadline, whereas both Flink and the original implementation fail to complete on time. Furthermore, as the number of workers increases, processing time remains stable because additional faster workers can compensate for slower ones. This allows StreamGuard to maintain nearly constant overhead across scales and consistently satisfy real-time constraints.

7.2.0.3 Robustness

Finally, we evaluate the robustness of StreamGuard by executing the workflow under combined failure and slowdown anomaly conditions at multiple scales, and comparing the results against both Flink and the ideal execution. The results are summarized in 10. We consider three anomaly settings: no slowdown (10 (a)), an average slowdown of 500 ms (10 (b)), and an average slowdown of 1000 ms (10 (c)). Each anomaly setting is evaluated under three failure configurations: no failures (blue), per-worker MTTF of 1000 s (orange), and MTTF of 100 s (green), while scaling the deployment from four to 32 workers. Bars represent the processing time of StreamGuard, while lines represent Flink processing time.

Consistent with observations from previous experiments, Flink’s synchronized checkpointing mechanism makes it increasingly sensitive to high failure rates, particularly at larger scales where the probability of failure increases with the number of workers. As scale and failure frequency increase, checkpoint coordination overhead grows, resulting in longer processing times. The presence of slowdown anomalies further amplifies this effect by extending processing duration, which in turn exposes Flink to additional failures within the same execution window. This compounding effect leads to rapidly increasing overhead, causing Flink to miss the processing deadline in most configurations once the average slowdown reaches 500 ms.

In contrast, StreamGuard maintains stable performance across all tested configurations. Failures and slowdown anomalies are handled by separate resilience mechanisms with minimal coupling, preventing their negative effects from amplifying each other. Dynamic checkpointing limits recovery overhead under frequent failures, while load balancing mitigates processing lag caused by slow workers. As a result, processing time remains close to the ideal execution even as scale, failure rate, and anomaly severity increase, allowing the workflow to consistently meet real-time deadlines across all evaluated settings.

8 Conclusions and Future Work↩︎

We presented StreamGuard, a resilience architecture for real-time scientific streaming workflows. By treating the producer–consumer pair as the unit of resilience and integrating dynamic non-blocking checkpointing with progress-aware load balancing as decoupled, composable modules, StreamGuard avoids global coordination while maintaining real-time deadlines under frequent failures, persistent slowdown anomalies, and increasing scale. Experiments on a real-world tomographic reconstruction workflow show that resilience can be achieved as a first-class system property without sacrificing scalability or performance transparency.

Several promising directions extend this work. First, StreamGuard assumes approximately uniform per-partition computational cost, which may not always hold in practice. Extending the load balancer’s capacity model with per-partition cost weights would broaden applicability to skewed workloads. Second, StreamGuard is evaluated with sufficient allocated resources for the experimental workload. Yet in practice, load spikes or miscalculations can cause transient under- or over-provisioning. Although our prior work has studied resource-workload mismatch [22], its setting does not target real-time streaming; studying such scenarios in a streaming context would help develop more robust and practical solutions. Finally, while the mechanisms presented here are stage-local by design, evaluating compound effects across multiple producer-consumer stages, including cross-stage I/O interference and backpressure, remains an important direction for future work.

References↩︎