1 Introduction↩︎

Modern particle physics detectors have grown significantly and are capable of generating extremely large volumes of data. To reduce the amount of data for offline storage, HEP experiments employ trigger systems, which are responsible for filtering, in real time, the data to be saved for further analyses. Because trigger applications must satisfy strict latency and throughput constraints, they are typically implemented on FPGAs.

In recent years, NN models have been increasingly integrated into trigger systems for a variety of tasks in several experiments, such as Belle II [1]–[3] and CERN Compact Muon Solenoid (CMS) detector [4], [5]. However, as these NN-based trigger systems are deployed close to the interaction point, they are exposed to radiation, caused by particle collisions in the detector [6].

Radiation-induced faults in hardware can lead to corrupted output data, stalled processing pipelines or even cause permanent hardware damage [7]. Given the scale of investment, failures in the trigger logic that cause downtime can result in substantial financial and scientific costs.

Prior fault-injection studies on NN accelerators have concentrated on SDC, while DUEs (hangs, deadlocks, timeouts) remain understudied [8]. For a real-time trigger, DUEs and packet-integrity violations are the most consequential failures. Existing studies also evaluate benchmark workloads on representative accelerators rather than full end-to-end deployed systems.

6pt

A timely and realistic case study for a fault tolerance analysis is the Belle II trigger system, presented in [3], which is an FPGA-based GNN dataflow accelerator for calorimeter clustering and trigger-bit generation. Its scale and complexity render traditional mitigation techniques such as TMR impractical, since the associated area overhead would prevent the design from fitting on a single FPGA. Moreover, the latency overhead of using multiple FPGAs would violate the strict timing constraints.

In this paper, we investigate how a complex hardware trigger can be affected by SEUs. Our contributions are as follows.

• We present, to the best of our knowledge, the first RTL FI study of a deployed L1 hardware trigger running a NN workload, targeting GNN-ETM in the Belle II trigger system.

• We characterise three types of failures most consequential to a real-time trigger pipeline: deadlocks, timeouts, and packet-integrity violations. Packet-integrity violations sit between the established SDC and DUE classes in terms of criticality.

• We propose inter-stage liveness monitoring as a more accurate alternative to output-only observation, showing that MTTF estimates from the two approaches differ by up to 78.7%. With the resulting data we identify the highest-priority hardening targets.

2 Related Work↩︎

Ahmadilivani et al. [8] systematically review 139 papers on NN reliability analysis, identifying three evaluation approaches (fault injection, analytical, hybrid). The survey finds that prevailing methodology focuses on SDC while DUEs (hangs, crashes, deadlocks) remain understudied, and identifies the control part of hardware NN accelerators as an understudied area.

Rech [13] reviews radiation effects in NN systems, covering fault propagation and hardening strategies, with a focus on GPUs and FPGAs. The survey notes that RTL fault injection provides full observability at the cost of longer simulation times, and that certain execution-flow failures are invisible to software-level fault injection.

Xu et al. [9] evaluated the reliability of a representative 2D PE array accelerator, by injecting stuck-at faults. The design is deployed on an AMD Zynq-7000 SoC platform, running four representative NNs under varying error rates. The authors observe that system exceptions (IO stalls, timeouts and early terminations) dominate the failures that occurred, compared to accuracy degradation. Applying TMR to the critical modules reduces the number of failures by a small amount, which is attributed to inaccessible to designers logic supplied by the vendor. However, two methodological limitations can be identified. First, faults are injected across the entire design, targeting only the memory, making it difficult to attribute failures to specific components. Additionally, system exceptions are observed only from the host processor perspective, which may delay their detection, consequently skewing MTTF estimates

Tonfat et al. [10] distinguish SDC and timeout outcomes in a configuration-memory FI campaign of HLS-generated matrix-multiplication variants on an AMD Artix-7. They show that control-flow-heavy designs fail through timeouts while aggressively pipelined data-flow variants fail mostly as single-output SDCs. Their emulation-based injection acts exclusively on configuration memory bits.

A closely related line of work, with respect to the injection approach, is provided by the recent studies of Jonckers et al. [11], [12], which apply RTL FI, to a NN accelerator, covering both the systolic-array core and the post-processing chain. In their 2024 study [11], the authors characterize the fault-propagation probability and fault magnitude of different register groups under constrained-random inputs, identifying the 32-bit systolic-array and accumulator registers as high-value hardening targets under Amdahl’s law. In a follow-up study [12], they extend the analysis to three image-classification NN workloads and distinguish output faults into non-critical perturbations and misclassifications, showing that both model redundancy and training accuracy, can shift the critical-fault rate by roughly an order of magnitude. It is worth noting that the failure taxonomy does not consider DUEs, the faults injected target only the Flip-Flops and a single fault is injected per inference run, which doesn’t capture the accumulation of faults during real world operation.

In this work, we study how faults lead to DUEs in a GNN accelerator deployed in the Belle II trigger system, as summarised in Table 1.

3 System Architecture↩︎

3.1 GNN-ETM Design↩︎

Belle II at SuperKEKB uses a hardware L1 trigger composed of dedicated subtriggers, one per participating subdetector, operating under a hard real-time deadline of \(4.4\,\mu\)s. GNN-ETM [3] is the L1 trigger module for the ECL subdetector. It performs real-time cluster reconstruction from TCs and generates the corresponding L1 trigger bits. The version used in this study differs from the original in two relevant ways: it uses a single GravNet block (vs.two) and includes the L1 trigger-bit generation logic. The full network architecture — CaloClusterNet, GravNet message passing, object-condensation training, and condensation-point selection — is described in [3].

GNN-ETM is implemented as a pipelined dataflow accelerator with four sub-components, shown in Fig. 2 and with signal counts summarised in Table 2. The Preprocessing Stage normalises and scales the incoming TC feature vectors and contributes \(385\,\)ns to the end-to-end latency. The GNN Dataflow Accelerator runs CaloClusterNet inference. It dominates both latency (\(821\,\)ns) and resource usage, consuming all available DSPs and approximately half the LUTs. The Postprocessing Stage performs the post-inference condensation-point selection, selecting cluster centroids from the GNN output, and contributes \(165\,\)ns. The Belle2Link MAC provides the Belle2Link protocol interface, multi-stream synchronisation, DAQ buffering, and arbitration, contributing the remaining \(475\,\)ns. The total end-to-end latency is \(1846\,\)ns. The system is implemented on a Universal Trigger Board (UT4) equipped with an AMD UltraScale XCVU190 FPGA.

The heterogeneous source code (HLS with AMD Vitis for the GNN Dataflow Accelerator, Chisel for the other three sub-components) is compiled to Verilog and integrated by FuseSoC into a single flat-netlist RTL design, which is the input to our FI campaigns.

3.2 Baseline Failure Detection↩︎

The existing failure-detection setup uses two mechanisms, both active only during simulation. An HLS-generated watchdog inside the GNN Dataflow Accelerator flags pipeline deadlocks within the GNN block. The Belle2LinkChecker, in the cocotb testbench at the design output, validates protocol conformance of the b2link byte stream and flags packet-integrity violations, in a similar way to the design-output observation approach of [9]. Together, these mechanisms cover the GNN block and the design output. Failures originating elsewhere in the pipeline are observed only indirectly. The cocotb testbench also instruments the inter-stage AXI-Stream interfaces with transaction monitors for functional verification, which are not part of the failure-detection setup. GNN-ETM itself currently includes no fault-tolerance mechanisms, such as configuration-memory scrubbing or module-level redundancy. This setup defines the baseline detection view used in the comparison study of Section 5.

4 Fault Injection Methodology↩︎

4.1 Simulation Setup and Fault Model↩︎

We analyse our DUT, GNN-ETM, by injecting faults through RTL simulation. Compared to alternatives such as beam testing and emulation, this enables more flexibility in choosing specific areas for FI and a target-platform-agnostic analysis [13]. An overview of the resulting fault-injection testbench is shown in Fig. 1.

Each simulation run takes approximately \(250\,000\) ns of simulated time with multiple faults injected per run. The FI follows a Poisson process, which is the model used for SEU ground testing [14] and in-detector FPGA operation [15]. The injection rate is set to one fault per \(1\,846\) ns, the design’s end-to-end latency. This is several orders of magnitude higher than the real radiation rate at the detector, which is why fault-injection studies normally inject a single fault per run. An exhaustive single-fault-per-run sweep across all \(211\,245\) signals would have required approximately 488 days to complete even with parallelism, so we trade absolute fidelity for simulation feasibility. The campaign therefore quantifies relative sub-component reliability rather than absolute in-detector failure rates.

We focus on faults that cause deadlocks, timeouts, or packet-integrity violations. A deadlock is a state in which the design, or one of its sub-components, stops making forward progress. A timeout is flagged when a simulation run exceeds the expected simulated time without completing. A packet-integrity violation corresponds to an output stream whose structure no longer conforms to the Belle2Link protocol and is therefore unreadable by the downstream DAQ. A simulation run is classified as failed if any of these is observed, and as passed otherwise.

Simulations ran in ModelSim [16] with the cocotb Python verification framework [17]. Faults were injected with the open-source cocotb fault injection tool [18] developed at CERN. The input stimulus consists of randomly generated TC data, with a fresh random seed per run, since the targeted failure modes are caused by faults in control signals rather than by the data being processed. All experiments ran on a server equipped with an AMD EPYC 7702P 64-core processor and 503 GB of RAM. Each fault is a single-bit SEU injected at signal level: a single bit of the targeted signal is flipped and remains flipped until the next normal write to that signal. Multi-bit upsets, relevant at advanced FPGA nodes [19], are not injected here but would propagate to the AXI-Stream interfaces identically and would be detected by the same inter-stage liveness monitors. We treat all RTL signals uniformly and do not report their storage-vs-combinational breakdown, since separating them requires synthesis that is either unreliable for the design’s HLS output and locked vendor IP (open-source flows) or tied to a specific toolchain (vendor flows). Faults target signals from both the full design and each sub-component individually. Injecting faults at the sub-component level is necessary since, when a failure occurs, it is otherwise difficult to trace which fault was responsible. For each target, we run enough simulations to inject at least as many faults as the number of signals it contains. The per-target run and fault counts are reported in Table 3.

4.2 Two Injection Campaigns↩︎

To analyze GNN-ETM, we ran two separate FI campaigns. The proportional-rate campaign mimics a uniform radiation environment: each sub-component is injected at a rate proportional to its share of the design’s signals. The fixed-rate campaign instead injects every target at the same absolute rate as the full design, so that MTTF is directly comparable across sub-components.

The proportional rate follows from the standard relationship between particle flux and device sensitivity. The particle flux \(\Phi\) is the number of particles crossing a unit area per unit time. The cross-section \(\sigma\) of a device is the effective sensitive area within which an incident particle produces a fault [13]. From these two quantities, the expected fault rate of the device, in faults per unit time, is \[\lambda = \Phi \cdot \sigma.\] We assume that the sensitive area is distributed uniformly across the design, meaning that every signal contributes an equal share of the total cross-section. Per-signal cross-sections cannot be characterised at RTL, since they depend on the specific FPGA mapping and require beam-irradiation measurements, making the uniform cross-section assumption necessary for a platform-agnostic study. Therefore, the cross-section of any portion of the device is calculated as \(\sigma = \sigma_s \cdot N\), where \(\sigma_s\) is the per-signal cross-section and \(N\) the number of signals. Having already fixed the injection rate of the full design to \(\lambda_{\text{full}}\), corresponding to one fault per \(1\,846\) ns (the end-to-end latency of the design), the same particle flux combined with the uniform-\(\sigma_s\) assumption yields the adjusted per-component rate \[\lambda_{\text{comp}} = \lambda_{\text{full}} \cdot \frac{N_{\text{comp}}}{N_{\text{full}}},\] with \(N_{\text{comp}}\) and \(N_{\text{full}}\) taken from Table 2. The proportional-rate campaign applies \(\lambda_{\text{comp}}\) to each sub-component in isolation, with the rest of the design left unfaulted. The relative rates across sub-components therefore mirror those of a uniform radiation environment, while absolute rates are anchored to the full-design rate chosen for simulation feasibility and are higher than the real-world equivalent.

The proportional-rate campaign captures relative exposure but mixes two effects: how often a sub-component is hit and how sensitive it is per fault. Small components like the Postprocessing Stage (\(6\,010\) signals) see only a handful of faults per run, which inflates their MTTF and makes per-component sensitivity hard to compare directly. The fixed-rate campaign complements it by equalising exposure across targets, isolating per-fault sensitivity.

4.3 Inter-Stage Liveness Analysis↩︎

The simulations themselves run under the baseline detection setup of Section 3.2. We additionally re-analyse the same simulation logs with an inter-stage liveness check, deriving four per-stage deadlock signatures from the cocotb transaction monitors at the inter-stage AXI-Stream interfaces. Section 5 compares the two views.

The four monitored interfaces, in pipeline order, are:

• Pre.header: the path from preprocessing to the Belle2Link header.

• GNN DF output: the output of CaloClusterNet inference.

• Post.CPS: the path from postprocessing to the Belle2Link CPS.

• Post.GDL: the path from postprocessing to the GDL.

During normal operation each stream emits a transaction roughly every \(125\,\)ns. A monitored stream is declared deadlocked if it remains silent for more than \(2\,\mu\)s after the \(1\,846\) ns end-to-end latency has elapsed, well above the normal interval, to avoid false positives. A tighter threshold would lower MTTF estimates at the cost of more false positives from natural backpressure, but the coverage gain over the baseline holds at any reasonable threshold. An equivalent live monitor would catch the same events with the same detection latency: the analysis only inspects AXI-Stream handshake events that a live monitor would also observe in real time, and detection is in both cases bounded by the \(2\,\mu\)s silence window.

When classifying each run, inter-stage deadlock signatures take priority. Baseline mechanisms (HLS watchdog, Belle2LinkChecker) classify the run if no signature fires.

5 Experimental Results↩︎

2.5pt

4pt

The two FI campaigns inject a total of \(1\,442\,840\) SEUs across the full GNN-ETM design and its four sub-components, as shown in Table 3. The Full Design row is identical between the two campaigns, so its faults are counted only once in this total. Table ¿tbl:tab:fi95results? summarises the outcomes of both campaigns under the baseline and inter-stage monitoring approaches: total runs, the per-class failure distribution as a percentage of total runs, the MTTF in microseconds of simulated time, and the percentage MTTF deviation between them. Each column of Table ¿tbl:tab:fi95results? corresponds to a separate batch of simulation runs with faults injected at a different target (full design, or only one sub-component), so percentages should be read within a column rather than compared across columns.

The Margin of error row in Table ¿tbl:tab:fi95results? gives the \(95\%\) confidence interval that applies to each percentage cell in the same column, computed following Leveugle’s statistical fault-injection methodology [20] with the conservative \(p = 0.5\) worst-case binomial assumption, as in recent studies [21].

5.1 Baseline Detection↩︎

The Baseline detection rows of Table ¿tbl:tab:fi95results? summarise what the existing verification setup observes. Failed-run rates exceed \(97\%\) for the full design and the GNN under both campaigns, and for the Preprocessing, Postprocessing, and Belle2Link MAC sub-components under fixed-rate injection. The Postprocessing Stage and Belle2Link MAC retain meaningful pass rates and longer MTTF under proportional-rate injection because of their smaller signal share, which means fewer faults per run. Under fixed-rate they drop to near-zero pass, confirming that the proportional-rate pass rates were a low-exposure artefact.

Deadlocks are observed exclusively in the GNN Dataflow Accelerator (\(63.58\%\) under proportional-rate and \(76.98\%\) under fixed-rate injection), since it is the only block currently covered by the HLS-generated watchdog. The Chisel-based sub-components (Preprocessing, Postprocessing, and Belle2Link MAC) have no dedicated deadlock detection. Their failures surface either as packet-integrity violations at the design output or as simulation timeouts, both detected only after long propagation latency.

5.2 Inter-stage Liveness Monitoring↩︎

The Inter-stage liveness monitoring rows of Table ¿tbl:tab:fi95results? reclassify the same simulation runs using the four cocotb transaction monitors at the AXI-Stream interfaces. With this approach, each deadlock is attributed to a specific stage: Pre.header, GNN DF output, Post.CPS, or Post.GDL. The timeout class drops below \(0.5\%\) on the Preprocessing and GNN Dataflow Accelerator targets, confirming that most baseline timeouts were deadlocks that the existing setup could not catch directly. MTTF deviations range from \(38.5\) to \(78.7\%\) across the Preprocessing, GNN Dataflow Accelerator, and Postprocessing sub-components, with Preprocessing under fixed-rate injection at the upper end, as shown in the MTTF deviation row.

The HLS watchdog row drops below \(1\%\) in every inter-stage column because the inter-stage monitors take priority and tag the same GNN deadlocks first. A stall in Postprocessing back-pressures the GNN-to-Postprocessing interface and freezes the GNN’s output stream. As a result, \(4.13\%\) of Postprocessing-injected runs under proportional-rate and \(33.14\%\) under fixed-rate are classified as GNN DF output deadlocks, even though the fault originated in Postprocessing.

5.3 Discussion↩︎

SDCs are naturally observed at the design output: intermediate errors do not affect the measurement as long as the output is correct, whether because no error occurred or because errors were masked along the way. For DUEs the opposite holds. A fault that stalls the pipeline at time \(t_d\) is only registered as a failure at \(t_d + \Delta\), where \(\Delta\) is the detection latency. Since the detection latency can not be removed from the MTTF calculation, the sooner an error is observed, the more accurate the measurement, especially when multiple faults are injected per run to accelerate the campaign. Output-only observation, as in [9], therefore inflates MTTF estimates for DUEs. Inter-stage monitors close this gap, except for the Belle2Link MAC, where all monitors are placed before it in the pipeline and \(\Delta\) remains identical to the baseline, leaving the MTTF deviation at zero.

Beyond closing detection gaps, the inter-stage MTTF data identifies hardening priorities for the design. Under proportional-rate injection, the Preprocessing Stage and GNN Dataflow Accelerator have the shortest MTTFs, making them the highest-priority hardening targets. The Belle2Link MAC and Postprocessing Stage have substantially longer MTTFs and contribute less to the Full Design’s failure rate. The fixed-rate campaign reaches the same conclusion, even though smaller sub-components are injected at fault rates up to 35x higher than in the proportional-rate campaign.

6 Conclusion↩︎

We have presented the first RTL fault-injection study of a deployed L1 hardware NN-trigger, targeting GNN-ETM in the Belle II trigger system. Through two campaigns, a proportional-rate campaign that reflects deployment-realistic exposure and a fixed-rate campaign that equalises exposure across sub-components, we injected \(1\,442\,840\) SEUs across the \(211\,245\) signals of the full design and its four sub-components.

We characterised three failure modes most consequential to a real-time trigger pipeline: deadlocks, timeouts, and packet-integrity violations. The existing verification infrastructure shows a monitoring asymmetry, as deadlocks are observable only inside the HLS-based GNN Dataflow Accelerator, while failures in the Chisel-based sub-components surface only as packet-integrity violations or simulation timeouts.

We proposed inter-stage liveness monitoring as a more accurate alternative to output-only observation. MTTF estimates from the two approaches differ by up to 78.7%, exposing the bias of measuring failures only at the design output. The Belle2Link MAC is not covered by this gain, since all four monitors sit upstream of it. A fifth monitor at the b2link output would close this gap, though with longer detection latency since b2link packets arrive less often. These monitors are not part of the accelerator pipeline, so they could be implemented as a hardware IP running in parallel with the accelerator without affecting the latency budget, and a reset on detected deadlock would restore normal operation most of the time. The per-stage attribution under proportional-rate injection identifies the GNN Dataflow Accelerator and the Preprocessing Stage as the highest-priority hardening targets, which will be the focus of our follow-up work.

Acknowledgment↩︎

This work is funded by the German Federal Ministry of Research, Technology and Space (BMFTR) in the framework of design tools for sovereign chip development with open source (DE:Sign DI-EDAI, grant number 16ME0990K) and by the DEEP consortium (05D25VK1) in the ErUM-Data action plan.

References↩︎