1 Motivation and design goals↩︎

Real-time dataflow processing in experimental particle physics places stringent demands on computing throughput, deterministic latency and I/O bandwidth. FPGA devices are particularly well suited to these requirements owing to their reconfigurable logic, tightly coupled memory and high-speed serial transceivers. The maturation of High-Level Synthesis (HLS) tools over the past decade has substantially lowered the entry barrier, allowing a wider community of physicists and engineers to exploit FPGA acceleration without resorting exclusively to Hardware Description Language workflows.

A significant limitation of present-day HLS environments, however, is their confinement to a single FPGA device. When the scale of a trigger or data-acquisition problem exceeds the capacity of one chip—as is common in modern experiments with high channel counts and event rates—developers must fall back on ad-hoc, manually crafted inter-FPGA communication layers. This gap motivated the creation of APEIRON: an integrated framework that extends the Xilinx Vitis HLS ecosystem to operate transparently across a network of interconnected FPGAs.

The principal design goals can be summarised as follows:

• provide a modular, topology-configurable, low-latency direct interconnect among FPGA nodes;

• offer a dataflow programming abstraction, drawing on Kahn Process Networks [1], in which processing tasks communicate through lightweight send/receive primitives regardless of their physical placement;

• automate the generation of all ancillary logic (routing, dispatching, aggregation) from a compact application description, so that users concentrate on the algorithmic C/C++ kernels;

• target both traditional low-level trigger systems and data-reduction stages in trigger-less or streaming readout experimental setups characterised by high event rates.

2 Platform architecture↩︎

2.1 Overall topology↩︎

At the system level, APEIRON models the data path of a trigger or data-reduction chain as m independent data sources (detectors or sub-detectors) feeding a cascade of n stream-processing stages. Each stage may reside on one or more FPGA nodes; the network fabric recombines data streams across stages as required by the physics algorithm. Figure 1 illustrates a representative configuration.

The scalability of this scheme rests on the network infrastructure: a packet-switched, dimension-order-routed mesh whose physical topology can be tailored to the application through configuration rather than redesign. The network constitutes the core enabling element of the platform, bridging the gap between single-device HLS and multi-device real-time systems.

2.2 Communication infrastructure↩︎

The communication subsystem descends from the APEnet [2] and EXANEST [3] network-on-chip designs originally developed for HPC, adapted here for low-latency stream processing in TDAQ applications. The Communication IP was co-designed with the APEIRON software stack to achieve low latency and scalable bandwidth between processing tasks.

Two classes of data movement are supported (see Figure 2):

Intra-node

transfers between kernels co-located on the same FPGA, routed through the local switch fabric.

Inter-node

transfers between kernels residing on different FPGAs, carried over high-speed serial transceivers and routed by the network.

Routing and switching. The Routing IP contains a crossbar switch, configuration/status registers and the inter-node and intra-node interfaces. Paths across the switch are set up dynamically by a router that applies dimension-order routing—offsets are resolved one coordinate at a time—while an arbiter resolves contention when multiple packets compete for the same egress port. The switching technique is Virtual Cut-Through [4]: forwarding of a packet begins as soon as the output direction is determined and sufficient buffer space is available. Deadlock freedom under dimension-order routing is ensured by providing two virtual channels per physical link [5]; fault tolerance is not currently addressed.

All transfers are packet-based. Each packet comprises a header carrying routing and addressing metadata, a variable-length payload and a trailing footer.

2.3 Kernel interface and programming model↩︎

To keep the developer’s task focused on the algorithm, APEIRON imposes only a minimal requirement on HLS kernels: each must expose AXI4-Stream [6] ports for its input and output channels, following the prototype:

void my_task(
    [optional kernel-specific parameters],
    message_stream_t message_data_in[N_INPUT_CHANNELS],
    message_stream_t message_data_out[N_OUTPUT_CHANNELS]);

Communication between any two kernels—co-located or remote—is expressed through a lightweight C++ API consisting of a non-blocking send() and a blocking receive():

size_t send(msg, size, dest_node, task_id, ch_id);
size_t receive(ch_id);

Here dest_node encodes the multi-dimensional coordinate of the target FPGA, task_id selects the receiving kernel within that node (up to four per node), and ch_id identifies one of up to 128 logical channels local to the receiving kernel. The API exploits AXI4-Stream side-channel signals to convey all the information needed to construct the packet header, allowing the developer to perform inter-kernel communication without knowledge of the underlying packet protocol.

Within each node, the Communication IP connects the user kernels to the Routing IP through two dedicated adapter blocks (Figure 3 and Figure 4):

• the Dispatcher, which inspects the header of incoming packets and steers them to the appropriate input channel FIFO of the target kernel;

• the Aggregator, which accepts outgoing data from the kernel, constructs the packet header and enqueues the result into the Routing IP’s transmission buffers.

2.4 Build flow↩︎

The mapping of the computational dataflow graph onto the physical FPGA network (illustrated in Figure [fig:mapping]) is specified in a YAML configuration file that lists, for every kernel, its input/output channel count and the intra-node port to which it is attached. From this description the APEIRON toolchain links the Communication and Routing IPs, wires them to the user kernels and produces the complete design bitstream. The designer is thus relieved of manual integration work and can iterate on the algorithmic content of the C/C++ kernels alone.

3 Validation: particle identification for NA62↩︎

3.1 Experimental context↩︎

NA62 is a fixed-target kaon-decay experiment operating at the CERN SPS North Area [7]. Its Ring Imaging Cherenkov (RICH) detector provides information that can be used for online particle identification (PID), which is a valuable input to the low-level trigger decision.

An earlier incarnation of the system, dubbed GPURICH, offloaded a geometry-based PID algorithm to a GPU, with a dedicated FPGA board running the NaNet [8]–[10] firmware to achieve direct, low-latency data transfer between the detector readout and GPU memory. In the present work we replace this heterogeneous GPU+FPGA pipeline with a single-FPGA solution—termed FPGA-RICH—built entirely within the APEIRON framework, using HLS.

3.2 Neural-network model↩︎

FPGA-RICH ingests RICH events in a continuous stream and must sustain a throughput exceeding 10 MHz, as dictated by experiment requirements. The PID task is cast as a neural-network inference problem using a seedless approach: the network receives only RICH hit data, without external track seeds.

The chosen architecture (Figure 5) is a fully connected network with three layers of 64, 16 and 4 neurons respectively. Up to 64 normalised photomultiplier identifiers per event serve as input features. To constrain FPGA resource usage, the trained model undergoes quantisation with QKeras [11] and is subsequently converted to synthesisable firmware using HLS4ML [12], yielding fixed-point representations of \(<\)​8, 1\(>\) for weights and biases and \(<\)​16, 6\(>\) for activations.

Two quantities are inferred per event: the multiplicity of charged particles (\(N_r\)) and the count of \(e^{\pm}\) (\(N_e\)). Training and validation datasets were assembled from NA62 physics runs using the experiment’s offline analysis framework; ground-truth labels were obtained from the seedless RichReco reconstruction algorithm.

3.3 Training strategy and results↩︎

Because the network output is intended to drive a trigger decision, inference performance is critical. The training set comprised approximately 3 million events from run 8011, chosen to be representative of online conditions. An independent validation set of 3.5 million events from run 8893 was used to assess generalisation. The ROC curves for \(N_r\) classification, shown in Figure [ROC], demonstrate satisfactory discriminating power.

Since the NA62 RICH detector is able to discriminate the type of charged particles only in the \(15-35\) \(GeV/c\) energy range, results for \(N_e\) are not equally satisfying.

3.4 Resource utilisation and throughput↩︎

Synthesis targeting a Xilinx VCU118 board at 150 MHz yielded a very compact implementation consuming 14 % of available LUTs and 2 % of DSP slices. The measured inference latency is 146.66 ns, corresponding to a sustained throughput of 18.75 MHz—comfortably above the 10 MHz requirement.

4 Conclusions and Future Work↩︎

We have presented the APEIRON framework, whose combination of a configurable low-latency FPGA network, a dataflow programming model and automated build tooling addresses a recurrent need in the design of trigger and data-acquisition systems for particle physics.

We are continuing the development of the APEIRON framework in order improve its performance and usability. We are also finalising the development of the FPGA-RICH system within the framework, encouraged by the good performance on the identification of charged particles. We envision a solution to improve results in identification of \(e^{\pm}\), using the LKr calorimeter online primitives that provide information related to the energy of the event.

References↩︎

References↩︎