Analyzing massive datasets from nuclear physics experiments can take hours or days to process, but researchers are working to radically reduce that time to mere seconds using special software being developed at the Department of Energy's Lawrence Berkeley and Oak Ridge national laboratories.
DELERIA - short for Distributed Event-Level Experiment Readout and Integrated Analysis - is a novel software platform designed specifically to support the GRETA spectrometer, a cutting-edge instrument for nuclear physics experiments. The Gamma Ray Energy Tracking Array, or GRETA , is currently under construction at Berkeley Lab and is scheduled to be installed in 2026 at the Facility for Rare Isotope Beams, or FRIB , at Michigan State University.
The software will enable GRETA to stream data directly to the nation's leading computing centers with the goal of analyzing large datasets in seconds. The data will be sent via the Energy Sciences Network, or ESnet. This real-time analysis will allow researchers to make critical adjustments to the experiment as it is taking place, leading to increased scientific productivity with significantly faster, more accurate results.
"Our primary goal is to establish a data pipeline for future experimental needs without having to build a local computing infrastructure. The pipeline will also allow us to increase the amount of analysis that is performed online," said Gustav Jansen, an ORNL computational nuclear physicist who has been developing DELERIA for the last two years. "But we also intend to show that DELERIA can be scaled to work with other research facilities and a broader range of scientific applications."
GRETA is a spherical array of 120 hyper-pure germanium crystals. Experimental samples are placed inside the instrument and bombarded with charged particles from the radioactive ion accelerator at the FRIB facility.
The collisions emit hundreds of thousands of photons every second. Each time a photon hits a crystal, the interaction is recorded and transmitted over the pipeline for analysis. This process is ultimately used to reveal how protons and neutrons in the nuclei of a sample are arranged and how they behave.
Significant computing resources are required to calculate the energies and coordinates of the photons and process the information in a way that is meaningful to researchers.
"Establishing a data pipeline allows us to offload the heavy computing to data centers at the national labs. So, instead of building a whole new computing system for a potential upgrade to GRETA, we can just simply connect to one," Jansen said. "And, from a nuclear physics standpoint, this presents us with options for upgrading GRETA, as well as instruments at other facilities that will also be able to leverage the DELERIA software platform."
Testing… Testing.
Together, Jansen and his colleagues at the Oak Ridge Leadership Computing Facility, a DOE Office of Science user facility located at ORNL, and Berkeley Lab built a testbed of the GRETA data pipeline. At Berkeley Lab, a virtual version of the GRETA detector array simulates photon collision events and transmits the data via ESnet , to a computing cluster at OLCF more than 2,000 miles away.
"The crystals inside the detector have lots of electrical contacts connected to them. When a photon hits one of the crystals, an electrical signal goes out on each contact point," Jansen said. "We need to figure out where inside the crystal the photons hit - the X, Y, Z coordinates. And we need to do that within 10 seconds of the collision. That's what we designed the pipeline to do."
The pipeline's processing power is provided by OLCF's Defiant , a 36-node AMD GPU-accelerated computing cluster. Defiant's job is to do the number crunching to determine the X, Y, Z coordinates of the photon collisions and send the processed data back to Berkeley Lab for storage or additional analysis. DELERIA also significantly reduces the volume of stored data by 97.5%, or a factor of 40×.
The testbed can generate and analyze approximately 480,000 photons collision events every second. The data must travel more than 4,000 miles there and back again and be processed all in 10 seconds or less.
The GRETA testbed is part of OLCF's Advanced Computing Ecosystem, or ACE , a testbed that provides computing and data resources for a wide range of system architectures. ACE is supported by DOE's Integrated Research Infrastructure , or IRI, an initiative to integrate DOE's leadership-class computing centers with its national research facilities.
Tricking time
In a series of demonstrations, the team has been able to process up to 35 gigabits (Gbps) of data per second, and transmit data at a rate of 35 gigabits per second, which actually surpasses GRETA's required maximum rate and is easily accommodated by ESnet's ESnet6 network, which has multiple 400 Gbps paths between Berkeley Lab and ORNL. The team intends to achieve even higher rates. According to Jansen, the big problem with moving data across the country is latency.
Currently, their system can process a single photon collision event in 5 milliseconds, or 0.005 seconds. But, traveling at light speed, it takes 120 milliseconds, or 0.12 seconds, to transmit data from Berkeley Lab to OLCF and back - more than an order of magnitude longer than it takes to process a single event.
"Of course, only in science fiction can you bypass the speed of light," Jansen said. "What we're working on now is to get a 10× speedup by tricking latency, so to speak. Instead of running one event at a time, the solution is to run events in parallel so that we can process one event, while others are being transferred. By finding the right balance, we can make sure that the computing cluster is always busy. Otherwise, our analysis will take 10 times longer than it should."
When it comes to moving faster than the speed of light, "Only Gustav can do that," said Tom Beck, jokingly. Beck is the section head for science engagement in ORNL's National Center for Computation Sciences. In addition to supporting Jansen, Beck is also working on other IRI projects that connect major DOE research facilities with leadership-class computing resources at the OLCF.
"This is really pioneering work in the IRI theme that's setting the standards for how we network the nation's leading centers of research. And the more use cases we have, the better we're going to get," Beck said. "One case at a time. And so far, this is really the first case that's up and working at this level of maturity that I'm aware of, which means we're off to a really great start."
In addition to Jansen, the project team members include Mario Cromaz (Berkeley Lab), Eli Dart, Eric Pouyoul, Ezra Kissel, Kiran Vasu, Seyoung Yu (ESnet), and Ross Miller (OLCF).
DELERIA is a Laboratory Directed Research and Development project led by Berkeley Lab. Support for this research also came from the DOE Office of Science's Advanced Scientific Computing Research program and the Office of Nuclear Physics. OLCF, ESnet and FRIB are DOE Office of Science user facilities.
