Supercomputers step particles through a virtual world to give a history of their movement. The results are compared to physical experiments. Credit: Jacquelyn DeMink/ORNL, U.S. Dept. of Energy
High energy physicists run on a treadmill that keeps speeding up. Their collider experiments smash particles at dazzling speeds and energies. Detectors identify and track the multitudes of smaller particles that fly out. With powerful new colliders crashing particles at ever-increasing energies, even more daughter particles are produced. The innovative Celeritas project , led by the Department of Energy's Oak Ridge National Laboratory, provides a software tool that makes sure simulations used to analyze particles can run on the fastest supercomputers, accelerating answers about the nature of the universe.
Celeritas is Latin for speed. The Celeritas code stands out for its ability to run primarily on graphics processing units (GPUs), which excel at parallel processing. It is a major upgrade from simulations relying on traditional central processing units (CPUs), which shine at sequential tasks. If Celeritas can quickly crunch big data, it could accelerate knowledge acquisition from some of the world's biggest physics experiments.
"Celeritas establishes ORNL as a focal point for high-performance computing in high energy physics," said Seth Johnson of ORNL's Computational Sciences and Engineering Division, who develops tools that optimize simulations running on leadership-class supercomputers.
From reactor models to particle experiments
The effort originated with nuclear reactor simulations led by Tom Evans, of ORNL's Computational Sciences and Engineering Division, for DOE's Exascale Computing Project. Marcel Demarteau, director of ORNL's Physics Division, launched the project and was its principal investigator. Johnson currently leads the project. Partners include DOE's Argonne National Laboratory and Fermi National Accelerator Laboratory (Fermilab).
Celeritas supports major experiments that rely on ginormous detectors. In the United States, Celeritas supports the Deep Underground Neutrino Experiment (DUNE), which shoots neutrinos from Fermilab in Illinois to the Sanford Underground Research Facility in South Dakota. Each of DUNE's two detector modules is as tall as a five-story building. Combined, they weigh as much as nearly four Eiffel Towers.
In Europe at CERN's Large Hadron Collider (LHC), Celeritas supports the Compact Muon Solenoid (CMS) and A Toroidal LHC Apparatus (ATLAS). CMS is as tall as a five-story building and heavier than the Eiffel tower. ATLAS is bigger but about the same weight.
Simulations create digital twins of these physical experiments. They step each particle through a virtual world to generate a history of its movement. Every second, up to a billion collisions produce particles. Because tracking this volume and variety of data is impossible, a "trigger" system filters potentially interesting events for further analysis. Even after down-selection, every day the CERN Data Center processes a petabyte (one million gigabytes) of data on average.
Tackling big data in high energy physics
Big data is a big problem for big science. However, thanks to DOE national labs, Celeritas is having a big impact. The CMS, ATLAS and DUNE collaborations each include thousands of researchers from hundreds of institutions and dozens of countries. Celeritas helps these communities make the most of their data.
"With our effort, the United States has been leading the way developing new methods to take advantage of GPUs," Johnson said. "American corporations developed GPU hardware in Silicon Valley, and DOE provided the major investment in GPU-led hardware for supercomputers. Because of those investments, our group at Oak Ridge made the connection between the GPU hardware and a gap in capability in these international physics projects."
For five years, ORNL-led researchers have collaborated with CERN scientists on Celeritas. "We rely on scientists at CERN to implement our work into their analysis code," Johnson said. "We've been competing and collaborating simultaneously. This dynamic has been productive for everyone involved."
Ensuring supercomputers can make the most of advanced GPU computer chips is an urgent need. A major upgrade of the LHC, the world's most powerful particle accelerator, is underway. Completion of the High-Luminosity LHC is expected in 2030.
"They're making the LHC brighter," Johnson said. "The brighter it is, the more data it makes. Because more particle events are happening at once, we have to do many more simultaneous calculations to see what might be going on."
Current codes calculate electromagnetic interactions of particles as they move through detectors. Celeritas is a critical step in reworking the way simulations and analyses are done in high energy physics. Johnson and colleagues performed initial simulations on ORNL's GPU-accelerated Frontier supercomputer at the Oak Ridge Leadership Computing Facility, a DOE Office of Science user facility. Boosting data throughput, the Celeritas code uses GPUs for massive parallel processing to simulate particle interactions based on state-of-the-art physical models. In one second, Frontier can complete a task that would take humanity more than five years, assuming each person on Earth could complete one calculation per second.
Testing on Frontier, the world's fastest supercomputer for open science
Researchers plug the Celeritas software tool into existing toolkits developed for large physics experiments. "Traditionally, the toolkits ran on CPUs only, which limits their ability to do large computations," Johnson said. "To use the newer hardware that's on every new supercomputer, they need to use our software to run on GPUs."
To validate theories of the Standard Model of particle physics, Celeritas researchers simulate the movements of particles crossing a detector and compare them with real data emerging from that detector.
CMS and ATLAS experiments follow particles produced in the collisions, and Celeritas has focused on the simulation of high-energy electromagnetic particles - photons, electrons and positrons. By contrast, for DUNE Celeritas will track low-energy photons, or light particles generated when neutrinos interact with liquid argon inside the detector.
"The interactions generate millions of light particles that we need to track to different detectors characterizing their energies and where they're going," Johnson said. "That's what makes this an especially good problem for GPUs. A lot of things are happening at the same time and about the same place."
The researchers have performed scoping-level simulations on Frontier. "We're doing things that are very new and have applications across multiple fields," Johnson said. "It's a really exciting thing to be doing for that reason alone."
Preparing for next-generation experiments
Looking ahead, the researchers aim to integrate Celeritas into existing software frameworks for production-level simulations. Stefano Tognini, a Celeritas team member from ORNL's Computational Sciences and Engineering Division, works closely with physicists of the COHERENT experiment. Jason Newby leads the experiment at ORNL, working with Physics Division colleagues Lorenzo Fabris, Brennan Hackett, Matthew Heath, Alfredo Galindo-Uribarri and Chang-Hong Yu, along with dozens of researchers from about 30 other institutions. This 2017 experiment at ORNL was the first to measure coherent scattering of low-energy neutrinos off nuclei.
Because Celeritas is new software, the researchers had to develop many fundamentally new algorithms to get it working. "Ultimately, we're going to use Celeritas to generate data to train new AI models," Johnson said.
In DOE's Office of Science, its Advanced Scientific Computing Research and High Energy Physics programs support the Celeritas project through its Scientific Discovery through Advanced Computing program.
UT-Battelle manages ORNL for the Department of Energy's Office of Science, the single largest supporter of basic research in the physical sciences in the United States. The Office of Science is working to address some of the most pressing challenges of our time. For more information, please visit energy.gov/science . - Dawn Levy
