El Capitan, the world's fastest supercomputer, may be new to scientists at Lawrence Livermore National Laboratory (LLNL), but it's already allowing them to explore physical systems in ways that weren't possible before.
With the arrival of El Capitan, LLNL researchers are entering a new era of scientific simulation - one in which they can model extreme physical events with unprecedented resolution, realism and speed. From capturing the chaotic spray of molten metal to the turbulence of fluid flows, the exascale machine is revealing worlds that were previously beyond reach, and it's doing so thanks to the close collaboration of hardware, software and science teams that makes LLNL uniquely equipped to lead in this space.
In one of the first examples of the new capability brought to bear by El Capitan - especially when it comes to how materials behave under extreme conditions - researchers used the machine to simulate what happens to a tin surface when it's struck by powerful shock waves and high-speed impacts.
LLNL physicist Kyle Mackay and his team used the Lab's ARES code - a sophisticated tool for modeling high-energy density physics and inertial confinement fusion (ICF) experiments - to run detailed simulations of the tin under stress.
"The shocks were strong enough to melt the metal and throw a spray of hot liquified tin, known as ejecta, ahead of the surface," Mackay explained. "The simulation was noteworthy for its high fidelity, employing advanced physics models for mechanisms like surface tension, detailed equations-of-state and especially its sub-micron mesh resolution."
The incredibly fine resolution the models achieved on El Capitan allowed the team to capture tiny features on the metal's surface, like machining grooves and internal voids. These small details have a big impact on how much ejecta gets produced, but they're usually missed in lower-resolution simulations. Even though the model covered just a few cubic millimeters of tin, the results were unprecedented, according to researchers.
"This could not have been done at this level of detail without the use of a machine like El Capitan," Mackay said.
But El Capitan doesn't work alone. The breakthroughs emerging from these early simulations are the result of a tightly integrated effort, driven by the skilled researchers behind the algorithms, models and multiphysics codes that are specifically engineered to take advantage of the cutting-edge hardware, producing a set of tools that are useable by a larger community. At LLNL, teams of domain scientists, code developers and computing experts work tirelessly, side-by-side, to co-design these capabilities from the ground up.
"This isn't about a machine, it's about this combination of deploying and managing advanced hardware with having all these world-class experts in hardware, software and science all under one roof, which is standard operating procedure at LLNL," said Weapon Simulation and Computing Associate Program Director Teresa Bailey. "This unique co-location and collaboration is what makes these kinds of advances possible."
Peering into plasma: shock-driven fluid instabilities captured in striking detail
One of the most stunning early advances enabled by El Capitan came from a set of simulations of shock-driven fluid instabilities, the kind that occur when extreme forces act across material boundaries.
Using LLNL's multiphysics code MARBL, a team led by Rob Rieben, and including Thomas Stitt, Aaron Skinner and Arturo Vargas, modeled the Kelvin-Helmholtz instability - a phenomenon that occurs when two fluids of different densities slide past each other under shear forces. The team replicated conditions from a previous laser experiment at the Omega laser facility, then scaled it into an ultra-high-resolution 3D simulation driven by El Capitan's enormous horsepower.
The shockwave in the model interacted with a prescribed ripple - a perturbation at the interface of two materials - producing a swirling, turbulent vortex as the materials mixed. These chaotic flows are incredibly difficult to capture, but thanks to 107 billion quadrature points and more than 8,000 AMD GPUs on El Capitan, the team was able to model the entire structure in extraordinary detail.
The result was a time-lapse of fluid behavior under intense energy conditions, revealing intricate shear and shock patterns that mirror - and in some cases go beyond - what's possible to observe in experiments.
"Experiments are the ultimate arbiter of physical truth but can be difficult to extract necessary data from," said Rieben. "High-fidelity simulations let us probe aspects of an experiment in a virtual manner that would not be possible to access in a real experiment. El Captain is a powerful scientific instrument for exploring physics via simulation at fidelities never seen before."
High-resolution turbulence reveals hidden dynamics in classic flow problem
Another high-resolution breakthrough used MARBL to simulate a classic but notoriously tricky problem in fluid dynamics: the lock-exchange. In this scenario, a heavy gas is held behind a barrier and then suddenly released into a lighter gas, triggering a rush of chaotic mixing, not unlike what happens during flows from volcanos, turbidity currents in the ocean or even flashover conditions in fires. Getting the physics right, especially the compressible turbulence that develops as the gases churn and interact with the container walls, requires precision, researchers said.
Jane Pratt and team ran a fully three-dimensional simulation using 1.8 billion quadrature points on El Capitan's 288-petaFLOP (288 quadrillion calculations per second) companion system Tuolumne, which shares an architecture with El Capitan but is about 1/10th the size. What made the run remarkable wasn't just the resolution, but the way the ALE-based code captured the decaying turbulence, offering a realistic depiction of how the turbulence interacts with shock waves.
"The lock-exchange problem is complicated because it involves a range of fluid instabilities interacting with layers of shear and with the walls, as gravity currents drive the turbulent entrainment of one fluid into another," Pratt said. "In the incompressible limit, our MARBL simulations compare closely with laboratory experiments."
"Using Tuolumne has allowed us to produce 3D simulations at extreme conditions that are difficult to simulate accurately because the flow conditions span a wide range of length and time scales," Pratt continued. "In the well-mixed end state, we study the properties of a truly turbulent flow using the framework of a modern ALE code, providing an exciting demonstration of ALE capabilities for the broader turbulence community."
Researchers said these kinds of early simulations run on El Capitan and Tuolumne help bridge the gap between theory, experiment and the real-world dynamics of extreme environments and hint at the kind of insight these next-generation machines are now putting within reach.
El Capitan transforms simulation and optimization workflows
In addition to allowing researchers to "zoom in" at much higher resolution, El Capitan makes it possible to simulate complex physical processes directly, rather than depending on simplified models to approximate them, Mackay said. By capturing the underlying physics in greater detail, researchers can reduce their reliance on assumptions - and even understand why certain models may fail under specific conditions - leading to more accurate and reliable predictions.
El Capitan's power also speeds up the process of running many different simulations at once - a method called ensemble generation. These kinds of studies, used to optimize designs or test how sensitive a system is to small changes, used to take months. Now, thanks to El Capitan, they can be done in mere days, or even hours. To illustrate the difference, Mackay used the analogy of designing a car engine.
"Imagine you're limited to running just one simulation per day and modeling only features larger than one inch due to limited computing power - you'd need to make numerous assumptions about what's occurring within the car's engine, slowing the optimization process significantly," he explained.
Thanks to El Capitan's 20-fold increase in computing power over its predecessor, Sierra, researchers can now run simulations much more frequently - in this hypothetical scenario, every hour instead of every day - and examine features at scales twenty times smaller.
"This allows for faster achievement of optimal designs and greater confidence in anticipated performance," Mackay explained.
For more on El Capitan, visit https://www.llnl.gov/news/highlights/el-capitan-high-performance-computing.
