Material properties such as sound insulation, resistance to extreme heat, and thermal expansion originate from how the zillions of microscopic building blocks (nuclei and electrons) interact at equilibrium and respond to perturbations. Atoms are typically about one ten-billionth of a metre across, so there can be a lot of parts to keep track of—a task that is complicated at the quantum mechanical level, where particles are neither here nor there until observed.
In recent years, researchers have turned to machine learning (ML) to overcome the challenges of tracking countless quantum particles while connecting these atomic-level details to observable physical properties. Models abound, but can they be trusted?
In a new paper published in Nature Communications, Michele Simoncelli , assistant professor of applied physics at Columbia, sets a benchmark for evaluating ML models that aim to predict the thermal and mechanical properties of different materials. The benchmark, developed with colleagues Balázs Póta, Paramvir Ahlawat, and Gábor Csányi at the University of Cambridge, adds a critical new level of physics-awareness to computer-driven outputs.
"We can call an atomistic ML model 'physics-aware' when it predicts the macroscopic properties of materials as a consequence of correctly describing the materials' atomistic physics—namely, their atomic vibrations," said Simoncelli. "There are cases in which ML models give apparently sensible predictions, but for the wrong reasons."
This new benchmark is setting the record straight, as it is already in use by a growing number of research groups and companies, including at Meta , Microsoft , and startups such as Radical AI and Orbital Materials .
Machine learning takes on quantum mechanics
The quantum behavior of a solid can be determined from the solution of the infamous Schrödinger equation, which is too complex to solve analytically for realistic materials and therefore requires computational solutions as accurate as possible. The Schrödinger equation describes the quantum behavior of particles such as electrons and nuclei. Its solutions describe the energy levels and "shape" (spatial probability distribution) of a quantum particle, allowing researchers to calculate the microscopic forces between atoms in a material. Obtaining accurate solutions becomes increasingly computationally expensive as the number of particles grows. Modeling a molecule or material can mean accounting for hundreds or even thousands of electrons.
These ML models, known as machine-learning interatomic potentials, attempt to speed up this process by learning from datasets of atomic positions, energies, forces, and stresses obtained from quantum-mechanical calculations. They predict atomic interactions without explicitly solving the Schrödinger equation each time. These models can run 1000 times faster than traditional approaches, but they aren't perfect.
"These models had been compared to each other mainly on their performance to predict energy, which is indeed the important quantity for most material properties," said Póta, the graduate student who is first author of the article. "But this can miss potential errors in forces, which determine the dynamics of atoms, which is particularly relevant to predict how they respond to temperature or mechanical perturbations."
The missing ingredient? More physics
Atoms in materials vibrate at the quantum mechanical level. These thermal vibrations are extraordinarily small, fast, and short-lived, but critical to properties like thermal conductivity and thermal expansion. Models that appear to be well-suited to make energy-based predictions have tended to distort the small effects of vibrations, missing important details about the material's underlying physics as a result—even when they happen to get the bigger picture mostly right.
The team began their benchmarking efforts about two years ago. At the time, ML models were attracting increasing attention for their out-of-the-box performance, and they wanted to see how well these could predict their property of interest, thermal conductivity.
They compared the predictions of several models available at the time against traditional quantum-mechanical calculations for a set of over 100 different crystalline materials. They were impressed by the results: among models with very similar accuracy in predicting formation energies and material stability, some accurately predicted both atomic vibrational properties and macroscopic thermal conductivity. Others were significantly off on atomic vibrations but still predicted thermal conductivity accurately because microscopic errors cancelled out, while others were off on both. With a small amount of additional, material-specific training, the team also achieved agreement within a few percent of reference calculations in selected materials and, for lithium bromide, agreement with experiments.
The article describes the shortcomings of these models in capturing atomic vibrations and how these can translate into inaccuracies in macroscopic properties. "Our benchmark provides a 'physics-aware' signal to ML developers to optimize their models," said Póta.
Interest—and adoption—have grown quickly.
Adopting the benchmark
In addition to being used by companies such as Meta and Microsoft , the benchmark has been incorporated into Matbench Discovery . This interactive leaderboard ranks ML models on their ability to predict crystal stability, structure, and thermal conductivity.
"I was very happy when Simoncelli's group offered to contribute their test set and metric design to diversify the Matbench leaderboard," said Janosh Riebesell, who created and maintains Matbench Discovery. "It revealed large performance gaps between models that looked nearly identical on crystal stability prediction, exposing weaknesses in models lacking strong physical constraints. Moreover, the thermo-mechanical properties that need to be modeled accurately to master heat conductivity are highly technologically relevant, so it gives us an early signal on how useful ML models are becoming for advancing materials-driven technologies."
Thermal properties are critical to thermal management across many areas of active research, from microchips, batteries, and quantum computers to jet engines, spacecraft heat shields, and nuclear reactors. "Thermal conductivity is especially sensitive to the microscopic vibrational physics of a material, but the general idea is relevant to other physical properties as well, a topic which we are currently exploring in the group," said Simoncelli.
Getting the Right Answer, For the Right Reasons
AI and machine learning models for atomic interactions are beginning to drive discovery itself. Models are proposing new materials, predicting their properties, and flagging the best candidates for synthesis, with robotic labs increasingly closing the loop.
In such a setting, a model that quietly gets aspects of a material's underlying physics wrong doesn't just score poorly on a benchmark: it can discard good options and promote bad ones, with no one checking each answer.
Efforts like those of Simoncelli and his colleagues will be critical to ensuring that ML models get their answers right…for the right reasons.
