Climate researchers, many affiliated with Rutgers-led consortium, call for changes in how risk projections are reported and replicated
Science is a crucial component of responding to climate change. When results are transparent and research is replicable, they can support more effective adaptation and mitigation.
But the opposite also is true, and according to a new Rutgers-coauthored paper published in the Proceedings of the National Academy of Sciences (PNAS), climate risk analysis is underperforming in some of the most basic expectations of openness.
The study found that of 258 top-cited climate risk papers published in leading climate risk journals in 2021-2022, only 4% shared both their data and the computer code used to produce their results. While many of the journals had guidelines governing data openness, they were at the time rarely enforced, the researchers found.
Robert Kopp, a Distinguished Professor in the Department of Earth and Planetary Sciences at the Rutgers School of Arts and Sciences and a coauthor of the PNAS Perspective, said this collective omission limits science's ability to interrogate and replicate relevant research with the urgency needed.
"When a risk analysis is based on algorithms or code that are not transparent or reusable, it cannot be properly verified," said Kopp, who directs the Megalopolitan Coastal Transformation Hub (MACH), a National Science Foundation-funded consortium at Rutgers whose researchers and affiliates led the study. "Thus, the public is really taking it on faith that the conclusions are valid."
Over the last decade, climate risk analytics have been a booming business. Projections about climate impacts are increasingly guiding insurance pricing, factoring into municipal credit and bond ratings, and informing portfolio allocations.
Yet data limitations and fundamental validation challenges raise important concerns about the reliability of such projections. A 2023 White House report warned that commercial risk predictions were sometimes "of questionable quality," in part because companies producing the assessments were not openly sharing their methodology.
To guard against these concerns in academia, peer-reviewed journals typically require researchers to share the data and computer code used to generate results and the process by which conclusions were reached. This information is then used by peer reviewers and other researchers to verify the findings and assess their robustness.
But as Kopp and fellow researchers discovered, source data and code aren't always made fully available to readers, which hurts public trust and stifles the scientific process, they said.
"When data and code are difficult or impossible to access, it slows down climate risk science -especially in application areas that require integrating many areas of expertise, methods and tools," said Adam Pollack, an assistant professor at the University of Iowa, an affiliated researcher at MACH and lead author of the study.
To quantify the transparency and usability of leading climate risk research, researchers from more than a dozen universities and institutions in the United States and Europe identified and reviewed the top-cited articles from 11 of the field's biggest journals - including Nature Climate Change, Earth's Future (of which Kopp is an editor), Communications Earth & Environment and Lancet Planetary Health - published during a 24-month period.
Articles were classified as having data and underlying programming code that were "open," "mostly open," "mostly closed" or "closed," based on data accessibility and the level of barriers to code reuse.
Even among the most influential climate risk publications, openness was severely limited. Of the 228 studies with code that was "closed" or "mostly closed" to researchers, 88% provided no explanation for falling short of their stated transparency standards.
The researchers stressed the lack of openness in the top climate risk publications doesn't indicate flawed research or undermine the big-picture findings of climate science. The Earth is warming, climate is changing, climate risk is increasing, and human activity is driving these shifts, they said.
And yet, the lack of transparency and reusability undermines what is necessary "for cumulative building of knowledge through community scrutiny, comparison and iteration," they wrote.
History offers powerful examples of what openness can achieve. For instance, as the researchers noted, Nobel laureate William Nordhaus' openly available Dynamic Integrated Climate-Economy model catalyzed decades of economic research refining estimates of the social cost of greenhouse gas emissions and directly informed federal environmental regulations.
Lisa Auermuller, executive director of MACH and a study coauthor, said, "In addition to citing and describing the challenges, we also provide a set of low-cost measures that researchers and journals can adopt now - measures we work to implement within our own research at MACH. These actions will increase transparency and reusability."
First, journals should enforce or strengthen transparency standards. Second, researchers should follow established norms for sharing datasets and code or provide clear statements to explain any limitations. Third, institutions, funders and professional societies must commit sustained investment to realize the full value of reusable climate risk science. Finally, studies like this one should be repeated frequently to gauge the field's progress.
"When foundational data and models are openly available and reusable, researchers, government agencies and service providers can make faster, more reliable progress in their respective roles," the researchers concluded.
Other coauthors affiliated with the Megalopolitan Coastal Transformation Hub include Moira Scheeler, a MACH program coordinator based at Rutgers; co-principal investigator Klaus Keller, the Hodgson Distinguished Professor at the Thayer School of Engineering at Dartmouth College;
Sönke Dangendorf, an assistant professor in the Department of River-Coastal Science and Engineering at Tulane University; Prabhat Hegde, a doctoral student at Dartmouth; Casey Helgeson, an associate research professor in the Earth and Environmental Systems Institute at Penn State; and Nancy Tuana, who holds the DuPont/Class of 1949 Professorship in Philosophy and Women's Studies at Penn State.
