A widely cited Nature cover article from 2023 claimed that scientific innovation has significantly declined in the last decades. This finding was based on citation analysis of large sets of metadata concerning scientific papers and patents. Researchers from Vrije Universiteit Brussel (VUB) and KU Leuven have found that incomplete reference data account for most of the alleged decline in scientific disruption.
A hidden peak and a software bug
'Shortly after its publication, I attempted to reproduce the study using open data,' says doctoral researcher Vincent Holst, first author of the study. 'I discovered a large peak of papers receiving the maximal disruption score. This was strange, since the histograms published in Nature showed no such peak.'
The Belgian team traced this disparity back to a bug in the visualisation software. The version of the plotting library that the authors had used contained a bug. Due to this bug, the largest data points had been omitted from the published histogram, without any warning. While invisible in the graphs, the underlying data remained included in the main analysis behind the headline result.
'If the papers with maximum scores were genuinely groundbreaking, the conclusion would still hold,' says Professor Vincent Ginis, who led the study. 'But we found that they mainly correspond to dataset artefacts. When the references from a publication are not listed in the database, the method that the authors used wrongly labels that publication as maximally disruptive.'
Older database records are more likely to have incomplete reference data. As the quality of database records improved over time, these false maximum scores became less common, too. This created the appearance that scientific disruption was declining. The declining curves essentially mirror how metadata quality has increased over time, conclude the authors.
'When we filter out the database errors, the decline largely disappears,' says Vincent Holst. 'This holds for all of the scientific databases that we examined. In the study's largest dataset, the artefacts account for 93% of the reported decline between 1945 and 2010.'
From headlines to science policy
The original claim that science was getting less disruptive has appeared in more than 300 news stories worldwide.
The Belgian researchers submitted their critique in October 2023 to Nature. It took the journal 32 months to finally accept it. While the critique was under review, Nature still published an editorial and ran a News Feature drawing on the original conclusion. In these years, the original paper was cited over a 1,000 times more and used in at least 16 policy documents.
'Whereas the software bug was fixed by the open-source community in months, it took years for the scientific correction to get published,' says professor Ginis. 'And all this time, the journal's policies prohibited us from speaking to the press while our critique was under submission.'
Other researchers in the field have suggested restricting the analysis to papers with at least ten references and citations. When they follow this guideline, the Belgian team even finds a slight, statistically significant increase in disruption. Still, this is not their conclusion.
'Based on the used measure, we simply cannot make conclusive statements about whether science is getting more or less disruptive,' says professor Ginis. 'It has been shown, however, that the methodology and conclusions from 2023 do not provide a sound basis for our science policy.'
Publication
Vincent Holst, Andres Algaba, Floriano Tori, Sylvia Wenmackers, and Vincent Ginis, "Dataset artefacts can partially drive the measured decline in disruption", Nature (2026). [volume, pages and DOI to be added upon publication]
