Two researchers from the University of Nottingham are part of an international collaboration of 865 researchers behind the SCORE (Systematising Confidence in Open Research and Evidence) programme, which has released major new findings on research credibility in three papers published in the world's leading science journal Nature.
SCORE is a large-scale, multi-method initiative offering new evidence on the reproducibility, robustness and replicability of research, and is designed to improve how scientific credibility is assessed in the social and behavioural sciences. SCORE is funded by the US Defence Advanced Research Projects Agency (DARPA).
The programme was led by an international team from the Centre for Open Science, the University of Melbourne, and Pennsylvania State University. It brought together teams from multiple institutions – including the University of Nottingham's Dr Joris Schroeder, Research Fellow in the School of Economics, and Dr Christopher Madan, Assistant Professor in the School of Psychology – to evaluate research using approaches such as reproduction and replication studies, expert assessments, and machine-learning models, while also creating one of the largest open datasets on research credibility.
When we read a published study, how confident can we really be in its findings? SCORE takes on this question at an unprecedented scale, and I'm glad to have been a small part of it.
Analysing claims from 3,900 papers across 62 journals (2009–2018), SCORE assessed the reliability of research (based on the three dimensions of reproducibility, robustness, and replicability), and the programme also tested whether humans and machines can predict which findings will replicate.
Key findings include:
- Reproducibility (same data, same analysis): Limited transparency hindered assessment – data was available for only 24 per cent of papers. Among those tested, 72 per cent reproduced approximately and 53 per cent exactly, with higher success when data and code were shared.
- Robustness (same data, different analyses): Results varied depending on analytical choices. Only 34 per cent of re-analyses closely matched original results, though 74 per cent reached similar overall conclusions.
- Replicability (new data): About 49 per cent of studies replicated successfully, with replication effects less than half the size of original findings.
It was great to be able to contribute to this project. Large field-wide collaborative projects like this are what truly move the field forward.
Additional findings show that credibility is multi-dimensional, with weak links between measures. Human predictions of replicability are fairly accurate, while machine predictions are less reliable. No single field consistently outperforms others, and there is no single metric for research credibility.
Overall, the results highlight that verifying research is complex but essential. The SCORE programme provides openly available data, tools, and materials to support future research and improve how scientific credibility is assessed.
The three research papers have been published in Nature.
Investigating the replicability of the social and behavioural sciences. Nature, 652(8108), 143–150.
