A team of EPFL scientists tested the reproducibility of five decades of research on fly immunity. Most of those research results proved to be valid, but the team made one surprising discovery: the non-reproducible results were most often found in highly prestigious journals.
These are tough times for reproducibility. While this principle underpins the credibility of science, it has come under threat over the past 15 years - and researchers can't do much about it. The reproducibility of research results is what ensures that a given finding can be obtained by another group of scientists working in a different environment. Yet in a range of fields - from biology to human behavior - a growing number of studies are proving to be non-reproducible. This is raising concerns about how reliable some of the scientific literature is and how the existing research system rewards certain practices.
To evaluate the extent of the problem, Bruno Lemaitre, an immunologist who heads EPFL's Lemaitre Lab, launched the ReproSci project. He and his team spent six years working with other research groups to meticulously review, unpack and test the reproducibility of 400 research papers published in his field of drosophila immunity between 1959 and 2011. For each of these "cold cases" as he calls them (see interview below), the team distilled the authors' scientific claims - identifying over 1,000 in all - and compared them with the results of subsequent studies on the same topics. For 45 key claims, no subsequent studies were found so the team reproduced the experiments themselves. The outcome of all this is available online (reprosci.epfl.ch) for the benefit of the entire scientific community.
A growing trend
The ReproSci findings have been written up in two articles which are now available on the bioRxiv server and are being reviewed for publication on eLife (see below). The project's findings were mostly encouraging, as the team was able to confirm 61% of the scientific claims. "Only" 7% of the claims proved to be non-reproducible. Around 24% of the claims had never been verified, which is what the EPFL team did for some of them. And of these, a significant number couldn't be reproduced, meaning that the actual percentage of non-reproducible claims is higher than 7% - probably between 15% and 18%. Taking all these figures into account, the research team concluded that a little over 80% of the published scientific claims were sound. This suggests that reproducibility may not be as big of a problem as many fear. Notwithstanding some non-reproducible findings, research results seemed to be generally reliable. However, the ReproSci project revealed an interesting incongruity: the unverifiable claims were often found where you would least expect them: in flagship journals such as Nature, Science and Cell and in papers written by scientists at prestigious institutes.
The second article provided further insight into these conclusions. The metastudy found no correlation between non-reproducibility and the authors' experience or number of publications. Instead, the problem occurred most often in studies with researchers who came from outside the field or were involved in the study on an exploratory or opportunistic basis. Another key finding was that the percentage of non-reproducible claims tended to increase over time as the field became more popular and got more media coverage.
An impact on several fields
The ReproSci findings were consistent with those from other reproducibility studies. For instance, the Reproducibility Project: Cancer Biology initiative published by eLife found that nearly 50% of the claims in cancer studies appearing in top journals were not reproducible.
With ReproSci, Lemaitre has provided a rare snapshot of reproducibility across an entire life-science field, with quantitative conclusions and data made available in open access. What's more, the conclusions are relevant not only to drosophila research. They show how natural human tendencies - such as wanting to boost visibility, raise funding and follow the latest research trends - can influence the reliability of a study's results. By compiling evidence, shedding light on hidden aspects of the research system and calling on other scientists to get involved, the ReproSci team is getting people to think more broadly about what makes science trustworthy. Lemaitre's approach goes beyond the technical aspects to remind scientists that reproducibility isn't just about following the right protocol but should be an integral part of their culture and common practices.
"In science, trust is key"
Lemaitre, originally from Lille, France, came to EPFL 18 years ago. Drawing on his keen interest in philosophy and psychology, he has written several books cutting across research fields. We met with this "meta-thinker" in his office to learn more about his pioneering work.
What exactly does "research reproducibility" mean?
The concept of reproducibility is intuitively easy to grasp. It means that studies published in scientific journals should have adequate descriptions of their experiments, and if you replicate those experiments or test the authors' claims then you'll get the same results. However, reproducibility in practice is much more complicated. All experiments are grounded in their specific context: the research lab, the people involved, the equipment, how results are interpreted and so on.
There are at least two kinds of reproducibility. The first is in the strictest sense - if you run the same experiment under the same conditions, you should get the same results. The second relates to the reproducibility of the overall concept: whether the authors' central claim holds true when you test it. Both of these types matter. A faulty experiment could lead to a valid conclusion, and a properly conducted experiment could produce a result that's misinterpreted. Our ReproSci project looked mainly at conceptual reproducibility. If a claim held true when tested using different approaches, we concluded that it's sound, even if the initial experiment was questionable.
How extensive is the problem with reproducibility?
Achieving reproducibility is a problem that's always existed, but today it seems to have gotten worse - especially in highly competitive fields. A growing number of published articles are putting forth claims that either don't hold water or are highly exaggerated. We've seen this for a while now in some fields like psychology and cancer biology.
What are the ramifications?
There are several. Scientists can find themselves wasting significant amounts of time and money, and those who are meticulous in their research may feel the system is unfair, as they watch colleagues build their careers on shaky results. What's more, there are ramifications for public-health policy. To cite a well-known example, doctors long thought that a high-fat diet was a main cause of obesity - but that was based on erroneous studies. It turns out the problem stems primarily from a diet high in sugar.
That said, we should be careful to not swing too far in the other direction. If journals published only studies they knew to be 100% accurate, that would stifle creativity. Science advances through risk-taking. The idea is to strike the right balance - to prevent abuses of the system without discouraging novel ways of thinking.
Where did the idea for the ReproSci project come from?
You could say I have an encyclopedic mind - I love digging into the details, especially when it comes to nature. And I have good memory for things related to my field of drosophila immunity. Over time, I'd built up a list in my head of problematic studies I'd read. Then I obtained funding from the Swiss National Science Foundation and hired a researcher, Hannah Westlake, who did an incredible amount of painstaking work. We analyzed 400 studies published between 1959 and 2011 and identified 1,006 scientific claims. Then we checked to see what had become of those claims 14 years later. This research took over six years and involved several other research groups.
What were your goals?
First of all, we wanted to bring clarity to our field by exposing research results that weren't sound, in order to prevent young scientists from wasting their time. We also wanted to understand why those results were non-reproducible: was it related to the research style, or a certain research institute or methodology? In addition, we felt it was important to create a public database for everyone in the scientific community. Another aspect of our project was to interview the scientists whose papers we reviewed to see what they thought of our reproducibility project. The results of these interviews will be published in a third article.
What kinds of non-reproducibility did you find?
We found various kinds: experiments with non-reproducible results, experimental methods that were described so poorly that we couldn't replicate them, conclusions that were too vague, and, in many cases, claims that were exaggerated. Most of these studies weren't entirely wrong, but the conclusions were stated as being more broadly applicable and groundbreaking than they actually were.
What do you think is behind this tendency to exaggerate?
The dynamic is a little like that of a startup seeking investors: the company initially promises a revolution in order to raise funding, but a decade later you see that the promises weren't kept and that what it actually achieved is much more modest. But in the meantime, the investors' checks were cashed! In research, this dynamic is encouraged by calls for projects, by scientific journals looking for attention-grabbing headlines and breakthroughs, and by universities eager for eye-catching communications about a scoop. Before, exaggerated discourse was the standard in politics, where it helped people get elected. Now it has become common in science. Some scientists are able to take advantage of the tendency to exaggerate. Look at Didier Raoult, for example, who claimed during the pandemic that chloroquine could be used to treat COVID-19. Many scientists knew he was a controversial figure, but his statements were relayed widely in the press.
You said that prestigious journals like Nature, Science and Cell were the ones that tended to have more of the questionable claims. Why do you think that is?
It could be because these journals often seek to publish spectacular headlines. But the more extraordinary a claim, the more likely it is to be non-reproducible. There's also the fact that these journals prefer fairly simplistic descriptions - nuanced conclusions generally make less of an impact. So authors are encouraged to smooth things out and gloss over things that might contradict their hypotheses or results. And for authors, getting published in a top-tier journal gives them amazing visibility. James Watson [the US geneticist who won the 1962 Nobel Prize in Physiology or Medicine for his breakthrough work on DNA] said: "a researcher is someone who gets published in Nature." It's a cynical remark, but a telling one with regard to how much a researcher's career can depend on these journals.
You've written a book on narcissism.* Do you think that's related to this issue?
There's a lot of talk today about whether we're seeing an increase in narcissism in society. If so, then that could also affect research. Narcissism is a complex phenomenon, but it might explain why some scientists place so much value on visibility. One narcissistic bias is a tendency to exaggerate and seek short-term attention. This could bring immediate benefits but it comes at a cost for society. Getting an article published in a famous journal brings major personal satisfaction. But if the claims prove to be worthless, the entire community pays the price. I've also seen that many esteemed scientists have big egos. There are some advantages to having a healthy ego - it stimulates creativity, fuels your passion and underpins your networking skills. One disadvantage however is that it also prompts you to exaggerate. The image of the lone scientist toiling away in a humble quest for knowledge is a myth. Science is actually done by human beings who compete with each other and strive for recognition, just like everyone else.
What was the reaction when your two articles were published?
Some people were happy to see our effort to bring more clarity. Others accused me of getting on my high horse and handing out good and bad marks to my colleagues. It's not clear whether scientists themselves like being put under the microscope. The interviews that we'll publish in an upcoming article showed that many scientists didn't really know which of the published claims actually held up. I'd say that around half of the scientists we spoke with who had published unverifiable claims admitted to it. On a more upbeat note, the most effective way such claims are "corrected" is that they're simply forgotten. Findings that can't be replicated typically disappear into oblivion 10 or 15 years on.
Do you think transparency initiatives like the Transparency and Openness Promotion (TOP) Guidelines will help move research in the right direction?
Such guidelines are a step forward, as they require scientists to include detailed descriptions of their experimental protocols, for example. They can prevent the non-reproducibility that comes from insufficient information about how experiments were conducted. But they also come with the risk of weighing down the administrative side of things. Research papers have already become extremely long and complicated, with dozens of pages of supplementary material. Too many rules can make science unwieldy. We need to find the right balance - enough detail so that experiments can be replicated, but without drowning studies in paperwork. Transparency is essential in science, but trust is key. This trust can be found when most members of the scientific community follow the implicit rules of research, but it can be undermined by non-reproducibility and exaggerated claims.
*Bruno Lemaitre, An Essay on Science and Narcissism: How do High-ego Personalities Drive Research in Life Sciences?, EPFL Press, 2020.
