Even researchers and people with extensive training in statistics often find it difficult to interpret statistical results correctly. This is shown by a new systematic review carried out by Uppsala University and others, in which researchers have analysed 55 studies involving over 15,000 participants.
Statistics are an important tool in many scientific fields. Using statistical methods, researchers can not only describe what has been observed but also draw more general conclusions. For example, this might involve investigating whether a medicine is effective or whether there is a link between air pollution and human health.
Drawing conclusions from statistical results is known as statistical inference. The way in which the results of statistical analyses are described and understood influences the conclusions that are actually drawn. Concepts such as the p-value, statistical significance and confidence intervals play a central role here.
But how well do we actually understand these concepts?
"We have found many studies that have examined this in various ways. What's striking is that, despite using different methods, the researchers have reached such similar conclusions: understanding of what can actually be inferred from statistical results is inadequate and misunderstandings are common," says Per Lytsy, affiliated researcher at the Department of Public Health and Caring Sciences, Uppsala University.
Misconceptions even amongst those with a knowledge of statistics
The systematic review comprises 55 studies published between 1963 and 2024. Most of the studies were questionnaire surveys in which participants were asked to interpret statistical results or to judge whether various statements were correct or incorrect. Other studies examined how statistical results influence confidence in scientific conclusions. Participants included students, researchers, healthcare professionals, statisticians and teachers of statistics.
A common misconception is that statistical significance is regarded as proof that a hypothesis is true or that an effect is significant. For example, a medicine may produce a statistically significant improvement without that improvement having any practical significance for patients.
One reason why misunderstandings are so common may be that statistical concepts are often difficult to understand and do not always correspond to the way we intuitively reason about probabilities and uncertainty. Furthermore, statistical methods are based on different theoretical foundations and are sometimes used in combination, which can make it even more difficult to understand which conclusions are justified.
"A common example is to interpret the p-value as the probability that the hypothesis we are testing is true or false. However, that conclusion cannot be reached on the basis of a p-value. Such misinterpretations can lead us to attribute a greater degree of certainty to research findings than is warranted," says Ronnie Pingel, senior lecturer at the Department of Statistics, Uppsala University.
Important for education and decisions
By compiling studies spanning several decades, the new review provides a broader picture of just how widespread and persistent these misconceptions are, even amongst people with extensive training in statistics. The researchers hope that the review will help to raise awareness of misinterpretations of statistics, improve the teaching of statistics and encourage a more critical examination of scientific findings.
"Being able to use statistical methods is not the same as understanding what the results mean. Our review shows that misunderstandings are common even among experienced researchers. This risks leading to incorrect conclusions and, ultimately, poorer decisions," says Lytsy.
