PROVIDENCE, R.I. [Brown University] — A statistical approach being used to support a new class of Alzheimer's drugs may lead to overstated claims about how the drugs work, according to a new study led by researchers at the Brown University School of Public Health.
Published in JAMA Neurology, the research letter focused on quantile aggregation, a new statistical technique that divides people into groups, averages their results together and then looks for patterns across those groupings.
The letter examined how the approach works when applied to cognition and amyloid, a protein that builds up in the brains of people with Alzheimer's disease. The approach was originally published in an analysis of Eli Lilly and Company's Alzheimer's drug donanemab.
"Many researchers believe reducing amyloid buildup could slow memory loss and cognitive decline associated with the disease, making it a major target for newer Alzheimer's drugs," said the study's senior author Sarah Ackley, who is an assistant professor of epidemiology at Brown's School of Public Health and runs the Computational Epidemiology Lab "The problem is that using this method to assess the effect of amyloid removal on cognition can produce misleading results."
The researcher's concern is that the approach can make the link between amyloid reduction and cognitive improvement appear much stronger than it is, according to the analysis. The study the researchers looked at was a reanalysis of the original data from the randomized control trial on donanemab. It was led by scientists affiliated with the drug maker.
"When we did simulations, we found that you could basically take a very weak relationship between amyloid and cognition and make it appear as something that looked really strong and important," Ackley said.
The team expected there might be problems with the method but were struck by how large the effects were.
In simulations that were designed to reflect the conditions from recent trials, the team found the method showed the relationship between amyloid and cognition to be 29 times higher than its actual magnitude.
The researchers said this happens because by combining large groups of patients and averaging their results together, the process hides variability in cognitive change between patients. That can make it look like reducing amyloid is more predictive of cognitive benefit than it is.
The method also combines patients who received the drug with those who received a placebo. Without that randomization, the analysis cannot reliably determine whether amyloid reduction is actually causing cognitive benefit or whether other factors are at play, according to the study.
To illustrate this, the team also tested the method using data from the Anti-Amyloid Treatment in Asymptomatic Alzheimer's Disease Study that ran from 2014 to 2023. That trial tested if the drug solanezumab could slow cognitive decline in older adults with elevated amyloid levels in their brains, an early sign associated with Alzheimer's disease.
The trial showed solanezumab did not slow cognitive decline, yet when the team ran the data from that trial through the analysis of donanemab using the quantile aggregation method, it came back showing a strong link between lower amyloid and better cognitive outcomes.
"We basically built a case that this method is going to give you misleading results," Ackley said. "It made a failed trial look like it had successfully removed amyloid and that the removal of amyloid had reduced cognitive decline. In reality, the drug did neither of these things."
Ackley emphasized that the findings do not settle the broader question of how the new Alzheimer's disease drugs work. Instead, she says, the work highlights a need for more rigorous statistical methods. She also emphasized the need for more data sharing in Alzheimer's research, especially as new treatments become more widely used and covered by public programs like Medicare.
"Our study was simple, but a great demonstration of the value of academic research," she said. "Working outside of industry incentives gave us the freedom to closely examine a methodological issue affecting how some of the most consequential new drugs are understood."
