OAK BROOK, Ill. – A hybrid reading strategy for screening mammography, developed by Dutch researchers and deployed retrospectively to more than 40,000 exams, reduced radiologist workload by 38% without changing recall or cancer detection rates. The study, which emphasizes AI confidence, was published today in Radiology , a journal of the Radiological Society of North America ( RSNA ).
"Although the overall performance of state-of-the-art AI models is very high, AI sometimes makes mistakes," said Sarah D. Verboom, M.Sc., a doctoral candidate in the Department of Medical Imaging at Radboud University Medical Center in the Netherlands. "Identifying exams in which AI interpretation is unreliable is crucial to allow for and optimize use of AI models in breast cancer screening programs."
The hybrid reading strategy involves using a combination of radiologist readers and a stand-alone AI interpretation of cases in which the AI model performs as well as, or better than, the radiologist.
"We can achieve this performance level if the AI model provides not only an assessment of the probability of malignancy (PoM) for a case but also a rating of its certainty of that assessment," Verboom said. "Unfortunately, the PoM itself is not always a good predictor of certainty because deep neural networks tend to be overconfident in their predictions."
To develop and evaluate a hybrid reading strategy, the researchers used a dataset of 41,469 screening mammography exams from 15,522 women (median age 59 years) with 332 screen-detected cancers and 34 interval cancers. The exams were performed between 2003 and 2018 in Utrecht, Netherlands, as part of the Dutch National Breast Cancer Screening Program.
The dataset was divided at the patient level into two equal groups with identical cancer detection, recall and interval cancer rates. The first group was used to determine the optimal thresholds for the hybrid reading strategy, while the second group was used to evaluate the reading strategies.
Of the uncertainty metrics evaluated by the researchers, the entropy of the mean PoM score of the most suspicious region produced a cancer detection rate of 6.6 per 1,000 cases and a recall rate of 23.7 per 1,000 cases, similar to rates of standard double-reading by radiologists.
The final hybrid reading strategy involved AI evaluating every screening mammogram to produce two outputs: the PoM and an uncertainty estimate of that prediction. When AI determined the PoM was below the established threshold with certainty, the case was considered normal. When AI detected a PoM above the established threshold, women were recalled for further testing, but only when that prediction was deemed confident. Otherwise, the exam was double-read by radiologists.
