Cybersecurity training programs as implemented today by most large companies do little to reduce the risk that employees will fall for phishing scams–the practice of sending malicious emails posing as legitimate to get victims to share personal information, such as their social security numbers.
That's the conclusion of a study evaluating the effectiveness of two different types of cybersecurity training during an eight-month, randomized controlled experiment. The experiment involved 10 different phishing email campaigns developed by the research team and sent to more than 19,500 employees at UC San Diego Health.
The team presented their research at the Blackhat conference Aug. 2 to 7 in Las Vegas. The team originally shared their work at the 46th IEEE Symposium on Security and Privacy in May in San Francisco.
Researchers found that there was no significant relationship between whether users had recently completed an annual, mandated cybersecurity training and the likelihood of falling for phishing emails. The team also examined the efficacy of embedded phishing training – the practice of sharing anti-phishing information after a user engages with a phishing email sent by their organization as a test. For this type of training, researchers found that the difference in failure rates between employees who had completed the training and those who did not was extremely low.
"Taken together, our results suggest that anti-phishing training programs, in their current and commonly deployed forms, are unlikely to offer significant practical value in reducing phishing risks," the researchers write.
Why is it important to combat phishing?
Whether phishing training is effective is an important question. In spite of 20 years of research and development into malicious email filtering techniques, a 2023 IBM study identifies phishing as the single largest source of successful cybersecurity breaches–16% overall, researchers write.
This threat is particularly challenging in the healthcare sector, where targeted data breaches have reached record highs. In 2023 alone, the U.S. Department of Health and Human Services (HHS) reported over 725 large data breach events, covering over 133 million health records, and 460 associated ransomware incidents.
As a result, it has become standard in many sectors to mandate both formal security training annually and to engage in unscheduled phishing exercises, in which employees are sent simulated phishing emails and then provided "embedded" training if they mistakenly click on the email's links.
Researchers were trying to understand which of these types of training are most effective. It turns out, as currently administered, that none of them are.
Why are cybersecurity trainings not effective?
One reason the trainings are not effective is that the majority of people do not engage with the embedded training materials, said Grant Ho, study co-author and a faculty member at the University of Chicago, who did some of this work as a postdoctoral researcher at UC San Diego. Overall, 75% of users engaged with the embedded training materials for a minute or less. One-third immediately closed the embedded training page without engaging with the material at all.
"This does lend some suggestion that these trainings, in their current form, are not effective," said Ariana Mirian, another paper co-author, who did the work as a Ph.D. student in the research group of UC San Diego computer science professors Stefan Savage and Geoff Voelker.
A study of 19,500 employees over eight months
To date, this is the largest study of the effectiveness of anti-phishing training, covering 19,500 employees at UC San Diego Health. In addition, it's one of only two studies that used a randomized control trial method to determine whether employees would receive training, and what kind of phishing emails–or lures–they would receive.
After sending 10 different types of phishing emails over the course of eight months, the researchers found that embedded phishing training only reduced the likelihood of clicking on a phishing link by 2%. This is particularly striking given the expense in time and effort that these trainings require, the researchers note.
Researchers also found that more employees fell for the phishing emails as time went on. In the first month of the study, only 10% of employees clicked on a phishing link. By the eighth month, more than half had clicked on at least one phishing link.
In addition, researchers found that some phishing emails were considerably more effective than others. For example, only 1.82% of recipients clicked on a phishing link to update their Outlook password. But 30.8% clicked on a link that purported to be an update to UC San Diego Health's vacation policy.
Given the results of the study, researchers recommend that organizations refocus their efforts to combat phishing on technical countermeasures. Specifically, two measures would have better return on investment: two-factor authentication for hardware and applications, as well as password managers that only work on correct domains, the researchers write.
This work was supported in part by funding from the University of California Office of the President "Be Smart About Safety" program–an effort focused on identifying best practices for reducing the frequency and severity of systemwide insurance losses. It was also supported in part by U.S. National Science Foundation grant CNS-2152644, the UCSD CSE Postdoctoral Fellows program, the Irwin Mark and Joan Klein Jacobs Chair in Information and Computer Science, the CSE Professorship in Internet Privacy and/or Internet Data Security, a generous gift from Google, and operational support from the UCSD Center for Networked Systems.
Understanding the Efficacy of Phishing Training in Practice
Grant Ho, Ariana Mirian, Elisa Luo, Stefan Savage and Geoffrey M. Voelker, Department of Computer Science and Engineering, UC San Diego
Grant Ho is currently a faculty member at the University of Chicago. Mirian is currently a senior security researcher at Censys.
Khang Tong, Euyhyun Lee and Lin Liu, Biostatistics, Epidemiology and Research Design, UC San Diego Health
Christopher A. Longhurst and Christian Dameff, Jacobs Center for Health Innovation and UC San Diego Health
