Researchers have demonstrated three new techniques to prevent so-called "backdoor attacks," which induce malicious or undesirable behavior in AI models. The researchers have also proposed a concept that would repurpose backdoor attack techniques to create an additional layer of security for users.
Backdoor attacks refer to instances in which someone has secretly embedded a "trigger" in an AI model that - when activated - forces the model to perform specific actions.
"The important point is that the model can behave normally when the trigger is absent, making the backdoor difficult to notice during ordinary use or testing," says Jung-Eun Kim, corresponding author of four papers describing the work and an assistant professor of computer science at North Carolina State University. "But when the trigger is activated, it can cause the model to do things an attacker intended such as generating unwanted text and executing malicious commands.
"Given that AI models are used for everything from financial transactions to software development, the ability to control behaviors of AI represents a significant vulnerability," says Kim.
As AI models have become more common, and are being used for more applications, there is an increasing need to "fine-tune" the models to perform these new tasks effectively. Fine-tuning refers to additional training for AI models, which involves using additional datasets. This creates new opportunities for backdoor attacks, which can be tailored to the specific functions that model is being fine-tuned to perform.
"This is a well-known vulnerability, but backdoor attacks are difficult to prevent because they are hard to identify until the trigger has been activated," says Jianwei Li, first author of all four papers and a Ph.D. student at NC State. "Our goal with this work was to prevent backdoor attacks before deployment in order to make AI safer and more trustworthy."
To address backdoor concerns, the researchers looked at three different levels of AI models: features, architecture, and weights.
"Weights are the parameters that make up a model," says Li. "Features are internal representations of the data in a model. And architecture refers to the internal operating components of the model.
"These are all essential elements of an AI model, and we've developed techniques that modify all three of those levels to remove or mitigate backdoor attacks without making significant sacrifices in performance," says Li.
The researchers also propose the use of "secret alignment," using triggers like those used in backdoor attacks to perform specific beneficial tasks. For example, a trigger could be installed that would be known only to management-level users, giving them restricted access to secure data or control functions.
"Overall, we've introduced a comprehensive suite of techniques that reduces the threat of backdoor attacks, all of which could be implemented immediately," says Kim. "And we've also outlined an approach for repurposing backdoor triggers into useful features."
The paper focusing on AI model features, "Backdoor Purification for LoRA-Tuned LLMs via Null-Space Projection," will be presented at the Fortieth Annual Conference on Neural Information Processing Systems (NeurIPS), being held in December in Sydney (Australia), Atlanta, (U.S.) and Paris (France). The paper focusing on AI model architecture, "Backdoor Containment via Expert Quarantine and Shutdown in LLMs," will also be presented at NeurIPS and was co-authored by NC State Ph.D. student Min-Seon Kim.
The paper focusing on AI model weights, "Purifying Generative LLMs from Backdoors without Prior Knowledge or Clean Reference," was presented at the Fourteenth International Conference on Learning Representations (ICLR), held April 23-27 in Rio de Janeiro, Brazil. The paper proposing the concept of "secret alignment," "Position: Retire the 'Positive Backdoor' Label - Secret Alignment Requires Strict and Systematic Evaluation," was presented at the Forty-Third International Conference on Machine Learning (ICML), held July 6-11 in Seoul, South Korea.
