Researchers have demonstrated a new way of attacking artificial intelligence computer vision systems, allowing them to control what the AI "sees." The research shows that the new technique, called RisingAttacK, is effective at manipulating all of the most widely used AI computer vision systems.
At issue are so-called "adversarial attacks," in which someone manipulates the data being fed into an AI system to control what the system sees, or does not see, in an image. For example, someone might manipulate an AI's ability to detect traffic signals, pedestrians or other cars - which would cause problems for autonomous vehicles. Or a hacker could install code on an X-ray machine that causes an AI system to make inaccurate diagnoses.
"We wanted to find an effective way of hacking AI vision systems because these vision systems are often used in contexts that can affect human health and safety - from autonomous vehicles to health technologies to security applications," says Tianfu Wu, co-corresponding author of a paper on the work and an associate professor of electrical and computer engineering at North Carolina State University. "That means it is very important for these AI systems to be secure. Identifying vulnerabilities is an important step in making these systems secure, since you must identify a vulnerability in order to defend against it."
RisingAttacK consists of a series of operations, with the goal of making the fewest changes to an image that will allow users to manipulate what the vision AI "sees."
First, RisingAttacK identifies all of the visual features in the image. The program also runs an operation to determine which of those features is most important to achieve the attack's goal.
"For example," says Wu, "if the goal of the attack is to stop the AI from identifying a car, what features in the image are most important for the AI to be able to identify a car in the image?"
RisingAttacK then calculates how sensitive the AI system is to changes in data and, more specifically, how sensitive the AI is to changes in data of the key features.
"This requires some computational power, but allows us to make very small, targeted changes to the key features that makes the attack successful," Wu says. "The end result is that two images may look identical to human eyes, and we might clearly see a car in both images. But due to RisingAttacK, the AI would see a car in the first image but would not see a car in the second image.
"And the nature of RisingAttacK means we can influence the AI's ability to see any of the top 20 or 30 targets it was trained to identify. So, that might be a car, a pedestrian, a bicycle, a stop sign, and so on."
The researchers tested RisingAttacK against the four most commonly used vision AI programs: ResNet-50, DenseNet-121, ViTB and DEiT-B. The technique was effective at manipulating all four programs.
"While we demonstrated RisingAttacK's ability to manipulate vision models, we are now in the process of determining how effective the technique is at attacking other AI systems, such as large language models," Wu says.
"Moving forward, the goal is to develop techniques that can successfully defend against such attacks."
The paper, "Adversarial Perturbations Are Formed by Iteratively Learning Linear Combinations of the Right Singular Vectors of the Adversarial Jacobian," will be presented July 15 at the International Conference of Machine Learning, being held in Vancouver, Canada. Co-corresponding author of the paper is Thomas Paniagua, a recent Ph.D. graduate from NC State. The paper was co-authored by Chinmay Savadikar, a Ph.D. student at NC State.
This work was done with support from the National Science Foundation under grants 1909644, 2024688 and 2013451; and from the Army Research Office under grants W911NF1810295 and W911NF2210010.
The research team has made RisingAttacK publicly available, so that the research community can use it to test neural networks for vulnerabilities. The program can be found here: https://github.com/ivmcl/ordered-topk-attack.
