For Nicolas Papernot, the secret to strengthening cybersecurity in the age of AI is transparency - sharing information about threats and allowing researchers to probe these systems before attackers do.
He practises what he preaches. An associate professor of electrical and computer engineering at the University of Toronto's Faculty of Applied Science & Engineering, Papernot drew global headlines in June when his lab demonstrated how to use free models to build an AI-powered worm that adapts its strategy as it spreads from one device to the next, taking over machines and stealing their computing power along the way.
After careful review, Papernot's team decided to go public with their findings to give defenders a head start to build safeguards against such threats, which are believed to be in development behind closed doors.
Papernot - who is also a faculty member at the Vector Institute, where he holds a Canada CIFAR AI Chair - will discuss what AI-powered threats mean for critical infrastructure, the economy and national security at a Schwartz Reisman Institute for Technology and Society event on Sept. 10 . The event will be livestreamed on YouTube .
Papernot spoke to U of T News reporter Adina Bresge ahead of the talk about the state of cybersecurity, the role universities can play and how everyday users can protect themselves.
How worried should we be about cybersecurity in the age of AI?
We should be concerned, but not just because AI suddenly created an entirely new class of threats. What AI is really doing is exposing how weak our existing cybersecurity already is.
Right now, we live in a precarious balance between attackers and defenders. Systems are protected just enough that it's not always worth it for attackers to target them. But with AI, the cost of mounting an attack drops dramatically, and that balance starts to tip.
The upside is that we already know many of the steps we need to take. It would actually be easier to invest in basic cybersecurity - things like stronger authentication and better protection of critical systems - than to hope a super-sophisticated AI defence system will magically fix everything for us.
Why do you place such an emphasis on transparency in AI?
People often describe AI as a "black box," and that's part of the problem. If nobody outside a company can see what these systems are doing, it's very hard to tell whether they're safe.
In an ideal world, we wouldn't need to take companies at their word. They'd show us evidence of how their systems work and what data they use. There are methods - like cryptography - that would allow them to do that without revealing trade secrets, but right now, they're too expensive to implement at scale.
That leaves us in a situation where we're relying on trust instead of proof. Transparency is how we bridge that gap: independent evaluations, clearer testing methods and more openness about how models are built and deployed.
How can universities to help solve this problem?
Universities are well positioned to act as a transparency bridge between companies, governments and the public.
We already operate under strict ethical and security guidelines, and our work is designed to be publicly shared and scrutinized. That makes universities a natural place to test powerful AI systems in contained environments, stress-test their safeguards and report honestly on what we find.
That independence benefits everyone. Companies gain credibility that is hard to achieve on their own. Governments can rely on the findings without raising concerns about interference. And the public knows these systems have been tested by experts with no stake in the results.
The goal isn't to criticize anyone. It's to help everyone understand the risks and how to manage them. We often hear promises about how AI is going to cure diseases or solve climate change if we just make it powerful enough. But we may never reach that optimistic future if, along the way, we suffer major security failures and large-scale attacks.
A prosperous AI future is a safe AI future. Strengthening cybersecurity isn't an obstacle to progress - it's what allows us to roll out AI systems widely and responsibly, so they can actually deliver the benefits we're hoping for.
How are the risks your research highlighted playing out in the real world?
We're seeing that some companies are running extremely powerful models with surprisingly weak safety measures around them. These companies should adopt today's cybersecurity practices to contain their experiments appropriately, even if it comes at the cost of decreased model performance.
Our team - Jonas Guan, Tom Blanchard, Hanna Foerster, Hengrui Jia and Gabriel Huang - built an AI-powered worm in a controlled environment to study how an autonomous system might move across a network and what it takes to keep one in check. That work showed us where things could go wrong if you weren't careful. We've now seen it happen .
I wouldn't say we had a crystal ball. But it does show the value of doing this kind of work at universities. By building and containing these systems carefully, we can find weak points and tell companies and regulators where to strengthen their defences - before something goes wrong.
How can people protect themselves today?
The cybersecurity hygiene we've been talking about for years matters more than ever - and we can't afford to put it off. Strong passwords, multi-factor authentication and up-to-date software are still your first line of defence.
What's changed is we're giving AI tools a lot of control over our digital lives, and we don't always know what they'll do with it. We have to think carefully about what we're handing over, and how that information can be passed between tools. If you give an AI agent access to your email and your calendar, for example, there's nothing to stop it from pulling a private message about your manager and attaching it to the invite for your next one-on-one.
On top of that, attackers can manipulate AI systems to act against you. An AI agent reading your email can't always tell the difference between a message from a colleague and malicious instructions that someone planted inside it. That command could tell the agent to comb through your inbox and send what it finds to your contact list.
Once you've given that access, you can't really take it back. So be cautious about which AI tools you use, what you share with them and what other services you let them control.
