In early September, Jacob Coxon, an AI model trainer for Anthropic, published a stark message on X: "The people building AI earnestly believe that it could kill us all by the end of the decade. This is not a marketing stunt. ... No other human activity poses this level of danger."
In his Sept. 8 thread, now viewed more than 175 million times, Coxon announced that he was resigning, saying that neither Anthropic nor his previous employer, OpenAI—two of the world's leading AI companies—were "acting responsibly."
Image caption: Gillian Hadfield
"Do not underestimate the power of this technology," Coxon wrote. "These will soon be superhuman systems that can hack anything, revolutionize any field overnight, and acquire real power and resources."
Coxon's viral warning, affirmed by at least two other Anthropic team members, jarred an industry already on edge following the so-called Hugging Face incident. Over the summer, autonomous AI agents broke out of a controlled testing environment and onto the open internet, where they breached the third-party Hugging Face platform and began communicating via an unsanctioned message board. In all, some 1,200 AI agents were in communication, exchanging more than 70,000 messages, actively collaborating to cheat on assigned tasks, and manipulating logs to cover their tracks.
These incidents set off alarm bells among those who best understand the awesome, rapidly evolving power and potential of artificial intelligence. Among them was Gillian Hadfield, Bloomberg Distinguished Professor of AI Alignment and Governance at Johns Hopkins University, with appointments in both the School of Government and Policy and the Whiting School of Engineering. She is also affiliated with the university's Data Science and AI Institute.
"I have been thinking about powerful AI for about 10 years, but I was not all that worried until this year, and I have become more worried since," Hadfield said recently.
Over the past decade, Hadfield's scholarship has focused on the development of the legal, regulatory, and technical infrastructure for artificial intelligence that operates within human normative frameworks—that is, how can we put effective, accountable systems in place to prevent the types of disastrous outcomes described above? Her ideas are foundational to AI governance legislation introduced in the U.S. House of Representatives this summer, and she was recently named to a panel of experts advising California on the development of its AI policies, as well as to Canada's new National Council on Artificial Intelligence, which advises Prime Minister Mark Carney.
Hadfield spoke with the Hub about recent events in the AI industry, the need for a rapid response, and the regulatory approach she has championed.
We've seen some alarming headlines about AI safety and security in recent weeks. Should we be concerned? How concerned are you?
Yes, we should be worried. The big lesson from what we have seen in the last few months is that when AI agents go out into the world, get onto the internet, and start doing things, which is what we are building them for, they do weird stuff. They do things you didn't want them to do. The truth of the matter is that we do not know how to test or control our most powerful models. With today's AI, the machine is writing its own program, and that means humans have been moved back a step in our control over how that program works. The gap between what is happening in this technology and the tools we have to make sure it goes the way we want is just getting wider and wider.
My version of the risk is that we simply break human systems, like the economy or social order, because we got wrong what it takes for AI to take part competently in complex human societies. I worry about disruption. I worry about chaos. I worry about the internet becoming unavailable. I worry about banks crashing. But I think we still have the capacity to change the course we're on to a much safer one. I think we have a window to do something now, and that window could close. That is why it is really quite critical that governments act right now, and there are things we can do. I have been working on proposals for what we can do, and we have shovel-ready approaches that we can adopt.
Based on where we are, what might common-sense guardrails on AI look like, and how achievable are they? Who is well-positioned and incentivized to put those guardrails in place?
Just like with most other products or services, you have to make sure that what you are releasing is safe enough that it is not going to cause disaster or chaos when you put it out there. Unfortunately, we don't know what those guardrails are for the increasingly powerful systems that are being released at an incredible rate. Yes, the companies releasing AI test these systems, but they are really just poking at them. We don't have good, predictive science to know how to test them reliably. This is a very, very new technology, and even figuring out what we should be testing for will take some time. But we absolutely know certain things that we can be looking for right now. The most important thing to do is to build the infrastructure to generate that science and regulatory technology, and to give government the capacity to be the one deciding what is too fast. And we can put the brakes on, to give us time to develop that science before we get even more powerful systems. I support a pause in development while we figure out how to make these systems safe, and especially a pause in the development of agents.
As for how achievable this is, you have to build the consensus, but there is some bipartisan agreement to start with. Our governments have been activated by recent events in ways that, honestly, I would have thought impossible even just a few months ago. The people who are closest to this technology have the best view of what we are facing and how fast it is going. But it is being invented pretty much entirely inside private technology companies, and we, the public, do not have visibility into that. The companies themselves are pulling the alarm and saying we really do need to slow this down. They need governments to act, and they need the public to step in to support that move. We need to incentivize greater investment, by private capital, philanthropy, and government, into the challenge of figuring out the science and technology of how we regulate this. Research universities have an enormously important role to play here. If we are going to develop the oversight and maintain the control that we want, that requires the kind of legitimacy and credibility that the research university brings.
You've proposed a regulatory approach to AI that was recently adopted by California and is part of bipartisan legislation drafted by Congress this past summer. Can you explain how this would work and what it would take to implement?
I started writing about this approach before I started thinking about AI, because I was thinking about how we keep up with fast-moving technology. I introduced the concept, which I call regulatory markets, in my 2016 book Rules for a Flat World. Jack Clark and I applied it to AI safety in 2019. The question was whether governments can come up with the rules we need to follow and say, do this test, do that test, to ensure AI is safe. The answer is no, we are not going to be able to do that, and I think that is even more clear now. We'll need to drive innovation in regulatory methods to figure out how to regulate AI effectively. That's the regulatory markets idea. Governments establish what is acceptable risk, and they license private actors who show that they have developed the methods and technology, the tests and the verification, to show a system is safe. The version of this in the current proposals is called independent verification organizations, or IVOs, which are independent of the AI companies.
Under this approach, a company that wants to sell a powerful AI model that poses serious risks would need an IVO to check that it has taken the steps in building the model to ensure that it stays within the level of risk the government has set. It also creates a market that can attract the investment we badly need to figure out how to predict what these models will do and which methods of building them actually work to keep risk within acceptable levels. For this to work, you need government to focus on effectively regulating the IVOs. Are the entities doing this work qualified? Do we have ways of establishing and maintaining their independence? Is there a threat that says, if you don't do this well, you lose your license?
California has passed a law, SB 813, that begins the process of creating a system of IVOs. The FRONTIER Act, the bipartisan bill from Rep. Obernolte and Rep. Trahan, would do it nationally, licensing independent verifiers and requiring the largest developers to retain one with the access to do the work. These are first steps, not the full system. It is still going to take time. It does not have to be super long, but it is not going to happen in weeks. Maybe months.
