At age 30, Ann Johnson had a lot going on. She taught math and physical education at a high school in Saskatchewan, Canada, where she also coached volleyball and basketball. She'd just had a baby a year earlier with her new husband, and had given a joyful 15-minute-long speech at their wedding.
Yet everything changed one sunny day in 2005, when Johnson suffered a brainstem stroke while playing volleyball with friends. The stroke caused extreme paralysis, and she lost the ability to speak and move any muscle in her body.
She had what's commonly known as locked-in syndrome, a rare condition when someone has near-complete paralysis and no ability to communicate naturally. She would try to speak, but her mouth wouldn't move and no sound would come out. It's most often caused by a stroke or the neurological disorder ALS.
Eighteen years went by before she heard her voice again.
That moment came during a clinical trial being conducted by researchers at UC Berkeley and UC San Francisco trying to restore people's ability to communicate using a brain-computer interface. The technology, the researchers say, has enormous potential to make the workforce and the world more accessible to people like Johnson.
Modeling the speech process
In 2015, Gopala Anumanchipalli began working as a postdoctoral researcher with Edward Chang, a neurosurgeon at UCSF, to understand how speech happens in the brain. They wanted to know what enables us to go from thinking something to actually saying it out loud.
"We were able to get a good sense of the part of the brain that is actually responsible for speech production," said Anumanchipalli, now an assistant professor of electrical engineering and computer sciences at UC Berkeley.
From there, they figured out how to computationally model the process so that they could synthesize from brain activity what someone is trying to say.
Essentially, they pinned down how to go to the source of knowledge - the brain - and then bypass what's broken - the connection to the body - and restore what's lost. In this case, they're using a neuroprosthesis that's reading from the part of the brain that processes speech.
They started the clinical trial in 2020, and Johnson joined as the third participant in 2022.
Although the population of people who lose their ability to speak in this way is relatively small, the researchers say, they are among the most vulnerable in terms of quality of life.
Since her stroke, Johnson has regained some muscle control. She now has full neck movement, and she can laugh and cry and smile. She communicates mostly using an eye-tracking system that allows her to select letters to spell words out on a computer screen. It's a slow process; she can only write about 14 words per minute, compared to conversational speech, which is closer to 160 words per minute.
So when she finally heard her thoughts out loud for the first time in nearly two decades, it was deeply emotional for her.
'We didn't want to read her mind'
"What do you think of my artificial voice?" Johnson asked, sitting next to her husband during the trial. "Tell me about yourself. I am doing well today."
Ph.D. student Kaylo Littlejohn, a co-lead on the study with Anumachipalli and Chang, remembers the moment well. As a researcher in the Berkeley Speech Group, part of the Berkeley AI Research Lab, he led the study's AI modeling efforts, training decoders so that the model accurately and effectively translated Johnson's brain activity.
To give Johnson an embodied experience, researchers had her choose from a selection of avatars, and they used a recording of her wedding speech to recreate her voice. An implant plugged into a computer nearby rested on top of the region of her brain that processes speech, acting as a kind of thought decoder. Then they showed her sentences and asked her to try to say them.
"She can't, because she has paralysis, but those signals are still being invoked from her brain, and the neural recording device is sensing those signals," said Littlejohn. The neural decoding device then sends them to the computer where the AI model resides, where they're translated. "Just like how Siri translates your voice to text, this AI model translates the brain activity into the text or the audio or the facial animation," he said.
While the model can reliably sense the intention to speak and then translate what's trying to be said, it can't read a person's errant thoughts. It only works when someone is making a concerted effort to say something.
"We didn't want to read her mind," said Anumanchipalli. "We really wanted to give her the agency to do this. In some sessions where she's doing nothing, we have the decoder running, and it does nothing because she's not trying to say anything. Only when she's attempting to say something do we hear a sound or action command."
But how realistic is it, really? Does it sound and look just like Johnson? Or is it more rudimentary and robotic? The answer, at least at this point, is somewhere in between.
Plug-and-play neuroprostheses and digital clones
When you watch a video of Johnson speaking with the brain-computer interface from when she first joined the clinical trial, you can hear her voice piecing together words in sing-songy tones, but it's not seamless. There's also an eight-second delay between the prompt and when the avatar speaks.
But this past March, the team published new research in Nature Neuroscience that dramatically decreased this delay. In 2023, the decoder used sequence-to-sequence architecture, which required that a user attempt an entire sentence before the model could convert the sentence to sound or movement. But now the decoder uses streaming architecture, which allows the models to actively listen in and translate brain activity to sound in real time, with only about a one-second delay.
We need to be able to have neuroprostheses be plug-and-play, so that it becomes a standard of care … That's where we need to be.
Gopala Anumanchipalli
In the 2023 study, the avatar moves its mouth when Johnson is talking, and makes little movements when she's asked to make a face, like a smile or a frown. Although the avatar wasn't used in the March study, researchers believe the streaming architecture will work with the avatar, too.
The avatar looks kind of like Johnson, but it's not a strong resemblance. In the near future, though, Anumanchipalli said it's possible there could be 3D photorealistic avatars.
"We can imagine that we could create a digital clone that is very much plugged in … with all the preferences, like how Zoom lets us have all these effects," he said. "All of this is possible."
Anumanchipalli said it could happen in just a few years, but research needs to happen in several areas. "It's not something that we have off-the-shelf models that we can use now," he said. "So development must happen in the science, in the technology, in the clinical translation, as well - all of them together to make this happen."
'Disabilities don't need to stop us or slow us down'
In February 2024, Johnson had her implant removed for a reason unrelated to the trial. But she continues to communicate with the research team. She sends eloquent emails using her current technology about what she felt during the trial and what she'd prefer to see in future iterations.
She enjoyed hearing her own voice, she told them, and the streaming synthesis approach made her feel in control. She also wants the implants to be wireless, instead of plugged into a computer - something the research team is working on.
Noah Berger, 2023
"Thinking further in the future, how do you imagine it working?" I asked Anumanchipalli. "Do you imagine a person, in real time, communicating exactly what they want with people around them?"
"It's hard to predict," he said, laughing. "But what I'm seeing are innovations that enable us to let people have the best quality of their lives. If that means they have a digital version of themselves communicating for them, that's what they need to be able to do.
"We need to be able to have neuroprostheses be plug-and-play, so that it becomes a standard of care and not a research experiment," he continued. "That's where we need to be."
Johnson hopes to one day become a counselor in a physical rehabilitation facility, ideally using a neuroprosthesis to talk with her clients. "I want patients there to see me and to know their lives are not over now," she wrote in response to a question from a UCSF reporter. "I want to show them that disabilities don't need to stop us or slow us down."
