How AI Gave Paralysed Woman Her Voice Back
A woman who had suffered a brainstem stroke and was severely paralyzed was able to communicate via a digital avatar thanks to research from UC San Francisco and UC Berkeley that resulted in the development of a brain-computer interface (BCI).
Speech and facial expressions have never before been artificially created from brain signals. The device can also translate these impulses into text at a rate of around 80 words per minute, which is a significant advance above currently available commercial technology.
In the near future, Edward Chang, MD, chair of neurological surgery at UCSF, hopes that this most recent research breakthrough—known as a brain computer interface, or BCI—will lead to an FDA-approved system that enables speech from brain signals. It was published on August 23, 2023, in Nature.
“Our goal is to restore a full, embodied way of communicating, which is really the most natural way for us to talk with others,” said Chang, who is a member of the UCSF Weill Institute for Neuroscience and the Jeanne Robertson Distinguished Professor in Psychiatry. “These advancements bring us much closer to making this a real solution for patients.”
Chang’s team has developed a system to decode brain signals into speech, including the movements that animate a person’s face during conversation. They implanted 253 electrodes onto a woman’s brain, intercepting brain signals that would have gone to muscles in her tongue, jaw, larynx, and face.
The electrodes were connected to computers, and the participant worked with the team to train the system’s artificial intelligence algorithms to recognize her unique brain signals for speech. Instead of training the AI to recognize whole words, they created a system that decodes words from phonemes, which form spoken words in the same way that letters form written words.
This approach enhanced the system’s accuracy and made it three times faster. The team also developed an algorithm for synthesizing speech, which they personalized to sound like her voice before the injury, using a recording of her speaking at her wedding.
The team animated the avatar with the help of software that simulates and animates muscle movements of the face, developed by Speech Graphics, a company that makes AI-driven facial animation. The researchers created customized machine-learning processes that allowed the company’s software to mesh with signals being sent from the woman’s brain as she was trying to speak and convert them into the movements on the avatar’s face, making the jaw open and close, the lips protrude and purse and the tongue go up and down, as well as the facial movements for happiness, sadness and surprise.
“We’re making up for the connections between the brain and vocal tract that have been severed by the stroke,” said Kaylo Littlejohn, a graduate student working with Chang and Gopala Anumanchipalli, PhD, a professor of electrical engineering and computer sciences at UC Berkeley. “When the subject first used this system to speak and move the avatar’s face in tandem, I knew that this was going to be something that would have a real impact.”
An important next step for the team is to create a wireless version that would not require the user to be physically connected to the BCI.
