The emotion of being heard again: A man with ALS recovers his voice

The smile of disbelief soon becomes a cry of emotion as 45-year-old Californian Casey Harrel hears his voice for the first time in four years when amyotrophic lateral sclerosis (ALS) began to affect the nerves that control the muscles in his throat. His wife, who is by his side with their daughter, is also overcome with emotion. The test has to stop for a few minutes, because some members of the team at the University of California, Davis, are also moved. The group has developed a brain-computer interface (BCI) that not only interprets what Harrel wants to say in real time but also picks up on his intonation and speaking style.

“It sounds a lot like me,” is heard through a speaker connected to the BCI that deciphers the patient’s neural activity when they try to say something, as shown in videos recording the process. In order for the device to be connected to the brain, a total of 256 1.5-millimeter electrodes were implanted in Harrel’s cerebral cortex, specifically in the region that controls speech, the jaw and other muscles belonging to the vocal tract. The paralysis had stopped the neural signals in that region from reaching their destination.

“In the sessions, we ask the participant to try to say the words that appear on the screen, so we know what they are trying to communicate. With this data, we train the artificial intelligence model to decode the neural signal and transform it into the sounds it wants to emit,” explains Maitreyee Wairagkar, a member of the research team. Harrel had difficulty communicating before: he had to repeat what he wanted to say several times to his care team, who interpreted it from the context. He also used a gyroscopic mouse that allowed him to move his head slightly to control a computer cursor and write one letter at a time.

Wairagkar and the group of scientists visit Harrel twice a week to continue the experiments. The researcher says that dialogue with the patient is much more fluid now. It takes about 25 milliseconds for him to hear his own voice, about the same as the average person, according to research published in June by Nature. Although there have been other BCIs, these converted brain signals into text: words that appeared on the screen with a certain delay. This method was more accurate in reproducing what the participant wants to say — the current voice system allows 60% to be understood — but at the cost of speed and fluidity.

Phonemes instead of words

The University of California presents its BCI not only as the first one capable of recreating the voice instantaneously, but as the only one that allows a person to speak “in their own words, tone and intonation.” The achievement has to do with the algorithm designed for the interface: instead of interpreting entire words, it decodes the phonemes that compose them. In other words, AI models were trained to detect the exact moment when each syllable is emitted; they focus more on vocalizations than on concepts.

“It brings people who haven’t heard me in a long time to tears,” Harrel says in one of the videos. The monitor displays the word or sentence that the system has decoded from brain signals. The patient is then presented with several options: they can click on a speaker symbol with a mouse to have their voice played. Or rate the computer’s interpretation of their neural activity: 100% correct, one word incorrect, nearly correct, or incorrect. If the latter is selected, the device presents the patient with up to six alternative words that could be what they were really trying to say. To provide more organic communication, the team cloned Harrel’s voice with artificial intelligence from recordings made before the sclerosis advanced.

The next step, Wairagkar points out, is to improve the intelligibility of the system: “Reaching 60% comprehension has been a big leap, because before only 4% was understood. But it can still be improved if we use more advanced neural recording devices, with more electrodes – for example, those that are being developed by different neurotechnology companies in the United States, which have more than 1,000 electrodes instead of the 256 we used in our study.” The aim is to extend the study to other people who have lost their speech.

“Each case is different. So far, we have only tested the BCI with one participant who has ALS. The brain changes according to the disease. For example, in a cerebrovascular accident (CVA), people can develop aphasia, which prevents them from producing language, and that is a different problem,” Wairagkar explains. Still, she and her team believe the technology could be used with people suffering from similar disabilities: “We just need to show that it works on more people,” she adds.

The University of California study is part of the BrainGate2 clinical trial, which evaluates the safety of using electrode arrays — known as Utah arrays — in humans. The purpose is for people with spinal cord injuries, strokes or ALS to undergo the surgical implantation of microelectrodes in the motor cortex, either to record the electrical activity of neurons or to stimulate them.

Harrel is happy to have signed up for the program as not being able to communicate produces a deep sense of loneliness and isolation. That’s why he tells the researchers around him, “I hope we’ve reached the point where everyone like me has this same opportunity. Let’s make that happen, alright?”