A greeting accompanied by a wave: researchers have connected attempted speech and movements in people with paralysis to the speech and gestures of a virtual avatar.
Article
We wave while saying hello, point while explaining a direction, or give a thumbs-up with good news. Conversation conveys much through the body as well as through spoken sentences.
If we were to create a communication device for people who have difficulty speaking or moving their limbs, would producing sentences alone be enough? Brain-interface research by Edward Chang’s team at UCSF extends that goal to conveying speech and gestures at the same time.
Place a hand gesture next to the sentence
Research introduced by the NIH on October 7 deals with a device that converts brain activity into commands for a virtual avatar. Edward Chang's research team at UCSF recorded signals from the motor cortex involved in speech and movement and linked these to speech and upper body movements. The paper was published in 『Nature Neuroscience』 on September 14th.
The study involved three people with severe speech paralysis. Two people had severe paralysis of their limbs and the other had moderate paralysis. The participants attempted conversational gestures such as waving and giving a thumbs-up, which the avatar then expressed.
Difficulty combining movements learned separately
Learning the moments when you only want to speak and the moments when you only want to make hand gestures does not mean that they will be interpreted equally well when you try both at the same time. The team compared data from separate attempts at speech and movement with data from attempts to do both together. Including the combined attempts in training improved the system’s ability to decode simultaneous actions.
In the test, a distinction was made between tasks involving imitating set movements and conversational questions and answers. No single performance figure therefore establishes that the system restores every aspect of natural conversation. This is a proof-of-concept study with a small number of participants, and this device is a wired system. The research team plans to develop it into a fully implanted wireless system in the future.
This device is not a machine that reads a person's entire thoughts. It learns signals when attempting specific words and movements and connects them to expressions. Waving with a greeting is something we rarely think about as two separate actions. In this study, restoring them together became part of the goal of restoring conversation.
https://creativecommons.org/licenses/by/4.0/
Image instructions: Representative images are actual screen excerpts from Supplementary Video 3 of the study. Brosler, Liu, Silva et al. / Nature Neuroscience · CC BY 4.0. This is an excerpt from one frame of the video. https://www.nature.com/articles/s41593-026-02446-2 https://creativecommons.org/licenses/by/4.0/
Sources and editorial notes
Nature Neuroscience · Research Papers
Editorial note: Facts were checked against the sources above; interpretations are MUADAILY’s editorial views. Image: Brosler, Liu, Silva et al. / UCSF · Nature Neuroscience · CC BY 4.0 · Frame from Supplementary Video 3.
Machine-translated from Korean. Read the original.
