Our brain-to-voice synthesis brain-computer interface paper was published in nature today! This neuroprosthesis synthesized the voice of a man with ALS instantaneously, enabling him to ‘speak’ flexibly and modulate the prosody of his BCI-voice. 1/7 Paper:

Spiking activity from 256 microelectrodes in speech motor cortex was translated into voice and played aloud continuously within 25 ms emulating a healthy speech-motor loop. His BCI-voice was ~60% intelligible – the highest to date for a voice synthesis BCI. 2/7

Decoding prosodic features simultaneously with brain-to-voice enabled the participant to modulate his intonation in closed-loop to ask a question, emphasize words, and even ‘sing’ simple melodies by varying his pitch, making his BCI-voice more expressive. 3/7

This BCI generalized to a variety of vocalizations including mimed speech, unseen words, self-initiated speech, interjections, made-up words, without training on these tasks. It allowed personalization of the BCI-voice to match the participant’s pre-ALS voice. 4/7

Mapping neural activity directly to sounds at each timepoint (without a language model or a discrete speech representation) empowered the participant to decide what to speak and how – a step towards a “digital vocal tract” for restoring fully expressive speech. 5/7

Special thanks to our study participant T15 for his invaluable contribution to this research. This study was conducted at @ucdavis @UCDneurosurgery as part of the BrainGate2 clinical trial @BrainGateTeam @MGH_CNTR. 6/7 #BCI #NeuroTech #ALS

Finally, thanks to co-senior authors @SergeyStavisky, @DrDavidBrandman and co-authors @NS_Card, @neuroleigh, @ca_rrina, Tyler Singer-Clark, Xianda Hou, and Lee M. Miller. This study wouldn't have been possible without their help! 7/7

@Nature Congratulations for a great work! It is very rewarding to see brain-machine interfaces helping people as dreamed 25 years ago! My heartfelt congratulatiosn to all of you!

@Nature Thank you for your kind words Dr. Nicolelis! The progress we’ve made has been built on the knowledge generated by the BCI research community over the past two decades.

@Nature Nice work. Wild that this is really happening.

@Nature Thank you!

@Nature Really cool work, congrats!!

@Nature amazing 👌🏽

@Nature Congratulations ! @Maitreyee_W @SergeyStavisky @DrDavidBrandman

@Nature @SergeyStavisky @DrDavidBrandman Thank you @GopalaSpeech !

@Nature amazing work!

@Nature What an amazing feat. So happy to learn this development.

@Nature Congratulations! Super cool :)

@Nature Thanks Sarah!

@Nature Congrats @Maitreyee_W, super impressive!

@Nature Thank you @matthew_willsey!

@Nature Wow!

@Nature lovely, simply lovely! huge congratulations to everyone involved

@Nature Congratulation @Maitreyee_W. Indeed great work and recognition by @Nature

@Nature Thank you!!

@Nature This is just out of the world!

@ShorterLab @Nature Incredible work!

@Nature Congrats, Dr. Wairagkar! Can this approach scale to users with no prior speech experience?

@Nature That's an open scientific question and will be very interesting to invertigate. Our current system decodes signals from the region of the brain that controls the movement of the vocal tract as the person 'tries' to speak.

@Nature So exciting, thanks Maitreyee! Keep up the great work!

@Nature Thanks!

@Nature Great!

@Nature @threadreaderapp unroll

@Nature