1/7 For the past decade, our team at Meta Reality Labs (previously CTRL-labs) has been dedicated to developing a neuromotor interface. Our goal is to address the Human Computer Interaction challenge of providing effortless, intuitive, and efficient input to computers.

2/7 We developed a wristband device that can be easily put on and removed to non-invasively sense muscle activations in the wrist and hand via surface electromyography (sEMG). The sEMG technology uses metal contacts on the skin to detect muscle activity, allowing us to transform intentional neuromotor commands into computer input.

3/7 We created generic wrist-based sEMG neural network decoding models trained on data from thousands of paid volunteers who participated in our study. These models generalize across people, eliminating the need for per-person or per-session calibration, which have traditionally been challenges for biosignal interfaces. Below, we show how offline handwriting decoding performance improves as the number of participants in the training dataset increases. For instance, when a model is trained on data from over 6000 users, it achieves an offline performance of 7% character error rate, which translates to about one error every 14 characters and is approaching error rates comparable with mobile typing.

4/7 To evaluate the performance of our decoders with naive users, we conducted closed-loop tests that included 1D continuous navigation (left), discrete gesture detection (middle), and handwriting (right).

5/7 Within minutes of first putting on the band, naive test users demonstrate closed-loop median performance of gesture decoding for: - 1D continuous navigation (wrist control): a target acquisition every two seconds - Discrete gestures: just under one gesture per second - Handwriting: 17.0 adjusted words per minute.

6/7 This is only the start! We’ve discovered that personalizing handwriting models with minimal fine-tuning for individual participants can lead to a 30% improvement in performance. As we gather more extensive training datasets and deploy models in situations where individuals can customize sEMG models to their unique writing style, sEMG decoding models will continue to advance.

7/7 As far as we know, this is the first high-bandwidth neuromotor interface that utilizes biosignals with effective out-of-the-box generalization across individuals. This achievement, led by Patrick Kaifosh and @TRReardon, is the result of the scientific and engineering efforts of hundreds of people and has been documented in a scientific journal preprint (link below).

When I tell people that working at Meta is like working at Hogwarts, CTRL Labs is what I’m talking about.

It's a bit sad how it could have been a dev kit for a community of hackers all those years ago, and then it disappeared into the black hole or Meta Reality Labs. I was very disappointed by that development.

Extremely impressive. Could it translates to prostheses?

We speculate towards clinical directions in the discussion of the linked preprint: