Today at Meta FAIR we’re announcing three new cutting-edge developments in robotics and touch perception — and releasing a collection of artifacts to empower the community to build on this work. Details on all of this new work ➡️ 1️⃣ Meta Sparsh is the first general-purpose encoder for vision-based... tactile sensing that works across many tactile sensors and many tasks. Trained on 460K+ tactile images using self-supervised learning. 2️⃣ Meta Digit 360 is a breakthrough artificial fingertip-based tactile sensor, equipped with 18+ sensing features to deliver detailed touch data with human-level precision and touch-sensing capabilities. 3️⃣ Meta Digit Plexus is a standardized platform for robotic sensor connections and interactions. It provides a hardware-software solution to integrate tactile sensors on a single robot hand and enables seamless data collection, control and analysis over a single cable. The potential impact of expanding capabilities and components like these for the open source community ranges from medical research to supply chain, manufacturing and much more. We’re excited to continue this work with the broader community.show more

To make these advancements more accessible for different applications, we’re partnering with @GelSight and Wonik Robotics to develop and commercialize these touch-sensing innovations. We’re excited about how this will enable the community to contribute and drive progress in this space.

Additionally, looking towards the future, we’re releasing PARTNR: a benchmark for Planning And Reasoning Tasks in humaN-Robot collaboration. Built on Habitat 3.0, it’s the largest benchmark of its kind to study and evaluate human-robot collaboration in household activities By providing a standardized benchmark and dataset we hope to enable new research on robots that can not only operate in isolation, but in collaboration with people. Details and code ➡️

insane, love the name as well: sparsh (in hindi) literally translates to “touch” we need more hindi names :)

Robotics research that is open source too? Holy shit I love you guys

::pokes you::

how long until this

Meta Sparsh: The paper introduces a family of general-purpose touch representations called "Sparsh" that are trained using self-supervised learning (SSL) techniques. The authors aim to develop touch representations that can work well across various vision-based tactile sensors and tasks, without the need for extensive labeled data. The authors find that the "Sparsh" representations, especially those trained using DINO and IJEPA, outperform task and sensor-specific end-to-end models by 95.1% on average across the "TacBench" tasks, when using limited labeled data (33-50%). "Sparsh" representations show strong performance in tasks like force estimation, slip detection, pose estimation, and grasp stability, even with as little as 10-33% of the labeled data. full paper:

When are the metabots coming?

Is it just me or y'all realize this is some groundbreaking stuff?

this is so cool -- excited to learn more about it