Loading video...

Video Failed to Load

There was a problem loading this video. This could be due to a temporary network issue or the video might be unavailable.

At #CVPR2024: Tactile-augmented Radiance Fields! We probe a scene with a touch sensor and localize each sample within a NeRF. We use diffusion to estimate the tactile signals for the points we didn't touch. w/ Yiming Dou, Antonio Loquercio, Fengyu Yang, Yi Liu

Andrew Owens

2,395 subscribers

38,628 views • 2 years ago •via X (Twitter)

Health & Wellness Science & Technology Education #CVPR2024

Anya Rossi• Live Now

Private livecam show

8 Comments

Andrew Owens2 years ago

One fun technical detail: we mount the touch sensor to an RGB-D camera using a selfie stick. Since "vision-based" touch sensors (like DIGIT, GelSight) are based on ordinary cameras, you can use multi-view geometry to estimate the relative pose between both sensors!

Andrew Owens2 years ago

Here's what the capturing procedure looks like.

Andrew Owens2 years ago

Project page: Paper:

Andrew Owens2 years ago

The idea of filling in a touch signal using a generative model is similar to recent work by @ShaohongZhong et al.: which uses robotic proprioception and GANs for object-scale reconstructions.

Yiming Dou2 years ago

@antoniloq Thanks Andrew! This project wouldn’t be possible without you advising on every single detail!😀

Igor Gilitschenski2 years ago

@_YimingDou @antoniloq That is amazing work! Congrats everyone!

Andrew Owens2 years ago

@_YimingDou @antoniloq Thanks so much, Igor!

Mustafa2 years ago

@_YimingDou @antoniloq Let's chat. Unfortunately can't dm you

Related Videos

We visited Paxini in Shenzhen, a company focused on the sense of touch for robots. We got hands-on with their dual-modal Dexter Hands (cameras + tactile sensors) and their Tora Double One humanoid. Humanoid robots will need this touch layer to get truly dexterous. Lena RoboHub🤖

We visited Paxini in Shenzhen, a company focused on the sense of touch for robots. We got hands-on with their dual-modal Dexter Hands (cameras + tactile sensors) and their Tora Double One humanoid. Humanoid robots will need this touch layer to get truly dexterous. Lena RoboHub🤖

The Humanoid Hub

27,349 views • 3 months ago

Today at Meta FAIR we’re announcing three new cutting-edge developments in robotics and touch perception — and releasing a collection of artifacts to empower the community to build on this work. Details on all of this new work ➡️ 1️⃣ Meta Sparsh is the first general-purpose encoder for vision-based tactile sensing that works across many tactile sensors and many tasks. Trained on 460K+ tactile images using self-supervised learning. 2️⃣ Meta Digit 360 is a breakthrough artificial fingertip-based tactile sensor, equipped with 18+ sensing features to deliver detailed touch data with human-level precision and touch-sensing capabilities. 3️⃣ Meta Digit Plexus is a standardized platform for robotic sensor connections and interactions. It provides a hardware-software solution to integrate tactile sensors on a single robot hand and enables seamless data collection, control and analysis over a single cable. The potential impact of expanding capabilities and components like these for the open source community ranges from medical research to supply chain, manufacturing and much more. We’re excited to continue this work with the broader community.

Today at Meta FAIR we’re announcing three new cutting-edge developments in robotics and touch perception — and releasing a collection of artifacts to empower the community to build on this work. Details on all of this new work ➡️ 1️⃣ Meta Sparsh is the first general-purpose encoder for vision-based tactile sensing that works across many tactile sensors and many tasks. Trained on 460K+ tactile images using self-supervised learning. 2️⃣ Meta Digit 360 is a breakthrough artificial fingertip-based tactile sensor, equipped with 18+ sensing features to deliver detailed touch data with human-level precision and touch-sensing capabilities. 3️⃣ Meta Digit Plexus is a standardized platform for robotic sensor connections and interactions. It provides a hardware-software solution to integrate tactile sensors on a single robot hand and enables seamless data collection, control and analysis over a single cable. The potential impact of expanding capabilities and components like these for the open source community ranges from medical research to supply chain, manufacturing and much more. We’re excited to continue this work with the broader community.

AI at Meta

453,860 views • 2 years ago

We just released TAVI -- a robotics framework that combines touch and vision to solve challenging dexterous tasks in under 1 hour. The key? Use human demonstrations to initialize a policy, followed by tactile-based online learning with vision-based rewards. Details in🧵(1/7)

We just released TAVI -- a robotics framework that combines touch and vision to solve challenging dexterous tasks in under 1 hour. The key? Use human demonstrations to initialize a policy, followed by tactile-based online learning with vision-based rewards. Details in🧵(1/7)

Lerrel Pinto

138,931 views • 3 years ago

Can we bring human-like Touch to robots🤖? Introducing our CoRL work on 3D-ViTac. Humans rely on both vision 👁️ and touch 🫳 for complex tasks. With combined visual-tactile sensing, robots can now tackle challenging tasks, like precise in-hand reorientation, fragile objects grasping. Website: #Robotics #CoRL2024 #Touch #tactile #AI #ML

Can we bring human-like Touch to robots🤖? Introducing our CoRL work on 3D-ViTac. Humans rely on both vision 👁️ and touch 🫳 for complex tasks. With combined visual-tactile sensing, robots can now tackle challenging tasks, like precise in-hand reorientation, fragile objects grasping. Website: #Robotics #CoRL2024 #Touch #tactile #AI #ML

Binghao Huang

49,714 views • 2 years ago

What can a robot do without vision? We gave our robot a sense of touch and showed it can recognize and retrieve objects with its cameras blindfolded. Introducing Rhoda’s new tactile sensor—it feels pressure, texture, vibration, slip, and temperature, just like your fingertips.

What can a robot do without vision? We gave our robot a sense of touch and showed it can recognize and retrieve objects with its cameras blindfolded. Introducing Rhoda’s new tactile sensor—it feels pressure, texture, vibration, slip, and temperature, just like your fingertips.

Rhoda AI

103,418 views • 13 days ago

(1/2) Mesh2NeRF: Direct Mesh Supervision for Neural Radiance Field Representation and Generation #ECCV2024! We show a theoretical derivation to create radiance fields directly from meshes. Thus, we can obtain GT training data for generative NeRF methods.

(1/2) Mesh2NeRF: Direct Mesh Supervision for Neural Radiance Field Representation and Generation #ECCV2024! We show a theoretical derivation to create radiance fields directly from meshes. Thus, we can obtain GT training data for generative NeRF methods.

Matthias Niessner

16,835 views • 2 years ago

We have plenty of human demonstration data for movement. We don’t have enough for touch. Without tactile feedback, foundation models can’t feel if an object is slipping, soft, or fragile during complex manipulation. Matrix Inno is bringing three data-collection gloves to IROS 2026: the RigidCore Tactile+EMF, the Fabric Tactile + EMF, and the Fabric Tactile + UltraLite Motion Glove. The hardware tracks pressure and hand pose simultaneously during human demos, mapping exactly where contact happens and how it shifts in real time. Wearables like this are how we finally get real, continuous contact data at scale. Source: Matrix Inno MX MATRIX

We have plenty of human demonstration data for movement. We don’t have enough for touch. Without tactile feedback, foundation models can’t feel if an object is slipping, soft, or fragile during complex manipulation. Matrix Inno is bringing three data-collection gloves to IROS 2026: the RigidCore Tactile+EMF, the Fabric Tactile + EMF, and the Fabric Tactile + UltraLite Motion Glove. The hardware tracks pressure and hand pose simultaneously during human demos, mapping exactly where contact happens and how it shifts in real time. Wearables like this are how we finally get real, continuous contact data at scale. Source: Matrix Inno MX MATRIX

RoboHub🤖

12,372 views • 1 month ago

More beautiful robot hands. This one from Sharpa The hand uses what they call a Dynamic Tactile Array, meaning vision and touch work together and their SaTA (Spatially-anchored Tactile Awareness) algorithm. Each fingertip includes a mini camera and 1,000+ tactile pixels, hitting 0.005N pressure sensitivity. With 22 DOF and 6D force sensing, it can do everything from eggs to tools.

More beautiful robot hands. This one from Sharpa The hand uses what they call a Dynamic Tactile Array, meaning vision and touch work together and their SaTA (Spatially-anchored Tactile Awareness) algorithm. Each fingertip includes a mini camera and 1,000+ tactile pixels, hitting 0.005N pressure sensitivity. With 22 DOF and 6D force sensing, it can do everything from eggs to tools.

Rohan Paul

41,775 views • 10 months ago

China's HexaCercle glove captures motion and touch with 408 tactile sensors and multi-axis force sensing. Its joint-angle data maps to AGIBOT’s 10-active/16-total-DoF OmniHand 2025 w/ Tactile.

China's HexaCercle glove captures motion and touch with 408 tactile sensors and multi-axis force sensing. Its joint-angle data maps to AGIBOT’s 10-active/16-total-DoF OmniHand 2025 w/ Tactile.

Eren Chen

28,962 views • 2 months ago

Introducing tactile skin sim-to-real for dexterous in-hand translation! We propose a simulation model for ReSkin, a magnetic tactile sensing skin. It can simulate ternary shear and binary normal forces. More:

Introducing tactile skin sim-to-real for dexterous in-hand translation! We propose a simulation model for ReSkin, a magnetic tactile sensing skin. It can simulate ternary shear and binary normal forces. More:

Haozhi Qi

33,562 views • 2 years ago

Progressively Optimized Local Radiance Fields for Robust View Synthesis paper page: present an algorithm for reconstructing the radiance field of a large-scale scene from a single casually captured video. The task poses two core challenges. First, most existing radiance field reconstruction approaches rely on accurate pre-estimated camera poses from Structure-from-Motion algorithms, which frequently fail on in-the-wild videos. Second, using a single, global radiance field with finite representational capacity does not scale to longer trajectories in an unbounded scene. For handling unknown poses, we jointly estimate the camera poses with radiance field in a progressive manner. We show that progressive optimization significantly improves the robustness of the reconstruction. For handling large unbounded scenes, we dynamically allocate new local radiance fields trained with frames within a temporal window. This further improves robustness (e.g., performs well even under moderate pose drifts) and allows us to scale to large scenes. Our extensive evaluation on the Tanks and Temples dataset and our collected outdoor dataset, Static Hikes, show that our approach compares favorably with the state-of-the-art.

Progressively Optimized Local Radiance Fields for Robust View Synthesis paper page: present an algorithm for reconstructing the radiance field of a large-scale scene from a single casually captured video. The task poses two core challenges. First, most existing radiance field reconstruction approaches rely on accurate pre-estimated camera poses from Structure-from-Motion algorithms, which frequently fail on in-the-wild videos. Second, using a single, global radiance field with finite representational capacity does not scale to longer trajectories in an unbounded scene. For handling unknown poses, we jointly estimate the camera poses with radiance field in a progressive manner. We show that progressive optimization significantly improves the robustness of the reconstruction. For handling large unbounded scenes, we dynamically allocate new local radiance fields trained with frames within a temporal window. This further improves robustness (e.g., performs well even under moderate pose drifts) and allows us to scale to large scenes. Our extensive evaluation on the Tanks and Temples dataset and our collected outdoor dataset, Static Hikes, show that our approach compares favorably with the state-of-the-art.

AK

140,624 views • 3 years ago

Touch alone isn’t enough. 🖐️ For robotics, tactile intelligence truly levels up when touch gains spatial meaning. At #ICRA2026, we dove into the core concept behind SaTA: Spatially-anchored Tactile Awareness for robust, dexterous manipulation. Read it here: The challenge is fundamental: a robot shouldn’t just register a touch, it needs to understand exactly where that contact occurs relative to its fingers, joints, and overall hand structure. This is the missing link that turns raw data into precise, real-time adjustments during manipulation. Why does this matter? Because the most complex part of any manipulation task happens when vision is at its least reliable, that final millimeter before insertion, sliding, gripping, or fine-tuning. At Sharpa, this is exactly why we’re building tactile hands and tactile AI in tandem. 🚀 #Sharpa #Robotics #EmbodiedAI #TactileIntelligence #ICRA2026 #DexterousManipulation 📷

Touch alone isn’t enough. 🖐️ For robotics, tactile intelligence truly levels up when touch gains spatial meaning. At #ICRA2026, we dove into the core concept behind SaTA: Spatially-anchored Tactile Awareness for robust, dexterous manipulation. Read it here: The challenge is fundamental: a robot shouldn’t just register a touch, it needs to understand exactly where that contact occurs relative to its fingers, joints, and overall hand structure. This is the missing link that turns raw data into precise, real-time adjustments during manipulation. Why does this matter? Because the most complex part of any manipulation task happens when vision is at its least reliable, that final millimeter before insertion, sliding, gripping, or fine-tuning. At Sharpa, this is exactly why we’re building tactile hands and tactile AI in tandem. 🚀 #Sharpa #Robotics #EmbodiedAI #TactileIntelligence #ICRA2026 #DexterousManipulation 📷

Sharpa

15,402 views • 4 months ago

What if you could give a robot a sense of touch for around $5? That's what makes eFlesh so interesting. It is a low-cost, 3D-printed tactile sensor that uses magnets and a magnetometer to sense contact, force and slipping. Put it on a robotic gripper and it could tell when an object is starting to slip, then adjust its grip instead of relying only on cameras. Cheap tactile sensing could make it much easier to add touch to robotic hands, grippers and other parts. Would you put a $5 tactile sensor on your robot? What would you make it do? Developed by Venkatesh, Lerrel Pinto and team

What if you could give a robot a sense of touch for around $5? That's what makes eFlesh so interesting. It is a low-cost, 3D-printed tactile sensor that uses magnets and a magnetometer to sense contact, force and slipping. Put it on a robotic gripper and it could tell when an object is starting to slip, then adjust its grip instead of relying only on cameras. Cheap tactile sensing could make it much easier to add touch to robotic hands, grippers and other parts. Would you put a $5 tactile sensor on your robot? What would you make it do? Developed by Venkatesh, Lerrel Pinto and team

Praveen Kumar Verma

21,321 views • 23 days ago

Robots need to feel the world to operate in it. Most manipulation policies today are tactile-blind. They either cannot interpret high-frequency tactile signals or treat them as a static channel. And the field lacks enough touch-rich datasets to train tactile-reactive policies at scale. T-Rex was built to answer both. Dantong Niu and team, advised by Jim Fan, Fei-Fei Li, Jitendra MALIK, Pieter Abbeel trevordarrell have tested whether a robot policy can react to high-frequency tactile signals the way human hands do, without giving up the generalization power of modern VLAs. The result: 65% average success rate across 12 real-world tasks. +30 absolute points over the strongest baseline. In one year, tactile VLAs have gone from promising to outperforming non-tactile baselines like pi0.5 on dexterous tasks. #Robotics #SharpaWave #Sharpa #EmbodiedAI #DexterousManipulation #TactileSensing #RobotLearning Project link:

Robots need to feel the world to operate in it. Most manipulation policies today are tactile-blind. They either cannot interpret high-frequency tactile signals or treat them as a static channel. And the field lacks enough touch-rich datasets to train tactile-reactive policies at scale. T-Rex was built to answer both. Dantong Niu and team, advised by Jim Fan, Fei-Fei Li, Jitendra MALIK, Pieter Abbeel trevordarrell have tested whether a robot policy can react to high-frequency tactile signals the way human hands do, without giving up the generalization power of modern VLAs. The result: 65% average success rate across 12 real-world tasks. +30 absolute points over the strongest baseline. In one year, tactile VLAs have gone from promising to outperforming non-tactile baselines like pi0.5 on dexterous tasks. #Robotics #SharpaWave #Sharpa #EmbodiedAI #DexterousManipulation #TactileSensing #RobotLearning Project link:

Sharpa

12,957 views • 4 months ago

Tactile Diffusion generates synthetic tactile images from sim data, capturing the complex illumination of the gel deformation. This research from UW & Meta AI is the first method using diffusion to close the sim2real gap for vision-based tactile sensing. Read the paper ⬇️

Tactile Diffusion generates synthetic tactile images from sim data, capturing the complex illumination of the gel deformation. This research from UW & Meta AI is the first method using diffusion to close the sim2real gap for vision-based tactile sensing. Read the paper ⬇️

AI at Meta

100,168 views • 3 years ago

Making basketball more accessible. OneCourt’s tactile broadcast devices use vibration and audio to help blind and low-vision fans experience Bucks games through touch. Together with Ticketmaster and onecourt_io, we are proud to bring these devices to Fiserv Forum. Reserve yours now:

Making basketball more accessible. OneCourt’s tactile broadcast devices use vibration and audio to help blind and low-vision fans experience Bucks games through touch. Together with Ticketmaster and onecourt_io, we are proud to bring these devices to Fiserv Forum. Reserve yours now:

Milwaukee Bucks

13,605 views • 8 months ago

Presenting research from Berkeley AI Research Humanoid Intelligence Center. DexTacWAM: A Visuo-Tactile World-Action Model for Dexterous Manipulation.🖐️🤖 Dexterous manipulation requires touch, yet multi-finger tactile data remain scarce and expensive to collect at scale. Through continual vision-to-touch learning, DexTacWAM adapts a pretrained video world model into a visuo-tactile world model for predictive multi-finger contact modeling and action generation. Across six contact-rich dexterous manipulation tasks, DexTacWAM achieves the highest score on every task, averaging 70.6 vs. 38.0 for the strongest baseline. 🔬 Why not simply inject tactile features into the action policy? With the same multi-finger tactile encoder, policy architecture, and training procedure, replacing the tactile world-model latent with direct tactile features drops the 4-task mean from 74.7 to 26.6. Takeaway: the benefit does not come simply from providing tactile observations to the policy, but from making multi-finger contact evolution part of the predicted world state. All code, model weights, and datasets are now open-sourced! 🌐 Project: 📄 Paper: 💻 Code: 🤗 Weights&Data: Author List: Haoran Yuan, Zekai Wang, Boning Shao Haoran Lu, trevordarrell, Ismini Lourentzou, Wei ZHAN

Presenting research from Berkeley AI Research Humanoid Intelligence Center. DexTacWAM: A Visuo-Tactile World-Action Model for Dexterous Manipulation.🖐️🤖 Dexterous manipulation requires touch, yet multi-finger tactile data remain scarce and expensive to collect at scale. Through continual vision-to-touch learning, DexTacWAM adapts a pretrained video world model into a visuo-tactile world model for predictive multi-finger contact modeling and action generation. Across six contact-rich dexterous manipulation tasks, DexTacWAM achieves the highest score on every task, averaging 70.6 vs. 38.0 for the strongest baseline. 🔬 Why not simply inject tactile features into the action policy? With the same multi-finger tactile encoder, policy architecture, and training procedure, replacing the tactile world-model latent with direct tactile features drops the 4-task mean from 74.7 to 26.6. Takeaway: the benefit does not come simply from providing tactile observations to the policy, but from making multi-finger contact evolution part of the predicted world state. All code, model weights, and datasets are now open-sourced! 🌐 Project: 📄 Paper: 💻 Code: 🤗 Weights&Data: Author List: Haoran Yuan, Zekai Wang, Boning Shao Haoran Lu, trevordarrell, Ismini Lourentzou, Wei ZHAN

Berkeley AI Research

112,249 views • 27 days ago

A touch-aware humanoid manipulation policy that cleans the lab for you🧹🧪 Introducing Humanoid Touch Dream: a real-world system for dexterous, contact-rich humanoid loco-manipulation. Our key idea is simple: the policy predicts future hand forces and tactile latents alongside actions, within a single-stage training framework. 1/7

A touch-aware humanoid manipulation policy that cleans the lab for you🧹🧪 Introducing Humanoid Touch Dream: a real-world system for dexterous, contact-rich humanoid loco-manipulation. Our key idea is simple: the policy predicts future hand forces and tactile latents alongside actions, within a single-stage training framework. 1/7

Yaru Niu

91,677 views • 6 months ago

In house tactile sensor prototype. Detects taps, motion, and multiple contact with force magnitudes. Built to be cheap, robust, and replaceable. Not in our demos yet, still WIP. We pair simple tactile sensing with torque transparent joints to recover rich contact without expensive sensors.

In house tactile sensor prototype. Detects taps, motion, and multiple contact with force magnitudes. Built to be cheap, robust, and replaceable. Not in our demos yet, still WIP. We pair simple tactile sensing with torque transparent joints to recover rich contact without expensive sensors.

Kyber Labs

39,735 views • 6 months ago