Video wird geladen...

Video konnte nicht geladen werden

Zur Startseite

Stereo depth models usually force a tradeoff. 📌 Paper + Code Either strong zero-shot generalization or real-time speed. Fast-FoundationStereo closes that gap. The model accelerates FoundationStereo by more than 10× while keeping comparable depth quality, making real-time stereo matching practical for robotics and XR. On an RTX 3090, it...

IlirAliu_'s profile picture

Ilir Aliu

56,421 subscribers

11,914 Aufrufe • vor 1 Monat •via X (Twitter)

LIVE

Anya Rossi• Live Now

Private livecam show

1 Kommentare

Profilbild von Gilbert
Gilbertvor 1 Monat

the pseudo-labeling pipeline is especially interesting because high-quality metric depth data has always been a major bottleneck.

Ähnliche Videos

A new milestone for real-time accurate 3D spatial computing! Introducing ⚡️Fast-FoundationStereo⚡️, a real-time zero-shot stereo depth estimation model that accelerates the original FoundationStereo by >10x with comparable quality. Details in threads 🧵 (1/N)
0:30

A new milestone for real-time accurate 3D spatial computing! Introducing ⚡️Fast-FoundationStereo⚡️, a real-time zero-shot stereo depth estimation model that accelerates the original FoundationStereo by >10x with comparable quality. Details in threads 🧵 (1/N)

Bowen Wen

77,271 Aufrufe • vor 10 Monaten

Excited to introduce StereoPolicy, led by Evans Han. 📷📷🤖StereoPolicy is an effective way to add geometric cues to modern robot policy models while keeping the strengths of pretrained 2D encoders. ⁉️Why stereo for robot manipulation? Monocular RGB often lacks the depth cues needed for precise manipulation, while RGB-D and point clouds can be noisy or brittle, especially on reflective and transparent objects in real-world deployment. Instead of explicitly reconstructing disparity, depth, or point clouds, StereoPolicy directly fuses synchronized left/right RGB views to learn implicit stereo cues, avoiding extra reconstruction latency that can make real-time manipulation difficult. Project Page:
0:50

Excited to introduce StereoPolicy, led by Evans Han. 📷📷🤖StereoPolicy is an effective way to add geometric cues to modern robot policy models while keeping the strengths of pretrained 2D encoders. ⁉️Why stereo for robot manipulation? Monocular RGB often lacks the depth cues needed for precise manipulation, while RGB-D and point clouds can be noisy or brittle, especially on reflective and transparent objects in real-world deployment. Instead of explicitly reconstructing disparity, depth, or point clouds, StereoPolicy directly fuses synchronized left/right RGB views to learn implicit stereo cues, avoiding extra reconstruction latency that can make real-time manipulation difficult. Project Page:

Ruohan Zhang

851,597 Aufrufe • vor 4 Monaten

Meet MapAnything – a transformer that directly regresses factored metric 3D scene geometry (from images, calibration, poses, or depth) in an end-to-end way. No pipelines, no extra stages. Just 3D geometry & cameras, straight from any type of input, delivering new state-of-the-art results 🚀 One universal model enables SoTA for: 🔥 Mono Depth Estimation 🔥 Multi-View SfM 🔥 Multi-View Stereo 🔥 Depth Completion 🔥 Registration … and many more possibilities! – plus everything is metric 🎯 We release code for data processing, training, benchmarking & ablations – everything Apache 2.0! Details & Links 👇
1:30

Meet MapAnything – a transformer that directly regresses factored metric 3D scene geometry (from images, calibration, poses, or depth) in an end-to-end way. No pipelines, no extra stages. Just 3D geometry & cameras, straight from any type of input, delivering new state-of-the-art results 🚀 One universal model enables SoTA for: 🔥 Mono Depth Estimation 🔥 Multi-View SfM 🔥 Multi-View Stereo 🔥 Depth Completion 🔥 Registration … and many more possibilities! – plus everything is metric 🎯 We release code for data processing, training, benchmarking & ablations – everything Apache 2.0! Details & Links 👇

Nikhil Keetha

123,641 Aufrufe • vor 1 Jahr

For the past 3 months, I've been heads down building and extending our multimodal data capture rig for Physical AI training. What you're seeing here is a single synchronized capture from our system — RGB, depth (stereo IR camera) feed; real-time 3D hand reconstruction, and a 23×20 tactile pressure grid from sensor gloves, all aligned and timestamped together. Getting here took a lot of unglamorous iteration. We reworked the pipeline to deliver clean, gap-free depth at scale after identifying early depth frame drops caused by data throughput bottlenecks in our capture box. We built per-channel calibration models that convert raw resistance signals from our flex sensor gloves into accurate, drift-corrected force values — solving the creep behavior that made a steady grip falsely appear to fade over time. Even though we’ve stayed quiet on socials, word of mouth spread — our rigs have already been deployed in data factories, manufacturing lines, and homes across the US, Europe, China, and Southeast Asia by our customers and ourselves. In the next few days, I'll be sharing more about how we're tackling rights-cleared data capture from the hardware level, and our closed-loop data-model flywheel — using model evaluation results to identify gaps and guide what data our customers need. More soon, stay tuned…
0:26

For the past 3 months, I've been heads down building and extending our multimodal data capture rig for Physical AI training. What you're seeing here is a single synchronized capture from our system — RGB, depth (stereo IR camera) feed; real-time 3D hand reconstruction, and a 23×20 tactile pressure grid from sensor gloves, all aligned and timestamped together. Getting here took a lot of unglamorous iteration. We reworked the pipeline to deliver clean, gap-free depth at scale after identifying early depth frame drops caused by data throughput bottlenecks in our capture box. We built per-channel calibration models that convert raw resistance signals from our flex sensor gloves into accurate, drift-corrected force values — solving the creep behavior that made a steady grip falsely appear to fade over time. Even though we’ve stayed quiet on socials, word of mouth spread — our rigs have already been deployed in data factories, manufacturing lines, and homes across the US, Europe, China, and Southeast Asia by our customers and ourselves. In the next few days, I'll be sharing more about how we're tackling rights-cleared data capture from the hardware level, and our closed-loop data-model flywheel — using model evaluation results to identify gaps and guide what data our customers need. More soon, stay tuned…

Grace Zhang

54,991 Aufrufe • vor 4 Monaten

🚨 BREAKING: Big news in the computer vision world! 🎥 Luxonis | Robotic Vision just dropped its new OAK 4 line, and it’s a big upgrade for edge computer vision. Instead of being “just a stereo camera,” OAK 4 is a fully standalone vision computer with 52 TOPS of on-device AI. Models run locally, depth is computed locally, and no external PC or cloud pipeline is required. This is why robotics teams love it: lower latency, lower cost, fewer failure points in the field. The hardware is built for the real-world. IP67, shock-resistant, wide-FOV RGB + stereo pair, IR projection, IMU, audio, and a patent-pending calibration system that keeps depth accurate even when conditions change. But the real move is the platform. With Luxonis Hub, you can deploy models, grab telemetry, push OTA updates, or collect data when performance drifts, all from a unified interface. It turns a single device into an end-to-end edge CV system. Most customers today in robotics are groups who just want something that works: AMRs, bin-picking systems, trailer-loading robots, and ag-tech. 🤖 And they all say the same thing, the appeal isn’t raw TOPS, it’s the all-in-one simplicity that lets them scale without building custom infrastructure. Feels like the direction edge vision has been waiting for: rugged hardware + high-throughput on-device compute + a real management layer. A next step toward “plug-and-deploy” perception for robots. 🔗 Find out more here: ~~ ♻️ Join the weekly robotics newsletter, and never miss any news →
1:12

🚨 BREAKING: Big news in the computer vision world! 🎥 Luxonis | Robotic Vision just dropped its new OAK 4 line, and it’s a big upgrade for edge computer vision. Instead of being “just a stereo camera,” OAK 4 is a fully standalone vision computer with 52 TOPS of on-device AI. Models run locally, depth is computed locally, and no external PC or cloud pipeline is required. This is why robotics teams love it: lower latency, lower cost, fewer failure points in the field. The hardware is built for the real-world. IP67, shock-resistant, wide-FOV RGB + stereo pair, IR projection, IMU, audio, and a patent-pending calibration system that keeps depth accurate even when conditions change. But the real move is the platform. With Luxonis Hub, you can deploy models, grab telemetry, push OTA updates, or collect data when performance drifts, all from a unified interface. It turns a single device into an end-to-end edge CV system. Most customers today in robotics are groups who just want something that works: AMRs, bin-picking systems, trailer-loading robots, and ag-tech. 🤖 And they all say the same thing, the appeal isn’t raw TOPS, it’s the all-in-one simplicity that lets them scale without building custom infrastructure. Feels like the direction edge vision has been waiting for: rugged hardware + high-throughput on-device compute + a real management layer. A next step toward “plug-and-deploy” perception for robots. 🔗 Find out more here: ~~ ♻️ Join the weekly robotics newsletter, and never miss any news →

Lukas Ziegler

41,955 Aufrufe • vor 10 Monaten

Forget lidar. One single camera. Runs in real time & is open source: A streaming 3D model that reconstructs scenes live, at ~20 FPS, over long sequences. End-to-end. Optimization tricks, cleanup steps? Nope. And it beats both streaming and even some offline methods. Perception is becoming software-first. Closer to machines that see and understand the world as it unfolds. Thanks for sharing, Yinghao Xu 📍Models: Project page: Code: Paper: —— Weekly robotics and AI insights. Subscribe free:
3:30

Forget lidar. One single camera. Runs in real time & is open source: A streaming 3D model that reconstructs scenes live, at ~20 FPS, over long sequences. End-to-end. Optimization tricks, cleanup steps? Nope. And it beats both streaming and even some offline methods. Perception is becoming software-first. Closer to machines that see and understand the world as it unfolds. Thanks for sharing, Yinghao Xu 📍Models: Project page: Code: Paper: —— Weekly robotics and AI insights. Subscribe free:

Ilir Aliu

112,718 Aufrufe • vor 6 Tagen

Forget lidar. One single camera. Runs in real time & is open source: A streaming 3D model that reconstructs scenes live, at ~20 FPS, over long sequences. End-to-end. Optimization tricks, cleanup steps? Nope. And it beats both streaming and even some offline methods. Perception is becoming software-first. Closer to machines that see and understand the world as it unfolds. Thanks for sharing, Yinghao Xu 📍Models: Project page: Code: Paper: —— Weekly robotics and AI insights. Subscribe free:
3:30

Forget lidar. One single camera. Runs in real time & is open source: A streaming 3D model that reconstructs scenes live, at ~20 FPS, over long sequences. End-to-end. Optimization tricks, cleanup steps? Nope. And it beats both streaming and even some offline methods. Perception is becoming software-first. Closer to machines that see and understand the world as it unfolds. Thanks for sharing, Yinghao Xu 📍Models: Project page: Code: Paper: —— Weekly robotics and AI insights. Subscribe free:

Ilir Aliu

596,256 Aufrufe • vor 3 Monaten

🥳Excited to share our latest work: "Diff-A-Riff"! 🥁 A Latent Diffusion Model that generates instrumental accompaniments for any musical input, specifically tailored for music producers! It's faster, lighter, and produces superior audio quality. Control via text/audio references. 48kHz sample rate, (pseudo) stereo, ~3Gb memory, takes 6 seconds to generate 90 seconds of music. Trained on a single GPU. 📜 🎶 🎸 "Diff-A-Riff" adapts to any musical input, following the artist's unique style. 🎛️ Optional controls via text prompts, audio references, interpolation slider, pseudo-stereo width and loop intensity. 🎚️ It produces state-of-the-art audio quality indistinguishable from real data by human raters and operates at unprecedented speed. 🧠 "Diff-A-Riff" is smaller and more efficient than previous models thanks to its Consistency Autoencoder, making it accessible and practical for various applications. Big shoutout to my outer space colleagues: Javier Nistal, the Machine in "machine learning" 🚄, Marco Pasini, the neural net whisperer 🤫, Cyran Aouameur, the troubleshootah 🛠️, Maarten Grachten, aka MaartenGPT 🤖. #Teamwork #AI #MusicTech #Innovation Javier Nistal Marco Pasini Cyran Aouameur SonyCSL(Paris)_Music Team
3:33

🥳Excited to share our latest work: "Diff-A-Riff"! 🥁 A Latent Diffusion Model that generates instrumental accompaniments for any musical input, specifically tailored for music producers! It's faster, lighter, and produces superior audio quality. Control via text/audio references. 48kHz sample rate, (pseudo) stereo, ~3Gb memory, takes 6 seconds to generate 90 seconds of music. Trained on a single GPU. 📜 🎶 🎸 "Diff-A-Riff" adapts to any musical input, following the artist's unique style. 🎛️ Optional controls via text prompts, audio references, interpolation slider, pseudo-stereo width and loop intensity. 🎚️ It produces state-of-the-art audio quality indistinguishable from real data by human raters and operates at unprecedented speed. 🧠 "Diff-A-Riff" is smaller and more efficient than previous models thanks to its Consistency Autoencoder, making it accessible and practical for various applications. Big shoutout to my outer space colleagues: Javier Nistal, the Machine in "machine learning" 🚄, Marco Pasini, the neural net whisperer 🤫, Cyran Aouameur, the troubleshootah 🛠️, Maarten Grachten, aka MaartenGPT 🤖. #Teamwork #AI #MusicTech #Innovation Javier Nistal Marco Pasini Cyran Aouameur SonyCSL(Paris)_Music Team

Stefan Lattner

17,746 Aufrufe • vor 2 Jahren

Tired of teleoperation? One human video → 1,000s of robot demos. (📍GitHub ) Scaling Robot Data Without Dynamics Simulation or Robot Hardware Real2Render2Real (R2R2R) is a new way to scale robot data without physics simulation or hardware. You take a phone scan + a single monocular human demo. It tracks the motion, renders photorealistic scenes, and generates diverse, robot-agnostic trajectories ready for training. > No teleop, no sim, no robot, just a phone and a video > Train VLA models and diffusion policies directly on the output > Supports multiple robot embodiments with kinematic consistency > 1000s of demos in 1/27 the time of real-world collection Thank you, Max Fu, for sharing!! Project: Paper: Code coming soon: It shows that with the right pipeline, you can scale robot learning data without touching a robot. One of the most interesting directions in scalable robotics today. —— Weekly robotics and AI insights. Subscribe free:
0:29

Tired of teleoperation? One human video → 1,000s of robot demos. (📍GitHub ) Scaling Robot Data Without Dynamics Simulation or Robot Hardware Real2Render2Real (R2R2R) is a new way to scale robot data without physics simulation or hardware. You take a phone scan + a single monocular human demo. It tracks the motion, renders photorealistic scenes, and generates diverse, robot-agnostic trajectories ready for training. > No teleop, no sim, no robot, just a phone and a video > Train VLA models and diffusion policies directly on the output > Supports multiple robot embodiments with kinematic consistency > 1000s of demos in 1/27 the time of real-world collection Thank you, Max Fu, for sharing!! Project: Paper: Code coming soon: It shows that with the right pipeline, you can scale robot learning data without touching a robot. One of the most interesting directions in scalable robotics today. —— Weekly robotics and AI insights. Subscribe free:

Ilir Aliu

42,864 Aufrufe • vor 8 Monaten

PERCEPTRON JUST DROPPED WHAT MIGHT BE THE BEST OPEN-WEIGHT MODEL FOR ROBOTICS Isaac 0.5 is an absolute beast. Instead of treating vision and movement as separate systems, it is a fully open foundation model that learns to process both at the exact same time. .. and by merging perception and control, it unlocks true real-time visual decision making. Some core stats: → 35 hardware embodiments → 100k hours of demonstrations → SOTA across core embodied reasoning benchmarks → Significantly lower inference FLOPs Best part? Perceptron AI open-sourced the ENTIRE stack! Training code, inference code, and weights are all public. Repo, weights, and paper in 🧵 ↓
6:12

PERCEPTRON JUST DROPPED WHAT MIGHT BE THE BEST OPEN-WEIGHT MODEL FOR ROBOTICS Isaac 0.5 is an absolute beast. Instead of treating vision and movement as separate systems, it is a fully open foundation model that learns to process both at the exact same time. .. and by merging perception and control, it unlocks true real-time visual decision making. Some core stats: → 35 hardware embodiments → 100k hours of demonstrations → SOTA across core embodied reasoning benchmarks → Significantly lower inference FLOPs Best part? Perceptron AI open-sourced the ENTIRE stack! Training code, inference code, and weights are all public. Repo, weights, and paper in 🧵 ↓

Charly Wargnier ♨️

47,979 Aufrufe • vor 1 Monat

Introducing Code Video Model 🎬 Amazing work led by Ziyu Wan and the team! Agentic coding as the engine for controllable video generation. Scene layout, motion, timing, camera trajectories, and physical dynamics, all in editable code. A coding agent turns prompts into executable programs that render proxy videos. Our training-free early-step conditioning lets a pretrained video model follow the proxy’s structure while adopting a reference image’s appearance. From physics-grounded motion and bullet time to 3D/4D reconstruction and robotics simulation, it all starts with code. Watch the demo and explore more visual results 👇 Project page: Paper: Code: Data: Youtube:
2:17

Introducing Code Video Model 🎬 Amazing work led by Ziyu Wan and the team! Agentic coding as the engine for controllable video generation. Scene layout, motion, timing, camera trajectories, and physical dynamics, all in editable code. A coding agent turns prompts into executable programs that render proxy videos. Our training-free early-step conditioning lets a pretrained video model follow the proxy’s structure while adopting a reference image’s appearance. From physics-grounded motion and bullet time to 3D/4D reconstruction and robotics simulation, it all starts with code. Watch the demo and explore more visual results 👇 Project page: Paper: Code: Data: Youtube:

Gordon (Guocheng) Qian

32,350 Aufrufe • vor 1 Tag

Been looking into RICE AI ( ◉ - ◉ ) lately, and it’s one of the few projects actually tackling a real challenge in robotics and AGI: the lack of high-quality training data. Robots can’t get truly intelligent without it. They building a decentralized way to collect and share robotics data, using a DePIN network to train physical AI foundation models. The project was even hand-picked by CZ as a winner of the Dubai BNB Chain Demo Day, and it’s backed by Floki while also being part of the NVIDIA Inception Program. They also launched FLOKI Minibot M1 - a small AI companion that doubles as a data node. Every interaction with it feeds the RICE AI Foundry, the first decentralized platform for Robotics Foundation Models, and contributors earn $RICE tokens in return. And it’s working: – 800 minibots sold in 24h – 33,000% oversubscribed Binance IDO – 500+ robots already deployed worldwide Feels like a project with real long-term potential. Worth keeping an eye on. RICE AI ( ◉ - ◉ )
1:33

Been looking into RICE AI ( ◉ - ◉ ) lately, and it’s one of the few projects actually tackling a real challenge in robotics and AGI: the lack of high-quality training data. Robots can’t get truly intelligent without it. They building a decentralized way to collect and share robotics data, using a DePIN network to train physical AI foundation models. The project was even hand-picked by CZ as a winner of the Dubai BNB Chain Demo Day, and it’s backed by Floki while also being part of the NVIDIA Inception Program. They also launched FLOKI Minibot M1 - a small AI companion that doubles as a data node. Every interaction with it feeds the RICE AI Foundry, the first decentralized platform for Robotics Foundation Models, and contributors earn $RICE tokens in return. And it’s working: – 800 minibots sold in 24h – 33,000% oversubscribed Binance IDO – 500+ robots already deployed worldwide Feels like a project with real long-term potential. Worth keeping an eye on. RICE AI ( ◉ - ◉ )

Christiaan

12,696 Aufrufe • vor 1 Jahr

An interactive world model developed by NVIDIA in collaboration with academic partners. - DreamDojo turns egocentric human video data into physical intelligence. - Human data is more scalable than robotics data but lacks action labels. - To solve this, a dedicated action model extracts latent actions by identifying physics and motion deltas between frames. Training - A massive 44k hours of video data are used for pre-training. - Post-training on small-scale robot datasets maps human physics to specific robot embodiments. - An additional distillation stage converts the model into an autoregressive, few-step diffusion model, enabling real-time, action-controllable simulation. Primary Use Cases - Live Teleoperation: Controlling a robot inside a world simulation in real-time. - Model-based Planning: Previewing and curating the best actions for improved success. - Policy Evaluation: Testing robot policies in realistic, out-of-distribution scenarios. Everything that's open-sourced: weights, code, post-training dataset, eval set, and details to reproduce.
1:14

An interactive world model developed by NVIDIA in collaboration with academic partners. - DreamDojo turns egocentric human video data into physical intelligence. - Human data is more scalable than robotics data but lacks action labels. - To solve this, a dedicated action model extracts latent actions by identifying physics and motion deltas between frames. Training - A massive 44k hours of video data are used for pre-training. - Post-training on small-scale robot datasets maps human physics to specific robot embodiments. - An additional distillation stage converts the model into an autoregressive, few-step diffusion model, enabling real-time, action-controllable simulation. Primary Use Cases - Live Teleoperation: Controlling a robot inside a world simulation in real-time. - Model-based Planning: Previewing and curating the best actions for improved success. - Policy Evaluation: Testing robot policies in realistic, out-of-distribution scenarios. Everything that's open-sourced: weights, code, post-training dataset, eval set, and details to reproduce.

The Humanoid Hub

11,575 Aufrufe • vor 7 Monaten

Introducing Open-TeleVision: with Fully Autonomous policy video👇. We can conduct a long-horizon task with inserting 12 cans nonstop without any interruptions. We offer: 🤖 Highly precise and smooth bimanual manipulation. 📺 Active egocentric vision (with a moving neck) feedback. It is achieved by imitation learning from teleoperation: We propose a VR-based REAL-TIME teleoperation that streams the stereo video observation from the robot camera to the VR device. The robot neck moves as the human head moves, the robot hands move as the human hands move, offering the operator an intuitive experience as the human herself becomes the robot. The devils are all in the details, and how to implement things right: ✅How to perform IK/retargeting for smooth and precise control. ✅How to do all these and also stream stereo video without no latency, all in real time. We released our code here: Active head hardware design: 1/n
0:33

Introducing Open-TeleVision: with Fully Autonomous policy video👇. We can conduct a long-horizon task with inserting 12 cans nonstop without any interruptions. We offer: 🤖 Highly precise and smooth bimanual manipulation. 📺 Active egocentric vision (with a moving neck) feedback. It is achieved by imitation learning from teleoperation: We propose a VR-based REAL-TIME teleoperation that streams the stereo video observation from the robot camera to the VR device. The robot neck moves as the human head moves, the robot hands move as the human hands move, offering the operator an intuitive experience as the human herself becomes the robot. The devils are all in the details, and how to implement things right: ✅How to perform IK/retargeting for smooth and precise control. ✅How to do all these and also stream stereo video without no latency, all in real time. We released our code here: Active head hardware design: 1/n

Xiaolong Wang

25,744 Aufrufe • vor 2 Jahren

I’m thrilled to announce that we just released GraspGen, a multi-year project we have been cooking at NVIDIA Robotics 🚀 GraspGen: A Diffusion-Based Framework for 6-DOF Grasping Grasping is a foundational challenge in robotics 🤖 — whether for industrial picking or general-purpose humanoids. VLA + real data collection is all the rage now but is expensive and scales poorly for this task. For every new gripper and/or scene, you’ll have to recollect the dataset in this paradigm for the best perf. 💡Key Idea: Since grasping is such a well-defined task in simulation - why can’t we just scale synthetic data generation and train a generative model for grasping? By embracing modularity and standardized grasp formats, we can make this a turnkey technology that works zero-shot for multiple settings. GraspGen is a modular framework for diffusion-based 6-DOF grasp generation that scales across embodiment types, observability conditions, clutter, task complexity. Key Features: ✅ Multi-embodiment support: suction, parallel-jaw, and multi-fingered grippers ✅ Generalization to partial + complete 3D point clouds ✅ Generalization to single-objects + cluttered scenes ✅ Modular design uses other robotics modules and foundation models (SAM2, cuRobo, FoundationStereo, FoundationPose). This allows GraspGen to focus on only one thing - grasp generation ✅ Training recipe: grasp discriminator is trained with On-Generator data from the diffusion model - so that it learns to correct the mistakes (if any) of the diffusion generator ✅ Real-time performance (~20 Hz) before any GPU acceleration; low memory footprint 📊 Results: • SOTA on the FetchBench [Han et al. CoRL 2024] benchmark • Zero-shot sim-to-real transfer on unknown objects and cluttered scenes • Dataset of 53M simulated grasps across 8K objects from Objaverse 📄 arXiv: 🌐 Website: 💻 Code: A huge thank you to everyone involved in this journey — excited to see what the community builds on top of it! Joint work with Clemens Eppner , Balakumar Sundaralingam , Yu-Wei, Jun Yamada Wentao Yuan and other collaborators #robotics #diffusionmodels #physicalAI #simtoreal
0:39

I’m thrilled to announce that we just released GraspGen, a multi-year project we have been cooking at NVIDIA Robotics 🚀 GraspGen: A Diffusion-Based Framework for 6-DOF Grasping Grasping is a foundational challenge in robotics 🤖 — whether for industrial picking or general-purpose humanoids. VLA + real data collection is all the rage now but is expensive and scales poorly for this task. For every new gripper and/or scene, you’ll have to recollect the dataset in this paradigm for the best perf. 💡Key Idea: Since grasping is such a well-defined task in simulation - why can’t we just scale synthetic data generation and train a generative model for grasping? By embracing modularity and standardized grasp formats, we can make this a turnkey technology that works zero-shot for multiple settings. GraspGen is a modular framework for diffusion-based 6-DOF grasp generation that scales across embodiment types, observability conditions, clutter, task complexity. Key Features: ✅ Multi-embodiment support: suction, parallel-jaw, and multi-fingered grippers ✅ Generalization to partial + complete 3D point clouds ✅ Generalization to single-objects + cluttered scenes ✅ Modular design uses other robotics modules and foundation models (SAM2, cuRobo, FoundationStereo, FoundationPose). This allows GraspGen to focus on only one thing - grasp generation ✅ Training recipe: grasp discriminator is trained with On-Generator data from the diffusion model - so that it learns to correct the mistakes (if any) of the diffusion generator ✅ Real-time performance (~20 Hz) before any GPU acceleration; low memory footprint 📊 Results: • SOTA on the FetchBench [Han et al. CoRL 2024] benchmark • Zero-shot sim-to-real transfer on unknown objects and cluttered scenes • Dataset of 53M simulated grasps across 8K objects from Objaverse 📄 arXiv: 🌐 Website: 💻 Code: A huge thank you to everyone involved in this journey — excited to see what the community builds on top of it! Joint work with Clemens Eppner , Balakumar Sundaralingam , Yu-Wei, Jun Yamada Wentao Yuan and other collaborators #robotics #diffusionmodels #physicalAI #simtoreal

Adithya Murali

24,547 Aufrufe • vor 1 Jahr

Introducing Praxis-1, an open-weight World Action Model that turns our video pre-training expertise into real world control for robots. Robot demonstration data is limited. But video data is infinite. Praxis-1 extends our bet that the best policy models will learn from video, and use that scale to bring embodied intelligence to every industry. Praxis-1 is built on our large-scale video pre-training and real-time infrastructure, and acts as a policy model for robotics developers and researchers that works across any embodiment or environment, no matter how complex. We are actively testing with early partners including Noble Machines, Standard Bots and Ultra, and will be releasing the weights publicly in the coming months. Learn more and request early access at
1:54

Introducing Praxis-1, an open-weight World Action Model that turns our video pre-training expertise into real world control for robots. Robot demonstration data is limited. But video data is infinite. Praxis-1 extends our bet that the best policy models will learn from video, and use that scale to bring embodied intelligence to every industry. Praxis-1 is built on our large-scale video pre-training and real-time infrastructure, and acts as a policy model for robotics developers and researchers that works across any embodiment or environment, no matter how complex. We are actively testing with early partners including Noble Machines, Standard Bots and Ultra, and will be releasing the weights publicly in the coming months. Learn more and request early access at

Runway

118,279 Aufrufe • vor 13 Tagen

It's working now 😁 The code is on GitHub: So this project is a 2D digital twin that synchronizes a physical object in this case an mpu6050 sensor to an Arduino with a virtual model. Here it's the robot image in real time. Some of the issues I fixed: - Shaky wires 😭 , when the USB cable shakes, the serial connection drops... I solved this by writing an auto reconnect function that searches for the Arduino without crashing and also fixed the position of the Arduino - There was also a time delay between the Arduino and python and a lag between the data received and the pygame interface... Too much data.
0:27

It's working now 😁 The code is on GitHub: So this project is a 2D digital twin that synchronizes a physical object in this case an mpu6050 sensor to an Arduino with a virtual model. Here it's the robot image in real time. Some of the issues I fixed: - Shaky wires 😭 , when the USB cable shakes, the serial connection drops... I solved this by writing an auto reconnect function that searches for the Arduino without crashing and also fixed the position of the Arduino - There was also a time delay between the Arduino and python and a lag between the data received and the pygame interface... Too much data.

Osasemwinhia Egharevba

23,855 Aufrufe • vor 7 Monaten

We might be solving the wrong problem in robotics. That’s what this makes clear. UMI → Universal Manipulation Interface A simple $400 gripper that lets you teach robots by demonstration. You hold it like a tool. Show the task. The robot learns. No teleoperation. No expensive hardware. No robot-specific data. Stanford open-sourced everything → hardware, code, datasets. What stands out to me is the bottleneck. Not algorithms. Data. Teleoperation → ~35 demos/hour UMI → ~111 demos/hour And the data transfers across robots → UR5, Franka, others. The design is surprisingly practical: → GoPro fisheye lens (155° FOV) + mirrors for depth → SLAM + IMU for precise 6DoF tracking → latency matching for dynamic tasks → diffusion policies for multimodal actions Then it scales. Cheng Chi takes this further with Sunday Robotics (with Tony Zhao). A $200 glove → deployed in 500+ homes → ~10 million real-world interactions. Not lab data. Real human behavior. Their robot learns dishes, laundry, espresso → with zero robot-specific data. This is where the shift becomes obvious. From training robots in controlled environments → to learning directly from humans at scale So here’s the real question: Will robotics be unlocked by better models… or by unlocking data? #ArtificialIntelligence #Robotics #AI #Innovation #FutureOfWork
0:20

We might be solving the wrong problem in robotics. That’s what this makes clear. UMI → Universal Manipulation Interface A simple $400 gripper that lets you teach robots by demonstration. You hold it like a tool. Show the task. The robot learns. No teleoperation. No expensive hardware. No robot-specific data. Stanford open-sourced everything → hardware, code, datasets. What stands out to me is the bottleneck. Not algorithms. Data. Teleoperation → ~35 demos/hour UMI → ~111 demos/hour And the data transfers across robots → UR5, Franka, others. The design is surprisingly practical: → GoPro fisheye lens (155° FOV) + mirrors for depth → SLAM + IMU for precise 6DoF tracking → latency matching for dynamic tasks → diffusion policies for multimodal actions Then it scales. Cheng Chi takes this further with Sunday Robotics (with Tony Zhao). A $200 glove → deployed in 500+ homes → ~10 million real-world interactions. Not lab data. Real human behavior. Their robot learns dishes, laundry, espresso → with zero robot-specific data. This is where the shift becomes obvious. From training robots in controlled environments → to learning directly from humans at scale So here’s the real question: Will robotics be unlocked by better models… or by unlocking data? #ArtificialIntelligence #Robotics #AI #Innovation #FutureOfWork

Pascal Bornet

186,836 Aufrufe • vor 6 Monaten

The first open-source unified world model for scalable robot manipulation: 5B-parameter open-source unified video-action world model that combines policy and world modeling to generate robot actions, predict future visuals, and evaluate task progress from observations, language, and state. The model is trained on 27.3K hours of heterogeneous data (17.8K real-robot teleop, 6.5K UMI demos, 3K egocentric human videos), enabling it to perform complex manipulation tasks like faucet connecting, bag packing, and toolbox storing as shown in demo videos. The approach supports test-time action refinement and points toward deployment-driven continuous improvement via fleet data. Thanks for sharing, Jianlan Luo (Jianlan Luo)! 📌 Resource links for τ0-WM: • Project page: • GitHub (code): • Hugging Face (model weights): • Paper (PDF): ——- Weekly robotics and AI insights. Subscribe free:
2:47

The first open-source unified world model for scalable robot manipulation: 5B-parameter open-source unified video-action world model that combines policy and world modeling to generate robot actions, predict future visuals, and evaluate task progress from observations, language, and state. The model is trained on 27.3K hours of heterogeneous data (17.8K real-robot teleop, 6.5K UMI demos, 3K egocentric human videos), enabling it to perform complex manipulation tasks like faucet connecting, bag packing, and toolbox storing as shown in demo videos. The approach supports test-time action refinement and points toward deployment-driven continuous improvement via fleet data. Thanks for sharing, Jianlan Luo (Jianlan Luo)! 📌 Resource links for τ0-WM: • Project page: • GitHub (code): • Hugging Face (model weights): • Paper (PDF): ——- Weekly robotics and AI insights. Subscribe free:

Ilir Aliu

28,276 Aufrufe • vor 4 Monaten