How can robots reliably place objects in diverse real-world... tasks? 🤖🔍 Placement is tough—objects vary in shape and placement modes (such as stacking, hanging, and insertion), making it a challenging problem. We introduce AnyPlace, a two-stage method trained purely on synthetic data to predict diverse placement poses of unseen objects for real-world tasks. Read on for more👇show more

#Battlefield6's Assault Ladder gadget has two modes. It can... be used as a "real" ladder or as a ramp depending on placement angle.show more

🤖 NVIDIA’s Gr00t N1.5 is now available in LeRobot!... This is the result of a great collaboration between the Hugging Face LeRobot team and NVIDIA Robotics ! Gr00t N1.5 highlights: 🦾 Cross-embodiment foundation model for robots 🧠 Multimodal inputs: vision, language, and proprioception 🪛Tested on the Libero benchmark and real-world hardware tasks 🌍Trained on real robot, synthetic, and internet-scale video data ⚙️ Flow matching action transformer for action predictionshow more

You can actually interact with the world simulator directly... in the browser. 🤖 Here is a quick screen recording (8x speed) of me playing with it: real-time action-conditioned video prediction across rigid objects, deformable objects, rope, and object piles. Try it yourself (no install required): Huge kudos to my student Yixuan Wang for making the interactive demo happen!show more

Working on a simple dialogue system for NPCs and... other objects in the world. #gamedevshow more

Skild AI's new S1 robot foundation model helps robots... learn previously unseen tasks from a single video demonstration. 🤖 See how NVIDIA technology supports S1 from training and simulation to real-world deployment. Learn more ➡️show more

A new approach uses diffused orientation fields to help... robots transfer skills across curved objects of diverse shapes and sizes, enabling them to peel and slice varied fruits and vegetables. Learn more in Science Robotics:show more

We’ve seen humanoid robots walk around for a while,... but when will they actually help with useful tasks in daily life? The challenge here is the diversity and complexity of real-world scenes. Our new work tackles this problem via 3D visuomotor policy learning. Using data from only 1 scene, our Improved 3D Diffusion Policy (iDP3) enables a full-sized humanoid robot to autonomously pick&place objects, pour water, and wipe tables, in the wild open world. (and all these skills are useful, right?) Web: Fully open-sourced code:show more

Spatial AI ( is building large real-world datasets to... teach robots how to navigate the world and complete tasks. Their first open source dataset, SEA (Spatial Everyday Activities), is the largest curated egocentric dataset of people carrying out real tasks, with 10,000 hours of data.show more

Dense tracking of deformable objects can unlock applications in... robotics, gen-AI and AR. We present DeformGS (previously MD-Splatting) and release the code and data. Join us at #WAFR where we will present new real-world results! 👇 1/9🧵show more

You can't 3D reconstruct glass from images... ...WRONG! Thanks... for video diffusion, now just about anything is possible! Introducing...Diffusion Knows Transparency (DKT) Transparent and reflective objects usually break robot vision and photogrammetry pipelines because they don't follow the "solid object" rules standard cameras expect. DKT is a new AI model that repurposes the "internal physics engine" found in video generation models to solve this problem. Researchers took a massive video diffusion model (WAN) and fine-tuned it using a custom-built synthetic dataset to turn it into a high-precision depth sensor. To train the AI, they built the first massive synthetic video library of transparent objects, 1.32 million frames of perfectly labeled glass and metal objects in motion. Without ever seeing a "real" labeled video of glass during training, the model (DKT) outperformed all previous specialized systems on real-world benchmarks (ClearPose, DREDS). They created a "lightweight" 1.3B parameter version that runs fast enough (0.17s per frame) to be used on actual robot hardware. Two reasons I find this project important: 1. It further proves that synthetic data will be essential for training the next generation vision models. 2. In real-world robotic tests, using DKT's depth maps nearly doubled the success rate of robot arms trying to pick up objects on tricky reflective or translucent surfaces. At home robots will need to interact with these types of objects on a daily basis. Check out the project page here: Code is LIVE! #Computervision #Robotics #AIshow more

MAKING AXIS TASKS FEEL EASY 👀🤖 I made a... quick video showing how to complete an Axis Robotics task step by step. The task gives you a clear goal, steps, controls and camera views to guide you. In this one, the goal is simple, Open the trash bin, pick up the onion, and place it inside. 🧅 Just follow the instructions, control the robot arm and complete the task. What I like is that even beginners can understand what they need to do. But the interesting part is what happens with these interactions after we complete them. Each completed task can contribute valuable robot training data, helping Physical Ai systems learn how to handle different objects and environments. Small tasks from the community can become useful data for teaching robots how to interact with the real world. That’s what makes Axis interesting to me.show more

🤖 Can a single robot policy manipulate diverse tools... without ever seeing them before? Introducing SimToolReal 🔨 : a generalist dexterous manipulation policy that transfers zero-shot sim→real to unseen tools + unseen tasks All videos are 1x speed (60 Hz control) 🧵👇show more

How can we responsibly deploy agentic AI robots in... real-world environments? As we close the gap between AI reasoning and physical action, developing a new safety framework is key to advancing helpful robots. 🤖show more

Check out this Stereo4D paper from Google DeepMind. It's... a pretty clever approach to a persistent problem in computer vision -- getting good training data for how things move in 3D. The key insight is using VR180 videos -- those stereo fisheye videos we launched back in 2017 for YouTubeVR. It was always clear that structured stereo datasets would be valuable for computer vision -- and we launched some powerful VR tools with it back in 2017 (link below). But what's the game changer now in 2024 is the scale -- they're providing 110K high quality clips :-) That's the kind of massive, real-world AI dataset that was just a dream back then! They're using it to train this model called DynaDUSt3R that can predict both 3D structure and motion from video frames. Which means it tracks how objects move between frames while simultaneously reconstructing their 3D shape. And given we're dealing with real stereoscopic content, results are notably better than synthetic data, giving you a faithful rendition of the real-world with a diverse set of subject matter. It's one of those through lines when tackling a timeless mission like mapping the world or spatial computing -- VR content created for immersion becoming the foundation for teaching machines to understand how the world moves. Sometimes innovation chains together in unexpected ways! Links to projects below⛓️show more

Trained on zero real-world data. Learned to walk, pick... up boxes, and follow multi-step instructions... in the REAL world. ( 📌 Paper below) Researchers from Amazon FAR, Berkeley, Stanford, and CMU scanned real rooms with an iPhone, rebuilt them as 3D Gaussian Splatting scenes, then generated 48,000 synthetic trajectories of a Unitree G1 walking, grasping, and placing objects inside those virtual replicas. They rendered the robot's first-person camera view from each run and paired it with the matching language instruction and motion data. That's the dataset every humanoid team needs and nobody has: synced egocentric video + language + kinematics, at scale. Instead of collecting it in the real world, they manufactured it. They trained a vision-language-kinematics policy on that synthetic data alone, then deployed it on the physical G1 across five task types: navigation to a named object, lifting boxes of three different sizes with no per-size tuning, chained multi-step tasks, robustness to mid-task layout changes and flickering lights, and multi-minute long-horizon runs. No real-world fine-tuning at any point. Real-world interaction data has been the hard limit on humanoid learning... slow, expensive, and small. If scanning a room once and synthesizing thousands of labeled interactions holds up as a general recipe, that limit moves. Data stops being the bottleneck robotics teams have to solve for. 📌 Paper: Project: ——- Weekly robotics and AI insights. Subscribe free:show more

We're currently testing new set design and camera placement... AI processes for our portfolio! We're using colored markers to place our "cameras" and characters in specific locations on a specific set! it's similar to preparing a set and scenery for a real film. AI Tools : Midjourney (Set Place 2D) /Freepik (now Magnific) : Nano Banana Pro (3D and Camera / character placement) / Grok + Kling (Animation) / Topaz (Upscale) / Characters (See process in quote)show more