Loading video...

Video Failed to Load

Go Home

๐Ÿค– Another zero-shot reward model is now in LeRobot: ROBOMETER. A general-purpose, zero-shot video-language reward model from University of South Carolina, UT Dallas, Massachusetts Institute of Technology (MIT), University of Washington, Ai2, and NVIDIA that predicts frame-level task progress. Trained on 1M+ trajectories from 21 robot embodiments, generalizes zero-shot...

32,625 views โ€ข 1 month ago โ€ขvia X (Twitter)

0 Comments

No comments available

Comments from the original post will appear here

Related Videos

Trained on zero real-world data. Learned to walk, pick up boxes, and follow multi-step instructions... in the REAL world. ( ๐Ÿ“Œ Paper below) Researchers from Amazon FAR, Berkeley, Stanford, and CMU scanned real rooms with an iPhone, rebuilt them as 3D Gaussian Splatting scenes, then generated 48,000 synthetic trajectories of a Unitree G1 walking, grasping, and placing objects inside those virtual replicas. They rendered the robot's first-person camera view from each run and paired it with the matching language instruction and motion data. That's the dataset every humanoid team needs and nobody has: synced egocentric video + language + kinematics, at scale. Instead of collecting it in the real world, they manufactured it. They trained a vision-language-kinematics policy on that synthetic data alone, then deployed it on the physical G1 across five task types: navigation to a named object, lifting boxes of three different sizes with no per-size tuning, chained multi-step tasks, robustness to mid-task layout changes and flickering lights, and multi-minute long-horizon runs. No real-world fine-tuning at any point. Real-world interaction data has been the hard limit on humanoid learning... slow, expensive, and small. If scanning a room once and synthesizing thousands of labeled interactions holds up as a general recipe, that limit moves. Data stops being the bottleneck robotics teams have to solve for. ๐Ÿ“Œ Paper: Project: โ€”โ€”- Weekly robotics and AI insights. Subscribe free:

Ilir Aliu

12,950 views โ€ข 9 days ago

๐—ฅ๐—ผ๐—ฏ๐—ผ๐˜๐˜€ ๐—ฑ๐—ผ๐—ปโ€™๐˜ ๐—ป๐—ฒ๐—ฒ๐—ฑ ๐—บ๐—ผ๐—ฟ๐—ฒ ๐—ฑ๐—ฒ๐—บ๐—ผ๐—ป๐˜€๐˜๐—ฟ๐—ฎ๐˜๐—ถ๐—ผ๐—ป๐˜€. ๐—ง๐—ต๐—ฒ๐˜† ๐—ป๐—ฒ๐—ฒ๐—ฑ ๐˜๐—ผ ๐—น๐—ฒ๐—ฎ๐—ฟ๐—ป ๐—ณ๐—ฟ๐—ผ๐—บ ๐—ณ๐—ฎ๐—ถ๐—น๐˜‚๐—ฟ๐—ฒ โ€” ๐—ฎ๐—ณ๐˜๐—ฒ๐—ฟ ๐˜„๐—ฎ๐˜๐—ฐ๐—ต๐—ถ๐—ป๐—ด ๐—ต๐˜‚๐—บ๐—ฎ๐—ป๐˜€. Most robot learning systems assume failure is the end of learning. In our new work, we study whether robots can improve after deployment by learning from their own failures, without any human intervention, teleoperation, or corrective labels. The key idea is simple: human videos contain structure about how the world works. We use them to learn cross-embodiment representations of action, dynamics, and value, enabling a shared predictive space between human behavior and robot experience. This allows a new learning loop: ๐Ÿ‘‰ pretrain on human videos ๐Ÿ‘‰ deploy robot policy ๐Ÿ‘‰ observe failures ๐Ÿ‘‰ reinterpret failures using human priors ๐Ÿ‘‰ improve autonomously We evaluate this across 7 real-world manipulation tasks, showing: ๐Ÿ“ˆ 40% โ†’ 81% success rate ๐Ÿ† Strong improvements over ฯ€0.6 RECAP and RISE โœ”๏ธ Zero human intervention during post-deployment improvement ๐Ÿงฌ Generalizes across robot embodiments and policy backbones A key finding is that explicit failure repair significantly outperforms failure reweighting, yielding substantially larger gains under identical data conditions (+25 pts vs +5 pts on the same ฯ€0.5 base policy). Overall, the results suggest a shift in how we think about robot learning: Human videos are not only for pretraining policies. They can provide the structure needed for continual self-improvement after deployment. ๐Ÿ“„ Paper: ๐ŸŒ Project: I am grateful for working with the fantastic leads Hanzhi Chen and Anran Zhang, and our collaborators Simon Schaefer, Kejia Chen, Shi Chen, Daniel Cremers. Special thanks to Stefan Leutenegger for co-advising this project with me. ETH Zรผrich TU Mรผnchen Microsoft Check out Hanzhi's ๐Ÿงต for more details

Oier Mees

12,277 views โ€ข 1 month ago

Yes they did. Btw here is what a .30-06 does do a resin skull encased in ballistic gel. Charlie Kirk was murdered by professional hitmen and the FBI blamed a patsy. Candace please discuss some of these inconsistencies. Amateur Shooter Skill Discrepancy: Official narrative claims Tyler Robinson, with no professional training, executed a precise 142-yard neck shot using a vintage Mauser 98 rifle. Ballistics data shows a miniscule success rate for novices. He had zero practice records or ammo traces. single-shot perfection is impossible without expert help. Invisible Rooftop Presence and Firing: No eyewitnesses saw Robinson climbing, positioning, or shooting despite ~3,000 attendees and visible rooftop in daylight. Witness statements point to conflicting shot directions. The probability of zero sightings 2 minutes Video shows fleeing figure, but actions exceed feasible time for amateur. Post-Escape Behavior: Calm demeanor post-shot he was at Dairy Queen posting Discord memes. Only arrested after dad's tip-off. His "Confession" text to vanished lover, unreleased geo-tracking data, amateur would show panic, not casual behavior. Audio Discrepancy across Multiple Sources: Videos show varying shot sounds (immediate crack in close views, delays/echoes in others). Physics predicts consistent crack-bang from a single rooftop. Variations imply closer/suppressed weapon or multiple shooters. Trajectory and Wound Mismatches: Blood spray arcs forward/left, not downward from elevated shot. No exit despite high velocity. Some Witnesses report "two shots." Evidence Tampering and Scene Handling: Stage/cameras dismantled ~5-10 mins post-shot. FBI delayed ~16 mins and no autopsy released. Videos show staff yanking SD cards, there are gag orders on witnesses and the scene was trampled by people who were supposed to be professionals. Pre-Event Foreknowledge Indicators: Google searches for "Tyler Robinson" spiked pre-shot. Hand signals in videos, scrubbed CIA-linked professors. Suggests coordination/setup.

Nacho Business

336,811 views โ€ข 9 months ago

Furniture assembly is the task everyone name-drops and nobody actually attempts at real scale. Every demo I have seen is a scaled down IKEA leg or a single arm on a toy chair. This paper does it properly, real scale, bimanual, up to 7 subtasks and 1,550 control steps per episode, and it is validated on a real Kinova Gen3, not just in sim. That real-robot number is the one that matters: only a 16 percent drop on the hardest task going from simulation to hardware. That is a small enough gap to take seriously, and it did not happen by accident. They built a VR teleoperation rig specifically for coordinated dual-arm collection, because generic single-arm teleop setups do not capture the coordination real assembly needs, and the model predicts a continuous progress signal alongside the action chunk rather than a discrete subtask label, letting it auto-transition and catch drift before it compounds into total failure. The simulation ablation is what got them there, 48 to 80 percent over baselines, with another 21 points from their perception and control design study alone, but that is groundwork, not the headline. Watch the video, there is a clip of the robot misgrasping the seat panel, reopening the gripper, and regrasping on its own. That is not scripted recovery behaviour, it emerged from training, and it emerged on hardware. Excellent work from the team from Mitsubishi Electric Research Laboratories, with Oxford and UNC Chapel Hill Clinical Laboratory Science. Video and project page in comments. #Robotics #Manipulation #VLA

Stephen James

14,952 views โ€ข 22 days ago

Wonderland: Navigating 3D Scenes from a Single Image Contributions: โ€ข First, we introduce a representation for controllable 3D generation by leveraging the generative priors from camera-guided video diffusion models. Unlike image models, video diffusion models are trained on extensive video datasets. This enables them to capture comprehensive spatial relationships within scenes across multiple views and embed a form of "3D awareness" in their latent space, which allows us to maintain 3D consistency in novel view synthesis. โ€ข Second, to achieve controllable novel view generation, we empower video models with precise control over specified camera motions. We introduce a novel dual-branch conditioning mechanism that effectively incorporates desired diverse camera trajectories into the video diffusion model. This enables expansion of a single image into a multi-view consistent capture of a 3D scene with precise pose control. โ€ข Third, to achieve efficient 3D reconstruction, we directly transform video latents into 3DGS. We propose a novel latent-based large reconstruction model (LaLRM) that lifts video latents to 3D in a feed-forward manner. With this design, during inference, our model directly predicts 3DGS from a single input image, effectively aligning the generation and reconstruction tasksโ€”and bridging image space and 3D spaceโ€”through the video latent space. Compared with reconstructing scenes from images, the video latent space offers a 256ร— spatial-temporal reduction while retaining essential and consistent 3D structural details. Such a high degree of compression is crucial, as it allows the LaLRM to handle a wider range of 3D scenes within the reconstruction framework, with the same memory constraints.

MrNeRF

52,849 views โ€ข 1 year ago

๐Ÿš€ Introducing EgoExo Forge - built on top of Rerun, Gradio, and Hugging Face hub (Iโ€™ll be in San Francisco July 21โ€“29 โ€” if youโ€™re into robotics, egocentric AI, large-scale data collection, or just want to chat, DM me!) In my opinion, large-scale, diverse, and high-quality data is still the largest bottleneck for generalized robotics deployment. I believe that some version of imitation learning from human examples will be the most scalable + clean way to train humanoid robots ๐Ÿค– (similar to what Tesla did for Full Self Driving). Teleop is too expensive to collect a large enough dataset in a reasonable manner, so passive collection via egocentric (and in certain cases, exocentric) views feels like the right bet. Over the past few months, I've been trying to build out the scaffolding for this and using Rerun as my underlying infrastructure. Data being collected needs to be easily inspectable + time series and rerun provides the right tooling for this. My goal is to first build out a ground truth representative dataset from already existing open source data, generate some reasonable baselines, and then go out and collect my own data that adheres to the defined schema. ๐Ÿ” Starting with open-source datasets 1. EgoDex from Apple 2. HOCap from Nvidia and the University of Texas at Dallas 3. Assembly101 from Meta All these different datasets have different sensor configurations + annotations, so my goal with egoexo-forge is to have one consistent labeling scheme + data layout. I built a data pipeline that aligns all of the different datasets in one general schema assuming the COCO133 keypoint layout that allows for exo+ego, ego only, or exo only Since the scaffolding is already there, it becomes MUCH easier to add other datasets. So the next ones that I'll be including are HD-EPIC kitchens dataset, HOT3D, and finally my own personal iPhone + insta360 go collection method. Once I have a diverse variety of datasets, I'll double down on what I believe to be the key algorithms required to make useful data for imitation learning ๐Ÿ“Š 1. Camera Pose estimation via SLAM/SFM for ego perspective (and automatic calibration for exo) 2. Human pose estimation for both egocentric + exocentric views 3. Metric 3D reconstruction + object tracking I'll be setting up reasonable open-source baselines for each of these to validate that these datasets work, and then finally try to use the generated datasets for some imitation learning via the pi0-lerobot repo I've been working on. I plan on making a blog post + providing more info on all of this in the near future so stay tuned

Pablo Vela

32,085 views โ€ข 1 year ago

Model-Free Reinforcement Learning (MFRL) has been alluring, especially with supercharged compute with physics on GPU. However, the methods use 0-th order gradients, and are often not the best optimizers. Can we do better than PPO in continuous control for robotics? Turns out yes! ๐Ÿฅณ tl;dr: Faster, better RL than PPO in continuous control ๐Ÿ’ช The answer lies in using more information from the simulation. We are juicing the simulation on GPU as it is, why not use it for gradients as well? This has been a driving question in a series of our works. We first studied this problem in ICLR 2022 paper on Short Horizon Actor Critic Naive gradient based methods are stuck in local minima and have exploding/vanishing gradients. SHAC solved this problem truncated rollouts and model based value estimation, where the model is Differentiable Sim. This boosted sample efficiency and wall-clock time immensely especially in high dimensional systems such as humanoids Yet, given enough compute PPO often caught up. Our follow up paper on on Adaptive Horizon Actor Critic at ICML 2024 discovers the cause and provides a fix. However, we find that even when given ground-truth dynamics, not all gradients are useful due to sample error. 1st-Order Model-Based Reinforcement Learning methods employing differentiable simulation provide gradients with reduced variance but are susceptible to bias in scenarios involving stiff dynamics, such as physical contact. We find that back-propagating through contact and long trajectories drastically reduces gradient accuracy. Using this insight, we propose AHAC to dynamically adapt its roll-out horizon to avoid differentiating through stiff contact. AHAC is a first-order model-based RL algorithm that learns high-dimensional tasks in minutes (wall clock) and outperforms PPO by 40%, even in the limit of data provided to PPO. This work is led by Ignat Georgiev alongside Krishnan Srinivasan, Jie Xu, Eric Heiden and ample assistance from warp team at NVIDIA Robotics (Miles Macklin)

Animesh Garg

52,300 views โ€ข 2 years ago

You can't 3D reconstruct glass from images... ...WRONG! Thanks for video diffusion, now just about anything is possible! Introducing...Diffusion Knows Transparency (DKT) Transparent and reflective objects usually break robot vision and photogrammetry pipelines because they don't follow the "solid object" rules standard cameras expect. DKT is a new AI model that repurposes the "internal physics engine" found in video generation models to solve this problem. Researchers took a massive video diffusion model (WAN) and fine-tuned it using a custom-built synthetic dataset to turn it into a high-precision depth sensor. To train the AI, they built the first massive synthetic video library of transparent objects, 1.32 million frames of perfectly labeled glass and metal objects in motion. Without ever seeing a "real" labeled video of glass during training, the model (DKT) outperformed all previous specialized systems on real-world benchmarks (ClearPose, DREDS). They created a "lightweight" 1.3B parameter version that runs fast enough (0.17s per frame) to be used on actual robot hardware. Two reasons I find this project important: 1. It further proves that synthetic data will be essential for training the next generation vision models. 2. In real-world robotic tests, using DKT's depth maps nearly doubled the success rate of robot arms trying to pick up objects on tricky reflective or translucent surfaces. At home robots will need to interact with these types of objects on a daily basis. Check out the project page here: Code is LIVE! #Computervision #Robotics #AI

Jonathan Stephens

17,712 views โ€ข 6 months ago

A Letter to Our Community: The Road Ahead for Robotics To our Community and Partners, As we step into 2026, our mission at Axis is clearer than ever: Constructing the definitive End-to-End Scaling Layer for Robotics. Our goal is to accelerate the transfer of diverse human intelligence into Robotics General Intelligence (RGI). By owning the critical path of intelligence creation, we are turning the physical limitations of robotics into a scalable, software-driven future. Here is our strategic outlook and roadmap for the year ahead. The Core Thesis: Simulation is the Only Way Out The path to RGI is currently blocked by Data Scarcity, Generalization Fragility, and Hardware Fragmentation. At Axis, we believe Simulation is the only way out. Our Simulation Data Platform and Data Augmentation Engine transform raw data into "Synthetic Gold". Backed by academic milestones like Roboverse, Skill Blending, and GraspVLA, we have proven that pure simulation can achieve the generalization required for the real world. We donโ€™t just collect data; we architect it. The Engine: Why Crypto? We believe RGI should come from all, not a few. Crypto is not just a feature; it is the primitive that powers our entire ecosystem flywheel: - Incentive Mechanism: Democratizing contribution and rewarding the trainers and developers. - Assetization: Turning proprietary data and refined models into liquid, ownable assets. - Verifiable Workflow: We are opening the "Black Box" of AI. By bringing total transparency to the Task Generation โ†’ Data Collection โ†’ Model Training pipeline, we ensure every byte of intelligence is verifiable, traceable, and secure. 2026 Strategic Deliverables This year, we are committed to delivering three foundational pillars: - The World's Largest Training Dataset for Robots: A robot training setโ€”diverse, high-quality interaction data at an unprecedented scale. - A Robotics Foundation Model: A universal robotic brain trained on our pure simulation and synthetic data, capable of robust cross-embodiment transfer and open-world adaptability. - Evolvable Robot Hardware: Robots deployed with Axis models that autonomously evolve through continuous interaction, turning every deployment into a self-improving node within our RGI network. The Ultimate Vision We are building more than models; we are architecting the Distributed Machine Economy. A future where every dataset, model, and robotic embodiment is a verifiable asset in a global, autonomous network. Thank you for building the future of intelligence with usโœŒ๏ธ๐Ÿ“ท

Axis Robotics

27,858 views โ€ข 6 months ago