Загрузка видео...

Не удалось загрузить видео

Возникла проблема при загрузке этого видео. Это может быть связано с временными проблемами сети или видео может быть недоступно.

На главную

NVIDIA DROPPED A MOTION DIFFUSION MODEL FOR HUMANOID ROBOTS trained on 700 hours of mocap data kimodo generates high-quality 3D human and robot motions from text prompts you control it with: → full-body pose keyframes → end-effector positions/rotations → 2D paths and waypoints works on human skeletons and unitree... show more

Vaishnavi

20,812 subscribers

17,572 просмотров • 3 месяцев назад •via X (Twitter)

Anya Rossi• Live Now

Private livecam show

Комментарии: 0

Нет доступных комментариев

Здесь появятся комментарии из оригинального поста

Похожие видео

We’ve seen humanoid robots walk around for a while, but when will they actually help with useful tasks in daily life? The challenge here is the diversity and complexity of real-world scenes. Our new work tackles this problem via 3D visuomotor policy learning. Using data from only 1 scene, our Improved 3D Diffusion Policy (iDP3) enables a full-sized humanoid robot to autonomously pick&place objects, pour water, and wipe tables, in the wild open world. (and all these skills are useful, right?) Web: Fully open-sourced code:

We’ve seen humanoid robots walk around for a while, but when will they actually help with useful tasks in daily life? The challenge here is the diversity and complexity of real-world scenes. Our new work tackles this problem via 3D visuomotor policy learning. Using data from only 1 scene, our Improved 3D Diffusion Policy (iDP3) enables a full-sized humanoid robot to autonomously pick&place objects, pour water, and wipe tables, in the wild open world. (and all these skills are useful, right?) Web: Fully open-sourced code:

Yanjie Ze

75,248 просмотров • 1 год назад

Imagine having a ping pong robot! 🏓 Researchers and developers building physical AI: meet Reachy 2 from Pollen Robotics, an open-source, humanoid robot for real-world experimentation. It’s a bimanual mobile manipulator: each 7-DOF arm mimics human proportions and can lift up to 3 kg, giving dexterity for object handling. It can be controlled with Python and ROS2 Humble, or go straight into VR teleoperation, use a headset to move Reachy’s arms, hands, and head, and see through its cameras as if you’re in the robot’s own body. Want it to move around? A mobile base with three omnidirectional wheels, rich sensors, and LiDAR lets Reachy 2 navigate and explore its surroundings smoothly. 🗺️ Under the hood, it’s powered by a CPU system that’s ready for machine learning, perfect for loading AI frameworks and testing new models from Hugging Face directly on the robot. Keep making robots more, and more accessible Pollen team! ... and keep making more open source models to make robots more mainstream clem 🤗!

Imagine having a ping pong robot! 🏓 Researchers and developers building physical AI: meet Reachy 2 from Pollen Robotics, an open-source, humanoid robot for real-world experimentation. It’s a bimanual mobile manipulator: each 7-DOF arm mimics human proportions and can lift up to 3 kg, giving dexterity for object handling. It can be controlled with Python and ROS2 Humble, or go straight into VR teleoperation, use a headset to move Reachy’s arms, hands, and head, and see through its cameras as if you’re in the robot’s own body. Want it to move around? A mobile base with three omnidirectional wheels, rich sensors, and LiDAR lets Reachy 2 navigate and explore its surroundings smoothly. 🗺️ Under the hood, it’s powered by a CPU system that’s ready for machine learning, perfect for loading AI frameworks and testing new models from Hugging Face directly on the robot. Keep making robots more, and more accessible Pollen team! ... and keep making more open source models to make robots more mainstream clem 🤗!

Lukas Ziegler

37,221 просмотров • 11 месяцев назад

$BOOM! Humanoid Robots Just Performed Surgery for the First Time! REAL VIDEO! In a groundbreaking preclinical breakthrough, researchers at UC San Diego have achieved what many thought was years away: teleoperated humanoid robots successfully completing live surgeries. Published in Nature, the study marks the world’s first use of humanoid robots for in-vivo laparoscopic procedures on large animals (pigs). Two separate surgeries were completed: Key Details •. Procedure: Laparoscopic gallbladder removal (cholecystectomy) •. Team 1: Human surgeon + one humanoid robot (the robot performed core tasks while the human assisted) •. Team 2: Two humanoid robots working together with no human at the operating table •. Robots: Custom “Surgie” humanoids (~5 ft tall, ~60 lbs) using standard surgical tools •. Control: Fully teleoperated by surgeons (remote human control, not autonomous) •. Significance: First demonstration of humanoid robots handling real surgical workflows in a live setting, proving compatibility with existing OR tools and spaces This proof shows humanoid robots could one day help address surgeon shortages, enable remote procedures in rural areas, battlefields, or even space all at a fraction of the cost and space of traditional surgical robots like da Vinci. Read the full publication here: Project page with video: The future of surgery just got a whole lot more interesting. And medical cost for the first time in decades will be scheduled to go down, much further down.$

BOOM! Humanoid Robots Just Performed Surgery for the First Time! REAL VIDEO! In a groundbreaking preclinical breakthrough, researchers at UC San Diego have achieved what many thought was years away: teleoperated humanoid robots successfully completing live surgeries. Published in Nature, the study marks the world’s first use of humanoid robots for in-vivo laparoscopic procedures on large animals (pigs). Two separate surgeries were completed: Key Details •. Procedure: Laparoscopic gallbladder removal (cholecystectomy) •. Team 1: Human surgeon + one humanoid robot (the robot performed core tasks while the human assisted) •. Team 2: Two humanoid robots working together with no human at the operating table •. Robots: Custom “Surgie” humanoids (~5 ft tall, ~60 lbs) using standard surgical tools •. Control: Fully teleoperated by surgeons (remote human control, not autonomous) •. Significance: First demonstration of humanoid robots handling real surgical workflows in a live setting, proving compatibility with existing OR tools and spaces This proof shows humanoid robots could one day help address surgeon shortages, enable remote procedures in rural areas, battlefields, or even space all at a fraction of the cost and space of traditional surgical robots like da Vinci. Read the full publication here: Project page with video: The future of surgery just got a whole lot more interesting. And medical cost for the first time in decades will be scheduled to go down, much further down.

Brian Roemmele

107,357 просмотров • 13 дней назад

Most robots still need markers, checkerboards, or long calibration rituals just to know where their arms are. Now it works from raw images in seconds. roboreg is a markerless multi arm localization toolkit that plugs into ROS 2 and RViz. No special hardware. No custom setup. You toggle between robot descriptions and the system figures out the rest. The idea is simple: ✅ Hand eye calibration from plain RGB or RGB D images ✅ Only three robot poses needed for millimeter accuracy ✅ Works with any ROS 2 compatible robot and camera ✅ Fully open source under Apache 2.0 It is powered by Hydra, a new marker free ICP variant that converges far more reliably than classical baselines and runs in under a second. If you want to try it: roboreg: ROS 2 roboreg: Hydra paper: pip install roboreg More details and discussion on Open Robotics Discourse:

Most robots still need markers, checkerboards, or long calibration rituals just to know where their arms are. Now it works from raw images in seconds. roboreg is a markerless multi arm localization toolkit that plugs into ROS 2 and RViz. No special hardware. No custom setup. You toggle between robot descriptions and the system figures out the rest. The idea is simple: ✅ Hand eye calibration from plain RGB or RGB D images ✅ Only three robot poses needed for millimeter accuracy ✅ Works with any ROS 2 compatible robot and camera ✅ Fully open source under Apache 2.0 It is powered by Hydra, a new marker free ICP variant that converges far more reliably than classical baselines and runs in under a second. If you want to try it: roboreg: ROS 2 roboreg: Hydra paper: pip install roboreg More details and discussion on Open Robotics Discourse:

Ilir Aliu

18,406 просмотров • 8 месяцев назад

This work makes a humanoid robot do simple parkour moves by looking with a depth camera and choosing the right move on the fly. The big deal is that it turns lots of small human moves into long, real-time robot behavior, without hand-coding every transition or retraining for each new course. A humanoid robot is usually good at steady walking, but it often fails when it has to do fast moves like jumping up, vaulting, or rolling, and then keep going to the next obstacle. The hard part is that you cannot easily collect training data for every possible obstacle shape, distance, and mistake, so robots end up learning a few moves that only work in a narrow setup. This work starts from short clips of real human parkour moves, like stepping over, vaulting, climbing, and rolling. It uses motion matching, which is basically a smart “pick the next clip that fits best right now” search, to stitch those short clips into a long, smooth plan that looks like a human doing a whole course. Then it trains a controller with reinforcement learning (RL), which means the robot learns by trial and error to copy that plan while staying balanced and not falling. After training separate expert controllers for different moves, it compresses them into 1 controller that uses only onboard depth sensing and a simple “go this fast in this direction” command. In real tests on a Unitree G1 humanoid, it can clear multiple obstacles in a row, adapt when obstacles get moved, and climb a wall up to 1.25m.

This work makes a humanoid robot do simple parkour moves by looking with a depth camera and choosing the right move on the fly. The big deal is that it turns lots of small human moves into long, real-time robot behavior, without hand-coding every transition or retraining for each new course. A humanoid robot is usually good at steady walking, but it often fails when it has to do fast moves like jumping up, vaulting, or rolling, and then keep going to the next obstacle. The hard part is that you cannot easily collect training data for every possible obstacle shape, distance, and mistake, so robots end up learning a few moves that only work in a narrow setup. This work starts from short clips of real human parkour moves, like stepping over, vaulting, climbing, and rolling. It uses motion matching, which is basically a smart “pick the next clip that fits best right now” search, to stitch those short clips into a long, smooth plan that looks like a human doing a whole course. Then it trains a controller with reinforcement learning (RL), which means the robot learns by trial and error to copy that plan while staying balanced and not falling. After training separate expert controllers for different moves, it compresses them into 1 controller that uses only onboard depth sensing and a simple “go this fast in this direction” command. In real tests on a Unitree G1 humanoid, it can clear multiple obstacles in a row, adapt when obstacles get moved, and climb a wall up to 1.25m.

Rohan Paul

37,121 просмотров • 5 месяцев назад

🦿Xpeng showed a humanoid robot called IRON whose movement looked so human that the team literally cut it open on stage to prove it is a machine. IRON uses a bionic body with a flexible spine, synthetic muscles, and soft skin so joints and torso can twist smoothly like a person. The system has 82 degrees of freedom in total with 22 in each hand for fine finger control. Compute runs on 3 custom AI chips rated at 2,250 TOPS (Tera Operations Per Second), which is far above typical laptop neural accelerators, so it can handle vision and motion planning on the robot. The AI stack focuses on turning camera input directly into body movement without routing through text, which reduces lag and makes the gait look natural. Xpeng staged the cut-open demo at AI Day in Guangzhou this week, addressing rumors that a performer was inside by exposing internal actuators, wiring, and cooling. Company materials also mention a large physical-world model and a multi-brain control setup for dialogue, perception, and locomotion, hinting at a path from stage demos to service work. Production is targeted for 2026, so near-term tasks will be limited, but the hardware shows a serious step toward human-scale manipulation.

🦿Xpeng showed a humanoid robot called IRON whose movement looked so human that the team literally cut it open on stage to prove it is a machine. IRON uses a bionic body with a flexible spine, synthetic muscles, and soft skin so joints and torso can twist smoothly like a person. The system has 82 degrees of freedom in total with 22 in each hand for fine finger control. Compute runs on 3 custom AI chips rated at 2,250 TOPS (Tera Operations Per Second), which is far above typical laptop neural accelerators, so it can handle vision and motion planning on the robot. The AI stack focuses on turning camera input directly into body movement without routing through text, which reduces lag and makes the gait look natural. Xpeng staged the cut-open demo at AI Day in Guangzhou this week, addressing rumors that a performer was inside by exposing internal actuators, wiring, and cooling. Company materials also mention a large physical-world model and a multi-brain control setup for dialogue, perception, and locomotion, hinting at a path from stage demos to service work. Production is targeted for 2026, so near-term tasks will be limited, but the hardware shows a serious step toward human-scale manipulation.

Rohan Paul

3,802,402 просмотров • 8 месяцев назад

🚨BREAKING Unitree's IPO application on the Science and Technology Innovation Board has been formally accepted by the Shanghai Stock Exchange, with a planned fundraising of 4.202 billion yuan. Of this, the intelligent robot model R&D project accounts for the majority, amounting to 2 billion yuan. In other words: For Unitree, hardware is no longer the problem,the real bottleneck that urgently needs to be overcome is the development of intelligent robot models. Notably, Unitree reports that it has sold over 5,500 humanoid robots,ranking first globally,making them the growth engine of its robotics business.

🚨BREAKING Unitree's IPO application on the Science and Technology Innovation Board has been formally accepted by the Shanghai Stock Exchange, with a planned fundraising of 4.202 billion yuan. Of this, the intelligent robot model R&D project accounts for the majority, amounting to 2 billion yuan. In other words: For Unitree, hardware is no longer the problem,the real bottleneck that urgently needs to be overcome is the development of intelligent robot models. Notably, Unitree reports that it has sold over 5,500 humanoid robots,ranking first globally,making them the growth engine of its robotics business.

CyberRobo

90,612 просмотров • 4 месяцев назад

The future of housework just leaked on GitHub and nobody is talking about it. knox byte just open sourced a framework that coordinates swarms of Unitree G1 humanoid robots to clean your entire house on their own. It's called ARGOS. You tell it "clean the bedroom" in plain English and 2+ G1 robots split the room into zones, sweep in parallel, and sync up for the tasks that need four hands like making the bed or moving furniture. The Claude API decomposes your sentence into a task graph. An auction system makes every robot bid on every task based on distance, battery, and current load. The cheapest robot wins. Cooperative jobs go to the cheapest team. Here's what makes this different from every demo video Boston Dynamics keeps teasing: → 12 cleaning tasks baked in sweeping, mopping, wiping, vacuuming, taking out trash, making the bed, changing sheets, moving furniture, sorting items → 3 policy architectures running underneath OpenVLA-7B for language tasks, Diffusion Policy for floor coverage, ACT for dexterous bimanual work → Train it on your own footage record yourself cleaning, run one command, it extracts poses, builds a LeRobot dataset, and LoRA fine-tunes the policy → PEFA protocol for cooperative work Propose, Execute, Feedback, Adjust. If one robot fails halfway through making the bed, the team replans and retries → Full MuJoCo simulation so you test policies before pushing them to real hardware → Silver and cyan terminal dashboard that shows live fleet status, zone maps, task queues, and battery levels in real time The G1 robots talk to each other over CycloneDDS mesh using Unitree's native SDK. No cloud. No middleware. The whole thing runs on a Jetson Orin inside each robot. The wildest part is the training pipeline. Drop cleaning videos into a folder, run argos train ingest, and the framework does the entire pipeline frame extraction, pose estimation, action labeling, HDF5 dataset, fine-tune, evaluate in sim, deploy to robot. One command per stage. Unitree G1s already exist. The framework to make them clean your house just hit GitHub. 52 stars. MIT License. 100% Opensource.

The future of housework just leaked on GitHub and nobody is talking about it. knox byte just open sourced a framework that coordinates swarms of Unitree G1 humanoid robots to clean your entire house on their own. It's called ARGOS. You tell it "clean the bedroom" in plain English and 2+ G1 robots split the room into zones, sweep in parallel, and sync up for the tasks that need four hands like making the bed or moving furniture. The Claude API decomposes your sentence into a task graph. An auction system makes every robot bid on every task based on distance, battery, and current load. The cheapest robot wins. Cooperative jobs go to the cheapest team. Here's what makes this different from every demo video Boston Dynamics keeps teasing: → 12 cleaning tasks baked in sweeping, mopping, wiping, vacuuming, taking out trash, making the bed, changing sheets, moving furniture, sorting items → 3 policy architectures running underneath OpenVLA-7B for language tasks, Diffusion Policy for floor coverage, ACT for dexterous bimanual work → Train it on your own footage record yourself cleaning, run one command, it extracts poses, builds a LeRobot dataset, and LoRA fine-tunes the policy → PEFA protocol for cooperative work Propose, Execute, Feedback, Adjust. If one robot fails halfway through making the bed, the team replans and retries → Full MuJoCo simulation so you test policies before pushing them to real hardware → Silver and cyan terminal dashboard that shows live fleet status, zone maps, task queues, and battery levels in real time The G1 robots talk to each other over CycloneDDS mesh using Unitree's native SDK. No cloud. No middleware. The whole thing runs on a Jetson Orin inside each robot. The wildest part is the training pipeline. Drop cleaning videos into a folder, run argos train ingest, and the framework does the entire pipeline frame extraction, pose estimation, action labeling, HDF5 dataset, fine-tune, evaluate in sim, deploy to robot. One command per stage. Unitree G1s already exist. The framework to make them clean your house just hit GitHub. 52 stars. MIT License. 100% Opensource.

Guri Singh

27,404 просмотров • 2 месяцев назад

China now has its own “Bolt” — a robot named after sprint legend Usain Bolt. A Chinese research team has unveiled the world’s first full-size humanoid robot to reach a peak speed of 10 meters per second, setting a new global benchmark for humanoid running. Bolt runs like a body pushed to the limit. Its joints and power systems work in tight coordination, keeping it balanced even at sprint speed. Built to match the build of an adult man—1.75 meters tall and 75 kilograms—it is a life-sized system operating at the edge of physics. Compared with Usain Bolt’s iconic 9.58-second 100-meter world record, which many experts believe may stand for decades, the gap between humans and machines is narrowing fast. Chinese robots are now challenging the ceiling of human performance—much as AlphaGo once challenged Go champion Ke Jie. The breakthrough builds on earlier world-record achievements in high-speed robotic running and marks a giant leap for China in humanoid motion and control. Beyond records, Bolt also carries practical value: robots are leaving the lab and stepping into real-world settings—sports training, emergency response, and demanding industrial tasks where speed, balance and control truly matter.

China now has its own “Bolt” — a robot named after sprint legend Usain Bolt. A Chinese research team has unveiled the world’s first full-size humanoid robot to reach a peak speed of 10 meters per second, setting a new global benchmark for humanoid running. Bolt runs like a body pushed to the limit. Its joints and power systems work in tight coordination, keeping it balanced even at sprint speed. Built to match the build of an adult man—1.75 meters tall and 75 kilograms—it is a life-sized system operating at the edge of physics. Compared with Usain Bolt’s iconic 9.58-second 100-meter world record, which many experts believe may stand for decades, the gap between humans and machines is narrowing fast. Chinese robots are now challenging the ceiling of human performance—much as AlphaGo once challenged Go champion Ke Jie. The breakthrough builds on earlier world-record achievements in high-speed robotic running and marks a giant leap for China in humanoid motion and control. Beyond records, Bolt also carries practical value: robots are leaving the lab and stepping into real-world settings—sports training, emergency response, and demanding industrial tasks where speed, balance and control truly matter.

Sinical

111,374 просмотров • 5 месяцев назад

I'll always root for a team that open-sources its best work, and Robbyant just did it properly. Robbyant, Ant Group's embodied-AI company, released LingBot-Vision, a vision foundation model for robots, and the part I love is the data. They trained it on 161M images, filtered down from 2B raw ones and mostly pulled straight from the open web, with no human labels, no edge detectors, no depth sensors anywhere in the loop. It learns the exact edges of objects from raw pixels. That's roughly a tenth of the data DINOv3 saw, and under a third of the training. And it shows in the results. On depth, working out how far away things are, the 1B model edges out a 7B on NYU-Depth. It also powers LingBot-Depth 2.0, which reads the surfaces cameras usually choke on, glass and mirrors, and halves indoor depth error. LingBot-Vision is fully open. Weights from the 1.1B flagship down to a tiny 21M version, code, and the paper. This is the timeline I want more of. Robbyant

I'll always root for a team that open-sources its best work, and Robbyant just did it properly. Robbyant, Ant Group's embodied-AI company, released LingBot-Vision, a vision foundation model for robots, and the part I love is the data. They trained it on 161M images, filtered down from 2B raw ones and mostly pulled straight from the open web, with no human labels, no edge detectors, no depth sensors anywhere in the loop. It learns the exact edges of objects from raw pixels. That's roughly a tenth of the data DINOv3 saw, and under a third of the training. And it shows in the results. On depth, working out how far away things are, the 1B model edges out a 7B on NYU-Depth. It also powers LingBot-Depth 2.0, which reads the surfaces cameras usually choke on, glass and mirrors, and halves indoor depth error. LingBot-Vision is fully open. Weights from the 1.1B flagship down to a tiny 21M version, code, and the paper. This is the timeline I want more of. Robbyant

Chubby♨️

48,249 просмотров • 14 дней назад

Wonderland: Navigating 3D Scenes from a Single Image Contributions: • First, we introduce a representation for controllable 3D generation by leveraging the generative priors from camera-guided video diffusion models. Unlike image models, video diffusion models are trained on extensive video datasets. This enables them to capture comprehensive spatial relationships within scenes across multiple views and embed a form of "3D awareness" in their latent space, which allows us to maintain 3D consistency in novel view synthesis. • Second, to achieve controllable novel view generation, we empower video models with precise control over specified camera motions. We introduce a novel dual-branch conditioning mechanism that effectively incorporates desired diverse camera trajectories into the video diffusion model. This enables expansion of a single image into a multi-view consistent capture of a 3D scene with precise pose control. • Third, to achieve efficient 3D reconstruction, we directly transform video latents into 3DGS. We propose a novel latent-based large reconstruction model (LaLRM) that lifts video latents to 3D in a feed-forward manner. With this design, during inference, our model directly predicts 3DGS from a single input image, effectively aligning the generation and reconstruction tasks—and bridging image space and 3D space—through the video latent space. Compared with reconstructing scenes from images, the video latent space offers a 256× spatial-temporal reduction while retaining essential and consistent 3D structural details. Such a high degree of compression is crucial, as it allows the LaLRM to handle a wider range of 3D scenes within the reconstruction framework, with the same memory constraints.

Wonderland: Navigating 3D Scenes from a Single Image Contributions: • First, we introduce a representation for controllable 3D generation by leveraging the generative priors from camera-guided video diffusion models. Unlike image models, video diffusion models are trained on extensive video datasets. This enables them to capture comprehensive spatial relationships within scenes across multiple views and embed a form of "3D awareness" in their latent space, which allows us to maintain 3D consistency in novel view synthesis. • Second, to achieve controllable novel view generation, we empower video models with precise control over specified camera motions. We introduce a novel dual-branch conditioning mechanism that effectively incorporates desired diverse camera trajectories into the video diffusion model. This enables expansion of a single image into a multi-view consistent capture of a 3D scene with precise pose control. • Third, to achieve efficient 3D reconstruction, we directly transform video latents into 3DGS. We propose a novel latent-based large reconstruction model (LaLRM) that lifts video latents to 3D in a feed-forward manner. With this design, during inference, our model directly predicts 3DGS from a single input image, effectively aligning the generation and reconstruction tasks—and bridging image space and 3D space—through the video latent space. Compared with reconstructing scenes from images, the video latent space offers a 256× spatial-temporal reduction while retaining essential and consistent 3D structural details. Such a high degree of compression is crucial, as it allows the LaLRM to handle a wider range of 3D scenes within the reconstruction framework, with the same memory constraints.

MrNeRF

52,801 просмотров • 1 год назад

90% of "AI developers" just download pre packaged GGUF files from Hugging Face, hit run, and call it a day. The top 10% know how to pull the raw safetensors, run the math, and quantize massive models into Q4_K_M themselves. If you think llama.cpp can only execute models, you’re missing the best part of the open source ecosystem. It’s a high performance optimization suite. Manually stripping 69% of the VRAM footprint off a brand new model architecture is where real infrastructure value is made. If you want to actually master local inference and deploy models like Google’s massive Gemma 4 12B it on consumer NVIDIA hardware using llama.cpp, you need to learn this pipeline. Let's build it. I just took the raw 22.7 GB Gemma 4 baseline and manually compressed it down to a 7.02 GB Q4_K_M GGUF artifact using llama.cpp. That is a 69% reduction in footprint. No quality loss. No VRAM bottlenecks. Just native, hardware accelerated C++ inference running a full 2,50,000 token context window on a dual NVIDIA Tesla T4 setup. Stop melting your VRAM on unoptimized weights and stop relying on other people's pipelines. Own your stack. I mapped this entire architecture from dynamic binary fetching to raw quantization and real time GPU streaming into a single, bulletproof notebook. Notebook link is in the comments below. Bookmark this blueprint for your next deployment and tell me which quantization works best for your workflow and model.

Alok

62,133 просмотров • 12 дней назад

🇨🇳 Another great Chinese Model, OmniHuman-1.5 from ByteDance Turns 1 image plus a voice track into expressive avatar video by pairing a System 1 and System 2 inspired planner with a Diffusion Transformer, Produces coherent motion for over 1 minute with moving camera and multi character scenes. Most avatar models move to the beat of the audio but miss meaning, so gestures feel generic and emotions feel shallow. The fix here is a Multimodal LLM planner that listens to the speech and drafts a structured plan describing intent, emotions, beats, and high level actions, which gives the motion engine clear semantic targets instead of only rhythm. The motion engine is a Multimodal Diffusion Transformer that fuses the plan with audio, the single reference image, and optional text prompts, then synthesizes continuous body, face, and head motion that matches both words and tone. A key trick is a Pseudo Last Frame, a synthetic target that summarizes the next expected state, which stabilizes fusion across modalities and keeps motion consistent over long spans. From just 1 image and speech, the system outputs speaking avatars with synchronized lips, context aware gestures, and continuous camera movement, and it also supports multi character interactions without manual choreography. Reported results show strong lip sync accuracy, high video quality, natural motion, and close match to text prompts, and the same setup works on nonhuman characters too.

Rohan Paul

63,859 просмотров • 10 месяцев назад

🚨 BREAKING: NVIDIA just announced the Isaac GR00T Reference Humanoid Robot. The first fully open humanoid robot reference design built on Jetson Thor, and it's going straight to the world's top research institutions. This is Jensen Huang's bet on open physical AI infrastructure. The hardware stack is serious: → Unitree H2 Plus chassis, 6 feet tall, 150 pounds, 31 degrees of freedom → Sharpa Wave tactile five-finger hands, 22 degrees of freedom, bringing total to 75 across the full body → NVIDIA Jetson AGX Thor onboard compute, 2,070 FP4 teraflops of AI performance, 128GB unified memory → Multi-view sensing, stereo head camera, wrist cameras, IMU Alongside this announcement, Unitree also introduced the H2 Plus as a standalone product, a frontier humanoid combining Unitree's own body, Sharpa's five-finger hands and NVIDIA Robotics Jetson Thor compute into one fully integrated research platform. The full Isaac GR00T software stack ships with it, teleoperation for data capture, open foundation models, Isaac Sim for training, Isaac Lab for evaluation, and accelerated ROS middleware for deployment. The complete loop from data to real-world robot in one unified platform. ETH Zürich, Stanford Robotics Center, UC San Diego and Ai2 are already on board as launch research partners. NVIDIA Robotics did to AI what it's now doing to robotics, build the platform, open the ecosystem, let the world build on top of it. Whoever owns the infrastructure layer wins. NVIDIA knows this better than anyone. 👀 Read more here: ~~ ♻️ Join the weekly robotics newsletter, and never miss any news →

🚨 BREAKING: NVIDIA just announced the Isaac GR00T Reference Humanoid Robot. The first fully open humanoid robot reference design built on Jetson Thor, and it's going straight to the world's top research institutions. This is Jensen Huang's bet on open physical AI infrastructure. The hardware stack is serious: → Unitree H2 Plus chassis, 6 feet tall, 150 pounds, 31 degrees of freedom → Sharpa Wave tactile five-finger hands, 22 degrees of freedom, bringing total to 75 across the full body → NVIDIA Jetson AGX Thor onboard compute, 2,070 FP4 teraflops of AI performance, 128GB unified memory → Multi-view sensing, stereo head camera, wrist cameras, IMU Alongside this announcement, Unitree also introduced the H2 Plus as a standalone product, a frontier humanoid combining Unitree's own body, Sharpa's five-finger hands and NVIDIA Robotics Jetson Thor compute into one fully integrated research platform. The full Isaac GR00T software stack ships with it, teleoperation for data capture, open foundation models, Isaac Sim for training, Isaac Lab for evaluation, and accelerated ROS middleware for deployment. The complete loop from data to real-world robot in one unified platform. ETH Zürich, Stanford Robotics Center, UC San Diego and Ai2 are already on board as launch research partners. NVIDIA Robotics did to AI what it's now doing to robotics, build the platform, open the ecosystem, let the world build on top of it. Whoever owns the infrastructure layer wins. NVIDIA knows this better than anyone. 👀 Read more here: ~~ ♻️ Join the weekly robotics newsletter, and never miss any news →

Lukas Ziegler

16,062 просмотров • 1 месяц назад

You can't 3D reconstruct glass from images... ...WRONG! Thanks for video diffusion, now just about anything is possible! Introducing...Diffusion Knows Transparency (DKT) Transparent and reflective objects usually break robot vision and photogrammetry pipelines because they don't follow the "solid object" rules standard cameras expect. DKT is a new AI model that repurposes the "internal physics engine" found in video generation models to solve this problem. Researchers took a massive video diffusion model (WAN) and fine-tuned it using a custom-built synthetic dataset to turn it into a high-precision depth sensor. To train the AI, they built the first massive synthetic video library of transparent objects, 1.32 million frames of perfectly labeled glass and metal objects in motion. Without ever seeing a "real" labeled video of glass during training, the model (DKT) outperformed all previous specialized systems on real-world benchmarks (ClearPose, DREDS). They created a "lightweight" 1.3B parameter version that runs fast enough (0.17s per frame) to be used on actual robot hardware. Two reasons I find this project important: 1. It further proves that synthetic data will be essential for training the next generation vision models. 2. In real-world robotic tests, using DKT's depth maps nearly doubled the success rate of robot arms trying to pick up objects on tricky reflective or translucent surfaces. At home robots will need to interact with these types of objects on a daily basis. Check out the project page here: Code is LIVE! #Computervision #Robotics #AI

You can't 3D reconstruct glass from images... ...WRONG! Thanks for video diffusion, now just about anything is possible! Introducing...Diffusion Knows Transparency (DKT) Transparent and reflective objects usually break robot vision and photogrammetry pipelines because they don't follow the "solid object" rules standard cameras expect. DKT is a new AI model that repurposes the "internal physics engine" found in video generation models to solve this problem. Researchers took a massive video diffusion model (WAN) and fine-tuned it using a custom-built synthetic dataset to turn it into a high-precision depth sensor. To train the AI, they built the first massive synthetic video library of transparent objects, 1.32 million frames of perfectly labeled glass and metal objects in motion. Without ever seeing a "real" labeled video of glass during training, the model (DKT) outperformed all previous specialized systems on real-world benchmarks (ClearPose, DREDS). They created a "lightweight" 1.3B parameter version that runs fast enough (0.17s per frame) to be used on actual robot hardware. Two reasons I find this project important: 1. It further proves that synthetic data will be essential for training the next generation vision models. 2. In real-world robotic tests, using DKT's depth maps nearly doubled the success rate of robot arms trying to pick up objects on tricky reflective or translucent surfaces. At home robots will need to interact with these types of objects on a daily basis. Check out the project page here: Code is LIVE! #Computervision #Robotics #AI

Jonathan Stephens

17,712 просмотров • 6 месяцев назад

OpenClaw, but built for normal people. Sim is an open-source platform that lets you build AI agent workflows on a drag-and-drop canvas. Connect them to channels like Telegram and WhatsApp and deploy without writing a single line of code. They also have a built-in Copilot that generates entire workflows from plain English, which you can then tweak and customize in the UI. Key features: - Free and open-source (Apache 2.0) - Vector store integration for RAG-grounded agents - Self-host with one command (`npx simstudio`) - Run fully local with Ollama, no API keys needed - Supports vLLM for production-grade self-hosted inference The thing I really like about Sim is the level of control you get. You can add conditional branching, parallel execution, human-in-the-loop approval gates, and even nest workflows inside other workflows. Everything is visible on the canvas, so you know exactly what your agent is doing at every step. And you can build a workflow in Sim, deploy it as an MCP server, and plug it into any agent, including OpenClaw. I've shared the link to Sim's GitHub repo in the next tweet.

OpenClaw, but built for normal people. Sim is an open-source platform that lets you build AI agent workflows on a drag-and-drop canvas. Connect them to channels like Telegram and WhatsApp and deploy without writing a single line of code. They also have a built-in Copilot that generates entire workflows from plain English, which you can then tweak and customize in the UI. Key features: - Free and open-source (Apache 2.0) - Vector store integration for RAG-grounded agents - Self-host with one command (`npx simstudio`) - Run fully local with Ollama, no API keys needed - Supports vLLM for production-grade self-hosted inference The thing I really like about Sim is the level of control you get. You can add conditional branching, parallel execution, human-in-the-loop approval gates, and even nest workflows inside other workflows. Everything is visible on the canvas, so you know exactly what your agent is doing at every step. And you can build a workflow in Sim, deploy it as an MCP server, and plug it into any agent, including OpenClaw. I've shared the link to Sim's GitHub repo in the next tweet.

Akshay 🚀

52,426 просмотров • 4 месяцев назад

I've been working a lot with SAM3 and the Momentum Human Rig (MHR). I finally integrated it into the data I'm working with Rerun. The progression I've taken looks as follows SAM3 + SAM3D-body on 1. a single image 2. a set of multiple images 3. a single video 4. A multiview video capture I took inspiration from the SAM3D-body paper and built a multiview fitting optimization pipeline. This pipeline involves using the 2D keypoints from the single-view pipeline, triangulating them, and employing an L1 loss between the 2D/3D keypoints. The temporal stability isn't great, so that's the next portion I'm going to focus on. One really frustrating thing about SAM3D-body is the lack of per-joint confidence values. It makes it harder to deal with occlusions. I'm probably going to need to use a separate model, or maybe add a confidence head.

I've been working a lot with SAM3 and the Momentum Human Rig (MHR). I finally integrated it into the data I'm working with Rerun. The progression I've taken looks as follows SAM3 + SAM3D-body on 1. a single image 2. a set of multiple images 3. a single video 4. A multiview video capture I took inspiration from the SAM3D-body paper and built a multiview fitting optimization pipeline. This pipeline involves using the 2D keypoints from the single-view pipeline, triangulating them, and employing an L1 loss between the 2D/3D keypoints. The temporal stability isn't great, so that's the next portion I'm going to focus on. One really frustrating thing about SAM3D-body is the lack of per-joint confidence values. It makes it harder to deal with occlusions. I'm probably going to need to use a separate model, or maybe add a confidence head.

Pablo Vela

42,267 просмотров • 6 месяцев назад

Force feedback demo Force feedback is when joystick is pushing on your hand when something is pushing on the robot arm. Feeling the force - so much helpful to control the robot, that done well it allows you to do tasks even without visual feed. You can make an experiment: close your eyes - you can easily get the headphones out of the case. Also, visual information is often not enough. For example, you're trying to pull out a usb connector, but you pull it at the wrong angle, causing it to get stuck. Visually, nothing changes, but the pressure is intense and you can break the connector. Surgical robots have been using force feedback for years, and there are also 3D styluses which use this feature, proving that the technology works and is useful. But in modern robots with AI, it's hardly ever implemented. Although it's useful for both teleoperation and AI model. That's one of the reasons why we are building our robotic arms starting with off the shelf motors rather than taking the whole off the shelf arm. There are still a range of easy wins that can be made iterating robot hardware.

Force feedback demo Force feedback is when joystick is pushing on your hand when something is pushing on the robot arm. Feeling the force - so much helpful to control the robot, that done well it allows you to do tasks even without visual feed. You can make an experiment: close your eyes - you can easily get the headphones out of the case. Also, visual information is often not enough. For example, you're trying to pull out a usb connector, but you pull it at the wrong angle, causing it to get stuck. Visually, nothing changes, but the pressure is intense and you can break the connector. Surgical robots have been using force feedback for years, and there are also 3D styluses which use this feature, proving that the technology works and is useful. But in modern robots with AI, it's hardly ever implemented. Although it's useful for both teleoperation and AI model. That's one of the reasons why we are building our robotic arms starting with off the shelf motors rather than taking the whole off the shelf arm. There are still a range of easy wins that can be made iterating robot hardware.

Igor Kulakov

18,773 просмотров • 1 год назад

It's 2030 and you are reviewing humanoid robots. A Tesla. A Google. An Apple. An OpenAI. A Meta. A Figure. And a bunch of Chinese-made ones. Which one is best, and why? I think the Tesla understands the world much better. Why? There were eight Teslas around me on the freeway today. Start there. No other robot company has that data. But my robot is parked at the local high school twice a day. Its cameras see humans in all of our weirdness. How we move. Where we go. Where we walk. Who we talk with. What you are wearing. Whether your hair was combed this morning. That data will lead to robotics breakthroughs. Apple might keep up with its Vision Pro data, but it is too freaked out by the privacy implications of using said data. (On the front are six cameras and a couple of TOF -- Time Of Flight -- sensors that can see everything in your home in great detail). Google has a lot of data, for sure. All my: 1. Email. 2. Calendars. 3. Photos. 4. TV watching behavior. 5. Contacts. 6. Documents and spreadsheets. 7. Files. 8. Location data. So I expect Google's robot will be attractive to many. But how do you see the others shake out over the next five years? Make some guesses. But remember what an AI pioneer told me years ago about AI: it's all about the data. The Chinese ones have huge advantages: the Chinese have more data on their citizens, and many more citizens to boot AND they can make robots cheaper than we can. But now that you know OpenAI is building its own robot you have caught wind of what I've heard from many in San Francisco and Silicon Valley: that humanoid robots are the real prize of AI and will be highly profitable for those that can make them and find customers willing to buy them. Here, too, I learned long ago never to bet against Elon Musk. Will you?

It's 2030 and you are reviewing humanoid robots. A Tesla. A Google. An Apple. An OpenAI. A Meta. A Figure. And a bunch of Chinese-made ones. Which one is best, and why? I think the Tesla understands the world much better. Why? There were eight Teslas around me on the freeway today. Start there. No other robot company has that data. But my robot is parked at the local high school twice a day. Its cameras see humans in all of our weirdness. How we move. Where we go. Where we walk. Who we talk with. What you are wearing. Whether your hair was combed this morning. That data will lead to robotics breakthroughs. Apple might keep up with its Vision Pro data, but it is too freaked out by the privacy implications of using said data. (On the front are six cameras and a couple of TOF -- Time Of Flight -- sensors that can see everything in your home in great detail). Google has a lot of data, for sure. All my: 1. Email. 2. Calendars. 3. Photos. 4. TV watching behavior. 5. Contacts. 6. Documents and spreadsheets. 7. Files. 8. Location data. So I expect Google's robot will be attractive to many. But how do you see the others shake out over the next five years? Make some guesses. But remember what an AI pioneer told me years ago about AI: it's all about the data. The Chinese ones have huge advantages: the Chinese have more data on their citizens, and many more citizens to boot AND they can make robots cheaper than we can. But now that you know OpenAI is building its own robot you have caught wind of what I've heard from many in San Francisco and Silicon Valley: that humanoid robots are the real prize of AI and will be highly profitable for those that can make them and find customers willing to buy them. Here, too, I learned long ago never to bet against Elon Musk. Will you?

Robert Scoble

33,804 просмотров • 1 год назад