Video wird geladen...

Video konnte nicht geladen werden

Beim Laden dieses Videos ist ein Problem aufgetreten. Dies könnte an einem vorübergehenden Netzwerkproblem liegen oder das Video ist möglicherweise nicht verfügbar.

Evolving dexterity with GPT-6 Astra 🖐️ Been trying Astra recently. Its zero-shot dexterous manipulation is already quite surprising. More interesting is seeing it learn and improve through simulation training, from pen spinning and Rubik's Cube to hammer use. The real goal would be to evolve this dexterity in the... show more

Jianglong Ye

1,304 subscribers

109,195 Aufrufe • vor 18 Tagen •via X (Twitter)

Bildung Wissenschaft & Technologie

Anya Rossi• Live Now

Private livecam show

18 Kommentare

Profilbild von Tongzhou Mu 🤖🦾🦿

Tongzhou Mu 🤖🦾🦿vor 18 Tagen

nice results. How was Astra used here exactly?

Profilbild von Jianglong Ye

Jianglong Yevor 18 Tagen

For example, with Rubik’s cube, Astra either scripts its motion from one prompt, or builds the simulation/training and composes learned turn-and-roll skills. I mainly check the final rendering.

Profilbild von Conor Mc Gartoll

Conor Mc Gartollvor 18 Tagen

nice! always have the most aesthetic renders

Profilbild von James Camarota

James Camarotavor 18 Tagen

The simulation progress looks promising. How much of the pen spinning or tool use transfers to a real hand when friction and object weight change?

Profilbild von Jianglong Ye

Jianglong Yevor 18 Tagen

I assume this simulation progress is just a small step toward complete sim2real transfer. Will try a real hand later. Should be fun!

Profilbild von Quanquan Peng

Quanquan Pengvor 17 Tagen

Amazing result! Beautiful rendering! astra’s sense of aesthetics is much better than mine…

Profilbild von Vishal Mandadi

Vishal Mandadivor 17 Tagen

Hey, I went through the repo. I don't see any interface/harness for zero-shot control with Astra. I am curious how Astra exactly works with the pen-spinning task in zero-shot :) Could you provide more details on it?

Profilbild von Jianglong Ye

Jianglong Yevor 17 Tagen

Yep, this repo is only the final artifact, not a harness. For pen spinning, I validated this prompt: "There's a pen simulator here. A robot hand lies palm-up with a pen on its fingers. Spin it one full turn without dropping it, ending steady. No PPO." When I tried it before, Astra could also set up the sim environment itself from scratch, so providing the sim env is not essential.

Profilbild von Vishal Mandadi

Vishal Mandadivor 17 Tagen

That's pretty cool! Any idea how it did it (underlying mechanism)? Like, I am guessing it probably used CEM-style planning

Profilbild von Vishal Mandadi

Vishal Mandadivor 17 Tagen

Oh yeah, I could see the CEM implemented in the repo. Nice result :)

Profilbild von clankr

clankrvor 17 Tagen

How exactly did it approach this? It feels like there are some shortcuts in there, with parts borrowed from previous work.

Profilbild von Jianglong Ye

Jianglong Yevor 17 Tagen

I assume the agent reused standard PPO impl and may have referred to open-source work for skills like pen spinning or cube manipulation. I didn't restrict that. The main work might be training skills for the specific embodiments/sim setups, then composing them.

Profilbild von Muhammad Ahmed

Muhammad Ahmedvor 17 Tagen

A dropped pen is cheap feedback; a misplaced hammer strike isn't. Moving this dexterity from simulation onto hardware will make force limits and safe exploration just as important as the task objective.

Profilbild von Dmytro Hrybov

Dmytro Hrybovvor 17 Tagen

what is actually being trained here?

Profilbild von Jianglong Ye

Jianglong Yevor 17 Tagen

The trained parts are the low-level primitive skills: pen spinning, three cube-rotation primitives, grasping/reorientation, etc. These primitives are not predefined either: the agent decides the skill decomposition itself, then trains the corresponding policies in simulation.

Profilbild von Rupesh Shrestha

Rupesh Shresthavor 17 Tagen

👏

Profilbild von awful

awfulvor 17 Tagen

i'm intrigued by the simulation training aspect, wondering how it'll translate to real-world dexterity applications

Profilbild von Sachin Bhadang

Sachin Bhadangvor 17 Tagen

Cool results!

Ähnliche Videos

This time I had GPT 6 Astra try to break the world record using a 5x5 Rubik's Cube... The (human) world record for a 5x5 is 29.49s which Astra just barely fell short of while using computer use. I still don't think people realize how insane that is.

This time I had GPT 6 Astra try to break the world record using a 5x5 Rubik's Cube... The (human) world record for a 5x5 is 29.49s which Astra just barely fell short of while using computer use. I still don't think people realize how insane that is.

Layton Gott

64,204 Aufrufe • vor 26 Tagen

People watching GPT-6 Astra demos on X may not realise how slow it actually is in real time. Here’s a rollout of GPT-6 Astra controlling the MolmoAct2 YAMs to complete this task in real-time. There is potential, but i don’t think it’s practical for real-world robotics deployment yet. I applaud your patience if you make it through the whole video.

People watching GPT-6 Astra demos on X may not realise how slow it actually is in real time. Here’s a rollout of GPT-6 Astra controlling the MolmoAct2 YAMs to complete this task in real-time. There is potential, but i don’t think it’s practical for real-world robotics deployment yet. I applaud your patience if you make it through the whole video.

Jiafei Duan

61,624 Aufrufe • vor 8 Tagen

We made GPT-6 Astra and Gemini 3.8 Flash play capture the flag on real robots while roasting each other and... Gemini won 70% of the time! Here is a flaming Gemini win, more in the thread This is surprising given the recent astra hype, but not entirely. Findings below ⬇️

We made GPT-6 Astra and Gemini 3.8 Flash play capture the flag on real robots while roasting each other and... Gemini won 70% of the time! Here is a flaming Gemini win, more in the thread This is surprising given the recent astra hype, but not entirely. Findings below ⬇️

Axel

14,798 Aufrufe • vor 14 Tagen

We're so cooked 🤯 GPT-6 Astra is automating video editing. It writes production LUTs from a sentence, work colorists spent 20 years learning to do. You describe the look you want, Astra writes a real .cube LUT file, and you import it into CapCut PC to apply it, adjust the intensity, and keep cutting with it. Same with animated subtitle styles. Here's the breakdown 🧵:

We're so cooked 🤯 GPT-6 Astra is automating video editing. It writes production LUTs from a sentence, work colorists spent 20 years learning to do. You describe the look you want, Astra writes a real .cube LUT file, and you import it into CapCut PC to apply it, adjust the intensity, and keep cutting with it. Same with animated subtitle styles. Here's the breakdown 🧵:

Jaynit Makwana

114,954 Aufrufe • vor 24 Tagen

Teaching robots real dexterity has always been a challenge. But what if they could handle tools like a human? DexterityGen (DexGen) is a new system that helps robots use their hands better. It improves how they grip, move, and handle objects… from holding a pen to using a screwdriver. DexGen learns in simulation and refines its skills in the real world, making robotic hands much more useful. What makes DexGen special? ✅ Smarter movements that refine rough actions into precise skills ✅ Trained on a massive collection of dexterous tasks for better learning ✅ Better teleoperation that makes robotic hand control easier and safer ✅ Handles real-world challenges like small objects, tricky angles, and gravity This moves robots closer to real dexterity. It makes tool use more natural, improves stability, and brings robotic hands one step closer to human-level skill. Seen at Zhao-Heng Yin 🫶 Github: Paper:

Teaching robots real dexterity has always been a challenge. But what if they could handle tools like a human? DexterityGen (DexGen) is a new system that helps robots use their hands better. It improves how they grip, move, and handle objects… from holding a pen to using a screwdriver. DexGen learns in simulation and refines its skills in the real world, making robotic hands much more useful. What makes DexGen special? ✅ Smarter movements that refine rough actions into precise skills ✅ Trained on a massive collection of dexterous tasks for better learning ✅ Better teleoperation that makes robotic hand control easier and safer ✅ Handles real-world challenges like small objects, tricky angles, and gravity This moves robots closer to real dexterity. It makes tool use more natural, improves stability, and brings robotic hands one step closer to human-level skill. Seen at Zhao-Heng Yin 🫶 Github: Paper:

Ilir Aliu

51,526 Aufrufe • vor 1 Jahr

Achieving human-like dexterity is the next frontier for robotics, and yet dexterity data is often subtly hard to scale. Real-world dexterity data, including things like finger-pose estimates, is often slightly off, making it physically invalid and hard to execute on real hardware and hard to learn from. DO AS I DO is an algorithm for reconstructing and retargeting monocular RGB videos to robot hands, outperforming the state of the art and even working from generated videos. Bhawna Paliwal Haritheja Will Liang and Mahi Shafiullah 🏠🤖 join us to tell us more. Watch Episode 107 of RoboPapers, with Chris Paxton and Jiafei Duan, now!

Achieving human-like dexterity is the next frontier for robotics, and yet dexterity data is often subtly hard to scale. Real-world dexterity data, including things like finger-pose estimates, is often slightly off, making it physically invalid and hard to execute on real hardware and hard to learn from. DO AS I DO is an algorithm for reconstructing and retargeting monocular RGB videos to robot hands, outperforming the state of the art and even working from generated videos. Bhawna Paliwal Haritheja Will Liang and Mahi Shafiullah 🏠🤖 join us to tell us more. Watch Episode 107 of RoboPapers, with Chris Paxton and Jiafei Duan, now!

RoboPapers

23,758 Aufrufe • vor 10 Tagen

I gave GPT-6 Astra Ultra one goal: build a rocket ship in The Universim 7h 50m, 6.1M tokens and $367.40 later, it did it It controlled the game through a CLI it built itself and Computer Use, with subagents handling threats, CLI usage and industry I didn't expect it to work because Astra is so slow, but it was smart enough to pause the game, plan its next moves and execute them Here's the gameplay at 100x speed Astra never fails to amaze me

I gave GPT-6 Astra Ultra one goal: build a rocket ship in The Universim 7h 50m, 6.1M tokens and $367.40 later, it did it It controlled the game through a CLI it built itself and Computer Use, with subagents handling threats, CLI usage and industry I didn't expect it to work because Astra is so slow, but it was smart enough to pause the game, plan its next moves and execute them Here's the gameplay at 100x speed Astra never fails to amaze me

Angel 🌼

40,842 Aufrufe • vor 19 Tagen

ChatGPT Work with GPT 6 Astra is incredible It is the best productivity tool on the planet right now I challenged myself this week to do all my work with JUST Work and I've never been more productive In this video I cover how to use it and how to super charge your productivity

ChatGPT Work with GPT 6 Astra is incredible It is the best productivity tool on the planet right now I challenged myself this week to do all my work with JUST Work and I've never been more productive In this video I cover how to use it and how to super charge your productivity

Alex Finn

59,391 Aufrufe • vor 22 Tagen

Pen spinning on WUJI Hand 2, from simulation training to real-world execution. We use motion-reference tracking and reinforcement learning to bring a simulation-trained policy to the physical hand. Code and setup guides are open source for developers and researchers to build on. Code:

Pen spinning on WUJI Hand 2, from simulation training to real-world execution. We use motion-reference tracking and reinforcement learning to bring a simulation-trained policy to the physical hand. Code and setup guides are open source for developers and researchers to build on. Code:

WUJI

12,065 Aufrufe • vor 6 Tagen

GPT-6 Astra is our best model yet for computer use, software engineering, and visual understanding to date. Hear from the builders putting it to work:

GPT-6 Astra is our best model yet for computer use, software engineering, and visual understanding to date. Hear from the builders putting it to work:

OpenAI Developers

798,627 Aufrufe • vor 1 Monat

How to edit videos with GPT-6 Astra: 1. Install the ChatCut plugin in ChatGPT/Codex 2. Select GPT-6 Astra 3. Upload your footage 4. Describe the video you want Astra understands your footage and edits it directly in ChatCut. Every cut, caption, and motion graphic stays editable on a real timeline. This screen recording shows the entire setup and a short-video edit from start to finish. Comment “Chat” to get the plugin.

How to edit videos with GPT-6 Astra: 1. Install the ChatCut plugin in ChatGPT/Codex 2. Select GPT-6 Astra 3. Upload your footage 4. Describe the video you want Astra understands your footage and edits it directly in ChatCut. Every cut, caption, and motion graphic stays editable on a real timeline. This screen recording shows the entire setup and a short-video edit from start to finish. Comment “Chat” to get the plugin.

ChatCut

44,460 Aufrufe • vor 26 Tagen

🧐Applying world models to improve real-world policy on challenging manipulation tasks used to be considered out of reach. 😌After sustained effort, we’re now seeing encouraging progress. 🚀Thrilled to introduce RISE: Self-Improving Robot Policy with Compositional World Model RISE is, to our knowledge, the first work to use a world model as an effective learning environment for challenging real-world manipulation, enabling policy improvement on tasks that demand high dynamics, dexterity, and precision. Incredible teamwork with KL Hongyang Li Yue Xiangyu Hao Zhao Chonghao Sima

🧐Applying world models to improve real-world policy on challenging manipulation tasks used to be considered out of reach. 😌After sustained effort, we’re now seeing encouraging progress. 🚀Thrilled to introduce RISE: Self-Improving Robot Policy with Compositional World Model RISE is, to our knowledge, the first work to use a world model as an effective learning environment for challenging real-world manipulation, enabling policy improvement on tasks that demand high dynamics, dexterity, and precision. Incredible teamwork with KL Hongyang Li Yue Xiangyu Hao Zhao Chonghao Sima

Jiazhi Yang

77,124 Aufrufe • vor 7 Monaten

This is sick! I built the entire city of Gurgaon in 3D using GPT-6 Astra I saw everyone building all sorts of cool demos with Astra, so I decided to give it a shot and needless to say, I was not dissapointed

This is sick! I built the entire city of Gurgaon in 3D using GPT-6 Astra I saw everyone building all sorts of cool demos with Astra, so I decided to give it a shot and needless to say, I was not dissapointed

Yajat Gulati

50,166 Aufrufe • vor 29 Tagen

The interesting part isn't just generating a product. Turns out GPT-6 Astra + Dreamina Seedance 2.5 is an actual pipeline. It's going from a draft → real-looking product → product page in one continuous workflow. I started with a simple product concept in Dreamina. Then GPT-6 Astra interpreted the draft, built out the actual product, and used it to create the final product detail page. Dreamina's pricing also makes it practical to generate a bunch of early concepts instead of overthinking the first one. This workflow is definitely worth experimenting with. #Dreamina #DreaminaPartner

The interesting part isn't just generating a product. Turns out GPT-6 Astra + Dreamina Seedance 2.5 is an actual pipeline. It's going from a draft → real-looking product → product page in one continuous workflow. I started with a simple product concept in Dreamina. Then GPT-6 Astra interpreted the draft, built out the actual product, and used it to create the final product detail page. Dreamina's pricing also makes it practical to generate a bunch of early concepts instead of overthinking the first one. This workflow is definitely worth experimenting with. #Dreamina #DreaminaPartner

Md Riyazuddin

61,601 Aufrufe • vor 24 Tagen

Want to improve your exchange? Check out the 1st drill in this progression by Hedges! Obviously it helps to have big, strong hands. ELITE finger dexterity! {Finger dexterity is the ability to move fingers and hands precisely, quickly, and with high coordination} Brian Whatley (DNA Catching)

Want to improve your exchange? Check out the 1st drill in this progression by Hedges! Obviously it helps to have big, strong hands. ELITE finger dexterity! {Finger dexterity is the ability to move fingers and hands precisely, quickly, and with high coordination} Brian Whatley (DNA Catching)

The Catching Guy

37,577 Aufrufe • vor 6 Monaten

A big part of scaling robot learning to solve real-world problems is that we somehow need to get enough diverse, high-quality data to train our robots to perform useful things. GPT and its fellow large language models were bootstrapped and proved out on a massive dataset of real-world language data. Unfortunately, despite our best efforts, similarly massive datasets don’t really exist for robotics — so, in our unending pursuit of high-quality, useful data, we turn to simulation. I compared a couple recent works on sim-to-real robot manipulation, which discuss how to train perception-driven manipulation policies in simulation, in such a way that they’re useful in the real world. - DextraH-RGB, from NVIDIA - Sim-and-Real Co-Training: A Simple Recipe for Vision-Based Robotic Manipulation, also from NVIDIA — specifically the GEAR lab - Sim-to-Real Reinforcement Learning for Vision-Based Dexterous Manipulation on Humanoids, another GEAR lab paper - Local Policies Enable Zero-shot Long-Horizon Manipulation, from CMU (video from DextrAH-RGB)

A big part of scaling robot learning to solve real-world problems is that we somehow need to get enough diverse, high-quality data to train our robots to perform useful things. GPT and its fellow large language models were bootstrapped and proved out on a massive dataset of real-world language data. Unfortunately, despite our best efforts, similarly massive datasets don’t really exist for robotics — so, in our unending pursuit of high-quality, useful data, we turn to simulation. I compared a couple recent works on sim-to-real robot manipulation, which discuss how to train perception-driven manipulation policies in simulation, in such a way that they’re useful in the real world. - DextraH-RGB, from NVIDIA - Sim-and-Real Co-Training: A Simple Recipe for Vision-Based Robotic Manipulation, also from NVIDIA — specifically the GEAR lab - Sim-to-Real Reinforcement Learning for Vision-Based Dexterous Manipulation on Humanoids, another GEAR lab paper - Local Policies Enable Zero-shot Long-Horizon Manipulation, from CMU (video from DextrAH-RGB)

Chris Paxton

20,486 Aufrufe • vor 1 Jahr

I had GPT 6 Astra create all the light fixtures I want to simulate based on the actual spec sheets. Finally I can rapidly iterate through stage designs in Sentinel and then bring it into the real world!

I had GPT 6 Astra create all the light fixtures I want to simulate based on the actual spec sheets. Finally I can rapidly iterate through stage designs in Sentinel and then bring it into the real world!

Spencer Sterling

14,937 Aufrufe • vor 28 Tagen

🎙️ Excited to introduce one of my favorite projects from the past year: TeleDexter, from the BIGAI dexterity team. It’s a stable, human-level dexterous teleoperation system and a suite of autonomous policies trained with it. Pen spinning, complex in-hand reorientation, and long-horizon tool use—once seen as the holy grail of manipulation—are now unlocked. 🧵👇 The hardware is already here; we have some incredible high-DoF robotic hands. The bottleneck? The controller. Most current systems are stuck in "quasi-static" grasping mode. Meanwhile, dynamic in-hand dexterity has remained severely limited. 🧠 To unlock the massive capabilities of human-like hands, we need to build an excellent "cerebellum" for dexterous hands. TeleDexter solves this with a novel co-tracking approach: it simultaneously tracks both human hand kinematics and object states, beautifully bridging the gap between human intent and robotic control. In order to train a better co-tracking policy that works robustly in the real world, we designed : (1) a hybrid reward design that combines consecutive goal reaching and dense tracking, (2) an action masking strategy during training that enhances sim2real performance, (3) a dexterous curriculum for learning the long-horizon interactions. Each design is inspired by numerous trials and countless real-world experiments. We’ve synthesized all the system details, engineering challenges, and core insights into our latest post. If you're interested in the future of dexterous manipulation, grab a coffee and check it out (9-min read): If you have more time, check out the paper:

🎙️ Excited to introduce one of my favorite projects from the past year: TeleDexter, from the BIGAI dexterity team. It’s a stable, human-level dexterous teleoperation system and a suite of autonomous policies trained with it. Pen spinning, complex in-hand reorientation, and long-horizon tool use—once seen as the holy grail of manipulation—are now unlocked. 🧵👇 The hardware is already here; we have some incredible high-DoF robotic hands. The bottleneck? The controller. Most current systems are stuck in "quasi-static" grasping mode. Meanwhile, dynamic in-hand dexterity has remained severely limited. 🧠 To unlock the massive capabilities of human-like hands, we need to build an excellent "cerebellum" for dexterous hands. TeleDexter solves this with a novel co-tracking approach: it simultaneously tracks both human hand kinematics and object states, beautifully bridging the gap between human intent and robotic control. In order to train a better co-tracking policy that works robustly in the real world, we designed : (1) a hybrid reward design that combines consecutive goal reaching and dense tracking, (2) an action masking strategy during training that enhances sim2real performance, (3) a dexterous curriculum for learning the long-horizon interactions. Each design is inspired by numerous trials and countless real-world experiments. We’ve synthesized all the system details, engineering challenges, and core insights into our latest post. If you're interested in the future of dexterous manipulation, grab a coffee and check it out (9-min read): If you have more time, check out the paper:

Siyuan Huang

12,746 Aufrufe • vor 2 Monaten

It’s long been a dream of roboticists to be able to teach a robot in simulation so as to skip the long and expensive process of collecting large amounts of real-world training data. However, building simulations for robot tasks is extremely hard. Ideally, we could go from real data to a useful simulation. This is exactly what Guangqi Jiang and his co-authors do. they use 3d Gaussian splatting to reconstructed scenes which let them create interactive environments that, when combined with a physcs engine, allow for training robot policies that show zero-shot sim-to-real transfer (i.e., using no real-world demonstrations). To learn more, watch Episode 56 of Robopapers with Michael Cho - Rbt/Acc and Chris Paxton now!

It’s long been a dream of roboticists to be able to teach a robot in simulation so as to skip the long and expensive process of collecting large amounts of real-world training data. However, building simulations for robot tasks is extremely hard. Ideally, we could go from real data to a useful simulation. This is exactly what Guangqi Jiang and his co-authors do. they use 3d Gaussian splatting to reconstructed scenes which let them create interactive environments that, when combined with a physcs engine, allow for training robot policies that show zero-shot sim-to-real transfer (i.e., using no real-world demonstrations). To learn more, watch Episode 56 of Robopapers with Michael Cho - Rbt/Acc and Chris Paxton now!

RoboPapers

20,434 Aufrufe • vor 9 Monaten

GPT-6 Astra smokes Claude Fable 5.1 in Blender 💨 We compared GPT-6 Astra and Fable 5.1 on a Porsche 911 GT3 RS build and the difference is visible from the very first shot Output: Astra: $132.03 • 2.58 hours Fable: $114.41 • 2.56 hours Astra's model is not only more detailed in every aspect, but also far closer to the original car, costing almost the same as Fable's blocky funny looking model, the difference is astonishing

GPT-6 Astra smokes Claude Fable 5.1 in Blender 💨 We compared GPT-6 Astra and Fable 5.1 on a Porsche 911 GT3 RS build and the difference is visible from the very first shot Output: Astra: $132.03 • 2.58 hours Fable: $114.41 • 2.56 hours Astra's model is not only more detailed in every aspect, but also far closer to the original car, costing almost the same as Fable's blocky funny looking model, the difference is astonishing

atomic.chat

132,877 Aufrufe • vor 28 Tagen