Jiafei Duan's banner

Jiafei Duan

@DJiafei • 8,037 subscribers

Assistant Professor at @NUScomputing | Robotics & AI PhD @uwcse | Host of @RoboPapers | Ex-@allen_ai robotics lead, @NVIDIA | Magician | my opinion is my alone.

Shorts

Singapore’s first humanoid robot fight really cool to see a knockout!

Singapore’s first humanoid robot fight really cool to see a knockout!

1,817,245 次观看

Humanoid fighting looks easy. Humanoid catching is hard. You have ~1 second to perceive, predict, and move your whole body to act. Meet Reflex: our framework for catching human-thrown boxes on a Unitree G1 using onboard RGB-D. Thread 👇

Humanoid fighting looks easy. Humanoid catching is hard. You have ~1 second to perceive, predict, and move your whole body to act. Meet Reflex: our framework for catching human-thrown boxes on a Unitree G1 using onboard RGB-D. Thread 👇

43,055 次观看

I trained a controller for Microduck using a 192-neuron Male CNS connectome subset. Watch its modeled neural activity drive exploratory head and body movements. Fun stuff.

I trained a controller for Microduck using a 192-neuron Male CNS connectome subset. Watch its modeled neural activity drive exploratory head and body movements. Fun stuff.

33,596 次观看

My prediction: we’re about to see hundreds of robotics papers built on SAM3D. Waiting for the first one to drop 👀 SAM3 & SAM3D from AI at Meta are just too good to be true!

My prediction: we’re about to see hundreds of robotics papers built on SAM3D. Waiting for the first one to drop 👀 SAM3 & SAM3D from AI at Meta are just too good to be true!

31,884 次观看

Can your VLA perform collision avoidance when explicitly instructed in language? The answer is no. However, with VLS steering it is possible now! The same policy can successfully avoid collisions when given the instruction.

Can your VLA perform collision avoidance when explicitly instructed in language? The answer is no. However, with VLS steering it is possible now! The same policy can successfully avoid collisions when given the instruction.

15,343 次观看

To be honest. Evaluation in robotics has always been a challenge unlike in vision or language: not enough DATA, not enough TASKS, and too much reliance on FIXED benchmarks. PlayGround pushes us beyond static datasets and predefined tasks, moving robotics evaluation into the era of dynamic evaluation in structure physical domain one at a time. *Can't wait to see how the community scale on this!

To be honest. Evaluation in robotics has always been a challenge unlike in vision or language: not enough DATA, not enough TASKS, and too much reliance on FIXED benchmarks. PlayGround pushes us beyond static datasets and predefined tasks, moving robotics evaluation into the era of dynamic evaluation in structure physical domain one at a time. *Can't wait to see how the community scale on this!

10,530 次观看

Robot paper of the day: RoboBallet: Planning for multirobot reaching with graph neural networks and reinforcement learning UCL + Google DeepMind + Intrinsic built an AI planner that choreographs teams of arms to work in tight spaces without collisions—planning in seconds, not days.

Robot paper of the day: RoboBallet: Planning for multirobot reaching with graph neural networks and reinforcement learning UCL + Google DeepMind + Intrinsic built an AI planner that choreographs teams of arms to work in tight spaces without collisions—planning in seconds, not days.

15,084 次观看

Videos

Anya Rossi

sweetdream.ai

SweetDream.ai•Sponsored•Livecam

Watch Anya Live

Anya is streaming live right now! Join her private show and enjoy exclusive content.

Exclusive private shows

1.2k viewers online

Private Show

Join now for exclusive access

Free preview available • Premium content

VLAs? WAMs? Are language and video models the right foundations for robotics? Introducing Grounded Action Model (GAM): a new paradigm that builds robot foundation models on top of a pretrained 3D grounding model. Ground first. Then learn to act. 🧵👇

VLAs? WAMs? Are language and video models the right foundations for robotics? Introducing Grounded Action Model (GAM): a new paradigm that builds robot foundation models on top of a pretrained 3D grounding model. Ground first. Then learn to act. 🧵👇

690,198 次观看 • 27 天前

Ran the first policy in my new lab at NUS Computing. Zero-shot, fresh out of the box.

Ran the first policy in my new lab at NUS Computing. Zero-shot, fresh out of the box.

455,655 次观看 • 2 个月前

People watching GPT-6 Astra demos on X may not realise how slow it actually is in real time. Here’s a rollout of GPT-6 Astra controlling the MolmoAct2 YAMs to complete this task in real-time. There is potential, but i don’t think it’s practical for real-world robotics deployment yet. I applaud your patience if you make it through the whole video.

People watching GPT-6 Astra demos on X may not realise how slow it actually is in real time. Here’s a rollout of GPT-6 Astra controlling the MolmoAct2 YAMs to complete this task in real-time. There is potential, but i don’t think it’s practical for real-world robotics deployment yet. I applaud your patience if you make it through the whole video.

61,959 次观看 • 22 天前

I think GPT-6 Astra will become the default baseline for future robotics papers. But for now, it would be interesting to find the boundary of its capabilities. At least for now, it still can’t solve this.

I think GPT-6 Astra will become the default baseline for future robotics papers. But for now, it would be interesting to find the boundary of its capabilities. At least for now, it still can’t solve this.

11,011 次观看 • 4 天前

Is Opus 5.5 better than GPT-6 Astra at Physical Intelligence "literally" ? put GPT-6 Astra and Claude Opus 5.5 in two Unitree G1s in MuJoCo Ring Fence. Every 2 sim seconds: state → LLM → joint targets → act. Goal: force the other out of the ring, and Astra won after 31 second.

Is Opus 5.5 better than GPT-6 Astra at Physical Intelligence "literally" ? put GPT-6 Astra and Claude Opus 5.5 in two Unitree G1s in MuJoCo Ring Fence. Every 2 sim seconds: state → LLM → joint targets → act. Goal: force the other out of the ring, and Astra won after 31 second.

54,246 次观看 • 26 天前

Was GPT-6 Astra trained on robot data? Got me curious. Borrowed Jay Chooi's code, ran an open-loop rollout on a MolmoAct2 dataset trajectory to find out. Here's what came out 👇

Was GPT-6 Astra trained on robot data? Got me curious. Borrowed Jay Chooi's code, ran an open-loop rollout on a MolmoAct2 dataset trajectory to find out. Here's what came out 👇

72,303 次观看 • 1 个月前

GPT-6 Astra vs Opus 5.5 vs Grok 4.7 vs MolmoAct2. Real time. No sped-up demos. Zero-shot. No cherry-picking. LLM robot control is progressing, but we’re definitely not there yet. Low-level control for robotics still matters according to Moravec’s paradox.

GPT-6 Astra vs Opus 5.5 vs Grok 4.7 vs MolmoAct2. Real time. No sped-up demos. Zero-shot. No cherry-picking. LLM robot control is progressing, but we’re definitely not there yet. Low-level control for robotics still matters according to Moravec’s paradox.

26,636 次观看 • 21 天前

I trained a controller for Microduck using a 192-neuron Male CNS connectome subset. Watch its modeled neural activity drive exploratory head and body movements. Fun stuff.

I trained a controller for Microduck using a 192-neuron Male CNS connectome subset. Watch its modeled neural activity drive exploratory head and body movements. Fun stuff.

33,596 次观看 • 1 个月前

Randomly stumbled on this video and wow. We humans are amazing manipulators. Memorization of skillful actions plays a part, but the sense of touch is what's truly crucial.

Randomly stumbled on this video and wow. We humans are amazing manipulators. Memorization of skillful actions plays a part, but the sense of touch is what's truly crucial.

56,975 次观看 • 2 个月前

Most capable generalist robotics models today are closed or at best, open weights. But robotics won’t reach its ChatGPT moment without real openness. That GPT moment was built on years of open tools and datasets such as Python, PyTorch, ImageNet and more, that let researchers inspect, reproduce, and build. Today, we’re introducing MolmoAct 2: a fully open-source action reasoning model for real-world robotics. We rethought and reshaped everything! 🧵👇

Most capable generalist robotics models today are closed or at best, open weights. But robotics won’t reach its ChatGPT moment without real openness. That GPT moment was built on years of open tools and datasets such as Python, PyTorch, ImageNet and more, that let researchers inspect, reproduce, and build. Today, we’re introducing MolmoAct 2: a fully open-source action reasoning model for real-world robotics. We rethought and reshaped everything! 🧵👇

117,636 次观看 • 5 个月前

Instead of asking a VLM to output progress, it reads the model’s internal belief directly from token logits. No in-context learning. No fine-tuning. No reward training. 📈 We introduce: TOPReward, a zero-shot reward modeling approach for robotics using token probabilities from pretrained video VLMs. The simplest way of doing reward modelling for robotics! Project: 🧵👇

Instead of asking a VLM to output progress, it reads the model’s internal belief directly from token logits. No in-context learning. No fine-tuning. No reward training. 📈 We introduce: TOPReward, a zero-shot reward modeling approach for robotics using token probabilities from pretrained video VLMs. The simplest way of doing reward modelling for robotics! Project: 🧵👇

108,387 次观看 • 7 个月前

Are you training your VLAs or WAMs correctly? Your action expert may be learning vision–action shortcuts that undermine generalization beyond the training distribution. Introducing Latent Interface Training (LIT 🔥): Learn to act first, then learn how to use vision. 🧵👇

Are you training your VLAs or WAMs correctly? Your action expert may be learning vision–action shortcuts that undermine generalization beyond the training distribution. Introducing Latent Interface Training (LIT 🔥): Learn to act first, then learn how to use vision. 🧵👇

16,420 次观看 • 1 个月前

Why do generalist robotic models fail when a cup is moved just two inches to the left? It’s not a lack of motor skill, it’s an alignment problem. Today, we introduce VLS: Vision-Language Steering of Pretrained Robot Policies, a training-free framework that guides robot behavior in real time. Check out the project: 👇🧵 (Watch till the end: VLS runs uncut, steering pretrained policies across long-horizon tasks.)

Why do generalist robotic models fail when a cup is moved just two inches to the left? It’s not a lack of motor skill, it’s an alignment problem. Today, we introduce VLS: Vision-Language Steering of Pretrained Robot Policies, a training-free framework that guides robot behavior in real time. Check out the project: 👇🧵 (Watch till the end: VLS runs uncut, steering pretrained policies across long-horizon tasks.)

74,089 次观看 • 8 个月前

Reasoning is central to purposeful action. Today we introduce MolmoAct — a fully open Action Reasoning Model (ARM) for robotics. Grounded in large-scale pre-training with action reasoning data, every predicted action is interpretable and user-steerable via visual trace. We are open-sourcing everything!

Reasoning is central to purposeful action. Today we introduce MolmoAct — a fully open Action Reasoning Model (ARM) for robotics. Grounded in large-scale pre-training with action reasoning data, every predicted action is interpretable and user-steerable via visual trace. We are open-sourcing everything!

99,944 次观看 • 1 年前

Given the strong community adoption and real-world deployment of MolmoAct2 on YAM, we're introducing zero-shot evaluation of MolmoAct2 Bimanual YAM in simulation. Now you can test out our models without a real-world robot and build on them! Code: Simulation built on Maniskill!

Given the strong community adoption and real-world deployment of MolmoAct2 on YAM, we're introducing zero-shot evaluation of MolmoAct2 Bimanual YAM in simulation. Now you can test out our models without a real-world robot and build on them! Code: Simulation built on Maniskill!

37,276 次观看 • 4 个月前

Humans use pointing to communicate plans intuitively. Compared to language, pointing gives more precise guidance to robot behaviors. Can we teach a robot how to point like humans? Introducing RoboPoint 🤖👉, an open-source VLM instruction-tuned to point. Check out our new work:

Humans use pointing to communicate plans intuitively. Compared to language, pointing gives more precise guidance to robot behaviors. Can we teach a robot how to point like humans? Introducing RoboPoint 🤖👉, an open-source VLM instruction-tuned to point. Check out our new work:

64,511 次观看 • 2 年前

Every time I watch this video, I can't help but wonder: why don't we have robot butlers in our homes yet? The hardware seemed capable 14 years ago with teleoperation. Is it just the "robot brain" we're missing, or is there more to the puzzle?

Every time I watch this video, I can't help but wonder: why don't we have robot butlers in our homes yet? The hardware seemed capable 14 years ago with teleoperation. Is it just the "robot brain" we're missing, or is there more to the puzzle?

47,387 次观看 • 1 年前

Humans learn and improve from failures. Similarly, foundation models adapt based on human feedback. Can we leverage this failure understanding to enhance robotics systems that use foundation models? Introducing AHA—a vision-language model for detecting and reasoning over failures in robotic manipulation. Project page: 🧵Thread👇 Aha!

Humans learn and improve from failures. Similarly, foundation models adapt based on human feedback. Can we leverage this failure understanding to enhance robotics systems that use foundation models? Introducing AHA—a vision-language model for detecting and reasoning over failures in robotic manipulation. Project page: 🧵Thread👇 Aha!

48,860 次观看 • 2 年前

Finally hooded! Collecting Egocentric data even during graduation! Huge thanks to my advisor Dieter Fox and Ranjay Krishna !

Finally hooded! Collecting Egocentric data even during graduation! Huge thanks to my advisor Dieter Fox and Ranjay Krishna !

11,439 次观看 • 4 个月前