Загрузка видео...

Не удалось загрузить видео

На главную

Introducing ACT-2 Preview The first robotics model to unify broad generalization with high reliability. A single fine-tuning example can teach Memo a new behavior that generalizes. Zero shot, real unseen homes, 99% success rate.

1,060,037 просмотров • 3 месяцев назад •via X (Twitter)

Комментарии: 61

Фото профиля Tony Zhao
Tony Zhao3 месяцев назад

We completed the most rigorous generalization test to date. Across 785 trials in 31 unseen environments, ACT-2 achieved 99.1% success in laundry folding. ACT-2 also achieved human-level fold quality: receiving an average rating of 4.72/5, with 98.3% earning four or five stars.

Фото профиля Tony Zhao
Tony Zhao3 месяцев назад

Tons of emergent behavior along the way: - Picking up clothes off the ground - Handling baby wear to 8XL shirts - Robustness against adversarial disturbances and lighting

Фото профиля Tony Zhao
Tony Zhao3 месяцев назад

We call this a Solve. Progress in robotics is difficult to measure because demos vary by setup. Demo ≠ Solved. A Solve declares two boundaries: scope and adaptation cost. Without both, 99% has no context.

Фото профиля Tony Zhao
Tony Zhao3 месяцев назад

We found a general recipe for Solves: scale pretraining, then hill-climb with minimal in-house data. For the first time, one fine-tuning example can teach a new behavior that generalizes. Below: 4 folding strategies, learned from one example each and tested on held-out setups.

Фото профиля Tony Zhao
Tony Zhao3 месяцев назад

Quantitatively, we measure the generalization gap as the difference between in-domain and out-of-domain performance. As we scale up pretraining, the gap falls sharply. This makes in-house performance a reliable predictor of performance in the wild.

Фото профиля Tony Zhao
Tony Zhao3 месяцев назад

This property allows us to hill-climb performance in our office, and trust those gains to hold in unseen homes Our fleet of Memos runs in parallel to rapidly advance reliability, quality, and speed. Left: fleet-scale improvement in-house Right: Memo working across unseen homes

Фото профиля Tony Zhao
Tony Zhao3 месяцев назад

Laundry is our first Solve of many. Our recipe is so general that scaling data and compute gives us predictable improvements. Unlocking one Solve accelerates the next Solve. The same ACT-2 model is learning to vacuum, organize toys, zip clothing, and turn pants inside out.

Фото профиля Tony Zhao
Tony Zhao3 месяцев назад

This fall, ACT-2’s first Solve enters homes through our Beta Program, the final step towards fully autonomous home robot deployment. Full technical report:

Фото профиля Sholto Douglas
Sholto Douglas3 месяцев назад

So cool, can't wait to have one at my place

Фото профиля Tony Zhao
Tony Zhao3 месяцев назад

Very soon 😉

Фото профиля Roei Herzig
Roei Herzig3 месяцев назад

Is it possible to get access to the pretrained model? We recently developed a mechanistic interpretability concept for fast-adaptation by tuning only the relevant parts of the network: If the pretrained model is strong, we can do magic. Give us access, and we can unlock continual learning together ;)

Фото профиля Tony Zhao
Tony Zhao3 месяцев назад

This is very cool, checking it out now!

Фото профиля Camilla Guo
Camilla Guo3 месяцев назад

sending this to my mom

Фото профиля Dhaval Shroff
Dhaval Shroff3 месяцев назад

Very cool to see this kind of generalization @tonyzzhao

Фото профиля Tony Zhao
Tony Zhao3 месяцев назад

Just ask for generalization! @ericjang11

Фото профиля Tim Zaman
Tim Zaman3 месяцев назад

But are you still shipping this year

Фото профиля Tony Zhao
Tony Zhao3 месяцев назад

Have you applied!

Фото профиля Yuanhao Qu
Yuanhao Qu3 месяцев назад

@timzaman Congrats @tonyzzhao on the launch!!! Can’t wait to have one at home.

Фото профиля Tony Zhao
Tony Zhao3 месяцев назад

@timzaman Thank you Yuanhao! Apply here!

Фото профиля Kate Park
Kate Park3 месяцев назад

Congrats! data engine end to end ftw 💪

Фото профиля Remi Cadene
Remi Cadene3 месяцев назад

Extremely cool ;)

Фото профиля Tony Zhao
Tony Zhao3 месяцев назад

Thank you Remi!! Congrats on the humanoid release. So cool.

Фото профиля Jason Ma
Jason Ma3 месяцев назад

congrats tony! awesome to see generalization + reliability

Фото профиля Tony Zhao
Tony Zhao3 месяцев назад

Thank you Jason! 🙏

Фото профиля Caitlin Kalinowski
Caitlin Kalinowski3 месяцев назад

The wanting intensifies! Does this mean I will be able to teach it to fold things my way? 😍

Фото профиля Tony Zhao
Tony Zhao3 месяцев назад

Exactly! I'm honestly quite surprised that it just works with SFT..!

Фото профиля Perry Jia
Perry Jia3 месяцев назад

Proud of the team for this achievement

Фото профиля Yu Xiang
Yu Xiang3 месяцев назад

@RemiCadene Very Impressive. Congratulations! There is a snapshot where the head camera is blocked, and the robot can still keep going. How to explain this behavior?

Фото профиля Tony Zhao
Tony Zhao3 месяцев назад

@RemiCadene ACT-2 takes all 5 cameras (one head, two on each hand) as input. And sometimes dropout is all you need!

Фото профиля Ryan Julian
Ryan Julian3 месяцев назад

Congratulations @sundayrobotics team! Incredible work

Фото профиля Tony Zhao
Tony Zhao3 месяцев назад

@sundayrobotics Thank you Ryan!

Фото профиля Machine Space
Machine Space3 месяцев назад

@Scobleizer I can’t believe nobody has asked for the holy grail of folding: The King Size Fitted Sheet.

Фото профиля Aaref Hilaly
Aaref Hilaly3 месяцев назад

big breakthrough - congrats!

Фото профиля Tony Zhao
Tony Zhao3 месяцев назад

Thank you Aaref for being part of this journey ❤️

Фото профиля Rahul
Rahul3 месяцев назад

This all seemed so far away till last year

Фото профиля Jiafei Duan
Jiafei Duan3 месяцев назад

Really cool work! It will be cooler if the weights could be open source, seems to be a common question here.

Фото профиля Lindon Gao
Lindon Gao3 месяцев назад

Congrats @tonyzzhao & team!

Фото профиля Tony Zhao
Tony Zhao3 месяцев назад

Thank you Lindon!!

Фото профиля Jake Roggenbuck
Jake Roggenbuck3 месяцев назад

Generalizing reliability!!

Фото профиля Owen Brake
Owen Brake3 месяцев назад

amazing, can you share the absolute number of episodes in the pre-training dataset?

Фото профиля clankr
clankr3 месяцев назад

These results are pretty impressive. So this is what the Sunday team meant when they said most robotics companies were collecting data the wrong way.

Фото профиля Ted Xiao
Ted Xiao3 месяцев назад

@RemiCadene Congrats, very exciting!

Фото профиля Tony Zhao
Tony Zhao3 месяцев назад

@RemiCadene Thank you Ted! 🙏

Фото профиля Max Mclaughlin
Max Mclaughlin3 месяцев назад

Impressive, but let's see it do all laundry, not just folding. Folding doesn't really change depending on where you are, full laundry definitely does

Фото профиля Sampriti Bhattacharyya
Sampriti Bhattacharyya3 месяцев назад

This is cool, definitely one of the hard problems

Фото профиля Tony Zhao
Tony Zhao3 месяцев назад

It was very difficult. We weren’t sure if it’s possible beginning of the year.

Фото профиля Eren Chen
Eren Chen3 месяцев назад

We all know what Green cap in Chinese culture means…

Фото профиля Ofir Ozeri
Ofir Ozeri3 месяцев назад

@philfung What about the emergent behavior you said you saw? That’s amazing but you left us curious!

Фото профиля Haoru Xue
Haoru Xue3 месяцев назад

Congrats! Looking solid!

Фото профиля Tony Zhao
Tony Zhao3 месяцев назад

Thank you Haoru! Precise posting after vague posting 😉

Фото профиля Ivan Kirigin
Ivan Kirigin3 месяцев назад

How do you think about a "solve" when the cost of errors scales? Like 99% in self driving is unlaunchable.

Фото профиля Tony Zhao
Tony Zhao3 месяцев назад

Precisely. The performance threshold could differ across applications. What a Solve highlights is that we should not omit Scope and Adaptation Budget when reporting the success rate. The Solve framework does not carry any opinion about the performance threshold itself.

Фото профиля Shuang Li
Shuang Li3 месяцев назад

Very cool congrats!

Фото профиля Tony Zhao
Tony Zhao3 месяцев назад

Thank you Shuang!

Фото профиля Dhruv Batra
Dhruv Batra3 месяцев назад

Very cool videos! I appreciate the detailed descriptions of the evals in the blog. Is there a model card somewhere?

Фото профиля Tony Zhao
Tony Zhao3 месяцев назад

Thank you Dhruv. We don't have it right now but might release it in the future with the full release 🫡

Фото профиля Dhruv Batra
Dhruv Batra3 месяцев назад

Look forward to reading it if/when you do. Also signed up for the waitlist — kudos on the launch!

Фото профиля Sourish Jasti
Sourish Jasti3 месяцев назад

so cool, congrats

Фото профиля Vikash Kumar
Vikash Kumar3 месяцев назад

@wenlong_huang @tonyzzhao - it’s incredible how far ACT is going! Let there be no boundaries. Kudos to entire @sundayrobotics team

Фото профиля Zeeshan Patel
Zeeshan Patel3 месяцев назад

@arthurallshire Congrats Tony, really cool results!

Фото профиля Tony Zhao
Tony Zhao3 месяцев назад

@arthurallshire Thank you Zeeshan! Hope all is well and congrats on the new journey!

Похожие видео