Video wird geladen...

Video konnte nicht geladen werden

Zur Startseite

Introducing ACT-2 Preview The first robotics model to unify broad generalization with high reliability. A single fine-tuning example can teach Memo a new behavior that generalizes. Zero shot, real unseen homes, 99% success rate.

1,060,037 Aufrufe • vor 3 Monaten •via X (Twitter)

61 Kommentare

Profilbild von Tony Zhao
Tony Zhaovor 3 Monaten

We completed the most rigorous generalization test to date. Across 785 trials in 31 unseen environments, ACT-2 achieved 99.1% success in laundry folding. ACT-2 also achieved human-level fold quality: receiving an average rating of 4.72/5, with 98.3% earning four or five stars.

Profilbild von Tony Zhao
Tony Zhaovor 3 Monaten

Tons of emergent behavior along the way: - Picking up clothes off the ground - Handling baby wear to 8XL shirts - Robustness against adversarial disturbances and lighting

Profilbild von Tony Zhao
Tony Zhaovor 3 Monaten

We call this a Solve. Progress in robotics is difficult to measure because demos vary by setup. Demo ≠ Solved. A Solve declares two boundaries: scope and adaptation cost. Without both, 99% has no context.

Profilbild von Tony Zhao
Tony Zhaovor 3 Monaten

We found a general recipe for Solves: scale pretraining, then hill-climb with minimal in-house data. For the first time, one fine-tuning example can teach a new behavior that generalizes. Below: 4 folding strategies, learned from one example each and tested on held-out setups.

Profilbild von Tony Zhao
Tony Zhaovor 3 Monaten

Quantitatively, we measure the generalization gap as the difference between in-domain and out-of-domain performance. As we scale up pretraining, the gap falls sharply. This makes in-house performance a reliable predictor of performance in the wild.

Profilbild von Tony Zhao
Tony Zhaovor 3 Monaten

This property allows us to hill-climb performance in our office, and trust those gains to hold in unseen homes Our fleet of Memos runs in parallel to rapidly advance reliability, quality, and speed. Left: fleet-scale improvement in-house Right: Memo working across unseen homes

Profilbild von Tony Zhao
Tony Zhaovor 3 Monaten

Laundry is our first Solve of many. Our recipe is so general that scaling data and compute gives us predictable improvements. Unlocking one Solve accelerates the next Solve. The same ACT-2 model is learning to vacuum, organize toys, zip clothing, and turn pants inside out.

Profilbild von Tony Zhao
Tony Zhaovor 3 Monaten

This fall, ACT-2’s first Solve enters homes through our Beta Program, the final step towards fully autonomous home robot deployment. Full technical report:

Profilbild von Sholto Douglas
Sholto Douglasvor 3 Monaten

So cool, can't wait to have one at my place

Profilbild von Tony Zhao
Tony Zhaovor 3 Monaten

Very soon 😉

Profilbild von Roei Herzig
Roei Herzigvor 3 Monaten

Is it possible to get access to the pretrained model? We recently developed a mechanistic interpretability concept for fast-adaptation by tuning only the relevant parts of the network: If the pretrained model is strong, we can do magic. Give us access, and we can unlock continual learning together ;)

Profilbild von Tony Zhao
Tony Zhaovor 3 Monaten

This is very cool, checking it out now!

Profilbild von Camilla Guo
Camilla Guovor 3 Monaten

sending this to my mom

Profilbild von Dhaval Shroff
Dhaval Shroffvor 3 Monaten

Very cool to see this kind of generalization @tonyzzhao

Profilbild von Tony Zhao
Tony Zhaovor 3 Monaten

Just ask for generalization! @ericjang11

Profilbild von Tim Zaman
Tim Zamanvor 3 Monaten

But are you still shipping this year

Profilbild von Tony Zhao
Tony Zhaovor 3 Monaten

Have you applied!

Profilbild von Yuanhao Qu
Yuanhao Quvor 3 Monaten

@timzaman Congrats @tonyzzhao on the launch!!! Can’t wait to have one at home.

Profilbild von Tony Zhao
Tony Zhaovor 3 Monaten

@timzaman Thank you Yuanhao! Apply here!

Profilbild von Kate Park
Kate Parkvor 3 Monaten

Congrats! data engine end to end ftw 💪

Profilbild von Remi Cadene
Remi Cadenevor 3 Monaten

Extremely cool ;)

Profilbild von Tony Zhao
Tony Zhaovor 3 Monaten

Thank you Remi!! Congrats on the humanoid release. So cool.

Profilbild von Jason Ma
Jason Mavor 3 Monaten

congrats tony! awesome to see generalization + reliability

Profilbild von Tony Zhao
Tony Zhaovor 3 Monaten

Thank you Jason! 🙏

Profilbild von Caitlin Kalinowski
Caitlin Kalinowskivor 3 Monaten

The wanting intensifies! Does this mean I will be able to teach it to fold things my way? 😍

Profilbild von Tony Zhao
Tony Zhaovor 3 Monaten

Exactly! I'm honestly quite surprised that it just works with SFT..!

Profilbild von Perry Jia
Perry Jiavor 3 Monaten

Proud of the team for this achievement

Profilbild von Yu Xiang
Yu Xiangvor 3 Monaten

@RemiCadene Very Impressive. Congratulations! There is a snapshot where the head camera is blocked, and the robot can still keep going. How to explain this behavior?

Profilbild von Tony Zhao
Tony Zhaovor 3 Monaten

@RemiCadene ACT-2 takes all 5 cameras (one head, two on each hand) as input. And sometimes dropout is all you need!

Profilbild von Ryan Julian
Ryan Julianvor 3 Monaten

Congratulations @sundayrobotics team! Incredible work

Profilbild von Tony Zhao
Tony Zhaovor 3 Monaten

@sundayrobotics Thank you Ryan!

Profilbild von Machine Space
Machine Spacevor 3 Monaten

@Scobleizer I can’t believe nobody has asked for the holy grail of folding: The King Size Fitted Sheet.

Profilbild von Aaref Hilaly
Aaref Hilalyvor 3 Monaten

big breakthrough - congrats!

Profilbild von Tony Zhao
Tony Zhaovor 3 Monaten

Thank you Aaref for being part of this journey ❤️

Profilbild von Rahul
Rahulvor 3 Monaten

This all seemed so far away till last year

Profilbild von Jiafei Duan
Jiafei Duanvor 3 Monaten

Really cool work! It will be cooler if the weights could be open source, seems to be a common question here.

Profilbild von Lindon Gao
Lindon Gaovor 3 Monaten

Congrats @tonyzzhao & team!

Profilbild von Tony Zhao
Tony Zhaovor 3 Monaten

Thank you Lindon!!

Profilbild von Jake Roggenbuck
Jake Roggenbuckvor 3 Monaten

Generalizing reliability!!

Profilbild von Owen Brake
Owen Brakevor 3 Monaten

amazing, can you share the absolute number of episodes in the pre-training dataset?

Profilbild von clankr
clankrvor 3 Monaten

These results are pretty impressive. So this is what the Sunday team meant when they said most robotics companies were collecting data the wrong way.

Profilbild von Ted Xiao
Ted Xiaovor 3 Monaten

@RemiCadene Congrats, very exciting!

Profilbild von Tony Zhao
Tony Zhaovor 3 Monaten

@RemiCadene Thank you Ted! 🙏

Profilbild von Max Mclaughlin
Max Mclaughlinvor 3 Monaten

Impressive, but let's see it do all laundry, not just folding. Folding doesn't really change depending on where you are, full laundry definitely does

Profilbild von Sampriti Bhattacharyya
Sampriti Bhattacharyyavor 3 Monaten

This is cool, definitely one of the hard problems

Profilbild von Tony Zhao
Tony Zhaovor 3 Monaten

It was very difficult. We weren’t sure if it’s possible beginning of the year.

Profilbild von Eren Chen
Eren Chenvor 3 Monaten

We all know what Green cap in Chinese culture means…

Profilbild von Ofir Ozeri
Ofir Ozerivor 3 Monaten

@philfung What about the emergent behavior you said you saw? That’s amazing but you left us curious!

Profilbild von Haoru Xue
Haoru Xuevor 3 Monaten

Congrats! Looking solid!

Profilbild von Tony Zhao
Tony Zhaovor 3 Monaten

Thank you Haoru! Precise posting after vague posting 😉

Profilbild von Ivan Kirigin
Ivan Kiriginvor 3 Monaten

How do you think about a "solve" when the cost of errors scales? Like 99% in self driving is unlaunchable.

Profilbild von Tony Zhao
Tony Zhaovor 3 Monaten

Precisely. The performance threshold could differ across applications. What a Solve highlights is that we should not omit Scope and Adaptation Budget when reporting the success rate. The Solve framework does not carry any opinion about the performance threshold itself.

Profilbild von Shuang Li
Shuang Livor 3 Monaten

Very cool congrats!

Profilbild von Tony Zhao
Tony Zhaovor 3 Monaten

Thank you Shuang!

Profilbild von Dhruv Batra
Dhruv Batravor 3 Monaten

Very cool videos! I appreciate the detailed descriptions of the evals in the blog. Is there a model card somewhere?

Profilbild von Tony Zhao
Tony Zhaovor 3 Monaten

Thank you Dhruv. We don't have it right now but might release it in the future with the full release 🫡

Profilbild von Dhruv Batra
Dhruv Batravor 3 Monaten

Look forward to reading it if/when you do. Also signed up for the waitlist — kudos on the launch!

Profilbild von Sourish Jasti
Sourish Jastivor 3 Monaten

so cool, congrats

Profilbild von Vikash Kumar
Vikash Kumarvor 3 Monaten

@wenlong_huang @tonyzzhao - it’s incredible how far ACT is going! Let there be no boundaries. Kudos to entire @sundayrobotics team

Profilbild von Zeeshan Patel
Zeeshan Patelvor 3 Monaten

@arthurallshire Congrats Tony, really cool results!

Profilbild von Tony Zhao
Tony Zhaovor 3 Monaten

@arthurallshire Thank you Zeeshan! Hope all is well and congrats on the new journey!

Ähnliche Videos