Загрузка видео...

Не удалось загрузить видео

Возникла проблема при загрузке этого видео. Это может быть связано с временными проблемами сети или видео может быть недоступно.

На главную

We found a general recipe for Solves: scale pretraining, then hill-climb with minimal in-house data. For the first time, one fine-tuning example can teach a new behavior that generalizes. Below: 4 folding strategies, learned from one example each and tested on held-out setups.

Tony Zhao

150,836 subscribers

171,531 просмотров • 3 месяцев назад •via X (Twitter)

Наука и технологии

Anya Rossi• Live Now

Private livecam show

Комментарии: 10

Фото профиля Tony Zhao

Tony Zhao3 месяцев назад

We call this a Solve. Progress in robotics is difficult to measure because demos vary by setup. Demo ≠ Solved. A Solve declares two boundaries: scope and adaptation cost. Without both, 99% has no context.

Фото профиля Tony Zhao

Tony Zhao3 месяцев назад

Tons of emergent behavior along the way: - Picking up clothes off the ground - Handling baby wear to 8XL shirts - Robustness against adversarial disturbances and lighting

Фото профиля Tony Zhao

Tony Zhao3 месяцев назад

We completed the most rigorous generalization test to date. Across 785 trials in 31 unseen environments, ACT-2 achieved 99.1% success in laundry folding. ACT-2 also achieved human-level fold quality: receiving an average rating of 4.72/5, with 98.3% earning four or five stars.

Фото профиля Tony Zhao

Tony Zhao3 месяцев назад

Introducing ACT-2 Preview The first robotics model to unify broad generalization with high reliability. A single fine-tuning example can teach Memo a new behavior that generalizes. Zero shot, real unseen homes, 99% success rate.

Фото профиля Tony Zhao

Tony Zhao3 месяцев назад

Quantitatively, we measure the generalization gap as the difference between in-domain and out-of-domain performance. As we scale up pretraining, the gap falls sharply. This makes in-house performance a reliable predictor of performance in the wild.

Фото профиля Tony Zhao

Tony Zhao3 месяцев назад

This property allows us to hill-climb performance in our office, and trust those gains to hold in unseen homes Our fleet of Memos runs in parallel to rapidly advance reliability, quality, and speed. Left: fleet-scale improvement in-house Right: Memo working across unseen homes

Фото профиля Tony Zhao

Tony Zhao3 месяцев назад

Laundry is our first Solve of many. Our recipe is so general that scaling data and compute gives us predictable improvements. Unlocking one Solve accelerates the next Solve. The same ACT-2 model is learning to vacuum, organize toys, zip clothing, and turn pants inside out.

Фото профиля Tony Zhao

Tony Zhao3 месяцев назад

This fall, ACT-2’s first Solve enters homes through our Beta Program, the final step towards fully autonomous home robot deployment. Full technical report:

Фото профиля Pierre Delecto Too 🙈🙉🙊

Pierre Delecto Too 🙈🙉🙊3 месяцев назад

Can it take my laundry downstairs, load it into the washer & dryer, and haul everything back upstairs?

Фото профиля Tensor Templar

Tensor Templar3 месяцев назад

How big is it

Похожие видео

Introducing ACT-2 Preview The first robotics model to unify broad generalization with high reliability. A single fine-tuning example can teach Memo a new behavior that generalizes. Zero shot, real unseen homes, 99% success rate.

Introducing ACT-2 Preview The first robotics model to unify broad generalization with high reliability. A single fine-tuning example can teach Memo a new behavior that generalizes. Zero shot, real unseen homes, 99% success rate.

Tony Zhao

1,061,729 просмотров • 3 месяцев назад

Introducing GEN-1.5, a one-shot learner. It can learn new tasks in a few seconds. Show it what to do, and it generalizes. This capability emerged from pretraining on physical data at scale, as a step towards our mission of building general intelligence for the physical world.

Introducing GEN-1.5, a one-shot learner. It can learn new tasks in a few seconds. Show it what to do, and it generalizes. This capability emerged from pretraining on physical data at scale, as a step towards our mission of building general intelligence for the physical world.

Generalist

3,446,526 просмотров • 2 месяцев назад

We found that to match the accuracy that S1 can achieve with just one example of prompting, current VLA models would need to be post-trained with 50-100 hours of data collection followed by fine-tuning!

We found that to match the accuracy that S1 can achieve with just one example of prompting, current VLA models would need to be post-trained with 50-100 hours of data collection followed by fine-tuning!

Skild AI

295,652 просмотров • 2 месяцев назад

For me the beginner lesson is fine. But the example was a bad one to teach it in.

For me the beginner lesson is fine. But the example was a bad one to teach it in.

chinedu🦀

19,155 просмотров • 4 месяцев назад

Introducing S1, our new foundation model that learns from one example. It can be taught 10-minute long tasks that it has never seen before, from one video prompt without any fine-tuning. Watch S1 operate in real-time via in-context learning:

Introducing S1, our new foundation model that learns from one example. It can be taught 10-minute long tasks that it has never seen before, from one video prompt without any fine-tuning. Watch S1 operate in real-time via in-context learning:

Skild AI

3,560,311 просмотров • 2 месяцев назад

New short course on Pretraining LLMs! Developed with Upstage and taught by their CEO Sung Kim and CSO Lucy Park. While prompting or fine-tuning existing models works well for many general language tasks, pretraining is valuable for specialized domains or languages with limited representation in current models. This course walks you through the LLM pretraining pipeline: 1. Data preparation: Learn to source, clean, and prepare training data using HuggingFace. 2. Model architecture: Configure transformer networks, including modifying existing models. 3. Training: Set up and run training using open-source libraries. 4. Evaluation: Benchmark performance using popular evaluation strategies. As an example use case, you'll also compare the output of a base model with its fine-tuned and further pretrained variants, to see the impact of pretraining on a model's ability to write Python. The course also explores an innovative technique called depth up-scaling, which Upstage used to train their Solar model family, reducing pretraining compute costs by up to 70%. This technique works by first duplicating layers of a smaller pretrained model to form a larger model, and then further pretraining the result. Sign up here!

New short course on Pretraining LLMs! Developed with Upstage and taught by their CEO Sung Kim and CSO Lucy Park. While prompting or fine-tuning existing models works well for many general language tasks, pretraining is valuable for specialized domains or languages with limited representation in current models. This course walks you through the LLM pretraining pipeline: 1. Data preparation: Learn to source, clean, and prepare training data using HuggingFace. 2. Model architecture: Configure transformer networks, including modifying existing models. 3. Training: Set up and run training using open-source libraries. 4. Evaluation: Benchmark performance using popular evaluation strategies. As an example use case, you'll also compare the output of a base model with its fine-tuned and further pretrained variants, to see the impact of pretraining on a model's ability to write Python. The course also explores an innovative technique called depth up-scaling, which Upstage used to train their Solar model family, reducing pretraining compute costs by up to 70%. This technique works by first duplicating layers of a smaller pretrained model to form a larger model, and then further pretraining the result. Sign up here!

Andrew Ng

85,781 просмотров • 2 лет назад

Evolution happens to populations. Useful genes become common in a group over generations, then spread to other groups that test them in their own environments. The speed depends on generation time, 20 minutes for a bacterium but about 25 years for an elephant. Using LLMs to evolve better models often looks like an isolated herd of elephants. Each search runs alone on one dataset, and for large models every generation costs a long training run on the full data. Our new paper, "Discovered, Not Designed," introduces Population Evolution. It has two goals, to discover better models and to discover them faster. To discover better model designs, the searches share what they find. Several searches run at the same time, each on its own dataset. When one finds a code change that helps, the same change is retrained on the others. While nature keeps no record of the genes that failed to help, Population Evolution does. All peer results, including weak or unsuccessful transfers, are added to a shared ledger, and the LLM reads that ledger before it writes the next change. To discover them faster, the searches do not train on the full data, because that is what makes each generation slow. We found that a search can run on as little as 2% or 5% of the data and still find changes that hold up in full training, as long as the other searches check each change first. In Population Evolution, a change that works on one slice must also work on at least two other slices, then on two larger 10% sets. Only then does it get a full training run. We tested Population Evolution on RMD-Bench, a benchmark we also introduce for evaluating model discovery. It covers five task families (LLM pretraining, VLM pretraining, RL algorithm discovery, ranking, and watch-time prediction) with 34 instances in total. Each one comes with a reference implementation and a fixed training and evaluation protocol, so a change found on one dataset or data scale can be retrained and scored on another, and other discovery methods can be compared under the same conditions. The numbers below are a sneak peek at the two most expensive families of experiments. The LLM pretraining task starts from a 185M-parameter Gemma 3-style model, and the VLM pretraining task starts from a Qwen3.5-style model with about 646M trainable parameters. The baseline runs the same kind of search directly on the full data, at the same total GPU compute. Population Evolution finds better models. The best relative accuracy gain for LLM pretraining goes from 0.92% to 2.48%. The best loss reduction for VLM pretraining goes from 5.05% to 8.78%. Population Evolution finds them faster. In LLM pretraining, each search gets 100 generations where a full-data search gets 14. In VLM pretraining, 57 of 61 full training runs met the success threshold, against 12 of 100. Evolution works because many groups pass along what helps and get many generations to do it, and Population Evolution gives LLM-driven model search these prerequisites. Read our paper at for full implementation details and results. The benchmark data and code will be made publicly available shortly.

Evolution happens to populations. Useful genes become common in a group over generations, then spread to other groups that test them in their own environments. The speed depends on generation time, 20 minutes for a bacterium but about 25 years for an elephant. Using LLMs to evolve better models often looks like an isolated herd of elephants. Each search runs alone on one dataset, and for large models every generation costs a long training run on the full data. Our new paper, "Discovered, Not Designed," introduces Population Evolution. It has two goals, to discover better models and to discover them faster. To discover better model designs, the searches share what they find. Several searches run at the same time, each on its own dataset. When one finds a code change that helps, the same change is retrained on the others. While nature keeps no record of the genes that failed to help, Population Evolution does. All peer results, including weak or unsuccessful transfers, are added to a shared ledger, and the LLM reads that ledger before it writes the next change. To discover them faster, the searches do not train on the full data, because that is what makes each generation slow. We found that a search can run on as little as 2% or 5% of the data and still find changes that hold up in full training, as long as the other searches check each change first. In Population Evolution, a change that works on one slice must also work on at least two other slices, then on two larger 10% sets. Only then does it get a full training run. We tested Population Evolution on RMD-Bench, a benchmark we also introduce for evaluating model discovery. It covers five task families (LLM pretraining, VLM pretraining, RL algorithm discovery, ranking, and watch-time prediction) with 34 instances in total. Each one comes with a reference implementation and a fixed training and evaluation protocol, so a change found on one dataset or data scale can be retrained and scored on another, and other discovery methods can be compared under the same conditions. The numbers below are a sneak peek at the two most expensive families of experiments. The LLM pretraining task starts from a 185M-parameter Gemma 3-style model, and the VLM pretraining task starts from a Qwen3.5-style model with about 646M trainable parameters. The baseline runs the same kind of search directly on the full data, at the same total GPU compute. Population Evolution finds better models. The best relative accuracy gain for LLM pretraining goes from 0.92% to 2.48%. The best loss reduction for VLM pretraining goes from 5.05% to 8.78%. Population Evolution finds them faster. In LLM pretraining, each search gets 100 generations where a full-data search gets 14. In VLM pretraining, 57 of 61 full training runs met the success threshold, against 12 of 100. Evolution works because many groups pass along what helps and get many generations to do it, and Population Evolution gives LLM-driven model search these prerequisites. Read our paper at for full implementation details and results. The benchmark data and code will be made publicly available shortly.

Zhuokai Zhao

925,744 просмотров • 28 дней назад

GEN-1.5, our latest embodied foundation model, can learn new tasks prompted with 3 - 12 seconds of a single demonstration, no gradient updates or fine-tuning. It generalizes prompts to new situations, recovers from mistakes, and improvises new strategies to reach the same goal.

GEN-1.5, our latest embodied foundation model, can learn new tasks prompted with 3 - 12 seconds of a single demonstration, no gradient updates or fine-tuning. It generalizes prompts to new situations, recovers from mistakes, and improvises new strategies to reach the same goal.

Generalist

138,196 просмотров • 2 месяцев назад

Introducing Natura-1, nature will's first general foundation model for the physical world. It is a single model that follows plain-language instructions zero-shot and carries them out with dexterous hands. - 50B MoE, pretrained on 1M hours of egocentric human experience. - Drives 20-DoF dexterous hands from plain language, zero-shot, in a 100ms loop. - Scaling human data from ~44K to 1M hours cut action MSE on held-out robot data by 67.7%. - Built on in-house data and compute infra, it handles fine-grained long-horizon tasks and a ~30-minute zero-shot interaction in one continuous take. The general model exists, and it can only get stronger.

Introducing Natura-1, nature will's first general foundation model for the physical world. It is a single model that follows plain-language instructions zero-shot and carries them out with dexterous hands. - 50B MoE, pretrained on 1M hours of egocentric human experience. - Drives 20-DoF dexterous hands from plain language, zero-shot, in a 100ms loop. - Scaling human data from ~44K to 1M hours cut action MSE on held-out robot data by 67.7%. - Built on in-house data and compute infra, it handles fine-grained long-horizon tasks and a ~30-minute zero-shot interaction in one continuous take. The general model exists, and it can only get stronger.

Ning Ding

21,938 просмотров • 22 дней назад

Can we scale 4D pretraining to learn general space-time representations that reconstruct an object from a few views at any time to any view at any other time? Introducing 4D-LRM: a Large Space-Time Reconstruction Model that ... 🔹 Predicts 4D Gaussian primitives directly from multi-view tokens (no motion vectors, no HexPlane); 🔹 Uses a clean, minimal Transformer backbone; 🔹 Generalizes with fast, high-quality feedforward rendering at any view and infinite frame rate. Check out more interactive demos and scaling behaviors on our homepage/paper. 👉Website: 👉Paper:

Can we scale 4D pretraining to learn general space-time representations that reconstruct an object from a few views at any time to any view at any other time? Introducing 4D-LRM: a Large Space-Time Reconstruction Model that ... 🔹 Predicts 4D Gaussian primitives directly from multi-view tokens (no motion vectors, no HexPlane); 🔹 Uses a clean, minimal Transformer backbone; 🔹 Generalizes with fast, high-quality feedforward rendering at any view and infinite frame rate. Check out more interactive demos and scaling behaviors on our homepage/paper. 👉Website: 👉Paper:

Martin Ziqiao Ma

21,851 просмотров • 1 год назад

Revolutionizing Move Programming with OpenLedger In this demo, we showcase how Move datasets contributed by data providers to OpenLedger’s datanets are used to fine-tune specialized models with LoRA fine-tuning. As seen in the video, we showcase an example on how builders can deploy a Move-specialized model that powers Co-pilot agents using our no-code model fine-tuning platform. This is the future of AI and Web3 innovation. Watch this space to see more specialised models and data feeds being built for next generation agents on top of OpenLedger #Move

Revolutionizing Move Programming with OpenLedger In this demo, we showcase how Move datasets contributed by data providers to OpenLedger’s datanets are used to fine-tune specialized models with LoRA fine-tuning. As seen in the video, we showcase an example on how builders can deploy a Move-specialized model that powers Co-pilot agents using our no-code model fine-tuning platform. This is the future of AI and Web3 innovation. Watch this space to see more specialised models and data feeds being built for next generation agents on top of OpenLedger #Move

OpenLedger

61,697 просмотров • 1 год назад

Fine-tuning Mistral 7B with LoRA on a 32 GB M1 (laptop!) in MLX Updated example uses less RAM + support for custom datasets 🚀

Fine-tuning Mistral 7B with LoRA on a 32 GB M1 (laptop!) in MLX Updated example uses less RAM + support for custom datasets 🚀

Awni Hannun

148,211 просмотров • 2 лет назад

Ever feel like God can’t forgive your failures? He can and He does. I’m thankful for that! Peter’s life is a beautiful example. It’s just one of the stories in my new book #NothingIsImpossibleWithGod - out one week from today and available for pre-order now!

Ever feel like God can’t forgive your failures? He can and He does. I’m thankful for that! Peter’s life is a beautiful example. It’s just one of the stories in my new book #NothingIsImpossibleWithGod - out one week from today and available for pre-order now!

Shannon Bream

16,024 просмотров • 8 месяцев назад

A British surgeon tells the BBC what he saw in Gaza. "I can remember one child for example who was 7, Amir... He heard a drone above him, turned round, and then it fired on him. And we had many many reports like that from children and adults."

A British surgeon tells the BBC what he saw in Gaza. "I can remember one child for example who was 7, Amir... He heard a drone above him, turned round, and then it fired on him. And we had many many reports like that from children and adults."

Paul Graham

379,262 просмотров • 3 месяцев назад

Introducing Natura-1, our first general foundation model for the physical world. It follows natural-language instructions zero-shot and turns them into action with dexterous hands. Natura-1 is a 50-billion-parameter mixture-of-experts (MoE) model pretrained on one million hours of egocentric human experience. On top of this foundation, we conduct efficient post-training on a limited amount of robot data to adapt pretrained world knowledge to dexterous manipulation, yielding a model that generalizes to a wide range of unseen scenarios. Natura-1 follows complex natural-language instructions zero-shot, reasoning through spatial constraints, multi-object relationships, and negative conditions. It can also autonomously carry out long and continuous workflows to perform long-horizon executions. Natura-1 provides early evidence that generalization—and even emergent behavior—can coexist with the complexity of dexterous hands. Further progress depends on the full stack: infrastructure for faster training and real-time inference, pretraining data at greater scale and coverage, and models that transfer to robots with less adaptation. We expect it to be a long-term research and engineering effort.

Introducing Natura-1, our first general foundation model for the physical world. It follows natural-language instructions zero-shot and turns them into action with dexterous hands. Natura-1 is a 50-billion-parameter mixture-of-experts (MoE) model pretrained on one million hours of egocentric human experience. On top of this foundation, we conduct efficient post-training on a limited amount of robot data to adapt pretrained world knowledge to dexterous manipulation, yielding a model that generalizes to a wide range of unseen scenarios. Natura-1 follows complex natural-language instructions zero-shot, reasoning through spatial constraints, multi-object relationships, and negative conditions. It can also autonomously carry out long and continuous workflows to perform long-horizon executions. Natura-1 provides early evidence that generalization—and even emergent behavior—can coexist with the complexity of dexterous hands. Further progress depends on the full stack: infrastructure for faster training and real-time inference, pretraining data at greater scale and coverage, and models that transfer to robots with less adaptation. We expect it to be a long-term research and engineering effort.

nature will

1,043,474 просмотров • 22 дней назад

Time for another recipe from my new cookbook Nights Out In The Kitchen. This time, Basque Cheesecake, which I first tried at Brat back in the day. This recipe is courtesy of James Jocky Petrie and it’s a cracker. You can order the book at the link below.

Time for another recipe from my new cookbook Nights Out In The Kitchen. This time, Basque Cheesecake, which I first tried at Brat back in the day. This recipe is courtesy of James Jocky Petrie and it’s a cracker. You can order the book at the link below.

Jay Rayner

29,083 просмотров • 2 месяцев назад

Every order with VistaPrint is a win. For example, we call this one the Wrexham AFC Hat on a Hat-Trick.

Every order with VistaPrint is a win. For example, we call this one the Wrexham AFC Hat on a Hat-Trick.

VistaPrint

2,015,033 просмотров • 3 лет назад

1/5 What does it take to scale pretraining to 20+ data modalities at once? Improved data mixing methods are one important ingredient. Today, we share how we scale data mixing to thousands of sources for DNA and natural language pretraining of Omnii, our multimodal genome model.

1/5 What does it take to scale pretraining to 20+ data modalities at once? Improved data mixing methods are one important ingredient. Today, we share how we scale data mixing to thousands of sources for DNA and natural language pretraining of Omnii, our multimodal genome model.

Radical Numerics

367,277 просмотров • 23 дней назад

How can you solve complex tasks using a Large Language Model? Here is a 2-minute introduction to everything you need to know to 10x the quality of your results. Let's talk about three techniques, in order of complexity, starting with the easiest one: • In-Context Learning • Indexing + In-Context Learning • Fine-tuning In-Context Learning The team that trained GPT-3 found something they couldn't explain: You can condition a model using examples of how you want it to behave. I included an example prompt in the attached video. You can "teach" the model how you want it to interpret questions, select the correct answers, and format the results by giving a few examples. You can also give specific knowledge to the model that will be helpful when formulating answers. We call this approach "grounding the model." There's another example in the video. Indexing + In-Context Learning Unfortunately, there is a limit to how much data you can include in a prompt. We call this the "context size." One version of GPT-4 supports a context of approximately 6,000 words, while the other supports 25,000 words. Although this sounds like a lot, many applications need more than that. Imagine you wrote a book and want to build an application to answer any questions about your story. What happens if your book is longer than the context? That's where Indexing comes in. Using a model, you can turn every book passage into an embedding. These are vectors, numbers that "encode" the passage's text. You can then store these embeddings in a particular database that supports fast retrieval of these vectors. You can then turn any question into an embedding and search the database for the list of passages that are similar to that query. Instead of using the entire book to ask the model, you can now use the relevant passages as in-context information, effectively working around the context size limitation. Fine-tuning Fine-tuning can give you an extra boost to get reliable outputs from your LLM. It is, however, the most complex approach on the list. There are different approaches to fine-tuning a model with your data. A popular technique is to process your data with your LLM and use the outputs to train a new classifier that solves your specific task. Notice that here you aren't modifying the LLM. Instead, you are chaining it with your trained classifier. Another approach is to modify the parameters of the LLM using your data. Think of this as "rewiring" the model in a way that solves your particular task. The results and costs will vary depending on how many layers you want to fine-tune from the original model. Many companies think that fine-tuning is the solution to their problems. In my experience, many will benefit from exploring the other two approaches. I love explaining Machine Learning and Artificial Intelligence ideas. If you enjoy in-depth content like this, follow me Santiago so you don't miss what comes next.

How can you solve complex tasks using a Large Language Model? Here is a 2-minute introduction to everything you need to know to 10x the quality of your results. Let's talk about three techniques, in order of complexity, starting with the easiest one: • In-Context Learning • Indexing + In-Context Learning • Fine-tuning In-Context Learning The team that trained GPT-3 found something they couldn't explain: You can condition a model using examples of how you want it to behave. I included an example prompt in the attached video. You can "teach" the model how you want it to interpret questions, select the correct answers, and format the results by giving a few examples. You can also give specific knowledge to the model that will be helpful when formulating answers. We call this approach "grounding the model." There's another example in the video. Indexing + In-Context Learning Unfortunately, there is a limit to how much data you can include in a prompt. We call this the "context size." One version of GPT-4 supports a context of approximately 6,000 words, while the other supports 25,000 words. Although this sounds like a lot, many applications need more than that. Imagine you wrote a book and want to build an application to answer any questions about your story. What happens if your book is longer than the context? That's where Indexing comes in. Using a model, you can turn every book passage into an embedding. These are vectors, numbers that "encode" the passage's text. You can then store these embeddings in a particular database that supports fast retrieval of these vectors. You can then turn any question into an embedding and search the database for the list of passages that are similar to that query. Instead of using the entire book to ask the model, you can now use the relevant passages as in-context information, effectively working around the context size limitation. Fine-tuning Fine-tuning can give you an extra boost to get reliable outputs from your LLM. It is, however, the most complex approach on the list. There are different approaches to fine-tuning a model with your data. A popular technique is to process your data with your LLM and use the outputs to train a new classifier that solves your specific task. Notice that here you aren't modifying the LLM. Instead, you are chaining it with your trained classifier. Another approach is to modify the parameters of the LLM using your data. Think of this as "rewiring" the model in a way that solves your particular task. The results and costs will vary depending on how many layers you want to fine-tune from the original model. Many companies think that fine-tuning is the solution to their problems. In my experience, many will benefit from exploring the other two approaches. I love explaining Machine Learning and Artificial Intelligence ideas. If you enjoy in-depth content like this, follow me Santiago so you don't miss what comes next.

Santiago

384,573 просмотров • 3 лет назад