Загрузка видео...

Не удалось загрузить видео

Возникла проблема при загрузке этого видео. Это может быть связано с временными проблемами сети или видео может быть недоступно.

На главную

Are robots ready to accelerate science labs? We created the WetLabs Benchmark to test 9 critical tasks, from easy to hard, and let 3 leading models take 20 attempts per task Astra leads but it comes close...

Josh

4,338 subscribers

69,086 просмотров • 14 дней назад •via X (Twitter)

Наука и технологии

Anya Rossi• Live Now

Private livecam show

Комментарии: 25

Фото профиля Josh

Josh14 дней назад

Lab work demands delicate and precise movements but Fable 5.1 crushed one of the beakers despite instructions to be gentle Other failure modes showed missed grasps due to shadows, grasps from inside the beaker which lead to cross contamination, and bad planning for the tasks

Фото профиля Josh

Josh14 дней назад

Accidents like a knocked over beaker didn’t end the run Opus 5.5 stood it back up and kept going. In another run, it correctly pulled a stirring rod from a fallen rack. Neither attempt fully succeeded, but both showed signs of mistake recovery and emergent behaviors

Фото профиля Josh

Josh14 дней назад

From successes, models were able to do a mix sequence, place a test tube, and pick a top plate from a stack of 24-well plates Human evaluators recorded 110 successes, 246 partials and 184 failures across 540 attempts Astra led with 25.6% and Opus 5.5 closely followed at 23.3%

Фото профиля Josh

Josh14 дней назад

Hallucination on task completion is still a challenge Models claimed 192 attempts as complete despite human evaluators finding 79 partial completions and 10 failures Astra hallucinated the least, whereas Opus 5.5 had to be corrected by humans on over half the completions

Фото профиля Josh

Josh14 дней назад

Astra exceeds Opus 5.5 on science lab tasks, but it burned ~4x the tokens It had the highest success rate and shortest median attempt time, but the highest mean estimated API cost: $8.64 per attempt Opus 5.5 nearly matched its success rate at $2.39 per attempt

Фото профиля Josh

Josh14 дней назад

Evals are the backbone of understanding and improving model capabilities The WetLab Benchmark will be updated as capabilities progress and we unlock a scale up of scientific work in the real world Explore the deep dive and all runs here:

Фото профиля Josh

Josh14 дней назад

Great work to the team across this effort @itsghvr @PranavKuppili @camrzadki @Kaiwen66lc Zac Gong @jasontheutopian

Фото профиля Josh

Josh14 дней назад

@PranavKuppili @camrzadki @Kaiwen66lc @jasontheutopian + shoutout @ryan_punamiya and friends for the reviews

Фото профиля TK Kong

TK Kong14 дней назад

Incredible work! Congrats Josh, @jasontheutopian, and team

Фото профиля Archie Sengupta

Archie Sengupta14 дней назад

@sohamsankaran

Фото профиля Sanskar Pandey

Sanskar Pandey14 дней назад

@TyneeWorld Great!

Фото профиля Don

Don14 дней назад

so sick!

Фото профиля Peter Haymond

Peter Haymond14 дней назад

@MeckaAI This is why I’m curious what SuperRadiant is doing

Фото профиля Yuri Namikawa

Yuri Namikawa14 дней назад

very needed for progress in robotics x labs. cool stuff!

Фото профиля vinny

vinny14 дней назад

So cool!

Фото профиля Tanvi

Tanvi14 дней назад

really cool stuff!

Фото профиля Rakesh Sahni

Rakesh Sahni14 дней назад

Not yet On the hard lab tasks, Opus 5.5 finished 2 of 60 tries and Astra and Fable 5.1 finished none When Opus 5.5 said a try was finished a person counted that try as finished 38 of 79 times Limit: Graders could see which model ran

Фото профиля steve jang

steve jang14 дней назад

🔥🔥🔥

Фото профиля Vai Viswanathan

Vai Viswanathan14 дней назад

Sick!

Фото профиля Jenny do Forno

Jenny do Forno14 дней назад

🤖 🔥 🚀

Фото профиля Healy Jones

Healy Jones14 дней назад

Very cool! 🤖🤖

Фото профиля Kane

Kane14 дней назад

wow that's dope fen kk

Фото профиля Andrea Wang

Andrea Wang14 дней назад

🚀🚀

Фото профиля Shryas Bhurat

Shryas Bhurat14 дней назад

🚀

Фото профиля Manoela Schmitz

Manoela Schmitz14 дней назад

Which specific tasks among the nine tested proved to be the most challenging for the models? Reviewing the detailed benchmark results shows how they performed across the different difficulty levels.

Похожие видео

Introducing Nav Arena, over 150 real and simulated environments to benchmark Agents on robots As a first experiment, we put Jev, Astra, Fable, Dimcode, and other models + harnesses to the test on 2000+ navigation tasks Fully open source! Code, data, and results below

Introducing Nav Arena, over 150 real and simulated environments to benchmark Agents on robots As a first experiment, we put Jev, Astra, Fable, Dimcode, and other models + harnesses to the test on 2000+ navigation tasks Fully open source! Code, data, and results below

stash

16,456 просмотров • 9 дней назад

Introducing the Open Axis Benchmark, a living benchmark engine for honest evaluation of robot manipulation models, built together with OpenRoboto. Robotic models evolve faster every month, while most benchmarks stay frozen. Models overfit to fixed task sets, scores stop reflecting real generalization, and that distorted signal misleads and holds back model evolution. Open Axis Benchmark isn't just larger. It lets evaluation itself keep pace with the models: each round locks a fresh task set drawn from the growing Axis Library, so models prove they can generalize instead of memorizing the test. Axis provides the data engine behind it, continuously producing tasks at the scale and diversity needed to genuinely test generalization. With Open Axis Benchmark, the same compounding loop that trains robots also powers how they are evaluated. Submit your model: Docs: See how it works ⬇️

Introducing the Open Axis Benchmark, a living benchmark engine for honest evaluation of robot manipulation models, built together with OpenRoboto. Robotic models evolve faster every month, while most benchmarks stay frozen. Models overfit to fixed task sets, scores stop reflecting real generalization, and that distorted signal misleads and holds back model evolution. Open Axis Benchmark isn't just larger. It lets evaluation itself keep pace with the models: each round locks a fresh task set drawn from the growing Axis Library, so models prove they can generalize instead of memorizing the test. Axis provides the data engine behind it, continuously producing tasks at the scale and diversity needed to genuinely test generalization. With Open Axis Benchmark, the same compounding loop that trains robots also powers how they are evaluated. Submit your model: Docs: See how it works ⬇️

Axis Robotics

37,902 просмотров • 14 дней назад

Can Jev Nav? We put Jev, Astra, Fable, Dimcode, and other models + harnesses to the test on 2000+ navigation tasks in 133 real and simulated environments How far are LMs from zero-shotting complex real-time control tasks? Full dataset and code open source ⤵️

Can Jev Nav? We put Jev, Astra, Fable, Dimcode, and other models + harnesses to the test on 2000+ navigation tasks in 133 real and simulated environments How far are LMs from zero-shotting complex real-time control tasks? Full dataset and code open source ⤵️

stash

20,068 просмотров • 11 дней назад

It's month close. AI models can pass the CPA exam, but do you trust them to close the books? Introducing APEX-Accounting, built in collaboration with Ramp and Ramp Labs, to test whether frontier AI can do real accounting work. It’s a new benchmark comprising 160 tasks across 10 simulated companies, authored and graded by accounting professionals with experience at firms like Deloitte, PwC, EY, and KPMG.

It's month close. AI models can pass the CPA exam, but do you trust them to close the books? Introducing APEX-Accounting, built in collaboration with Ramp and Ramp Labs, to test whether frontier AI can do real accounting work. It’s a new benchmark comprising 160 tasks across 10 simulated companies, authored and graded by accounting professionals with experience at firms like Deloitte, PwC, EY, and KPMG.

Mercor

142,214 просмотров • 2 месяцев назад

Today, we publicly released RoboCasa365, a large-scale simulation benchmark for training and systematically evaluating generalist robot models. Built upon our original RoboCasa framework, it offers: • 2,500 realistic kitchen environments; • 365 everyday tasks (basic skills + long-horizon mobile manipulation); • Over 3,200 objects with many articulated fixtures/appliances. All are designed for fully controlled, reproducible benchmarking of robotic policies. Progress in robotic foundation models is real. But it’s still hard to answer basic questions like: How close are we to general-purpose autonomy? What factors drive generalization? What are the model/data scaling curves like? Real-world eval is slow and noisy, and existing sims (like LIBERO, which we built 3 years ago) often lack sufficient task and scene diversity. This benchmark comes with 2,200+ hours of demonstrations and 500K+ trajectories to support studies of multi-task training, pretraining, and continual learning at scale. Check it out at

Today, we publicly released RoboCasa365, a large-scale simulation benchmark for training and systematically evaluating generalist robot models. Built upon our original RoboCasa framework, it offers: • 2,500 realistic kitchen environments; • 365 everyday tasks (basic skills + long-horizon mobile manipulation); • Over 3,200 objects with many articulated fixtures/appliances. All are designed for fully controlled, reproducible benchmarking of robotic policies. Progress in robotic foundation models is real. But it’s still hard to answer basic questions like: How close are we to general-purpose autonomy? What factors drive generalization? What are the model/data scaling curves like? Real-world eval is slow and noisy, and existing sims (like LIBERO, which we built 3 years ago) often lack sufficient task and scene diversity. This benchmark comes with 2,200+ hours of demonstrations and 500K+ trajectories to support studies of multi-task training, pretraining, and continual learning at scale. Check it out at

Yuke Zhu

24,518 просмотров • 7 месяцев назад

Announcing Agent Composer: AI that works when it actually IS rocket science. Technical teams can now automate the routine (but complex) tasks that used to take hours every week—root cause analysis, production planning, test code generation— and reduce them down to minutes, so you can get your real work done. Here's how we solved the AI context problem for hard engineering tasks (like rocket science 🚀) 🧵

Announcing Agent Composer: AI that works when it actually IS rocket science. Technical teams can now automate the routine (but complex) tasks that used to take hours every week—root cause analysis, production planning, test code generation— and reduce them down to minutes, so you can get your real work done. Here's how we solved the AI context problem for hard engineering tasks (like rocket science 🚀) 🧵

Contextual AI

1,552,705 просмотров • 8 месяцев назад

The Robots are Coming We are “2 to 3 years” from mass adoption of robots — Elon thinks it could happen even faster. How will it change our lives — and what will happen to jobs.

The Robots are Coming We are “2 to 3 years” from mass adoption of robots — Elon thinks it could happen even faster. How will it change our lives — and what will happen to jobs.

Peter St Onge, Ph.D.

202,398 просмотров • 1 год назад

For general-purpose robots to be useful, we must be able to teach them new tasks quickly and repurpose them to new roles. But most techniques for teaching robots — even from human demonstrations — take a great deal of new data. But it turns out that it is possible, in part through decomposing tasks into component subtasks, that it’s possible to learn a robot skills quickly. Kamil Dreczkowski Pietro Vitiello and Edward Johns join us to talk about their extensive study on how to teach robots new skills efficiently, while also generalizing to novel object instances, using a combination of task decomposition and retrieval, work that was published in Science Robotics. Watch episode #94 of RoboPapers now, with Michael Cho - Rbt/Acc and Jiafei Duan, to learn more!

For general-purpose robots to be useful, we must be able to teach them new tasks quickly and repurpose them to new roles. But most techniques for teaching robots — even from human demonstrations — take a great deal of new data. But it turns out that it is possible, in part through decomposing tasks into component subtasks, that it’s possible to learn a robot skills quickly. Kamil Dreczkowski Pietro Vitiello and Edward Johns join us to talk about their extensive study on how to teach robots new skills efficiently, while also generalizing to novel object instances, using a combination of task decomposition and retrieval, work that was published in Science Robotics. Watch episode #94 of RoboPapers now, with Michael Cho - Rbt/Acc and Jiafei Duan, to learn more!

RoboPapers

19,495 просмотров • 2 месяцев назад

We created a game for America's Navy but it's so hard we forgot how to beat it. Tap the link below to play from your desktop and let us know how far you get Play: #AmericasNavyPartner

We created a game for America's Navy but it's so hard we forgot how to beat it. Tap the link below to play from your desktop and let us know how far you get Play: #AmericasNavyPartner

Dexerto

465,757 просмотров • 3 лет назад

Usyk is Ready to Pay. Are you? As the co-founder and world champion, Oleksandr Usyk knows that success comes from making the right choices. That’s why we are created Ready to Pay - the card he trusts to pay anywhere, anytime. So, are you ready? Join now:

Usyk is Ready to Pay. Are you? As the co-founder and world champion, Oleksandr Usyk knows that success comes from making the right choices. That’s why we are created Ready to Pay - the card he trusts to pay anywhere, anytime. So, are you ready? Join now:

RTPay

17,536 просмотров • 1 год назад

OpenAI researcher David Holtz reveals workers who don't adapt their prompting to newer models leave 50% of the alpha on the table because half of AI's value comes from how you interact with it: "We ran a lab experiment. We give people the same task to do, but we give them models of different capabilities. How much of the improvement when we give people more capable models comes from the fact that we gave them a better model vs the fact that they actually update and adapt the way they prompt and interact with these models?" "For tasks where precision matters, where you have some output you're trying to get and you're trying to steer the model, which is different from a creative task like brainstorming." "For these steering-based tasks, when you give people a better model, about 50% of the improvement we see in how well they're able to complete the task comes from changes in how they interact with the model." "If you just kept the way you interact with the model static and just made the capabilities better and better, you'd leave 50% of the alpha basically on the table." OpenAI

OpenAI researcher David Holtz reveals workers who don't adapt their prompting to newer models leave 50% of the alpha on the table because half of AI's value comes from how you interact with it: "We ran a lab experiment. We give people the same task to do, but we give them models of different capabilities. How much of the improvement when we give people more capable models comes from the fact that we gave them a better model vs the fact that they actually update and adapt the way they prompt and interact with these models?" "For tasks where precision matters, where you have some output you're trying to get and you're trying to steer the model, which is different from a creative task like brainstorming." "For these steering-based tasks, when you give people a better model, about 50% of the improvement we see in how well they're able to complete the task comes from changes in how they interact with the model." "If you just kept the way you interact with the model static and just made the capabilities better and better, you'd leave 50% of the alpha basically on the table." OpenAI

MTS

15,590 просмотров • 1 месяц назад

We are excited to join Nvidia's Nemotron Coalition of leading AI labs working together to advance open frontier foundation models. To celebrate we have partnered with NVIDIA and Nebius Token Factory to provide 2 free weeks of the new Nemotron 3 Ultra model on the Nous Portal!

We are excited to join Nvidia's Nemotron Coalition of leading AI labs working together to advance open frontier foundation models. To celebrate we have partnered with NVIDIA and Nebius Token Factory to provide 2 free weeks of the new Nemotron 3 Ultra model on the Nous Portal!

Nous Research

1,501,926 просмотров • 4 месяцев назад

GPT-6 Astra “uses Unity to assemble a city scene from existing assets, allowing users to easily create an immersive 3D environment to explore that matches their vision.” “Alongside Astra, we are also updating the Codex harness to significantly improve the speed of computer use. Combined with Astra's efficiency, this translates to a 1.9x faster task completion compared to the current GPT-5.6 Sol experience, on the Mind2Web benchmark. “ “The model's improvements on speed mean it can take on many time consuming life tasks for you, faster than you can.*” I’ll admit, others and I have created far more impressive demos, but keep in mind this was done in Unity, which is a much more difficult environment for models to work with. If they weren’t using computer use which would’ve been extremely slow they were likely manipulating the project programmatically through C# and/or Unity’s YAML-serialized scene files which again models are pretty bad at. So this is a solid upgrade from 5.6 - from what I’ve seen unity!

GPT-6 Astra “uses Unity to assemble a city scene from existing assets, allowing users to easily create an immersive 3D environment to explore that matches their vision.” “Alongside Astra, we are also updating the Codex harness to significantly improve the speed of computer use. Combined with Astra's efficiency, this translates to a 1.9x faster task completion compared to the current GPT-5.6 Sol experience, on the Mind2Web benchmark. “ “The model's improvements on speed mean it can take on many time consuming life tasks for you, faster than you can.*” I’ll admit, others and I have created far more impressive demos, but keep in mind this was done in Unity, which is a much more difficult environment for models to work with. If they weren’t using computer use which would’ve been extremely slow they were likely manipulating the project programmatically through C# and/or Unity’s YAML-serialized scene files which again models are pretty bad at. So this is a solid upgrade from 5.6 - from what I’ve seen unity!

Chris

22,655 просмотров • 1 месяц назад

We’re launching Optima. Now anyone can create a custom benchmark for their use case, leveraging Artificial Analysis’ leading research and platform Building and running benchmarks is difficult. We have distilled Artificial Analysis’ research and experience developing benchmarks into Optima, a new platform for benchmarking models on your own workloads and comparing performance, speed and cost efficiency. Optima allows you to find the best model for your task, or an equally performant alternative to your current setup at 10x lower cost or time per task. We’ve integrated Artificial Analysis' research and experience in benchmarks across the Optima workflow: ➤ Build benchmarks based on your own data and use cases: There are three ways to build a benchmark with Optima. Upload an existing evaluation dataset from your own files or Hugging Face, or import agent traces from platforms including Arize AI, Braintrust and langfuse.com. Install the Optima skill to build a benchmark using context from your coding environment and previous sessions. Or simply describe your use case and provide example inputs and outputs, and Optima will build the benchmark for you ➤ Run across the latest models: Run the same benchmark across leading models in a single click, and keep your leaderboard up to date as soon as new models are released ➤ Bring Artificial Analysis grading to your own benchmark: Evaluate responses against objective rubric criteria or using the same pairwise judging approach used for Artificial Analysis benchmarks including GDPval-AA and AA-Briefcase. For pairwise judging, select your preferred responses from a sample and Optima uses those preferences to rank models across your test set ➤ Compare performance, cost and time efficiency: Optima measures more than model performance. Cost per Task and Time per Task are tracked alongside benchmark scores, with category-level results and support for custom metrics, allowing you to compare the tradeoffs between models for your specific use case Ahead of launch, here are examples questions our beta testers answered with Optima: ➤ Which model can save me 10x the cost without a meaningful decrease in quality for my finance & accounting agent? ➤ Which model best matches the writing style of lawyers for my legal agent? ➤ Which model can best identify different elements in my custom image dataset? Optima is available today. Build your own benchmark at

We’re launching Optima. Now anyone can create a custom benchmark for their use case, leveraging Artificial Analysis’ leading research and platform Building and running benchmarks is difficult. We have distilled Artificial Analysis’ research and experience developing benchmarks into Optima, a new platform for benchmarking models on your own workloads and comparing performance, speed and cost efficiency. Optima allows you to find the best model for your task, or an equally performant alternative to your current setup at 10x lower cost or time per task. We’ve integrated Artificial Analysis' research and experience in benchmarks across the Optima workflow: ➤ Build benchmarks based on your own data and use cases: There are three ways to build a benchmark with Optima. Upload an existing evaluation dataset from your own files or Hugging Face, or import agent traces from platforms including Arize AI, Braintrust and langfuse.com. Install the Optima skill to build a benchmark using context from your coding environment and previous sessions. Or simply describe your use case and provide example inputs and outputs, and Optima will build the benchmark for you ➤ Run across the latest models: Run the same benchmark across leading models in a single click, and keep your leaderboard up to date as soon as new models are released ➤ Bring Artificial Analysis grading to your own benchmark: Evaluate responses against objective rubric criteria or using the same pairwise judging approach used for Artificial Analysis benchmarks including GDPval-AA and AA-Briefcase. For pairwise judging, select your preferred responses from a sample and Optima uses those preferences to rank models across your test set ➤ Compare performance, cost and time efficiency: Optima measures more than model performance. Cost per Task and Time per Task are tracked alongside benchmark scores, with category-level results and support for custom metrics, allowing you to compare the tradeoffs between models for your specific use case Ahead of launch, here are examples questions our beta testers answered with Optima: ➤ Which model can save me 10x the cost without a meaningful decrease in quality for my finance & accounting agent? ➤ Which model best matches the writing style of lawyers for my legal agent? ➤ Which model can best identify different elements in my custom image dataset? Optima is available today. Build your own benchmark at

Artificial Analysis

133,508 просмотров • 1 месяц назад

We never get to see the big picture in the beginning. And we cannot let our inability to fully understand everything keep us from valuing others and leading them right where we are. Take what is right in front of you and do it well!

We never get to see the big picture in the beginning. And we cannot let our inability to fully understand everything keep us from valuing others and leading them right where we are. Take what is right in front of you and do it well!

John C. Maxwell

12,753 просмотров • 3 лет назад

We voted to take our ball home and not let anyone else play with it and that’s ok. But why take a knife to the ball so we can’t play with it too? £1bn quid to make back £100bn in imports and exports per year without rejoining? I’d take it.

We voted to take our ball home and not let anyone else play with it and that’s ok. But why take a knife to the ball so we can’t play with it too? £1bn quid to make back £100bn in imports and exports per year without rejoining? I’d take it.

Matthew

42,156 просмотров • 5 месяцев назад

One of the biggest surprises in AI over the last few years has been how well coding agents generalize beyond software. In a recent essay, MIT professor Philip Isola argued that we may be entering the era of robot-use agents: general-purpose models that can control different robots, write policies, and learn new physical tasks with little or no robot-specific training. In this episode of Decoded, we're joined by the founders of Waddle Labs and Robocurve PBC, two of the startups whose work helped drive this realization. They're working at the frontier of using general-purpose models to control robots, and their recent demos helped inspire the growing conversation around robot-use agents. Together, we dig into the research behind that idea, from code-as-policies and vision-language-action models to the harnesses and evals needed to make these systems work in the real world. 00:00 — What Are Robot-Use Agents? 02:11 — From Vision-Language-Action Models to General LLMs 04:07 — The Bitter Lesson for Robotics 07:04 — How Coding Agents Learned to Control Robots 10:41 — How Robots Learn From Experience 14:22 — The Harness as a Form of Robot Intelligence 16:14 — Watching Astra Control a Robot 20:55 — Why General Models May Win in Robotics 26:05 — How Close Are General-Purpose Robots?

One of the biggest surprises in AI over the last few years has been how well coding agents generalize beyond software. In a recent essay, MIT professor Philip Isola argued that we may be entering the era of robot-use agents: general-purpose models that can control different robots, write policies, and learn new physical tasks with little or no robot-specific training. In this episode of Decoded, we're joined by the founders of Waddle Labs and Robocurve PBC, two of the startups whose work helped drive this realization. They're working at the frontier of using general-purpose models to control robots, and their recent demos helped inspire the growing conversation around robot-use agents. Together, we dig into the research behind that idea, from code-as-policies and vision-language-action models to the harnesses and evals needed to make these systems work in the real world. 00:00 — What Are Robot-Use Agents? 02:11 — From Vision-Language-Action Models to General LLMs 04:07 — The Bitter Lesson for Robotics 07:04 — How Coding Agents Learned to Control Robots 10:41 — How Robots Learn From Experience 14:22 — The Harness as a Form of Robot Intelligence 16:14 — Watching Astra Control a Robot 20:55 — Why General Models May Win in Robotics 26:05 — How Close Are General-Purpose Robots?

Y Combinator

93,369 просмотров • 13 дней назад