Video yükleniyor...

Video Yüklenemedi

Ana Sayfaya Dön

Excited to release PostTrainBench v1.0! This benchmark evaluates the ability of frontier AI agents to post-train language models in a simplified setting. We believe this is a first step toward tracking progress in recursive self-improvement 🧵:

karinanguyen's profile picture

Karina

44,124 subscribers

195,478 görüntüleme • 5 ay önce •via X (Twitter)

EğitimBilim & Teknoloji
LIVE

Anya Rossi• Live Now

Private livecam show

0 Yorum

Yorum bulunmuyor

Orijinal gönderinin yorumları burada görünecek

Benzer Videolar

Recently, PostTrainBench showed how well AI agents can post-train models. Meta Harness showed that the harness itself can improve. What happens if a harness is improvising itself and the improvised harness is post training language models, all in a loop? Viv on this:
2:31

Recently, PostTrainBench showed how well AI agents can post-train models. Meta Harness showed that the harness itself can improve. What happens if a harness is improvising itself and the improvised harness is post training language models, all in a loop? Viv on this:

himanshu

29,325 görüntüleme • 4 ay önce

A challenge from Daniel Kokotajlo to AI company employees who believe recursive self-improvement will kick off in the next couple of years:
0:27

A challenge from Daniel Kokotajlo to AI company employees who believe recursive self-improvement will kick off in the next couple of years:

Palisade Research

31,338 görüntüleme • 6 gün önce

LLM agents have demonstrated promise in their ability to automate computer tasks, but face challenges with multi-step reasoning and planning. Towards addressing this, we propose an inference-time tree search algorithm for LLM agents to explicitly perform exploration and multi-step planning in interactive web environments. It is the first tree search algorithm for LLM agents that shows effectiveness on realistic and complex web environments: on the challenging VisualWebArena benchmark, applying our search algorithm on top of a GPT-4o agent yields a 39.7% relative increase in success rate compared to the same baseline without search, setting a state-of-the-art success rate of 26.4%. On WebArena, search also yields a 28.0% relative improvement over a baseline agent, setting a competitive success rate of 19.2%.
0:20

LLM agents have demonstrated promise in their ability to automate computer tasks, but face challenges with multi-step reasoning and planning. Towards addressing this, we propose an inference-time tree search algorithm for LLM agents to explicitly perform exploration and multi-step planning in interactive web environments. It is the first tree search algorithm for LLM agents that shows effectiveness on realistic and complex web environments: on the challenging VisualWebArena benchmark, applying our search algorithm on top of a GPT-4o agent yields a 39.7% relative increase in success rate compared to the same baseline without search, setting a state-of-the-art success rate of 26.4%. On WebArena, search also yields a 28.0% relative improvement over a baseline agent, setting a competitive success rate of 19.2%.

Jing Yu Koh

124,894 görüntüleme • 2 yıl önce

🚀 Excited to announce the first release of a novel open source programming language and platform for language model interaction! Combining prompts, constraints & scripting, LMQL elevates the capabilities of large language models. 🧵1/6 A quick tour.
0:20

🚀 Excited to announce the first release of a novel open source programming language and platform for language model interaction! Combining prompts, constraints & scripting, LMQL elevates the capabilities of large language models. 🧵1/6 A quick tour.

LMQL (Language Model Query Language)

198,966 görüntüleme • 3 yıl önce

I'm excited to share our $200M Series B at a $1B+ valuation to develop AI systems that reason! We believe reasoning is the main blocker to effective AI agents. We train large models tailor-made for reasoning on our ~10K GPU cluster. On top, we prototype agents we use every day.
0:24

I'm excited to share our $200M Series B at a $1B+ valuation to develop AI systems that reason! We believe reasoning is the main blocker to effective AI agents. We train large models tailor-made for reasoning on our ~10K GPU cluster. On top, we prototype agents we use every day.

Kanjun 🐙

1,692,171 görüntüleme • 3 yıl önce

🚀 Excited to release AIBuildAI — an AI agent that automatically builds AI models 🏆 #1 on OpenAI MLE-Bench 💻 GitHub: Building AI models still requires a lot of manual work — designing models, writing code, training them, tuning hyperparameters, and iterating on results. We developed AIBuildAI to automate much of this process. The system runs an agent loop that automatically: • analyzes the task • designs models • writes code to implement them • trains the models • tunes hyperparameters • evaluates model performance • iteratively improves the models 🏆 On OpenAI’s MLE-Bench benchmark, AIBuildAI ranked #1:
1:33

🚀 Excited to release AIBuildAI — an AI agent that automatically builds AI models 🏆 #1 on OpenAI MLE-Bench 💻 GitHub: Building AI models still requires a lot of manual work — designing models, writing code, training them, tuning hyperparameters, and iterating on results. We developed AIBuildAI to automate much of this process. The system runs an agent loop that automatically: • analyzes the task • designs models • writes code to implement them • trains the models • tunes hyperparameters • evaluates model performance • iteratively improves the models 🏆 On OpenAI’s MLE-Bench benchmark, AIBuildAI ranked #1:

Pengtao Xie

152,252 görüntüleme • 5 ay önce

For over a century, we believed evolution was just natural selection and mutation. We were wrong. The third, hidden pillar of evolution is cooperation. We tested whether SOTA language models possess this trait or if they default to greed. Introducing Commons Keeper v1.0 🧵
1:15

For over a century, we believed evolution was just natural selection and mutation. We were wrong. The third, hidden pillar of evolution is cooperation. We tested whether SOTA language models possess this trait or if they default to greed. Introducing Commons Keeper v1.0 🧵

Abhishek Anand

14,471 görüntüleme • 9 ay önce

Mustafa Suleyman says recursive self-improvement is the threshold moment, when AI improves itself without humans Some see this as a path to "foom" — an intelligence explosion with unbounded compute and no human control "we're taking steps toward riskier and riskier stuff"
1:25

Mustafa Suleyman says recursive self-improvement is the threshold moment, when AI improves itself without humans Some see this as a path to "foom" — an intelligence explosion with unbounded compute and no human control "we're taking steps toward riskier and riskier stuff"

Haider.

18,349 görüntüleme • 8 ay önce

AI researcher Prof. Roman Yampolskiy: 100% of code for the most powerful AIs is now written by AI, an early form of recursive self-improvement. He says it's a hyperexponential process that results in superintelligent AI, "machine gods", which we won't be able to control.
1:24

AI researcher Prof. Roman Yampolskiy: 100% of code for the most powerful AIs is now written by AI, an early form of recursive self-improvement. He says it's a hyperexponential process that results in superintelligent AI, "machine gods", which we won't be able to control.

ControlAI

11,606 görüntüleme • 5 ay önce

Introducing Managed Auth: a standardized way for agents to log in and stay logged in across the internet. This is a big step toward finally solving auth for agents.
1:44

Introducing Managed Auth: a standardized way for agents to log in and stay logged in across the internet. This is a big step toward finally solving auth for agents.

KERNEL

33,390 görüntüleme • 6 ay önce

.Poetiq is a new startup that recently achieved a major jump on the ARC-AGI benchmark by layering a recursive self-improvement system on top of existing models. In this episode of the Lightcone Podcast, Poetiq's Founder & CEO Ian Fischer joined us to discuss how small teams can build “reasoning harnesses” that outperform base models, what that means for startups and why automating prompt engineering may be one of the most powerful levers in AI today. 00:00 – Intro 00:40 – What Is Poetiq? 01:07 – Recursive Self-Improvement Explained 02:07 – The Fine-Tuning Trap 02:59 – “Stilts” for LLMs 03:14 – Recursive Self-Improvement vs. Fine-Tuning 05:05 – Taking the Top Spot on ARC-AGI 06:37 – Beating Claude on Humanity’s Last Exam 08:40 – How the Meta-System Works 10:26 – Beyond RL: A New S-Curve 11:32 – Automating Prompt Engineering 13:37 – From 5% to 95% Performance 14:50 – Early Access & Putting Your Agent on Stilts 16:17 – From YC Founder to DeepMind Researcher 18:29 – Advice for Engineers in the AI Era
19:45

.Poetiq is a new startup that recently achieved a major jump on the ARC-AGI benchmark by layering a recursive self-improvement system on top of existing models. In this episode of the Lightcone Podcast, Poetiq's Founder & CEO Ian Fischer joined us to discuss how small teams can build “reasoning harnesses” that outperform base models, what that means for startups and why automating prompt engineering may be one of the most powerful levers in AI today. 00:00 – Intro 00:40 – What Is Poetiq? 01:07 – Recursive Self-Improvement Explained 02:07 – The Fine-Tuning Trap 02:59 – “Stilts” for LLMs 03:14 – Recursive Self-Improvement vs. Fine-Tuning 05:05 – Taking the Top Spot on ARC-AGI 06:37 – Beating Claude on Humanity’s Last Exam 08:40 – How the Meta-System Works 10:26 – Beyond RL: A New S-Curve 11:32 – Automating Prompt Engineering 13:37 – From 5% to 95% Performance 14:50 – Early Access & Putting Your Agent on Stilts 16:17 – From YC Founder to DeepMind Researcher 18:29 – Advice for Engineers in the AI Era

Y Combinator

156,371 görüntüleme • 6 ay önce

This week we’re getting GPT 5.5 and I couldn’t be more excited to see the first new pre train base in a while Before Spud releases, here’s the last video you need to watch - Greg Brockman perfectly indicated to us 3 important things that perfectly encapsulate the lineage of OpenAI and the mission. 1.This is the culmination of 2 years of research and is a fresh base pre-train that we haven’t gotten for a long time. 2.This is an “exciting release” and it will bring new capabilities (sort of a given but still very grateful nonetheless) & that it is AN EARLY VERSION of what we have coming - meaning they have a lot of RL to do to make this model even better since it’s a fresh pre-train. 3.THIS IS THE BEGINNING OF THE FLYWHEEL. As he said toward the end, “we have this engine of progress that just moves faster and faster and Spud is just one step along the way.” Depending on the capabilities we see this week from Spud, we may be able to map out if AGI will be sooner rather than later, judging by how much recursive work can exist within this model and the harness. In other words, the flywheel.
0:42

This week we’re getting GPT 5.5 and I couldn’t be more excited to see the first new pre train base in a while Before Spud releases, here’s the last video you need to watch - Greg Brockman perfectly indicated to us 3 important things that perfectly encapsulate the lineage of OpenAI and the mission. 1.This is the culmination of 2 years of research and is a fresh base pre-train that we haven’t gotten for a long time. 2.This is an “exciting release” and it will bring new capabilities (sort of a given but still very grateful nonetheless) & that it is AN EARLY VERSION of what we have coming - meaning they have a lot of RL to do to make this model even better since it’s a fresh pre-train. 3.THIS IS THE BEGINNING OF THE FLYWHEEL. As he said toward the end, “we have this engine of progress that just moves faster and faster and Spud is just one step along the way.” Depending on the capabilities we see this week from Spud, we may be able to map out if AGI will be sooner rather than later, judging by how much recursive work can exist within this model and the harness. In other words, the flywheel.

Chris

103,198 görüntüleme • 4 ay önce

Today, we’re introducing Decide Agent, a frontier-level AI agent for Excel analysis. Decide Agent scored 82.50% on SpreadsheetBench, the public standard benchmark for Excel AI agents, ranking #4 globally. This is the same leaderboard used by Claude, OpenAI, and Microsoft to evaluate their spreadsheet agents. We’re here to push the frontier in Excel and data analytics. Decide Agent is now generally available at trydecide . ai
1:05

Today, we’re introducing Decide Agent, a frontier-level AI agent for Excel analysis. Decide Agent scored 82.50% on SpreadsheetBench, the public standard benchmark for Excel AI agents, ranking #4 globally. This is the same leaderboard used by Claude, OpenAI, and Microsoft to evaluate their spreadsheet agents. We’re here to push the frontier in Excel and data analytics. Decide Agent is now generally available at trydecide . ai

Ab.

40,512 görüntüleme • 7 ay önce

How well do today’s frontier models handle long-horizon, multi-step web agent tasks, such as identifying the top 25 U.S. CS PhD programs with ML/AI faculty likely accepting students and compiling the results into a structured sheet? Check out our new work on Odysseys: Benchmarking Web Agents on Realistic Long Horizon Tasks Paper: Leaderboard: We introduce Odysseys, a benchmark of 200 long-horizon tasks derived from real browsing sessions and evaluated on the live Internet. We show that binary pass/fail is inadequate in this setting and propose rubric-based evaluation, which better aligns with human judgment and provides more informative signals. Across leading models, the best achieves only 44.5% success, leaving substantial headroom. We further introduce a Trajectory Efficiency metric (rubric score per step) and find efficiency remains extremely low (1.15%), highlighting a key bottleneck. Odysseys provides a realistic benchmark for measuring progress toward web agents capable of sustained, efficient, real-world operation. See a more detailed thread by Jing Yu Koh.
1:12

How well do today’s frontier models handle long-horizon, multi-step web agent tasks, such as identifying the top 25 U.S. CS PhD programs with ML/AI faculty likely accepting students and compiling the results into a structured sheet? Check out our new work on Odysseys: Benchmarking Web Agents on Realistic Long Horizon Tasks Paper: Leaderboard: We introduce Odysseys, a benchmark of 200 long-horizon tasks derived from real browsing sessions and evaluated on the live Internet. We show that binary pass/fail is inadequate in this setting and propose rubric-based evaluation, which better aligns with human judgment and provides more informative signals. Across leading models, the best achieves only 44.5% success, leaving substantial headroom. We further introduce a Trajectory Efficiency metric (rubric score per step) and find efficiency remains extremely low (1.15%), highlighting a key bottleneck. Odysseys provides a realistic benchmark for measuring progress toward web agents capable of sustained, efficient, real-world operation. See a more detailed thread by Jing Yu Koh.

Russ Salakhutdinov

22,518 görüntüleme • 4 ay önce

Compared to conventional VLAs, S1 is a step-change improvement. For known tasks, S1 can match the performance of language-prompted VLAs. For novel tasks, S1 exponentially outperforms any existing language-prompted VLAs as we scale the pre-training. We believe this is a promising direction towards establishing scaling laws for robotics.
0:27

Compared to conventional VLAs, S1 is a step-change improvement. For known tasks, S1 can match the performance of language-prompted VLAs. For novel tasks, S1 exponentially outperforms any existing language-prompted VLAs as we scale the pre-training. We believe this is a promising direction towards establishing scaling laws for robotics.

Skild AI

302,433 görüntüleme • 9 gün önce

"to me, it's not about the human in the loop, but the human at the center." i sat down with Stefania Druga, staff research scientist Sakana AI, at our Browserbase booth at AI Engineer. after Massachusetts Institute of Technology (MIT) and Google DeepMind, stefania recently joined sakana's recursive self-improvement (rsi) lab in tokyo to build self-improving AI for the next frontier. we got into: - catastrophic forgetting in day-long agent research runs, and how memory-first harnesses help - sovereign AI and open-weights: building on models you control instead of locking into one provider - why computer use still frustrates people even as models get smarter - the domains ripe for self-improving research agents: math, genomics, rare conditions - my fav moment at the end: what tasks are better left human, and what are we going to do as humans after everything is automated? with open frontier models arriving fast, Stefania Druga's case for sovereign AI only gets stronger. loved this conversation. thanks for joining us at the Browserbase booth!
16:07

"to me, it's not about the human in the loop, but the human at the center." i sat down with Stefania Druga, staff research scientist Sakana AI, at our Browserbase booth at AI Engineer. after Massachusetts Institute of Technology (MIT) and Google DeepMind, stefania recently joined sakana's recursive self-improvement (rsi) lab in tokyo to build self-improving AI for the next frontier. we got into: - catastrophic forgetting in day-long agent research runs, and how memory-first harnesses help - sovereign AI and open-weights: building on models you control instead of locking into one provider - why computer use still frustrates people even as models get smarter - the domains ripe for self-improving research agents: math, genomics, rare conditions - my fav moment at the end: what tasks are better left human, and what are we going to do as humans after everything is automated? with open frontier models arriving fast, Stefania Druga's case for sovereign AI only gets stronger. loved this conversation. thanks for joining us at the Browserbase booth!

justine

19,853 görüntüleme • 1 ay önce

It’s time to go beyond language models. Introducing Odyssey-2 Max, our most powerful world model yet. It materially advances the SOTA in physical accuracy. This is a big step toward models that simulate and interact with the world in real time. A new intelligence entirely!
1:08

It’s time to go beyond language models. Introducing Odyssey-2 Max, our most powerful world model yet. It materially advances the SOTA in physical accuracy. This is a big step toward models that simulate and interact with the world in real time. A new intelligence entirely!

Odyssey

2,185,951 görüntüleme • 4 ay önce

Earlier this week at GTC, we announced our partnership with Nvidia. We will work with Nvidia to build strong, American open-source models that are at the frontier of scientific reasoning. These models will be essential for the US to compete with China on science in the coming decades. Jensen is committing to spend tens of billions of dollars developing open-source models, and we are excited to be a partner with them in figuring out how to benchmark, train and use those agents to accelerate scientific research. We have already open-sourced some of the work we have done with them, and are looking forward to open-sourcing more. There are few things today that are more important. See our blog post below, and watch the video to learn more, narrated by the man himself.
1:44

Earlier this week at GTC, we announced our partnership with Nvidia. We will work with Nvidia to build strong, American open-source models that are at the frontier of scientific reasoning. These models will be essential for the US to compete with China on science in the coming decades. Jensen is committing to spend tens of billions of dollars developing open-source models, and we are excited to be a partner with them in figuring out how to benchmark, train and use those agents to accelerate scientific research. We have already open-sourced some of the work we have done with them, and are looking forward to open-sourcing more. There are few things today that are more important. See our blog post below, and watch the video to learn more, narrated by the man himself.

Sam Rodriques

23,763 görüntüleme • 5 ay önce