Загрузка видео...

Не удалось загрузить видео

Возникла проблема при загрузке этого видео. Это может быть связано с временными проблемами сети или видео может быть недоступно.

На главную

We’re sharing new results showing that frontier AI models are able to outperform human experts on financial predictions for the first time. We built an environment for the challenging task of Earnings Predictions, finding that the most recent frontier models (Opus 5.5, Fable 5.1, Astra) with Samaya’s harness, meaningfully... outperform expert consensus predictions on commonly reported earnings metrics such as company revenue, gross margin and more. We built this environment to have a “point-in-time” gate in the harness to ensure no information leakage, and created financial retrieval and data tools that would respect this gate and give the models access to the same information available to experts. We also adjusted for bias in expert consensus to create a hard, bias-corrected baseline, where outperformance has real signal. Only the most recent frontier AI models are able to beat this hard baseline, pointing to an important inflection point in AI capabilities in finance. Like with FrontierFinance, we see substantial headroom for further RL training in this environment — the models show large variations in error across different rollouts. We’re continuing research on using earnings prediction and other predictive tasks to post-train models to learn from their mistakes. Get in touch if you’d like to collaborate or try out an alpha version. Link to the full research below.show more

Maithra Raghu

22,654 subscribers

25,779 просмотров • 1 день назад •via X (Twitter)

Финансы Наука и технологии

Anya Rossi• Live Now

Private livecam show

Комментарии: 14

Фото профиля Jeff Dean

Jeff Dean1 день назад

Congrats on these very nice results!

Фото профиля Maithra Raghu

Maithra Raghu1 день назад

Thank you Jeff!!🙏🏼

Фото профиля Maithra Raghu

Maithra Raghu1 день назад

Link to full research:

Фото профиля Nima Alidoust

Nima Alidoust1 день назад

really cool

Фото профиля solmaz shariat

solmaz shariat1 день назад

A great example of what the combination of good labeled data + harness on frontier models can achieve. Fantistic work!

Фото профиля generatorman

generatorman1 день назад

>We built this environment to have a “point-in-time” gate in the harness to ensure no information leakage if you've checked this less than 20 times then you've got it wrong

Фото профиля Sani Ai Tech

Sani Ai Tech1 день назад

This feels like a major inflection point for AI-driven financial forecasting

Фото профиля Adithya Giridharan

Adithya Giridharan1 день назад

amazing @maithra_raghu , would love to try the alpha release , where can i express my interest/register?

Фото профиля Future Alpha

Future Alpha1 день назад

Companies have to beat buy side whisper consensus, not just sell side estimates.

Фото профиля Vera Ai | Tools & Updates

Vera Ai | Tools & Updates1 день назад

The point-in-time safeguards make these financial prediction results especially compelling for future model training

Фото профиля Nathan Benaich

Nathan Benaich1 день назад

this is v cool!

Фото профиля James Kaplan

James Kaplan1 день назад

Super cool

Фото профиля I Wo Main Yo

I Wo Main Yo1 день назад

At what point are SI models considered insiders, with exactly the same trading restrictions as real insiders?

Фото профиля Veyra | AI & Tech

Veyra | AI & Tech1 день назад

Point in time harness shows frontier models beat expert earnings

Похожие видео

Today we're announcing Base Labs, a dedicated research organization focused on advancing open-source AI. We believe in a healthy, open frontier model ecosystem. To enable this, we are working on: - Blue-sky research on continual learning, the science of RL, and how models learn across their full lifecycle, with every experiment and recipe shared openly. - The BaseHub Data Foundry: the highest-quality open RL environments, training data, and real-world benchmarks, built for anyone to train and benchmark on. - Post-post training: taking open-source models and making them better, safer, and more aligned through continual post-training, built on our research, and deployed with our frontier safety stack so organizations can use open-source models with confidence. - Making models cheaper and more performant through our model performance research. This is a mission-driven research effort, not a commercial product. We believe the health of the open-source AI ecosystem matters and that the best way to advance it is to do science in the open. We’re hiring engineers, researchers, and research fellows to advance this mission.

Today we're announcing Base Labs, a dedicated research organization focused on advancing open-source AI. We believe in a healthy, open frontier model ecosystem. To enable this, we are working on: - Blue-sky research on continual learning, the science of RL, and how models learn across their full lifecycle, with every experiment and recipe shared openly. - The BaseHub Data Foundry: the highest-quality open RL environments, training data, and real-world benchmarks, built for anyone to train and benchmark on. - Post-post training: taking open-source models and making them better, safer, and more aligned through continual post-training, built on our research, and deployed with our frontier safety stack so organizations can use open-source models with confidence. - Making models cheaper and more performant through our model performance research. This is a mission-driven research effort, not a commercial product. We believe the health of the open-source AI ecosystem matters and that the best way to advance it is to do science in the open. We’re hiring engineers, researchers, and research fellows to advance this mission.

Base Labs

243,728 просмотров • 1 месяц назад

“We want to orchestrate the world’s compute, in the same way Bitcoin did, but to train models that can rival ChatGPT” crux on the This Week in Startups podcast, sharing the motivations behind IOTA ・ SN9: to train frontier models efficiently and at scale, in an age where CapEx for centralized training is reaching unsustainable highs.

“We want to orchestrate the world’s compute, in the same way Bitcoin did, but to train models that can rival ChatGPT” crux on the This Week in Startups podcast, sharing the motivations behind IOTA ・ SN9: to train frontier models efficiently and at scale, in an age where CapEx for centralized training is reaching unsustainable highs.

Macrocosmos

14,545 просмотров • 5 месяцев назад

one of the most challenging tasks for frontier models is being able to extract thousands of values from extremely dense tables in documents. our new Extract v2.5 agents are able to get 93%-96%+ on long-list extraction, including cells that fall in between pages. in contrast, astra gets ~30% accuracy over this benchmark. in these cases, frontier VLMs can stop early or miss repeated records. they also struggle to attribute each value back to the source. we are able to attribute every single extracted value back to the source. check out the video below, the blog post, and llamaparse! blog: llamaparse:

one of the most challenging tasks for frontier models is being able to extract thousands of values from extremely dense tables in documents. our new Extract v2.5 agents are able to get 93%-96%+ on long-list extraction, including cells that fall in between pages. in contrast, astra gets ~30% accuracy over this benchmark. in these cases, frontier VLMs can stop early or miss repeated records. they also struggle to attribute each value back to the source. we are able to attribute every single extracted value back to the source. check out the video below, the blog post, and llamaparse! blog: llamaparse:

Jerry Liu

13,103 просмотров • 7 дней назад

Today, Mistral AI and Microsoft are expanding our partnership to bring Mistral’s frontier models across Azure, Microsoft Foundry, Copilot Studio, and Azure Local. Europe should have access to the world’s most capable AI without compromising control over their data, operations, or digital future. By bringing Mistral’s frontier European models into our environment, we are providing customers with a trusted foundation for AI, one that offers both leading-edge innovation and the freedom to operate on their own terms. Learn more:

Today, Mistral AI and Microsoft are expanding our partnership to bring Mistral’s frontier models across Azure, Microsoft Foundry, Copilot Studio, and Azure Local. Europe should have access to the world’s most capable AI without compromising control over their data, operations, or digital future. By bringing Mistral’s frontier European models into our environment, we are providing customers with a trusted foundation for AI, one that offers both leading-edge innovation and the freedom to operate on their own terms. Learn more:

Brad Smith

322,248 просмотров • 2 месяцев назад

.L3Harris says Palantir helped them beat frontier AI models in less than 2 days, at 95% lower cost: "When we fine-tuned open source models trained on our own data, we were able to outperform the frontier models in less than 48 hours." "The cost of our fine-tuned open source model was 95% lower than the frontier models we were using." "AI is a commodity. It's all about the data. We view our data as a corporate asset. It's our unique hard-earned knowledge." "We believe American defense companies should not be a vassal for frontier AI labs, handing over our data and institutional knowledge, hoping to rent back the intelligence it creates." " We own the model, we own the compute, we own the advantage."

.L3Harris says Palantir helped them beat frontier AI models in less than 2 days, at 95% lower cost: "When we fine-tuned open source models trained on our own data, we were able to outperform the frontier models in less than 48 hours." "The cost of our fine-tuned open source model was 95% lower than the frontier models we were using." "AI is a commodity. It's all about the data. We view our data as a corporate asset. It's our unique hard-earned knowledge." "We believe American defense companies should not be a vassal for frontier AI labs, handing over our data and institutional knowledge, hoping to rent back the intelligence it creates." " We own the model, we own the compute, we own the advantage."

Jawwwn

1,105,799 просмотров • 29 дней назад

Today, we're launching Vals Public Sector: independent AI evaluation for government. Recent events have made one thing clear: the government has a stake in understanding how frontier AI models perform on day-to-day work, and the risks they carry. This work is central to who we are. We've spent years building industry-leading benchmarks alongside the top AI labs and enterprise domain experts. Now, we’re bringing that same rigor to support government where AI matters most, from public benefits to national security. We are excited for the work ahead!

Today, we're launching Vals Public Sector: independent AI evaluation for government. Recent events have made one thing clear: the government has a stake in understanding how frontier AI models perform on day-to-day work, and the risks they carry. This work is central to who we are. We've spent years building industry-leading benchmarks alongside the top AI labs and enterprise domain experts. Now, we’re bringing that same rigor to support government where AI matters most, from public benefits to national security. We are excited for the work ahead!

Vals AI

11,538 просмотров • 3 месяцев назад

"It all comes down to compute." Dmitri Alperovitch from Silverado Policy Accelerator explains: "The single most important input to winning is compute the processing power used to train and run AI models. Let me say that plainly, because it is defining that everything that congress and this committee do on AI policy. The binding constraint in this competition is not talent, it is not data, and it is not cash. Both the United States and China have talented researchers and financial resources, and we have learned that frontier models trained on generic data routinely outperform smaller models, trained on more specialized inputs. It all comes down to compute."

"It all comes down to compute." Dmitri Alperovitch from Silverado Policy Accelerator explains: "The single most important input to winning is compute the processing power used to train and run AI models. Let me say that plainly, because it is defining that everything that congress and this committee do on AI policy. The binding constraint in this competition is not talent, it is not data, and it is not cash. Both the United States and China have talented researchers and financial resources, and we have learned that frontier models trained on generic data routinely outperform smaller models, trained on more specialized inputs. It all comes down to compute."

Select Committee on China

41,578 просмотров • 5 месяцев назад

Interesting that in three of my recent conversations with Krishna at Anthropic, Dylan Patel, and Gavin Baker, each said that frontier tokens are capturing the majority of the economic value. Gavin: "An overwhelming amount of the economic returns to AI at the model layer have been at the frontier. That's surprising to me, and I think it's been surprising to a lot of people. This is one of the most important questions to be answered, and you need to have a hypothesis on it as an investor." Krishna: "We think the returns to frontier intelligence are extremely high. Customers invest really heavily in more tokens with the newer models. The ones at the frontier clearly are capturing this economic value, driving meaningful ROI for customers. The returns to frontier intelligence are not slowing down." Dylan: "No one gives a crap about GPT-4 class models. They want the frontier because the frontier lets them create the economically valuable things."

Interesting that in three of my recent conversations with Krishna at Anthropic, Dylan Patel, and Gavin Baker, each said that frontier tokens are capturing the majority of the economic value. Gavin: "An overwhelming amount of the economic returns to AI at the model layer have been at the frontier. That's surprising to me, and I think it's been surprising to a lot of people. This is one of the most important questions to be answered, and you need to have a hypothesis on it as an investor." Krishna: "We think the returns to frontier intelligence are extremely high. Customers invest really heavily in more tokens with the newer models. The ones at the frontier clearly are capturing this economic value, driving meaningful ROI for customers. The returns to frontier intelligence are not slowing down." Dylan: "No one gives a crap about GPT-4 class models. They want the frontier because the frontier lets them create the economically valuable things."

Patrick OShaughnessy

121,392 просмотров • 4 месяцев назад

Today we are introducing Tara. Biological datasets are a source of insights and a means to train biological AI models. As the ability to reason at scale emerges, they take on a new role: the ground truth for testing what reasoning models produce, and the environment in which those models operate, get feedback, and improve. Tara, our autonomous research agent, is embedded in our ever-expanding datasets, lab-generated and synthetic, and built to test and evolve the hypotheses frontier models generate, matching the pace at which they produce new ideas. By keeping those models grounded in a vast space of high-precision biological data, we believe we can compound biological reasoning and close the impedance mismatch between hypothesis generation and validation.

Today we are introducing Tara. Biological datasets are a source of insights and a means to train biological AI models. As the ability to reason at scale emerges, they take on a new role: the ground truth for testing what reasoning models produce, and the environment in which those models operate, get feedback, and improve. Tara, our autonomous research agent, is embedded in our ever-expanding datasets, lab-generated and synthetic, and built to test and evolve the hypotheses frontier models generate, matching the pace at which they produce new ideas. By keeping those models grounded in a vast space of high-precision biological data, we believe we can compound biological reasoning and close the impedance mismatch between hypothesis generation and validation.

Nima Alidoust

30,780 просмотров • 3 месяцев назад

AI only has pre-schooler level capabilities for visual tasks. "The frontier models—the best of them—still reason around the age of a preschooler, and any elementary school kid in that benchmark was able to beat all the frontier models on these visual tasks." "These are tasks that are not just object recognition. It's about spatial reasoning... tasks like navigating through the maze…” — Andrew M. Dai, CEO of Elorian

AI only has pre-schooler level capabilities for visual tasks. "The frontier models—the best of them—still reason around the age of a preschooler, and any elementary school kid in that benchmark was able to beat all the frontier models on these visual tasks." "These are tasks that are not just object recognition. It's about spatial reasoning... tasks like navigating through the maze…” — Andrew M. Dai, CEO of Elorian

The Information

26,229 просмотров • 3 месяцев назад

Today I’m excited to introduce micro1 Intelligence, the world’s most advanced platform for training frontier AI models. Achieving AGI is bottlenecked by one main thing: high-quality data. Data based on real-world environments that capture human expert workflows, complex decision-making, and reward signals models need to learn. With micro1 Intelligence, frontier labs can train on RL environments across every subject matter all in one place.

Today I’m excited to introduce micro1 Intelligence, the world’s most advanced platform for training frontier AI models. Achieving AGI is bottlenecked by one main thing: high-quality data. Data based on real-world environments that capture human expert workflows, complex decision-making, and reward signals models need to learn. With micro1 Intelligence, frontier labs can train on RL environments across every subject matter all in one place.

Ali Ansari

333,257 просмотров • 10 месяцев назад

World Labs CEO Dr. Fei-Fei Li & SceniX Co-Founder Yunzhu Li on the data bottleneck in robotics and how world models help: "The lack of data in training, the lack of data in evaluation, this is very, very different from language models, where data is abundant on the internet." "And we know that in order for robotics to work, we have to somehow unlock the power of scaling law. But where does that come from?" "It's a profound problem that everybody's battling with in robotics." "We see a lot of unlock in being able to do this whole process through the modeling of the environments." "Being able to create these digital worlds... that's just going to unlock so much more potential for being able to replace all the costly and the unsafe data in the real environments with the data generated from the worlds for robots to be able to do scalable learning and evaluations." Fei-Fei Li Yunzhu Li martin_casado

World Labs CEO Dr. Fei-Fei Li & SceniX Co-Founder Yunzhu Li on the data bottleneck in robotics and how world models help: "The lack of data in training, the lack of data in evaluation, this is very, very different from language models, where data is abundant on the internet." "And we know that in order for robotics to work, we have to somehow unlock the power of scaling law. But where does that come from?" "It's a profound problem that everybody's battling with in robotics." "We see a lot of unlock in being able to do this whole process through the modeling of the environments." "Being able to create these digital worlds... that's just going to unlock so much more potential for being able to replace all the costly and the unsafe data in the real environments with the data generated from the worlds for robots to be able to do scalable learning and evaluations." Fei-Fei Li Yunzhu Li martin_casado

a16z

78,216 просмотров • 2 месяцев назад

Anthropic CEO Dario Amodei: I’m not a doomer about this. I think this technology can be built safely. And when we’ve looked at, you know, incidents that we have faced, in cases where our models haven’t behaved as perfectly as they could behave, because I think this to some extent has happened to everyone, we’re often able to look at causes, and we look and we say, we had this stage where we train the AI, right? And their behavior at the time that they act, at the time that they do these things, is shaped by their behavior while they’re being trained. And when we look back, what we find is, oops, we included some environments that encourage the models to cheat, that encourage the models to hack out by mistake. And now that we understand, we’re going back, we’re taking more time to make sure that we always have tried to filter out these bad training environments, but we’re trying to do an even more rigorous job of them than we have in the past.

Anthropic CEO Dario Amodei: I’m not a doomer about this. I think this technology can be built safely. And when we’ve looked at, you know, incidents that we have faced, in cases where our models haven’t behaved as perfectly as they could behave, because I think this to some extent has happened to everyone, we’re often able to look at causes, and we look and we say, we had this stage where we train the AI, right? And their behavior at the time that they act, at the time that they do these things, is shaped by their behavior while they’re being trained. And when we look back, what we find is, oops, we included some environments that encourage the models to cheat, that encourage the models to hack out by mistake. And now that we understand, we’re going back, we’re taking more time to make sure that we always have tried to filter out these bad training environments, but we’re trying to do an even more rigorous job of them than we have in the past.

Acyn

25,877 просмотров • 25 дней назад

Earlier this week at GTC, we announced our partnership with Nvidia. We will work with Nvidia to build strong, American open-source models that are at the frontier of scientific reasoning. These models will be essential for the US to compete with China on science in the coming decades. Jensen is committing to spend tens of billions of dollars developing open-source models, and we are excited to be a partner with them in figuring out how to benchmark, train and use those agents to accelerate scientific research. We have already open-sourced some of the work we have done with them, and are looking forward to open-sourcing more. There are few things today that are more important. See our blog post below, and watch the video to learn more, narrated by the man himself.

Earlier this week at GTC, we announced our partnership with Nvidia. We will work with Nvidia to build strong, American open-source models that are at the frontier of scientific reasoning. These models will be essential for the US to compete with China on science in the coming decades. Jensen is committing to spend tens of billions of dollars developing open-source models, and we are excited to be a partner with them in figuring out how to benchmark, train and use those agents to accelerate scientific research. We have already open-sourced some of the work we have done with them, and are looking forward to open-sourcing more. There are few things today that are more important. See our blog post below, and watch the video to learn more, narrated by the man himself.

Sam Rodriques

24,421 просмотров • 6 месяцев назад

🚨 NEW: DNI Director Jay Clayton tells Fox’s Bret Baier on AI: "I think this is a race that we need to win.” “And the reason... is if you talk to people who have seen what these models can do and are projecting what they will be able to do in the future, the power is immense.” “I believe that if we are the strongest country in the world, the world will be a much better place. And I look at AI frontier models in the very same framework."

🚨 NEW: DNI Director Jay Clayton tells Fox’s Bret Baier on AI: "I think this is a race that we need to win.” “And the reason... is if you talk to people who have seen what these models can do and are projecting what they will be able to do in the future, the power is immense.” “I believe that if we are the strongest country in the world, the world will be a much better place. And I look at AI frontier models in the very same framework."

TV News Now

10,946 просмотров • 28 дней назад

As AI labs race to train and deploy new frontier models, existing models become more affordable with better tokenomics. ✨ "Everybody's trying to get to the next frontier. And every time they get to the next frontier, the last generation AI tokens, the cost starts to decline about a factor of 10x every year," said NVIDIA CEO Jensen Huang in a recent keynote. Model optimization techniques such as speculative decoding and multi-token prediction, combined with inference serving platforms like NVIDIA Dynamo on NVIDIA Blackwell NVL72 systems, enable AI factories to boost throughput by 10x with one-tenth of the cost per token. Learn more about AI factory tokenomics ➡️

As AI labs race to train and deploy new frontier models, existing models become more affordable with better tokenomics. ✨ "Everybody's trying to get to the next frontier. And every time they get to the next frontier, the last generation AI tokens, the cost starts to decline about a factor of 10x every year," said NVIDIA CEO Jensen Huang in a recent keynote. Model optimization techniques such as speculative decoding and multi-token prediction, combined with inference serving platforms like NVIDIA Dynamo on NVIDIA Blackwell NVL72 systems, enable AI factories to boost throughput by 10x with one-tenth of the cost per token. Learn more about AI factory tokenomics ➡️

NVIDIA AI

16,053 просмотров • 8 месяцев назад

learned a lot from this conversation with Simon Mo and Matt Bornstein. biggest takeaways for me: -there are a lot of reasons why we should like open-weight models. a lot of these arguments stop at handwavy things like "what if the labs stop releasing frontier models to the public" or "it's lower cost." but simon's position as lead maintainer of vLLM and CEO of Inferact give him authority to talk about some of the other, more interesting and concrete reasons to pay attention to open-weight models, namely that they allow end-users to calibrate latency / other performance metrics with way more customizability than what any of the frontier closed-source labs offer (and without the fear that your job might be met with a refusal at some random point where you're deep in a 2 hour job) -re: the above point...for this reason, a lot of US companies (inferact included!) choose to use open-weight models over their closed-source alternatives. this also isn't limited to internal workloads / research - on a recent a16z podcast the team at Decagon spoke about how something like 90% of their customer service ai agents run on open-weight models that they've fine-tuned. -we should really appreciate how many companies/teams came out researchers fascinated by the wave of very small open-weight models that were being distilled from e.g. gpt-3.5 and earlier models in 2022/2023 (prior to the release of chatGPT!). these small models motivated the development of pagedattention, which then led to vlmm/inferact (at other layers of the stack with similar origin stories, you can look at teams like openrouter or ollama). in other words, we have open-weight models to thank for a bunch of the orchestration infra we now rely on. i think yet another, indirect, way we can point to open-source/weight infra pushing the frontier forward. anyway, a lot more in this convo, it was a lot of fun!

learned a lot from this conversation with Simon Mo and Matt Bornstein. biggest takeaways for me: -there are a lot of reasons why we should like open-weight models. a lot of these arguments stop at handwavy things like "what if the labs stop releasing frontier models to the public" or "it's lower cost." but simon's position as lead maintainer of vLLM and CEO of Inferact give him authority to talk about some of the other, more interesting and concrete reasons to pay attention to open-weight models, namely that they allow end-users to calibrate latency / other performance metrics with way more customizability than what any of the frontier closed-source labs offer (and without the fear that your job might be met with a refusal at some random point where you're deep in a 2 hour job) -re: the above point...for this reason, a lot of US companies (inferact included!) choose to use open-weight models over their closed-source alternatives. this also isn't limited to internal workloads / research - on a recent a16z podcast the team at Decagon spoke about how something like 90% of their customer service ai agents run on open-weight models that they've fine-tuned. -we should really appreciate how many companies/teams came out researchers fascinated by the wave of very small open-weight models that were being distilled from e.g. gpt-3.5 and earlier models in 2022/2023 (prior to the release of chatGPT!). these small models motivated the development of pagedattention, which then led to vlmm/inferact (at other layers of the stack with similar origin stories, you can look at teams like openrouter or ollama). in other words, we have open-weight models to thank for a bunch of the orchestration infra we now rely on. i think yet another, indirect, way we can point to open-source/weight infra pushing the frontier forward. anyway, a lot more in this convo, it was a lot of fun!

Elena

12,922 просмотров • 2 месяцев назад

BREAKING: Only ~1% of Bending Spoons’ AI requests & tokens goes to frontier models. 99% runs on self-hosted open-weight models. CEO Luca Ferrari on the shift to Sovereign AI: “We end up using open-weight models we self-host, so these are basically free. I mean, there is a little bit of cloud cost, but for 99% of the requests and tokens.” “And the frontier models, generally closed weight, through APIs for maybe 1% of the requests, only for the most complex tasks.” “Sometimes we use them automatically to check the work that got done by the slightly less intelligent models, as a more senior engineer would with a more junior engineer.”

BREAKING: Only ~1% of Bending Spoons’ AI requests & tokens goes to frontier models. 99% runs on self-hosted open-weight models. CEO Luca Ferrari on the shift to Sovereign AI: “We end up using open-weight models we self-host, so these are basically free. I mean, there is a little bit of cloud cost, but for 99% of the requests and tokens.” “And the frontier models, generally closed weight, through APIs for maybe 1% of the requests, only for the most complex tasks.” “Sometimes we use them automatically to check the work that got done by the slightly less intelligent models, as a more senior engineer would with a more junior engineer.”

Molly O’Shea

243,526 просмотров • 21 дней назад