Video yükleniyor...

Video Yüklenemedi

Ana Sayfaya Dön

Are open source models catching up to proprietary models? We’ve looked back at 3 years of Arena’s data to show how the race has evolved. For comparison, we’ve taken the top 20% of the models and uncovered the following: - Before mid 2024: The gap was between 100-150 points...

arena's profile picture

Arena.ai

153,108 subscribers

22,447 görüntüleme • 5 ay önce •via X (Twitter)

LIVE

Anya Rossi• Live Now

Private livecam show

0 Yorum

Yorum bulunmuyor

Orijinal gönderinin yorumları burada görünecek

Benzer Videolar

5 patterns in Text Arena's price–performance Pareto frontier since 2023: 1. GPT-4-level quality is now ~500x lower cost. - From a ~$50 blended price per million tokens in 2023 to ~$0.10 today. 2. The higher-price end is both better and lower-priced since 2023. - The leading Arena score has climbed ~170 points (1,330 → 1,500). While the price of the higher-end frontier models dropped from ~$50 to ~$20 per million tokens. 3. The low-cost end gained the most. - Under $0.20 per million tokens, the best available model went from ~1,000 Arena score in 2023 to ~1,440 today. 4. The low-cost/top performance gap has nearly closed. - In 2023, sub-$0.20 models trailed the leader by ~350 Arena points. Today, ~60. 5. The cast has rotated quite a bit. - - OpenAI set the 2023–24 benchmark. - AI at Meta strengthened the low-cost end in 2024. - Google DeepMind drove the 2025 jump. - Anthropic holds the peak in 2026. - xAI and Chinese labs like DeepSeek AI, Z.ai, Kimi.ai, Xiaomi MiMo, and Qwen are continuing to push the mid-price frontier.
0:26

5 patterns in Text Arena's price–performance Pareto frontier since 2023: 1. GPT-4-level quality is now ~500x lower cost. - From a ~$50 blended price per million tokens in 2023 to ~$0.10 today. 2. The higher-price end is both better and lower-priced since 2023. - The leading Arena score has climbed ~170 points (1,330 → 1,500). While the price of the higher-end frontier models dropped from ~$50 to ~$20 per million tokens. 3. The low-cost end gained the most. - Under $0.20 per million tokens, the best available model went from ~1,000 Arena score in 2023 to ~1,440 today. 4. The low-cost/top performance gap has nearly closed. - In 2023, sub-$0.20 models trailed the leader by ~350 Arena points. Today, ~60. 5. The cast has rotated quite a bit. - - OpenAI set the 2023–24 benchmark. - AI at Meta strengthened the low-cost end in 2024. - Google DeepMind drove the 2025 jump. - Anthropic holds the peak in 2026. - xAI and Chinese labs like DeepSeek AI, Z.ai, Kimi.ai, Xiaomi MiMo, and Qwen are continuing to push the mid-price frontier.

Arena.ai

58,133 görüntüleme • 3 ay önce

4 years ago we were on the brink of AI becoming proprietary and centralized, when OpenAI kept GPT3 closed and VCs started dumping money on researchers. From fully open science, to fully closed, in a matter of months. It was scary, and 1,000+ leading researchers and scientists banded together to show the world that it was possible to do the same work in the open, and build an ecosystem that benefits everyone. That was the BigScience Research Workshop BLOOM project, and it put us back on track to open science, starting with forward-thinking organizations like Meta releasing OPT. Look at us now. Open models have not only caught up, they're state of the art now. Not just LLMs, but models for document AI, speech to text, text to speech, generating images and more. We're closing in on 2 million open weight models on Hugging Face. Thanks for the reminder Thomas Wolf .
2:20

4 years ago we were on the brink of AI becoming proprietary and centralized, when OpenAI kept GPT3 closed and VCs started dumping money on researchers. From fully open science, to fully closed, in a matter of months. It was scary, and 1,000+ leading researchers and scientists banded together to show the world that it was possible to do the same work in the open, and build an ecosystem that benefits everyone. That was the BigScience Research Workshop BLOOM project, and it put us back on track to open science, starting with forward-thinking organizations like Meta releasing OPT. Look at us now. Open models have not only caught up, they're state of the art now. Not just LLMs, but models for document AI, speech to text, text to speech, generating images and more. We're closing in on 2 million open weight models on Hugging Face. Thanks for the reminder Thomas Wolf .

Jeff Boudier 🤗

21,947 görüntüleme • 1 yıl önce

.Josh Wolfe: Anybody Using DeepSeek App Is 'Absolute Fool' "Anybody using the DeepSeek app is an absolute fool. If you're using DeepSeek on companies like Together Compute, one of Lux's companies, which can get rid of the CCP censorship, then it's probably okay. But remember, the open-source movement is something we deeply believe in. Most great technologists, entrepreneurs, and venture capitalists are on the side of open source. The closed-source models that have consumed tens of billions of dollars are the ones that are really going to be at risk. When you look at Hugging Face, a major repository, or Together Compute, Runway ML, and a lot of Lux's companies, they have been pioneers in open source. Now, why am I not worried about open source, even with the DeepSeek model? As long as you don't have the CCP censorship on it, the models with their open weights allow people to run on their proprietary data. This means companies like pharma or defense companies that have their own siloed, proprietary data—think about Bloomberg with their proprietary longitudinal data, or Meta with their data—are the ones who will have the edge. Even as open source takes hold, these companies will still dominate. I’m not worried about open source being the problem. I’m more concerned about people overfunding closed models with no proprietary source. A lot of capital is going to be burned there, and we’re already seeing that with people worried about OpenAI in some aspects."
1:27

.Josh Wolfe: Anybody Using DeepSeek App Is 'Absolute Fool' "Anybody using the DeepSeek app is an absolute fool. If you're using DeepSeek on companies like Together Compute, one of Lux's companies, which can get rid of the CCP censorship, then it's probably okay. But remember, the open-source movement is something we deeply believe in. Most great technologists, entrepreneurs, and venture capitalists are on the side of open source. The closed-source models that have consumed tens of billions of dollars are the ones that are really going to be at risk. When you look at Hugging Face, a major repository, or Together Compute, Runway ML, and a lot of Lux's companies, they have been pioneers in open source. Now, why am I not worried about open source, even with the DeepSeek model? As long as you don't have the CCP censorship on it, the models with their open weights allow people to run on their proprietary data. This means companies like pharma or defense companies that have their own siloed, proprietary data—think about Bloomberg with their proprietary longitudinal data, or Meta with their data—are the ones who will have the edge. Even as open source takes hold, these companies will still dominate. I’m not worried about open source being the problem. I’m more concerned about people overfunding closed models with no proprietary source. A lot of capital is going to be burned there, and we’re already seeing that with people worried about OpenAI in some aspects."

Josh Caplan

40,039 görüntüleme • 1 yıl önce

“We have a crisis of open source models in the Western world. Outside of China, there are no good open source models. We don't even have any in the US now. The talent, the capital and the focus to be best-in-class at pre-training, mid-training, post-training is an extremely scarce skillset. The answer for a lot of countries may just be to take an open weights model from China, post-train or fine-tune their own version, and have that be what they start from." Everett Randle Why does the West have such poor open source models and is it a national security threat to rely on open-source Chinese models Demis Hassabis Yann LeCun Aidan Gomez Lilian Weng Jan Leike
1:20

“We have a crisis of open source models in the Western world. Outside of China, there are no good open source models. We don't even have any in the US now. The talent, the capital and the focus to be best-in-class at pre-training, mid-training, post-training is an extremely scarce skillset. The answer for a lot of countries may just be to take an open weights model from China, post-train or fine-tune their own version, and have that be what they start from." Everett Randle Why does the West have such poor open source models and is it a national security threat to rely on open-source Chinese models Demis Hassabis Yann LeCun Aidan Gomez Lilian Weng Jan Leike

Harry Stebbings

63,738 görüntüleme • 2 ay önce

✨Spotlight: Top 4 Code Arena open models Open models are moving fast, and this community is directly shaping how that progress shows up in real use. Your builds, votes, and experiments surface what frontier models can actually do in practice. To celebrate the momentum, we’re spotlighting the top open models in Code Arena (WebDev) 🧵👇 🔥 MiniMax (official) — MiniMax-M2.1 🤖 Z.ai — GLM-4.7 ✨ DeepSeek — Deepseek-v3.2-thinking 🚀 Xiaomi MiMo — Mimo-v2-flash (non-thinking) Let’s keep building in the open and raising the bar together 🚀
1:07

✨Spotlight: Top 4 Code Arena open models Open models are moving fast, and this community is directly shaping how that progress shows up in real use. Your builds, votes, and experiments surface what frontier models can actually do in practice. To celebrate the momentum, we’re spotlighting the top open models in Code Arena (WebDev) 🧵👇 🔥 MiniMax (official) — MiniMax-M2.1 🤖 Z.ai — GLM-4.7 ✨ DeepSeek — Deepseek-v3.2-thinking 🚀 Xiaomi MiMo — Mimo-v2-flash (non-thinking) Let’s keep building in the open and raising the bar together 🚀

Arena.ai

16,798 görüntüleme • 7 ay önce

Today, we’re introducing Forge, a system for enterprises to build frontier-grade AI models grounded in their proprietary knowledge. 🌎 Forge bridges the gap between generic AI and enterprise-specific needs. Instead of relying on broad, public data, organizations can train models that understand their internal context embedded within systems, workflows, and policies, aligning AI with their unique operations. We have already partnered with world-leading organizations, like ASML, DSO National Laboratories Singapore, Ericsson, European Space Agency, Home Team Science and Technology Agency (HTX) Singapore and Reply to train models on the proprietary data that powers their most complex systems and future-defining technologies.
1:28

Today, we’re introducing Forge, a system for enterprises to build frontier-grade AI models grounded in their proprietary knowledge. 🌎 Forge bridges the gap between generic AI and enterprise-specific needs. Instead of relying on broad, public data, organizations can train models that understand their internal context embedded within systems, workflows, and policies, aligning AI with their unique operations. We have already partnered with world-leading organizations, like ASML, DSO National Laboratories Singapore, Ericsson, European Space Agency, Home Team Science and Technology Agency (HTX) Singapore and Reply to train models on the proprietary data that powers their most complex systems and future-defining technologies.

Mistral AI

444,201 görüntüleme • 5 ay önce

CHINA IS CATCHING UP IN AI 🇨🇳 China’s new open-source AI model, GLM-5.2 by Zhipu AI, reportedly matches Anthropic’s Claude Mythos in security bug detection. The model outperformed some Claude versions in independent cybersecurity benchmarks and can achieve Mythos-level performance. Unlike Claude Mythos, GLM-5.2 is open-source and significantly cheaper to run. The gap between US and Chinese AI models is narrowing rapidly.
0:29

CHINA IS CATCHING UP IN AI 🇨🇳 China’s new open-source AI model, GLM-5.2 by Zhipu AI, reportedly matches Anthropic’s Claude Mythos in security bug detection. The model outperformed some Claude versions in independent cybersecurity benchmarks and can achieve Mythos-level performance. Unlike Claude Mythos, GLM-5.2 is open-source and significantly cheaper to run. The gap between US and Chinese AI models is narrowing rapidly.

CryptoGoos

71,944 görüntüleme • 2 ay önce

Sam on Kimi, distillation, and open source: “I have always assumed that there are going to be great cheap models in the world, and we better be the greatest and the cheapest. You get a better deal today, at least at a particular latency, using OpenAI's models than Kimi. We distill our own models. That’s how we make smaller, cheaper models. That’s a very good thing to do. There will clearly be an important place for open source models in the world and people that will want their own weights. We have so much usage of our models that we do not need to be a gigantically high-margin business to be able to afford model training. I would rather people not distill from us, for sure. But this is not in my top ten list of worries.”
2:12

Sam on Kimi, distillation, and open source: “I have always assumed that there are going to be great cheap models in the world, and we better be the greatest and the cheapest. You get a better deal today, at least at a particular latency, using OpenAI's models than Kimi. We distill our own models. That’s how we make smaller, cheaper models. That’s a very good thing to do. There will clearly be an important place for open source models in the world and people that will want their own weights. We have so much usage of our models that we do not need to be a gigantically high-margin business to be able to afford model training. I would rather people not distill from us, for sure. But this is not in my top ten list of worries.”

Patrick OShaughnessy

269,663 görüntüleme • 1 ay önce

Open source AI is gaining traction due to the White House’s clampdown on Anthropic & OpenAI. “A lot of founders and developers have been burned by trying to build apps on top of these really advanced closed source models, like Fable for instance…” “… and then seeing those models get taken down or removed from the market just days later.” — Stephanie Palazzolo, AI Reporter
0:57

Open source AI is gaining traction due to the White House’s clampdown on Anthropic & OpenAI. “A lot of founders and developers have been burned by trying to build apps on top of these really advanced closed source models, like Fable for instance…” “… and then seeing those models get taken down or removed from the market just days later.” — Stephanie Palazzolo, AI Reporter

The Information

19,314 görüntüleme • 2 ay önce

Jensen Huang was asked whether AI's future is open source or proprietary the question the whole industry keeps fighting over. He didn't pick a side. He said something that quietly ends the fight. "models is a technology, not a product. Model is a technology, not a service" "For the vast majority of consumers, the horizontal layer, the general intelligence, I would really, really love not to go fine-tune my own" "all these industries, their domain expertise, their specialization has to be channeled, has to be captured in a way that they can control, and that it can only come from open models" "the open model industry we're contributing tremendously to it is near the frontier" Everyone else is arguing about which lab wins OpenAI or Anthropic, who has the smartest frontier model this quarter. Jensen is standing on stage saying that's the sideshow. The proprietary model is the convenience layer you rent when you don't care. The open model is the thing you own when the work is yours. Two different products for two different needs not two horses in one race. open models are the second-biggest in the room. Ahead of Anthropic. Behind only OpenAI.
2:51

Jensen Huang was asked whether AI's future is open source or proprietary the question the whole industry keeps fighting over. He didn't pick a side. He said something that quietly ends the fight. "models is a technology, not a product. Model is a technology, not a service" "For the vast majority of consumers, the horizontal layer, the general intelligence, I would really, really love not to go fine-tune my own" "all these industries, their domain expertise, their specialization has to be channeled, has to be captured in a way that they can control, and that it can only come from open models" "the open model industry we're contributing tremendously to it is near the frontier" Everyone else is arguing about which lab wins OpenAI or Anthropic, who has the smartest frontier model this quarter. Jensen is standing on stage saying that's the sideshow. The proprietary model is the convenience layer you rent when you don't care. The open model is the thing you own when the work is yours. Two different products for two different needs not two horses in one race. open models are the second-biggest in the room. Ahead of Anthropic. Behind only OpenAI.

Vikram M

12,078 görüntüleme • 1 ay önce

The First 100 begins now. Verify Quality is live on PrismaX. For the first time, anyone can score the robot training data that models learn from, earn points, and compete to become one of The First 100. Better data. Better models. The standard starts with you.
0:34

The First 100 begins now. Verify Quality is live on PrismaX. For the first time, anyone can score the robot training data that models learn from, earn points, and compete to become one of The First 100. Better data. Better models. The standard starts with you.

PrismaX

225,269 görüntüleme • 2 ay önce

JUST IN: 🇨🇳 China’s new open-source AI model, GLM-5.2 by Zhipu AI, reportedly matches Anthropic’s Claude Mythos in security bug detection. The model outperformed some Claude versions in independent cybersecurity benchmarks and can achieve Mythos-level performance. Unlike Claude Mythos, GLM-5.2 is open-source and significantly cheaper to run. The gap between US and Chinese AI models is narrowing rapidly.
0:32

JUST IN: 🇨🇳 China’s new open-source AI model, GLM-5.2 by Zhipu AI, reportedly matches Anthropic’s Claude Mythos in security bug detection. The model outperformed some Claude versions in independent cybersecurity benchmarks and can achieve Mythos-level performance. Unlike Claude Mythos, GLM-5.2 is open-source and significantly cheaper to run. The gap between US and Chinese AI models is narrowing rapidly.

CryptoTweets

62,983 görüntüleme • 2 ay önce

Factory (Factory) CEO Matan Grinberg (Matan Grinberg) says "In the next 12 months, 90% of tokens will be going to open models." Enterprises are already shifting rapidly.. "At the beginning of the year, <1% of their tokens were going to open models." "Around March, it crossed 1%... & in May it crossed 10%." "These open models are incredibly performant, incredibly cheap, incredibly fast." While frontier closed models aren't going away: "They'll still have a place, but I think that place will be shrinking—at least in token share." And by the end of 2026? "At least in the enterprise, we'll probably cross the 50% threshold."
2:40

Factory (Factory) CEO Matan Grinberg (Matan Grinberg) says "In the next 12 months, 90% of tokens will be going to open models." Enterprises are already shifting rapidly.. "At the beginning of the year, <1% of their tokens were going to open models." "Around March, it crossed 1%... & in May it crossed 10%." "These open models are incredibly performant, incredibly cheap, incredibly fast." While frontier closed models aren't going away: "They'll still have a place, but I think that place will be shrinking—at least in token share." And by the end of 2026? "At least in the enterprise, we'll probably cross the 50% threshold."

Molly O’Shea

120,949 görüntüleme • 1 ay önce

China is once again unsettling its American competitors in the AI race. Chinese labs have returned to the spotlight with the unveiling of GLM 5.2, described as the most powerful AI model ever trained in China — and one of the leading open‑source models worldwide. 🔍 Key highlights of “GLM 5.2”: ♦️ It is considered the strongest AI model developed in China to date, and ranks among the smartest open‑source models available. ♦️ It operates at a cost less than one‑tenth that of Anthropic’s latest models, making it a far more economical option. ♦️ It is likely to attract developers and companies seeking powerful, low‑cost models — especially amid growing technical and geopolitical restrictions. ♦️ Some evaluations place the model among the top globally, trailing OpenAI’s models while outperforming several competitors in specialized benchmarks and performance tests.
0:42

China is once again unsettling its American competitors in the AI race. Chinese labs have returned to the spotlight with the unveiling of GLM 5.2, described as the most powerful AI model ever trained in China — and one of the leading open‑source models worldwide. 🔍 Key highlights of “GLM 5.2”: ♦️ It is considered the strongest AI model developed in China to date, and ranks among the smartest open‑source models available. ♦️ It operates at a cost less than one‑tenth that of Anthropic’s latest models, making it a far more economical option. ♦️ It is likely to attract developers and companies seeking powerful, low‑cost models — especially amid growing technical and geopolitical restrictions. ♦️ Some evaluations place the model among the top globally, trailing OpenAI’s models while outperforming several competitors in specialized benchmarks and performance tests.

China pulse 🇨🇳

14,293 görüntüleme • 2 ay önce

We got attacked by secret unreleased proprietary models and defended ourselves with an open model, more precisely the NVIDIA quantized version of GLM 5.2 coming from Z.ai. Banning any open model would hurt first cyber security defenders, startups, small companies, researchers and everyone who's not a frontier lab and need on-prem affordable controlable models to compete and protect themselves. Let's not do that!
1:21

We got attacked by secret unreleased proprietary models and defended ourselves with an open model, more precisely the NVIDIA quantized version of GLM 5.2 coming from Z.ai. Banning any open model would hurt first cyber security defenders, startups, small companies, researchers and everyone who's not a frontier lab and need on-prem affordable controlable models to compete and protect themselves. Let's not do that!

clem 🤗

393,801 görüntüleme • 1 ay önce

Announcing Artificial Analysis Video Arena - the first crowdsourced comparison for Text to Video models Text to Video models are accelerating rapidly and crossing quality thresholds every month. We created Video Arena to compare them using the only source of truth for visual media - human preference! Video Arena includes hundreds of videos from the leading video models, including: - Runway's Runway Gen 3 Alpha - Pika's Pika 1.5 - Luma's Dream Machine - MiniMax / Hailuo AI (MiniMax) - Kling AI's Kling 1.0 - Zhipu AI's CogVideoX-5B Voting is open now and we’ll be announcing the first leaderboard results within 24 hours. Any predictions? In the meantime, you can see your own ‘personal leaderboard’ of how you’ve ranked the video models after 30 votes. Link to the Artificial Analysis Video Arena in the below tweet! 👇
0:20

Announcing Artificial Analysis Video Arena - the first crowdsourced comparison for Text to Video models Text to Video models are accelerating rapidly and crossing quality thresholds every month. We created Video Arena to compare them using the only source of truth for visual media - human preference! Video Arena includes hundreds of videos from the leading video models, including: - Runway's Runway Gen 3 Alpha - Pika's Pika 1.5 - Luma's Dream Machine - MiniMax / Hailuo AI (MiniMax) - Kling AI's Kling 1.0 - Zhipu AI's CogVideoX-5B Voting is open now and we’ll be announcing the first leaderboard results within 24 hours. Any predictions? In the meantime, you can see your own ‘personal leaderboard’ of how you’ve ranked the video models after 30 votes. Link to the Artificial Analysis Video Arena in the below tweet! 👇

Artificial Analysis

27,108 görüntüleme • 1 yıl önce

Earlier this week at GTC, we announced our partnership with Nvidia. We will work with Nvidia to build strong, American open-source models that are at the frontier of scientific reasoning. These models will be essential for the US to compete with China on science in the coming decades. Jensen is committing to spend tens of billions of dollars developing open-source models, and we are excited to be a partner with them in figuring out how to benchmark, train and use those agents to accelerate scientific research. We have already open-sourced some of the work we have done with them, and are looking forward to open-sourcing more. There are few things today that are more important. See our blog post below, and watch the video to learn more, narrated by the man himself.
1:44

Earlier this week at GTC, we announced our partnership with Nvidia. We will work with Nvidia to build strong, American open-source models that are at the frontier of scientific reasoning. These models will be essential for the US to compete with China on science in the coming decades. Jensen is committing to spend tens of billions of dollars developing open-source models, and we are excited to be a partner with them in figuring out how to benchmark, train and use those agents to accelerate scientific research. We have already open-sourced some of the work we have done with them, and are looking forward to open-sourcing more. There are few things today that are more important. See our blog post below, and watch the video to learn more, narrated by the man himself.

Sam Rodriques

23,763 görüntüleme • 5 ay önce

Today we are introducing Tara. Biological datasets are a source of insights and a means to train biological AI models. As the ability to reason at scale emerges, they take on a new role: the ground truth for testing what reasoning models produce, and the environment in which those models operate, get feedback, and improve. Tara, our autonomous research agent, is embedded in our ever-expanding datasets, lab-generated and synthetic, and built to test and evolve the hypotheses frontier models generate, matching the pace at which they produce new ideas. By keeping those models grounded in a vast space of high-precision biological data, we believe we can compound biological reasoning and close the impedance mismatch between hypothesis generation and validation.
4:03

Today we are introducing Tara. Biological datasets are a source of insights and a means to train biological AI models. As the ability to reason at scale emerges, they take on a new role: the ground truth for testing what reasoning models produce, and the environment in which those models operate, get feedback, and improve. Tara, our autonomous research agent, is embedded in our ever-expanding datasets, lab-generated and synthetic, and built to test and evolve the hypotheses frontier models generate, matching the pace at which they produce new ideas. By keeping those models grounded in a vast space of high-precision biological data, we believe we can compound biological reasoning and close the impedance mismatch between hypothesis generation and validation.

Nima Alidoust

30,037 görüntüleme • 2 ay önce

.Bill Gurley “I don’t think politicians understand how powerful open source is as a competitive weapon. If you have ten companies all doing open source, the minute an idea is shared, it's available to the other ten and they absorb it immediately. I think the odds that the U.S., with two proprietary models, can stay ahead of China, with ten open source ones, is very low.”
1:48

.Bill Gurley “I don’t think politicians understand how powerful open source is as a competitive weapon. If you have ten companies all doing open source, the minute an idea is shared, it's available to the other ten and they absorb it immediately. I think the odds that the U.S., with two proprietary models, can stay ahead of China, with ten open source ones, is very low.”

Codie Sanchez

65,669 görüntüleme • 5 ay önce

Something cracked in 2016. Since the late 1970s, the share of teenagers unsure about having children has consistently stayed below 20%. For decades, the gap between boys and girls remained very small, typically just 2 percentage points or less. Boys’ uncertainty has stayed largely stable over time. But girls’ uncertainty skyrocketed after 2016, widening the gender gap to 5 percentage points and continuing to grow, reaching a 9-point difference by 2024. The bottom line is boys haven’t changed much. Girls have.
0:26

Something cracked in 2016. Since the late 1970s, the share of teenagers unsure about having children has consistently stayed below 20%. For decades, the gap between boys and girls remained very small, typically just 2 percentage points or less. Boys’ uncertainty has stayed largely stable over time. But girls’ uncertainty skyrocketed after 2016, widening the gender gap to 5 percentage points and continuing to grow, reaching a 9-point difference by 2024. The bottom line is boys haven’t changed much. Girls have.

Rothmus 🏴

16,915 görüntüleme • 3 ay önce