Video wird geladen...

Video konnte nicht geladen werden

Zur Startseite

Arena AutoEval can quickly evaluate new models, allowing model labs to leverage results faster. It compresses the evaluation feedback loop from weeks to hours, enabling rapid, human-preference-based insight into a model’s capabilities. For more details, see the full video in the post below.

arena's profile picture

Arena.ai

229,647 subscribers

21,583 Aufrufe • vor 3 Monaten •via X (Twitter)

Wissenschaft & Technologie
LIVE

Anya Rossi• Live Now

Private livecam show

6 Kommentare

Profilbild von Arena.ai
Arena.aivor 3 Monaten

Watch the full video at and find more details our AutoEval methodology in the article below

Profilbild von Sergio
Sergiovor 3 Monaten

so when are you gonna start using that crap? still no DS Pro results, also no DS Flash image to website results

Profilbild von NDIZEYE
NDIZEYEvor 3 Monaten

These benchmarks are starting to lose credibility at some point, we will need better approaches to evaluate model intelligence not just how well a model performs on a benchmark.

Profilbild von EricLeifYang
EricLeifYangvor 3 Monaten

where is deepseek-v4-pro？😂

Profilbild von cheesylikeme
cheesylikemevor 3 Monaten

Hours not weeks is the claim. AutoEval still has to lose to a human on a task the rubric did not anticipate, or it is just a faster echo.

Profilbild von Aloya
Aloyavor 3 Monaten

What's interesting here is that compressing the feedback loop from weeks to hours doesn't just save time - it changes the nature of the decisions being made. When results arrive in hours, teams iterate. When they arrive in weeks, teams commit. Those are fundamentally different cognitive modes, and the second one is where a lot of model development debt gets quietly accumulated. The speed improvement here isn't cosmetic.

Ähnliche Videos

Arena AutoEval uses tens of millions of historical Arena battles from humans, to train a reward model that can estimate human preferences at scale, and deliver evaluation results at frontier speed. Unlike LLM-as-judge methods, AutoEval remains a live benchmark: it is derived from existing real-world battles, with the reward model scoring from human prompts and model responses. Arena ML Engineers, Haoran Guo and Wayne Zhang, discuss AutoEval below. Watch the full video to learn more about AutoEval’s methodology and model lab use cases in the post below.
2:29

Arena AutoEval uses tens of millions of historical Arena battles from humans, to train a reward model that can estimate human preferences at scale, and deliver evaluation results at frontier speed. Unlike LLM-as-judge methods, AutoEval remains a live benchmark: it is derived from existing real-world battles, with the reward model scoring from human prompts and model responses. Arena ML Engineers, Haoran Guo and Wayne Zhang, discuss AutoEval below. Watch the full video to learn more about AutoEval’s methodology and model lab use cases in the post below.

Arena.ai

34,516 Aufrufe • vor 3 Monaten

ChatGPT learned from human feedback. The next generation of AI will learn from the laws of nature. Liam Fedus (co-creator of ChatGPT, now at Periodic Labs) says models will train through “experiment in the loop”, using real-world results, not human preference, as their reward function. Liam Fedus Periodic Labs
1:40

ChatGPT learned from human feedback. The next generation of AI will learn from the laws of nature. Liam Fedus (co-creator of ChatGPT, now at Periodic Labs) says models will train through “experiment in the loop”, using real-world results, not human preference, as their reward function. Liam Fedus Periodic Labs

a16z

73,184 Aufrufe • vor 1 Jahr

What happens when robot world models learn from human experience at scale? 🤔 DreamDojo from NVIDIA Research is a generalist robot world model pretrained on 44K hours of egocentric human videos and then post-trained on robot data to generalize across new objects and environments. After distillation, it runs at 10 FPS for live teleoperation, policy evaluation, and model-based planning. Read the ICML paper to learn more 📄
0:32

What happens when robot world models learn from human experience at scale? 🤔 DreamDojo from NVIDIA Research is a generalist robot world model pretrained on 44K hours of egocentric human videos and then post-trained on robot data to generalize across new objects and environments. After distillation, it runs at 10 FPS for live teleoperation, policy evaluation, and model-based planning. Read the ICML paper to learn more 📄

NVIDIA Robotics

23,580 Aufrufe • vor 4 Monaten

Announcing Artificial Analysis Video Arena - the first crowdsourced comparison for Text to Video models Text to Video models are accelerating rapidly and crossing quality thresholds every month. We created Video Arena to compare them using the only source of truth for visual media - human preference! Video Arena includes hundreds of videos from the leading video models, including: - Runway's Runway Gen 3 Alpha - Pika's Pika 1.5 - Luma's Dream Machine - MiniMax / Hailuo AI (MiniMax) - Kling AI's Kling 1.0 - Zhipu AI's CogVideoX-5B Voting is open now and we’ll be announcing the first leaderboard results within 24 hours. Any predictions? In the meantime, you can see your own ‘personal leaderboard’ of how you’ve ranked the video models after 30 votes. Link to the Artificial Analysis Video Arena in the below tweet! 👇
0:20

Announcing Artificial Analysis Video Arena - the first crowdsourced comparison for Text to Video models Text to Video models are accelerating rapidly and crossing quality thresholds every month. We created Video Arena to compare them using the only source of truth for visual media - human preference! Video Arena includes hundreds of videos from the leading video models, including: - Runway's Runway Gen 3 Alpha - Pika's Pika 1.5 - Luma's Dream Machine - MiniMax / Hailuo AI (MiniMax) - Kling AI's Kling 1.0 - Zhipu AI's CogVideoX-5B Voting is open now and we’ll be announcing the first leaderboard results within 24 hours. Any predictions? In the meantime, you can see your own ‘personal leaderboard’ of how you’ve ranked the video models after 30 votes. Link to the Artificial Analysis Video Arena in the below tweet! 👇

Artificial Analysis

27,108 Aufrufe • vor 2 Jahren

🟩 Tasks that took hours can now be done with simple commands. Saksham Choudhary from Humyn Labs explains what it really means to be the human in the loop.
0:37

🟩 Tasks that took hours can now be done with simple commands. Saksham Choudhary from Humyn Labs explains what it really means to be the human in the loop.

KGeN Foundation 🟩

103,082 Aufrufe • vor 6 Monaten

The more AI can do, the more we need to ask what it should and shouldn’t do. OpenAI researcher Jason Wolfe joins host Andrew Mayne to explore the Model Spec, the public framework that defines how models are intended to behave. They break down how it works in practice, from the chain of command that resolves conflicting instructions to the way it evolves over time through real-world use, feedback, and new model capabilities.
37:26

The more AI can do, the more we need to ask what it should and shouldn’t do. OpenAI researcher Jason Wolfe joins host Andrew Mayne to explore the Model Spec, the public framework that defines how models are intended to behave. They break down how it works in practice, from the chain of command that resolves conflicting instructions to the way it evolves over time through real-world use, feedback, and new model capabilities.

OpenAI

226,532 Aufrufe • vor 7 Monaten

We've reduced the time it takes to go from physical prompt → robot behavior. The faster anyone can teach a robot to do something new, the easier it becomes to scale physical work. Read more about GEN-1.5 in our blog post in the comments below.
0:21

We've reduced the time it takes to go from physical prompt → robot behavior. The faster anyone can teach a robot to do something new, the easier it becomes to scale physical work. Read more about GEN-1.5 in our blog post in the comments below.

Generalist

27,829 Aufrufe • vor 2 Monaten

Humans learn and improve from failures. Similarly, foundation models adapt based on human feedback. Can we leverage this failure understanding to enhance robotics systems that use foundation models? Introducing AHA—a vision-language model for detecting and reasoning over failures in robotic manipulation. Project page: 🧵Thread👇 Aha!
1:29

Humans learn and improve from failures. Similarly, foundation models adapt based on human feedback. Can we leverage this failure understanding to enhance robotics systems that use foundation models? Introducing AHA—a vision-language model for detecting and reasoning over failures in robotic manipulation. Project page: 🧵Thread👇 Aha!

Jiafei Duan

48,860 Aufrufe • vor 2 Jahren

Introducing Prompt-to-leaderboard (P2L): a real-time LLM leaderboard tailored exactly to your use case! P2L trains an LLM to generate "prompt-specific" leaderboards, so you can input a prompt and get a leaderboard specifically for that prompt. The model is trained on the 2M human preference votes from Chatbot Arena. P2L Highlights: 🔹Instant leaderboard for any prompt 🗿 🔹Optimal model routing (hit #1 on Chatbot Arena in Jan 2025 with 1395 score 🧏) 🔹Fine-grained model strength & weakness analysis 🤓 Check out our demo and thread below for more details!
1:06

Introducing Prompt-to-leaderboard (P2L): a real-time LLM leaderboard tailored exactly to your use case! P2L trains an LLM to generate "prompt-specific" leaderboards, so you can input a prompt and get a leaderboard specifically for that prompt. The model is trained on the 2M human preference votes from Chatbot Arena. P2L Highlights: 🔹Instant leaderboard for any prompt 🗿 🔹Optimal model routing (hit #1 on Chatbot Arena in Jan 2025 with 1395 score 🧏) 🔹Fine-grained model strength & weakness analysis 🤓 Check out our demo and thread below for more details!

Arena.ai

122,599 Aufrufe • vor 1 Jahr

Demo 2: human + agent design loop inside Codex. Not every workflow needs the agent to drive everything. You design in Open Design. Codex reviews the output. It can inspect files, open pages, test interactions, critique the design, and turn feedback into the next prompt. Design becomes a human + agent feedback loop.
1:08

Demo 2: human + agent design loop inside Codex. Not every workflow needs the agent to drive everything. You design in Open Design. Codex reviews the output. It can inspect files, open pages, test interactions, critique the design, and turn feedback into the next prompt. Design becomes a human + agent feedback loop.

Open Design

20,103 Aufrufe • vor 5 Monaten

UPDATE! Human Synapse Decoder has a massive AI model update! In the video below, this is me learning something in realtime! I go from active listening to inner contemplation as the AI feeds out a single word at a time for me to read and hear. This is the Symbiotic Feedback loop of the Human Synapse Decoder were the AI adjusts outputs based on MY cognition. I use off the shelf brainwave detection circuits in a pipeline of 5 AI models decoding the 36 signals from my crude cap. Reading brainwave is not new. What is new, for the first time built a closed loop system where the AI can understand if the outputs are being useful and adjust in real-time dynamically. The float from Attention to Meditation is vital to understanding. You hear a point, you consider a point. Someday your Local AI will do this and you may just let your AI use other AI. More soon…
2:29

UPDATE! Human Synapse Decoder has a massive AI model update! In the video below, this is me learning something in realtime! I go from active listening to inner contemplation as the AI feeds out a single word at a time for me to read and hear. This is the Symbiotic Feedback loop of the Human Synapse Decoder were the AI adjusts outputs based on MY cognition. I use off the shelf brainwave detection circuits in a pipeline of 5 AI models decoding the 36 signals from my crude cap. Reading brainwave is not new. What is new, for the first time built a closed loop system where the AI can understand if the outputs are being useful and adjust in real-time dynamically. The float from Attention to Meditation is vital to understanding. You hear a point, you consider a point. Someday your Local AI will do this and you may just let your AI use other AI. More soon…

Brian Roemmele

24,986 Aufrufe • vor 1 Monat

Today we’re announcing two new updates in our computer vision work — a new, expanded license for our DINOv2 model and the release of FACET, a comprehensive new benchmark dataset to help evaluate and improve fairness in vision models. More details ➡️ 🧵
0:27

Today we’re announcing two new updates in our computer vision work — a new, expanded license for our DINOv2 model and the release of FACET, a comprehensive new benchmark dataset to help evaluate and improve fairness in vision models. More details ➡️ 🧵

AI at Meta

454,160 Aufrufe • vor 3 Jahren

Everyday for the past 2 weeks, we've been sharing something new from GEN-1, our latest milestone in scaling robot learning. This has never been done before. Going from ideas to skills in days (or faster) is what physical AI models should deliver. More coming. Stay tuned. Read more about it in our blog post in the comments below ↓
0:42

Everyday for the past 2 weeks, we've been sharing something new from GEN-1, our latest milestone in scaling robot learning. This has never been done before. Going from ideas to skills in days (or faster) is what physical AI models should deliver. More coming. Stay tuned. Read more about it in our blog post in the comments below ↓

Generalist

40,860 Aufrufe • vor 6 Monaten

Introducing Aiur Labs The new parent company to JRNY CLUB 🚨 To celebrate I'm doing a giveaway with 3 winners of $300 of Ethereum To enter just repost this, check out the video, follow Aiur Labs and comment your feedback below Winners in 48 hours :)
4:16

Introducing Aiur Labs The new parent company to JRNY CLUB 🚨 To celebrate I'm doing a giveaway with 3 winners of $300 of Ethereum To enter just repost this, check out the video, follow Aiur Labs and comment your feedback below Winners in 48 hours :)

JRNY Crypto

144,510 Aufrufe • vor 2 Jahren

Introducing Hedra Model Factory: an autonomous inference and training platform for building models for the real world. The recursive self-improvement loop behind frontier research has been locked inside a few labs. We're opening it to teams in robotics, manufacturing, biology and defense. Our researchers come from Stanford, NVIDIA and Meta Superintelligence Labs. Now we work alongside your team to post-train and optimize your models.
1:58

Introducing Hedra Model Factory: an autonomous inference and training platform for building models for the real world. The recursive self-improvement loop behind frontier research has been locked inside a few labs. We're opening it to teams in robotics, manufacturing, biology and defense. Our researchers come from Stanford, NVIDIA and Meta Superintelligence Labs. Now we work alongside your team to post-train and optimize your models.

Hedra

2,438,700 Aufrufe • vor 2 Tagen

An interactive world model developed by NVIDIA in collaboration with academic partners. - DreamDojo turns egocentric human video data into physical intelligence. - Human data is more scalable than robotics data but lacks action labels. - To solve this, a dedicated action model extracts latent actions by identifying physics and motion deltas between frames. Training - A massive 44k hours of video data are used for pre-training. - Post-training on small-scale robot datasets maps human physics to specific robot embodiments. - An additional distillation stage converts the model into an autoregressive, few-step diffusion model, enabling real-time, action-controllable simulation. Primary Use Cases - Live Teleoperation: Controlling a robot inside a world simulation in real-time. - Model-based Planning: Previewing and curating the best actions for improved success. - Policy Evaluation: Testing robot policies in realistic, out-of-distribution scenarios. Everything that's open-sourced: weights, code, post-training dataset, eval set, and details to reproduce.
1:14

An interactive world model developed by NVIDIA in collaboration with academic partners. - DreamDojo turns egocentric human video data into physical intelligence. - Human data is more scalable than robotics data but lacks action labels. - To solve this, a dedicated action model extracts latent actions by identifying physics and motion deltas between frames. Training - A massive 44k hours of video data are used for pre-training. - Post-training on small-scale robot datasets maps human physics to specific robot embodiments. - An additional distillation stage converts the model into an autoregressive, few-step diffusion model, enabling real-time, action-controllable simulation. Primary Use Cases - Live Teleoperation: Controlling a robot inside a world simulation in real-time. - Model-based Planning: Previewing and curating the best actions for improved success. - Policy Evaluation: Testing robot policies in realistic, out-of-distribution scenarios. Everything that's open-sourced: weights, code, post-training dataset, eval set, and details to reproduce.

The Humanoid Hub

11,575 Aufrufe • vor 8 Monaten

Did you know there is 1 setting that makes a world of difference for a lightning fast trading DOM? If you compare the 2 DOMs in the video below, you should be able to clearly see how one appears much faster than the other. I'll share more details in the post below 🧵
0:42

Did you know there is 1 setting that makes a world of difference for a lightning fast trading DOM? If you compare the 2 DOMs in the video below, you should be able to clearly see how one appears much faster than the other. I'll share more details in the post below 🧵

Ziggy

12,276 Aufrufe • vor 1 Jahr

Announcing our new event - Buildathon: The Rapid Engineering Competition. See the video for details, and please apply to participate!
1:42

Announcing our new event - Buildathon: The Rapid Engineering Competition. See the video for details, and please apply to participate!

Andrew Ng

98,392 Aufrufe • vor 1 Jahr

In case you missed it, we recently launched "Post-training of LLMs," a short course where you'll: ✅ Understand when and why to use post-training methods like Supervised Fine-Tuning (SFT), Direct Preference Optimization (DPO), and Online Reinforcement Learning. ✅ Learn the concepts underlying the three post-training methods of SFT, DPO, and Online RL, their common use-cases, and how to curate high-quality data to effectively train a model using each method. ✅ Download a pre-trained model and implement post-training pipelines to turn a base model into an instruct model, change the identity of a chat assistant, and improve a model’s math capabilities. Learn more and enroll for free:
1:31

In case you missed it, we recently launched "Post-training of LLMs," a short course where you'll: ✅ Understand when and why to use post-training methods like Supervised Fine-Tuning (SFT), Direct Preference Optimization (DPO), and Online Reinforcement Learning. ✅ Learn the concepts underlying the three post-training methods of SFT, DPO, and Online RL, their common use-cases, and how to curate high-quality data to effectively train a model using each method. ✅ Download a pre-trained model and implement post-training pipelines to turn a base model into an instruct model, change the identity of a chat assistant, and improve a model’s math capabilities. Learn more and enroll for free:

DeepLearning.AI

16,779 Aufrufe • vor 1 Jahr