Loading video...

Video Failed to Load

There was a problem loading this video. This could be due to a temporary network issue or the video might be unavailable.

Arena AutoEval can quickly evaluate new models, allowing model labs to leverage results faster. It compresses the evaluation feedback loop from weeks to hours, enabling rapid, human-preference-based insight into a model’s capabilities. For more details, see the full video in the post below.

Arena.ai

218,789 subscribers

21,381 views • 14 days ago •via X (Twitter)

Anya Rossi• Live Now

Private livecam show

0 Comments

No comments available

Comments from the original post will appear here

Related Videos

What happens when robot world models learn from human experience at scale? 🤔 DreamDojo from NVIDIA Research is a generalist robot world model pretrained on 44K hours of egocentric human videos and then post-trained on robot data to generalize across new objects and environments. After distillation, it runs at 10 FPS for live teleoperation, policy evaluation, and model-based planning. Read the ICML paper to learn more 📄

What happens when robot world models learn from human experience at scale? 🤔 DreamDojo from NVIDIA Research is a generalist robot world model pretrained on 44K hours of egocentric human videos and then post-trained on robot data to generalize across new objects and environments. After distillation, it runs at 10 FPS for live teleoperation, policy evaluation, and model-based planning. Read the ICML paper to learn more 📄

NVIDIA Robotics

22,958 views • 1 month ago

Announcing Artificial Analysis Video Arena - the first crowdsourced comparison for Text to Video models Text to Video models are accelerating rapidly and crossing quality thresholds every month. We created Video Arena to compare them using the only source of truth for visual media - human preference! Video Arena includes hundreds of videos from the leading video models, including: - Runway's Runway Gen 3 Alpha - Pika's Pika 1.5 - Luma's Dream Machine - MiniMax / Hailuo AI (MiniMax) - Kling AI's Kling 1.0 - Zhipu AI's CogVideoX-5B Voting is open now and we’ll be announcing the first leaderboard results within 24 hours. Any predictions? In the meantime, you can see your own ‘personal leaderboard’ of how you’ve ranked the video models after 30 votes. Link to the Artificial Analysis Video Arena in the below tweet! 👇

Announcing Artificial Analysis Video Arena - the first crowdsourced comparison for Text to Video models Text to Video models are accelerating rapidly and crossing quality thresholds every month. We created Video Arena to compare them using the only source of truth for visual media - human preference! Video Arena includes hundreds of videos from the leading video models, including: - Runway's Runway Gen 3 Alpha - Pika's Pika 1.5 - Luma's Dream Machine - MiniMax / Hailuo AI (MiniMax) - Kling AI's Kling 1.0 - Zhipu AI's CogVideoX-5B Voting is open now and we’ll be announcing the first leaderboard results within 24 hours. Any predictions? In the meantime, you can see your own ‘personal leaderboard’ of how you’ve ranked the video models after 30 votes. Link to the Artificial Analysis Video Arena in the below tweet! 👇

Artificial Analysis

27,108 views • 1 year ago

The more AI can do, the more we need to ask what it should and shouldn’t do. OpenAI researcher Jason Wolfe joins host Andrew Mayne to explore the Model Spec, the public framework that defines how models are intended to behave. They break down how it works in practice, from the chain of command that resolves conflicting instructions to the way it evolves over time through real-world use, feedback, and new model capabilities.

The more AI can do, the more we need to ask what it should and shouldn’t do. OpenAI researcher Jason Wolfe joins host Andrew Mayne to explore the Model Spec, the public framework that defines how models are intended to behave. They break down how it works in practice, from the chain of command that resolves conflicting instructions to the way it evolves over time through real-world use, feedback, and new model capabilities.

OpenAI

225,838 views • 5 months ago

Humans learn and improve from failures. Similarly, foundation models adapt based on human feedback. Can we leverage this failure understanding to enhance robotics systems that use foundation models? Introducing AHA—a vision-language model for detecting and reasoning over failures in robotic manipulation. Project page: 🧵Thread👇 Aha!

Humans learn and improve from failures. Similarly, foundation models adapt based on human feedback. Can we leverage this failure understanding to enhance robotics systems that use foundation models? Introducing AHA—a vision-language model for detecting and reasoning over failures in robotic manipulation. Project page: 🧵Thread👇 Aha!

Jiafei Duan

48,793 views • 1 year ago

Demo 2: human + agent design loop inside Codex. Not every workflow needs the agent to drive everything. You design in Open Design. Codex reviews the output. It can inspect files, open pages, test interactions, critique the design, and turn feedback into the next prompt. Design becomes a human + agent feedback loop.

Demo 2: human + agent design loop inside Codex. Not every workflow needs the agent to drive everything. You design in Open Design. Codex reviews the output. It can inspect files, open pages, test interactions, critique the design, and turn feedback into the next prompt. Design becomes a human + agent feedback loop.

Open Design

20,103 views • 3 months ago

Today we’re announcing two new updates in our computer vision work — a new, expanded license for our DINOv2 model and the release of FACET, a comprehensive new benchmark dataset to help evaluate and improve fairness in vision models. More details ➡️ 🧵

Today we’re announcing two new updates in our computer vision work — a new, expanded license for our DINOv2 model and the release of FACET, a comprehensive new benchmark dataset to help evaluate and improve fairness in vision models. More details ➡️ 🧵

AI at Meta

454,030 views • 3 years ago

Introducing Prompt-to-leaderboard (P2L): a real-time LLM leaderboard tailored exactly to your use case! P2L trains an LLM to generate "prompt-specific" leaderboards, so you can input a prompt and get a leaderboard specifically for that prompt. The model is trained on the 2M human preference votes from Chatbot Arena. P2L Highlights: 🔹Instant leaderboard for any prompt 🗿 🔹Optimal model routing (hit #1 on Chatbot Arena in Jan 2025 with 1395 score 🧏) 🔹Fine-grained model strength & weakness analysis 🤓 Check out our demo and thread below for more details!

Introducing Prompt-to-leaderboard (P2L): a real-time LLM leaderboard tailored exactly to your use case! P2L trains an LLM to generate "prompt-specific" leaderboards, so you can input a prompt and get a leaderboard specifically for that prompt. The model is trained on the 2M human preference votes from Chatbot Arena. P2L Highlights: 🔹Instant leaderboard for any prompt 🗿 🔹Optimal model routing (hit #1 on Chatbot Arena in Jan 2025 with 1395 score 🧏) 🔹Fine-grained model strength & weakness analysis 🤓 Check out our demo and thread below for more details!

Arena.ai

122,525 views • 1 year ago

Everyday for the past 2 weeks, we've been sharing something new from GEN-1, our latest milestone in scaling robot learning. This has never been done before. Going from ideas to skills in days (or faster) is what physical AI models should deliver. More coming. Stay tuned. Read more about it in our blog post in the comments below ↓

Everyday for the past 2 weeks, we've been sharing something new from GEN-1, our latest milestone in scaling robot learning. This has never been done before. Going from ideas to skills in days (or faster) is what physical AI models should deliver. More coming. Stay tuned. Read more about it in our blog post in the comments below ↓

Generalist

39,593 views • 4 months ago

Introducing Aiur Labs The new parent company to JRNY CLUB 🚨 To celebrate I'm doing a giveaway with 3 winners of $300 of Ethereum To enter just repost this, check out the video, follow Aiur Labs and comment your feedback below Winners in 48 hours :)

Introducing Aiur Labs The new parent company to JRNY CLUB 🚨 To celebrate I'm doing a giveaway with 3 winners of $300 of Ethereum To enter just repost this, check out the video, follow Aiur Labs and comment your feedback below Winners in 48 hours :)

JRNY Crypto

144,493 views • 2 years ago

An interactive world model developed by NVIDIA in collaboration with academic partners. - DreamDojo turns egocentric human video data into physical intelligence. - Human data is more scalable than robotics data but lacks action labels. - To solve this, a dedicated action model extracts latent actions by identifying physics and motion deltas between frames. Training - A massive 44k hours of video data are used for pre-training. - Post-training on small-scale robot datasets maps human physics to specific robot embodiments. - An additional distillation stage converts the model into an autoregressive, few-step diffusion model, enabling real-time, action-controllable simulation. Primary Use Cases - Live Teleoperation: Controlling a robot inside a world simulation in real-time. - Model-based Planning: Previewing and curating the best actions for improved success. - Policy Evaluation: Testing robot policies in realistic, out-of-distribution scenarios. Everything that's open-sourced: weights, code, post-training dataset, eval set, and details to reproduce.

An interactive world model developed by NVIDIA in collaboration with academic partners. - DreamDojo turns egocentric human video data into physical intelligence. - Human data is more scalable than robotics data but lacks action labels. - To solve this, a dedicated action model extracts latent actions by identifying physics and motion deltas between frames. Training - A massive 44k hours of video data are used for pre-training. - Post-training on small-scale robot datasets maps human physics to specific robot embodiments. - An additional distillation stage converts the model into an autoregressive, few-step diffusion model, enabling real-time, action-controllable simulation. Primary Use Cases - Live Teleoperation: Controlling a robot inside a world simulation in real-time. - Model-based Planning: Previewing and curating the best actions for improved success. - Policy Evaluation: Testing robot policies in realistic, out-of-distribution scenarios. Everything that's open-sourced: weights, code, post-training dataset, eval set, and details to reproduce.

The Humanoid Hub

11,575 views • 6 months ago

How do we make robot policy evaluation reproducible across labs? Announcing OpenArm 02: a fully open-source dual arm platform, with a unified setup for reproducible evaluation: controlled tasks, backgrounds, lighting, and camera angles. ✅ Fully open-source, so anyone can replicate the setup anywhere ✅ AutoEval: 24/7 real-world evaluation with minimal human intervention (building on work by Paul Zhou) ✅ Fair side-by-side policy comparison OpenArm is the open foundation to turn isolated results into shared progress. Releasing soon. Stay tuned.

How do we make robot policy evaluation reproducible across labs? Announcing OpenArm 02: a fully open-source dual arm platform, with a unified setup for reproducible evaluation: controlled tasks, backgrounds, lighting, and camera angles. ✅ Fully open-source, so anyone can replicate the setup anywhere ✅ AutoEval: 24/7 real-world evaluation with minimal human intervention (building on work by Paul Zhou) ✅ Fair side-by-side policy comparison OpenArm is the open foundation to turn isolated results into shared progress. Releasing soon. Stay tuned.

Enactic

64,759 views • 5 months ago

Did you know there is 1 setting that makes a world of difference for a lightning fast trading DOM? If you compare the 2 DOMs in the video below, you should be able to clearly see how one appears much faster than the other. I'll share more details in the post below 🧵

Did you know there is 1 setting that makes a world of difference for a lightning fast trading DOM? If you compare the 2 DOMs in the video below, you should be able to clearly see how one appears much faster than the other. I'll share more details in the post below 🧵

Ziggy

12,276 views • 1 year ago

NEWS: Waymo has released a new blog post detailing their AI strategy and how it’s allowing them to bring service to more riders faster. "Achieving demonstrably safe AI — where safety is proven, not just promised — requires a holistic approach. Beyond a smart and capable Driver, you also need a closed-loop, realistic Simulator to train and rigorously test the Driver in a myriad of challenging situations, and a sharp Critic to evaluate the Driver's performance and identify areas for improvement." Waymo says that autonomous driving isn’t just a matter of building a “smart driver,” but rather creating a full AI ecosystem centered on safety from the ground up. At the core is the Waymo Foundation Model, a unified world-model that powers all major components of Waymo’s autonomous stack (Driver, Simulator, Critic). "By using a “Think Fast/Think Slow” architecture (combining rapid sensor-fusion with deep semantic reasoning), this system enables the car to detect complex and rare road scenarios (e.g. a burning vehicle ahead), reason about them, and choose safe behavior. Waymo trains large “Teacher” AI-models for driving, simulation, and evaluation, then distills them into smaller, efficient “Student” models suitable for real-world deployment, while keeping safety validation tightly integrated. The result is a continuous “flywheel” of learning: driving data (real and simulated) generate feedback, which leads to refinements, more simulation, more data, and only when safety checks pass is new code deployed. Having already exceeded 100 million fully autonomous miles, Waymo reports a more than ten-fold reduction in severe-injury crashes compared to human drivers." Full blog post:

NEWS: Waymo has released a new blog post detailing their AI strategy and how it’s allowing them to bring service to more riders faster. "Achieving demonstrably safe AI — where safety is proven, not just promised — requires a holistic approach. Beyond a smart and capable Driver, you also need a closed-loop, realistic Simulator to train and rigorously test the Driver in a myriad of challenging situations, and a sharp Critic to evaluate the Driver's performance and identify areas for improvement." Waymo says that autonomous driving isn’t just a matter of building a “smart driver,” but rather creating a full AI ecosystem centered on safety from the ground up. At the core is the Waymo Foundation Model, a unified world-model that powers all major components of Waymo’s autonomous stack (Driver, Simulator, Critic). "By using a “Think Fast/Think Slow” architecture (combining rapid sensor-fusion with deep semantic reasoning), this system enables the car to detect complex and rare road scenarios (e.g. a burning vehicle ahead), reason about them, and choose safe behavior. Waymo trains large “Teacher” AI-models for driving, simulation, and evaluation, then distills them into smaller, efficient “Student” models suitable for real-world deployment, while keeping safety validation tightly integrated. The result is a continuous “flywheel” of learning: driving data (real and simulated) generate feedback, which leads to refinements, more simulation, more data, and only when safety checks pass is new code deployed. Having already exceeded 100 million fully autonomous miles, Waymo reports a more than ten-fold reduction in severe-injury crashes compared to human drivers." Full blog post:

Sawyer Merritt

83,635 views • 8 months ago

Testing robot policies on hardware is slow, expensive and hard to scale. World models offer a promising path to accelerating robot policy development. We're sharing new research from the Runway Robotics team, in which we simulated 8 robot policies inside our General World Model and found 0.95 correlation with real-world results. Those early results point to world model simulation as a practical substitute for hardware evaluation, comparing favorably to existing real-to-sim approaches. Learn more at the link below.

Testing robot policies on hardware is slow, expensive and hard to scale. World models offer a promising path to accelerating robot policy development. We're sharing new research from the Runway Robotics team, in which we simulated 8 robot policies inside our General World Model and found 0.95 correlation with real-world results. Those early results point to world model simulation as a practical substitute for hardware evaluation, comparing favorably to existing real-to-sim approaches. Learn more at the link below.

Runway

13,464 views • 6 months ago

In case you missed it, we recently launched "Post-training of LLMs," a short course where you'll: ✅ Understand when and why to use post-training methods like Supervised Fine-Tuning (SFT), Direct Preference Optimization (DPO), and Online Reinforcement Learning. ✅ Learn the concepts underlying the three post-training methods of SFT, DPO, and Online RL, their common use-cases, and how to curate high-quality data to effectively train a model using each method. ✅ Download a pre-trained model and implement post-training pipelines to turn a base model into an instruct model, change the identity of a chat assistant, and improve a model’s math capabilities. Learn more and enroll for free:

In case you missed it, we recently launched "Post-training of LLMs," a short course where you'll: ✅ Understand when and why to use post-training methods like Supervised Fine-Tuning (SFT), Direct Preference Optimization (DPO), and Online Reinforcement Learning. ✅ Learn the concepts underlying the three post-training methods of SFT, DPO, and Online RL, their common use-cases, and how to curate high-quality data to effectively train a model using each method. ✅ Download a pre-trained model and implement post-training pipelines to turn a base model into an instruct model, change the identity of a chat assistant, and improve a model’s math capabilities. Learn more and enroll for free:

DeepLearning.AI

16,779 views • 1 year ago

Today, we’re thrilled to unveil the Arcee.ai Foundation Models, a new family of GenAI models designed from the ground up for enterprise reality. The first release—AFM-4.5B—is a 4.5-billion-parameter frontier model that delivers excellent accuracy, strict compliance, and very high cost efficiency. In short: enterprise-grade intelligence that can run anywhere—on a smartphone, at the edge, or in the cloud. For a quick taste, you can test AFM-4.5B in our playground and on For a deeper dive into the model’s training pipeline and benchmarks, details are available in our technical blog post. ➡️ Arcee AI playground: ➡️ Together AI playground: ➡️ Launch blog post: ➡️ Technical blog post: PS: As we’ll now focus on our AFM foundation models, and because we love open-source, we're opening up access to our previously closed-source language models. Details in the tech blog post!

Today, we’re thrilled to unveil the Arcee.ai Foundation Models, a new family of GenAI models designed from the ground up for enterprise reality. The first release—AFM-4.5B—is a 4.5-billion-parameter frontier model that delivers excellent accuracy, strict compliance, and very high cost efficiency. In short: enterprise-grade intelligence that can run anywhere—on a smartphone, at the edge, or in the cloud. For a quick taste, you can test AFM-4.5B in our playground and on For a deeper dive into the model’s training pipeline and benchmarks, details are available in our technical blog post. ➡️ Arcee AI playground: ➡️ Together AI playground: ➡️ Launch blog post: ➡️ Technical blog post: PS: As we’ll now focus on our AFM foundation models, and because we love open-source, we're opening up access to our previously closed-source language models. Details in the tech blog post!

Arcee.ai

21,934 views • 1 year ago

Hardware Supply Chain Nicolas Dessaigne In Shenzhen, a team can go from design to a new physical part in a day. In the US, that same loop often takes weeks, and that gap compounds. The overall stack for rapid hardware iteration still doesn't exist in America, and we want to fund the startups building it.

Hardware Supply Chain Nicolas Dessaigne In Shenzhen, a team can go from design to a new physical part in a day. In the US, that same loop often takes weeks, and that gap compounds. The overall stack for rapid hardware iteration still doesn't exist in America, and we want to fund the startups building it.

Y Combinator

500,044 views • 4 months ago

We're helping AI to see the 3D world in motion as humans do. 🌐 Enter D4RT: a unified model that turns video into 4D representations faster than previous methods - enabling it to understand space and time. This is how it works 🧵

We're helping AI to see the 3D world in motion as humans do. 🌐 Enter D4RT: a unified model that turns video into 4D representations faster than previous methods - enabling it to understand space and time. This is how it works 🧵

Google DeepMind

369,517 views • 7 months ago

🎥 Today we’re premiering Meta Movie Gen: the most advanced media foundation models to-date. Developed by AI research teams at Meta, Movie Gen delivers state-of-the-art results across a range of capabilities. We’re excited for the potential of this line of research to usher in entirely new possibilities for casual creators and creative professionals alike. More details and examples of what Movie Gen can do ➡️ 🛠️ Movie Gen models and capabilities Movie Gen Video: 30B parameter transformer model that can generate high-quality and high-definition images and videos from a single text prompt. Movie Gen Audio: A 13B parameter transformer model that can take a video input along with optional text prompts for controllability to generate high-fidelity audio synced to the video. It can generate ambient sound, instrumental background music and foley sound — delivering state-of-the-art results in audio quality, video-to-audio alignment and text-to-audio alignment. Precise video editing: Using a generated or existing video and accompanying text instructions as an input it can perform localized edits such as adding, removing or replacing elements — or global changes like background or style changes. Personalized videos: Using an image of a person and a text prompt, the model can generate a video with state-of-the-art results on character preservation and natural movement in video. We’re continuing to work closely with creative professionals from across the field to integrate their feedback as we work towards a potential release. We look forward to sharing more on this work and the creative possibilities it will enable in the future.

🎥 Today we’re premiering Meta Movie Gen: the most advanced media foundation models to-date. Developed by AI research teams at Meta, Movie Gen delivers state-of-the-art results across a range of capabilities. We’re excited for the potential of this line of research to usher in entirely new possibilities for casual creators and creative professionals alike. More details and examples of what Movie Gen can do ➡️ 🛠️ Movie Gen models and capabilities Movie Gen Video: 30B parameter transformer model that can generate high-quality and high-definition images and videos from a single text prompt. Movie Gen Audio: A 13B parameter transformer model that can take a video input along with optional text prompts for controllability to generate high-fidelity audio synced to the video. It can generate ambient sound, instrumental background music and foley sound — delivering state-of-the-art results in audio quality, video-to-audio alignment and text-to-audio alignment. Precise video editing: Using a generated or existing video and accompanying text instructions as an input it can perform localized edits such as adding, removing or replacing elements — or global changes like background or style changes. Personalized videos: Using an image of a person and a text prompt, the model can generate a video with state-of-the-art results on character preservation and natural movement in video. We’re continuing to work closely with creative professionals from across the field to integrate their feedback as we work towards a potential release. We look forward to sharing more on this work and the creative possibilities it will enable in the future.

AI at Meta

2,265,771 views • 1 year ago