Loading video...

Video Failed to Load

There was a problem loading this video. This could be due to a temporary network issue or the video might be unavailable.

In flow matching, a coupling determines how noise and data samples are paired during training. The choice of coupling is important because it influences the geometry of trajectories at inference time. The simplest choice is the independent coupling, where noise and data points are paired arbitrarily. This can lead... show more

Alec Helbling

10,778 subscribers

65,484 views • 4 months ago •via X (Twitter)

Education Science & Technology

Anya Rossi• Live Now

Private livecam show

0 Comments

No comments available

Comments from the original post will appear here

Related Videos

System identification (sysid) is the process of finding the physical parameters that make a simulation match reality. If you're training an RL locomotion policy in simulation, the accuracy of your motor model directly affects how well the policy transfers to the real robot. A recent git commit by Kevin Zakka added a sysid toolbox to MuJoCo which automates this process: you provide recorded motor data and a MuJoCo model, and it optimizes the model parameters to minimize the difference between simulated and real trajectories. For my RobStride Dynamics RS02 QDD motors (17 Nm peak, 7.75:1 gear), I built a Rust tool that sends multi-sine torque excitation at 1 kHz and records position/velocity feedback. I then feed this data into MuJoCo's sysid optimizer.

System identification (sysid) is the process of finding the physical parameters that make a simulation match reality. If you're training an RL locomotion policy in simulation, the accuracy of your motor model directly affects how well the policy transfers to the real robot. A recent git commit by Kevin Zakka added a sysid toolbox to MuJoCo which automates this process: you provide recorded motor data and a MuJoCo model, and it optimizes the model parameters to minimize the difference between simulated and real trajectories. For my RobStride Dynamics RS02 QDD motors (17 Nm peak, 7.75:1 gear), I built a Rust tool that sends multi-sine torque excitation at 1 kHz and records position/velocity feedback. I then feed this data into MuJoCo's sysid optimizer.

David Bar

48,347 views • 5 months ago

Holy sh!t ! OpenAI will have their custom inference chips ready in just a few months and deployed at scale by the end of the year! 🤯 Training chip = The heavy lifters that require massive amounts of data and power to build and teach the AI models from scratch. Inference chip = The specialized, highly efficient chips that actually run the AI and generate the answers in real-time when you use it. This is going to help OpenAI drastically cut down their massive compute costs, speed up model reasoning times, and finally break free from relying entirely on Nvidia to scale their operations.

Holy sh!t ! OpenAI will have their custom inference chips ready in just a few months and deployed at scale by the end of the year! 🤯 Training chip = The heavy lifters that require massive amounts of data and power to build and teach the AI models from scratch. Inference chip = The specialized, highly efficient chips that actually run the AI and generate the answers in real-time when you use it. This is going to help OpenAI drastically cut down their massive compute costs, speed up model reasoning times, and finally break free from relying entirely on Nvidia to scale their operations.

Chris

60,278 views • 6 months ago

$Transformer and Mixture of Experts, explained visually! Mixture of Experts (MoE) is a popular architecture that uses different experts to improve Transformer models. Transformer and MoE differ in the decoder block: - Transformer uses a feed-forward network. - MoE uses experts, which are feed-forward networks but smaller compared to those Transformer. During inference, a subset of experts are selected. This makes inference faster in MoE. Also, since the network has multiple decoder layers: - The text passes through different experts across layers. - The chosen experts also differ between tokens. But how does the model decide which experts should be ideal? The router does that. It is a multi-class classifier that produces softmax scores over experts to select the top K experts. The router is trained with the network, and it learns to select the best experts. But it isn't straightforward. There are challenges! Challenge 1) Notice this pattern at the start of training: - Say, the model selects "Expert 2" - This expert gets a bit better - It may get selected again since it's the "best" - It learns more - It gets selected again in the next iteration - It learns more, and so on! This means many experts can go under-trained due to the overselection of a few experts! We solve this in two steps: - Add noise to the feed-forward output of the router so that other experts can get higher logits. - Set all but the top K logits to -infinity. After softmax, these scores become zero. This way, other experts also get the opportunity to train. Challenge 2) Some experts may get exposed to more tokens than others, leading to under-trained experts. We prevent this by limiting the number of tokens an expert can process. If an expert reaches the limit, the token is passed to the next best expert. Overall, MoEs have more parameters to load. But a fraction of them are activated during inference. This leads to faster inference. Mixtral 8x7B and Llama 4 are two popular MoE-based LLMs. Have you used MoEs in production yet? To dive deeper into how MoE inference works in production, we wrote a full article covering token dispatch, grouped expert computation, model-weight memory, multi-GPU communication, expert placement, load imbalance, and performance diagnosis. Read it below.$

Transformer and Mixture of Experts, explained visually! Mixture of Experts (MoE) is a popular architecture that uses different experts to improve Transformer models. Transformer and MoE differ in the decoder block: - Transformer uses a feed-forward network. - MoE uses experts, which are feed-forward networks but smaller compared to those Transformer. During inference, a subset of experts are selected. This makes inference faster in MoE. Also, since the network has multiple decoder layers: - The text passes through different experts across layers. - The chosen experts also differ between tokens. But how does the model decide which experts should be ideal? The router does that. It is a multi-class classifier that produces softmax scores over experts to select the top K experts. The router is trained with the network, and it learns to select the best experts. But it isn't straightforward. There are challenges! Challenge 1) Notice this pattern at the start of training: - Say, the model selects "Expert 2" - This expert gets a bit better - It may get selected again since it's the "best" - It learns more - It gets selected again in the next iteration - It learns more, and so on! This means many experts can go under-trained due to the overselection of a few experts! We solve this in two steps: - Add noise to the feed-forward output of the router so that other experts can get higher logits. - Set all but the top K logits to -infinity. After softmax, these scores become zero. This way, other experts also get the opportunity to train. Challenge 2) Some experts may get exposed to more tokens than others, leading to under-trained experts. We prevent this by limiting the number of tokens an expert can process. If an expert reaches the limit, the token is passed to the next best expert. Overall, MoEs have more parameters to load. But a fraction of them are activated during inference. This leads to faster inference. Mixtral 8x7B and Llama 4 are two popular MoE-based LLMs. Have you used MoEs in production yet? To dive deeper into how MoE inference works in production, we wrote a full article covering token dispatch, grouped expert computation, model-weight memory, multi-GPU communication, expert placement, load imbalance, and performance diagnosis. Read it below.

Daily Dose of Data Science

53,989 views • 4 days ago

This animation shows how a uniform grid of points is deformed by a flow matching model. The visualization highlights how the learned flow warps the underlying space—stretching, compressing, and bending it—to transform one probability distribution into another.

This animation shows how a uniform grid of points is deformed by a flow matching model. The visualization highlights how the learned flow warps the underlying space—stretching, compressing, and bending it—to transform one probability distribution into another.

Alec Helbling

43,008 views • 1 year ago

We’re proud to share that Tamarind Bio has been selected to build, host, and operate the inference infrastructure layer for TuneLab2.0, the next evolution of the platform. Eli Lilly and Company TuneLab is a first-of-its-kind, collaborative AI/ML drug discovery platform, bringing models trained on over $1B worth of Lilly proprietary data to the biotech ecosystem. Tamarind will power TuneLab’s scalable drug discovery workflows and model inference.

We’re proud to share that Tamarind Bio has been selected to build, host, and operate the inference infrastructure layer for TuneLab2.0, the next evolution of the platform. Eli Lilly and Company TuneLab is a first-of-its-kind, collaborative AI/ML drug discovery platform, bringing models trained on over $1B worth of Lilly proprietary data to the biotech ecosystem. Tamarind will power TuneLab’s scalable drug discovery workflows and model inference.

Deniz Kavi

43,702 views • 3 months ago

The next World Cup star is playing right now, and nobody is watching. The 2026 World Cup is on, and millions of kids are dreaming of one day playing for their own flag. But the path there is almost impossible: no consolidated data, too many games, not enough scouts, human bias, and wrong place, wrong time. Any one of them can bury a generational talent before a scout ever sees them. That is what we are building to fix. So we developed a full automated annotation + action spotting model where it actually matters: grassroots football. Shaky cameras, awful angles, terrible lighting. But now nothing on the market beats sn44's v2.0 model on it. Check the last GT line, it’s human annotations. Simply because miners' outputs + post-process = magic. Post-process is a deliberate design choice. We point miners at the hardest 80% of the workflow, the part that is genuinely difficult to fake, and handle the rest downstream. Fewer places to game the system means a network you can trust. Every single challenge on sn44 can be an independent startup now. Let that sink in.

The next World Cup star is playing right now, and nobody is watching. The 2026 World Cup is on, and millions of kids are dreaming of one day playing for their own flag. But the path there is almost impossible: no consolidated data, too many games, not enough scouts, human bias, and wrong place, wrong time. Any one of them can bury a generational talent before a scout ever sees them. That is what we are building to fix. So we developed a full automated annotation + action spotting model where it actually matters: grassroots football. Shaky cameras, awful angles, terrible lighting. But now nothing on the market beats sn44's v2.0 model on it. Check the last GT line, it’s human annotations. Simply because miners' outputs + post-process = magic. Post-process is a deliberate design choice. We point miners at the hardest 80% of the workflow, the part that is genuinely difficult to fake, and handle the rest downstream. Fewer places to game the system means a network you can trust. Every single challenge on sn44 can be an independent startup now. Let that sink in.

Score - Subnet 44

430,045 views • 3 months ago

Amazon’s machine learning model collects 300 million data points per season and can now predict which players are likely to blitz before the snap. This is a look into the future of broadcasting.

Amazon’s machine learning model collects 300 million data points per season and can now predict which players are likely to blitz before the snap. This is a look into the future of broadcasting.

Joe Pompliano

688,472 views • 2 years ago

Prediction: By 2030, global data center workloads are projected to split into: • 50% traditional workloads • 37% AI inference • 13% AI training Inference is set to become nearly 3× larger than training. As more AI applications move into production, builders will need infrastructure that can support lower-cost inference, reliable execution, faster routing and private workloads at scale. This is where FAR AI enters the market. Source: JLL Research, 2025

Prediction: By 2030, global data center workloads are projected to split into: • 50% traditional workloads • 37% AI inference • 13% AI training Inference is set to become nearly 3× larger than training. As more AI applications move into production, builders will need infrastructure that can support lower-cost inference, reliable execution, faster routing and private workloads at scale. This is where FAR AI enters the market. Source: JLL Research, 2025

FAR Labs

20,706 views • 2 months ago

What if you kept asking an LLM to "make it better"? In some recent work at FAIR, we investigate how we can efficiently use RL to fine-tune LLMs to iteratively self-improve on their previous solutions at inference-time. Training for iterated self-improvement can be costly. The naive approach to training for K self-improvement steps leads to K times the number of rollout steps per episode. We introduce Exploratory Iteration (ExIt), an RL-based automatic curriculum method that bootstraps diverse training distributions of self-improvement tasks by upcycling the LLM's own responses at previous turns as the starting points for both self-improvement and *self-divergence.* In order to decide what task to train on next, the curriculum prioritizes sampling of partial turn histories that led to higher return variance in its GRPO group (a learnability score that comes for free). This automatic curriculum over the bootstrapped task space teaches the model how to perform iterated self-improvement while only ever training the model on single-step self-improvement tasks. We look at ExIt's impact in both single-turn (contest math problems) and multi-turn (BFCLv3 multi-turn tasks), as well as MLE-bench, where the LLM is run in a search scaffold to produce solutions to real Kaggle competitions. Across these eval settings, we find ExIt produces models with greater capacity for inference-time self-improvement compared to GRPO. Notably, ExIt models can self-improve on test tasks for many more steps than the typical solution depth encountered during training, including a 22% improvement in MLE-bench performance compared to GRPO.

What if you kept asking an LLM to "make it better"? In some recent work at FAIR, we investigate how we can efficiently use RL to fine-tune LLMs to iteratively self-improve on their previous solutions at inference-time. Training for iterated self-improvement can be costly. The naive approach to training for K self-improvement steps leads to K times the number of rollout steps per episode. We introduce Exploratory Iteration (ExIt), an RL-based automatic curriculum method that bootstraps diverse training distributions of self-improvement tasks by upcycling the LLM's own responses at previous turns as the starting points for both self-improvement and self-divergence. In order to decide what task to train on next, the curriculum prioritizes sampling of partial turn histories that led to higher return variance in its GRPO group (a learnability score that comes for free). This automatic curriculum over the bootstrapped task space teaches the model how to perform iterated self-improvement while only ever training the model on single-step self-improvement tasks. We look at ExIt's impact in both single-turn (contest math problems) and multi-turn (BFCLv3 multi-turn tasks), as well as MLE-bench, where the LLM is run in a search scaffold to produce solutions to real Kaggle competitions. Across these eval settings, we find ExIt produces models with greater capacity for inference-time self-improvement compared to GRPO. Notably, ExIt models can self-improve on test tasks for many more steps than the typical solution depth encountered during training, including a 22% improvement in MLE-bench performance compared to GRPO.

Minqi Jiang

41,147 views • 1 year ago

OpenAI's Deep Research is getting a run for its money. Deep Lake was just released, and it's a different take on an AI system that can do deep research on your own data. You can use Deep Lake to build AI search with reasoning on your private and public data. (Look at the attached videos to get an idea of how it works.) If you want to research proprietary and sensitive data, Deep Research won't help you because it's limited to public data. Deep Lake, however, will allow you to use your private data. On top of that, Deep Lake supports multi-modal retrieval from the ground up. It uses vision language models for data ingestion and retrieval so that you can connect any data (PDFs, images, videos, structured data, etc.) You can even use mixed-data queries! Deep Lake can search your data from S3, Dropbox, and GCP. It learns from your queries over time, making the results as relevant to your work as possible!

OpenAI's Deep Research is getting a run for its money. Deep Lake was just released, and it's a different take on an AI system that can do deep research on your own data. You can use Deep Lake to build AI search with reasoning on your private and public data. (Look at the attached videos to get an idea of how it works.) If you want to research proprietary and sensitive data, Deep Research won't help you because it's limited to public data. Deep Lake, however, will allow you to use your private data. On top of that, Deep Lake supports multi-modal retrieval from the ground up. It uses vision language models for data ingestion and retrieval so that you can connect any data (PDFs, images, videos, structured data, etc.) You can even use mixed-data queries! Deep Lake can search your data from S3, Dropbox, and GCP. It learns from your queries over time, making the results as relevant to your work as possible!

Santiago

171,340 views • 1 year ago

BIG NEWS. The Blockworks website has evolved: yesterday we were the home of news, today we are the home of onchain data. Head over to the site to see for yourself, but here's a little snippet of what you can expect: 1. Sector leaderboards (chains, DEXes, borrow lend, DATs, etc...) 2. Comprehensive data dashboards protocols 3. The ability to compare pricing and onchain data easily (coming soon) We're doing this because the industry still has a gigantic data problem. As investors get more sophisticated and fundamentals driven, basic high level facts are no longer sufficient. Investors need to be able to trust the data they are seeing and go much deeper than the surface level info that's available today. Additionally, because many data providers allow companies to essentially self report, you can't trust what you are seeing. This site is our contribution to fixing that problem and to ensuring clear, accurate data for investors. Blockworks is fully dedicated to becoming the most comprehensive data company in crypto in 2026. This is the first of many, many announcements like this this year, stay tuned.

BIG NEWS. The Blockworks website has evolved: yesterday we were the home of news, today we are the home of onchain data. Head over to the site to see for yourself, but here's a little snippet of what you can expect: 1. Sector leaderboards (chains, DEXes, borrow lend, DATs, etc...) 2. Comprehensive data dashboards protocols 3. The ability to compare pricing and onchain data easily (coming soon) We're doing this because the industry still has a gigantic data problem. As investors get more sophisticated and fundamentals driven, basic high level facts are no longer sufficient. Investors need to be able to trust the data they are seeing and go much deeper than the surface level info that's available today. Additionally, because many data providers allow companies to essentially self report, you can't trust what you are seeing. This site is our contribution to fixing that problem and to ensuring clear, accurate data for investors. Blockworks is fully dedicated to becoming the most comprehensive data company in crypto in 2026. This is the first of many, many announcements like this this year, stay tuned.

Mippo 🟪

91,927 views • 8 months ago

AI has had exactly two scaling axes that worked so far, and the second one is starting to look finite too the first one was pretraining: with scaling parameters and data, we got world knowledge (i.e. ChatGPT had read enough to know things), but it started saturating a while ago the second one was RL, and people had been doing RL the whole time before that: RLHF is RL but it never scaled far because it was trying to control the exact output, which tokens come out, how the text reads, but you can only push that so far before you’re just polishing RLVR dropped that constraint: giving the model a task, then checking whether the final answer is right, and ignoring everything in between -- so the model does whatever it wants in the middle and only the endpoint gets graded, and that’s much closer to actual RL and it’s what bought us planning and reasoning (arguably, tool use sits around 2.5 on this list -- while useful, it's not a different kind of thing) so one axis gave knowledge, the other gave reasoning, and both of them are one model working alone the next axis is how many models you can get working on the same problem, which is a different kind of axis than the previous two we know that multi-agent RL has always been the harder problem: I spent years in that literature and the gap between single-agent and multi-agent is definitely not incremental -- it’s a whole different class of difficulty! which is also why the derivatives are steep at the start, nobody has picked the easy wins yet... and the thing that gates this multi-agent coordination is communication: models can only coordinate as well as they can exchange information, and right now they do that by writing sentences to each other imagine what could we possibly achieve if we properly open that third axis development by letting models to exchange information in their native "language" without loosing any computational data that they produce during inference

AI has had exactly two scaling axes that worked so far, and the second one is starting to look finite too the first one was pretraining: with scaling parameters and data, we got world knowledge (i.e. ChatGPT had read enough to know things), but it started saturating a while ago the second one was RL, and people had been doing RL the whole time before that: RLHF is RL but it never scaled far because it was trying to control the exact output, which tokens come out, how the text reads, but you can only push that so far before you’re just polishing RLVR dropped that constraint: giving the model a task, then checking whether the final answer is right, and ignoring everything in between -- so the model does whatever it wants in the middle and only the endpoint gets graded, and that’s much closer to actual RL and it’s what bought us planning and reasoning (arguably, tool use sits around 2.5 on this list -- while useful, it's not a different kind of thing) so one axis gave knowledge, the other gave reasoning, and both of them are one model working alone the next axis is how many models you can get working on the same problem, which is a different kind of axis than the previous two we know that multi-agent RL has always been the harder problem: I spent years in that literature and the gap between single-agent and multi-agent is definitely not incremental -- it’s a whole different class of difficulty! which is also why the derivatives are steep at the start, nobody has picked the easy wins yet... and the thing that gates this multi-agent coordination is communication: models can only coordinate as well as they can exchange information, and right now they do that by writing sentences to each other imagine what could we possibly achieve if we properly open that third axis development by letting models to exchange information in their native "language" without loosing any computational data that they produce during inference

Sasha Malysheva

14,445 views • 1 month ago

Oswaldo Cabrera and Luis Arraez need to be rotating at the leadoff spot for the Yankees, depending on who is hitting first. Then, whoever isn't at the top of the lineup can bat second. It's time to put the analytics down like Old Yeller; otherwise, how are the Yankees ever going to beat the Dodgers in the World Series? This lineup is missing PURE contact hitters, and that's where Ozzy and Arraez come in. In a lot of ways, they're both more important than Aaron Judge. If you do this, Judge can take his time because the lineup would be LOADED.

Oswaldo Cabrera and Luis Arraez need to be rotating at the leadoff spot for the Yankees, depending on who is hitting first. Then, whoever isn't at the top of the lineup can bat second. It's time to put the analytics down like Old Yeller; otherwise, how are the Yankees ever going to beat the Dodgers in the World Series? This lineup is missing PURE contact hitters, and that's where Ozzy and Arraez come in. In a lot of ways, they're both more important than Aaron Judge. If you do this, Judge can take his time because the lineup would be LOADED.

Luis Feld

46,733 views • 1 month ago

If you’ve been ignoring Axis Robotics because it looks like another random points farm, read this. In the last 48 hours, Unitree said robotics is approaching its "ChatGPT moment", while the chairman of ACE Robotics believes it could happen by the end of 2027. But for robots to reach that level, they need a crazy amount of training data. That’s exactly what Axis is building. You control simulated robots directly from your browser, complete simple tasks and earn points. Those movements also help create data that can be used to train real robots. And this isn’t some tiny experiment anymore: - $12M raised - 123K+ contributors - 3M+ robot trajectories - Community data already used to train a real robot So yeah, we’re basically farming a potential airdrop while teaching our future robot servants how to work 😂 If you haven’t started yet, it’s completely free. You only need a tiny amount of gas on Base to sign your completed tasks. ✅ Start farming Axis points: Important: Sign every completed task from the History page, otherwise you won’t receive the points.

If you’ve been ignoring Axis Robotics because it looks like another random points farm, read this. In the last 48 hours, Unitree said robotics is approaching its "ChatGPT moment", while the chairman of ACE Robotics believes it could happen by the end of 2027. But for robots to reach that level, they need a crazy amount of training data. That’s exactly what Axis is building. You control simulated robots directly from your browser, complete simple tasks and earn points. Those movements also help create data that can be used to train real robots. And this isn’t some tiny experiment anymore: - $12M raised - 123K+ contributors - 3M+ robot trajectories - Community data already used to train a real robot So yeah, we’re basically farming a potential airdrop while teaching our future robot servants how to work 😂 If you haven’t started yet, it’s completely free. You only need a tiny amount of gas on Base to sign your completed tasks. ✅ Start farming Axis points: Important: Sign every completed task from the History page, otherwise you won’t receive the points.

Pranjal Bora 🧭

29,321 views • 1 month ago

Fancy winning a pair of #EFL tickets for a home game of your choice? 😍 Reply with the answer to the following question and use #Super6Christmas for a chance to win 🎟️ How many points are Leicester City on in the league this season? 🦊 Winner announced at 4pm, T&Cs apply ⌚️

Fancy winning a pair of #EFL tickets for a home game of your choice? 😍 Reply with the answer to the following question and use #Super6Christmas for a chance to win 🎟️ How many points are Leicester City on in the league this season? 🦊 Winner announced at 4pm, T&Cs apply ⌚️

Super 6

53,500 views • 2 years ago

Super excited about Hydra-0 from Hongyu Li and team! The key idea is to use flow as a shared visual interface across embodiments/objects for controllable video generation, allowing a single generalist world model to learn from human, handheld-gripper, and robot interaction data. My favorite result is the video below: start from a real video of a human doing the task (left), extract the desired object flow, and condition the model on that flow (right). The model then hallucinates a plausible robot motion that could produce the same object motion. Very cool glimpse of how a generalist world model can bridge human demonstrations and robot control.

Super excited about Hydra-0 from Hongyu Li and team! The key idea is to use flow as a shared visual interface across embodiments/objects for controllable video generation, allowing a single generalist world model to learn from human, handheld-gripper, and robot interaction data. My favorite result is the video below: start from a real video of a human doing the task (left), extract the desired object flow, and condition the model on that flow (right). The model then hallucinates a plausible robot motion that could produce the same object motion. Very cool glimpse of how a generalist world model can bridge human demonstrations and robot control.

Yunzhu Li

10,850 views • 1 month ago

AI Is Moving Beyond “Generating Videos” — Toward “Generating Worlds” Over the past two years, AI video models have advanced at an astonishing pace. From Runway and Pika to Sora and Veo, AI-generated videos have become increasingly realistic and more consistent with the physical laws of the real world. Many people believe the next objective is simply to generate videos that are longer, sharper, and more lifelike. But if we take a step back, we can see that the real transformation is not happening in video itself. It is happening in world models. What Is a World Model? In 1943, psychologist Kenneth Craik proposed an idea that would influence artificial intelligence research for decades. He argued that the human brain does not merely react to the outside world. Instead, it maintains an internal model of how the world works. Because we have this internal model, we can predict the outcome of an action before we actually take it. Before crossing a road, we estimate whether a car will pass by. Before catching a ball, we predict its trajectory. These abilities come from continuously simulating the world in our minds, rather than relying entirely on trial and error. This idea later became known by a more formal term: World Model. A world model does not describe a single image or a fixed video clip. It is an internal representation capable of continuously simulating the rules and dynamics of the real world. Why Is AI Research Turning Toward World Models? Because predicting “what comes next” is becoming increasingly central to how AI systems work. Language models predict the next token. Image models predict the next step in the denoising process. Video models predict the next frame. A world model, however, attempts to predict something broader: What should the world look like in the next moment? In 2018, David Ha and Jürgen Schmidhuber proposed in their paper World Models that an intelligent agent could first learn a model of the world, and then use that internal model to plan its actions. The Dreamer series later demonstrated that many complex tasks could be learned by training agents inside an “imagined world.” At the same time, the development of video models such as Sora and Veo led researchers to another realization: A model capable of continuously generating video has already learned, at least implicitly, many of the rules governing the real world. As a result, these two research directions have gradually begun to converge. But Video Is Not Yet a World This is where the distinction is often misunderstood. For a world model to support meaningful real-time interaction, it must solve several critical problems. Most video models today are essentially answering one question: What should the next frame look like? A true world model needs to answer much more: What happens if I take one step forward? If I walk behind a building and then return, will the building still be there? If I suddenly change the camera angle, will the entire space remain consistent? If I enter a command such as: “Summon a dragon.” Will the world respond immediately? In other words, a world model must do more than generate content. It must understand space. It must understand time. It must understand causality. And it must understand interaction. Moving from watching to participating is where the real difficulty of world models begins. World Models Are Entering the Interactive Era One of the latest attempts in this direction is Alaya World, recently open-sourced by Alaya World, or Alaya Lab. Instead of generating a fixed video clip, it generates a world that users can explore in real time. Users can begin with text, an image, or a video, enter the generated scene, move freely through it, and introduce new prompts at any moment during generation. The world responds immediately. According to the publicly released information, Alaya World provides: Real-time streaming generation at 720p and 24 FPS Stable continuous exploration for more than one minute The ability to switch prompts and trigger skills or events during generation Model weights and inference code released under the Apache 2.0 License Training code and datasets planned for future release What makes these capabilities important is not simply the technical specifications. It is that the generated “world” can now support continuous interaction. The official demo shows that users can genuinely control, transform, and explore the generated environment. AI Is Evolving From a Tool Into an Environment Over the past few years, most discussions around AI have focused on content generation. Generating text. Generating images. Generating videos. But world models raise a fundamentally different question: Can AI generate an environment that people can inhabit, explore, and continuously evolve? If the answer is yes, the impact will extend far beyond video generation. Game development, robotics training, embodied intelligence, digital twins, virtual production, and many other fields could be transformed by the development of world models. World models are still at a very early stage. Yet from Craik’s proposal of an internal mental model more than eighty years ago to the emergence of today’s interactive world-generation systems, a clear evolutionary path is beginning to take shape. Perhaps what AI is ultimately learning has never been limited to images, videos, or language. Perhaps it is learning the world itself. References GitHub: Technical Report:

AI Is Moving Beyond “Generating Videos” — Toward “Generating Worlds” Over the past two years, AI video models have advanced at an astonishing pace. From Runway and Pika to Sora and Veo, AI-generated videos have become increasingly realistic and more consistent with the physical laws of the real world. Many people believe the next objective is simply to generate videos that are longer, sharper, and more lifelike. But if we take a step back, we can see that the real transformation is not happening in video itself. It is happening in world models. What Is a World Model? In 1943, psychologist Kenneth Craik proposed an idea that would influence artificial intelligence research for decades. He argued that the human brain does not merely react to the outside world. Instead, it maintains an internal model of how the world works. Because we have this internal model, we can predict the outcome of an action before we actually take it. Before crossing a road, we estimate whether a car will pass by. Before catching a ball, we predict its trajectory. These abilities come from continuously simulating the world in our minds, rather than relying entirely on trial and error. This idea later became known by a more formal term: World Model. A world model does not describe a single image or a fixed video clip. It is an internal representation capable of continuously simulating the rules and dynamics of the real world. Why Is AI Research Turning Toward World Models? Because predicting “what comes next” is becoming increasingly central to how AI systems work. Language models predict the next token. Image models predict the next step in the denoising process. Video models predict the next frame. A world model, however, attempts to predict something broader: What should the world look like in the next moment? In 2018, David Ha and Jürgen Schmidhuber proposed in their paper World Models that an intelligent agent could first learn a model of the world, and then use that internal model to plan its actions. The Dreamer series later demonstrated that many complex tasks could be learned by training agents inside an “imagined world.” At the same time, the development of video models such as Sora and Veo led researchers to another realization: A model capable of continuously generating video has already learned, at least implicitly, many of the rules governing the real world. As a result, these two research directions have gradually begun to converge. But Video Is Not Yet a World This is where the distinction is often misunderstood. For a world model to support meaningful real-time interaction, it must solve several critical problems. Most video models today are essentially answering one question: What should the next frame look like? A true world model needs to answer much more: What happens if I take one step forward? If I walk behind a building and then return, will the building still be there? If I suddenly change the camera angle, will the entire space remain consistent? If I enter a command such as: “Summon a dragon.” Will the world respond immediately? In other words, a world model must do more than generate content. It must understand space. It must understand time. It must understand causality. And it must understand interaction. Moving from watching to participating is where the real difficulty of world models begins. World Models Are Entering the Interactive Era One of the latest attempts in this direction is Alaya World, recently open-sourced by Alaya World, or Alaya Lab. Instead of generating a fixed video clip, it generates a world that users can explore in real time. Users can begin with text, an image, or a video, enter the generated scene, move freely through it, and introduce new prompts at any moment during generation. The world responds immediately. According to the publicly released information, Alaya World provides: Real-time streaming generation at 720p and 24 FPS Stable continuous exploration for more than one minute The ability to switch prompts and trigger skills or events during generation Model weights and inference code released under the Apache 2.0 License Training code and datasets planned for future release What makes these capabilities important is not simply the technical specifications. It is that the generated “world” can now support continuous interaction. The official demo shows that users can genuinely control, transform, and explore the generated environment. AI Is Evolving From a Tool Into an Environment Over the past few years, most discussions around AI have focused on content generation. Generating text. Generating images. Generating videos. But world models raise a fundamentally different question: Can AI generate an environment that people can inhabit, explore, and continuously evolve? If the answer is yes, the impact will extend far beyond video generation. Game development, robotics training, embodied intelligence, digital twins, virtual production, and many other fields could be transformed by the development of world models. World models are still at a very early stage. Yet from Craik’s proposal of an internal mental model more than eighty years ago to the emergence of today’s interactive world-generation systems, a clear evolutionary path is beginning to take shape. Perhaps what AI is ultimately learning has never been limited to images, videos, or language. Perhaps it is learning the world itself. References GitHub: Technical Report:

雪踏乌云

113,347 views • 2 months ago

At Dreamforce, the exploration of the possibilities of AI is incredible. At the same time, it's important that we understand that policies never move at the same speed as technology. So having a grasp over what is happening and how the negative aspects of AI don't proliferate in a big way, to minimize collateral damage is most important. I hope the companies and governments across the world wake up to this and ensure the best of AI possibilities will reach the people, as this is a phenomenal enabler for humanity like never before. -Sg Marc Benioff Dreamforce Salesforce #DF24

At Dreamforce, the exploration of the possibilities of AI is incredible. At the same time, it's important that we understand that policies never move at the same speed as technology. So having a grasp over what is happening and how the negative aspects of AI don't proliferate in a big way, to minimize collateral damage is most important. I hope the companies and governments across the world wake up to this and ensure the best of AI possibilities will reach the people, as this is a phenomenal enabler for humanity like never before. -Sg Marc Benioff Dreamforce Salesforce #DF24

Sadhguru

68,330 views • 2 years ago

Depth Any Video with Scalable Synthetic Data AI physicists and chemists continue to make strides in depth estimation from video. Check out this new paper featuring some impressive examples. See the thread for more details (unfortunately no code yet). Abstract: Video depth estimation has long been hindered by the scarcity of consistent and scalable ground truth data, leading to inconsistent and unreliable results. In this paper, we introduce Depth Any Video, a model that tackles the challenge through two key innovations. First, we develop a scalable synthetic data pipeline, capturing real-time video depth data from diverse game environments, yielding 40,000 video clips of 5-second duration, each with precise depth annotations. Second, we leverage the powerful priors of generative video diffusion models to handle real-world videos effectively, integrating advanced techniques such as rotary position encoding and flow matching to further enhance flexibility and efficiency. Unlike previous models, which are limited to fixed-length video sequences, our approach introduces a novel mixed-duration training strategy that handles videos of varying lengths and performs robustly across different frame rates 0 - even on single frames. At inference, we propose a depth interpolation method that enables our model to infer high-resolution video depth across sequences of up to 150 frames. Our model outperforms all previous generative depth models in terms of spatial accuracy and temporal consistency.

Depth Any Video with Scalable Synthetic Data AI physicists and chemists continue to make strides in depth estimation from video. Check out this new paper featuring some impressive examples. See the thread for more details (unfortunately no code yet). Abstract: Video depth estimation has long been hindered by the scarcity of consistent and scalable ground truth data, leading to inconsistent and unreliable results. In this paper, we introduce Depth Any Video, a model that tackles the challenge through two key innovations. First, we develop a scalable synthetic data pipeline, capturing real-time video depth data from diverse game environments, yielding 40,000 video clips of 5-second duration, each with precise depth annotations. Second, we leverage the powerful priors of generative video diffusion models to handle real-world videos effectively, integrating advanced techniques such as rotary position encoding and flow matching to further enhance flexibility and efficiency. Unlike previous models, which are limited to fixed-length video sequences, our approach introduces a novel mixed-duration training strategy that handles videos of varying lengths and performs robustly across different frame rates 0 - even on single frames. At inference, we propose a depth interpolation method that enables our model to infer high-resolution video depth across sequences of up to 150 frames. Our model outperforms all previous generative depth models in terms of spatial accuracy and temporal consistency.

MrNeRF

27,428 views • 1 year ago