Video wird geladen...

Video konnte nicht geladen werden

Beim Laden dieses Videos ist ein Problem aufgetreten. Dies könnte an einem vorübergehenden Netzwerkproblem liegen oder das Video ist möglicherweise nicht verfügbar.

Video world models today have a very limited context length. Mode Seeking meets Mean Seeking (MMM) unlocks long-context, persistent video world models through a unified representation. 1/8 🧵

Gordon Wetzstein

5,368 subscribers

43,742 Aufrufe • vor 5 Monaten •via X (Twitter)

Anya Rossi• Live Now

Private livecam show

0 Kommentare

Keine Kommentare verfügbar

Kommentare vom Original-Post werden hier angezeigt

Ähnliche Videos

Large language models reason through text. Vision‑language‑action models reason through the real world. By fusing perception, context, and action from live video, VLAs deliver the awareness physical AI needs for next‑gen robotics and edge systems.

Large language models reason through text. Vision‑language‑action models reason through the real world. By fusing perception, context, and action from live video, VLAs deliver the awareness physical AI needs for next‑gen robotics and edge systems.

Intel

15,931 Aufrufe • vor 6 Monaten

We open-source Action Images — a new representation that translates 7-DoF robot actions into interpretable images. Video models are emerging as powerful robotic foundation models, but a key challenge remains: how can we seamlessly integrate robot policies into video models? Instead of representing actions as low-dimensional control tokens, Action Images provide a pixel-grounded action representation, reframing policy learning as a visual tracking problem! By unifying observations and actions in the same video space, Action Images enable a unified robotics world model that supports video-action joint generation, action-conditioned video generation, and action labeling! Code: Paper:

We open-source Action Images — a new representation that translates 7-DoF robot actions into interpretable images. Video models are emerging as powerful robotic foundation models, but a key challenge remains: how can we seamlessly integrate robot policies into video models? Instead of representing actions as low-dimensional control tokens, Action Images provide a pixel-grounded action representation, reframing policy learning as a visual tracking problem! By unifying observations and actions in the same video space, Action Images enable a unified robotics world model that supports video-action joint generation, action-conditioned video generation, and action labeling! Code: Paper:

Chuang Gan

19,963 Aufrufe • vor 1 Monat

Today @GoogleMaps is getting its biggest upgrade in over a decade. By combining our Gemini models with a deep understanding of the world, Maps now unlocks entirely new possibilities for how you navigate and explore. Here’s what you need to know 🧵

Today @GoogleMaps is getting its biggest upgrade in over a decade. By combining our Gemini models with a deep understanding of the world, Maps now unlocks entirely new possibilities for how you navigate and explore. Here’s what you need to know 🧵

Google

28,855,848 Aufrufe • vor 5 Monaten

Can video generative models exhibit visuospatial intelligence? 🤔 Introducing Video4Spatial — a video-only framework that tackles spatial tasks. With just video context, our model can: 🔍 Ground objects by planning geometry-consistent paths 📸 Follow camera-pose instructions for scene navigation 🌐 Generalize to long contexts & unseen outdoor scenes A step toward video models as visual-spatial reasoners. Project: arXiv:

Can video generative models exhibit visuospatial intelligence? 🤔 Introducing Video4Spatial — a video-only framework that tackles spatial tasks. With just video context, our model can: 🔍 Ground objects by planning geometry-consistent paths 📸 Follow camera-pose instructions for scene navigation 🌐 Generalize to long contexts & unseen outdoor scenes A step toward video models as visual-spatial reasoners. Project: arXiv:

Xingang Pan

15,946 Aufrufe • vor 8 Monaten

Yann LeCun says that within a year to 18 months, we'll have a general method for training hierarchical world models These models would learn from video and real-world data, then help plan actions in robotics, healthcare, and other areas "then scale them toward a universal world model"

Yann LeCun says that within a year to 18 months, we'll have a general method for training hierarchical world models These models would learn from video and real-world data, then help plan actions in robotics, healthcare, and other areas "then scale them toward a universal world model"

Haider.

201,090 Aufrufe • vor 3 Monaten

Punchline: World models == VQA (about the future)! Planning with world models can be powerful for robotics/control. But most world models are video generators trained to predict everything, including irrelevant pixels and distractions. We ask - what if a world model only predicted the semantic information necessary for decision-making? Introducing Semantic World Models (SWM). Given an observation and an action sequence, SWMs cast modeling as answering textual questions about the future outcome resulting from the actions. Recasting world modeling as a VQA problem lets us directly leverage the pretrained knowledge and machinery of VLMs for generalizable modeling. We had a lot of fun thinking about how this work helps connect these two seemingly very different fields of study - VLMs and world models! 🧵(1/6) Paper: Fun demo:

Punchline: World models == VQA (about the future)! Planning with world models can be powerful for robotics/control. But most world models are video generators trained to predict everything, including irrelevant pixels and distractions. We ask - what if a world model only predicted the semantic information necessary for decision-making? Introducing Semantic World Models (SWM). Given an observation and an action sequence, SWMs cast modeling as answering textual questions about the future outcome resulting from the actions. Recasting world modeling as a VQA problem lets us directly leverage the pretrained knowledge and machinery of VLMs for generalizable modeling. We had a lot of fun thinking about how this work helps connect these two seemingly very different fields of study - VLMs and world models! 🧵(1/6) Paper: Fun demo:

Abhishek Gupta

61,390 Aufrufe • vor 9 Monaten

There's a lot of speculation about whether OpenAI's video generation model Sora has a 'physics engine' (bolstered by OAI's own claims about 'world simulation'). Like the debate about world models in LLMs, this question is both genuinely interesting and somewhat ill-defined. 🧵1/

There's a lot of speculation about whether OpenAI's video generation model Sora has a 'physics engine' (bolstered by OAI's own claims about 'world simulation'). Like the debate about world models in LLMs, this question is both genuinely interesting and somewhat ill-defined. 🧵1/

Raphaël Millière

617,441 Aufrufe • vor 2 Jahren

Today we announced RTFM (Real-Time Frame Model) — a world model that generates frames in real time from any camera viewpoint. Unlike standard video models, RTFM understands 3D geometry. You can literally move the camera through the generated world. 🎥👇

Today we announced RTFM (Real-Time Frame Model) — a world model that generates frames in real time from any camera viewpoint. Unlike standard video models, RTFM understands 3D geometry. You can literally move the camera through the generated world. 🎥👇

Keunhong Park

15,823 Aufrufe • vor 10 Monaten

SITUATION EXPLAINED: A new generation of AI labs are betting that world models are the path to general intelligence that can interact with the real world. We had General Intuition CEO Pim de Witte on MTS to explain what world models are, and why intuitively world models offer a clearer path towards general intelligence than LLMs. For humans, text is a technology for compressing information about the unfolding, three-dimensional world around us and communicating it to other people. When LLMs reason about the world, they do so by reconstructing the world through text. World models work by modeling the three-dimensional world at the architecture level and then allowing agents to interact with that world. Unlike an LLM, which understands space and time through text, world models perform spatial and temporal reasoning natively. General Intuition hopes to use world models to create embodied AI—machines that can reason and act in the real world. But other companies are also pushing the frontier in this space. Odyssey ML today announced a multi-agent world model, Agora-1. Their launch video shows four participants interacting with a real-time world model simulation of GoldenEye, a first-person shooter for the N64. A playable preview of the GoldenEye simulation is available on their site.

SITUATION EXPLAINED: A new generation of AI labs are betting that world models are the path to general intelligence that can interact with the real world. We had General Intuition CEO Pim de Witte on MTS to explain what world models are, and why intuitively world models offer a clearer path towards general intelligence than LLMs. For humans, text is a technology for compressing information about the unfolding, three-dimensional world around us and communicating it to other people. When LLMs reason about the world, they do so by reconstructing the world through text. World models work by modeling the three-dimensional world at the architecture level and then allowing agents to interact with that world. Unlike an LLM, which understands space and time through text, world models perform spatial and temporal reasoning natively. General Intuition hopes to use world models to create embodied AI—machines that can reason and act in the real world. But other companies are also pushing the frontier in this space. Odyssey ML today announced a multi-agent world model, Agora-1. Their launch video shows four participants interacting with a real-time world model simulation of GoldenEye, a first-person shooter for the N64. A playable preview of the GoldenEye simulation is available on their site.

MTS

11,641 Aufrufe • vor 3 Monaten

🚀 Why am I, launching the Ephemeral Ethernal Mint Pass? It's a unique blend of a Mint Pass & Premium Membership that unlocks the universe of projects I've been involved in! We're seeking 100 patrons to help bridge the physical & digital world. Dive in to learn more!👇 🧵1/

🚀 Why am I, launching the Ephemeral Ethernal Mint Pass? It's a unique blend of a Mint Pass & Premium Membership that unlocks the universe of projects I've been involved in! We're seeking 100 patrons to help bridge the physical & digital world. Dive in to learn more!👇 🧵1/

Vhils

11,006 Aufrufe • vor 3 Jahren

New research from BlackForestLabsAI - Unofficial 🥳 Meet Self-Flow: our self-supervised framework for image, audio, video & world models 🤖 Do generative models really need DINO to learn strong representations? We propose teaching them directly via a joint framework instead 🧵

New research from BlackForestLabsAI - Unofficial 🥳 Meet Self-Flow: our self-supervised framework for image, audio, video & world models 🤖 Do generative models really need DINO to learn strong representations? We propose teaching them directly via a joint framework instead 🧵

Hila Chefer

68,791 Aufrufe • vor 5 Monaten

Six months ago, on the first-ever episode of The Humanoid Hub, Eric Jang discussed World Models as something that wasn't being taken seriously as the core of AI systems. Today, 1X introduced a robotics policy that turns that vision into reality, converting video generation from World Models into real-world robot actions.

Six months ago, on the first-ever episode of The Humanoid Hub, Eric Jang discussed World Models as something that wasn't being taken seriously as the core of AI systems. Today, 1X introduced a robotics policy that turns that vision into reality, converting video generation from World Models into real-world robot actions.

The Humanoid Hub

30,933 Aufrufe • vor 7 Monaten

Never bet against Cursor. Subagents solve so many issues between different models it's kind of insane, especially with Composer 1. Composer 1 can quickly gather context for a model like GPT-5.2 Codex, which otherwise could take on the order of 10 mins just to search files. Definitely one of the most underrated models right now. Now you get the benefits of it's speed, but can handoff to a smarter model at implementation. The video I attached is just a simple example, but you can see how long GPT 5.2 takes to gather context on a very simple problem. For harder problems, the search time grows significantly.

Never bet against Cursor. Subagents solve so many issues between different models it's kind of insane, especially with Composer 1. Composer 1 can quickly gather context for a model like GPT-5.2 Codex, which otherwise could take on the order of 10 mins just to search files. Definitely one of the most underrated models right now. Now you get the benefits of it's speed, but can handoff to a smarter model at implementation. The video I attached is just a simple example, but you can see how long GPT 5.2 takes to gather context on a very simple problem. For harder problems, the search time grows significantly.

ostyn

19,708 Aufrufe • vor 6 Monaten

Today at #GoogleIO we've announced: 🤖 Project Astra: our vision for AI assistants 🎨 Imagen 3 & Veo: our new image and video gen models ♊ Gemini 1.5 Flash: a light-weight multimodal model with long context & 1.5 Pro with 2M tokens... "Tuesday" as we call it Google DeepMind

Today at #GoogleIO we've announced: 🤖 Project Astra: our vision for AI assistants 🎨 Imagen 3 & Veo: our new image and video gen models ♊ Gemini 1.5 Flash: a light-weight multimodal model with long context & 1.5 Pro with 2M tokens... "Tuesday" as we call it Google DeepMind

Alexandre Moufarek

111,189 Aufrufe • vor 2 Jahren

Introducing WorldForge: testable world-model workflows for physical AI systems. You can think of it, loosely, as “LangChain for world models”. The problem is that “world model” has become an overloaded label. Depending on context, it can mean a video generator, a cost model, a robot policy, a JEPA-style latent predictor, etc. They share almost nothing, different inputs, runtimes, failure modes. I built WorldForge to stop pretending they're interchangeable. Front-door demo: a real Hugging Face LeRobot (LeRobot) diffusion_pusht policy combined with LeWorldModel by Lucas Maes checkpoint for scoring. Both run locally on my MacBook in the demo video. LeWM is extremely efficient (~15M params), can plan up to 48× faster, and runs on commodity hardware. WorldForge wires the loop: policy → candidates → score → select Replay happens in a local TUI today, but the same loop could drive a real robot in the world. Would love feedback from people working on world models, physical AI, robotics, ML infra, and adjacent tooling. Fully open source. Contributions very welcome. Plan in the dream, replay in real world.

Introducing WorldForge: testable world-model workflows for physical AI systems. You can think of it, loosely, as “LangChain for world models”. The problem is that “world model” has become an overloaded label. Depending on context, it can mean a video generator, a cost model, a robot policy, a JEPA-style latent predictor, etc. They share almost nothing, different inputs, runtimes, failure modes. I built WorldForge to stop pretending they're interchangeable. Front-door demo: a real Hugging Face LeRobot (LeRobot) diffusion_pusht policy combined with LeWorldModel by Lucas Maes checkpoint for scoring. Both run locally on my MacBook in the demo video. LeWM is extremely efficient (~15M params), can plan up to 48× faster, and runs on commodity hardware. WorldForge wires the loop: policy → candidates → score → select Replay happens in a local TUI today, but the same loop could drive a real robot in the world. Would love feedback from people working on world models, physical AI, robotics, ML infra, and adjacent tooling. Fully open source. Contributions very welcome. Plan in the dream, replay in real world.

abdel

17,001 Aufrufe • vor 3 Monaten

Today we release LFM2.5-Encoder-230M and LFM2.5-Encoder-350M: bidirectional encoders that stay fast at long context, even on CPU. > LFM2.5-Encoder-230M: about 3.7x faster than ModernBERT-base on CPU at 8,192 tokens. Under 30s per forward pass, versus over a minute and a half. > LFM2.5-Encoder-350M: 4th of 14 models on GLUE, SuperGLUE, and multilingual classification, behind only three larger models, one of them nearly 10x its size. 🧵

Today we release LFM2.5-Encoder-230M and LFM2.5-Encoder-350M: bidirectional encoders that stay fast at long context, even on CPU. > LFM2.5-Encoder-230M: about 3.7x faster than ModernBERT-base on CPU at 8,192 tokens. Under 30s per forward pass, versus over a minute and a half. > LFM2.5-Encoder-350M: 4th of 14 models on GLUE, SuperGLUE, and multilingual classification, behind only three larger models, one of them nearly 10x its size. 🧵

Liquid AI

197,707 Aufrufe • vor 22 Tagen

This is a single uncut video, showing a robot learning several tasks instantly, after just one demonstration each ... This is possible because we've now been able to achieve in-context learning for everyday robotics tasks, and I'm very excited to announce our latest paper: 🎆 Instant Policy: In-Context Imitation Learning via Graph Diffusion 🎆 (1/6) 🧵👇

This is a single uncut video, showing a robot learning several tasks instantly, after just one demonstration each ... This is possible because we've now been able to achieve in-context learning for everyday robotics tasks, and I'm very excited to announce our latest paper: 🎆 Instant Policy: In-Context Imitation Learning via Graph Diffusion 🎆 (1/6) 🧵👇

Edward Johns

74,683 Aufrufe • vor 1 Jahr

🚀 We open-sourced LongLive — interactive, real-time long-video generation. 👥Generates video in real time as users enter text prompts. ⚡️20.7 FPS on a single H100,⏱️up to 240s per clip. 🎬Fine-tunes SOTA short-video models (e.g., Wan) into long-video generators. 🌍One step closer to World Models. All code for training & inference, model weights, demo page, and videos released! Paper: Code: Model: Demo Page: Introduction Video:

🚀 We open-sourced LongLive — interactive, real-time long-video generation. 👥Generates video in real time as users enter text prompts. ⚡️20.7 FPS on a single H100,⏱️up to 240s per clip. 🎬Fine-tunes SOTA short-video models (e.g., Wan) into long-video generators. 🌍One step closer to World Models. All code for training & inference, model weights, demo page, and videos released! Paper: Code: Model: Demo Page: Introduction Video:

Yukang Chen

11,915 Aufrufe • vor 10 Monaten

Urban Company emerged as a beacon of hope for women wanting to gain economic independence in India. Today, they feel trapped by the app. Watch the first episode of Gig Workers Rising, a video collaboration from Context News and Rest of World:

Urban Company emerged as a beacon of hope for women wanting to gain economic independence in India. Today, they feel trapped by the app. Watch the first episode of Gig Workers Rising, a video collaboration from Context News and Rest of World:

Context News

13,812 Aufrufe • vor 2 Jahren

Demis Hassabis says world models are his longest standing passion and explains benefits vs. language models: ▫️ “I think language models are able to understand a lot about the world. More than we expected because language is actually probably richer than we thought. But there's still a lot about the spatial dynamics of the world, spatial awareness and the physical context we're in — and how that works mechanically — that is hard to describe in words and isn't generally described in corpuses of words. Alot of this is allied to learning from experience. There's a lot of things which you can't really describe something. You have to just experience it. Maybe the senses and so on are very hard to put into words. Whether that's motor angles and smell and these kinds of senses, it's very difficult to describe that in any kind of language.”▫️ This is what Demis and Google Deepmind is trying to solve with Genie. He also says that the video models (Veo) will play a part in training the world models and this is all key for AI robotics.

Demis Hassabis says world models are his longest standing passion and explains benefits vs. language models: ▫️ “I think language models are able to understand a lot about the world. More than we expected because language is actually probably richer than we thought. But there's still a lot about the spatial dynamics of the world, spatial awareness and the physical context we're in — and how that works mechanically — that is hard to describe in words and isn't generally described in corpuses of words. Alot of this is allied to learning from experience. There's a lot of things which you can't really describe something. You have to just experience it. Maybe the senses and so on are very hard to put into words. Whether that's motor angles and smell and these kinds of senses, it's very difficult to describe that in any kind of language.”▫️ This is what Demis and Google Deepmind is trying to solve with Genie. He also says that the video models (Veo) will play a part in training the world models and this is all key for AI robotics.

Bearly AI

123,703 Aufrufe • vor 8 Monaten