正在加载视频...

视频加载失败

加载此视频时出现问题。这可能是由于临时网络问题，或视频可能不可用。

Been thinking about what this paper really means. "Video diffusion" and "World Models" are becoming synonymous. Neural Computers are basically video diffusion world models for terminal envs and GUI. Lots of talk last week about automating Manim videos. In theory, we should be able to train these world models... show more

AVB

14,542 subscribers

120,301 次观看 • 7 个月前 •via X (Twitter)

教育科学技术

Anya Rossi• Live Now

Private livecam show

17 条评论

AVB 的头像

AVB7 个月前

To be clear, I don’t particularly believe this is gonna happen or it can replace the quality of human Manim. Then again I’m a known skeptic. I think video gen models will be more creative than LLMs writing Manim code. It will be less precise with math though. There’s probably a middle ground where the best solution lies.

Zain Shah 的头像

Zain Shah7 个月前

I have 10k hours of LLM generated manim videos for Have been playing around with interactive world models as well but maybe the manim videos would be good to train on too. Interesting idea

Dr. Rachid EJJAMI 的头像

Dr. Rachid EJJAMI7 个月前

Interesting how confidence in output doesn’t always match correctness.

AVB 的头像

AVB7 个月前

True for both humans and agents!

DeFi Maria 的头像

DeFi Maria7 个月前

wild if true

pixel pusher 的头像

pixel pusher7 个月前

interesting

osufever 的头像

osufever7 个月前

these worldmodels dont train on video. no one scraped youtube

Vai Viswanathan 的头像

Vai Viswanathan7 个月前

Video diffusion and world models are not synonymous. Video diffusion is one approach for World Modeling

Far 的头像

Far7 个月前

so we're just training a model on manim to predict terminal outputs now? because if it can hallucinate the whole video without code, i assume it'll also hallucinate the gpu drivers.

AVB 的头像

AVB7 个月前

Nah they just trained on terminal outputs (and GUI). The manim thing is just something I'm ideating

Ranko Mosic 的头像

Ranko Mosic7 个月前

NC is much wider than the current Wan 2.1 POC. The idea seems to be to create a learned runtime to replace von Neumann based stack.

AVB 的头像

AVB7 个月前

Yeah for sure. The CNC stuff seems definitely very utopian from the distance. Let’s see where it goes, but at its current state it’s a video diffusion model with action primitives… which is definitely also pretty cool!

Duoduo 的头像

Duoduo7 个月前

A neural computer is not just a better browser agent. It's the point where the environment stops being external tooling and starts becoming part of the model's body.

크롱 的头像

크롱7 个月前

비디오 디퓨전이랑 월드 모델이 결국 한 몸이 되어가는 분위기네. GUI 환경에서 단순히 화면을 재현하는 걸 넘어 진짜 시스템의 논리를 이해하는 수준까지 갈 수 있을지가 관건일 듯.

OpenLedger Intern 的头像

OpenLedger Intern7 个月前

so we’re slowly moving from “write code to make video” to “just describe it and hope the model cooks”

Noke - SF Nov 21-22 的头像

Noke - SF Nov 21-227 个月前

Although it might be interesting seeing the quasi-random outputs - this should be taken as more be a means for an AI system to explore an operations state space, not something that a human would gain much utility out of

Aran Nayebi 的头像

Aran Nayebi7 个月前

It's a computer-use example of a broader principle: capable agents *must* build world models internally, even under partial observability. More details here:

相关视频

Dreamix: Video Diffusion Models are General Video Editors abs: project page: present diffusion-based method that is able to perform text-based motion and appearance editing of general videos

Dreamix: Video Diffusion Models are General Video Editors abs: project page: present diffusion-based method that is able to perform text-based motion and appearance editing of general videos

AK

398,258 次观看 • 3 年前

Video diffusion models generate high-quality videos but are too slow for interactive applications. We MIT CSAIL Adobe Research introduce CausVid, a fast autoregressive video diffusion model that starts playing the moment you hit "Generate"! A thread 🧵

Video diffusion models generate high-quality videos but are too slow for interactive applications. We MIT CSAIL Adobe Research introduce CausVid, a fast autoregressive video diffusion model that starts playing the moment you hit "Generate"! A thread 🧵

Tianwei Yin

84,074 次观看 • 1 年前

Punchline: World models == VQA (about the future)! Planning with world models can be powerful for robotics/control. But most world models are video generators trained to predict everything, including irrelevant pixels and distractions. We ask - what if a world model only predicted the semantic information necessary for decision-making? Introducing Semantic World Models (SWM). Given an observation and an action sequence, SWMs cast modeling as answering textual questions about the future outcome resulting from the actions. Recasting world modeling as a VQA problem lets us directly leverage the pretrained knowledge and machinery of VLMs for generalizable modeling. We had a lot of fun thinking about how this work helps connect these two seemingly very different fields of study - VLMs and world models! 🧵(1/6) Paper: Fun demo:

Punchline: World models == VQA (about the future)! Planning with world models can be powerful for robotics/control. But most world models are video generators trained to predict everything, including irrelevant pixels and distractions. We ask - what if a world model only predicted the semantic information necessary for decision-making? Introducing Semantic World Models (SWM). Given an observation and an action sequence, SWMs cast modeling as answering textual questions about the future outcome resulting from the actions. Recasting world modeling as a VQA problem lets us directly leverage the pretrained knowledge and machinery of VLMs for generalizable modeling. We had a lot of fun thinking about how this work helps connect these two seemingly very different fields of study - VLMs and world models! 🧵(1/6) Paper: Fun demo:

Abhishek Gupta

61,539 次观看 • 1 年前

Every home is different. That means that to build a useful home robot, we must be able to perform zero-shot generalization on a wide range of tasks. Humanoid company 1X has a solution: world models. 1X Director of Evaluations Daniel Ho joins us on RoboPapers to talk about: - why world models are the future for scaling robot learning - how to use world models for robot control - what world models unlock for evaluating robot model performance - how we can hill-climb from here to general purpose robots Watch Episode #61 of RoboPapers, with Michael Cho - Rbt/Acc and Chris Paxton, now!

Every home is different. That means that to build a useful home robot, we must be able to perform zero-shot generalization on a wide range of tasks. Humanoid company 1X has a solution: world models. 1X Director of Evaluations Daniel Ho joins us on RoboPapers to talk about: - why world models are the future for scaling robot learning - how to use world models for robot control - what world models unlock for evaluating robot model performance - how we can hill-climb from here to general purpose robots Watch Episode #61 of RoboPapers, with Michael Cho - Rbt/Acc and Chris Paxton, now!

RoboPapers

28,039 次观看 • 9 个月前

LLM-grounded Diffusion: Enhancing Prompt Understanding of Text-to-Image Diffusion Models with Large Language Models paper page: github: Recent advancements in text-to-image generation with diffusion models have yielded remarkable results synthesizing highly realistic and diverse images. However, these models still encounter difficulties when generating images from prompts that demand spatial or common sense reasoning. We propose to equip diffusion models with enhanced reasoning capabilities by using off-the-shelf pretrained large language models (LLMs) in a novel two-stage generation process. First, we adapt an LLM to be a text-guided layout generator through in-context learning. When provided with an image prompt, an LLM outputs a scene layout in the form of bounding boxes along with corresponding individual descriptions. Second, we steer a diffusion model with a novel controller to generate images conditioned on the layout. Both stages utilize frozen pretrained models without any LLM or diffusion model parameter optimization. We validate the superiority of our design by demonstrating its ability to outperform the base diffusion model in accurately generating images according to prompts that necessitate both language and spatial reasoning. Additionally, our method naturally allows dialog-based scene specification and is able to handle prompts in a language that is not well-supported by the underlying diffusion model.

LLM-grounded Diffusion: Enhancing Prompt Understanding of Text-to-Image Diffusion Models with Large Language Models paper page: github: Recent advancements in text-to-image generation with diffusion models have yielded remarkable results synthesizing highly realistic and diverse images. However, these models still encounter difficulties when generating images from prompts that demand spatial or common sense reasoning. We propose to equip diffusion models with enhanced reasoning capabilities by using off-the-shelf pretrained large language models (LLMs) in a novel two-stage generation process. First, we adapt an LLM to be a text-guided layout generator through in-context learning. When provided with an image prompt, an LLM outputs a scene layout in the form of bounding boxes along with corresponding individual descriptions. Second, we steer a diffusion model with a novel controller to generate images conditioned on the layout. Both stages utilize frozen pretrained models without any LLM or diffusion model parameter optimization. We validate the superiority of our design by demonstrating its ability to outperform the base diffusion model in accurately generating images according to prompts that necessitate both language and spatial reasoning. Additionally, our method naturally allows dialog-based scene specification and is able to handle prompts in a language that is not well-supported by the underlying diffusion model.

AK

83,681 次观看 • 3 年前

China is all about Propaganda and Censorship. Without the Firewall and Censorship, the entire world would be able to see videos like this one. As a result, all of their “best infrastructure in the world” propaganda would be busted.

China is all about Propaganda and Censorship. Without the Firewall and Censorship, the entire world would be able to see videos like this one. As a result, all of their “best infrastructure in the world” propaganda would be busted.

Pseudo Prophet

170,430 次观看 • 9 个月前

If you look at the best image/video diffusion models, they are still not able to get the hands quite right, especially when interacting with objects ✍️ We present HOIDiffusion #CVPR2024, a way to generate accurate and realistic hand-object interaction images, in diverse poses, scenes, and styles. This is not just for synthesizing beautiful pixels, but also a way to generate infinite pairs of hand-object images and 3D GTs data.

If you look at the best image/video diffusion models, they are still not able to get the hands quite right, especially when interacting with objects ✍️ We present HOIDiffusion #CVPR2024, a way to generate accurate and realistic hand-object interaction images, in diverse poses, scenes, and styles. This is not just for synthesizing beautiful pixels, but also a way to generate infinite pairs of hand-object images and 3D GTs data.

Xiaolong Wang

33,673 次观看 • 2 年前

Are these simulations "AI"? AI animations are a new type of AI-automated content. They're made by a different process than other AI videos, which typically rely on diffusion models. These videos use large language models, like ChatGPT and Claude, to operate rendering software like Unreal Engine and Blender, or code their own software from scratch. This is a big advancement in automated content, and I think lot of automated videos will move toward this process over time.

Are these simulations "AI"? AI animations are a new type of AI-automated content. They're made by a different process than other AI videos, which typically rely on diffusion models. These videos use large language models, like ChatGPT and Claude, to operate rendering software like Unreal Engine and Blender, or code their own software from scratch. This is a big advancement in automated content, and I think lot of automated videos will move toward this process over time.

Jeremy Carrasco

456,300 次观看 • 1 个月前

V3D Video Diffusion Models are Effective 3D Generators Automatic 3D generation has recently attracted widespread attention. Recent methods have greatly accelerated the generation speed, but usually produce less-detailed objects due to limited model capacity or 3D data. Motivated by recent advancements in video diffusion models, we introduce V3D, which leverages the world simulation capacity of pre-trained video diffusion models to facilitate 3D generation. To fully unleash the potential of video diffusion to perceive the 3D world, we further introduce geometrical consistency prior and extend the video diffusion model to a multi-view consistent 3D generator. Benefiting from this, the state-of-the-art video diffusion model could be fine-tuned to generate 360degree orbit frames surrounding an object given a single image. With our tailored reconstruction pipelines, we can generate high-quality meshes or 3D Gaussians within 3 minutes. Furthermore, our method can be extended to scene-level novel view synthesis, achieving precise control over the camera path with sparse input views. Extensive experiments demonstrate the superior performance of the proposed approach, especially in terms of generation quality and multi-view consistency

V3D Video Diffusion Models are Effective 3D Generators Automatic 3D generation has recently attracted widespread attention. Recent methods have greatly accelerated the generation speed, but usually produce less-detailed objects due to limited model capacity or 3D data. Motivated by recent advancements in video diffusion models, we introduce V3D, which leverages the world simulation capacity of pre-trained video diffusion models to facilitate 3D generation. To fully unleash the potential of video diffusion to perceive the 3D world, we further introduce geometrical consistency prior and extend the video diffusion model to a multi-view consistent 3D generator. Benefiting from this, the state-of-the-art video diffusion model could be fine-tuned to generate 360degree orbit frames surrounding an object given a single image. With our tailored reconstruction pipelines, we can generate high-quality meshes or 3D Gaussians within 3 minutes. Furthermore, our method can be extended to scene-level novel view synthesis, achieving precise control over the camera path with sparse input views. Extensive experiments demonstrate the superior performance of the proposed approach, especially in terms of generation quality and multi-view consistency

AK

31,997 次观看 • 2 年前

TokenFlow: Consistent Diffusion Features for Consistent Video Editing paper page: The generative AI revolution has recently expanded to videos. Nevertheless, current state-of-the-art video models are still lagging behind image models in terms of visual quality and user control over the generated content. In this work, we present a framework that harnesses the power of a text-to-image diffusion model for the task of text-driven video editing. Specifically, given a source video and a target text-prompt, our method generates a high-quality video that adheres to the target text, while preserving the spatial layout and motion of the input video. Our method is based on a key observation that consistency in the edited video can be obtained by enforcing consistency in the diffusion feature space. We achieve this by explicitly propagating diffusion features based on inter-frame correspondences, readily available in the model. Thus, our framework does not require any training or fine-tuning, and can work in conjunction with any off-the-shelf text-to-image editing method. We demonstrate state-of-the-art editing results on a variety of real-world videos.

TokenFlow: Consistent Diffusion Features for Consistent Video Editing paper page: The generative AI revolution has recently expanded to videos. Nevertheless, current state-of-the-art video models are still lagging behind image models in terms of visual quality and user control over the generated content. In this work, we present a framework that harnesses the power of a text-to-image diffusion model for the task of text-driven video editing. Specifically, given a source video and a target text-prompt, our method generates a high-quality video that adheres to the target text, while preserving the spatial layout and motion of the input video. Our method is based on a key observation that consistency in the edited video can be obtained by enforcing consistency in the diffusion feature space. We achieve this by explicitly propagating diffusion features based on inter-frame correspondences, readily available in the model. Thus, our framework does not require any training or fine-tuning, and can work in conjunction with any off-the-shelf text-to-image editing method. We demonstrate state-of-the-art editing results on a variety of real-world videos.

AK

143,565 次观看 • 3 年前

Over the last few months, we’ve been thinking about how to learn from “off-domain” data - data from non-robot sources like video or simulation. These data sources are not quite good enough to learn policies (even monolithic VLA models) directly, but they still contain lots of information that can be useful for generalizable robot control. How can we develop robot learning models that are able to make use of this type of data for generalizable control? In new work, that we call HAMSTER, we show that VLMs can be useful for enabling robotic learning from off-domain data, but specifically when used through hierarchical VLA architectures. We show that this class of models can learn generalizable robot policies for the real world from large-scale, off-domain data. A 🧵 (1/10)

Over the last few months, we’ve been thinking about how to learn from “off-domain” data - data from non-robot sources like video or simulation. These data sources are not quite good enough to learn policies (even monolithic VLA models) directly, but they still contain lots of information that can be useful for generalizable robot control. How can we develop robot learning models that are able to make use of this type of data for generalizable control? In new work, that we call HAMSTER, we show that VLMs can be useful for enabling robotic learning from off-domain data, but specifically when used through hierarchical VLA architectures. We show that this class of models can learn generalizable robot policies for the real world from large-scale, off-domain data. A 🧵 (1/10)

Abhishek Gupta

11,994 次观看 • 1 年前

VSTAR Generative Temporal Nursing for Longer Dynamic Video Synthesis Despite tremendous progress in the field of text-to-video (T2V) synthesis, open-sourced T2V diffusion models struggle to generate longer videos with dynamically varying and evolving content. They tend to

VSTAR Generative Temporal Nursing for Longer Dynamic Video Synthesis Despite tremendous progress in the field of text-to-video (T2V) synthesis, open-sourced T2V diffusion models struggle to generate longer videos with dynamically varying and evolving content. They tend to

AK

37,003 次观看 • 2 年前

Generative Rendering: Controllable 4D-Guided Video Generation with 2D Diffusion Models paper page: Traditional 3D content creation tools empower users to bring their imagination to life by giving them direct control over a scene's geometry, appearance, motion, and camera path. Creating computer-generated videos, however, is a tedious manual process, which can be automated by emerging text-to-video diffusion models. Despite great promise, video diffusion models are difficult to control, hindering a user to apply their own creativity rather than amplifying it. To address this challenge, we present a novel approach that combines the controllability of dynamic 3D meshes with the expressivity and editability of emerging diffusion models. For this purpose, our approach takes an animated, low-fidelity rendered mesh as input and injects the ground truth correspondence information obtained from the dynamic mesh into various stages of a pre-trained text-to-image generation model to output high-quality and temporally consistent frames. We demonstrate our approach on various examples where motion can be obtained by animating rigged assets or changing the camera path.

Generative Rendering: Controllable 4D-Guided Video Generation with 2D Diffusion Models paper page: Traditional 3D content creation tools empower users to bring their imagination to life by giving them direct control over a scene's geometry, appearance, motion, and camera path. Creating computer-generated videos, however, is a tedious manual process, which can be automated by emerging text-to-video diffusion models. Despite great promise, video diffusion models are difficult to control, hindering a user to apply their own creativity rather than amplifying it. To address this challenge, we present a novel approach that combines the controllability of dynamic 3D meshes with the expressivity and editability of emerging diffusion models. For this purpose, our approach takes an animated, low-fidelity rendered mesh as input and injects the ground truth correspondence information obtained from the dynamic mesh into various stages of a pre-trained text-to-image generation model to output high-quality and temporally consistent frames. We demonstrate our approach on various examples where motion can be obtained by animating rigged assets or changing the camera path.

AK

117,091 次观看 • 2 年前

Generate images and video directly from your terminal with Claude + our new genmedia CLI. • Search and run fal models from CLI • Generate structured commands with Claude • Run async media workflows • Download outputs locally • Use installable agent skills Watch the full tutorial:

Generate images and video directly from your terminal with Claude + our new genmedia CLI. • Search and run fal models from CLI • Generate structured commands with Claude • Run async media workflows • Download outputs locally • Use installable agent skills Watch the full tutorial:

fal

11,549 次观看 • 6 个月前

🚨 NEW: DNI Director Jay Clayton tells Fox’s Bret Baier on AI: "I think this is a race that we need to win.” “And the reason... is if you talk to people who have seen what these models can do and are projecting what they will be able to do in the future, the power is immense.” “I believe that if we are the strongest country in the world, the world will be a much better place. And I look at AI frontier models in the very same framework."

🚨 NEW: DNI Director Jay Clayton tells Fox’s Bret Baier on AI: "I think this is a race that we need to win.” “And the reason... is if you talk to people who have seen what these models can do and are projecting what they will be able to do in the future, the power is immense.” “I believe that if we are the strongest country in the world, the world will be a much better place. And I look at AI frontier models in the very same framework."

TV News Now

10,946 次观看 • 2 个月前

🔥Excited to share the first released work from our IEI lab! Congrats to AnteaWu 🎉 This work is motivated by the lack of quantitative evaluation for physics alignment in video world models. With tools like MegaSam and CoTracker, we can directly reconstruct dynamic 3D scenes, enabling quantitative evaluation of physical alignment. Both code and data are released — feel free to try it out! It should work, but if it doesn’t, contact AnteaWu directly : )

🔥Excited to share the first released work from our IEI lab! Congrats to AnteaWu 🎉 This work is motivated by the lack of quantitative evaluation for physics alignment in video world models. With tools like MegaSam and CoTracker, we can directly reconstruct dynamic 3D scenes, enabling quantitative evaluation of physical alignment. Both code and data are released — feel free to try it out! It should work, but if it doesn’t, contact AnteaWu directly : )

Xueyan Zou

10,239 次观看 • 5 个月前

World Labs CEO Fei-Fei Li on the need for world models: "There's no language out there. You don't go out in nature and there's words written in the sky for you." "There is a 3D world out there that follows laws of physics, that has its own structures due to materials. To fundamentally back that information out and be able to represent it and be able to generate it is just fundamentally quite a different problem." "We will be borrowing useful ideas from language and LLMs, but this is fundamentally, philosophically a different problem."

World Labs CEO Fei-Fei Li on the need for world models: "There's no language out there. You don't go out in nature and there's words written in the sky for you." "There is a 3D world out there that follows laws of physics, that has its own structures due to materials. To fundamentally back that information out and be able to represent it and be able to generate it is just fundamentally quite a different problem." "We will be borrowing useful ideas from language and LLMs, but this is fundamentally, philosophically a different problem."

a16z

204,772 次观看 • 2 个月前

I finally released my new video on YouTube about Diffusion Models / Score-Based Generative Models. Literally planned this for a year and put so much work in. I think this approach to diffusion models is so intuitive and highly recommend giving that a go! Video is 38min long, so you will need some time to watch that haha.

I finally released my new video on YouTube about Diffusion Models / Score-Based Generative Models. Literally planned this for a year and put so much work in. I think this approach to diffusion models is so intuitive and highly recommend giving that a go! Video is 38min long, so you will need some time to watch that haha.

dome | Outlier

54,611 次观看 • 2 年前

Diffusion models are now live on Darkwoods. Text to image and text to video generation now available privately alongside LLM chat in a single interface. Two modes available - "TEE" for fully private image and video generation and "anonymous" for anonymised prompting with leading models. Create your world.

Diffusion models are now live on Darkwoods. Text to image and text to video generation now available privately alongside LLM chat in a single interface. Two modes available - "TEE" for fully private image and video generation and "anonymous" for anonymised prompting with leading models. Create your world.

darkwoods

24,274 次观看 • 1 个月前