Loading video...

Video Failed to Load

Go Home

NeoVerse Enhancing 4D World Model with in-the-wild Monocular Videos

_akhaliq's profile picture

AK

506,624 subscribers

23,598 views • 6 months ago •via X (Twitter)

LIVE

Anya Rossi• Live Now

Private livecam show

0 Comments

No comments available

Comments from the original post will appear here

Related Videos

4DGT: Learning a 4D Gaussian Transformer Using Real-World Monocular Videos Abstract: We propose 4DGT, a 4D Gaussian-based Transformer model for dynamic scene reconstruction, trained entirely on real-world monocular posed videos. Using 4D Gaussian as an inductive bias, 4DGT unifies static and dynamic components, enabling the modeling of complex, time-varying environments with varying object lifespans. We introduced a novel density control strategy in training, which allows our 4DGT to handle longer space-time input while maintaining efficient rendering at runtime. Our model processes 64 consecutive posed frames in a rolling-window fashion, predicting consistent 4D Gaussians in the scene. Unlike optimization-based methods, 4DGT performs purely feed-forward inference, reducing reconstruction time from hours to seconds and scaling effectively to long video sequences. Trained only on large-scale monocular posed video datasets, 4DGT can significantly outperform prior Gaussian-based networks in real-world videos and achieve on-par accuracy with optimization-based methods on cross-domain videos.
5:19

4DGT: Learning a 4D Gaussian Transformer Using Real-World Monocular Videos Abstract: We propose 4DGT, a 4D Gaussian-based Transformer model for dynamic scene reconstruction, trained entirely on real-world monocular posed videos. Using 4D Gaussian as an inductive bias, 4DGT unifies static and dynamic components, enabling the modeling of complex, time-varying environments with varying object lifespans. We introduced a novel density control strategy in training, which allows our 4DGT to handle longer space-time input while maintaining efficient rendering at runtime. Our model processes 64 consecutive posed frames in a rolling-window fashion, predicting consistent 4D Gaussians in the scene. Unlike optimization-based methods, 4DGT performs purely feed-forward inference, reducing reconstruction time from hours to seconds and scaling effectively to long video sequences. Trained only on large-scale monocular posed video datasets, 4DGT can significantly outperform prior Gaussian-based networks in real-world videos and achieve on-par accuracy with optimization-based methods on cross-domain videos.

MrNeRF

34,782 views • 1 year ago

Robot Learning needs 4D world models! Robot Learning needs 4D world models! Robot Learning needs 4D world models! We introduce TesserAct, a 4D embodied world model that can simulate how agents interact with the 3D world over time! We achieve this by simply extending a pre-trained 2D video generation model to jointly predict RGB, depth, and surface normals. It enables: 1️⃣ Much better policy learning in the wild 2️⃣ Temporal + spatial coherence in 4D dynamic prediction 3️⃣ Novel view synthesis for embodied scenes Code: Paper Link: Project page:
0:32

Robot Learning needs 4D world models! Robot Learning needs 4D world models! Robot Learning needs 4D world models! We introduce TesserAct, a 4D embodied world model that can simulate how agents interact with the 3D world over time! We achieve this by simply extending a pre-trained 2D video generation model to jointly predict RGB, depth, and surface normals. It enables: 1️⃣ Much better policy learning in the wild 2️⃣ Temporal + spatial coherence in 4D dynamic prediction 3️⃣ Novel view synthesis for embodied scenes Code: Paper Link: Project page:

Chuang Gan

43,265 views • 1 year ago

Feature4X Bridging Any Monocular Video to 4D Agentic AI with Versatile Gaussian Feature Fields
1:16

Feature4X Bridging Any Monocular Video to 4D Agentic AI with Versatile Gaussian Feature Fields

AK

13,901 views • 1 year ago

VerseCrafter Dynamic Realistic Video World Model with 4D Geometric Control
0:44

VerseCrafter Dynamic Realistic Video World Model with 4D Geometric Control

AK

26,856 views • 6 months ago

Today we're releasing WildDet3D—an open model for monocular 3D object detection in the wild. It works with text, clicks, or 2D boxes, and on zero-shot evals it nearly doubles the best prior scores. 🧵
0:57

Today we're releasing WildDet3D—an open model for monocular 3D object detection in the wild. It works with text, clicks, or 2D boxes, and on zero-shot evals it nearly doubles the best prior scores. 🧵

Ai2

86,317 views • 3 months ago

Introducing WildDet3D, a grounding model for monocular 3D object detection in the wild. A question I keep coming back to is: what is the right backbone for robotics foundation models? Should it be a video model, a language model, or perhaps a grounding model? WildDet3D is our first step in exploring that direction.
2:10

Introducing WildDet3D, a grounding model for monocular 3D object detection in the wild. A question I keep coming back to is: what is the right backbone for robotics foundation models? Should it be a video model, a language model, or perhaps a grounding model? WildDet3D is our first step in exploring that direction.

Jiafei Duan

12,072 views • 3 months ago

Check out this BIM 4D sim by Thiery Peleias, owner of Virtuart4d 🚧 Imported into UE5 for the animation aspect, the project model was created in Revit, the 4D planning animation in Synchro 4D Pro and the environment captured from Cesium with Google Maps Photorealistic 3D Tiles!
1:47

Check out this BIM 4D sim by Thiery Peleias, owner of Virtuart4d 🚧 Imported into UE5 for the animation aspect, the project model was created in Revit, the 4D planning animation in Synchro 4D Pro and the environment captured from Cesium with Google Maps Photorealistic 3D Tiles!

Unreal Engine

20,772 views • 1 year ago

Absolutely wild. Interactive 4D video live in your browser.
0:25

Absolutely wild. Interactive 4D video live in your browser.

Tim Soret

56,285 views • 1 year ago

We are excited to share EMDB, a novel dataset of 3D human poses for in-the-wild monocular videos, including global trajectories. Data and toolkit code is now available. More details in the thread below. Project Page:
0:36

We are excited to share EMDB, a novel dataset of 3D human poses for in-the-wild monocular videos, including global trajectories. Data and toolkit code is now available. More details in the thread below. Project Page:

AIT Lab

14,601 views • 2 years ago

Kyle Berkshire’s swing in 4D 🤯 The 3x World Long Drive Champion stopped by the Radiant Images 4D stage at the PGA Championship … and didn’t disappoint!
0:22

Kyle Berkshire’s swing in 4D 🤯 The 3x World Long Drive Champion stopped by the Radiant Images 4D stage at the PGA Championship … and didn’t disappoint!

PGA of America

95,333 views • 1 month ago

The best video model in the world just got better. You can now generate 1080p videos with Seedance 2.0 on Venice. Available anonymously.
0:48

The best video model in the world just got better. You can now generate 1080p videos with Seedance 2.0 on Venice. Available anonymously.

Venice

11,280 views • 2 months ago

forget about physical camera stabilizations why don't we reconstruct the world in 4d and rerecord the footage:
1:19

forget about physical camera stabilizations why don't we reconstruct the world in 4d and rerecord the footage:

Dreaming Tulpa 🥓👑

106,083 views • 1 year ago

4D LangSplat: 4D Language Gaussian Splatting via Multimodal Large Language Models Contributions: • We introduce 4D LangSplat for open-vocabulary 4D spatial-temporal queries. To the best of our knowledge, we are the first to construct 4D language fields with object textual captions generated by MLLMs. • To model smooth transitions across states for objects in 4D scenes, we propose a status deformable network to capture continuous temporal changes. • Experiential results show that our method attains state-of-the-art performance for both time-agnostic and time-sensitive open-vocabulary queries.
0:29

4D LangSplat: 4D Language Gaussian Splatting via Multimodal Large Language Models Contributions: • We introduce 4D LangSplat for open-vocabulary 4D spatial-temporal queries. To the best of our knowledge, we are the first to construct 4D language fields with object textual captions generated by MLLMs. • To model smooth transitions across states for objects in 4D scenes, we propose a status deformable network to capture continuous temporal changes. • Experiential results show that our method attains state-of-the-art performance for both time-agnostic and time-sensitive open-vocabulary queries.

MrNeRF

10,953 views • 1 year ago

🔥Unbounded 4D City Generation🔥 #CityDreamer4D is a generative model of unbounded 4D cities that decouples static and dynamic scenes with realism and controllability - Project: - Code: - CityTopia Dataset:
1:02

🔥Unbounded 4D City Generation🔥 #CityDreamer4D is a generative model of unbounded 4D cities that decouples static and dynamic scenes with realism and controllability - Project: - Code: - CityTopia Dataset:

Ziwei Liu

24,326 views • 1 year ago

Bytedance first to the holodeck with a steerable real time video generation world model. Up to a minute long videos 24 fps in HD resolution on 8xH100s. It's only an 8 billion param model...
0:22

Bytedance first to the holodeck with a steerable real time video generation world model. Up to a minute long videos 24 fps in HD resolution on 8xH100s. It's only an 8 billion param model...

Prakash (Ate-a-Pi)

34,144 views • 1 year ago

We’ve upgraded Stable Video Diffusion 4D to Stable Video 4D 2.0 (SV4D 2.0), improving the quality of 4D outputs generated from a single object-centric video. While 3D provides a static view of an object’s shape and size; 4D extends this by including time, showing how the object moves. This multi-view video diffusion model generates a 4D output in three steps: 1️⃣ Starts with an input video of a moving person or object 2️⃣ Generates novel views of the subject from unseen angles 3️⃣ Constructs a single dynamic 4D video output with spatial and temporal consistency You can learn more here: (1/4)
0:30

We’ve upgraded Stable Video Diffusion 4D to Stable Video 4D 2.0 (SV4D 2.0), improving the quality of 4D outputs generated from a single object-centric video. While 3D provides a static view of an object’s shape and size; 4D extends this by including time, showing how the object moves. This multi-view video diffusion model generates a 4D output in three steps: 1️⃣ Starts with an input video of a moving person or object 2️⃣ Generates novel views of the subject from unseen angles 3️⃣ Constructs a single dynamic 4D video output with spatial and temporal consistency You can learn more here: (1/4)

Stability AI

35,974 views • 1 year ago

Interactive 4D Time-Lapse for #VDC in #Construction / #AEC Gauzilla Pro allows for creating a 4D time-lapse from a series of videos. You can convert drone footage into an interactive historical record in 4D running 100% on the web. ✅ 4D time-lapse drastically reduces the need for physical site visits and also enhances communications among teams and stakeholders.
1:02

Interactive 4D Time-Lapse for #VDC in #Construction / #AEC Gauzilla Pro allows for creating a 4D time-lapse from a series of videos. You can convert drone footage into an interactive historical record in 4D running 100% on the web. ✅ 4D time-lapse drastically reduces the need for physical site visits and also enhances communications among teams and stakeholders.

Gauzilla Pro

14,432 views • 26 days ago

Diffuman4D 4D Consistent Human View Synthesis from Sparse-View Videos with Spatio-Temporal Diffusion Models
0:58

Diffuman4D 4D Consistent Human View Synthesis from Sparse-View Videos with Spatio-Temporal Diffusion Models

AK

12,518 views • 1 year ago

A thread of videos of the celebrations tonight in downtown LA after the Dodgers won the World Series 🧵 Starting with the moment the Dodgers won the game and the crowd outside Tom’s went wild:
0:57

A thread of videos of the celebrations tonight in downtown LA after the Dodgers won the World Series 🧵 Starting with the moment the Dodgers won the game and the crowd outside Tom’s went wild:

Jonathan Green

107,843 views • 8 months ago

GameFactory Creating New Games with Generative Interactive Videos present GameFactory, a generalizable world model that learns from a small-scale dataset of Minecraft game videos. By leveraging the prior knowledge of a pretrained video diffusion model, it can create new games in an open domain.
1:14

GameFactory Creating New Games with Generative Interactive Videos present GameFactory, a generalizable world model that learns from a small-scale dataset of Minecraft game videos. By leveraging the prior knowledge of a pretrained video diffusion model, it can create new games in an open domain.

AK

70,029 views • 1 year ago