Video wird geladen...

Video konnte nicht geladen werden

Zur Startseite

Can we scale 4D pretraining to learn general space-time representations that reconstruct an object from a few views at any time to any view at any other time? Introducing 4D-LRM: a Large Space-Time Reconstruction Model that ... 🔹 Predicts 4D Gaussian primitives directly from multi-view tokens (no motion vectors,...

ziqiao_ma's profile picture

Martin Ziqiao Ma

4,436 subscribers

21,787 Aufrufe • vor 11 Monaten •via X (Twitter)

LIVE

Anya Rossi• Live Now

Private livecam show

2 Kommentare

Profilbild von Martin Ziqiao Ma
Martin Ziqiao Mavor 11 Monaten

Thanks to all of my amazing co-authors @ChenXuweiyi @shoubin621 @Sai__Bi @KaiZhang9546 @chenziwee @6SihanXu @jed_yang @zexiangxu Kalyan Sunkavalli @mohitban47 @SLED_AI @HaoTan5!♥️ Wrapped up an amazing year @AdobeResearch — beyond grateful for this risky journey from an NLP guy to a graphics amateur, and for the mentorship, support, and inspiration from some of the best minds in the field. Can’t thank the team enough! 🙏✨

Profilbild von Martin Ziqiao Ma
Martin Ziqiao Mavor 11 Monaten

@shoubin621 @Sai__Bi @KaiZhang9546 @chenziwee @6SihanXu @jed_yang @zexiangxu @mohitban47 @SLED_AI @HaoTan5 @AdobeResearch Also huge thanks to my academic collaborators @michigan_AI and @unc_ai_group — the exchange between academia and industry has been incredibly valuable. Grateful for the chance to learn from both worlds! 🙏

Ähnliche Videos

4DGT: Learning a 4D Gaussian Transformer Using Real-World Monocular Videos Abstract: We propose 4DGT, a 4D Gaussian-based Transformer model for dynamic scene reconstruction, trained entirely on real-world monocular posed videos. Using 4D Gaussian as an inductive bias, 4DGT unifies static and dynamic components, enabling the modeling of complex, time-varying environments with varying object lifespans. We introduced a novel density control strategy in training, which allows our 4DGT to handle longer space-time input while maintaining efficient rendering at runtime. Our model processes 64 consecutive posed frames in a rolling-window fashion, predicting consistent 4D Gaussians in the scene. Unlike optimization-based methods, 4DGT performs purely feed-forward inference, reducing reconstruction time from hours to seconds and scaling effectively to long video sequences. Trained only on large-scale monocular posed video datasets, 4DGT can significantly outperform prior Gaussian-based networks in real-world videos and achieve on-par accuracy with optimization-based methods on cross-domain videos.
5:19

4DGT: Learning a 4D Gaussian Transformer Using Real-World Monocular Videos Abstract: We propose 4DGT, a 4D Gaussian-based Transformer model for dynamic scene reconstruction, trained entirely on real-world monocular posed videos. Using 4D Gaussian as an inductive bias, 4DGT unifies static and dynamic components, enabling the modeling of complex, time-varying environments with varying object lifespans. We introduced a novel density control strategy in training, which allows our 4DGT to handle longer space-time input while maintaining efficient rendering at runtime. Our model processes 64 consecutive posed frames in a rolling-window fashion, predicting consistent 4D Gaussians in the scene. Unlike optimization-based methods, 4DGT performs purely feed-forward inference, reducing reconstruction time from hours to seconds and scaling effectively to long video sequences. Trained only on large-scale monocular posed video datasets, 4DGT can significantly outperform prior Gaussian-based networks in real-world videos and achieve on-par accuracy with optimization-based methods on cross-domain videos.

MrNeRF

34,782 Aufrufe • vor 1 Jahr

We’ve upgraded Stable Video Diffusion 4D to Stable Video 4D 2.0 (SV4D 2.0), improving the quality of 4D outputs generated from a single object-centric video. While 3D provides a static view of an object’s shape and size; 4D extends this by including time, showing how the object moves. This multi-view video diffusion model generates a 4D output in three steps: 1️⃣ Starts with an input video of a moving person or object 2️⃣ Generates novel views of the subject from unseen angles 3️⃣ Constructs a single dynamic 4D video output with spatial and temporal consistency You can learn more here: (1/4)
0:30

We’ve upgraded Stable Video Diffusion 4D to Stable Video 4D 2.0 (SV4D 2.0), improving the quality of 4D outputs generated from a single object-centric video. While 3D provides a static view of an object’s shape and size; 4D extends this by including time, showing how the object moves. This multi-view video diffusion model generates a 4D output in three steps: 1️⃣ Starts with an input video of a moving person or object 2️⃣ Generates novel views of the subject from unseen angles 3️⃣ Constructs a single dynamic 4D video output with spatial and temporal consistency You can learn more here: (1/4)

Stability AI

35,974 Aufrufe • vor 1 Jahr

[1/9] 🚀 We introduce 4Real-Video-V2, a method that can generate 4D scenes from a simple text prompt, viewable from any angle at any moment in time. It’s fast, photorealistic, and works on full scenes. Here's how it works and why it matters. 👇
0:56

[1/9] 🚀 We introduce 4Real-Video-V2, a method that can generate 4D scenes from a simple text prompt, viewable from any angle at any moment in time. It’s fast, photorealistic, and works on full scenes. Here's how it works and why it matters. 👇

Ashkan Mirzaei

10,012 Aufrufe • vor 11 Monaten

Our walkthrough video for the Certified GVNR NFT Mint is here. 🔹 Watch. 🔹 Learn. 🔹 Mint with ANY ASSET on ANY CHAIN. 🔹 Secure your $GVNR allocation. Available here It's your time. View now 👇
1:33

Our walkthrough video for the Certified GVNR NFT Mint is here. 🔹 Watch. 🔹 Learn. 🔹 Mint with ANY ASSET on ANY CHAIN. 🔹 Secure your $GVNR allocation. Available here It's your time. View now 👇

GVNR

184,238 Aufrufe • vor 1 Jahr

🤖 wake up, new 4d tool! Take any normal 2D video → instantly turn it into fully explorable 3D space. You can now orbit, tilt, and view the scene from angles that literally never existed in the original footage. This is 4D Gaussian Splatting (from 4dv) and it’s next-level.
0:22

🤖 wake up, new 4d tool! Take any normal 2D video → instantly turn it into fully explorable 3D space. You can now orbit, tilt, and view the scene from angles that literally never existed in the original footage. This is 4D Gaussian Splatting (from 4dv) and it’s next-level.

filipe

64,555 Aufrufe • vor 22 Tagen

📢Face Anything: 4D Face Reconstruction from Any Image Sequence Transformer model for 4D face reconstruction and dense tracking: - predict canonical facial coordinates per pixel - tracking as reconstruction in canonical space - geometry + correspondences in one forward pass Key idea: a shared canonical space across frames - correspondences as nearest neighbors - no motion or deformation estimation Stable geometry and tracking, even under large expressions and viewpoint changes - check out our results! 🌐 ▶️ Great work by Umut Kocasarı, Simon Giebenhain, Richard Shaw
0:42

📢Face Anything: 4D Face Reconstruction from Any Image Sequence Transformer model for 4D face reconstruction and dense tracking: - predict canonical facial coordinates per pixel - tracking as reconstruction in canonical space - geometry + correspondences in one forward pass Key idea: a shared canonical space across frames - correspondences as nearest neighbors - no motion or deformation estimation Stable geometry and tracking, even under large expressions and viewpoint changes - check out our results! 🌐 ▶️ Great work by Umut Kocasarı, Simon Giebenhain, Richard Shaw

Matthias Niessner

61,383 Aufrufe • vor 1 Monat

SceNeRFlow: Time-Consistent Reconstruction of General Dynamic Scenes abs: paper page: Existing methods for the 4D reconstruction of general, non-rigidly deforming objects focus on novel-view synthesis and neglect correspondences. However, time consistency enables advanced downstream tasks like 3D editing, motion analysis, or virtual-asset creation. We propose SceNeRFlow to reconstruct a general, non-rigid scene in a time-consistent manner. Our dynamic-NeRF method takes multi-view RGB videos and background images from static cameras with known camera parameters as input. It then reconstructs the deformations of an estimated canonical model of the geometry and appearance in an online fashion. Since this canonical model is time-invariant, we obtain correspondences even for long-term, long-range motions. We employ neural scene representations to parametrize the components of our method. Like prior dynamic-NeRF methods, we use a backwards deformation model. We find non-trivial adaptations of this model necessary to handle larger motions: We decompose the deformations into a strongly regularized coarse component and a weakly regularized fine component, where the coarse component also extends the deformation field into the space surrounding the object, which enables tracking over time. We show experimentally that, unlike prior work that only handles small motion, our method enables the reconstruction of studio-scale motions.
0:26

SceNeRFlow: Time-Consistent Reconstruction of General Dynamic Scenes abs: paper page: Existing methods for the 4D reconstruction of general, non-rigidly deforming objects focus on novel-view synthesis and neglect correspondences. However, time consistency enables advanced downstream tasks like 3D editing, motion analysis, or virtual-asset creation. We propose SceNeRFlow to reconstruct a general, non-rigid scene in a time-consistent manner. Our dynamic-NeRF method takes multi-view RGB videos and background images from static cameras with known camera parameters as input. It then reconstructs the deformations of an estimated canonical model of the geometry and appearance in an online fashion. Since this canonical model is time-invariant, we obtain correspondences even for long-term, long-range motions. We employ neural scene representations to parametrize the components of our method. Like prior dynamic-NeRF methods, we use a backwards deformation model. We find non-trivial adaptations of this model necessary to handle larger motions: We decompose the deformations into a strongly regularized coarse component and a weakly regularized fine component, where the coarse component also extends the deformation field into the space surrounding the object, which enables tracking over time. We show experimentally that, unlike prior work that only handles small motion, our method enables the reconstruction of studio-scale motions.

AK

76,380 Aufrufe • vor 2 Jahren

4D LangSplat: 4D Language Gaussian Splatting via Multimodal Large Language Models Contributions: • We introduce 4D LangSplat for open-vocabulary 4D spatial-temporal queries. To the best of our knowledge, we are the first to construct 4D language fields with object textual captions generated by MLLMs. • To model smooth transitions across states for objects in 4D scenes, we propose a status deformable network to capture continuous temporal changes. • Experiential results show that our method attains state-of-the-art performance for both time-agnostic and time-sensitive open-vocabulary queries.
0:29

4D LangSplat: 4D Language Gaussian Splatting via Multimodal Large Language Models Contributions: • We introduce 4D LangSplat for open-vocabulary 4D spatial-temporal queries. To the best of our knowledge, we are the first to construct 4D language fields with object textual captions generated by MLLMs. • To model smooth transitions across states for objects in 4D scenes, we propose a status deformable network to capture continuous temporal changes. • Experiential results show that our method attains state-of-the-art performance for both time-agnostic and time-sensitive open-vocabulary queries.

MrNeRF

10,953 Aufrufe • vor 1 Jahr

Synthesia research releases HumanRF: High-Fidelity Neural Radiance Fields for Humans in Motion a 4D dynamic neural scene representation that captures full-body appearance in motion from multi-view video input, and enables playback from novel, unseen viewpoints project page: paper page:
3:06

Synthesia research releases HumanRF: High-Fidelity Neural Radiance Fields for Humans in Motion a 4D dynamic neural scene representation that captures full-body appearance in motion from multi-view video input, and enables playback from novel, unseen viewpoints project page: paper page:

AK

160,700 Aufrufe • vor 3 Jahren

Robot Learning needs 4D world models! Robot Learning needs 4D world models! Robot Learning needs 4D world models! We introduce TesserAct, a 4D embodied world model that can simulate how agents interact with the 3D world over time! We achieve this by simply extending a pre-trained 2D video generation model to jointly predict RGB, depth, and surface normals. It enables: 1️⃣ Much better policy learning in the wild 2️⃣ Temporal + spatial coherence in 4D dynamic prediction 3️⃣ Novel view synthesis for embodied scenes Code: Paper Link: Project page:
0:32

Robot Learning needs 4D world models! Robot Learning needs 4D world models! Robot Learning needs 4D world models! We introduce TesserAct, a 4D embodied world model that can simulate how agents interact with the 3D world over time! We achieve this by simply extending a pre-trained 2D video generation model to jointly predict RGB, depth, and surface normals. It enables: 1️⃣ Much better policy learning in the wild 2️⃣ Temporal + spatial coherence in 4D dynamic prediction 3️⃣ Novel view synthesis for embodied scenes Code: Paper Link: Project page:

Chuang Gan

43,265 Aufrufe • vor 1 Jahr

Ever wondered how roses grow and wither in your backyard?🌹 Our latest work on generating 4D temporal object intrinsics lets you explore a rose's entire lifecycle—from birth to death—under any environment light, from any viewpoint, at any moment. Project page: (🧵 1/7)
0:33

Ever wondered how roses grow and wither in your backyard?🌹 Our latest work on generating 4D temporal object intrinsics lets you explore a rose's entire lifecycle—from birth to death—under any environment light, from any viewpoint, at any moment. Project page: (🧵 1/7)

Chen Geng

35,649 Aufrufe • vor 1 Jahr

🎉Excited to partner with Scroll to bring Chain Abstraction to Scroll apps! 📜 Here’s what this unlocks for dApps and users: 🔹Ability to spend your USDC, USDT, and ETH balances from any chain directly on Scroll, without bridging. 🔹A native multi-chain experience that works seamlessly with any wallet like Metamask, Rabby, Arcana, etc. 🔹Freedom to pay gas fees in USDC/USDT without worrying about native gas tokens. Enable Chain Abstraction in your app with our SDK: 👉Try it: 👉Docs: Learn more:
3:47

🎉Excited to partner with Scroll to bring Chain Abstraction to Scroll apps! 📜 Here’s what this unlocks for dApps and users: 🔹Ability to spend your USDC, USDT, and ETH balances from any chain directly on Scroll, without bridging. 🔹A native multi-chain experience that works seamlessly with any wallet like Metamask, Rabby, Arcana, etc. 🔹Freedom to pay gas fees in USDC/USDT without worrying about native gas tokens. Enable Chain Abstraction in your app with our SDK: 👉Try it: 👉Docs: Learn more:

Arcana Network

30,911 Aufrufe • vor 1 Jahr

Most multi-view reconstruction models need full supervision. We show they can self-improve without any ground truth labels. Introducing SelfEvo: Self-Improving 4D Perception via Self-Distillation. Up to +36.5% in video depth, +20.1% in camera estimation, zero annotation.
0:45

Most multi-view reconstruction models need full supervision. We show they can self-improve without any ground truth labels. Introducing SelfEvo: Self-Improving 4D Perception via Self-Distillation. Up to +36.5% in video depth, +20.1% in camera estimation, zero annotation.

Qianqian Wang

24,309 Aufrufe • vor 2 Monaten

Fast View Synthesis of Casual Videos paper page: Novel view synthesis from an in-the-wild video is difficult due to challenges like scene dynamics and lack of parallax. While existing methods have shown promising results with implicit neural radiance fields, they are slow to train and render. This paper revisits explicit video representations to synthesize high-quality novel views from a monocular video efficiently. We treat static and dynamic video content separately. Specifically, we build a global static scene model using an extended plane-based scene representation to synthesize temporally coherent novel video. Our plane-based scene representation is augmented with spherical harmonics and displacement maps to capture view-dependent effects and model non-planar complex surface geometry. We opt to represent the dynamic content as per-frame point clouds for efficiency. While such representations are inconsistency-prone, minor temporal inconsistencies are perceptually masked due to motion. We develop a method to quickly estimate such a hybrid video representation and render novel views in real time. Our experiments show that our method can render high-quality novel views from an in-the-wild video with comparable quality to state-of-the-art methods while being 100x faster in training and enabling real-time rendering.
0:24

Fast View Synthesis of Casual Videos paper page: Novel view synthesis from an in-the-wild video is difficult due to challenges like scene dynamics and lack of parallax. While existing methods have shown promising results with implicit neural radiance fields, they are slow to train and render. This paper revisits explicit video representations to synthesize high-quality novel views from a monocular video efficiently. We treat static and dynamic video content separately. Specifically, we build a global static scene model using an extended plane-based scene representation to synthesize temporally coherent novel video. Our plane-based scene representation is augmented with spherical harmonics and displacement maps to capture view-dependent effects and model non-planar complex surface geometry. We opt to represent the dynamic content as per-frame point clouds for efficiency. While such representations are inconsistency-prone, minor temporal inconsistencies are perceptually masked due to motion. We develop a method to quickly estimate such a hybrid video representation and render novel views in real time. Our experiments show that our method can render high-quality novel views from an in-the-wild video with comparable quality to state-of-the-art methods while being 100x faster in training and enabling real-time rendering.

AK

20,651 Aufrufe • vor 2 Jahren

MeshLRM Large Reconstruction Model for High-Quality Mesh We propose MeshLRM, a novel LRM-based approach that can reconstruct a high-quality mesh from merely four input images in less than one second. Different from previous large reconstruction models (LRMs) that focus on
0:25

MeshLRM Large Reconstruction Model for High-Quality Mesh We propose MeshLRM, a novel LRM-based approach that can reconstruct a high-quality mesh from merely four input images in less than one second. Different from previous large reconstruction models (LRMs) that focus on

AK

69,111 Aufrufe • vor 2 Jahren

🔹 Untold stories from Ethan Laird 🔹 Remembering our Irish stars 🔹 A social view from Admin's Corner It's Thursday. It's time for Blue Rising 🍿
0:23

🔹 Untold stories from Ethan Laird 🔹 Remembering our Irish stars 🔹 A social view from Admin's Corner It's Thursday. It's time for Blue Rising 🍿

Birmingham City FC

11,404 Aufrufe • vor 2 Monaten

We're helping AI to see the 3D world in motion as humans do. 🌐 Enter D4RT: a unified model that turns video into 4D representations faster than previous methods - enabling it to understand space and time. This is how it works 🧵
0:29

We're helping AI to see the 3D world in motion as humans do. 🌐 Enter D4RT: a unified model that turns video into 4D representations faster than previous methods - enabling it to understand space and time. This is how it works 🧵

Google DeepMind

368,824 Aufrufe • vor 4 Monaten

A lot of prophecy has been fulfilled, and the time is near; Jesus can come at any time, any day. We need to walk away from evil.
1:25

A lot of prophecy has been fulfilled, and the time is near; Jesus can come at any time, any day. We need to walk away from evil.

Man of God, Bishop Mari Emmanuel.

17,678 Aufrufe • vor 1 Jahr

⏰ we brought time travel to replit! 1. see a global history of all your checkpoints 2. view a fully interactive version of your app at any point in that history 3. rollback and rollforward to any point in history (with your database data and schema, h/t Neon Postgres)
1:17

⏰ we brought time travel to replit! 1. see a global history of all your checkpoints 2. view a fully interactive version of your app at any point in that history 3. rollback and rollforward to any point in history (with your database data and schema, h/t Neon Postgres)

Jacky

161,672 Aufrufe • vor 1 Jahr

🚀 Xaman 4.0 is here! A major update with: 🔹 Seamless swaps – any asset, any time 🔹 10x faster onboarding – start in seconds 🔹 Blazing performance 🔹 Opt-in push notifications – stay updated instantly 🔹 And much more! Watch now ⬇️
0:31

🚀 Xaman 4.0 is here! A major update with: 🔹 Seamless swaps – any asset, any time 🔹 10x faster onboarding – start in seconds 🔹 Blazing performance 🔹 Opt-in push notifications – stay updated instantly 🔹 And much more! Watch now ⬇️

Xaman® Wallet 🪝

219,601 Aufrufe • vor 1 Jahr