正在加载视频...

视频加载失败

加载此视频时出现问题。这可能是由于临时网络问题，或视频可能不可用。

Check out MultiDiff #CVPR2024! From a single RGB image, MultiDiff enables scene-level novel view synthesis with free camera control. Great work by @normanisation Katja Schwarz Barbara Roessle, L Porzi, S Rota Bulò, P Kontschieder

Matthias Niessner

47,444 subscribers

39,053 次观看 • 1 年前 •via X (Twitter)

教育科学技术新闻政治 #CVPR2024

Anya Rossi• Live Now

Private livecam show

0 条评论

暂无评论

原始帖子的评论将显示在这里

相关视频

Wonderland: Navigating 3D Scenes from a Single Image Contributions: • First, we introduce a representation for controllable 3D generation by leveraging the generative priors from camera-guided video diffusion models. Unlike image models, video diffusion models are trained on extensive video datasets. This enables them to capture comprehensive spatial relationships within scenes across multiple views and embed a form of "3D awareness" in their latent space, which allows us to maintain 3D consistency in novel view synthesis. • Second, to achieve controllable novel view generation, we empower video models with precise control over specified camera motions. We introduce a novel dual-branch conditioning mechanism that effectively incorporates desired diverse camera trajectories into the video diffusion model. This enables expansion of a single image into a multi-view consistent capture of a 3D scene with precise pose control. • Third, to achieve efficient 3D reconstruction, we directly transform video latents into 3DGS. We propose a novel latent-based large reconstruction model (LaLRM) that lifts video latents to 3D in a feed-forward manner. With this design, during inference, our model directly predicts 3DGS from a single input image, effectively aligning the generation and reconstruction tasks—and bridging image space and 3D space—through the video latent space. Compared with reconstructing scenes from images, the video latent space offers a 256× spatial-temporal reduction while retaining essential and consistent 3D structural details. Such a high degree of compression is crucial, as it allows the LaLRM to handle a wider range of 3D scenes within the reconstruction framework, with the same memory constraints.

Wonderland: Navigating 3D Scenes from a Single Image Contributions: • First, we introduce a representation for controllable 3D generation by leveraging the generative priors from camera-guided video diffusion models. Unlike image models, video diffusion models are trained on extensive video datasets. This enables them to capture comprehensive spatial relationships within scenes across multiple views and embed a form of "3D awareness" in their latent space, which allows us to maintain 3D consistency in novel view synthesis. • Second, to achieve controllable novel view generation, we empower video models with precise control over specified camera motions. We introduce a novel dual-branch conditioning mechanism that effectively incorporates desired diverse camera trajectories into the video diffusion model. This enables expansion of a single image into a multi-view consistent capture of a 3D scene with precise pose control. • Third, to achieve efficient 3D reconstruction, we directly transform video latents into 3DGS. We propose a novel latent-based large reconstruction model (LaLRM) that lifts video latents to 3D in a feed-forward manner. With this design, during inference, our model directly predicts 3DGS from a single input image, effectively aligning the generation and reconstruction tasks—and bridging image space and 3D space—through the video latent space. Compared with reconstructing scenes from images, the video latent space offers a 256× spatial-temporal reduction while retaining essential and consistent 3D structural details. Such a high degree of compression is crucial, as it allows the LaLRM to handle a wider range of 3D scenes within the reconstruction framework, with the same memory constraints.

MrNeRF

52,801 次观看 • 1 年前

We're releasing OVIE, a novel view generation model trained entirely on single images. No multi-view datasets needed. Given a single image, it generates novel views of any scene in real time, running orders of magnitude faster than competing approaches.

We're releasing OVIE, a novel view generation model trained entirely on single images. No multi-view datasets needed. Given a single image, it generates novel views of any scene in real time, running orders of magnitude faster than competing approaches.

kyutai

30,957 次观看 • 2 个月前

Bai et al., "Positional Encoding Field" Make your RoPE encoding 3D by including a z axis, then manipulate your image by simply manipulating your positional encoding in 3D --> novel view synthesis. Neat idea.

Bai et al., "Positional Encoding Field" Make your RoPE encoding 3D by including a z axis, then manipulate your image by simply manipulating your positional encoding in 3D --> novel view synthesis. Neat idea.

Kwang Moo Yi

46,855 次观看 • 7 个月前

Accepted at #ECCV2024 🇮🇹 See you in Milan! PolyOculus: Simultaneous Multi-view Image-based Novel View Synthesis Jason Yu Tristan Aumentado-Armstrong Fereshteh Forghani Marcus Brubaker and me Paper:

Accepted at #ECCV2024 🇮🇹 See you in Milan! PolyOculus: Simultaneous Multi-view Image-based Novel View Synthesis Jason Yu Tristan Aumentado-Armstrong Fereshteh Forghani Marcus Brubaker and me Paper:

Kosta Derpanis (sabbatical in Munich)

28,772 次观看 • 1 年前

Pixel3DMM is a new method that can reconstruct 3D human faces from a single RGB image or multiple frames

Pixel3DMM is a new method that can reconstruct 3D human faces from a single RGB image or multiple frames

Dreaming Tulpa 🥓👑

32,565 次观看 • 1 年前

RIP Ken Burns effect? Post camera moves will never be the same. Convert and image or video into a 3D scene and animate a virtual camera to create a new move. Camera control by Moonvalley

RIP Ken Burns effect? Post camera moves will never be the same. Convert and image or video into a 3D scene and animate a virtual camera to create a new move. Camera control by Moonvalley

Paul Trillo

13,716 次观看 • 11 个月前

Super excited to announce our new work: Synthesizing Moving People with 3D Control (3DHM)💡 Why is 3DHM unique? With 3D Control, 3DHM can animate a 𝗿𝗮𝗻𝗱𝗼𝗺 human photo with 𝗮𝗻𝘆 poses in a 𝟯𝟲𝟬-𝗱𝗲𝗴𝗿𝗲𝗲 camera view and 𝗮𝗻𝘆 camera azimuths from 𝗮𝗻𝘆 video!

Super excited to announce our new work: Synthesizing Moving People with 3D Control (3DHM)💡 Why is 3DHM unique? With 3D Control, 3DHM can animate a 𝗿𝗮𝗻𝗱𝗼𝗺 human photo with 𝗮𝗻𝘆 poses in a 𝟯𝟲𝟬-𝗱𝗲𝗴𝗿𝗲𝗲 camera view and 𝗮𝗻𝘆 camera azimuths from 𝗮𝗻𝘆 video!

Boyi Li

74,410 次观看 • 2 年前

Create anime from a single image. With Gen-4 and ChatGPT, you can turn any image into a multi-scene cartoon. Here’s how 👇

Create anime from a single image. With Gen-4 and ChatGPT, you can turn any image into a multi-scene cartoon. Here’s how 👇

Framer 🇱🇹

779,106 次观看 • 1 年前

(1/3) Can we turn text-to-image models into photorealistic 3D generators? ViewDiff (#CVPR2024) produces realistic, multi-view consistent images of real-world 3D objects in authentic surroundings. Website Video How does it work?

(1/3) Can we turn text-to-image models into photorealistic 3D generators? ViewDiff (#CVPR2024) produces realistic, multi-view consistent images of real-world 3D objects in authentic surroundings. Website Video How does it work?

Matthias Niessner

34,749 次观看 • 2 年前

SplatVoxel: History-Aware Novel View Streaming without Temporal Training Contributions: • We propose a hybrid Splat-Voxel feed-forward reconstruction framework that leverages historical information to enable novel view streaming, without relying on multi-view video datasets for training. • We develop an efficient sparse voxel transformer with a coarse-to-fine voxel representation, outperforming existing feed-forward Gaussian splatting methods. • Experiment results demonstrate that our proposed framework enhances novel view synthesis for streaming scene reconstruction, providing better visual quality and reduced temporal artifacts through history-aware modeling.

SplatVoxel: History-Aware Novel View Streaming without Temporal Training Contributions: • We propose a hybrid Splat-Voxel feed-forward reconstruction framework that leverages historical information to enable novel view streaming, without relying on multi-view video datasets for training. • We develop an efficient sparse voxel transformer with a coarse-to-fine voxel representation, outperforming existing feed-forward Gaussian splatting methods. • Experiment results demonstrate that our proposed framework enhances novel view synthesis for streaming scene reconstruction, providing better visual quality and reduced temporal artifacts through history-aware modeling.

MrNeRF

10,823 次观看 • 1 年前

AI is so trippy. This entire scene, including the exterior, was created from a single image of the ovens in the interior. This is a multi-view reconstruction, not a video. No v2v used. Prompt image in reply.

AI is so trippy. This entire scene, including the exterior, was created from a single image of the ovens in the interior. This is a multi-view reconstruction, not a video. No v2v used. Prompt image in reply.

martin_casado

54,363 次观看 • 10 个月前

DimensionX: Create Any 3D and 4D Scenes from a Single Image with Controllable Video Diffusion TL;DR: Create 3/4DGS from Video Diffusion Note: Some first inference code released (not all yet). Contributions (cited): • We present DimensionX, a novel framework for generating photorealistic 3D and 4D scenes from only a single image using controllable video diffusion. • We propose ST-Director, which decouples the spatial and temporal priors in video diffusion models by learning (spatial and temporal) dimension-aware modules with our curated datasets. We further enhance the hybriddimension control with a training-free composition approach according to the essence of video diffusion denoising process. • To bridge the gap between video diffusion and real-world scenes, we design a trajectory-aware mechanism for 3D generation and an identity-preserving denoising approach for 4D generation, enabling more realistic and controllable scene synthesis. • Extensive experiments manifest that our DimensionX delivers superior performance in video, 3D, and 4D generation compared with baseline methods.

DimensionX: Create Any 3D and 4D Scenes from a Single Image with Controllable Video Diffusion TL;DR: Create 3/4DGS from Video Diffusion Note: Some first inference code released (not all yet). Contributions (cited): • We present DimensionX, a novel framework for generating photorealistic 3D and 4D scenes from only a single image using controllable video diffusion. • We propose ST-Director, which decouples the spatial and temporal priors in video diffusion models by learning (spatial and temporal) dimension-aware modules with our curated datasets. We further enhance the hybriddimension control with a training-free composition approach according to the essence of video diffusion denoising process. • To bridge the gap between video diffusion and real-world scenes, we design a trajectory-aware mechanism for 3D generation and an identity-preserving denoising approach for 4D generation, enabling more realistic and controllable scene synthesis. • Extensive experiments manifest that our DimensionX delivers superior performance in video, 3D, and 4D generation compared with baseline methods.

MrNeRF

17,028 次观看 • 1 年前

WildGS-SLAM: Monocular Gaussian Splatting SLAM in Dynamic Environments TL;DR: We present WildGS-SLAM, a robust monocular RGB SLAM system. - Utilizes uncertainty-aware tracking and mapping to handle dynamic scenes. - Leverages DINOv2-based uncertainty maps for dynamic object removal. - Improves tracking and mapping. - Enables high-quality view synthesis.

WildGS-SLAM: Monocular Gaussian Splatting SLAM in Dynamic Environments TL;DR: We present WildGS-SLAM, a robust monocular RGB SLAM system. - Utilizes uncertainty-aware tracking and mapping to handle dynamic scenes. - Leverages DINOv2-based uncertainty maps for dynamic object removal. - Improves tracking and mapping. - Enables high-quality view synthesis.

MrNeRF

17,382 次观看 • 1 年前

Here’s how to create this cool 3D scene from a single image! 📸 Midjourney (isometric image generation) Trellis (Image to 3D Gaussian Splat) Browser Lab (3D Editor Splat Import)

Here’s how to create this cool 3D scene from a single image! 📸 Midjourney (isometric image generation) Trellis (Image to 3D Gaussian Splat) Browser Lab (3D Editor Splat Import)

Ian Curtis

117,646 次观看 • 1 年前

This AI 3D tool is on another level. A dragon head and a dungeon NPC. Both from a single image. Here is how:

This AI 3D tool is on another level. A dragon head and a dungeon NPC. Both from a single image. Here is how:

AI Highlight

141,349 次观看 • 1 天前

U P W A R D S P I R A L

U P W A R D S P I R A L

Angelo฿TC

113,177 次观看 • 2 年前

The camera motion in Imagine is insanely next-level...woah! I've been exploring a few prompts with the token 'image sequence'

The camera motion in Imagine is insanely next-level...woah! I've been exploring a few prompts with the token 'image sequence'

Kiri

390,472 次观看 • 4 个月前

Seedance 2.0 is insane. It created a full anime edit from a single prompt. Camera work, timing, effects, impact — everything feels finished, like a professional edit. The level this AI has reached is unreal.

Seedance 2.0 is insane. It created a full anime edit from a single prompt. Camera work, timing, effects, impact — everything feels finished, like a professional edit. The level this AI has reached is unreal.

nachos2d

25,098 次观看 • 4 个月前

Check out this view of the Bank from Saturday’s flyover by the MD National Guard!

Check out this view of the Bank from Saturday’s flyover by the MD National Guard!

Baltimore Ravens

53,587 次观看 • 1 年前

R A W S A L E R T S A better view.

R A W S A L E R T S A better view.

Jon Nicosia

26,311 次观看 • 1 年前