Check out MultiDiff #CVPR2024! From a single RGB image,... MultiDiff enables scene-level novel view synthesis with free camera control. Great work by @normanisation Katja Schwarz Barbara Roessle, L Porzi, S Rota Bulò, P Kontschiedershow more

Wonderland: Navigating 3D Scenes from a Single Image Contributions:... • First, we introduce a representation for controllable 3D generation by leveraging the generative priors from camera-guided video diffusion models. Unlike image models, video diffusion models are trained on extensive video datasets. This enables them to capture comprehensive spatial relationships within scenes across multiple views and embed a form of "3D awareness" in their latent space, which allows us to maintain 3D consistency in novel view synthesis. • Second, to achieve controllable novel view generation, we empower video models with precise control over specified camera motions. We introduce a novel dual-branch conditioning mechanism that effectively incorporates desired diverse camera trajectories into the video diffusion model. This enables expansion of a single image into a multi-view consistent capture of a 3D scene with precise pose control. • Third, to achieve efficient 3D reconstruction, we directly transform video latents into 3DGS. We propose a novel latent-based large reconstruction model (LaLRM) that lifts video latents to 3D in a feed-forward manner. With this design, during inference, our model directly predicts 3DGS from a single input image, effectively aligning the generation and reconstruction tasks—and bridging image space and 3D space—through the video latent space. Compared with reconstructing scenes from images, the video latent space offers a 256× spatial-temporal reduction while retaining essential and consistent 3D structural details. Such a high degree of compression is crucial, as it allows the LaLRM to handle a wider range of 3D scenes within the reconstruction framework, with the same memory constraints.show more

We're releasing OVIE, a novel view generation model trained... entirely on single images. No multi-view datasets needed. Given a single image, it generates novel views of any scene in real time, running orders of magnitude faster than competing approaches.show more

Check out TriPlaneNet! From a single image, we predict... EG3D latents & offsets, thus obtaining high-fidelity 3D models. Works in real time! Great work by our stellar MA student Ananta R. Bhattarai advised by Artem Sevastopolsky #WACV2024show more

Bai et al., "Positional Encoding Field" Make your RoPE... encoding 3D by including a z axis, then manipulate your image by simply manipulating your positional encoding in 3D --> novel view synthesis. Neat idea.show more

Accepted at #ECCV2024 🇮🇹 See you in Milan! PolyOculus:... Simultaneous Multi-view Image-based Novel View Synthesis Jason Yu Tristan Aumentado-Armstrong Fereshteh Forghani Marcus Brubaker and me Paper:show more

ReViV reconstructs viewer-centric human motion (body, hand, and gaze)... and view-centric scene geometry (camera and depth) from a single egocentric RGB video in a unified feed-forward model. It formulates the task as learning the full joint probability distribution over multimodal signals, including RGB video, camera trajectory, gaze direction, full-body motion, hand motion, and depth. Powered by a Masked Generative Egocentric Transformer, ReViV operates within a single feed-forward architecture to simultaneously reconstruct the temporally consistent 4D reconstruction across the viewer and the view with fast inference speed. Paper Title: ReViV: Reconstructing the Viewer and the View in 4D from Monocular Project: Link:show more

I am blown away 🤯. Check this out! CameraCtrl... II: Dynamic Scene Exploration via Camera-controlled Video Diffusion Models TL;DR: "To enable broader exploration of dynamic scenes, our model can generate new video clips of the same scene based on previously generated content and user-provided camera trajectories. This approach maintains dynamic capabilities, accurate camera control, and scene consistency throughout the extended exploration." "Our model enables precise camera control across diverse scenarios while preserving dynamic scene elements, e.g." "Our method can generate videos with strong 3D consistency, which enables high-quality 3D reconstruction using the camera-controlled videos." Contributions: 1) A systematic data curation pipeline for constructing a dynamic video dataset with camera trajectory annotations; 2) A lightweight camera control injection module and corresponding training strategy that preserves dynamic video generation capabilities while adding camera control effect; 3) A clip-wise autoregressive generation recipe that enables extended range exploration of generated scenes.show more

Super excited to announce our new work: Synthesizing Moving... People with 3D Control (3DHM)💡 Why is 3DHM unique? With 3D Control, 3DHM can animate a 𝗿𝗮𝗻𝗱𝗼𝗺 human photo with 𝗮𝗻𝘆 poses in a 𝟯𝟲𝟬-𝗱𝗲𝗴𝗿𝗲𝗲 camera view and 𝗮𝗻𝘆 camera azimuths from 𝗮𝗻𝘆 video!show more

Create anime from a single image. With Gen-4 and... ChatGPT, you can turn any image into a multi-scene cartoon. Here’s how 👇show more

A single image. An entire 3D world. And every... important object is editable. Hyper3D WorldGen turns a single scene image into a structured 3D environment with: - Independent assets - Spatial relationships - Physical properties - The freedom to move, replace, refine, and interact with them This is scene-level 3D generation. 🧵👇show more

(1/3) Can we turn text-to-image models into photorealistic 3D... generators? ViewDiff (#CVPR2024) produces realistic, multi-view consistent images of real-world 3D objects in authentic surroundings. Website Video How does it work?show more

AI is so trippy. This entire scene, including the... exterior, was created from a single image of the ovens in the interior. This is a multi-view reconstruction, not a video. No v2v used. Prompt image in reply.show more

A scene from a novel?

DimensionX: Create Any 3D and 4D Scenes from a... Single Image with Controllable Video Diffusion TL;DR: Create 3/4DGS from Video Diffusion Note: Some first inference code released (not all yet). Contributions (cited): • We present DimensionX, a novel framework for generating photorealistic 3D and 4D scenes from only a single image using controllable video diffusion. • We propose ST-Director, which decouples the spatial and temporal priors in video diffusion models by learning (spatial and temporal) dimension-aware modules with our curated datasets. We further enhance the hybriddimension control with a training-free composition approach according to the essence of video diffusion denoising process. • To bridge the gap between video diffusion and real-world scenes, we design a trajectory-aware mechanism for 3D generation and an identity-preserving denoising approach for 4D generation, enabling more realistic and controllable scene synthesis. • Extensive experiments manifest that our DimensionX delivers superior performance in video, 3D, and 4D generation compared with baseline methods.show more

WildGS-SLAM: Monocular Gaussian Splatting SLAM in Dynamic Environments TL;DR:... We present WildGS-SLAM, a robust monocular RGB SLAM system. - Utilizes uncertainty-aware tracking and mapping to handle dynamic scenes. - Leverages DINOv2-based uncertainty maps for dynamic object removal. - Improves tracking and mapping. - Enables high-quality view synthesis.show more

Introducing Latent Labs - a Text to 360 Image... Generator! Type in a prompt, generate, and control the camera within the image. Free to try at: ... #AIart #AIArtwork #AIArtworksshow more

MiniMax H3 Max Multi-Angle is now on Venice. Move... the camera around a single reference image with precise orbit, elevation, and distance while preserving the scene. Available privately.show more