#ECCV24 We introduce Controllable Human-Object Interaction Synthesis (CHOIS) to... generate interactions given text in a 3D scene. w alex, Roozbeh Mottaghi Jiajun Wu Xavier Puig Karen. code: project page:show more

Checkout ObjectForesight! We tackle the problem of predicting future... 3D object trajectories given a past context, purely from human videos of everyday interactions! [1/n]show more

Excited to present GenZI at #CVPR2024! Lei Li introduces... GenZI, the first zero-shot approach to creating realistic 3D human-scene interactions by leveraging interaction priors from large VLMs. Code and data on our website!show more

If you’re not in freefall, you’re likely contacting something.... Yet 3D human-object interaction (HOI) reconstruction remains underexplored. PICO (#CVPR2025) recovers humans 🏃‍♂️, objects 🏓, and their interactions 👉🍎- all in 3D, from just a single internet image. 1/11show more

TokenHSI is out on Hugging Face Unified Synthesis of... Physical Human-Scene Interactions through Task Tokenizationshow more

🛠Made a tool for creators That's Generate 3D text... in secs..show more

We have added a demo Colab notebook for Neuralangelo... 🗿🏛! The entire package -- setup, training, and 3D visualization -- all in your browser 😁 full code: project page:show more

WeatherEdit: Controllable Weather Editing with 4D Gaussian Field Contributions:... 1. Based on our analysis of weather editing characteristics, we introduce WeatherEdit, a comprehensive and efficient framework for realistic and controllable weather generation. Compared with existing methods that focus on either background editing or static weather effects, a progressive 2D-to-4D transformation process in WeatherEdit enhances adaptability across a wider range of scenarios. 2. We introduce an all-in-one adapter to enable a diffusion model for multi-weather (snowy, rainy, and fog) synthesis, along with a Temporal-View attention to ensure consistent editing across multi-frame and multi-view. 3. We design a 4D Gaussian field for weather particle modeling, enabling plausible simulation of raindrops, snowflakes, and fog with controllable severity. 4. We demonstrate WeatherEdit’s effectiveness in generating realistic, consistent, and controllable weather effects in 3D driving scenes, showcasing its applicability to real-world scenarios.show more

DreamPose: Fashion Image-to-Video Synthesis via Stable Diffusion abs: project... page: github:show more

🎶Time to start NeRFing audio. Introducing NVAS3D for novel... acoustic synthesis in 3D scenes. Watch with AirPods. arxiv: code:show more

DimensionX: Create Any 3D and 4D Scenes from a... Single Image with Controllable Video Diffusion TL;DR: Create 3/4DGS from Video Diffusion Note: Some first inference code released (not all yet). Contributions (cited): • We present DimensionX, a novel framework for generating photorealistic 3D and 4D scenes from only a single image using controllable video diffusion. • We propose ST-Director, which decouples the spatial and temporal priors in video diffusion models by learning (spatial and temporal) dimension-aware modules with our curated datasets. We further enhance the hybriddimension control with a training-free composition approach according to the essence of video diffusion denoising process. • To bridge the gap between video diffusion and real-world scenes, we design a trajectory-aware mechanism for 3D generation and an identity-preserving denoising approach for 4D generation, enabling more realistic and controllable scene synthesis. • Extensive experiments manifest that our DimensionX delivers superior performance in video, 3D, and 4D generation compared with baseline methods.show more

We introduce TurboEdit -- simple text-based image editing in... 1/2 sec! We leverage few-step diffusion models, mapping real images into noise with an encoder. Please see our #ECCV2024 paper: Work w/ Zongze Wu, Nick Kolkin, Jon Brandt, Eli Shechtman 1/show more

UPDATE: Spline's 3D Code API for SwiftUI is GA!... Create custom 3D interactions for your iOS/macOS projects. - Trigger events and transitions - Change object properties - Update variables - and more! Try it now.show more

LLMs, trained only on text, might already know more... about other modalities than we realized; we just need to find ways elicit it. project page: w/ Phillip Isola and Brian Cheungshow more

📢We introduce “RefFusion”, a novel inpainting method for scenes... reconstructed using 3D Gaussian Splatting. 🔗 TLDR: we personalize an image diffusion model to a given reference image and distill its knowledge to 3D through score distillation sampling.show more

📢🚗✨ Excited to announce InfiniCube, our scalable generative model... for dynamic 3D driving scene generation with high fidelity and controllability! InfiniCube generates very large-scale (300m×400m ~ 100,000m^2), dynamic 3D driving scenes given HD maps, 3D bounding boxes, and text prompts as controls. Website: Props to Yifan Lu for leading this project!!!! [1/N]show more

Wonderland: Navigating 3D Scenes from a Single Image Contributions:... • First, we introduce a representation for controllable 3D generation by leveraging the generative priors from camera-guided video diffusion models. Unlike image models, video diffusion models are trained on extensive video datasets. This enables them to capture comprehensive spatial relationships within scenes across multiple views and embed a form of "3D awareness" in their latent space, which allows us to maintain 3D consistency in novel view synthesis. • Second, to achieve controllable novel view generation, we empower video models with precise control over specified camera motions. We introduce a novel dual-branch conditioning mechanism that effectively incorporates desired diverse camera trajectories into the video diffusion model. This enables expansion of a single image into a multi-view consistent capture of a 3D scene with precise pose control. • Third, to achieve efficient 3D reconstruction, we directly transform video latents into 3DGS. We propose a novel latent-based large reconstruction model (LaLRM) that lifts video latents to 3D in a feed-forward manner. With this design, during inference, our model directly predicts 3DGS from a single input image, effectively aligning the generation and reconstruction tasks—and bridging image space and 3D space—through the video latent space. Compared with reconstructing scenes from images, the video latent space offers a 256× spatial-temporal reduction while retaining essential and consistent 3D structural details. Such a high degree of compression is crucial, as it allows the LaLRM to handle a wider range of 3D scenes within the reconstruction framework, with the same memory constraints.show more

We are presenting **SAD** (Segment Any RGBD): SAD is... able to perform 3D segmentation (segment out any 3D object) with RGBD inputs (or rendered depth images only). - Code: - Demo Hugging Face:show more

Humans are social creatures and physical interaction with other... humans plays an important role in our everyday life. In this thread, we explain how such interactions can be reconstructed in 3D from a single image utilising a denoising diffusion probabilistic model. (1/n)show more

RGB Human Videos -> Dexterous Manipulation Policies No teleoperation.... No wearables. No robot interaction. The 🔑: Reliable 3D hand-object trajectory reconstruction and efficient action augmentation Fun collaboration led by Hongyi Chenshow more