📢MeshPad: Interactive Sketch-Conditioned Artist-Designed Mesh Generation and Editing📢 Users... can interactively design 3D models just from a sketch-based interface - check out the demo :) We break down the design process into addition with an autoregressive generator and deletion operations enabled by a classifier. To speed-up predictions, we propose a mesh-specific speculator such that users get immediate within a few seconds. Project: Video: Great work by Haoxuan Li Ziya Erkoç Lei Li Daniele Sirigatti V. Rosov Angela Daishow more

Can we use motion to prompt diffusion models? Our... #ICLR2024 paper does just that. We propose Motion Guidance, a technique that allows users to edit an image by specifying “where things should move.”show more

Wonderland: Navigating 3D Scenes from a Single Image Contributions:... • First, we introduce a representation for controllable 3D generation by leveraging the generative priors from camera-guided video diffusion models. Unlike image models, video diffusion models are trained on extensive video datasets. This enables them to capture comprehensive spatial relationships within scenes across multiple views and embed a form of "3D awareness" in their latent space, which allows us to maintain 3D consistency in novel view synthesis. • Second, to achieve controllable novel view generation, we empower video models with precise control over specified camera motions. We introduce a novel dual-branch conditioning mechanism that effectively incorporates desired diverse camera trajectories into the video diffusion model. This enables expansion of a single image into a multi-view consistent capture of a 3D scene with precise pose control. • Third, to achieve efficient 3D reconstruction, we directly transform video latents into 3DGS. We propose a novel latent-based large reconstruction model (LaLRM) that lifts video latents to 3D in a feed-forward manner. With this design, during inference, our model directly predicts 3DGS from a single input image, effectively aligning the generation and reconstruction tasks—and bridging image space and 3D space—through the video latent space. Compared with reconstructing scenes from images, the video latent space offers a 256× spatial-temporal reduction while retaining essential and consistent 3D structural details. Such a high degree of compression is crucial, as it allows the LaLRM to handle a wider range of 3D scenes within the reconstruction framework, with the same memory constraints.show more

DimensionX: Create Any 3D and 4D Scenes from a... Single Image with Controllable Video Diffusion TL;DR: Create 3/4DGS from Video Diffusion Note: Some first inference code released (not all yet). Contributions (cited): • We present DimensionX, a novel framework for generating photorealistic 3D and 4D scenes from only a single image using controllable video diffusion. • We propose ST-Director, which decouples the spatial and temporal priors in video diffusion models by learning (spatial and temporal) dimension-aware modules with our curated datasets. We further enhance the hybriddimension control with a training-free composition approach according to the essence of video diffusion denoising process. • To bridge the gap between video diffusion and real-world scenes, we design a trajectory-aware mechanism for 3D generation and an identity-preserving denoising approach for 4D generation, enabling more realistic and controllable scene synthesis. • Extensive experiments manifest that our DimensionX delivers superior performance in video, 3D, and 4D generation compared with baseline methods.show more

The context size of video world models is only... a few frames. Like a human with severe memory loss! We design a long-term memory for world models based on explicit 3D representations inspired by the human mind. This enables long-term consistency. 1/3show more

📢GeomHair: Reconstruction of Hair Strands from Colorless 3D Scans📢... We present a novel method to reconstruct hair strands from colorless 3D scans by extracting orientation cues directly from the mesh surface geometry by finding local characteristic lines and from shaded renderings using a neural 2D line detector. We enhance the reconstruction with a diffusion prior trained on synthetic hair data and adapted to each scan using a tailored text prompt, allowing us to recover both simple and complex hairstyles without relying on color input. To support further research, we also introduce Strands400, the largest publicly available dataset of 3D hair strand reconstructions from real-world scans of 400 different people, featuring complicated hairstyles, such as ponytails and buns. 🌍 📷 Great work by Rachmadio Noval L. Artem Sevastopolsky Egor Zakharov @ness_prisshow more

Introducing pixelSplat: feed-forward Gaussian splats from image pairs! Led... by David Charatan and Lester Li, collaborating with Andrea Tagliasacchi @CVPR! We propose a memory-efficient, fast and editable alternative to pixelNeRF based on 3D Gaussian Splatting! 1/nshow more

📢Introducing Generated Reality📢 A world model for XR that... turns your tracked hand and head poses into an interactive, generative video experience. Take world models to the next level by interacting with the world using your own body! 🔗 1/4show more

(1/n) 🚀 With FastVideo, you can now generate a... 5-second video in 5 seconds on a single H200 GPU! Introducing FastWan series, a family of fast video generation models trained via a new recipe we term as “sparse distillation”, to speed up video denoising time by 70X! 🖥️ Live demo: (Thanks to @gmicloud for the support!) 🔗 Blog: 🔓 We fully open-source our models, code, and data with Apache-2.0 licensesshow more

This was just a sketch from a few weeks... ago, but I suddenly got inspired and ended up polishing and editing it 😔show more

🤗Another breakdown from an older project made for Lands... of Noara . It was one of my first 2D VFX on a shot and a great project to direct and work on. I also made the motion design. Animated by the fantastic Makadam #blender #animation #3D #p2designshow more

3DGS render blender add-on v3.0 ( was just open-sourced!... With KIRI tools, you can use "3DGS to mesh" function to get a mesh from Gaussian splatting, and now you can convert the mesh back to Gaussian splatting. It's kind of ironic, but we just try to bring handy tools to everyone. #GaussianSplatting #Blendershow more

Competition 2: Mesh Generation 🦾 Until now, our subnet... has focused on Gaussian splats. That work led to integrations with Unity, building the world’s largest 3D dataset, and early application deployments with real users and organic traffic growth. 🕹️ The next step is clear. Developers want native mesh generation that drops straight into existing game pipelines and unlocks real production use today. 🤖 Our competition format lets us move flexibly between goals, optimizing for near term commercial utility with meshes and balancing longer term research with Gaussian splat world models and digital twins. 📈 With significant open source mesh models releasing in the past weeks, the timing is right and the bar for miner innovation has never been higher. Mesh competition launches in 2 weeks!show more

Someone just built a desktop app that that generates... 3D models from images and runs 100% locally. It's called Modly. It runs entirely on your GPU, no cloud, no API bills. Just drop an image and get a 3D mesh. 100% Open Source.show more

📢📢📢 Check out XCube, a generative model for very-high... resolution 3D shapes and scenes! Project Page: Project Video: Props to Xuanchi Ren for leading this project!!!! [1/N]show more

Huge drop by Meta: ActionMesh turns any video into... an animated 3D mesh. Demo available on Hugging Face ⬇️show more

The #1 AI 3D Model Generator just got a... game-changing upgrade — and anyone can use it. No 3D design experience? No problem. From game assets to product design — this tool does it all. 👇🧵show more

ImmerseGen: Agent-Guided Immersive World Generation with Alpha-Textured Proxies Contributions:... 1) We propose ImmerseGen, a novel agent-guided 3D environment generation framework. It uses simplified geometric proxies with alpha-textured meshes to produce compact, photorealistic worlds ready for real-time mobile VR rendering. 2) We propose a novel RGBA texturing paradigm. It first synthesizes 8K terrain textures using a geometry-conditioned panorama generator via user-centric mapping, and then directly generates alpha-textured proxy assets, avoiding fidelity loss typically resulting from mesh decimation. 3) To automate scene creation from user prompts, we introduce VLM-based modeling agents equipped with a novel grid-based semantic analysis. This enables 3D spatial reasoning from 2D observations and ensures accurate asset placement. ImmerseGen further enhances immersion with dynamic effects and ambient audio for a multisensory experience. 4) Experiments on multiple scene-generation scenarios and live mobile VR applications show that ImmerseGen outperforms previous methods in visual quality, realism, spatial coherence, and rendering efficiency for immersive real-time VR experiences.show more

🚀New paper out - We present Video-MSG (Multimodal Sketch... Guidance), a novel planning-based training-free guidance method for T2V models, improving control of spatial layout and object trajectories. 🔧 Key idea: • Generate a Video Sketch — a spatio-temporal plan with background, foreground, and motion in the pixel space. • Encode this structure directly into the latent space of the diffusion model during generation, which does not require fine-tuning or additional memory during inference. 🧵show more

We call this feature the Graphic Design Generator. To... make your own design, just set up a frame, position your text and elements exactly where you want them, type in a prompt, and hit ‘Recraft.’ Voilà — your design is ready! Try it out here: #RecraftAI #red_panda #GraphicDesignGeneratorshow more

📢 Our lab has been exploring 3D world models... for years — and we’re thrilled to share **PhysTwin**: a milestone that reconstructs object appearance, geometry, and dynamics from just a few seconds of interaction! Led by the amazing Hanxiao Jiang 👉 PhysTwin combines **Gaussian splatting** with **inverse dynamics optimization** based on simple **spring-mass** systems. ⚙️ The result? Real-time, action-conditioned 3D video prediction under novel interactions (i.e., 3D world models). 🔑 A few key takeaways: 1. Having the right structure (e.g., particles/masses) helps navigate the trade-off between sample efficiency, generalization, and broad applicability. 2. Visual foundation models (VFMs) have matured to the point where they can provide rich supervision for world modeling (e.g., tracking, shape completion). 3. Beyond VFMs, many crucial components have come together in recent years: Gaussian splats for rendering, NVIDIA Warp for high-performance simulation, and scene/asset generation from a wide range of labs and companies. The future of 3D world models is looking bright! ✨ 4. The resulting digital twin supports a wide range of downstream applications—especially in data generation and policy evaluation, thanks to its realistic rendering and simulation capabilities. 🎥 All code and data to reproduce the results, along with interactive demos, are available on the website. Check the following visualizations of: (1) observations, (2) reconstructed state/actions, (3) interactive digital twins, and (4) the overlays between real-world robot teleoperation and our model’s open-loop predictions.show more