Loading video...

Video Failed to Load

Go Home

(1/2) LightIt: Illumination Modeling and Control for Diffusion Models! #CVPR2024 We facilitate lighting control for novel image generation from text prompts. We can also edit lighting for a given input image. Video: Project:

MattNiessner's profile picture

Matthias Niessner

47,444 subscribers

19,835 views • 2 years ago •via X (Twitter)

Science & TechnologyEducation#CVPR2024
LIVE

Anya Rossi• Live Now

Private livecam show

4 Comments

Matthias Niessner's profile picture
Matthias Niessner2 years ago

(2/2) Algorithmically, we propose a control reconstruction loss that constrains identity preservation (i.e., same image content) while being able to change the lighting. Great work by @Peter4AI from his internship advised by @JulienPhilip2, Kalyan Sunkavalli, @yannickhold!

Lukas's profile picture
Lukas2 years ago

Normalize more papers with clear explanations. Very nice video!

CubanHodLer🇨🇺's profile picture
CubanHodLer🇨🇺2 years ago

No code available?

MMU's profile picture
MMU2 years ago

Fantastic work.

Related Videos

DiffSplat Repurposing Image Diffusion Models for Scalable Gaussian Splat Generation DiffSplat is a generative framework to synthesize 3D Gaussian Splats from text prompts & single-view images in ⚡️ 1~2 seconds. It is fine-tuned directly from a pretrained text-to-image diffusion model.
0:24

DiffSplat Repurposing Image Diffusion Models for Scalable Gaussian Splat Generation DiffSplat is a generative framework to synthesize 3D Gaussian Splats from text prompts & single-view images in ⚡️ 1~2 seconds. It is fine-tuned directly from a pretrained text-to-image diffusion model.

AK

38,416 views • 1 year ago

Wan2.6 Image Generation is ready for Commercial Use! - Interleaved Text-and-Image Output - Multi-Image Conditioned Generation - Commercial-Grade ID Preservation - Extract Creative Elements - Precise Control of Camera Angles and Lighting Try Wan2.6 today!
1:01

Wan2.6 Image Generation is ready for Commercial Use! - Interleaved Text-and-Image Output - Multi-Image Conditioned Generation - Commercial-Grade ID Preservation - Extract Creative Elements - Precise Control of Camera Angles and Lighting Try Wan2.6 today!

Alibaba Cloud

145,741 views • 4 months ago

🔥🔥We propose #VideoBooth to enable **customized video generation** with image prompts, which provide more accurate and direct content control beyond the text prompts. - Project: - Code: - Video:
1:05

🔥🔥We propose #VideoBooth to enable **customized video generation** with image prompts, which provide more accurate and direct content control beyond the text prompts. - Project: - Code: - Video:

Ziwei Liu

26,316 views • 2 years ago

TurboEdit Instant text-based image editing discuss: We address the challenges of precise image inversion and disentangled image editing in the context of few-step diffusion models. We introduce an encoder based iterative inversion technique. The inversion network is conditioned on the input image and the reconstructed image from the previous step, allowing for correction of the next reconstruction towards the input image. We demonstrate that disentangled controls can be easily achieved in the few-step diffusion model by conditioning on an (automatically generated) detailed text prompt. To manipulate the inverted image, we freeze the noise maps and modify one attribute in the text prompt (either manually or via instruction based editing driven by an LLM), resulting in the generation of a new image similar to the input image with only one attribute changed. It can further control the editing strength and accept instructive text prompt. Our approach facilitates realistic text-guided image edits in real-time, requiring only 8 number of functional evaluations (NFEs) in inversion (one-time cost) and 4 NFEs per edit. Our method is not only fast, but also significantly outperforms state-of-the-art multi-step diffusion editing techniques.
0:22

TurboEdit Instant text-based image editing discuss: We address the challenges of precise image inversion and disentangled image editing in the context of few-step diffusion models. We introduce an encoder based iterative inversion technique. The inversion network is conditioned on the input image and the reconstructed image from the previous step, allowing for correction of the next reconstruction towards the input image. We demonstrate that disentangled controls can be easily achieved in the few-step diffusion model by conditioning on an (automatically generated) detailed text prompt. To manipulate the inverted image, we freeze the noise maps and modify one attribute in the text prompt (either manually or via instruction based editing driven by an LLM), resulting in the generation of a new image similar to the input image with only one attribute changed. It can further control the editing strength and accept instructive text prompt. Our approach facilitates realistic text-guided image edits in real-time, requiring only 8 number of functional evaluations (NFEs) in inversion (one-time cost) and 4 NFEs per edit. Our method is not only fast, but also significantly outperforms state-of-the-art multi-step diffusion editing techniques.

AK

16,062 views • 1 year ago

👀 Pixel perfect 💎✨ 🖼️ Edify Image from #NVIDIAResearch is a family of diffusion models that supports a wide range of applications, including text-to-image synthesis, 4K upsampling, ControlNets, 360° HDR panorama generation, and finetuning for image customization. 🧵 1/2
2:48

👀 Pixel perfect 💎✨ 🖼️ Edify Image from #NVIDIAResearch is a family of diffusion models that supports a wide range of applications, including text-to-image synthesis, 4K upsampling, ControlNets, 360° HDR panorama generation, and finetuning for image customization. 🧵 1/2

NVIDIA AI Developer

14,747 views • 1 year ago

Struggling with slow inference of diffusion and flow models? Check out the video below—I’ve been using our new FastGen library to achieve 7-28x acceleration for text-2-image and {text,image,video}-2-video generation without sacrificing visual fidelity!
2:20

Struggling with slow inference of diffusion and flow models? Check out the video below—I’ve been using our new FastGen library to achieve 7-28x acceleration for text-2-image and {text,image,video}-2-video generation without sacrificing visual fidelity!

Julius Berner

13,463 views • 3 months ago

introducing Qwen Edit. this new models offers incredible image editing capabilities from text prompts. try it now for free!
0:25

introducing Qwen Edit. this new models offers incredible image editing capabilities from text prompts. try it now for free!

Krea

76,657 views • 9 months ago

Image Generation has landed🎨🤖 Introducing Image Arena - a battle of image generation models like FLUX, Stable Diffusion, Dall-E, Recraft, Ideogram, and more! Who will reign supreme? 1. Describe your desired image🎨 2. Two anonymous models output images 3. Vote for the winner! Enjoy! We will be releasing the leaderboard soon! More examples below👇
0:46

Image Generation has landed🎨🤖 Introducing Image Arena - a battle of image generation models like FLUX, Stable Diffusion, Dall-E, Recraft, Ideogram, and more! Who will reign supreme? 1. Describe your desired image🎨 2. Two anonymous models output images 3. Vote for the winner! Enjoy! We will be releasing the leaderboard soon! More examples below👇

lmarena.ai (formerly lmsys.org)

106,853 views • 1 year ago

EasyControl just dropped on Hugging Face Adding Efficient and Flexible Control for Diffusion Transformer Free chatgpt style ghibli image generation with easy control
0:34

EasyControl just dropped on Hugging Face Adding Efficient and Flexible Control for Diffusion Transformer Free chatgpt style ghibli image generation with easy control

AK

222,119 views • 1 year ago

(1/4) Excited to share our #ICCV2023 paper Text2Room! We generate scene-scale textured 3D meshes from a given text prompt leveraging 2D text-to-image models such as StableDiffusion. Project: Code: Video:
0:24

(1/4) Excited to share our #ICCV2023 paper Text2Room! We generate scene-scale textured 3D meshes from a given text prompt leveraging 2D text-to-image models such as StableDiffusion. Project: Code: Video:

Matthias Niessner

74,788 views • 2 years ago

New Kling, Runway, Luma competitor? Open text and image video generation model Pyramidal Flow Matching for Efficient Video Generative Modeling
2:23

New Kling, Runway, Luma competitor? Open text and image video generation model Pyramidal Flow Matching for Efficient Video Generative Modeling

AK

90,965 views • 1 year ago

LLM-grounded Diffusion: Enhancing Prompt Understanding of Text-to-Image Diffusion Models with Large Language Models paper page: github: Recent advancements in text-to-image generation with diffusion models have yielded remarkable results synthesizing highly realistic and diverse images. However, these models still encounter difficulties when generating images from prompts that demand spatial or common sense reasoning. We propose to equip diffusion models with enhanced reasoning capabilities by using off-the-shelf pretrained large language models (LLMs) in a novel two-stage generation process. First, we adapt an LLM to be a text-guided layout generator through in-context learning. When provided with an image prompt, an LLM outputs a scene layout in the form of bounding boxes along with corresponding individual descriptions. Second, we steer a diffusion model with a novel controller to generate images conditioned on the layout. Both stages utilize frozen pretrained models without any LLM or diffusion model parameter optimization. We validate the superiority of our design by demonstrating its ability to outperform the base diffusion model in accurately generating images according to prompts that necessitate both language and spatial reasoning. Additionally, our method naturally allows dialog-based scene specification and is able to handle prompts in a language that is not well-supported by the underlying diffusion model.
5:11

LLM-grounded Diffusion: Enhancing Prompt Understanding of Text-to-Image Diffusion Models with Large Language Models paper page: github: Recent advancements in text-to-image generation with diffusion models have yielded remarkable results synthesizing highly realistic and diverse images. However, these models still encounter difficulties when generating images from prompts that demand spatial or common sense reasoning. We propose to equip diffusion models with enhanced reasoning capabilities by using off-the-shelf pretrained large language models (LLMs) in a novel two-stage generation process. First, we adapt an LLM to be a text-guided layout generator through in-context learning. When provided with an image prompt, an LLM outputs a scene layout in the form of bounding boxes along with corresponding individual descriptions. Second, we steer a diffusion model with a novel controller to generate images conditioned on the layout. Both stages utilize frozen pretrained models without any LLM or diffusion model parameter optimization. We validate the superiority of our design by demonstrating its ability to outperform the base diffusion model in accurately generating images according to prompts that necessitate both language and spatial reasoning. Additionally, our method naturally allows dialog-based scene specification and is able to handle prompts in a language that is not well-supported by the underlying diffusion model.

AK

83,657 views • 2 years ago

We introduce W.A.L.T, a diffusion model for photorealistic video generation. Our model is a transformer trained on image and video generation in a shared latent space. 🧵👇
0:43

We introduce W.A.L.T, a diffusion model for photorealistic video generation. Our model is a transformer trained on image and video generation in a shared latent space. 🧵👇

Agrim Gupta

431,113 views • 2 years ago

Generate videos — or control them in chat. AI avatars, face swap, voice & audio built in. Edit anything: objects, lighting, camera, even the story. One input → series + variations. Consistent characters, cinematic multi-shot videos. Text/image → full video. Try 10 generations.
0:51

Generate videos — or control them in chat. AI avatars, face swap, voice & audio built in. Edit anything: objects, lighting, camera, even the story. One input → series + variations. Consistent characters, cinematic multi-shot videos. Text/image → full video. Try 10 generations.

Neuravision App

4,094,929 views • 2 months ago

Generate videos — or control them in chat. AI avatars, face swap, voice & audio built in. Edit anything: objects, lighting, camera, even the story. One input → series + variations. Consistent characters, cinematic multi-shot videos. Text/image → full video. Try 10 generations.
0:32

Generate videos — or control them in chat. AI avatars, face swap, voice & audio built in. Edit anything: objects, lighting, camera, even the story. One input → series + variations. Consistent characters, cinematic multi-shot videos. Text/image → full video. Try 10 generations.

Neuravision App

4,579,863 views • 2 months ago

Phidias A Generative Model for Creating 3D Content from Text, Image, and 3D Conditions with Reference-Augmented Diffusion discuss: In 3D modeling, designers often use an existing 3D model as a reference to create new ones. This practice has inspired the development of Phidias, a novel generative model that uses diffusion for reference-augmented 3D generation. Given an image, our method leverages a retrieved or user-provided 3D reference model to guide the generation process, thereby enhancing the generation quality, generalization ability, and controllability. Our model integrates three key components: 1) meta-ControlNet that dynamically modulates the conditioning strength, 2) dynamic reference routing that mitigates misalignment between the input image and 3D reference, and 3) self-reference augmentations that enable self-supervised training with a progressive curriculum. Collectively, these designs result in a clear improvement over existing methods. Phidias establishes a unified framework for 3D generation using text, image, and 3D conditions with versatile applications.
1:53

Phidias A Generative Model for Creating 3D Content from Text, Image, and 3D Conditions with Reference-Augmented Diffusion discuss: In 3D modeling, designers often use an existing 3D model as a reference to create new ones. This practice has inspired the development of Phidias, a novel generative model that uses diffusion for reference-augmented 3D generation. Given an image, our method leverages a retrieved or user-provided 3D reference model to guide the generation process, thereby enhancing the generation quality, generalization ability, and controllability. Our model integrates three key components: 1) meta-ControlNet that dynamically modulates the conditioning strength, 2) dynamic reference routing that mitigates misalignment between the input image and 3D reference, and 3) self-reference augmentations that enable self-supervised training with a progressive curriculum. Collectively, these designs result in a clear improvement over existing methods. Phidias establishes a unified framework for 3D generation using text, image, and 3D conditions with versatile applications.

AK

25,120 views • 1 year ago

ByteDance just unveiled HuMo, a unified framework for human-centric video generation. With HuMo you can: • Generate videos from text + image • Create audio-synced videos from text + audio • Combine text, image, and audio for maximum control • Preserve consistent subjects across frames • Achieve natural audio-visual synchronization This isn’t just another release , it’s a big step toward fully controllable, coherent, and cinematic AI video.
1:51

ByteDance just unveiled HuMo, a unified framework for human-centric video generation. With HuMo you can: • Generate videos from text + image • Create audio-synced videos from text + audio • Combine text, image, and audio for maximum control • Preserve consistent subjects across frames • Achieve natural audio-visual synchronization This isn’t just another release , it’s a big step toward fully controllable, coherent, and cinematic AI video.

DStudioproject

29,408 views • 9 months ago

Interactive 3D posing and smart lighting prompts. With this nodepack in ComfyUI, it's now easier for you to control the image's pose and lighting:
1:09

Interactive 3D posing and smart lighting prompts. With this nodepack in ComfyUI, it's now easier for you to control the image's pose and lighting:

Daxiong (Lin)

13,294 views • 4 months ago

Wan2.6: Commercial-Grade Image Output Wan2.6-Image is now available. 🖼️ Interleaved Text-and-Image Output：Generate interleaved text-and-image content with logical reasoning capabilities — enabling layered, narrative-driven visual storytelling. 🖼️ Multi-Image Conditioned Generation：Support flexible referencing, combining, and replacement of multiple images, integrating varied visual inspirations to generate novel and compelling results 🖼️ Commercial-Grade ID Preservation：High consistency in characters, styles, and elements for commercial scenarios. 🖼️ Extract creative elements — such as color, style, and composition — from reference images to enable aesthetically driven image generation. 🖼️ Precise Control of Camera Angles and Lighting：Support specifying camera perspective, spatial depth (foreground/background), and lighting details — enabling precise control over spatial composition and atmospheric mood. Try Wan2.6 today.
1:00

Wan2.6: Commercial-Grade Image Output Wan2.6-Image is now available. 🖼️ Interleaved Text-and-Image Output：Generate interleaved text-and-image content with logical reasoning capabilities — enabling layered, narrative-driven visual storytelling. 🖼️ Multi-Image Conditioned Generation：Support flexible referencing, combining, and replacement of multiple images, integrating varied visual inspirations to generate novel and compelling results 🖼️ Commercial-Grade ID Preservation：High consistency in characters, styles, and elements for commercial scenarios. 🖼️ Extract creative elements — such as color, style, and composition — from reference images to enable aesthetically driven image generation. 🖼️ Precise Control of Camera Angles and Lighting：Support specifying camera perspective, spatial depth (foreground/background), and lighting details — enabling precise control over spatial composition and atmospheric mood. Try Wan2.6 today.

Wan

33,220 views • 5 months ago

Grok Imagine API just released A world-class video generation + video editing model Text-to-Video: Turn simple prompts into rich video clips with audio Image Generation + Editing: Bring ideas to life with visuals from scratch Video Editing Tools: Restyle scenes, add/remove props, control motion Best-in-Class Quality + Low Latency: Designed to deliver fast, cost-efficient results API pricing: Image input: $0.002 Video input : $0.01 Video output : $0.05
0:22

Grok Imagine API just released A world-class video generation + video editing model Text-to-Video: Turn simple prompts into rich video clips with audio Image Generation + Editing: Bring ideas to life with visuals from scratch Video Editing Tools: Restyle scenes, add/remove props, control motion Best-in-Class Quality + Low Latency: Designed to deliver fast, cost-efficient results API pricing: Image input: $0.002 Video input : $0.01 Video output : $0.05

X Freeze

15,078 views • 4 months ago