Loading video...

Video Failed to Load

There was a problem loading this video. This could be due to a temporary network issue or the video might be unavailable.

ControlNet is a neural network structure to control diffusion models by adding extra conditions, processing any of photo, text, or logo and getting hundreds of design options in just a couple of clicks

Massimo

4,437,680 subscribers

213,965 views • 3 years ago •via X (Twitter)

Anya Rossi• Live Now

Private livecam show

9 Comments

dudeguy3 years ago

okay but what the fuck is that picture you picked

Jelena Perfiljeva3 years ago

Always love your tweets but can we please not promote that picture? How do you think it makes many women feel to see this?

Chiguy3 years ago

This is scary because older people, and most people really, will get to the point we will never know what’s real and what’s not anymore. All the doctored videos and pics people make that are fake to rile up everyone.

Liquirizia3 years ago

This is the entrance of Witherun lol

Deviant Weather Observer3 years ago

😁

Jimmy Wine3 years ago

Whiterun

Blaze3 years ago

graphic design schools and studios in 10 years

🇺🇦Sonia!3 years ago

Damn, it's an unsubscribe for me. Promoting AI based on theft is cringe at this point.

brena se ha ido a blusky3 years ago

i love the stuff you guys post but pls stop posting Ais related content

Related Videos

Excited to share our work on Neural Assets: a new method for enabling 3D asset-level control in image diffusion models – scalable & without any 3D inductive biases. Neural Assets goes beyond text or pixel-based control & provides an interface inspired by 3D graphics tools. 🧵

Excited to share our work on Neural Assets: a new method for enabling 3D asset-level control in image diffusion models – scalable & without any 3D inductive biases. Neural Assets goes beyond text or pixel-based control & provides an interface inspired by 3D graphics tools. 🧵

Thomas Kipf

98,036 views • 2 years ago

Does GPT understand the world? Here is what Ilya Sutskever, co-founder of OpenAI, says during a discussion with Jensen Huang, CEO of Nvidia: (1) When we train a large neural network to accurately predict the next word in lots of different texts from the internet, the AI is learning a world model. (2) On the surface, it may look like learning correlations in text, but it turns out that to 'just learn' statistical correlations in text, to compress information really well, what the neural network learns is some representation of the process that produced the text. (3) This text is a projection of the world...what the neural network is learning is aspects of the world, of people, of the human conditions, their hopes, dreams, motivations, their interactions...the situations we are in. The neural network learns a compressed, abstract, usable representation." Do you think learning representations = understanding? Are large language models simply stochastic parrots, or are they much more?

Does GPT understand the world? Here is what Ilya Sutskever, co-founder of OpenAI, says during a discussion with Jensen Huang, CEO of Nvidia: (1) When we train a large neural network to accurately predict the next word in lots of different texts from the internet, the AI is learning a world model. (2) On the surface, it may look like learning correlations in text, but it turns out that to 'just learn' statistical correlations in text, to compress information really well, what the neural network learns is some representation of the process that produced the text. (3) This text is a projection of the world...what the neural network is learning is aspects of the world, of people, of the human conditions, their hopes, dreams, motivations, their interactions...the situations we are in. The neural network learns a compressed, abstract, usable representation." Do you think learning representations = understanding? Are large language models simply stochastic parrots, or are they much more?

Alex Ker 🔭

1,367,231 views • 3 years ago

LLM-grounded Diffusion: Enhancing Prompt Understanding of Text-to-Image Diffusion Models with Large Language Models paper page: github: Recent advancements in text-to-image generation with diffusion models have yielded remarkable results synthesizing highly realistic and diverse images. However, these models still encounter difficulties when generating images from prompts that demand spatial or common sense reasoning. We propose to equip diffusion models with enhanced reasoning capabilities by using off-the-shelf pretrained large language models (LLMs) in a novel two-stage generation process. First, we adapt an LLM to be a text-guided layout generator through in-context learning. When provided with an image prompt, an LLM outputs a scene layout in the form of bounding boxes along with corresponding individual descriptions. Second, we steer a diffusion model with a novel controller to generate images conditioned on the layout. Both stages utilize frozen pretrained models without any LLM or diffusion model parameter optimization. We validate the superiority of our design by demonstrating its ability to outperform the base diffusion model in accurately generating images according to prompts that necessitate both language and spatial reasoning. Additionally, our method naturally allows dialog-based scene specification and is able to handle prompts in a language that is not well-supported by the underlying diffusion model.

LLM-grounded Diffusion: Enhancing Prompt Understanding of Text-to-Image Diffusion Models with Large Language Models paper page: github: Recent advancements in text-to-image generation with diffusion models have yielded remarkable results synthesizing highly realistic and diverse images. However, these models still encounter difficulties when generating images from prompts that demand spatial or common sense reasoning. We propose to equip diffusion models with enhanced reasoning capabilities by using off-the-shelf pretrained large language models (LLMs) in a novel two-stage generation process. First, we adapt an LLM to be a text-guided layout generator through in-context learning. When provided with an image prompt, an LLM outputs a scene layout in the form of bounding boxes along with corresponding individual descriptions. Second, we steer a diffusion model with a novel controller to generate images conditioned on the layout. Both stages utilize frozen pretrained models without any LLM or diffusion model parameter optimization. We validate the superiority of our design by demonstrating its ability to outperform the base diffusion model in accurately generating images according to prompts that necessitate both language and spatial reasoning. Additionally, our method naturally allows dialog-based scene specification and is able to handle prompts in a language that is not well-supported by the underlying diffusion model.

AK

83,681 views • 3 years ago

TokenFlow: Consistent Diffusion Features for Consistent Video Editing paper page: The generative AI revolution has recently expanded to videos. Nevertheless, current state-of-the-art video models are still lagging behind image models in terms of visual quality and user control over the generated content. In this work, we present a framework that harnesses the power of a text-to-image diffusion model for the task of text-driven video editing. Specifically, given a source video and a target text-prompt, our method generates a high-quality video that adheres to the target text, while preserving the spatial layout and motion of the input video. Our method is based on a key observation that consistency in the edited video can be obtained by enforcing consistency in the diffusion feature space. We achieve this by explicitly propagating diffusion features based on inter-frame correspondences, readily available in the model. Thus, our framework does not require any training or fine-tuning, and can work in conjunction with any off-the-shelf text-to-image editing method. We demonstrate state-of-the-art editing results on a variety of real-world videos.

TokenFlow: Consistent Diffusion Features for Consistent Video Editing paper page: The generative AI revolution has recently expanded to videos. Nevertheless, current state-of-the-art video models are still lagging behind image models in terms of visual quality and user control over the generated content. In this work, we present a framework that harnesses the power of a text-to-image diffusion model for the task of text-driven video editing. Specifically, given a source video and a target text-prompt, our method generates a high-quality video that adheres to the target text, while preserving the spatial layout and motion of the input video. Our method is based on a key observation that consistency in the edited video can be obtained by enforcing consistency in the diffusion feature space. We achieve this by explicitly propagating diffusion features based on inter-frame correspondences, readily available in the model. Thus, our framework does not require any training or fine-tuning, and can work in conjunction with any off-the-shelf text-to-image editing method. We demonstrate state-of-the-art editing results on a variety of real-world videos.

AK

143,469 views • 3 years ago

Exploring animated intros with text/logo #warpfusion Stable Diffusion #midjourney #ai #stablediffusion #controlnet #animation #aiar

Mrboofy

180,012 views • 3 years ago

A lot of people think that diffusion LLMs == BERT. But masked dLLMs are just one form of text diffusion, popularized by Subham Sahoo and his group at Cornell. There's much more to explore in text diffusion land! Check out our conversation (link below)!

A lot of people think that diffusion LLMs == BERT. But masked dLLMs are just one form of text diffusion, popularized by Subham Sahoo and his group at Cornell. There's much more to explore in text diffusion land! Check out our conversation (link below)!

Julia Turc

26,124 views • 6 months ago

"In the future, you won't need any software at all. The neural network will completely replace it." - stephen balaban, co-founder of Lambda "Here's the idea: instead of generating a program, just go to ChatGPT and say, 'Hi, please behave like a calculator or a spreadsheet.'" "'Generate an ASCII user interface for me. And I want you to essentially just respond, implement the logic of that program in your mind.'" "This is what I call neural software. Normal software is really brittle. 'Neural software' can't have bugs, only misunderstandings." "Large language models will be the software you interact with." From Stephen's appearance on the show in May.

"In the future, you won't need any software at all. The neural network will completely replace it." - stephen balaban, co-founder of Lambda "Here's the idea: instead of generating a program, just go to ChatGPT and say, 'Hi, please behave like a calculator or a spreadsheet.'" "'Generate an ASCII user interface for me. And I want you to essentially just respond, implement the logic of that program in your mind.'" "This is what I call neural software. Normal software is really brittle. 'Neural software' can't have bugs, only misunderstandings." "Large language models will be the software you interact with." From Stephen's appearance on the show in May.

TBPN

28,665 views • 11 months ago

Google just released Muse: a text to image model achieving state of the art performance. It is significantly more efficient than diffusion models. Muse also allows for inpainting, outpainting, and mask-free editing without finetuning or inverting the model.

Google just released Muse: a text to image model achieving state of the art performance. It is significantly more efficient than diffusion models. Muse also allows for inpainting, outpainting, and mask-free editing without finetuning or inverting the model.

bleedingedge.ai

183,766 views • 3 years ago

Generative Rendering: Controllable 4D-Guided Video Generation with 2D Diffusion Models paper page: Traditional 3D content creation tools empower users to bring their imagination to life by giving them direct control over a scene's geometry, appearance, motion, and camera path. Creating computer-generated videos, however, is a tedious manual process, which can be automated by emerging text-to-video diffusion models. Despite great promise, video diffusion models are difficult to control, hindering a user to apply their own creativity rather than amplifying it. To address this challenge, we present a novel approach that combines the controllability of dynamic 3D meshes with the expressivity and editability of emerging diffusion models. For this purpose, our approach takes an animated, low-fidelity rendered mesh as input and injects the ground truth correspondence information obtained from the dynamic mesh into various stages of a pre-trained text-to-image generation model to output high-quality and temporally consistent frames. We demonstrate our approach on various examples where motion can be obtained by animating rigged assets or changing the camera path.

Generative Rendering: Controllable 4D-Guided Video Generation with 2D Diffusion Models paper page: Traditional 3D content creation tools empower users to bring their imagination to life by giving them direct control over a scene's geometry, appearance, motion, and camera path. Creating computer-generated videos, however, is a tedious manual process, which can be automated by emerging text-to-video diffusion models. Despite great promise, video diffusion models are difficult to control, hindering a user to apply their own creativity rather than amplifying it. To address this challenge, we present a novel approach that combines the controllability of dynamic 3D meshes with the expressivity and editability of emerging diffusion models. For this purpose, our approach takes an animated, low-fidelity rendered mesh as input and injects the ground truth correspondence information obtained from the dynamic mesh into various stages of a pre-trained text-to-image generation model to output high-quality and temporally consistent frames. We demonstrate our approach on various examples where motion can be obtained by animating rigged assets or changing the camera path.

AK

117,091 views • 2 years ago

🏃Motion Tokenizer Bridging Semantic and Kinematic Conditions🏃‍♀️ #MoTok is a diffusion-based tokenizer that unifies *perception-planning-control*, combining the strengths of continuous diffusion & discrete tokens - Project: - Code:

🏃Motion Tokenizer Bridging Semantic and Kinematic Conditions🏃‍♀️ #MoTok is a diffusion-based tokenizer that unifies perception-planning-control, combining the strengths of continuous diffusion & discrete tokens - Project: - Code:

Ziwei Liu

19,259 views • 7 months ago

Stable Diffusion is now capable of creating photo realistic full 3D models from single images. The amount of ways it could be used in video games and the metaverse blows my mind. AI is getting closer and closer to futuristic sci-fi movies!

Stable Diffusion is now capable of creating photo realistic full 3D models from single images. The amount of ways it could be used in video games and the metaverse blows my mind. AI is getting closer and closer to futuristic sci-fi movies!

CONTROLLER TALK

224,122 views • 3 years ago

In this experiment you can bring your scribbles to life with just a prompt! 🎨 I've used Stable Diffusion, ControlNet, a 360° Lora and 8th Wall's Sky segmentation. #stablediffusion #controlnet #threejs #8thwall

In this experiment you can bring your scribbles to life with just a prompt! 🎨 I've used Stable Diffusion, ControlNet, a 360° Lora and 8th Wall's Sky segmentation. #stablediffusion #controlnet #threejs #8thwall

Stijn Spanhove

39,212 views • 3 years ago

This is a big day. Meta is open-sourcing AudioCraft. You can now generate incredible music and sounds with a single prompt. It includes the most performant Generative AI Model (audio) on the market, the "Llama" of Audio. The research framework contains the weights and code of these models: ▸ MusicGen: controllable text-to-music model. ▸ AudioGen: text-to-sound model. ▸ EnCodec: high fidelity neural audio codec. ▸ Multi Band Diffusion: An EnCodec compatible decoder using diffusion. This is going to tremendously speed up audio research 👏

This is a big day. Meta is open-sourcing AudioCraft. You can now generate incredible music and sounds with a single prompt. It includes the most performant Generative AI Model (audio) on the market, the "Llama" of Audio. The research framework contains the weights and code of these models: ▸ MusicGen: controllable text-to-music model. ▸ AudioGen: text-to-sound model. ▸ EnCodec: high fidelity neural audio codec. ▸ Multi Band Diffusion: An EnCodec compatible decoder using diffusion. This is going to tremendously speed up audio research 👏

Lior Alexander

231,729 views • 3 years ago

visualization of a simple neural network

visualization of a simple neural network

ℏεsam

79,716 views • 1 year ago

Dreamix: Video Diffusion Models are General Video Editors abs: project page: present diffusion-based method that is able to perform text-based motion and appearance editing of general videos

Dreamix: Video Diffusion Models are General Video Editors abs: project page: present diffusion-based method that is able to perform text-based motion and appearance editing of general videos

AK

398,254 views • 3 years ago

I spent the weekend hacking an AI room makeover using ControlNet & Stable Diffusion, restyling my parent's "drawing room." How soon until this workflow is standard for home décor & interior design? ControlNet is a glimpse into the AI-infused future of 3D rendering & AR.

I spent the weekend hacking an AI room makeover using ControlNet & Stable Diffusion, restyling my parent's "drawing room." How soon until this workflow is standard for home décor & interior design? ControlNet is a glimpse into the AI-infused future of 3D rendering & AR.

Bilawal Sidhu

473,319 views • 3 years ago

CATransformers is a carbon-driven neural architecture and system hardware co-design framework. Using CATransformers, we discover greener CLIP models that achieve an average of 9.1% reduction potential in total lifecycle carbon emissions while maintaining accuracy (or increasing accuracy) and latency. This research is the first to look into carbon-driven neural architecture and system hardware co-design. It is enabled by a first-of-its-kind architectural carbon modeling tool – ACT, which we developed at FAIR. Check out: Our paper ➡️ code repository ➡️ and the additional carbon design tools and research artifacts in Sustainable AI ➡️

CATransformers is a carbon-driven neural architecture and system hardware co-design framework. Using CATransformers, we discover greener CLIP models that achieve an average of 9.1% reduction potential in total lifecycle carbon emissions while maintaining accuracy (or increasing accuracy) and latency. This research is the first to look into carbon-driven neural architecture and system hardware co-design. It is enabled by a first-of-its-kind architectural carbon modeling tool – ACT, which we developed at FAIR. Check out: Our paper ➡️ code repository ➡️ and the additional carbon design tools and research artifacts in Sustainable AI ➡️

AI at Meta

32,738 views • 1 year ago