Loading video...

Video Failed to Load

There was a problem loading this video. This could be due to a temporary network issue or the video might be unavailable.

[1/5] Always wondered what people see when looking at a Rorschach test? SpaText - our recent #CVPR2023 paper from @MetaAI may give you a sneak peek! TL;DR: We extend text-to-image models with region-specific textual controllability. Project Page:

Omri Avrahami

1,196 subscribers

19,389 views • 3 years ago •via X (Twitter)

Anya Rossi• Live Now

Private livecam show

7 Comments

Omri Avrahami3 years ago

[2/5] Recent text-to-image diffusion models are able to generate convincing results of unprecedented quality. However, it is nearly impossible to control the shapes of different regions/objects or their layout in a fine-grained fashion.

Omri Avrahami3 years ago

[3/5] SpaText is a new method for text-to-image generation using open-vocabulary scene control. In addition to a global text prompt that describes the entire scene, the user provides a segmentation map where each region of interest is annotated by a free-form text description

Omri Avrahami3 years ago

[4/5] Due to lack of large-scale datasets that have a detailed textual description for each region in the image, we choose to leverage the current large-scale text-to-image datasets and base our approach on a novel CLIP-based spatio-textual representation

Omri Avrahami3 years ago

[5/5] Thanks to my great collaborators at @MetaAI: @THayes427, Oran Gafni, Sonal Gupta, Yaniv Taigman, @deviparikh, @xi_yin_, and to my supervisors @DaniLischinski and @ohadf! Special thanks to @m_mmandel for the Rorschach video art. #GenerativeAI #AIart #AIArtwork

Ori Gordon3 years ago

@MetaAI Wow

Or Tal3 years ago

@MetaAI Interesting read!

Moshe Mandel3 years ago

@MetaAI Thank you @OmriAvr! Please watch video with audio!

Related Videos

Designing an Encoder for Fast Personalization of Text-to-Image Models TL;DR: use an encoder to personalize a text-to-image model to new concepts with a single image and 5-15 tuning steps abs: project page:

Designing an Encoder for Fast Personalization of Text-to-Image Models TL;DR: use an encoder to personalize a text-to-image model to new concepts with a single image and 5-15 tuning steps abs: project page:

AK

165,168 views • 3 years ago

(1/4) Excited to share our #ICCV2023 paper Text2Room! We generate scene-scale textured 3D meshes from a given text prompt leveraging 2D text-to-image models such as StableDiffusion. Project: Code: Video:

(1/4) Excited to share our #ICCV2023 paper Text2Room! We generate scene-scale textured 3D meshes from a given text prompt leveraging 2D text-to-image models such as StableDiffusion. Project: Code: Video:

Matthias Niessner

74,907 views • 3 years ago

(1/5) Great to be at Sequoia Capital to give a sneak peek of one of our research directions! TL;DR one path to data-efficiency may be to “abuse GPUs like they’ve never been abused before”

(1/5) Great to be at Sequoia Capital to give a sneak peek of one of our research directions! TL;DR one path to data-efficiency may be to “abuse GPUs like they’ve never been abused before”

Flapping Airplanes

197,848 views • 3 months ago

Ever wondered what happens behind the scenes at a cover shoot with Leigh-Anne Pinnock? We’ll give you a sneak peek… 👀 Leigh-Anne is our real life Little Mermaid. Listen to her narrate the tale on Audible now:

Ever wondered what happens behind the scenes at a cover shoot with Leigh-Anne Pinnock? We’ll give you a sneak peek… 👀 Leigh-Anne is our real life Little Mermaid. Listen to her narrate the tale on Audible now:

Audible UK 🎧

17,132 views • 3 years ago

Google announces PALP Prompt Aligned Personalization of Text-to-Image Models paper page: Content creators often aim to create personalized images using personal subjects that go beyond the capabilities of conventional text-to-image models. Additionally, they may want the resulting image to encompass a specific location, style, ambiance, and more. Existing personalization methods may compromise personalization ability or the alignment to complex textual prompts. This trade-off can impede the fulfillment of user prompts and subject fidelity. We propose a new approach focusing on personalization methods for a single prompt to address this issue. We term our approach prompt-aligned personalization. While this may seem restrictive, our method excels in improving text alignment, enabling the creation of images with complex and intricate prompts, which may pose a challenge for current techniques. In particular, our method keeps the personalized model aligned with a target prompt using an additional score distillation sampling term. We demonstrate the versatility of our method in multi- and single-shot settings and further show that it can compose multiple subjects or use inspiration from reference images, such as artworks. We compare our approach quantitatively and qualitatively with existing baselines and state-of-the-art techniques.

Google announces PALP Prompt Aligned Personalization of Text-to-Image Models paper page: Content creators often aim to create personalized images using personal subjects that go beyond the capabilities of conventional text-to-image models. Additionally, they may want the resulting image to encompass a specific location, style, ambiance, and more. Existing personalization methods may compromise personalization ability or the alignment to complex textual prompts. This trade-off can impede the fulfillment of user prompts and subject fidelity. We propose a new approach focusing on personalization methods for a single prompt to address this issue. We term our approach prompt-aligned personalization. While this may seem restrictive, our method excels in improving text alignment, enabling the creation of images with complex and intricate prompts, which may pose a challenge for current techniques. In particular, our method keeps the personalized model aligned with a target prompt using an additional score distillation sampling term. We demonstrate the versatility of our method in multi- and single-shot settings and further show that it can compose multiple subjects or use inspiration from reference images, such as artworks. We compare our approach quantitatively and qualitatively with existing baselines and state-of-the-art techniques.

AK

90,199 views • 2 years ago

While still heavy work in progress (as you can see, we recorded directly from Unreal Engine Editor), we wanted to give you a sneak peek into our process of making these characters more fun, powerful, and feel good when playing! 🪓 Valkyrie & ⚔️ Barbarian:

While still heavy work in progress (as you can see, we recorded directly from Unreal Engine Editor), we wanted to give you a sneak peek into our process of making these characters more fun, powerful, and feel good when playing! 🪓 Valkyrie & ⚔️ Barbarian:

Project R.I.S.E

62,548 views • 1 year ago

T2I-CompBench: A Comprehensive Benchmark for Open-world Compositional Text-to-image Generation paper page: Despite the stunning ability to generate high-quality images by recent text-to-image models, current approaches often struggle to effectively compose objects with different attributes and relationships into a complex and coherent scene. We propose T2I-CompBench, a comprehensive benchmark for open-world compositional text-to-image generation, consisting of 6,000 compositional text prompts from 3 categories (attribute binding, object relationships, and complex compositions) and 6 sub-categories (color binding, shape binding, texture binding, spatial relationships, non-spatial relationships, and complex compositions). We further propose several evaluation metrics specifically designed to evaluate compositional text-to-image generation. We introduce a new approach, Generative mOdel fine-tuning with Reward-driven Sample selection (GORS), to boost the compositional text-to-image generation abilities of pretrained text-to-image models. Extensive experiments and evaluations are conducted to benchmark previous methods on T2I-CompBench, and to validate the effectiveness of our proposed evaluation metrics and GORS approach.

T2I-CompBench: A Comprehensive Benchmark for Open-world Compositional Text-to-image Generation paper page: Despite the stunning ability to generate high-quality images by recent text-to-image models, current approaches often struggle to effectively compose objects with different attributes and relationships into a complex and coherent scene. We propose T2I-CompBench, a comprehensive benchmark for open-world compositional text-to-image generation, consisting of 6,000 compositional text prompts from 3 categories (attribute binding, object relationships, and complex compositions) and 6 sub-categories (color binding, shape binding, texture binding, spatial relationships, non-spatial relationships, and complex compositions). We further propose several evaluation metrics specifically designed to evaluate compositional text-to-image generation. We introduce a new approach, Generative mOdel fine-tuning with Reward-driven Sample selection (GORS), to boost the compositional text-to-image generation abilities of pretrained text-to-image models. Extensive experiments and evaluations are conducted to benchmark previous methods on T2I-CompBench, and to validate the effectiveness of our proposed evaluation metrics and GORS approach.

AK

17,503 views • 3 years ago

LLM-grounded Diffusion: Enhancing Prompt Understanding of Text-to-Image Diffusion Models with Large Language Models paper page: github: Recent advancements in text-to-image generation with diffusion models have yielded remarkable results synthesizing highly realistic and diverse images. However, these models still encounter difficulties when generating images from prompts that demand spatial or common sense reasoning. We propose to equip diffusion models with enhanced reasoning capabilities by using off-the-shelf pretrained large language models (LLMs) in a novel two-stage generation process. First, we adapt an LLM to be a text-guided layout generator through in-context learning. When provided with an image prompt, an LLM outputs a scene layout in the form of bounding boxes along with corresponding individual descriptions. Second, we steer a diffusion model with a novel controller to generate images conditioned on the layout. Both stages utilize frozen pretrained models without any LLM or diffusion model parameter optimization. We validate the superiority of our design by demonstrating its ability to outperform the base diffusion model in accurately generating images according to prompts that necessitate both language and spatial reasoning. Additionally, our method naturally allows dialog-based scene specification and is able to handle prompts in a language that is not well-supported by the underlying diffusion model.

LLM-grounded Diffusion: Enhancing Prompt Understanding of Text-to-Image Diffusion Models with Large Language Models paper page: github: Recent advancements in text-to-image generation with diffusion models have yielded remarkable results synthesizing highly realistic and diverse images. However, these models still encounter difficulties when generating images from prompts that demand spatial or common sense reasoning. We propose to equip diffusion models with enhanced reasoning capabilities by using off-the-shelf pretrained large language models (LLMs) in a novel two-stage generation process. First, we adapt an LLM to be a text-guided layout generator through in-context learning. When provided with an image prompt, an LLM outputs a scene layout in the form of bounding boxes along with corresponding individual descriptions. Second, we steer a diffusion model with a novel controller to generate images conditioned on the layout. Both stages utilize frozen pretrained models without any LLM or diffusion model parameter optimization. We validate the superiority of our design by demonstrating its ability to outperform the base diffusion model in accurately generating images according to prompts that necessitate both language and spatial reasoning. Additionally, our method naturally allows dialog-based scene specification and is able to handle prompts in a language that is not well-supported by the underlying diffusion model.

AK

83,657 views • 3 years ago

While we are still working hard on the 1.50 update, today we bring you a sneak peek of what is about to come! In cooperation with our friends from Scania Group, we will add the famous Scania Demo Centre, in Södertälje, Sweden, to ETS2! 🇸🇪 See more 👇

While we are still working hard on the 1.50 update, today we bring you a sneak peek of what is about to come! In cooperation with our friends from Scania Group, we will add the famous Scania Demo Centre, in Södertälje, Sweden, to ETS2! 🇸🇪 See more 👇

SCS Software

26,813 views • 2 years ago

Wondering what you missed in the first night of celebration of #NexusTurnsOne ? Here's a sneak peek of what it looks like to party with Nexus! We would like to thank everyone who joined us in our celebration! We'd love to see your pictures too! Don't forget to tag us when you post with #NexusTurnsOne

Wondering what you missed in the first night of celebration of #NexusTurnsOne ? Here's a sneak peek of what it looks like to party with Nexus! We would like to thank everyone who joined us in our celebration! We'd love to see your pictures too! Don't forget to tag us when you post with #NexusTurnsOne

Nexus Spa

30,587 views • 1 year ago

When we launched Flow in May, we had a simple hope: to see what you'd create. And you DELIVERED 🫡🫡🫡 We are blown away by the creativity and imagination of this community. Here are just a few recent favorites from our Discord community:

When we launched Flow in May, we had a simple hope: to see what you'd create. And you DELIVERED 🫡🫡🫡 We are blown away by the creativity and imagination of this community. Here are just a few recent favorites from our Discord community:

Google Labs

51,670 views • 1 year ago

StyleDrop: Text-to-Image Generation in Any Style introduce StyleDrop, a method that enables the synthesis of images that faithfully follow a specific style using a text-to-image model. The proposed method is extremely versatile and captures nuances and details of a user-provided style, such as color schemes, shading, design patterns, and local and global effects. It efficiently learns a new style by fine-tuning very few trainable parameters (less than 1% of total model parameters) and improving the quality via iterative training with either human or automated feedback. Better yet, StyleDrop is able to deliver impressive results even when the user supplies only a single image that specifies the desired style. An extensive study shows that, for the task of style tuning text-to-image models, StyleDrop implemented on Muse convincingly outperforms other methods, including DreamBooth and textual inversion on Imagen or Stable Diffusion. paper page:

StyleDrop: Text-to-Image Generation in Any Style introduce StyleDrop, a method that enables the synthesis of images that faithfully follow a specific style using a text-to-image model. The proposed method is extremely versatile and captures nuances and details of a user-provided style, such as color schemes, shading, design patterns, and local and global effects. It efficiently learns a new style by fine-tuning very few trainable parameters (less than 1% of total model parameters) and improving the quality via iterative training with either human or automated feedback. Better yet, StyleDrop is able to deliver impressive results even when the user supplies only a single image that specifies the desired style. An extensive study shows that, for the task of style tuning text-to-image models, StyleDrop implemented on Muse convincingly outperforms other methods, including DreamBooth and textual inversion on Imagen or Stable Diffusion. paper page:

AK

56,377 views • 3 years ago

When you think you’re about to give a testimonial about what it’s like to work for Virgin, but instead we sing your praises! 🎉 What better way to honour them than with a surprise that strikes a chord, quite literally. Have you got the secret sauce that we’re looking for? Our careers page is waiting for you

When you think you’re about to give a testimonial about what it’s like to work for Virgin, but instead we sing your praises! 🎉 What better way to honour them than with a surprise that strikes a chord, quite literally. Have you got the secret sauce that we’re looking for? Our careers page is waiting for you

Virgin

132,494 views • 2 years ago

Blended-NeRF: Zero-Shot Object Generation and Blending in Existing Neural Radiance Fields paper page: Editing a local region or a specific object in a 3D scene represented by a NeRF is challenging, mainly due to the implicit nature of the scene representation. Consistently blending a new realistic object into the scene adds an additional level of difficulty. We present Blended-NeRF, a robust and flexible framework for editing a specific region of interest in an existing NeRF scene, based on text prompts or image patches, along with a 3D ROI box. Our method leverages a pretrained language-image model to steer the synthesis towards a user-provided text prompt or image patch, along with a 3D MLP model initialized on an existing NeRF scene to generate the object and blend it into a specified region in the original scene. We allow local editing by localizing a 3D ROI box in the input scene, and seamlessly blend the content synthesized inside the ROI with the existing scene using a novel volumetric blending technique. To obtain natural looking and view-consistent results, we leverage existing and new geometric priors and 3D augmentations for improving the visual fidelity of the final result. We test our framework both qualitatively and quantitatively on a variety of real 3D scenes and text prompts, demonstrating realistic multi-view consistent results with much flexibility and diversity compared to the baselines. Finally, we show the applicability of our framework for several 3D editing applications, including adding new objects to a scene, removing/replacing/altering existing objects, and texture conversion.

Blended-NeRF: Zero-Shot Object Generation and Blending in Existing Neural Radiance Fields paper page: Editing a local region or a specific object in a 3D scene represented by a NeRF is challenging, mainly due to the implicit nature of the scene representation. Consistently blending a new realistic object into the scene adds an additional level of difficulty. We present Blended-NeRF, a robust and flexible framework for editing a specific region of interest in an existing NeRF scene, based on text prompts or image patches, along with a 3D ROI box. Our method leverages a pretrained language-image model to steer the synthesis towards a user-provided text prompt or image patch, along with a 3D MLP model initialized on an existing NeRF scene to generate the object and blend it into a specified region in the original scene. We allow local editing by localizing a 3D ROI box in the input scene, and seamlessly blend the content synthesized inside the ROI with the existing scene using a novel volumetric blending technique. To obtain natural looking and view-consistent results, we leverage existing and new geometric priors and 3D augmentations for improving the visual fidelity of the final result. We test our framework both qualitatively and quantitatively on a variety of real 3D scenes and text prompts, demonstrating realistic multi-view consistent results with much flexibility and diversity compared to the baselines. Finally, we show the applicability of our framework for several 3D editing applications, including adding new objects to a scene, removing/replacing/altering existing objects, and texture conversion.

AK

62,768 views • 3 years ago

🗒️ WIP Wednesday #12 🗒️ - Prototype Peek - Talking Process with our Creative Director 👀 For this weeks 'Work In Progress Wednesday' we'd like to hand you over to Benjamin Cleary our creative director & co-founder of Mr Kite. He wants to say hi & give you a peek at some of our prototype testing with some insight into the creative process when it comes to design. As always, everything you see is all a work in progress. This is a prototype!

🗒️ WIP Wednesday #12 🗒️ - Prototype Peek - Talking Process with our Creative Director 👀 For this weeks 'Work In Progress Wednesday' we'd like to hand you over to Benjamin Cleary our creative director & co-founder of Mr Kite. He wants to say hi & give you a peek at some of our prototype testing with some insight into the creative process when it comes to design. As always, everything you see is all a work in progress. This is a prototype!

Youmio

26,985 views • 3 years ago

If you're building a PDF RAG pipeline: Should you be using OCR and 𝘁𝗲𝘅𝘁-𝗯𝗮𝘀𝗲𝗱 𝗿𝗲𝘁𝗿𝗶𝗲𝘃𝗮𝗹 methods, or just 𝗲𝗺𝗯𝗲𝗱 𝗶𝗺𝗮𝗴𝗲𝘀 𝗱𝗶𝗿𝗲𝗰𝘁𝗹𝘆 using late interaction models? This paper says the answer might actually be 𝘣𝘰𝘵𝘩. My colleagues at Weaviate released IRPAPERS, a benchmark comparing 𝗶𝗺𝗮𝗴𝗲-𝗯𝗮𝘀𝗲𝗱 and 𝘁𝗲𝘅𝘁-𝗯𝗮𝘀𝗲𝗱 retrieval over 3,230 pages from 166 scientific papers. The setup: Take the same PDFs and process them two ways. For text, run OCR with GPT-4.1 and embed with Arctic 2.0 + BM25 hybrid search. For images, embed raw page images with ColModernVBERT multi-vector embeddings. Test both on 180 needle-in-the-haystack questions. 𝗧𝗵𝗲 𝗿𝗲𝘀𝘂𝗹𝘁𝘀: Text edges out images at the top rank: 46% vs 43% Recall@1 But images match or exceed text at deeper recall: 93% vs 91% Recall@20 But text and image based methods actually fail on 𝘥𝘪𝘧𝘧𝘦𝘳𝘦𝘯𝘁 𝘲𝘶𝘦𝘳𝘪𝘦𝘴. At Recall@1: • 22 queries succeed with text but fail with images • 18 queries succeed with images but fail with text This complementarity is what makes 𝗠𝘂𝗹𝘁𝗶𝗺𝗼𝗱𝗮𝗹 𝗛𝘆𝗯𝗿𝗶𝗱 𝗦𝗲𝗮𝗿𝗰𝗵 work. By fusing scores from both text and image retrieval, they achieved: • 49% Recall@1 (beating either modality alone) • 81% Recall@5 • 95% Recall@20 More in the video below 🔽 Dataset: Paper: Code:

If you're building a PDF RAG pipeline: Should you be using OCR and 𝘁𝗲𝘅𝘁-𝗯𝗮𝘀𝗲𝗱 𝗿𝗲𝘁𝗿𝗶𝗲𝘃𝗮𝗹 methods, or just 𝗲𝗺𝗯𝗲𝗱 𝗶𝗺𝗮𝗴𝗲𝘀 𝗱𝗶𝗿𝗲𝗰𝘁𝗹𝘆 using late interaction models? This paper says the answer might actually be 𝘣𝘰𝘵𝘩. My colleagues at Weaviate released IRPAPERS, a benchmark comparing 𝗶𝗺𝗮𝗴𝗲-𝗯𝗮𝘀𝗲𝗱 and 𝘁𝗲𝘅𝘁-𝗯𝗮𝘀𝗲𝗱 retrieval over 3,230 pages from 166 scientific papers. The setup: Take the same PDFs and process them two ways. For text, run OCR with GPT-4.1 and embed with Arctic 2.0 + BM25 hybrid search. For images, embed raw page images with ColModernVBERT multi-vector embeddings. Test both on 180 needle-in-the-haystack questions. 𝗧𝗵𝗲 𝗿𝗲𝘀𝘂𝗹𝘁𝘀: Text edges out images at the top rank: 46% vs 43% Recall@1 But images match or exceed text at deeper recall: 93% vs 91% Recall@20 But text and image based methods actually fail on 𝘥𝘪𝘧𝘧𝘦𝘳𝘦𝘯𝘁 𝘲𝘶𝘦𝘳𝘪𝘦𝘴. At Recall@1: • 22 queries succeed with text but fail with images • 18 queries succeed with images but fail with text This complementarity is what makes 𝗠𝘂𝗹𝘁𝗶𝗺𝗼𝗱𝗮𝗹 𝗛𝘆𝗯𝗿𝗶𝗱 𝗦𝗲𝗮𝗿𝗰𝗵 work. By fusing scores from both text and image retrieval, they achieved: • 49% Recall@1 (beating either modality alone) • 81% Recall@5 • 95% Recall@20 More in the video below 🔽 Dataset: Paper: Code:

Victoria Slocum

44,145 views • 4 months ago

You have a new Dev Playground for testing existing and upcoming FLUX models: 𝗖𝗼𝗺𝗽𝗮𝗿𝗲 𝗺𝗼𝗱𝗲𝗹𝘀 𝘀𝗶𝗱𝗲-𝗯𝘆-𝘀𝗶𝗱𝗲. Run the same prompt across multiple FLUX models at once and see the differences. 𝗖𝗼𝗻𝘁𝗿𝗼𝗹 𝗽𝗮𝗿𝗮𝗺𝗲𝘁𝗲𝗿𝘀. Adjust API parameters from the UI, so what you test is exactly what you build. 𝗘𝘃𝗮𝗹𝘂𝗮𝘁𝗲. Test quality, latency, and cost. 𝗔𝗹𝘄𝗮𝘆𝘀 𝗰𝘂𝗿𝗿𝗲𝗻𝘁. It launches with our latest models and we'll keep adding new models. Link to try it below.

You have a new Dev Playground for testing existing and upcoming FLUX models: 𝗖𝗼𝗺𝗽𝗮𝗿𝗲 𝗺𝗼𝗱𝗲𝗹𝘀 𝘀𝗶𝗱𝗲-𝗯𝘆-𝘀𝗶𝗱𝗲. Run the same prompt across multiple FLUX models at once and see the differences. 𝗖𝗼𝗻𝘁𝗿𝗼𝗹 𝗽𝗮𝗿𝗮𝗺𝗲𝘁𝗲𝗿𝘀. Adjust API parameters from the UI, so what you test is exactly what you build. 𝗘𝘃𝗮𝗹𝘂𝗮𝘁𝗲. Test quality, latency, and cost. 𝗔𝗹𝘄𝗮𝘆𝘀 𝗰𝘂𝗿𝗿𝗲𝗻𝘁. It launches with our latest models and we'll keep adding new models. Link to try it below.

Black Forest Labs

14,788 views • 1 month ago

I want to experiment with short form video. Here’s hasty attempt #1, a 1 min interview with Mint portco Kaaj, who happened to be in our office when we came up with the idea. It’s rough- I’d love feedback. What should we do better? What would you like to see from me?

I want to experiment with short form video. Here’s hasty attempt #1, a 1 min interview with Mint portco Kaaj, who happened to be in our office when we came up with the idea. It’s rough- I’d love feedback. What should we do better? What would you like to see from me?

Sheel Mohnot

10,190 views • 1 year ago

Welcome to 2026! We kick off the New Year with a mini DF Direct, looking back at the best hardware of 2025 and what we expect to see from 2026. Oh... and our first hands-on of the remarkable GPD Win 5 Strix Halo handheld:

Welcome to 2026! We kick off the New Year with a mini DF Direct, looking back at the best hardware of 2025 and what we expect to see from 2026. Oh... and our first hands-on of the remarkable GPD Win 5 Strix Halo handheld:

Digital Foundry

85,916 views • 7 months ago