Gezinmeyi atla
LLM-grounded Diffusion: Enhancing Prompt Understanding of Text-to-Image Diffusion Models with Large…
0:00 / 5:11

LLM-grounded Diffusion: Enhancing Prompt Understanding of Text-to-Image Diffusion Models with Large…

AK

@_akhaliq524 B takipçi

X profili
367 beğeniİndirOrijinal gönderi

83.681 görüntüleme

LLM-grounded Diffusion: Enhancing Prompt Understanding of Text-to-Image Diffusion Models with Large Language Models paper page: github: Recent advancements in text-to-image generation with diffusion models have yielded remarkable results synthesizing highly realistic and diverse images. However, these models still encounter difficulties when generating images from prompts that demand spatial or common sense reasoning. We propose to equip diffusion models with enhanced reasoning capabilities by using off-the-shelf pretrained large language models (LLMs) in a novel two-stage generation process. First, we adapt an LLM to be a text-guided layout generator through in-context learning. When provided with an image prompt, an LLM outputs a scene layout in the form of bounding boxes along with corresponding individual descriptions. Second, we steer a diffusion model with a novel controller to generate images conditioned on the layout. Both stages utilize frozen pretrained models without any LLM or diffusion model parameter optimization. We validate the superiority of our design by demonstrating its ability to outperform the base diffusion model in accurately generating images according to prompts that necessitate both language and spatial reasoning. Additionally, our method naturally allows dialog-based scene specification and is able to handle prompts in a language that is not well-supported by the underlying diffusion model.

6 Yorum

Yorumlar X (Twitter) üzerindeki orijinal gönderiden alınmıştır

  • Boyi Li profil fotoğrafı

    @Boyiliee

    Thanks @_akhaliq for sharing our work!

    3 beğeni

  • zorr0 (ττ) profil fotoğrafı

    @aizorr0

    @replytensor

    2 beğeni

  • haareblond profil fotoğrafı

    @HaareBlond

    cool but still feels hacky

    1 beğeni

  • Takomo AI profil fotoğrafı

    @Takomo_ai

    That's great progress!

  • Cavit Erginsoy profil fotoğrafı

    @caviterginsoy

    @yuliangxiu I saw this about a month ago and had played around with it, is the same or a parallel dev? Wish someone built an extension for A1111

  • VIJAY KUMAR REDDY BOMMIREDDY profil fotoğrafı

    @Vijay60606060

    Impressive work! Expanding the text-to-image domain with diffusion models showcases great potential. Looking forward to exploring the paper and GitHub repository. Keep up the great work! 👍