✨New preprint: Dual-Process Image Generation! We distill *feedback from a VLM* into *feed-forward image generation*, at inference time. The result is flexible control: parameterize tasks as multimodal inputs, visually inspect the images with the VLM, and update the generator.🧵

(2/n) Our method re-uses the visual instruction tuning loss originally used to train the VLM, to instead optimize the weights of the image generator.

(3/n) You can get pretty creative, because VLMs afford a flexible interface. We play around with implementing spatial controls via *visual prompting*, where we overlay the control over the image and ask the VLM if they match, then optimize for an image that matches the control.

(4/n) Check out our paper + code! Our codebase lets you mix-and-match different off-the-shelf image generators and VLMs, and can run on Nvidia RTX 4090s. Page: Paper: Code:

(5/n) Research is non-linear! We started this project more than a year ago – first by training a diffusion captioner (a VLM that encodes images with diffusion hyperfeatures). We’re not working on that direction anymore, but here’s a peek at that first prototype:

(6/n) Special thanks to my co-authors @jongranskog, @holynski_, @trevordarrell! This was a great academic collaboration with @runwayml, especially @agermanidis, who was super supportive of our very experimental ideas throughout the entire process.

congrats, really great work

Nice!

Cool work! Did you try to optimize the prior noise sample instead of the generator?

Yes! Check out the paper to see a controlled comparison of the efficiency of the two settings -- you can either update the noise or the weights with the same gradient. One cool thing about weight optimization, unlike noise optimization, is that it generalizes!

Love it!

This is very cool! Kinda reminded me a bit of CycleGANs with all the latent-space self-consistency training.

@ariG23498 check this out

Thanks for using CommonsenseT2I! Interested in how sensitive the gradient is to how the VQA questions are phrased

Thanks for releasing CommonsenseT2I; it's a great benchmark! You might be interested in Table 4 of the Appendix (which compares auto-generated vs. 'ground-truth' VQA questions). We didn't study phrasing, but that's a good idea!

Congrats @graceluo_ 🫶 Super cool work as usual 😉

Very cool work

Incredible that the backprop at inference time is so smooth - it’s escaping local minima surprisingly quickly. Very nice work

Very interesting trick. I am glad it even works, opens so many possibilities. Quote 1: "We primarily word questions such that they have a clear Yes or No answer, and append “Answer with Yes or No.” to the question. We find this yields interpretable probabilities for the extent to which the control was satisfied, simply by exponentiating the VQA loss.... " Quote 2: "Most importantly, we still compute gradients through the VLM, which gives a denser signal than reinforcement learning." Quote3: how they compute loss

Cool work!

nice work!!

Congrats! The results are very compelling!

Amazing work! The results are so fun to look through

Very cool! ..I've had a feeling that openai's new image gen worked a little like this actually, what do you think?

I actually don’t think this is how they do it, but maybe it’ll be in the next version 😉

Implicit feedback no prompting brilliant applicable across AI model to model sharing latent / conceptual information with no intermediary.

Cool results!

Hi there! Your work on dual-process image generation is fascinating. It reminded me of our recent paper, " Learning Flow Fields in Attention for Controllable Person Image Generation,"The link is:

AI artistry just leveled up. Bravo!

@yyujjinii 😆😆🤗🤗