Loading video...

Video Failed to Load

Go Home

Can a small academic team build a strong text-to-image model using only public datasets? Introducing i1: a simple, fully open recipe for strong text-to-image models

71,178 views • 3 months ago •via X (Twitter)

21 Comments

Zhuang Liu's profile picture
Zhuang Liu3 months ago

We study the design space of text-to-image diffusion models w/ 300+ controlled experiments. The insights are combined to build i1-3B, which outperforms the best prior fully open model by 29.5 absolute percentage points on average across 5 benchmarks.

Zhuang Liu's profile picture
Zhuang Liu3 months ago

We release the checkpoint, training/inference code, dataset, data pipeline, and full recipe. This is not just another checkpoint. It is a strong baseline people can study, reproduce, modify, and build on.

Zhuang Liu's profile picture
Zhuang Liu3 months ago

At 1024px, i1-3B is competitive with leading open-weight T2I models across our benchmark suite, and outperforms models like FLUX.1 [Dev] and HiDream-I1-Full. This is not just on older benchmarks (GenEval and DPG-Bench). It also holds on newer benchmarks (PRISM, CVTG-2K, and LongText-Bench) used by leading models (e.g., Z-Image, Ideogram 4.0) and better reflect modern T2I capabilities.

Zhuang Liu's profile picture
Zhuang Liu3 months ago

Alongside strong general image generation, i1-3B is the first fully open T2I model with competitive text rendering performance (to the best of our knowledge). With 3B parameters, it scores 92% on LongText-Bench. For comparison, the best prior fully open model scores 15%.

Zhuang Liu's profile picture
Zhuang Liu3 months ago

The strong performance of i1-3B comes from 300+ controlled experiments carefully ablating modeling and data designs. We highlight some of our findings below:

Zhuang Liu's profile picture
Zhuang Liu3 months ago

Modeling findings: 1) a single strong text encoder + larger text encoder adapter > combining multiple text encoders 2) AdaLN provides marginal benefit for T2I relative to its large number of parameters 3) dual-stream DiT + long skip connections is a strong backbone design

Zhuang Liu's profile picture
Zhuang Liu3 months ago

Data findings: 1) train on long captions + inference-time prompt rewrite > train on short captions, even for short inference prompts 2) equal weighting is a strong default for mixing curated datasets 3) with a diverse mix of datasets, fewer images and more epochs don’t hurt performance much 4) strong high-resolution performance doesn't require broad coverage in high-resolution training data

Zhuang Liu's profile picture
Zhuang Liu3 months ago

i1 shows that strong text-to-image models do not have to rely on opaque recipes, inaccessible data, or highly specialized new components. A simple, fully open recipe built from public datasets and carefully validated design choices can be competitive with leading open-weight models. Find many more results and analyses in our paper! Led by @zeng_boya with many others contributing, Tianze Luo, Jucheng Shen, Shu Pu, @TaimingLu and @GabrielSarch

Zhuang Liu's profile picture
Zhuang Liu3 months ago

Check i1 out: Model: Data: Project Page: Code: Paper:

Lucas Beyer (bl16)'s profile picture
Lucas Beyer (bl16)3 months ago

@XingyuFu2 Nice work and write-up, congrats!

Zhuang Liu's profile picture
Zhuang Liu3 months ago

@XingyuFu2 Thanks Lucas!

Saining Xie's profile picture
Saining Xie3 months ago

great work!

Grigoris Chrysos's profile picture
Grigoris Chrysos3 months ago

Great work!

Hang Gao's profile picture
Hang Gao3 months ago

Nice results and ablations! Congrats!

Nikhil Lamba's profile picture
Nikhil Lamba3 months ago

Super cool

Nikhil Lamba's profile picture
Nikhil Lamba3 months ago

Definitely the trend seems the same. As it was for CNNs and vision. First very deep models, then the refined ones with only imp layers that can run on edge devices.

kenji's profile picture
kenji3 months ago

@jon_barron Do the team plan to release 1b model soon?

Jiawei (Joe) Zhou's profile picture
Jiawei (Joe) Zhou3 months ago

Great work for democratizing T2I model training! Will definitely take a look.

XuDong Wang's profile picture
XuDong Wang3 months ago

Nice work Zhuang! Congratulations!

Zhuang Liu's profile picture
Zhuang Liu3 months ago

Thanks Xudong!

Saad Gul's profile picture
Saad Gul3 months ago

@ricklamers 😎

Related Videos

[CLIP] by Hand ✍️ The CLIP (Contrastive Language–Image Pre-training) model, a groundbreaking work by OpenAI, redefines the intersection of computer vision and natural language processing. It is the basis of all the multi-modal foundation models we see today. How does CLIP work? Goal: 🟨 Learn a shared embedding space for text and image [1] Given ↳ A mini batch of 3 text-image pairs ↳ OpenAI used 400 million text-image pairs to train its original CLIP model. Process 1st pair: "big table" [2] 🟪 Text → 2 Vectors (3D) ↳ Look up word embedding vectors using word2vec. [3] 🟩 Image → 2 Vectors (4D) ↳ Divide the image into two patches. ↳ Flatten each patch [4] Process other pairs ↳ Repeat [2]-[3] [5] 🟪 Text Encoder & 🟩 Image Encoder ↳ Encode input vectors into feature vectors ↳ Here, both encoders are simple one layer perceptron (linear + ReLU) ↳ In practice, the encoders are usually transformer models. [6] 🟪 🟩 Mean Pooling: 2 → 1 vector ↳ Average 2 feature vectors into a single vector by averaging across the columns ↳ The goal is to have one vector to represent each image or text [7] 🟪 🟩 -> 🟨 Projection ↳ Note that the text and image feature vectors from the encoders have different dimensions (3D vs. 4D). ↳ Use a linear layer to project image and text vectors to a 2D shared embedding space. 🏋️ Contrastive Pre-training 🏋️ [8] Prepare for MatMul ↳ Copy text vectors (T1,T2,T3) ↳ Copy the transpose of image vectors (I1,I2,I3) ↳ They are all in the 2D shared embedding space. [9] 🟦 MatMul ↳ Multiply T and I matrices. ↳ This is equivalent to taking dot product between every pair of image and text vectors. ↳ The purpose is to use dot product to estimate the similarity between a pair of image-text. [10] 🟦 Softmax: e^x ↳ Raise e to the power of the number in each cell ↳ To simplify hand calculation, we approximate e^□ with 3^□. [11] 🟦 Softmax: ∑ ↳ Sum each row for 🟩 image→🟪 text ↳ Sum each column for 🟪 text→ 🟩 image [12] 🟦 Softmax: 1 / sum ↳ Divide each element by the column sum to obtain a similarity matrix for 🟪 text→🟩 image ↳ Divide each element by the row sum to obtain a similarity matrix for 🟩 image→🟪 text [13] 🟥 Loss Gradients ↳ The "Targets" for the similarity matrices are Identity Matrices. ↳ Why? If I and T come from the same pair (i=j), we want the highest value, which is 1, and 0 otherwise. ↳ Apply the simple equation of [Similarity - Target] to compute gradients of for both directions. ↳ Why so simple? Because when Softmax and Cross-Entropy Loss are used together, the math magically works out that way. ↳ These gradients kick off the backpropagation process to update weights and biases of the encoders and projection layers (red borders).

Tom Yeh

67,896 views • 2 years ago