Video yükleniyor...

Video Yüklenemedi

Ana Sayfaya Dön

Can a small academic team build a strong text-to-image model using only public datasets? Introducing i1: a simple, fully open recipe for strong text-to-image models

71,178 görüntüleme • 3 ay önce •via X (Twitter)

21 Yorum

Zhuang Liu profil fotoğrafı
Zhuang Liu3 ay önce

We study the design space of text-to-image diffusion models w/ 300+ controlled experiments. The insights are combined to build i1-3B, which outperforms the best prior fully open model by 29.5 absolute percentage points on average across 5 benchmarks.

Zhuang Liu profil fotoğrafı
Zhuang Liu3 ay önce

We release the checkpoint, training/inference code, dataset, data pipeline, and full recipe. This is not just another checkpoint. It is a strong baseline people can study, reproduce, modify, and build on.

Zhuang Liu profil fotoğrafı
Zhuang Liu3 ay önce

At 1024px, i1-3B is competitive with leading open-weight T2I models across our benchmark suite, and outperforms models like FLUX.1 [Dev] and HiDream-I1-Full. This is not just on older benchmarks (GenEval and DPG-Bench). It also holds on newer benchmarks (PRISM, CVTG-2K, and LongText-Bench) used by leading models (e.g., Z-Image, Ideogram 4.0) and better reflect modern T2I capabilities.

Zhuang Liu profil fotoğrafı
Zhuang Liu3 ay önce

Alongside strong general image generation, i1-3B is the first fully open T2I model with competitive text rendering performance (to the best of our knowledge). With 3B parameters, it scores 92% on LongText-Bench. For comparison, the best prior fully open model scores 15%.

Zhuang Liu profil fotoğrafı
Zhuang Liu3 ay önce

The strong performance of i1-3B comes from 300+ controlled experiments carefully ablating modeling and data designs. We highlight some of our findings below:

Zhuang Liu profil fotoğrafı
Zhuang Liu3 ay önce

Modeling findings: 1) a single strong text encoder + larger text encoder adapter > combining multiple text encoders 2) AdaLN provides marginal benefit for T2I relative to its large number of parameters 3) dual-stream DiT + long skip connections is a strong backbone design

Zhuang Liu profil fotoğrafı
Zhuang Liu3 ay önce

Data findings: 1) train on long captions + inference-time prompt rewrite > train on short captions, even for short inference prompts 2) equal weighting is a strong default for mixing curated datasets 3) with a diverse mix of datasets, fewer images and more epochs don’t hurt performance much 4) strong high-resolution performance doesn't require broad coverage in high-resolution training data

Zhuang Liu profil fotoğrafı
Zhuang Liu3 ay önce

i1 shows that strong text-to-image models do not have to rely on opaque recipes, inaccessible data, or highly specialized new components. A simple, fully open recipe built from public datasets and carefully validated design choices can be competitive with leading open-weight models. Find many more results and analyses in our paper! Led by @zeng_boya with many others contributing, Tianze Luo, Jucheng Shen, Shu Pu, @TaimingLu and @GabrielSarch

Zhuang Liu profil fotoğrafı
Zhuang Liu3 ay önce

Check i1 out: Model: Data: Project Page: Code: Paper:

Lucas Beyer (bl16) profil fotoğrafı
Lucas Beyer (bl16)3 ay önce

@XingyuFu2 Nice work and write-up, congrats!

Zhuang Liu profil fotoğrafı
Zhuang Liu3 ay önce

@XingyuFu2 Thanks Lucas!

Saining Xie profil fotoğrafı
Saining Xie3 ay önce

great work!

Grigoris Chrysos profil fotoğrafı
Grigoris Chrysos3 ay önce

Great work!

Hang Gao profil fotoğrafı
Hang Gao3 ay önce

Nice results and ablations! Congrats!

Nikhil Lamba profil fotoğrafı
Nikhil Lamba3 ay önce

Super cool

Nikhil Lamba profil fotoğrafı
Nikhil Lamba3 ay önce

Definitely the trend seems the same. As it was for CNNs and vision. First very deep models, then the refined ones with only imp layers that can run on edge devices.

kenji profil fotoğrafı
kenji3 ay önce

@jon_barron Do the team plan to release 1b model soon?

Jiawei (Joe) Zhou profil fotoğrafı
Jiawei (Joe) Zhou3 ay önce

Great work for democratizing T2I model training! Will definitely take a look.

XuDong Wang profil fotoğrafı
XuDong Wang3 ay önce

Nice work Zhuang! Congratulations!

Zhuang Liu profil fotoğrafı
Zhuang Liu3 ay önce

Thanks Xudong!

Saad Gul profil fotoğrafı
Saad Gul3 ay önce

@ricklamers 😎

Benzer Videolar

[CLIP] by Hand ✍️ The CLIP (Contrastive Language–Image Pre-training) model, a groundbreaking work by OpenAI, redefines the intersection of computer vision and natural language processing. It is the basis of all the multi-modal foundation models we see today. How does CLIP work? Goal: 🟨 Learn a shared embedding space for text and image [1] Given ↳ A mini batch of 3 text-image pairs ↳ OpenAI used 400 million text-image pairs to train its original CLIP model. Process 1st pair: "big table" [2] 🟪 Text → 2 Vectors (3D) ↳ Look up word embedding vectors using word2vec. [3] 🟩 Image → 2 Vectors (4D) ↳ Divide the image into two patches. ↳ Flatten each patch [4] Process other pairs ↳ Repeat [2]-[3] [5] 🟪 Text Encoder & 🟩 Image Encoder ↳ Encode input vectors into feature vectors ↳ Here, both encoders are simple one layer perceptron (linear + ReLU) ↳ In practice, the encoders are usually transformer models. [6] 🟪 🟩 Mean Pooling: 2 → 1 vector ↳ Average 2 feature vectors into a single vector by averaging across the columns ↳ The goal is to have one vector to represent each image or text [7] 🟪 🟩 -> 🟨 Projection ↳ Note that the text and image feature vectors from the encoders have different dimensions (3D vs. 4D). ↳ Use a linear layer to project image and text vectors to a 2D shared embedding space. 🏋️ Contrastive Pre-training 🏋️ [8] Prepare for MatMul ↳ Copy text vectors (T1,T2,T3) ↳ Copy the transpose of image vectors (I1,I2,I3) ↳ They are all in the 2D shared embedding space. [9] 🟦 MatMul ↳ Multiply T and I matrices. ↳ This is equivalent to taking dot product between every pair of image and text vectors. ↳ The purpose is to use dot product to estimate the similarity between a pair of image-text. [10] 🟦 Softmax: e^x ↳ Raise e to the power of the number in each cell ↳ To simplify hand calculation, we approximate e^□ with 3^□. [11] 🟦 Softmax: ∑ ↳ Sum each row for 🟩 image→🟪 text ↳ Sum each column for 🟪 text→ 🟩 image [12] 🟦 Softmax: 1 / sum ↳ Divide each element by the column sum to obtain a similarity matrix for 🟪 text→🟩 image ↳ Divide each element by the row sum to obtain a similarity matrix for 🟩 image→🟪 text [13] 🟥 Loss Gradients ↳ The "Targets" for the similarity matrices are Identity Matrices. ↳ Why? If I and T come from the same pair (i=j), we want the highest value, which is 1, and 0 otherwise. ↳ Apply the simple equation of [Similarity - Target] to compute gradients of for both directions. ↳ Why so simple? Because when Softmax and Cross-Entropy Loss are used together, the math magically works out that way. ↳ These gradients kick off the backpropagation process to update weights and biases of the encoders and projection layers (red borders).

Tom Yeh

67,896 görüntüleme • 2 yıl önce