Video wird geladen...

Video konnte nicht geladen werden

Zur Startseite

Can a small academic team build a strong text-to-image model using only public datasets? Introducing i1: a simple, fully open recipe for strong text-to-image models

71,178 Aufrufe • vor 3 Monaten •via X (Twitter)

21 Kommentare

Profilbild von Zhuang Liu
Zhuang Liuvor 3 Monaten

We study the design space of text-to-image diffusion models w/ 300+ controlled experiments. The insights are combined to build i1-3B, which outperforms the best prior fully open model by 29.5 absolute percentage points on average across 5 benchmarks.

Profilbild von Zhuang Liu
Zhuang Liuvor 3 Monaten

We release the checkpoint, training/inference code, dataset, data pipeline, and full recipe. This is not just another checkpoint. It is a strong baseline people can study, reproduce, modify, and build on.

Profilbild von Zhuang Liu
Zhuang Liuvor 3 Monaten

At 1024px, i1-3B is competitive with leading open-weight T2I models across our benchmark suite, and outperforms models like FLUX.1 [Dev] and HiDream-I1-Full. This is not just on older benchmarks (GenEval and DPG-Bench). It also holds on newer benchmarks (PRISM, CVTG-2K, and LongText-Bench) used by leading models (e.g., Z-Image, Ideogram 4.0) and better reflect modern T2I capabilities.

Profilbild von Zhuang Liu
Zhuang Liuvor 3 Monaten

Alongside strong general image generation, i1-3B is the first fully open T2I model with competitive text rendering performance (to the best of our knowledge). With 3B parameters, it scores 92% on LongText-Bench. For comparison, the best prior fully open model scores 15%.

Profilbild von Zhuang Liu
Zhuang Liuvor 3 Monaten

The strong performance of i1-3B comes from 300+ controlled experiments carefully ablating modeling and data designs. We highlight some of our findings below:

Profilbild von Zhuang Liu
Zhuang Liuvor 3 Monaten

Modeling findings: 1) a single strong text encoder + larger text encoder adapter > combining multiple text encoders 2) AdaLN provides marginal benefit for T2I relative to its large number of parameters 3) dual-stream DiT + long skip connections is a strong backbone design

Profilbild von Zhuang Liu
Zhuang Liuvor 3 Monaten

Data findings: 1) train on long captions + inference-time prompt rewrite > train on short captions, even for short inference prompts 2) equal weighting is a strong default for mixing curated datasets 3) with a diverse mix of datasets, fewer images and more epochs don’t hurt performance much 4) strong high-resolution performance doesn't require broad coverage in high-resolution training data

Profilbild von Zhuang Liu
Zhuang Liuvor 3 Monaten

i1 shows that strong text-to-image models do not have to rely on opaque recipes, inaccessible data, or highly specialized new components. A simple, fully open recipe built from public datasets and carefully validated design choices can be competitive with leading open-weight models. Find many more results and analyses in our paper! Led by @zeng_boya with many others contributing, Tianze Luo, Jucheng Shen, Shu Pu, @TaimingLu and @GabrielSarch

Profilbild von Zhuang Liu
Zhuang Liuvor 3 Monaten

Check i1 out: Model: Data: Project Page: Code: Paper:

Profilbild von Lucas Beyer (bl16)
Lucas Beyer (bl16)vor 3 Monaten

@XingyuFu2 Nice work and write-up, congrats!

Profilbild von Zhuang Liu
Zhuang Liuvor 3 Monaten

@XingyuFu2 Thanks Lucas!

Profilbild von Saining Xie
Saining Xievor 3 Monaten

great work!

Profilbild von Grigoris Chrysos
Grigoris Chrysosvor 3 Monaten

Great work!

Profilbild von Hang Gao
Hang Gaovor 3 Monaten

Nice results and ablations! Congrats!

Profilbild von Nikhil Lamba
Nikhil Lambavor 3 Monaten

Super cool

Profilbild von Nikhil Lamba
Nikhil Lambavor 3 Monaten

Definitely the trend seems the same. As it was for CNNs and vision. First very deep models, then the refined ones with only imp layers that can run on edge devices.

Profilbild von kenji
kenjivor 3 Monaten

@jon_barron Do the team plan to release 1b model soon?

Profilbild von Jiawei (Joe) Zhou
Jiawei (Joe) Zhouvor 3 Monaten

Great work for democratizing T2I model training! Will definitely take a look.

Profilbild von XuDong Wang
XuDong Wangvor 3 Monaten

Nice work Zhuang! Congratulations!

Profilbild von Zhuang Liu
Zhuang Liuvor 3 Monaten

Thanks Xudong!

Profilbild von Saad Gul
Saad Gulvor 3 Monaten

@ricklamers 😎

Ähnliche Videos

[CLIP] by Hand ✍️ The CLIP (Contrastive Language–Image Pre-training) model, a groundbreaking work by OpenAI, redefines the intersection of computer vision and natural language processing. It is the basis of all the multi-modal foundation models we see today. How does CLIP work? Goal: 🟨 Learn a shared embedding space for text and image [1] Given ↳ A mini batch of 3 text-image pairs ↳ OpenAI used 400 million text-image pairs to train its original CLIP model. Process 1st pair: "big table" [2] 🟪 Text → 2 Vectors (3D) ↳ Look up word embedding vectors using word2vec. [3] 🟩 Image → 2 Vectors (4D) ↳ Divide the image into two patches. ↳ Flatten each patch [4] Process other pairs ↳ Repeat [2]-[3] [5] 🟪 Text Encoder & 🟩 Image Encoder ↳ Encode input vectors into feature vectors ↳ Here, both encoders are simple one layer perceptron (linear + ReLU) ↳ In practice, the encoders are usually transformer models. [6] 🟪 🟩 Mean Pooling: 2 → 1 vector ↳ Average 2 feature vectors into a single vector by averaging across the columns ↳ The goal is to have one vector to represent each image or text [7] 🟪 🟩 -> 🟨 Projection ↳ Note that the text and image feature vectors from the encoders have different dimensions (3D vs. 4D). ↳ Use a linear layer to project image and text vectors to a 2D shared embedding space. 🏋️ Contrastive Pre-training 🏋️ [8] Prepare for MatMul ↳ Copy text vectors (T1,T2,T3) ↳ Copy the transpose of image vectors (I1,I2,I3) ↳ They are all in the 2D shared embedding space. [9] 🟦 MatMul ↳ Multiply T and I matrices. ↳ This is equivalent to taking dot product between every pair of image and text vectors. ↳ The purpose is to use dot product to estimate the similarity between a pair of image-text. [10] 🟦 Softmax: e^x ↳ Raise e to the power of the number in each cell ↳ To simplify hand calculation, we approximate e^□ with 3^□. [11] 🟦 Softmax: ∑ ↳ Sum each row for 🟩 image→🟪 text ↳ Sum each column for 🟪 text→ 🟩 image [12] 🟦 Softmax: 1 / sum ↳ Divide each element by the column sum to obtain a similarity matrix for 🟪 text→🟩 image ↳ Divide each element by the row sum to obtain a similarity matrix for 🟩 image→🟪 text [13] 🟥 Loss Gradients ↳ The "Targets" for the similarity matrices are Identity Matrices. ↳ Why? If I and T come from the same pair (i=j), we want the highest value, which is 1, and 0 otherwise. ↳ Apply the simple equation of [Similarity - Target] to compute gradients of for both directions. ↳ Why so simple? Because when Softmax and Cross-Entropy Loss are used together, the math magically works out that way. ↳ These gradients kick off the backpropagation process to update weights and biases of the encoders and projection layers (red borders).

Tom Yeh

67,896 Aufrufe • vor 2 Jahren