正在加载视频...

视频加载失败

Can a small academic team build a strong text-to-image model using only public datasets? Introducing i1: a simple, fully open recipe for strong text-to-image models

71,178 次观看 • 3 个月前 •via X (Twitter)

21 条评论

Zhuang Liu 的头像
Zhuang Liu3 个月前

We study the design space of text-to-image diffusion models w/ 300+ controlled experiments. The insights are combined to build i1-3B, which outperforms the best prior fully open model by 29.5 absolute percentage points on average across 5 benchmarks.

Zhuang Liu 的头像
Zhuang Liu3 个月前

We release the checkpoint, training/inference code, dataset, data pipeline, and full recipe. This is not just another checkpoint. It is a strong baseline people can study, reproduce, modify, and build on.

Zhuang Liu 的头像
Zhuang Liu3 个月前

At 1024px, i1-3B is competitive with leading open-weight T2I models across our benchmark suite, and outperforms models like FLUX.1 [Dev] and HiDream-I1-Full. This is not just on older benchmarks (GenEval and DPG-Bench). It also holds on newer benchmarks (PRISM, CVTG-2K, and LongText-Bench) used by leading models (e.g., Z-Image, Ideogram 4.0) and better reflect modern T2I capabilities.

Zhuang Liu 的头像
Zhuang Liu3 个月前

Alongside strong general image generation, i1-3B is the first fully open T2I model with competitive text rendering performance (to the best of our knowledge). With 3B parameters, it scores 92% on LongText-Bench. For comparison, the best prior fully open model scores 15%.

Zhuang Liu 的头像
Zhuang Liu3 个月前

The strong performance of i1-3B comes from 300+ controlled experiments carefully ablating modeling and data designs. We highlight some of our findings below:

Zhuang Liu 的头像
Zhuang Liu3 个月前

Modeling findings: 1) a single strong text encoder + larger text encoder adapter > combining multiple text encoders 2) AdaLN provides marginal benefit for T2I relative to its large number of parameters 3) dual-stream DiT + long skip connections is a strong backbone design

Zhuang Liu 的头像
Zhuang Liu3 个月前

Data findings: 1) train on long captions + inference-time prompt rewrite > train on short captions, even for short inference prompts 2) equal weighting is a strong default for mixing curated datasets 3) with a diverse mix of datasets, fewer images and more epochs don’t hurt performance much 4) strong high-resolution performance doesn't require broad coverage in high-resolution training data

Zhuang Liu 的头像
Zhuang Liu3 个月前

i1 shows that strong text-to-image models do not have to rely on opaque recipes, inaccessible data, or highly specialized new components. A simple, fully open recipe built from public datasets and carefully validated design choices can be competitive with leading open-weight models. Find many more results and analyses in our paper! Led by @zeng_boya with many others contributing, Tianze Luo, Jucheng Shen, Shu Pu, @TaimingLu and @GabrielSarch

Zhuang Liu 的头像
Zhuang Liu3 个月前

Check i1 out: Model: Data: Project Page: Code: Paper:

Lucas Beyer (bl16) 的头像
Lucas Beyer (bl16)3 个月前

@XingyuFu2 Nice work and write-up, congrats!

Zhuang Liu 的头像
Zhuang Liu3 个月前

@XingyuFu2 Thanks Lucas!

Saining Xie 的头像
Saining Xie3 个月前

great work!

Grigoris Chrysos 的头像
Grigoris Chrysos3 个月前

Great work!

Hang Gao 的头像
Hang Gao3 个月前

Nice results and ablations! Congrats!

Nikhil Lamba 的头像
Nikhil Lamba3 个月前

Super cool

Nikhil Lamba 的头像
Nikhil Lamba3 个月前

Definitely the trend seems the same. As it was for CNNs and vision. First very deep models, then the refined ones with only imp layers that can run on edge devices.

kenji 的头像
kenji3 个月前

@jon_barron Do the team plan to release 1b model soon?

Jiawei (Joe) Zhou 的头像
Jiawei (Joe) Zhou3 个月前

Great work for democratizing T2I model training! Will definitely take a look.

XuDong Wang 的头像
XuDong Wang3 个月前

Nice work Zhuang! Congratulations!

Zhuang Liu 的头像
Zhuang Liu3 个月前

Thanks Xudong!

Saad Gul 的头像
Saad Gul3 个月前

@ricklamers 😎

相关视频

[CLIP] by Hand ✍️ The CLIP (Contrastive Language–Image Pre-training) model, a groundbreaking work by OpenAI, redefines the intersection of computer vision and natural language processing. It is the basis of all the multi-modal foundation models we see today. How does CLIP work? Goal: 🟨 Learn a shared embedding space for text and image [1] Given ↳ A mini batch of 3 text-image pairs ↳ OpenAI used 400 million text-image pairs to train its original CLIP model. Process 1st pair: "big table" [2] 🟪 Text → 2 Vectors (3D) ↳ Look up word embedding vectors using word2vec. [3] 🟩 Image → 2 Vectors (4D) ↳ Divide the image into two patches. ↳ Flatten each patch [4] Process other pairs ↳ Repeat [2]-[3] [5] 🟪 Text Encoder & 🟩 Image Encoder ↳ Encode input vectors into feature vectors ↳ Here, both encoders are simple one layer perceptron (linear + ReLU) ↳ In practice, the encoders are usually transformer models. [6] 🟪 🟩 Mean Pooling: 2 → 1 vector ↳ Average 2 feature vectors into a single vector by averaging across the columns ↳ The goal is to have one vector to represent each image or text [7] 🟪 🟩 -> 🟨 Projection ↳ Note that the text and image feature vectors from the encoders have different dimensions (3D vs. 4D). ↳ Use a linear layer to project image and text vectors to a 2D shared embedding space. 🏋️ Contrastive Pre-training 🏋️ [8] Prepare for MatMul ↳ Copy text vectors (T1,T2,T3) ↳ Copy the transpose of image vectors (I1,I2,I3) ↳ They are all in the 2D shared embedding space. [9] 🟦 MatMul ↳ Multiply T and I matrices. ↳ This is equivalent to taking dot product between every pair of image and text vectors. ↳ The purpose is to use dot product to estimate the similarity between a pair of image-text. [10] 🟦 Softmax: e^x ↳ Raise e to the power of the number in each cell ↳ To simplify hand calculation, we approximate e^□ with 3^□. [11] 🟦 Softmax: ∑ ↳ Sum each row for 🟩 image→🟪 text ↳ Sum each column for 🟪 text→ 🟩 image [12] 🟦 Softmax: 1 / sum ↳ Divide each element by the column sum to obtain a similarity matrix for 🟪 text→🟩 image ↳ Divide each element by the row sum to obtain a similarity matrix for 🟩 image→🟪 text [13] 🟥 Loss Gradients ↳ The "Targets" for the similarity matrices are Identity Matrices. ↳ Why? If I and T come from the same pair (i=j), we want the highest value, which is 1, and 0 otherwise. ↳ Apply the simple equation of [Similarity - Target] to compute gradients of for both directions. ↳ Why so simple? Because when Softmax and Cross-Entropy Loss are used together, the math magically works out that way. ↳ These gradients kick off the backpropagation process to update weights and biases of the encoders and projection layers (red borders).

Tom Yeh

67,896 次观看 • 2 年前