Загрузка видео...

Не удалось загрузить видео

На главную

Can a small academic team build a strong text-to-image model using only public datasets? Introducing i1: a simple, fully open recipe for strong text-to-image models

71,178 просмотров • 3 месяцев назад •via X (Twitter)

Комментарии: 21

Фото профиля Zhuang Liu
Zhuang Liu3 месяцев назад

We study the design space of text-to-image diffusion models w/ 300+ controlled experiments. The insights are combined to build i1-3B, which outperforms the best prior fully open model by 29.5 absolute percentage points on average across 5 benchmarks.

Фото профиля Zhuang Liu
Zhuang Liu3 месяцев назад

We release the checkpoint, training/inference code, dataset, data pipeline, and full recipe. This is not just another checkpoint. It is a strong baseline people can study, reproduce, modify, and build on.

Фото профиля Zhuang Liu
Zhuang Liu3 месяцев назад

At 1024px, i1-3B is competitive with leading open-weight T2I models across our benchmark suite, and outperforms models like FLUX.1 [Dev] and HiDream-I1-Full. This is not just on older benchmarks (GenEval and DPG-Bench). It also holds on newer benchmarks (PRISM, CVTG-2K, and LongText-Bench) used by leading models (e.g., Z-Image, Ideogram 4.0) and better reflect modern T2I capabilities.

Фото профиля Zhuang Liu
Zhuang Liu3 месяцев назад

Alongside strong general image generation, i1-3B is the first fully open T2I model with competitive text rendering performance (to the best of our knowledge). With 3B parameters, it scores 92% on LongText-Bench. For comparison, the best prior fully open model scores 15%.

Фото профиля Zhuang Liu
Zhuang Liu3 месяцев назад

The strong performance of i1-3B comes from 300+ controlled experiments carefully ablating modeling and data designs. We highlight some of our findings below:

Фото профиля Zhuang Liu
Zhuang Liu3 месяцев назад

Modeling findings: 1) a single strong text encoder + larger text encoder adapter > combining multiple text encoders 2) AdaLN provides marginal benefit for T2I relative to its large number of parameters 3) dual-stream DiT + long skip connections is a strong backbone design

Фото профиля Zhuang Liu
Zhuang Liu3 месяцев назад

Data findings: 1) train on long captions + inference-time prompt rewrite > train on short captions, even for short inference prompts 2) equal weighting is a strong default for mixing curated datasets 3) with a diverse mix of datasets, fewer images and more epochs don’t hurt performance much 4) strong high-resolution performance doesn't require broad coverage in high-resolution training data

Фото профиля Zhuang Liu
Zhuang Liu3 месяцев назад

i1 shows that strong text-to-image models do not have to rely on opaque recipes, inaccessible data, or highly specialized new components. A simple, fully open recipe built from public datasets and carefully validated design choices can be competitive with leading open-weight models. Find many more results and analyses in our paper! Led by @zeng_boya with many others contributing, Tianze Luo, Jucheng Shen, Shu Pu, @TaimingLu and @GabrielSarch

Фото профиля Zhuang Liu
Zhuang Liu3 месяцев назад

Check i1 out: Model: Data: Project Page: Code: Paper:

Фото профиля Lucas Beyer (bl16)
Lucas Beyer (bl16)3 месяцев назад

@XingyuFu2 Nice work and write-up, congrats!

Фото профиля Zhuang Liu
Zhuang Liu3 месяцев назад

@XingyuFu2 Thanks Lucas!

Фото профиля Saining Xie
Saining Xie3 месяцев назад

great work!

Фото профиля Grigoris Chrysos
Grigoris Chrysos3 месяцев назад

Great work!

Фото профиля Hang Gao
Hang Gao3 месяцев назад

Nice results and ablations! Congrats!

Фото профиля Nikhil Lamba
Nikhil Lamba3 месяцев назад

Super cool

Фото профиля Nikhil Lamba
Nikhil Lamba3 месяцев назад

Definitely the trend seems the same. As it was for CNNs and vision. First very deep models, then the refined ones with only imp layers that can run on edge devices.

Фото профиля kenji
kenji3 месяцев назад

@jon_barron Do the team plan to release 1b model soon?

Фото профиля Jiawei (Joe) Zhou
Jiawei (Joe) Zhou3 месяцев назад

Great work for democratizing T2I model training! Will definitely take a look.

Фото профиля XuDong Wang
XuDong Wang3 месяцев назад

Nice work Zhuang! Congratulations!

Фото профиля Zhuang Liu
Zhuang Liu3 месяцев назад

Thanks Xudong!

Фото профиля Saad Gul
Saad Gul3 месяцев назад

@ricklamers 😎

Похожие видео

[CLIP] by Hand ✍️ The CLIP (Contrastive Language–Image Pre-training) model, a groundbreaking work by OpenAI, redefines the intersection of computer vision and natural language processing. It is the basis of all the multi-modal foundation models we see today. How does CLIP work? Goal: 🟨 Learn a shared embedding space for text and image [1] Given ↳ A mini batch of 3 text-image pairs ↳ OpenAI used 400 million text-image pairs to train its original CLIP model. Process 1st pair: "big table" [2] 🟪 Text → 2 Vectors (3D) ↳ Look up word embedding vectors using word2vec. [3] 🟩 Image → 2 Vectors (4D) ↳ Divide the image into two patches. ↳ Flatten each patch [4] Process other pairs ↳ Repeat [2]-[3] [5] 🟪 Text Encoder & 🟩 Image Encoder ↳ Encode input vectors into feature vectors ↳ Here, both encoders are simple one layer perceptron (linear + ReLU) ↳ In practice, the encoders are usually transformer models. [6] 🟪 🟩 Mean Pooling: 2 → 1 vector ↳ Average 2 feature vectors into a single vector by averaging across the columns ↳ The goal is to have one vector to represent each image or text [7] 🟪 🟩 -> 🟨 Projection ↳ Note that the text and image feature vectors from the encoders have different dimensions (3D vs. 4D). ↳ Use a linear layer to project image and text vectors to a 2D shared embedding space. 🏋️ Contrastive Pre-training 🏋️ [8] Prepare for MatMul ↳ Copy text vectors (T1,T2,T3) ↳ Copy the transpose of image vectors (I1,I2,I3) ↳ They are all in the 2D shared embedding space. [9] 🟦 MatMul ↳ Multiply T and I matrices. ↳ This is equivalent to taking dot product between every pair of image and text vectors. ↳ The purpose is to use dot product to estimate the similarity between a pair of image-text. [10] 🟦 Softmax: e^x ↳ Raise e to the power of the number in each cell ↳ To simplify hand calculation, we approximate e^□ with 3^□. [11] 🟦 Softmax: ∑ ↳ Sum each row for 🟩 image→🟪 text ↳ Sum each column for 🟪 text→ 🟩 image [12] 🟦 Softmax: 1 / sum ↳ Divide each element by the column sum to obtain a similarity matrix for 🟪 text→🟩 image ↳ Divide each element by the row sum to obtain a similarity matrix for 🟩 image→🟪 text [13] 🟥 Loss Gradients ↳ The "Targets" for the similarity matrices are Identity Matrices. ↳ Why? If I and T come from the same pair (i=j), we want the highest value, which is 1, and 0 otherwise. ↳ Apply the simple equation of [Similarity - Target] to compute gradients of for both directions. ↳ Why so simple? Because when Softmax and Cross-Entropy Loss are used together, the math magically works out that way. ↳ These gradients kick off the backpropagation process to update weights and biases of the encoders and projection layers (red borders).

Tom Yeh

67,896 просмотров • 2 лет назад