how should multimodal models encode visual input? → pretrained encoder? → encoder-free? → before the LLM? published models bundle these choices together. we trained 200+ models from scratch to see which of these choices matters only one of them does 🧵 [1/n]

🌐 📄 see @choang333's thread for a deep dive 👇 [2/n]

most multimodal models start from a pretrained encoder like SigLIP 2. Kimi K3 trains its encoder from scratch at the same compute and data, which is better? we trained everything from scratch, encoder pretraining counted in the budget [3/n]

1️⃣ pretrained encoder? optional we pretrained encoders 4 ways (contrastive, SSL, captioning). at best they tie an encoder trained from scratch with the LLM even ignoring its cost, more encoder pretraining raises ImageNet but not necessarily VQA [4/n]

2️⃣ encoder-free? trails at every size, ~3× the compute to catch up > but isn't it just smaller? give the LLM the encoder's 0.4B params (wider, deeper, or both) → text gets better → VQA doesn't [5/n]

3️⃣ does that capacity have to come before the LLM? we test whether an encoder's capacity can move inside the backbone: in its first layers, image tokens get vision-only branches on every attention projection + FFN text tokens never touch them [6/n]

🔑 reserve them for vision same 0.4B params, same layers: → widen the shared FFNs: 40.1 → add a branch for all tokens: 42.3 → make that branch vision-only: 45.0 an encoder gets 44.4 [7/n]

🔑 put them early the same vision-only budget in every layer, spread through depth, or in the last layers → no better than encoder-free in the first layers, it's on par with the encoder, at every size [8/n]

and it's faster an encoder has to finish before the LLM can start. in-fusion has no separate stage to wait for → 13% faster to first token at the encoder's 0.4B → and the gap grows with capacity: an encoder's added time climbs much faster than in-fusion's [9/n]

so of those three questions, only one matters: does vision get its own parameters? not its own pretraining, and not its own encoder. vision-only params in the LLM's first layers do the job [10/n]

Inkling's "encoder-free" stem, a 4-layer patch MLP, is still vision-only capacity, built to compress hi-res images into few tokens but Inkling 2 Small brought back an encoder 🧐🧐 [11/n]

🤝@choang333 this has been a truly amazing collaboration. stay tuned for more 🔜 thank you to all of our collaborators @JasonQSY @ShangwenLi1 @ml_perception @gh_marjan @LukeZettlemoyer @giffmana @mengyer Jakob Verbeek @sainingxie @Hu_Hsu [11/n]

@JasonQSY @ShangwenLi1 @ml_perception @gh_marjan @LukeZettlemoyer @giffmana @mengyer @sainingxie @Hu_Hsu 🧵[n/n] 📄 Paper: 🌐 Blog:

@JasonQSY @ShangwenLi1 @ml_perception @gh_marjan @LukeZettlemoyer @mengyer @sainingxie @Hu_Hsu amazing results! congrats

@mihirp98 @JasonQSY @ShangwenLi1 @ml_perception @gh_marjan @LukeZettlemoyer @mengyer @sainingxie @Hu_Hsu amazing work!!

@JasonQSY @ShangwenLi1 @ml_perception @gh_marjan @LukeZettlemoyer @mengyer @sainingxie @Hu_Hsu Great work!

@JasonQSY @ShangwenLi1 @ml_perception @gh_marjan @LukeZettlemoyer @mengyer @sainingxie @Hu_Hsu We actually went deeper on these architectural choices in our recent study of 200+ models:

@JasonQSY @ShangwenLi1 @ml_perception @gh_marjan @LukeZettlemoyer @mengyer @sainingxie @Hu_Hsu 200+ models trained from scratch is the right way to settle this, since most vision-language papers change five things at once. If only one choice really matters, expect the encoder-free camp to get quiet fast. Which one was it?