Test-time scaling, reasoning, and generally search-like processes clearly drive significant gains in LLMs. Largely owed to the structure of language. One would think the same could apply to non-linguistic domains, like image generation, but that obviously depends on whether the structure of the domain's representation lends itself to search.... 1D ordered tokens (e.g., image FlexTok, video FlexTok) seem like a natural fit since they enable a step-by-step coarse-to-fine generation. We investigated that and found they indeed enable search and scale far better with test-time compute than 2D grids. See the visuals on the webpage. Appearing in ICML Conference 2026. 🔗 📄show more

Unlike the standard 2D grid tokenization, which encodes images into a fixed number of tokens, 1D ordered tokenizers encode images into variable-length sequences with a coarse-to-fine structure. Examples of such 1D ordered tokenizers are and This coarse-to-fine structure changes how these models scale at test time.

This coarse-to-fine structure yields intermediate readouts from 1D ordered tokens, which are easier to verify than those from 2D grids. This provides a signal for steering the model via test-time search. The animation shows beam search over a 1D ordered tokenizer.

To study this topic, we define a search-over-tokens (SoTo) framework with three key questions: 1️⃣ Which search algorithm works best for a given token structure? 2️⃣ Which verifiers can guide search, and which work best? 3️⃣ How much does the AR prior matter?

We compare 1D ordered and 2D grid-based tokens with controlled AR models across three standard search algorithms and find: Both models scale similarly under best-of-N and lookahead search, but ordered tokens benefit the most from beam search, and ultimately scale better.

Interestingly, it is possible to search directly over 1D ordered tokens, even without training an AR model entirely, and still generate reasonable images. The video shows beam search directly over the FlexTok token space, guided by an ImageReward verifier.

Search also enables zero-shot control. By swapping in an image-image verifier (e.g., DreamSim), we can guide the model to generate objects similar to a reference image, even though the AR model was only trained on text-image pairs.

We also study search with a range of image-text verifiers. All of them improve over the no-search baseline, but an ensemble achieves the strongest results overall across all metrics. Some example visualizations:

We also compare different AR priors. While unconditional and uniform priors can generate reasonable images via guided search, a strong conditional prior narrows the search space, converging on the final image concepts much faster.

The interactive tools on the website and the demo are really useful to get a quick understanding. The code is open-source. It is extensible to other AR models, verifiers, and search algorithms. There are starter notebook examples in the repo. Website:  Paper: Code: Demo: Congrats to the great team at @EPFL_en and @Apple: @zhitong_gao, @parham_rezaei_, @califyn, @MingqiaoY, @nale_jovanovic, @JRAllardice, @afshin_dn, @zamir_ar, @roman__bachmann, @oguzhanthefatih!

A significant step toward Search-over-Tokens for vision. The real insight here is that 1D ordered tokens transform image generation from a rigid raster-scan into a hierarchical decision tree, making test-time compute finally effective. The scaling curves (530M + search vs. 3.4B) clearly show that representation structure, not just model size, is the bottleneck for visual reasoning. Congrats on the @icmlconf 2026 acceptance!

@m1nj12 Awesome! It always seemed like semanticist/flextok should enable some cool new things built on top.

Really compelling insight. The idea that test-time scaling in non-linguistic domains depends heavily on whether the representation itself supports structured search feels extremely important especially as reasoning-style inference expands beyond text.

All diffusion models too enable step-by-step coarse-to-fine generation