GPTs are generating sequences in a left-to-right order. Is there another way? With François Fleuret and @evanncourdier, in partnership with SkySoft-ATM, we developed σ-GPT, capable of generating sequences in any order chosen dynamically at inference time. 1/6

We only need to shuffle the input sequence and to use a double positional encoding, so that the model is informed of what token to generate at each position of the output sequence. 2/6

This opens a multitude of possibilities. Such as: (A) Conditioning on any subset of tokens from a sequence or (B) Estimating the probability of the remaining tokens at once. 3/6

(A) is done by putting the conditioning tokens “first”, and (B) is done by putting all the remaining tokens as “the next”. The latter requires carving the attention matrix. 4/6

We combine these two functionalities into a burst-sampling scheme with rejection. It generates all the remaining tokens in one shot, and keeps only the ones that are self-consistent. This allows sublinear sequence generation similar to diffusion. 5/6

With the advantages that simple tasks can be fully completed in a few steps. 6/6 (with a diffusion comparison)

More detail about this work in our paper, that was just accepted at @ECMLPKDD. Paper: Demo: Code: in preparation.

@francoisfleuret @EvannCourdier @SkysoftATM so the first words that come out are the most important and limit the following possibilities. cool.

@francoisfleuret @EvannCourdier @SkysoftATM I agree that it can be limiting in a way, but it can also be useful for the model: fixing some tokens might give the overall structure of the text it want to generate, and then it only has to fill in between.

@francoisfleuret @EvannCourdier @SkysoftATM We think multidimensionally, but we communicate serially. Intuitively this feels like it has legs.

@francoisfleuret @EvannCourdier @SkysoftATM I'm sorry what? Half an hour ago I was tweeting about an architecture just like this combined with type checking/linting information that is tokenized and injected to improve code generation.. And you just post a paper on the difficult part of that project? Just.. right now? O.o