We just brought flow maps to language modeling for one-step sequence generation 💥 Discrete diffusion is not necessary -- continuous flows over one-hot encodings achieve SoTA performance and ≥8.3× faster generation 🔥 We believe this is a major step forward for discrete generative modeling and language modeling alike. 🚀... Full thread from first author Chanhyuk Lee:show more

Beyond our core empirical results, we map out the complete design space of flow maps for discrete data, opening the door to a suite of new algorithms: 📐 Progressive / Shortcut-style approaches 🌊 Eulerian / Mean Flow-style approaches 🎯 Lagrangian / Terminal Velocity Matching approaches

Tagging some folks who might be interested 👇 @sedielem @_albertgu @zicokolter @RickyTQChen @max_simchowitz @StefanoErmon @DrYangSong @msalbergo @FrankNoeBerlin @electronickale @DaniloJRezende @karsten_kreis @bose_joey @lipmanya @ZhengyangGeng @baaadas @jaschasd @poolio @ValentinDeBort1 @ArashVahdat @du_yilun @emiel_hoogeboom @dpkingma @robrombach @DavidDuvenaud

@_albertgu @zicokolter @RickyTQChen @max_simchowitz @StefanoErmon @DrYangSong @msalbergo @FrankNoeBerlin @electronickale @DaniloJRezende @karsten_kreis @bose_joey @lipmanya @ZhengyangGeng @baaadas @jaschasd @poolio further info below: 📰arXiv: 📷project page: 📷code:

Don't these continuous flows introduce a whole new layer of complexity in terms of understanding and debugging the generated sequences? Are there any efforts being made to provide more interpretability of these continuous flows, or is that a trade-off for the increased speed and performance?

can you explain what you mean a bit further? it's actually quite a lot simpler than the standard CTMC formulation of a discrete diffusion!

Congrats, Nicholas! I read the paper and found it pretty constructive, particularly the boundary condition under the CE loss part.

I thought the results should be evaluated on benchmark datasets like gsm8k minerva_math right

LM1B and OWT are pretty standard here -- those are math datasets.