James Whittington's banner
James Whittington's profile picture

James Whittington

@jcrwhittington • 3,725 subscribers

Neuroscience, AI, physics, medicine.

Shorts

Today we are happy to share one of our first open research releases: Latent Sparse Attention (LSA), a step towards subquadratic attention that matches accuracy of full attention at ultra-long context. Sparse attention attends to only a few tokens per query, but it has to find the right ones. At 10M tokens, that search takes 98% of the attention layer's time. LSA makes sparse attention up to 22.7× faster and full attention 72.4x faster. With Ales Kubicek Andrea Caciolai Liangzu Peng Sukrit Sumant Ben Hoover Fraser Ruairidh Battleday Radim Urban Torsten Hoefler 🇨🇭 James Whittington

Today we are happy to share one of our first open research releases: Latent Sparse Attention (LSA), a step towards subquadratic attention that matches accuracy of full attention at ultra-long context. Sparse attention attends to only a few tokens per query, but it has to find the right ones. At 10M tokens, that search takes 98% of the attention layer's time. LSA makes sparse attention up to 22.7× faster and full attention 72.4x faster. With Ales Kubicek Andrea Caciolai Liangzu Peng Sukrit Sumant Ben Hoover Fraser Ruairidh Battleday Radim Urban Torsten Hoefler 🇨🇭 James Whittington

324,232 Aufrufe