Alexander Johansen's banner
Alexander Johansen's profile picture

Alexander Johansen

@AlexRoseJo • 1,396 subscribers

CS PhD @Stanford || Ask me about NVFP4 training || Proofs, Bounds, and Better agents

Shorts

🥐 Hi everyone! FlashAttention's backward pass has the same zero-sum bug Jon Barron just fixed in softmax. With FP4 stochastic rounding it's wrong on every single draw. We fixed it in our ICLR'27 submission, CroissAttn and unlock e2e FP4 training.

🥐 Hi everyone! FlashAttention's backward pass has the same zero-sum bug Jon Barron just fixed in softmax. With FP4 stochastic rounding it's wrong on every single draw. We fixed it in our ICLR'27 submission, CroissAttn and unlock e2e FP4 training.

73,156 views