Skip navigation
Today's Training Data episode takes us BTS on the infrastructure challenges required to do large RL…
0:00 / 45:33

Today's Training Data episode takes us BTS on the infrastructure challenges required to do large RL…

Sonya Huang 🐥

@sonyatweetybird30K followers

Visit on X
345 likesDownloadOriginal post

81,490 views

Today's Training Data episode takes us BTS on the infrastructure challenges required to do large RL runs at scale, featuring Federico Cassano (Composer Lead at Cursor) and Dmytro Dzhulgakov (Co-Founder at Fireworks). The Cursor team trained Composer 2 on Fireworks by starting with a strong base model (Kimi 2.5) and performing large-scale mid-training on code tokens and web data to learn common patterns and libraries, followed by a large-scale Reinforcement Learning run to learn how to navigate the Cursor harness, call tools, and write correct code. Today's episode dives into the systems and infrastructure challenges of making that large RL run happening, and there were many (!!), from numerical mismatch to global distribution to synchronizing rollouts across asynchronous pipelines to keeping track of expert activation across runs and more. Extremely nerdy in-the-weeds challenges that Federico and Dima were delighted to nerd out on together :) Beyond RL infra, we also discussed Online vs Simulated rollouts, self-summarization for long-horizon agents, environment design ("the most powerful RL environment is the product itself"), and other technical nuggets. PS: We filmed this episode before the SpaceX news, while the Cursor team was still compute-constrained. While Cursor now has *all* the flops, the takeaways and hurdles crossed ring true for any serious application-level company that is racing to post-train their own models. I believe that more serious application companies will go the way of Cursor and post-train their own models. 00:00 Introduction 00:53 Why Cursor Trained Composer 2 04:55 Specialization vs Bitter Lesson 06:16 Composer 2 Training Recipe 16:32 Scaling RL Infrastructure Globally 23:32 Floating Point Drift 25:11 MoE Sensitivity Explained 26:25 Router Replay Fix 27:19 Real Time RL Loop 31:49 Long Horizon Agents 34:29 Why RL Everywhere 37:34 LLM as Judge Rewards 39:14 RL in Hard Domains 40:13 Build Your Own Environments 44:34 Closing Thoughts

14 Comments

Comments from the original post on X (Twitter)

  • elie's profile picture

    @eliebakouch

    @ellev3n11 @cursor_ai @dzhulgakov @FireworksAI_HQ oh huge, can’t wait to watch this

    5 likes

  • Kartik's profile picture

    @ayyar

    @ellev3n11 @cursor_ai @dzhulgakov @FireworksAI_HQ Awesome work @ellev3n11 ! 😀

    1 like

  • Steve Hong's profile picture

    @steve_steve_hg

    @ellev3n11 @cursor_ai @dzhulgakov @FireworksAI_HQ @agentictac

  • Honest Opinion's profile picture

    @justmy2cents111

    @ellev3n11 @cursor_ai @dzhulgakov @FireworksAI_HQ Why are you just uploading this?

  • Sonya Huang 🐥's profile picture

    @sonyatweetybird

    @ellev3n11 @cursor_ai @dzhulgakov @FireworksAI_HQ this was filmed a month ago. standard backlog.

  • Harshal's profile picture

    @twtHarshal

    @ellev3n11 @cursor_ai @dzhulgakov @FireworksAI_HQ Meaning all vertical companies fold into @SpaceX when they are the landlord?

  • Sonya Huang 🐥's profile picture

    @sonyatweetybird

    @ellev3n11 @cursor_ai @dzhulgakov @FireworksAI_HQ @SpaceX AWS is the landlord but doesn't own all of SaaS

  • Harshal's profile picture

    @twtHarshal

    @ellev3n11 @cursor_ai @dzhulgakov @FireworksAI_HQ @SpaceX It changes with AI, Spacex will own coz AWS compute cost economics is different then AI tokeneconomics, built more you go bust, built less you deal beg! You don’t own SAAS literally, but have the pricing power to OWN them and make them winners or losers! E.g cursor, Antropic?

  • Stjepan's profile picture

    @StjepanZRPavic

    @ellev3n11 @cursor_ai @dzhulgakov @FireworksAI_HQ girls all wet from all the masculine chad energies

  • brandon galang ▲'s profile picture

    @brandon_galang

    @ellev3n11 @cursor_ai @dzhulgakov @FireworksAI_HQ Is this an old interview? Composer 2 or 2.5?

  • Sonya Huang 🐥's profile picture

    @sonyatweetybird

    @ellev3n11 @cursor_ai @dzhulgakov @FireworksAI_HQ ah shoot Composer 2.5! typo!

    3 likes

  • brandon galang ▲'s profile picture

    @brandon_galang

    @ellev3n11 @cursor_ai @dzhulgakov @FireworksAI_HQ I thought so 😄 Thanks for hosting this, was a great conversation

    1 like

  • Gregor's profile picture

    @bygregorr

    @ellev3n11 @cursor_ai @dzhulgakov @FireworksAI_HQ not sure training RL at scale on a third-party provider is what i'd expect i figured iteration speed would force you to own the infra. did Fireworks solve a specific bottleneck or was it mostly just GPU cost?

  • Sonya Huang 🐥's profile picture

    @sonyatweetybird

    @ellev3n11 @cursor_ai @dzhulgakov @FireworksAI_HQ you can BYOC to fireworks (which is what cursor does)

    2 likes