Skip navigation
I hand-wrote a 500-LoC RL stack to make hacking on RL research much easier.
0:00 / 1:53

I hand-wrote a 500-LoC RL stack to make hacking on RL research much easier.

omkaar

@omkizzy4.3K followers

Visit on X
175 likesDownloadOriginal post

17,954 views

I hand-wrote a 500-LoC RL stack to make hacking on RL research much easier. Most RL stacks are either massive and unhackable, or duct-taped research scripts. I am open-sourcing Mithrl, a modular RLVR stack. Next items on my checklist: adding more complex environment examples, supporting multi-gpu + async RL, and QoL fixes. I might scrap external runtime dependencies (Huggingface PEFT + vLLM) and write purpose-built, simpler versions from scratch if I feel the need. If you want to experiment with RL and are looking to own sovereign tools, I’d love to get on call, understand your requirements and help integrate for free.

36 Comments

Comments from the original post on X (Twitter)

  • omkaar's profile picture

    @omkizzy

    cc for feedback: @willccb @QGallouedec @_rajanagarwal @yacinelearning @danielhanchen (learnt vllm sleep/wake from here) @karpathy (easy to port to autoresearch which is what I am doing next)

    4 likes

  • omkaar's profile picture

    @omkizzy

    github:

    3 likes

  • parth's profile picture

    @parthsareen

    hell yeah

    2 likes

  • omkaar's profile picture

    @omkizzy

    ship squad

    1 like

  • surya's profile picture

    @suryasure05

    keep cooking omkizzy

    2 likes

  • omkaar's profile picture

    @omkizzy

    me and you forever

    1 like

  • Jibran's profile picture

    @Jibran_05

    @hamostaf04 this is sick

    2 likes

  • omkaar's profile picture

    @omkizzy

    @hamostaf04 yessir thank you

    2 likes

  • mikael haji's profile picture

    @mikaelhaji

    lets gooo this is fire

    2 likes

  • omkaar's profile picture

    @omkizzy

    appreciate you brother

  • saksham's profile picture

    @sakshambatraa

    yoooo that’s tuff

    1 like

  • omkaar's profile picture

    @omkizzy

    thank you brother

    1 like

  • ari dutilh's profile picture

    @aridutilh

    man this is awesome

    1 like

  • omkaar's profile picture

    @omkizzy

    i appreciate it man, lmk if you have ppl experimenting w LLM RL I could talk to

    1 like

  • ari dutilh's profile picture

    @aridutilh

    @novasarc01 potentially

    1 like

  • omkaar's profile picture

    @omkizzy

    @novasarc01 hi @novasarc01, would love to chat if this is interesting

    1 like

  • Damon Crockett's profile picture

    @DamonCrockett

    Hand wrote, nice

    1 like

  • toki's profile picture

    @tokifyi

    this is fire

    1 like

  • omkaar's profile picture

    @omkizzy

    appreciate it brother, lmk if you run any experiments on top

  • Madhav Singhal's profile picture

    @madhavsinghal_

    this is dope

    1 like