Wanna know whether different LLM providers serve the same LLama 3.1 70B? I sure did! So I ran a quick eval to get some surprising results + open sourced my code 👇 Check out my comparison between Groq Inc Fireworks AI octoaicloud DeepInfra and Together AI

If you don't want to watch all that, check out the results yourself, the project is open on my @weights_biases Weave dashboard, here's the direct link to the comparison view 👉

And the code is live on Github, it's a simple python notebook + some instrumentation via @GroqInc and @OpenRouterAI which is amazing for these types of comparisons (thanks @xanderatallah for the assist!)

If you're interested to try Weave out, super easy to get started, check out the docs at or lmk if I can help

@GroqInc @FireworksAI_HQ @OctoAICloud @DeepInfra @togethercompute Great. I also noticed differences between providers. It looks like the choice of provider is quite important. Would be great to have a continuously updated leaderboard for that. Wouldn't this be something cool to sponsor for W&B? It would also showcase the W&B tooling,

@GroqInc @FireworksAI_HQ @OctoAICloud @DeepInfra @togethercompute Interesting! Not to mention apart from latency, we pay for tokens in and out! So when they spit out more than the others, we wait longer and pay more.

@GroqInc @FireworksAI_HQ @OctoAICloud @DeepInfra @togethercompute Yeah, we need a better integration with OpenRouter (@xanderatallah maybe we should work together on this?) to support per provider pricing in the UI so we can actually estimate correct provider pricing and do an apples to apples comparison for that as well!

@GroqInc @FireworksAI_HQ @OctoAICloud @DeepInfra @togethercompute wow, so every openrouter llama 3.1 70b provider has different results at temperature 0... that's strange

@GroqInc @FireworksAI_HQ @OctoAICloud @DeepInfra @togethercompute Very interesting!

@GroqInc @FireworksAI_HQ @OctoAICloud @DeepInfra @togethercompute Thanks for the analysis, @altryne! We are excited see you independently validate @OctoAICloud as a leading provider of Llama3.1 70b!

@GroqInc @FireworksAI_HQ @OctoAICloud @DeepInfra @togethercompute Thanks for taking the time to review Luis! Are you guys providing access to full 128K context already? cc Would love to collab with @OctoAICloud 👏