Loading video...

Video Failed to Load

There was a problem loading this video. This could be due to a temporary network issue or the video might be unavailable.

We keep getting asked by users if they can use the 70B parameter model in production. We're serving the chat variant of Llama-2 70B on 2xA100 and getting pretty great throughput — it's cooking!

Tuhin Srivastava

3,324 subscribers

20,037 views • 3 years ago •via X (Twitter)

Anya Rossi• Live Now

Private livecam show

6 Comments

Dwayne3 years ago

It's such a shame that Llama 2 is so neutered with all of the excessive safety they wrapped it in. Hopefully, we see an unlocked 70b Llama 2 soon.

Tuhin Srivastava3 years ago

@DwayneCodes Yes, I'm sure it's coming. The non-chat model has less of it, I believe.

Bill Njoroge3 years ago

how many it/s?

Debargha Ganguly3 years ago

Will give this a try tomorrow, the readme on the repo says 4xA100 vs 2xA100 in your tweet. What’s the throughput you’re seeing?

Tuhin Srivastava3 years ago

I’ll push an updated truss tonight!

Muskan Khandelwal3 years ago

Hey Tuhin, is there some way I can get to contribute to Baseten ? I can learn and deliver at blazing fast speed.

Related Videos

Finally! Codellama-70b as Copilot in VSCode! 😱 You can now harness the power of the most advanced code generation model, Code Llama 70B, right in Visual Studio Code with Perplexity This model has outperformed even GPT-4 in code writing, and it's now seamlessly integrated into our editor 🥳 Do you anticipate the emergence of more open-source models that surpass GPT-4?

Finally! Codellama-70b as Copilot in VSCode! 😱 You can now harness the power of the most advanced code generation model, Code Llama 70B, right in Visual Studio Code with Perplexity This model has outperformed even GPT-4 in code writing, and it's now seamlessly integrated into our editor 🥳 Do you anticipate the emergence of more open-source models that surpass GPT-4?

Daniel San

69,499 views • 2 years ago

Our Llama-3.1-Nemotron-70B-Instruct model is a leading model on the 🏆 Arena Hard benchmark (85) from Arena. Arena Hard uses a data pipeline to build high-quality benchmarks from live data in Chatbot Arena, and is known for its predictive ability of Chatbot Arena Elo score as well as separability between helpful and less helpful models. Use our customized model Llama-3.1-Nemotron-70B to improve the helpfulness of LLM generated responses in your applications. 📥 Try on our API catalog: 📥 On GitHub: 📥 Or on Hugging Face:

Our Llama-3.1-Nemotron-70B-Instruct model is a leading model on the 🏆 Arena Hard benchmark (85) from Arena. Arena Hard uses a data pipeline to build high-quality benchmarks from live data in Chatbot Arena, and is known for its predictive ability of Chatbot Arena Elo score as well as separability between helpful and less helpful models. Use our customized model Llama-3.1-Nemotron-70B to improve the helpfulness of LLM generated responses in your applications. 📥 Try on our API catalog: 📥 On GitHub: 📥 Or on Hugging Face:

NVIDIA AI Developer

140,756 views • 1 year ago

"One of the biggest misconceptions" Cerebras CFO Bob Komin pushes back on the small-models narrative. "We serve all models, and there is no limit to the size of the models that we can serve. Today, we're serving trillion parameter models. We're serving trillion parameter models that are internal for OpenAI today. We are currently running OpenAI 5.4 and 5.5 with them."

"One of the biggest misconceptions" Cerebras CFO Bob Komin pushes back on the small-models narrative. "We serve all models, and there is no limit to the size of the models that we can serve. Today, we're serving trillion parameter models. We're serving trillion parameter models that are internal for OpenAI today. We are currently running OpenAI 5.4 and 5.5 with them."

Deirdre Bosa

84,422 views • 2 months ago

NVIDIA silently released the Llama 3.1 70B Instruct Nemotron model which is topping leaderboards. You can try it here: I sat down with one of the researchers to learn how it performed so well and why they built this 🤙 Anything else I should ask him?

NVIDIA silently released the Llama 3.1 70B Instruct Nemotron model which is topping leaderboards. You can try it here: I sat down with one of the researchers to learn how it performed so well and why they built this 🤙 Anything else I should ask him?

Nader Khalil🍊

48,344 views • 1 year ago

This is the fastest I've seen Llama 3.3 running anywhere! Llama 3.3 70B running at 652 t/s is lightning fast. And if you want Llama 3.1, here are the speeds I was able to get: • Llama 3.1 8B: 1006 t/s • Llama 3.1 70B: 709 t/s • Llama 3.1 405B: 206 t/s (You can access all of these models for free! See the link below.) This speed is incredible, but the interesting part is what's happening behind the scenes: These models aren't running on a GPU! In this video, I'm using the SambaNova cloud to access these models. They built a custom chip (SN40L) optimized for AI workflows. A single SN40L chip can hold hundreds of models (trillions of parameters) in memory! The speed alone is a huge deal, but the big advantage is for agentic workflows running multiple specialized models. A GPU can only host a single model and switch (unload and load) to a different model if necessary. An SN40L, on the other hand, can host every model at once, making it much faster. Here is the video where you can see how fast these chips are:

This is the fastest I've seen Llama 3.3 running anywhere! Llama 3.3 70B running at 652 t/s is lightning fast. And if you want Llama 3.1, here are the speeds I was able to get: • Llama 3.1 8B: 1006 t/s • Llama 3.1 70B: 709 t/s • Llama 3.1 405B: 206 t/s (You can access all of these models for free! See the link below.) This speed is incredible, but the interesting part is what's happening behind the scenes: These models aren't running on a GPU! In this video, I'm using the SambaNova cloud to access these models. They built a custom chip (SN40L) optimized for AI workflows. A single SN40L chip can hold hundreds of models (trillions of parameters) in memory! The speed alone is a huge deal, but the big advantage is for agentic workflows running multiple specialized models. A GPU can only host a single model and switch (unload and load) to a different model if necessary. An SN40L, on the other hand, can host every model at once, making it much faster. Here is the video where you can see how fast these chips are:

Santiago

97,528 views • 1 year ago

Using LLaMA 3 70B on Groq Inc to instantly refactor and document code. The implications for software engineering are wild. Gone are the days of waiting on an LLM for suggestions or code changes. Now, it's an instant feedback loop. Demo link in the comments:

Using LLaMA 3 70B on Groq Inc to instantly refactor and document code. The implications for software engineering are wild. Gone are the days of waiting on an LLM for suggestions or code changes. Now, it's an instant feedback loop. Demo link in the comments:

Matt Shumer

442,695 views • 2 years ago

"The reason we're seeing this extreme amount of hatred and violence is because we're actually succeeding in getting rid of corruption and waste, If we weren't succeeding in getting rid of corruption and waste, they wouldn't care, but we are succeeding." -Elon Musk

"The reason we're seeing this extreme amount of hatred and violence is because we're actually succeeding in getting rid of corruption and waste, If we weren't succeeding in getting rid of corruption and waste, they wouldn't care, but we are succeeding." -Elon Musk

SMX 🇺🇸

77,274 views • 1 year ago

"The reason we're seeing this extreme amount of hatred and violence is because we're actually succeeding in getting rid of corruption and waste. If we weren't succeeding in getting rid of corruption and waste, they wouldn't care. But we are succeeding." -Elon Musk

"The reason we're seeing this extreme amount of hatred and violence is because we're actually succeeding in getting rid of corruption and waste. If we weren't succeeding in getting rid of corruption and waste, they wouldn't care. But we are succeeding." -Elon Musk

SMX 🇺🇸

579,166 views • 1 year ago

President Trump on trade deal with Japan: "So we're paying zero (tariff). They're paying 15. We're getting $550 billion, and to be exact, it's 90% of that but we control the whole lot of it. And it's really been great."

President Trump on trade deal with Japan: "So we're paying zero (tariff). They're paying 15. We're getting $550 billion, and to be exact, it's 90% of that but we control the whole lot of it. And it's really been great."

CSPAN

17,610 views • 1 year ago

DeepSeek R1 671B is ridiculously good, but running at 158 tokens per second is something out of this world. A model *this good* running *this fast* is 3 years ahead of this time. The secret: DeepSeek in this short video isn't running on GPUs. It's running on custom-made chips! This is running on SambaNova and their SN40L chips, which are designed and optimized for running AI workflows. SambaNova is sponsoring my work. We designed GPUs to run games and later realized we could use them for AI. This is very different from creating a chip specifically for AI from the ground up. Just think about this for a second: One single SNL40 chip can simultaneously hold 100+ models (trillions of parameters) in memory! An agentic workflow that uses multiple models simultaneously won't need to swap models from memory even once! By the way, the Llama family of models *screams* on the SambaNova Cloud. Here are some numbers from my latest tests: • Llama 3.3 70B: 367.32 t/s • Llama 3.2 1B: 2381.82 t/s • Llama 3.2 3B: 1335.71 t/s • Llama 3.1 8B: 929.58 t/s • Llama 3.1 70B: 369.16 t/s • Llama 3.1 405B: 103.37 t/s You can access all of these models for free!

DeepSeek R1 671B is ridiculously good, but running at 158 tokens per second is something out of this world. A model this good running this fast is 3 years ahead of this time. The secret: DeepSeek in this short video isn't running on GPUs. It's running on custom-made chips! This is running on SambaNova and their SN40L chips, which are designed and optimized for running AI workflows. SambaNova is sponsoring my work. We designed GPUs to run games and later realized we could use them for AI. This is very different from creating a chip specifically for AI from the ground up. Just think about this for a second: One single SNL40 chip can simultaneously hold 100+ models (trillions of parameters) in memory! An agentic workflow that uses multiple models simultaneously won't need to swap models from memory even once! By the way, the Llama family of models screams on the SambaNova Cloud. Here are some numbers from my latest tests: • Llama 3.3 70B: 367.32 t/s • Llama 3.2 1B: 2381.82 t/s • Llama 3.2 3B: 1335.71 t/s • Llama 3.1 8B: 929.58 t/s • Llama 3.1 70B: 369.16 t/s • Llama 3.1 405B: 103.37 t/s You can access all of these models for free!

Santiago

130,934 views • 1 year ago

Fox Host: If you can… take advantage of the oil, that is real positive. It could affect the war in Ukraine. Russia is banking on the oil it's getting. If you can bring oil prices down, by getting this oil…

Fox Host: If you can… take advantage of the oil, that is real positive. It could affect the war in Ukraine. Russia is banking on the oil it's getting. If you can bring oil prices down, by getting this oil…

Acyn

42,241 views • 6 months ago

Terry Francona reflecting on the 2025 Reds: "We aren't the finished product, but they competed like crazy and that is what we asked them to do...even the last couple nights -- we're kind of getting our ass handed to us -- but they keep going. That's a good trait."

Terry Francona reflecting on the 2025 Reds: "We aren't the finished product, but they competed like crazy and that is what we asked them to do...even the last couple nights -- we're kind of getting our ass handed to us -- but they keep going. That's a good trait."

Charlie Clifford

303,874 views • 9 months ago

lets chat about the first week of SMITE 2 Ranked and Matchmaking, and some of what to expect in the future spoiler: there is matchmaking, we are working on it (and it's slowly getting better)

lets chat about the first week of SMITE 2 Ranked and Matchmaking, and some of what to expect in the future spoiler: there is matchmaking, we are working on it (and it's slowly getting better)

innocentrabbit

65,474 views • 1 year ago

😳 "It's every week we're talking about VAR getting it wrong." 📺 "It's time they start getting people in who've played football." Jermaine Pennant calls on the PGMOL to use ex-footballers to help VAR

😳 "It's every week we're talking about VAR getting it wrong." 📺 "It's time they start getting people in who've played football." Jermaine Pennant calls on the PGMOL to use ex-footballers to help VAR

talkSPORT

65,598 views • 10 months ago

🚨🇺🇸ELON: DEMS HATE US BECAUSE WE’RE GETTING RID OF CORRUPTION “The reason we're seeing this extreme amount of hatred and violence is because we're actually succeeding in getting rid of corruption and waste. If we weren't succeeding in getting rid of corruption and waste, they wouldn't care. But we are succeeding. Without President Trump, this would be impossible. I'd just like to say, it's only possible with this president. We've got this narrow window of opportunity with a House, a Senate and a popular vote—and really, we're just implementing the will of the people—where we can finally take corrective action. We can fix the government, fix the country, and create a great future. And again, it's only possible because of President Trump. I can't say enough good things about him.” Source: Fox News, March 18, 2025

🚨🇺🇸ELON: DEMS HATE US BECAUSE WE’RE GETTING RID OF CORRUPTION “The reason we're seeing this extreme amount of hatred and violence is because we're actually succeeding in getting rid of corruption and waste. If we weren't succeeding in getting rid of corruption and waste, they wouldn't care. But we are succeeding. Without President Trump, this would be impossible. I'd just like to say, it's only possible with this president. We've got this narrow window of opportunity with a House, a Senate and a popular vote—and really, we're just implementing the will of the people—where we can finally take corrective action. We can fix the government, fix the country, and create a great future. And again, it's only possible because of President Trump. I can't say enough good things about him.” Source: Fox News, March 18, 2025

Mario Nawfal

99,911 views • 1 year ago

We asked Chris Best about Substack's business model. "It's completely free to publish on Substack." "We make money when you make money. We're tuning the algorithm to introduce people to things they deeply value and might pay for." "The emergent effect of that is very different than getting people to spend as much time on the platform as possible."

We asked Chris Best about Substack's business model. "It's completely free to publish on Substack." "We make money when you make money. We're tuning the algorithm to introduce people to things they deeply value and might pay for." "The emergent effect of that is very different than getting people to spend as much time on the platform as possible."

TBPN

12,785 views • 1 year ago