Загрузка видео...

Не удалось загрузить видео

На главную

We keep getting asked by users if they can use the 70B parameter model in production. We're serving the chat variant of Llama-2 70B on 2xA100 and getting pretty great throughput — it's cooking!

tuhinone's profile picture

Tuhin Srivastava

3,324 subscribers

20,037 просмотров • 3 лет назад •via X (Twitter)

LIVE

Anya Rossi• Live Now

Private livecam show

Комментарии: 6

Фото профиля Dwayne
Dwayne3 лет назад

It's such a shame that Llama 2 is so neutered with all of the excessive safety they wrapped it in. Hopefully, we see an unlocked 70b Llama 2 soon.

Фото профиля Tuhin Srivastava
Tuhin Srivastava3 лет назад

@DwayneCodes Yes, I'm sure it's coming. The non-chat model has less of it, I believe.

Фото профиля Bill Njoroge
Bill Njoroge3 лет назад

how many it/s?

Фото профиля Debargha Ganguly
Debargha Ganguly3 лет назад

Will give this a try tomorrow, the readme on the repo says 4xA100 vs 2xA100 in your tweet. What’s the throughput you’re seeing?

Фото профиля Tuhin Srivastava
Tuhin Srivastava3 лет назад

I’ll push an updated truss tonight!

Фото профиля Muskan Khandelwal
Muskan Khandelwal3 лет назад

Hey Tuhin, is there some way I can get to contribute to Baseten ? I can learn and deliver at blazing fast speed.

Похожие видео

Finally! Codellama-70b as Copilot in VSCode! 😱 You can now harness the power of the most advanced code generation model, Code Llama 70B, right in Visual Studio Code with Perplexity This model has outperformed even GPT-4 in code writing, and it's now seamlessly integrated into our editor 🥳 Do you anticipate the emergence of more open-source models that surpass GPT-4?
0:33

Finally! Codellama-70b as Copilot in VSCode! 😱 You can now harness the power of the most advanced code generation model, Code Llama 70B, right in Visual Studio Code with Perplexity This model has outperformed even GPT-4 in code writing, and it's now seamlessly integrated into our editor 🥳 Do you anticipate the emergence of more open-source models that surpass GPT-4?

Daniel San

69,499 просмотров • 2 лет назад

Our Llama-3.1-Nemotron-70B-Instruct model is a leading model on the 🏆 Arena Hard benchmark (85) from Arena. Arena Hard uses a data pipeline to build high-quality benchmarks from live data in Chatbot Arena, and is known for its predictive ability of Chatbot Arena Elo score as well as separability between helpful and less helpful models. Use our customized model Llama-3.1-Nemotron-70B to improve the helpfulness of LLM generated responses in your applications. 📥 Try on our API catalog: 📥 On GitHub: 📥 Or on Hugging Face:
0:43

Our Llama-3.1-Nemotron-70B-Instruct model is a leading model on the 🏆 Arena Hard benchmark (85) from Arena. Arena Hard uses a data pipeline to build high-quality benchmarks from live data in Chatbot Arena, and is known for its predictive ability of Chatbot Arena Elo score as well as separability between helpful and less helpful models. Use our customized model Llama-3.1-Nemotron-70B to improve the helpfulness of LLM generated responses in your applications. 📥 Try on our API catalog: 📥 On GitHub: 📥 Or on Hugging Face:

NVIDIA AI Developer

140,756 просмотров • 1 год назад

"One of the biggest misconceptions" Cerebras CFO Bob Komin pushes back on the small-models narrative. "We serve all models, and there is no limit to the size of the models that we can serve. Today, we're serving trillion parameter models. We're serving trillion parameter models that are internal for OpenAI today. We are currently running OpenAI 5.4 and 5.5 with them."
0:50

"One of the biggest misconceptions" Cerebras CFO Bob Komin pushes back on the small-models narrative. "We serve all models, and there is no limit to the size of the models that we can serve. Today, we're serving trillion parameter models. We're serving trillion parameter models that are internal for OpenAI today. We are currently running OpenAI 5.4 and 5.5 with them."

Deirdre Bosa

84,422 просмотров • 2 месяцев назад

NVIDIA silently released the Llama 3.1 70B Instruct Nemotron model which is topping leaderboards. You can try it here: I sat down with one of the researchers to learn how it performed so well and why they built this 🤙 Anything else I should ask him?
9:59

NVIDIA silently released the Llama 3.1 70B Instruct Nemotron model which is topping leaderboards. You can try it here: I sat down with one of the researchers to learn how it performed so well and why they built this 🤙 Anything else I should ask him?

Nader Khalil🍊

48,344 просмотров • 1 год назад

This is the fastest I've seen Llama 3.3 running anywhere! Llama 3.3 70B running at 652 t/s is lightning fast. And if you want Llama 3.1, here are the speeds I was able to get: • Llama 3.1 8B: 1006 t/s • Llama 3.1 70B: 709 t/s • Llama 3.1 405B: 206 t/s (You can access all of these models for free! See the link below.) This speed is incredible, but the interesting part is what's happening behind the scenes: These models aren't running on a GPU! In this video, I'm using the SambaNova cloud to access these models. They built a custom chip (SN40L) optimized for AI workflows. A single SN40L chip can hold hundreds of models (trillions of parameters) in memory! The speed alone is a huge deal, but the big advantage is for agentic workflows running multiple specialized models. A GPU can only host a single model and switch (unload and load) to a different model if necessary. An SN40L, on the other hand, can host every model at once, making it much faster. Here is the video where you can see how fast these chips are:
7:24

This is the fastest I've seen Llama 3.3 running anywhere! Llama 3.3 70B running at 652 t/s is lightning fast. And if you want Llama 3.1, here are the speeds I was able to get: • Llama 3.1 8B: 1006 t/s • Llama 3.1 70B: 709 t/s • Llama 3.1 405B: 206 t/s (You can access all of these models for free! See the link below.) This speed is incredible, but the interesting part is what's happening behind the scenes: These models aren't running on a GPU! In this video, I'm using the SambaNova cloud to access these models. They built a custom chip (SN40L) optimized for AI workflows. A single SN40L chip can hold hundreds of models (trillions of parameters) in memory! The speed alone is a huge deal, but the big advantage is for agentic workflows running multiple specialized models. A GPU can only host a single model and switch (unload and load) to a different model if necessary. An SN40L, on the other hand, can host every model at once, making it much faster. Here is the video where you can see how fast these chips are:

Santiago

97,528 просмотров • 1 год назад

Using LLaMA 3 70B on Groq Inc to instantly refactor and document code. The implications for software engineering are wild. Gone are the days of waiting on an LLM for suggestions or code changes. Now, it's an instant feedback loop. Demo link in the comments:
0:21

Using LLaMA 3 70B on Groq Inc to instantly refactor and document code. The implications for software engineering are wild. Gone are the days of waiting on an LLM for suggestions or code changes. Now, it's an instant feedback loop. Demo link in the comments:

Matt Shumer

442,695 просмотров • 2 лет назад

"The reason we're seeing this extreme amount of hatred and violence is because we're actually succeeding in getting rid of corruption and waste, If we weren't succeeding in getting rid of corruption and waste, they wouldn't care, but we are succeeding." -Elon Musk
0:45

"The reason we're seeing this extreme amount of hatred and violence is because we're actually succeeding in getting rid of corruption and waste, If we weren't succeeding in getting rid of corruption and waste, they wouldn't care, but we are succeeding." -Elon Musk

SMX 🇺🇸

77,274 просмотров • 1 год назад

"The reason we're seeing this extreme amount of hatred and violence is because we're actually succeeding in getting rid of corruption and waste. If we weren't succeeding in getting rid of corruption and waste, they wouldn't care. But we are succeeding." -Elon Musk
0:44

"The reason we're seeing this extreme amount of hatred and violence is because we're actually succeeding in getting rid of corruption and waste. If we weren't succeeding in getting rid of corruption and waste, they wouldn't care. But we are succeeding." -Elon Musk

SMX 🇺🇸

579,166 просмотров • 1 год назад

President Trump on trade deal with Japan: "So we're paying zero (tariff). They're paying 15. We're getting $550 billion, and to be exact, it's 90% of that but we control the whole lot of it. And it's really been great."
2:06

President Trump on trade deal with Japan: "So we're paying zero (tariff). They're paying 15. We're getting $550 billion, and to be exact, it's 90% of that but we control the whole lot of it. And it's really been great."

CSPAN

17,610 просмотров • 1 год назад

DeepSeek R1 671B is ridiculously good, but running at 158 tokens per second is something out of this world. A model *this good* running *this fast* is 3 years ahead of this time. The secret: DeepSeek in this short video isn't running on GPUs. It's running on custom-made chips! This is running on SambaNova and their SN40L chips, which are designed and optimized for running AI workflows. SambaNova is sponsoring my work. We designed GPUs to run games and later realized we could use them for AI. This is very different from creating a chip specifically for AI from the ground up. Just think about this for a second: One single SNL40 chip can simultaneously hold 100+ models (trillions of parameters) in memory! An agentic workflow that uses multiple models simultaneously won't need to swap models from memory even once! By the way, the Llama family of models *screams* on the SambaNova Cloud. Here are some numbers from my latest tests: • Llama 3.3 70B: 367.32 t/s • Llama 3.2 1B: 2381.82 t/s • Llama 3.2 3B: 1335.71 t/s • Llama 3.1 8B: 929.58 t/s • Llama 3.1 70B: 369.16 t/s • Llama 3.1 405B: 103.37 t/s You can access all of these models for free!
1:34

DeepSeek R1 671B is ridiculously good, but running at 158 tokens per second is something out of this world. A model *this good* running *this fast* is 3 years ahead of this time. The secret: DeepSeek in this short video isn't running on GPUs. It's running on custom-made chips! This is running on SambaNova and their SN40L chips, which are designed and optimized for running AI workflows. SambaNova is sponsoring my work. We designed GPUs to run games and later realized we could use them for AI. This is very different from creating a chip specifically for AI from the ground up. Just think about this for a second: One single SNL40 chip can simultaneously hold 100+ models (trillions of parameters) in memory! An agentic workflow that uses multiple models simultaneously won't need to swap models from memory even once! By the way, the Llama family of models *screams* on the SambaNova Cloud. Here are some numbers from my latest tests: • Llama 3.3 70B: 367.32 t/s • Llama 3.2 1B: 2381.82 t/s • Llama 3.2 3B: 1335.71 t/s • Llama 3.1 8B: 929.58 t/s • Llama 3.1 70B: 369.16 t/s • Llama 3.1 405B: 103.37 t/s You can access all of these models for free!

Santiago

130,934 просмотров • 1 год назад

Fox Host: If you can… take advantage of the oil, that is real positive. It could affect the war in Ukraine. Russia is banking on the oil it's getting. If you can bring oil prices down, by getting this oil…
0:37

Fox Host: If you can… take advantage of the oil, that is real positive. It could affect the war in Ukraine. Russia is banking on the oil it's getting. If you can bring oil prices down, by getting this oil…

Acyn

42,241 просмотров • 6 месяцев назад

Terry Francona reflecting on the 2025 Reds: "We aren't the finished product, but they competed like crazy and that is what we asked them to do...even the last couple nights -- we're kind of getting our ass handed to us -- but they keep going. That's a good trait."
0:34

Terry Francona reflecting on the 2025 Reds: "We aren't the finished product, but they competed like crazy and that is what we asked them to do...even the last couple nights -- we're kind of getting our ass handed to us -- but they keep going. That's a good trait."

Charlie Clifford

303,874 просмотров • 9 месяцев назад

lets chat about the first week of SMITE 2 Ranked and Matchmaking, and some of what to expect in the future spoiler: there is matchmaking, we are working on it (and it's slowly getting better)
2:19

lets chat about the first week of SMITE 2 Ranked and Matchmaking, and some of what to expect in the future spoiler: there is matchmaking, we are working on it (and it's slowly getting better)

innocentrabbit

65,474 просмотров • 1 год назад

😳 "It's every week we're talking about VAR getting it wrong." 📺 "It's time they start getting people in who've played football." Jermaine Pennant calls on the PGMOL to use ex-footballers to help VAR
0:58

😳 "It's every week we're talking about VAR getting it wrong." 📺 "It's time they start getting people in who've played football." Jermaine Pennant calls on the PGMOL to use ex-footballers to help VAR

talkSPORT

65,598 просмотров • 10 месяцев назад

🚨🇺🇸ELON: DEMS HATE US BECAUSE WE’RE GETTING RID OF CORRUPTION “The reason we're seeing this extreme amount of hatred and violence is because we're actually succeeding in getting rid of corruption and waste. If we weren't succeeding in getting rid of corruption and waste, they wouldn't care. But we are succeeding. Without President Trump, this would be impossible. I'd just like to say, it's only possible with this president. We've got this narrow window of opportunity with a House, a Senate and a popular vote—and really, we're just implementing the will of the people—where we can finally take corrective action. We can fix the government, fix the country, and create a great future. And again, it's only possible because of President Trump. I can't say enough good things about him.” Source: Fox News, March 18, 2025
0:52

🚨🇺🇸ELON: DEMS HATE US BECAUSE WE’RE GETTING RID OF CORRUPTION “The reason we're seeing this extreme amount of hatred and violence is because we're actually succeeding in getting rid of corruption and waste. If we weren't succeeding in getting rid of corruption and waste, they wouldn't care. But we are succeeding. Without President Trump, this would be impossible. I'd just like to say, it's only possible with this president. We've got this narrow window of opportunity with a House, a Senate and a popular vote—and really, we're just implementing the will of the people—where we can finally take corrective action. We can fix the government, fix the country, and create a great future. And again, it's only possible because of President Trump. I can't say enough good things about him.” Source: Fox News, March 18, 2025

Mario Nawfal

99,911 просмотров • 1 год назад

We asked Chris Best about Substack's business model. "It's completely free to publish on Substack." "We make money when you make money. We're tuning the algorithm to introduce people to things they deeply value and might pay for." "The emergent effect of that is very different than getting people to spend as much time on the platform as possible."
1:24

We asked Chris Best about Substack's business model. "It's completely free to publish on Substack." "We make money when you make money. We're tuning the algorithm to introduce people to things they deeply value and might pay for." "The emergent effect of that is very different than getting people to spend as much time on the platform as possible."

TBPN

12,785 просмотров • 1 год назад