正在加载视频...

视频加载失败

加载此视频时出现问题。这可能是由于临时网络问题，或视频可能不可用。

We keep getting asked by users if they can use the 70B parameter model in production. We're serving the chat variant of Llama-2 70B on 2xA100 and getting pretty great throughput — it's cooking!

Tuhin Srivastava

3,324 subscribers

20,037 次观看 • 3 年前 •via X (Twitter)

Anya Rossi• Live Now

Private livecam show

6 条评论

Dwayne 的头像

Dwayne3 年前

It's such a shame that Llama 2 is so neutered with all of the excessive safety they wrapped it in. Hopefully, we see an unlocked 70b Llama 2 soon.

Tuhin Srivastava 的头像

Tuhin Srivastava3 年前

@DwayneCodes Yes, I'm sure it's coming. The non-chat model has less of it, I believe.

Bill Njoroge 的头像

Bill Njoroge3 年前

how many it/s?

Debargha Ganguly 的头像

Debargha Ganguly3 年前

Will give this a try tomorrow, the readme on the repo says 4xA100 vs 2xA100 in your tweet. What’s the throughput you’re seeing?

Tuhin Srivastava 的头像

Tuhin Srivastava3 年前

I’ll push an updated truss tonight!

Muskan Khandelwal 的头像

Muskan Khandelwal3 年前

Hey Tuhin, is there some way I can get to contribute to Baseten ? I can learn and deliver at blazing fast speed.

相关视频

Finally! Codellama-70b as Copilot in VSCode! 😱 You can now harness the power of the most advanced code generation model, Code Llama 70B, right in Visual Studio Code with Perplexity This model has outperformed even GPT-4 in code writing, and it's now seamlessly integrated into our editor 🥳 Do you anticipate the emergence of more open-source models that surpass GPT-4?

Finally! Codellama-70b as Copilot in VSCode! 😱 You can now harness the power of the most advanced code generation model, Code Llama 70B, right in Visual Studio Code with Perplexity This model has outperformed even GPT-4 in code writing, and it's now seamlessly integrated into our editor 🥳 Do you anticipate the emergence of more open-source models that surpass GPT-4?

Daniel San

69,499 次观看 • 2 年前

Our Llama-3.1-Nemotron-70B-Instruct model is a leading model on the 🏆 Arena Hard benchmark (85) from Arena. Arena Hard uses a data pipeline to build high-quality benchmarks from live data in Chatbot Arena, and is known for its predictive ability of Chatbot Arena Elo score as well as separability between helpful and less helpful models. Use our customized model Llama-3.1-Nemotron-70B to improve the helpfulness of LLM generated responses in your applications. 📥 Try on our API catalog: 📥 On GitHub: 📥 Or on Hugging Face:

Our Llama-3.1-Nemotron-70B-Instruct model is a leading model on the 🏆 Arena Hard benchmark (85) from Arena. Arena Hard uses a data pipeline to build high-quality benchmarks from live data in Chatbot Arena, and is known for its predictive ability of Chatbot Arena Elo score as well as separability between helpful and less helpful models. Use our customized model Llama-3.1-Nemotron-70B to improve the helpfulness of LLM generated responses in your applications. 📥 Try on our API catalog: 📥 On GitHub: 📥 Or on Hugging Face:

NVIDIA AI Developer

140,756 次观看 • 1 年前

"One of the biggest misconceptions" Cerebras CFO Bob Komin pushes back on the small-models narrative. "We serve all models, and there is no limit to the size of the models that we can serve. Today, we're serving trillion parameter models. We're serving trillion parameter models that are internal for OpenAI today. We are currently running OpenAI 5.4 and 5.5 with them."

"One of the biggest misconceptions" Cerebras CFO Bob Komin pushes back on the small-models narrative. "We serve all models, and there is no limit to the size of the models that we can serve. Today, we're serving trillion parameter models. We're serving trillion parameter models that are internal for OpenAI today. We are currently running OpenAI 5.4 and 5.5 with them."

Deirdre Bosa

84,422 次观看 • 2 个月前

NVIDIA silently released the Llama 3.1 70B Instruct Nemotron model which is topping leaderboards. You can try it here: I sat down with one of the researchers to learn how it performed so well and why they built this 🤙 Anything else I should ask him?

NVIDIA silently released the Llama 3.1 70B Instruct Nemotron model which is topping leaderboards. You can try it here: I sat down with one of the researchers to learn how it performed so well and why they built this 🤙 Anything else I should ask him?

Nader Khalil🍊

48,344 次观看 • 1 年前

This is the fastest I've seen Llama 3.3 running anywhere! Llama 3.3 70B running at 652 t/s is lightning fast. And if you want Llama 3.1, here are the speeds I was able to get: • Llama 3.1 8B: 1006 t/s • Llama 3.1 70B: 709 t/s • Llama 3.1 405B: 206 t/s (You can access all of these models for free! See the link below.) This speed is incredible, but the interesting part is what's happening behind the scenes: These models aren't running on a GPU! In this video, I'm using the SambaNova cloud to access these models. They built a custom chip (SN40L) optimized for AI workflows. A single SN40L chip can hold hundreds of models (trillions of parameters) in memory! The speed alone is a huge deal, but the big advantage is for agentic workflows running multiple specialized models. A GPU can only host a single model and switch (unload and load) to a different model if necessary. An SN40L, on the other hand, can host every model at once, making it much faster. Here is the video where you can see how fast these chips are:

This is the fastest I've seen Llama 3.3 running anywhere! Llama 3.3 70B running at 652 t/s is lightning fast. And if you want Llama 3.1, here are the speeds I was able to get: • Llama 3.1 8B: 1006 t/s • Llama 3.1 70B: 709 t/s • Llama 3.1 405B: 206 t/s (You can access all of these models for free! See the link below.) This speed is incredible, but the interesting part is what's happening behind the scenes: These models aren't running on a GPU! In this video, I'm using the SambaNova cloud to access these models. They built a custom chip (SN40L) optimized for AI workflows. A single SN40L chip can hold hundreds of models (trillions of parameters) in memory! The speed alone is a huge deal, but the big advantage is for agentic workflows running multiple specialized models. A GPU can only host a single model and switch (unload and load) to a different model if necessary. An SN40L, on the other hand, can host every model at once, making it much faster. Here is the video where you can see how fast these chips are:

Santiago

97,528 次观看 • 1 年前

Using LLaMA 3 70B on Groq Inc to instantly refactor and document code. The implications for software engineering are wild. Gone are the days of waiting on an LLM for suggestions or code changes. Now, it's an instant feedback loop. Demo link in the comments:

Using LLaMA 3 70B on Groq Inc to instantly refactor and document code. The implications for software engineering are wild. Gone are the days of waiting on an LLM for suggestions or code changes. Now, it's an instant feedback loop. Demo link in the comments:

Matt Shumer

442,695 次观看 • 2 年前

"The reason we're seeing this extreme amount of hatred and violence is because we're actually succeeding in getting rid of corruption and waste, If we weren't succeeding in getting rid of corruption and waste, they wouldn't care, but we are succeeding." -Elon Musk

"The reason we're seeing this extreme amount of hatred and violence is because we're actually succeeding in getting rid of corruption and waste, If we weren't succeeding in getting rid of corruption and waste, they wouldn't care, but we are succeeding." -Elon Musk

SMX 🇺🇸

77,274 次观看 • 1 年前

"The reason we're seeing this extreme amount of hatred and violence is because we're actually succeeding in getting rid of corruption and waste. If we weren't succeeding in getting rid of corruption and waste, they wouldn't care. But we are succeeding." -Elon Musk

"The reason we're seeing this extreme amount of hatred and violence is because we're actually succeeding in getting rid of corruption and waste. If we weren't succeeding in getting rid of corruption and waste, they wouldn't care. But we are succeeding." -Elon Musk

SMX 🇺🇸

579,166 次观看 • 1 年前

President Trump on trade deal with Japan: "So we're paying zero (tariff). They're paying 15. We're getting $550 billion, and to be exact, it's 90% of that but we control the whole lot of it. And it's really been great."

President Trump on trade deal with Japan: "So we're paying zero (tariff). They're paying 15. We're getting $550 billion, and to be exact, it's 90% of that but we control the whole lot of it. And it's really been great."

CSPAN

17,610 次观看 • 1 年前

DeepSeek R1 671B is ridiculously good, but running at 158 tokens per second is something out of this world. A model *this good* running *this fast* is 3 years ahead of this time. The secret: DeepSeek in this short video isn't running on GPUs. It's running on custom-made chips! This is running on SambaNova and their SN40L chips, which are designed and optimized for running AI workflows. SambaNova is sponsoring my work. We designed GPUs to run games and later realized we could use them for AI. This is very different from creating a chip specifically for AI from the ground up. Just think about this for a second: One single SNL40 chip can simultaneously hold 100+ models (trillions of parameters) in memory! An agentic workflow that uses multiple models simultaneously won't need to swap models from memory even once! By the way, the Llama family of models *screams* on the SambaNova Cloud. Here are some numbers from my latest tests: • Llama 3.3 70B: 367.32 t/s • Llama 3.2 1B: 2381.82 t/s • Llama 3.2 3B: 1335.71 t/s • Llama 3.1 8B: 929.58 t/s • Llama 3.1 70B: 369.16 t/s • Llama 3.1 405B: 103.37 t/s You can access all of these models for free!

DeepSeek R1 671B is ridiculously good, but running at 158 tokens per second is something out of this world. A model this good running this fast is 3 years ahead of this time. The secret: DeepSeek in this short video isn't running on GPUs. It's running on custom-made chips! This is running on SambaNova and their SN40L chips, which are designed and optimized for running AI workflows. SambaNova is sponsoring my work. We designed GPUs to run games and later realized we could use them for AI. This is very different from creating a chip specifically for AI from the ground up. Just think about this for a second: One single SNL40 chip can simultaneously hold 100+ models (trillions of parameters) in memory! An agentic workflow that uses multiple models simultaneously won't need to swap models from memory even once! By the way, the Llama family of models screams on the SambaNova Cloud. Here are some numbers from my latest tests: • Llama 3.3 70B: 367.32 t/s • Llama 3.2 1B: 2381.82 t/s • Llama 3.2 3B: 1335.71 t/s • Llama 3.1 8B: 929.58 t/s • Llama 3.1 70B: 369.16 t/s • Llama 3.1 405B: 103.37 t/s You can access all of these models for free!

Santiago

130,934 次观看 • 1 年前

Fox Host: If you can… take advantage of the oil, that is real positive. It could affect the war in Ukraine. Russia is banking on the oil it's getting. If you can bring oil prices down, by getting this oil…

Fox Host: If you can… take advantage of the oil, that is real positive. It could affect the war in Ukraine. Russia is banking on the oil it's getting. If you can bring oil prices down, by getting this oil…

Acyn

42,241 次观看 • 6 个月前

Terry Francona reflecting on the 2025 Reds: "We aren't the finished product, but they competed like crazy and that is what we asked them to do...even the last couple nights -- we're kind of getting our ass handed to us -- but they keep going. That's a good trait."

Terry Francona reflecting on the 2025 Reds: "We aren't the finished product, but they competed like crazy and that is what we asked them to do...even the last couple nights -- we're kind of getting our ass handed to us -- but they keep going. That's a good trait."

Charlie Clifford

303,874 次观看 • 9 个月前

lets chat about the first week of SMITE 2 Ranked and Matchmaking, and some of what to expect in the future spoiler: there is matchmaking, we are working on it (and it's slowly getting better)

lets chat about the first week of SMITE 2 Ranked and Matchmaking, and some of what to expect in the future spoiler: there is matchmaking, we are working on it (and it's slowly getting better)

innocentrabbit

65,474 次观看 • 1 年前

😳 "It's every week we're talking about VAR getting it wrong." 📺 "It's time they start getting people in who've played football." Jermaine Pennant calls on the PGMOL to use ex-footballers to help VAR

😳 "It's every week we're talking about VAR getting it wrong." 📺 "It's time they start getting people in who've played football." Jermaine Pennant calls on the PGMOL to use ex-footballers to help VAR

talkSPORT

65,598 次观看 • 10 个月前

🚨🇺🇸ELON: DEMS HATE US BECAUSE WE’RE GETTING RID OF CORRUPTION “The reason we're seeing this extreme amount of hatred and violence is because we're actually succeeding in getting rid of corruption and waste. If we weren't succeeding in getting rid of corruption and waste, they wouldn't care. But we are succeeding. Without President Trump, this would be impossible. I'd just like to say, it's only possible with this president. We've got this narrow window of opportunity with a House, a Senate and a popular vote—and really, we're just implementing the will of the people—where we can finally take corrective action. We can fix the government, fix the country, and create a great future. And again, it's only possible because of President Trump. I can't say enough good things about him.” Source: Fox News, March 18, 2025

🚨🇺🇸ELON: DEMS HATE US BECAUSE WE’RE GETTING RID OF CORRUPTION “The reason we're seeing this extreme amount of hatred and violence is because we're actually succeeding in getting rid of corruption and waste. If we weren't succeeding in getting rid of corruption and waste, they wouldn't care. But we are succeeding. Without President Trump, this would be impossible. I'd just like to say, it's only possible with this president. We've got this narrow window of opportunity with a House, a Senate and a popular vote—and really, we're just implementing the will of the people—where we can finally take corrective action. We can fix the government, fix the country, and create a great future. And again, it's only possible because of President Trump. I can't say enough good things about him.” Source: Fox News, March 18, 2025

Mario Nawfal

99,911 次观看 • 1 年前

We asked Chris Best about Substack's business model. "It's completely free to publish on Substack." "We make money when you make money. We're tuning the algorithm to introduce people to things they deeply value and might pay for." "The emergent effect of that is very different than getting people to spend as much time on the platform as possible."

We asked Chris Best about Substack's business model. "It's completely free to publish on Substack." "We make money when you make money. We're tuning the algorithm to introduce people to things they deeply value and might pay for." "The emergent effect of that is very different than getting people to spend as much time on the platform as possible."

TBPN

12,785 次观看 • 1 年前