正在加载视频...

视频加载失败

返回首页

Llama 3 8B running 1.89 tokens/s on a Raspberry Pi 5 is pretty CRAZY

adamcohenhillel's profile picture

Adam

10,569 subscribers

931,465 次观看 • 2 年前 •via X (Twitter)

LIVE

Anya Rossi• Live Now

Private livecam show

10 条评论

Gillinghammer 的头像
Gillinghammer2 年前

every house needs one of these in the event that civilization falls and the internet dies

Adam C.H. 的头像
Adam C.H.2 年前

Yep, working on it

Ravi__ 的头像
Ravi__2 年前

Wow ! What is the context window again?

Adam C.H. 的头像
Adam C.H.2 年前

8k!

b4rtaz 的头像
b4rtaz2 年前

On 2 x Raspberry Pi it should be ~3.5 tokens/s. 7B performs 295ms/token

Daniel Merja 的头像
Daniel Merja2 年前

the maxed ram version?

Adam C.H. 的头像
Adam C.H.2 年前

Yep, 8gb

Kevin Loi 的头像
Kevin Loi2 年前

You're cooking!

deepak 🤖 的头像
deepak 🤖2 年前

super impressive, are you using an external fan for your Pi?

Adam C.H. 的头像
Adam C.H.2 年前

not yet, but I should haha

相关视频

MM8108 HaLow is now running on Raspberry Pi 5 with OpenWRT
9:05

MM8108 HaLow is now running on Raspberry Pi 5 with OpenWRT

Data Slayer

38,646 次观看 • 3 个月前

And here we go - running Llama 3.2 3B on Raspberry Pi 5 with AI HAT+ 2. This thing is not going to break any speed records, but it’s incredible that we can now get such a powerful model on such a constrained system. Just a little over 3 years since ChatGPT launched.
0:40

And here we go - running Llama 3.2 3B on Raspberry Pi 5 with AI HAT+ 2. This thing is not going to break any speed records, but it’s incredible that we can now get such a powerful model on such a constrained system. Just a little over 3 years since ChatGPT launched.

Bojan Tunguz

27,108 次观看 • 7 个月前

NEW Raspberry Pi AI Hat Is a Massive Upgrade Raspberry Pi
2:08

NEW Raspberry Pi AI Hat Is a Massive Upgrade Raspberry Pi

NetworkChuck

225,296 次观看 • 8 个月前

Can You ACTUALLY Run AI on a Raspberry Pi? Raspberry Pi
0:28

Can You ACTUALLY Run AI on a Raspberry Pi? Raspberry Pi

NetworkChuck

11,444 次观看 • 7 个月前

The Raspberry Pi 5 Compute Module 5 (CM5) is here! 🧵
15:56

The Raspberry Pi 5 Compute Module 5 (CM5) is here! 🧵

Jeff Geerling

47,406 次观看 • 1 年前

Raspberry Pi power meets industrial smarts with SECO Pi Vision 10.1 CM5. Pi Vision 10.1 CM5 is a rugged, ready-to-roll platform built around Raspberry Pi Compute Module 5. It’s tailor-made for IoT developers and system integrators. This isn’t just any old human–machine interface. It’s a Raspberry Pi gone pro.
1:27

Raspberry Pi power meets industrial smarts with SECO Pi Vision 10.1 CM5. Pi Vision 10.1 CM5 is a rugged, ready-to-roll platform built around Raspberry Pi Compute Module 5. It’s tailor-made for IoT developers and system integrators. This isn’t just any old human–machine interface. It’s a Raspberry Pi gone pro.

Raspberry Pi

16,618 次观看 • 10 个月前

DeepSeek R1 671B is ridiculously good, but running at 158 tokens per second is something out of this world. A model *this good* running *this fast* is 3 years ahead of this time. The secret: DeepSeek in this short video isn't running on GPUs. It's running on custom-made chips! This is running on SambaNova and their SN40L chips, which are designed and optimized for running AI workflows. SambaNova is sponsoring my work. We designed GPUs to run games and later realized we could use them for AI. This is very different from creating a chip specifically for AI from the ground up. Just think about this for a second: One single SNL40 chip can simultaneously hold 100+ models (trillions of parameters) in memory! An agentic workflow that uses multiple models simultaneously won't need to swap models from memory even once! By the way, the Llama family of models *screams* on the SambaNova Cloud. Here are some numbers from my latest tests: • Llama 3.3 70B: 367.32 t/s • Llama 3.2 1B: 2381.82 t/s • Llama 3.2 3B: 1335.71 t/s • Llama 3.1 8B: 929.58 t/s • Llama 3.1 70B: 369.16 t/s • Llama 3.1 405B: 103.37 t/s You can access all of these models for free!
1:34

DeepSeek R1 671B is ridiculously good, but running at 158 tokens per second is something out of this world. A model *this good* running *this fast* is 3 years ahead of this time. The secret: DeepSeek in this short video isn't running on GPUs. It's running on custom-made chips! This is running on SambaNova and their SN40L chips, which are designed and optimized for running AI workflows. SambaNova is sponsoring my work. We designed GPUs to run games and later realized we could use them for AI. This is very different from creating a chip specifically for AI from the ground up. Just think about this for a second: One single SNL40 chip can simultaneously hold 100+ models (trillions of parameters) in memory! An agentic workflow that uses multiple models simultaneously won't need to swap models from memory even once! By the way, the Llama family of models *screams* on the SambaNova Cloud. Here are some numbers from my latest tests: • Llama 3.3 70B: 367.32 t/s • Llama 3.2 1B: 2381.82 t/s • Llama 3.2 3B: 1335.71 t/s • Llama 3.1 8B: 929.58 t/s • Llama 3.1 70B: 369.16 t/s • Llama 3.1 405B: 103.37 t/s You can access all of these models for free!

Santiago

131,077 次观看 • 1 年前

OpenAI's nightmare: DeepSeek R1 on a Raspberry Pi
4:17

OpenAI's nightmare: DeepSeek R1 on a Raspberry Pi

Jeff Geerling

23,736 次观看 • 1 年前

A brand new edition of The Official Raspberry Pi Beginner's Guide is OUT NOW! Updated for Raspberry Pi 500+ and OS Trixie:
0:31

A brand new edition of The Official Raspberry Pi Beginner's Guide is OUT NOW! Updated for Raspberry Pi 500+ and OS Trixie:

Raspberry Pi

18,003 次观看 • 7 个月前

How a Raspberry Pi is made. Full video here:
0:57

How a Raspberry Pi is made. Full video here:

Jeff Geerling

1,164,052 次观看 • 3 年前

Llama3.1 on a raspberry pi Thanks for Llamafile 😍
0:36

Llama3.1 on a raspberry pi Thanks for Llamafile 😍

Mike Bird

126,760 次观看 • 2 年前

Duupr Clock ⏰ built with Flutter Firebase running on a Raspberry Pi loaded w/ PrimeOS 👍Animations in Rive 💙 #duupr #clock #raspberrypi
0:59

Duupr Clock ⏰ built with Flutter Firebase running on a Raspberry Pi loaded w/ PrimeOS 👍Animations in Rive 💙 #duupr #clock #raspberrypi

Roman Jaquez

32,774 次观看 • 1 年前

🚨 Meet Raspberry Pi Pico 2, our new $5 microcontroller board 🚨 The next big tiny thing, built using RP2350: a high-performance, secure microcontroller designed here at Raspberry Pi.
0:33

🚨 Meet Raspberry Pi Pico 2, our new $5 microcontroller board 🚨 The next big tiny thing, built using RP2350: a high-performance, secure microcontroller designed here at Raspberry Pi.

Raspberry Pi

876,366 次观看 • 2 年前

Running Qwen3 8B thinking on an iPhone Air with MLX. The model is quantized to 4-bit and runs pretty well.
0:45

Running Qwen3 8B thinking on an iPhone Air with MLX. The model is quantized to 4-bit and runs pretty well.

Awni Hannun

215,597 次观看 • 11 个月前

I can't believe this works, but I got DeepSeek-V4-Flash (284B params) running on a Raspberry Pi 5 (8GB edition) at >1tok/s @ ~8W during full-tilt inference! It uses an untouched copy of antirez's GGUF. Took 160+ experiments over 5 days between GPT-5.5 xhigh and Opus 4.8 max.
0:32

I can't believe this works, but I got DeepSeek-V4-Flash (284B params) running on a Raspberry Pi 5 (8GB edition) at >1tok/s @ ~8W during full-tilt inference! It uses an untouched copy of antirez's GGUF. Took 160+ experiments over 5 days between GPT-5.5 xhigh and Opus 4.8 max.

Dan Woods

185,249 次观看 • 3 个月前

GPS-free visual-inertial navigation in real time on Raspberry Pi 5 with a consumer-grade IMU and camera (and this is not even the slowest CPU this runs on!)
0:59

GPS-free visual-inertial navigation in real time on Raspberry Pi 5 with a consumer-grade IMU and camera (and this is not even the slowest CPU this runs on!)

Otto Seiskari

264,690 次观看 • 1 年前

Multimodal Ichigo Llama 3.1 - Real Time Voice AI 🔥 > WhisperSpeech X Llama 3.1 8B > Trained on 50K hours of speech (7 languages) > Continually trained on 45hrs 10x A1000s > MLS -> WhisperVQ tokens -> Llama 3.1 > Instruction tuned on 1.89M samples > 70% speech, 20% transcription, 10% text > Apache 2.0 licensed ⚡ Architecture: > WhisperSpeech/ VQ for Semantic Tokens > Llama 3.1 8B Instruct for Text backbone > Early fusion (Chameleon) I'm super bullish on Homebrew to Menlo and early fusion, audio and text, multimodal models! (P.S. Play with the demo on Hugging Face)
0:32

Multimodal Ichigo Llama 3.1 - Real Time Voice AI 🔥 > WhisperSpeech X Llama 3.1 8B > Trained on 50K hours of speech (7 languages) > Continually trained on 45hrs 10x A1000s > MLS -> WhisperVQ tokens -> Llama 3.1 > Instruction tuned on 1.89M samples > 70% speech, 20% transcription, 10% text > Apache 2.0 licensed ⚡ Architecture: > WhisperSpeech/ VQ for Semantic Tokens > Llama 3.1 8B Instruct for Text backbone > Early fusion (Chameleon) I'm super bullish on Homebrew to Menlo and early fusion, audio and text, multimodal models! (P.S. Play with the demo on Hugging Face)

Vaibhav (VB) Srivastav

82,170 次观看 • 1 年前

Testing #RePlayOS Alpha in Raspberry Pi 5. Interlaced video, 24bit audio quality, on-the-fly instant resolution changes, and more… Powered by RGB-Pi 💪
2:13

Testing #RePlayOS Alpha in Raspberry Pi 5. Interlaced video, 24bit audio quality, on-the-fly instant resolution changes, and more… Powered by RGB-Pi 💪

RTA

10,218 次观看 • 2 年前