Video yükleniyor...

Video Yüklenemedi

Ana Sayfaya Dön

Llama 3 8B running 1.89 tokens/s on a Raspberry Pi 5 is pretty CRAZY

adamcohenhillel's profile picture

Adam

10,490 subscribers

931,383 görüntüleme • 2 yıl önce •via X (Twitter)

LIVE

Anya Rossi• Live Now

Private livecam show

10 Yorum

Gillinghammer profil fotoğrafı
Gillinghammer2 yıl önce

every house needs one of these in the event that civilization falls and the internet dies

Adam C.H. profil fotoğrafı
Adam C.H.2 yıl önce

Yep, working on it

Ravi__ profil fotoğrafı
Ravi__2 yıl önce

Wow ! What is the context window again?

Adam C.H. profil fotoğrafı
Adam C.H.2 yıl önce

8k!

b4rtaz profil fotoğrafı
b4rtaz2 yıl önce

On 2 x Raspberry Pi it should be ~3.5 tokens/s. 7B performs 295ms/token

Daniel Merja profil fotoğrafı
Daniel Merja2 yıl önce

the maxed ram version?

Adam C.H. profil fotoğrafı
Adam C.H.2 yıl önce

Yep, 8gb

Kevin Loi profil fotoğrafı
Kevin Loi2 yıl önce

You're cooking!

deepak 🤖 profil fotoğrafı
deepak 🤖2 yıl önce

super impressive, are you using an external fan for your Pi?

Adam C.H. profil fotoğrafı
Adam C.H.2 yıl önce

not yet, but I should haha

Benzer Videolar

MM8108 HaLow is now running on Raspberry Pi 5 with OpenWRT
9:05

MM8108 HaLow is now running on Raspberry Pi 5 with OpenWRT

Data Slayer

38,214 görüntüleme • 1 ay önce

What can you do with Llama quality and Groq speed? Instant. That's what. 3 months back: Llama 8B running at 750 Tokens/sec Now: Llama 70B model running at 3,200 Tokens/sec We're still going to get a liiiiiiitle bit faster, but this is our V1 14nm LPU - how fast will V2 be? 😉
0:39

What can you do with Llama quality and Groq speed? Instant. That's what. 3 months back: Llama 8B running at 750 Tokens/sec Now: Llama 70B model running at 3,200 Tokens/sec We're still going to get a liiiiiiitle bit faster, but this is our V1 14nm LPU - how fast will V2 be? 😉

Jonathan Ross

179,815 görüntüleme • 1 yıl önce

NEW Raspberry Pi AI Hat Is a Massive Upgrade Raspberry Pi
2:08

NEW Raspberry Pi AI Hat Is a Massive Upgrade Raspberry Pi

NetworkChuck

225,055 görüntüleme • 6 ay önce

And here we go - running Llama 3.2 3B on Raspberry Pi 5 with AI HAT+ 2. This thing is not going to break any speed records, but it’s incredible that we can now get such a powerful model on such a constrained system. Just a little over 3 years since ChatGPT launched.
0:40

And here we go - running Llama 3.2 3B on Raspberry Pi 5 with AI HAT+ 2. This thing is not going to break any speed records, but it’s incredible that we can now get such a powerful model on such a constrained system. Just a little over 3 years since ChatGPT launched.

Bojan Tunguz

27,108 görüntüleme • 6 ay önce

Can You ACTUALLY Run AI on a Raspberry Pi? Raspberry Pi
0:28

Can You ACTUALLY Run AI on a Raspberry Pi? Raspberry Pi

NetworkChuck

11,444 görüntüleme • 6 ay önce

The Raspberry Pi 5 Compute Module 5 (CM5) is here! 🧵
15:56

The Raspberry Pi 5 Compute Module 5 (CM5) is here! 🧵

Jeff Geerling

47,279 görüntüleme • 1 yıl önce

Raspberry Pi power meets industrial smarts with SECO Pi Vision 10.1 CM5. Pi Vision 10.1 CM5 is a rugged, ready-to-roll platform built around Raspberry Pi Compute Module 5. It’s tailor-made for IoT developers and system integrators. This isn’t just any old human–machine interface. It’s a Raspberry Pi gone pro.
1:27

Raspberry Pi power meets industrial smarts with SECO Pi Vision 10.1 CM5. Pi Vision 10.1 CM5 is a rugged, ready-to-roll platform built around Raspberry Pi Compute Module 5. It’s tailor-made for IoT developers and system integrators. This isn’t just any old human–machine interface. It’s a Raspberry Pi gone pro.

Raspberry Pi

16,610 görüntüleme • 8 ay önce

DeepSeek R1 671B is ridiculously good, but running at 158 tokens per second is something out of this world. A model *this good* running *this fast* is 3 years ahead of this time. The secret: DeepSeek in this short video isn't running on GPUs. It's running on custom-made chips! This is running on SambaNova and their SN40L chips, which are designed and optimized for running AI workflows. SambaNova is sponsoring my work. We designed GPUs to run games and later realized we could use them for AI. This is very different from creating a chip specifically for AI from the ground up. Just think about this for a second: One single SNL40 chip can simultaneously hold 100+ models (trillions of parameters) in memory! An agentic workflow that uses multiple models simultaneously won't need to swap models from memory even once! By the way, the Llama family of models *screams* on the SambaNova Cloud. Here are some numbers from my latest tests: • Llama 3.3 70B: 367.32 t/s • Llama 3.2 1B: 2381.82 t/s • Llama 3.2 3B: 1335.71 t/s • Llama 3.1 8B: 929.58 t/s • Llama 3.1 70B: 369.16 t/s • Llama 3.1 405B: 103.37 t/s You can access all of these models for free!
1:34

DeepSeek R1 671B is ridiculously good, but running at 158 tokens per second is something out of this world. A model *this good* running *this fast* is 3 years ahead of this time. The secret: DeepSeek in this short video isn't running on GPUs. It's running on custom-made chips! This is running on SambaNova and their SN40L chips, which are designed and optimized for running AI workflows. SambaNova is sponsoring my work. We designed GPUs to run games and later realized we could use them for AI. This is very different from creating a chip specifically for AI from the ground up. Just think about this for a second: One single SNL40 chip can simultaneously hold 100+ models (trillions of parameters) in memory! An agentic workflow that uses multiple models simultaneously won't need to swap models from memory even once! By the way, the Llama family of models *screams* on the SambaNova Cloud. Here are some numbers from my latest tests: • Llama 3.3 70B: 367.32 t/s • Llama 3.2 1B: 2381.82 t/s • Llama 3.2 3B: 1335.71 t/s • Llama 3.1 8B: 929.58 t/s • Llama 3.1 70B: 369.16 t/s • Llama 3.1 405B: 103.37 t/s You can access all of these models for free!

Santiago

131,019 görüntüleme • 1 yıl önce

OpenAI's nightmare: DeepSeek R1 on a Raspberry Pi
4:17

OpenAI's nightmare: DeepSeek R1 on a Raspberry Pi

Jeff Geerling

23,725 görüntüleme • 1 yıl önce

A brand new edition of The Official Raspberry Pi Beginner's Guide is OUT NOW! Updated for Raspberry Pi 500+ and OS Trixie:
0:31

A brand new edition of The Official Raspberry Pi Beginner's Guide is OUT NOW! Updated for Raspberry Pi 500+ and OS Trixie:

Raspberry Pi

18,003 görüntüleme • 5 ay önce

How a Raspberry Pi is made. Full video here:
0:57

How a Raspberry Pi is made. Full video here:

Jeff Geerling

1,163,918 görüntüleme • 3 yıl önce

Llama3.1 on a raspberry pi Thanks for Llamafile 😍
0:36

Llama3.1 on a raspberry pi Thanks for Llamafile 😍

Mike Bird

126,760 görüntüleme • 2 yıl önce

Duupr Clock ⏰ built with Flutter Firebase running on a Raspberry Pi loaded w/ PrimeOS 👍Animations in Rive 💙 #duupr #clock #raspberrypi
0:59

Duupr Clock ⏰ built with Flutter Firebase running on a Raspberry Pi loaded w/ PrimeOS 👍Animations in Rive 💙 #duupr #clock #raspberrypi

Roman Jaquez

32,734 görüntüleme • 1 yıl önce

Running Qwen3 8B thinking on an iPhone Air with MLX. The model is quantized to 4-bit and runs pretty well.
0:45

Running Qwen3 8B thinking on an iPhone Air with MLX. The model is quantized to 4-bit and runs pretty well.

Awni Hannun

215,589 görüntüleme • 10 ay önce

🚨 Meet Raspberry Pi Pico 2, our new $5 microcontroller board 🚨 The next big tiny thing, built using RP2350: a high-performance, secure microcontroller designed here at Raspberry Pi.
0:33

🚨 Meet Raspberry Pi Pico 2, our new $5 microcontroller board 🚨 The next big tiny thing, built using RP2350: a high-performance, secure microcontroller designed here at Raspberry Pi.

Raspberry Pi

876,158 görüntüleme • 1 yıl önce

I can't believe this works, but I got DeepSeek-V4-Flash (284B params) running on a Raspberry Pi 5 (8GB edition) at >1tok/s @ ~8W during full-tilt inference! It uses an untouched copy of antirez's GGUF. Took 160+ experiments over 5 days between GPT-5.5 xhigh and Opus 4.8 max.
0:32

I can't believe this works, but I got DeepSeek-V4-Flash (284B params) running on a Raspberry Pi 5 (8GB edition) at >1tok/s @ ~8W during full-tilt inference! It uses an untouched copy of antirez's GGUF. Took 160+ experiments over 5 days between GPT-5.5 xhigh and Opus 4.8 max.

Dan Woods

184,322 görüntüleme • 1 ay önce

GPS-free visual-inertial navigation in real time on Raspberry Pi 5 with a consumer-grade IMU and camera (and this is not even the slowest CPU this runs on!)
0:59

GPS-free visual-inertial navigation in real time on Raspberry Pi 5 with a consumer-grade IMU and camera (and this is not even the slowest CPU this runs on!)

Otto Seiskari

264,613 görüntüleme • 1 yıl önce

Multimodal Ichigo Llama 3.1 - Real Time Voice AI 🔥 > WhisperSpeech X Llama 3.1 8B > Trained on 50K hours of speech (7 languages) > Continually trained on 45hrs 10x A1000s > MLS -> WhisperVQ tokens -> Llama 3.1 > Instruction tuned on 1.89M samples > 70% speech, 20% transcription, 10% text > Apache 2.0 licensed ⚡ Architecture: > WhisperSpeech/ VQ for Semantic Tokens > Llama 3.1 8B Instruct for Text backbone > Early fusion (Chameleon) I'm super bullish on Homebrew to Menlo and early fusion, audio and text, multimodal models! (P.S. Play with the demo on Hugging Face)
0:32

Multimodal Ichigo Llama 3.1 - Real Time Voice AI 🔥 > WhisperSpeech X Llama 3.1 8B > Trained on 50K hours of speech (7 languages) > Continually trained on 45hrs 10x A1000s > MLS -> WhisperVQ tokens -> Llama 3.1 > Instruction tuned on 1.89M samples > 70% speech, 20% transcription, 10% text > Apache 2.0 licensed ⚡ Architecture: > WhisperSpeech/ VQ for Semantic Tokens > Llama 3.1 8B Instruct for Text backbone > Early fusion (Chameleon) I'm super bullish on Homebrew to Menlo and early fusion, audio and text, multimodal models! (P.S. Play with the demo on Hugging Face)

Vaibhav (VB) Srivastav

82,126 görüntüleme • 1 yıl önce

Testing #RePlayOS Alpha in Raspberry Pi 5. Interlaced video, 24bit audio quality, on-the-fly instant resolution changes, and more… Powered by RGB-Pi 💪
2:13

Testing #RePlayOS Alpha in Raspberry Pi 5. Interlaced video, 24bit audio quality, on-the-fly instant resolution changes, and more… Powered by RGB-Pi 💪

RTA

10,175 görüntüleme • 2 yıl önce