Video yükleniyor...

Video Yüklenemedi

Ana Sayfaya Dön

Demo for the sharded inference engine on: gpt-oss-120b on 4090's, max input context 95k, max output context infinite (tok/s degrades with context) queue of max 12 prompts concurrently, use it wisely

leyten's profile picture

leyten

6,857 subscribers

90,800 görüntüleme • 1 ay önce •via X (Twitter)

Bilim & Teknoloji
LIVE

Anya Rossi• Live Now

Private livecam show

0 Yorum

Yorum bulunmuyor

Orijinal gönderinin yorumları burada görünecek

Benzer Videolar

introducing simple-llm: a ~950 line, powerful & extensible inference engine that performs on par with vllm. enjoy :) performance (gpt-oss-120b, on an h100): - batch=1: 135 tok/s (vllm: 138) - batch=64: 4,041 tok/s (vllm: 3,846)
0:25

introducing simple-llm: a ~950 line, powerful & extensible inference engine that performs on par with vllm. enjoy :) performance (gpt-oss-120b, on an h100): - batch=1: 135 tok/s (vllm: 138) - batch=64: 4,041 tok/s (vllm: 3,846)

naklecha

59,730 görüntüleme • 6 ay önce

GLM-5.1-478B-NVFP4 Running on: - 4x RTX Pro 6000 - Sglang - 370,000 max tokens (1.75x full context) - p10 27.7 | p90 45.6 tok/s decode (gen) - 1340 tok/s prefill I could get 2x decode if I limit to 64k context (100 tok/s) In this video it operates Figma (:
1:00

GLM-5.1-478B-NVFP4 Running on: - 4x RTX Pro 6000 - Sglang - 370,000 max tokens (1.75x full context) - p10 27.7 | p90 45.6 tok/s decode (gen) - 1340 tok/s prefill I could get 2x decode if I limit to 64k context (100 tok/s) In this video it operates Figma (:

0xSero

74,772 görüntüleme • 3 ay önce

DFlash v0.1.4 : custom Metal verify kernels for quantized Qwen3 hybrid models, plus significant peak memory reduction at long context. M5 Max 40-core GPU, 64GB, stock mlx_lm baseline: Qwen3.6-35B-A3B-4bit: ► @ 1024 · 138.3 → 300.3 tok/s (2.20x) ► @ 2048 · 135.6 → 246.4 tok/s (1.81x) ► @ 4096 · 134.5 → 208.4 tok/s (1.56x) ► @ 8192 · 133.2 → 177.4 tok/s (1.33x) Qwen3.5-27B-4bit: ► @ 1024 · 33.5 → 79.0 tok/s (2.37x) ► @ 2048 · 33.1 → 70.2 tok/s (2.12x) ► @ 4096 · 31.5 → 55.7 tok/s (1.77x) ► @ 8192 · 33.9 → 45.3 tok/s (1.34x) Working on making this usable for agentic workloads goal is to never drop below baseline at any context depth. LLM decode is memory-bandwidth bound. M5 Max runs at 614 GB/s, that's 1.5x more than M1-M4 Max (400-410 GB/s). Results will vary on lower bandwidth chips.
0:32

DFlash v0.1.4 : custom Metal verify kernels for quantized Qwen3 hybrid models, plus significant peak memory reduction at long context. M5 Max 40-core GPU, 64GB, stock mlx_lm baseline: Qwen3.6-35B-A3B-4bit: ► @ 1024 · 138.3 → 300.3 tok/s (2.20x) ► @ 2048 · 135.6 → 246.4 tok/s (1.81x) ► @ 4096 · 134.5 → 208.4 tok/s (1.56x) ► @ 8192 · 133.2 → 177.4 tok/s (1.33x) Qwen3.5-27B-4bit: ► @ 1024 · 33.5 → 79.0 tok/s (2.37x) ► @ 2048 · 33.1 → 70.2 tok/s (2.12x) ► @ 4096 · 31.5 → 55.7 tok/s (1.77x) ► @ 8192 · 33.9 → 45.3 tok/s (1.34x) Working on making this usable for agentic workloads goal is to never drop below baseline at any context depth. LLM decode is memory-bandwidth bound. M5 Max runs at 614 GB/s, that's 1.5x more than M1-M4 Max (400-410 GB/s). Results will vary on lower bandwidth chips.

bstn 👁️

23,120 görüntüleme • 3 ay önce

Nemotron 3 Nano runs nicely with mlx-lm on an M4 Max. Could be a great model for local use on Mac: MoE + hybrid attention make it fast even for very long context. Generating in realtime with 4-bit model:
0:29

Nemotron 3 Nano runs nicely with mlx-lm on an M4 Max. Could be a great model for local use on Mac: MoE + hybrid attention make it fast even for very long context. Generating in realtime with 4-bit model:

Awni Hannun

51,070 görüntüleme • 7 ay önce

DFlash speculative decoding on Apple Silicon Qwen3.5-9B bf16 · M5 Max · greedy exact match ▸ 85 tok/s, 3.3× at 1024 tokens (runtime) ▸ ~70 tok/s, 2.6× in the video (terminal I/O overhead) ▸ 80 tok/s, 3.1× at 2048 tokens (runtime) Currently working on: → Long context (speedup degrades past 4K tokens, KV cache growth) → Int4 quantized models (27B class) Built on MLX, no CUDA, single machine. Draft generates 16 tokens in parallel, target verifies in one forward pass. Will open source when ready.
0:39

DFlash speculative decoding on Apple Silicon Qwen3.5-9B bf16 · M5 Max · greedy exact match ▸ 85 tok/s, 3.3× at 1024 tokens (runtime) ▸ ~70 tok/s, 2.6× in the video (terminal I/O overhead) ▸ 80 tok/s, 3.1× at 2048 tokens (runtime) Currently working on: → Long context (speedup degrades past 4K tokens, KV cache growth) → Int4 quantized models (27B class) Built on MLX, no CUDA, single machine. Draft generates 16 tokens in parallel, target verifies in one forward pass. Will open source when ready.

bstn 👁️

36,942 görüntüleme • 3 ay önce

AMD has created the worlds most powerful AI APU The AMD Ryzen™ AI Max+ PRO 395 runs Nvidia's custom Llama 3.1 70B at 5.94 tkns/s Context: The average person reads at 5.2 tkns/s The 4090 runs the same model at 2.7 tkns/s HP is releasing it as a powerful mini PC.
0:46

AMD has created the worlds most powerful AI APU The AMD Ryzen™ AI Max+ PRO 395 runs Nvidia's custom Llama 3.1 70B at 5.94 tkns/s Context: The average person reads at 5.2 tkns/s The 4090 runs the same model at 2.7 tkns/s HP is releasing it as a powerful mini PC.

AMDGPU

42,665 görüntüleme • 1 yıl önce

Qwen 3.6 Plus Preview just dropped on OpenRouter for Free. > 1M token context. > $0 input. $0 output. Meanwhile: - Claude Opus 4.6 → $5 / $25 per 1M - GPT-5.4 → charges for 1M context Qwen just nuked the pricing model. Open source isn’t slowing down. It’s taking over.
0:44

Qwen 3.6 Plus Preview just dropped on OpenRouter for Free. > 1M token context. > $0 input. $0 output. Meanwhile: - Claude Opus 4.6 → $5 / $25 per 1M - GPT-5.4 → charges for 1M context Qwen just nuked the pricing model. Open source isn’t slowing down. It’s taking over.

WorldofAI

146,277 görüntüleme • 3 ay önce

Everyone's comparing the DGX Spark to a 5090 and calling it slow. I think that's the wrong comparison. I ran Qwen3.6 35B-A3B FP8 with the full 262K context window enabled (~96GB RAM) — something gaming GPUs can't really do. Results: 🟢No context: 51.3 tok/s, TTFT 110ms 🟣200K prefill: 34.6 tok/s, TTFT 85s (~2,341 tok/s prefill) Prefill is way faster than a Mac. And 35 tok/s deep into 200K context, on a model this strong, is genuinely usable. The Spark plays a different game.
1:04

Everyone's comparing the DGX Spark to a 5090 and calling it slow. I think that's the wrong comparison. I ran Qwen3.6 35B-A3B FP8 with the full 262K context window enabled (~96GB RAM) — something gaming GPUs can't really do. Results: 🟢No context: 51.3 tok/s, TTFT 110ms 🟣200K prefill: 34.6 tok/s, TTFT 85s (~2,341 tok/s prefill) Prefill is way faster than a Mac. And 35 tok/s deep into 200K context, on a model this strong, is genuinely usable. The Spark plays a different game.

stevibe

33,301 görüntüleme • 2 ay önce

Nemotron-3-Ultra running on 4x 6000s edits my latest demo video.. - 75 tok/s decode - 8x concurrency - 256k context - 899 tok/s prefill - 20k tok/s prefill cache - NVFP4 Setting it up to be my Hermes driver. It's good enough at most things and doesn't talk like a moron.
1:26

Nemotron-3-Ultra running on 4x 6000s edits my latest demo video.. - 75 tok/s decode - 8x concurrency - 256k context - 899 tok/s prefill - 20k tok/s prefill cache - NVFP4 Setting it up to be my Hermes driver. It's good enough at most things and doesn't talk like a moron.

0xSero

15,674 görüntüleme • 1 ay önce

For a little bit of context as to how this top deck alone compares to Pad 1's deluge, the red line is Pad 1's max strength (pushing outward (not height)) from the middle of the launch mount. It is crazy how much power this plate has behind it. NSF - NASASpaceflight.com
0:48

For a little bit of context as to how this top deck alone compares to Pad 1's deluge, the red line is Pad 1's max strength (pushing outward (not height)) from the middle of the launch mount. It is crazy how much power this plate has behind it. NSF - NASASpaceflight.com

StarbaseTracking

20,038 görüntüleme • 5 ay önce

Mad Max on 14.2.1 is PATHETIC. Traffic on this road moves around 55 to 60. No issue with going 42 to 45 but stay out of the left lane! You can see a car pass me on the right. For context, 13.2.9 in Standard with max speed temporarily set to 60 would go 52 to 55. Perfect.
0:24

Mad Max on 14.2.1 is PATHETIC. Traffic on this road moves around 55 to 60. No issue with going 42 to 45 but stay out of the left lane! You can see a car pass me on the right. For context, 13.2.9 in Standard with max speed temporarily set to 60 would go 52 to 55. Perfect.

JS

36,812 görüntüleme • 7 ay önce

Qwen3.5 runs quite well in mlx-lm. Awesome that we have a frontier-level hybrid model. The context gets longer but the inference speed and memory use barely change. Here's the Q4 generating a space invaders game on an M3 Ultra. Generated 4,120 tokens at 37.6 tok/s.
0:28

Qwen3.5 runs quite well in mlx-lm. Awesome that we have a frontier-level hybrid model. The context gets longer but the inference speed and memory use barely change. Here's the Q4 generating a space invaders game on an M3 Ultra. Generated 4,120 tokens at 37.6 tok/s.

Awni Hannun

48,673 görüntüleme • 5 ay önce

llama.cpp with MTP support makes local models fast enough to use as daily drivers 🚀 Qwen3.6-27B dense generation (on A10G): From 25 tok/s → 45 tok/s (+78%). Two flags on llama-server: --spec-type draft-mtp --spec-draft-n-max 2
0:38

llama.cpp with MTP support makes local models fast enough to use as daily drivers 🚀 Qwen3.6-27B dense generation (on A10G): From 25 tok/s → 45 tok/s (+78%). Two flags on llama-server: --spec-type draft-mtp --spec-draft-n-max 2

Victor M

171,818 görüntüleme • 2 ay önce

context... Context... CONTEXT! Yeah, not the smoking gun the left thought it was... However, let this be a lesson for all of us... Stay on point. Don't give the left any ammo, as they will use anything that even resembles ammo against you.
1:17

context... Context... CONTEXT! Yeah, not the smoking gun the left thought it was... However, let this be a lesson for all of us... Stay on point. Don't give the left any ammo, as they will use anything that even resembles ammo against you.

Stephen Davis

68,513 görüntüleme • 5 ay önce

A voice agent powered by gpt-oss. Running locally on my macBook. Demo recorded in a Waymo with WiFi turned off. I'm still on my space game voice AI kick, obviously. Code link below. For conversational voice AI, you want to set the gpt-oss reasoning behavior to "low". (The default is "medium".) Notes on how to do that and a jinja template you can use are in the repo. The LLM in the demo video is the big, 120B version of gpt-oss. You can use the smaller, 20B model for this, of course. But OpenAI really did a cool thing here designing the 120B model to run in "just" 80GB of VRAM. And the llama.cpp mlx inference is fast: ~250ms TTFT. Running a big model on-device feels like a time warp into the future of AI.
4:35

A voice agent powered by gpt-oss. Running locally on my macBook. Demo recorded in a Waymo with WiFi turned off. I'm still on my space game voice AI kick, obviously. Code link below. For conversational voice AI, you want to set the gpt-oss reasoning behavior to "low". (The default is "medium".) Notes on how to do that and a jinja template you can use are in the repo. The LLM in the demo video is the big, 120B version of gpt-oss. You can use the smaller, 20B model for this, of course. But OpenAI really did a cool thing here designing the 120B model to run in "just" 80GB of VRAM. And the llama.cpp mlx inference is fast: ~250ms TTFT. Running a big model on-device feels like a time warp into the future of AI.

kwindla

202,147 görüntüleme • 11 ay önce

Apple Silicon + Gemma 4 fans: this is for you. Pico AI Server now supports continuous batching with MLX-Swift. 43 tok/s on 1 stream. 26 tok/s per stream on 2 concurrent streams. That’s 52 tok/s total. a 21% throughput gain on a six-year-old MacBook Pro M1 Max!
1:41

Apple Silicon + Gemma 4 fans: this is for you. Pico AI Server now supports continuous batching with MLX-Swift. 43 tok/s on 1 stream. 26 tok/s per stream on 2 concurrent streams. That’s 52 tok/s total. a 21% throughput gain on a six-year-old MacBook Pro M1 Max!

Ronald Mannak

50,180 görüntüleme • 3 ay önce

Minimax-M3 running on 4x RTX Pro 6000s - 800k context - 4x concurrency at 250k - 70-120 tok/s - 2000 tok/s prefill no cache - 376gb vram - mxfp4 It's working on improving the audio on one of my videos, it's actually doing a good job in researching solutions. Good model
0:33

Minimax-M3 running on 4x RTX Pro 6000s - 800k context - 4x concurrency at 250k - 70-120 tok/s - 2000 tok/s prefill no cache - 376gb vram - mxfp4 It's working on improving the audio on one of my videos, it's actually doing a good job in researching solutions. Good model

0xSero

24,634 görüntüleme • 1 ay önce

🗣️ Mikel Arteta on substituting on Max Dowman: “Probably not in my head, I had a gut feeling that it was a moment for him. Probably because he doesn’t seem to be fazed by the occasion, the moment, the context, the opponent.” ❌🧠
0:35

🗣️ Mikel Arteta on substituting on Max Dowman: “Probably not in my head, I had a gut feeling that it was a moment for him. Probably because he doesn’t seem to be fazed by the occasion, the moment, the context, the opponent.” ❌🧠

DailyAFC

41,811 görüntüleme • 4 ay önce

First video about MTPLX it works! 🔥 Gonna do my context benchmarks and some coding on it ASAP! In my very first test here: 🥇 M3 Ultra on the right: 33 --> 68 tps 🥈 M5 Max on the left: 27 --> 55 tps But M5 Max has 2 external monitors connected and is doing ton of stuff in parallel. Amazing job by Youssof Al Toukhi here!
2:12

First video about MTPLX it works! 🔥 Gonna do my context benchmarks and some coding on it ASAP! In my very first test here: 🥇 M3 Ultra on the right: 33 --> 68 tps 🥈 M5 Max on the left: 27 --> 55 tps But M5 Max has 2 external monitors connected and is doing ton of stuff in parallel. Amazing job by Youssof Al Toukhi here!

Ivan Fioravanti ᯅ

14,267 görüntüleme • 2 ay önce