Video wird geladen...

Video konnte nicht geladen werden

Beim Laden dieses Videos ist ein Problem aufgetreten. Dies könnte an einem vorübergehenden Netzwerkproblem liegen oder das Video ist möglicherweise nicht verfügbar.

Demo for the sharded inference engine on: gpt-oss-120b on 4090's, max input context 95k, max output context infinite (tok/s degrades with context) queue of max 12 prompts concurrently, use it wisely

leyten

6,552 subscribers

89,747 Aufrufe • vor 14 Tagen •via X (Twitter)

Wissenschaft & Technologie

Anya Rossi• Live Now

Private livecam show

0 Kommentare

Keine Kommentare verfügbar

Kommentare vom Original-Post werden hier angezeigt

Ähnliche Videos

introducing simple-llm: a ~950 line, powerful & extensible inference engine that performs on par with vllm. enjoy :) performance (gpt-oss-120b, on an h100): - batch=1: 135 tok/s (vllm: 138) - batch=64: 4,041 tok/s (vllm: 3,846)

introducing simple-llm: a ~950 line, powerful & extensible inference engine that performs on par with vllm. enjoy :) performance (gpt-oss-120b, on an h100): - batch=1: 135 tok/s (vllm: 138) - batch=64: 4,041 tok/s (vllm: 3,846)

naklecha

59,730 Aufrufe • vor 5 Monaten

GLM-5.1-478B-NVFP4 Running on: - 4x RTX Pro 6000 - Sglang - 370,000 max tokens (1.75x full context) - p10 27.7 | p90 45.6 tok/s decode (gen) - 1340 tok/s prefill I could get 2x decode if I limit to 64k context (100 tok/s) In this video it operates Figma (:

GLM-5.1-478B-NVFP4 Running on: - 4x RTX Pro 6000 - Sglang - 370,000 max tokens (1.75x full context) - p10 27.7 | p90 45.6 tok/s decode (gen) - 1340 tok/s prefill I could get 2x decode if I limit to 64k context (100 tok/s) In this video it operates Figma (:

0xSero

74,772 Aufrufe • vor 2 Monaten

DFlash v0.1.4 : custom Metal verify kernels for quantized Qwen3 hybrid models, plus significant peak memory reduction at long context. M5 Max 40-core GPU, 64GB, stock mlx_lm baseline: Qwen3.6-35B-A3B-4bit: ► @ 1024 · 138.3 → 300.3 tok/s (2.20x) ► @ 2048 · 135.6 → 246.4 tok/s (1.81x) ► @ 4096 · 134.5 → 208.4 tok/s (1.56x) ► @ 8192 · 133.2 → 177.4 tok/s (1.33x) Qwen3.5-27B-4bit: ► @ 1024 · 33.5 → 79.0 tok/s (2.37x) ► @ 2048 · 33.1 → 70.2 tok/s (2.12x) ► @ 4096 · 31.5 → 55.7 tok/s (1.77x) ► @ 8192 · 33.9 → 45.3 tok/s (1.34x) Working on making this usable for agentic workloads goal is to never drop below baseline at any context depth. LLM decode is memory-bandwidth bound. M5 Max runs at 614 GB/s, that's 1.5x more than M1-M4 Max (400-410 GB/s). Results will vary on lower bandwidth chips.

DFlash v0.1.4 : custom Metal verify kernels for quantized Qwen3 hybrid models, plus significant peak memory reduction at long context. M5 Max 40-core GPU, 64GB, stock mlx_lm baseline: Qwen3.6-35B-A3B-4bit: ► @ 1024 · 138.3 → 300.3 tok/s (2.20x) ► @ 2048 · 135.6 → 246.4 tok/s (1.81x) ► @ 4096 · 134.5 → 208.4 tok/s (1.56x) ► @ 8192 · 133.2 → 177.4 tok/s (1.33x) Qwen3.5-27B-4bit: ► @ 1024 · 33.5 → 79.0 tok/s (2.37x) ► @ 2048 · 33.1 → 70.2 tok/s (2.12x) ► @ 4096 · 31.5 → 55.7 tok/s (1.77x) ► @ 8192 · 33.9 → 45.3 tok/s (1.34x) Working on making this usable for agentic workloads goal is to never drop below baseline at any context depth. LLM decode is memory-bandwidth bound. M5 Max runs at 614 GB/s, that's 1.5x more than M1-M4 Max (400-410 GB/s). Results will vary on lower bandwidth chips.

bstn 👁️

23,120 Aufrufe • vor 2 Monaten

∞ Introducing World #1 Infinite Agent - infinite steps, infinite context, infinite output on cloud. See for yourself:

∞ Introducing World #1 Infinite Agent - infinite steps, infinite context, infinite output on cloud. See for yourself:

Flowith

446,995 Aufrufe • vor 1 Jahr

gpt-oss 120b running at 50 tokens/s locally on my macbook

gpt-oss 120b running at 50 tokens/s locally on my macbook

ADAM

81,339 Aufrufe • vor 11 Monaten

"To put it into context, he hasn't even done his GCSE's yet." 🤯 Jamie Redknapp on Max Dowman 🌟

"To put it into context, he hasn't even done his GCSE's yet." 🤯 Jamie Redknapp on Max Dowman 🌟

Sky Sports Premier League

227,385 Aufrufe • vor 3 Monaten

Nemotron 3 Nano runs nicely with mlx-lm on an M4 Max. Could be a great model for local use on Mac: MoE + hybrid attention make it fast even for very long context. Generating in realtime with 4-bit model:

Nemotron 3 Nano runs nicely with mlx-lm on an M4 Max. Could be a great model for local use on Mac: MoE + hybrid attention make it fast even for very long context. Generating in realtime with 4-bit model:

Awni Hannun

51,029 Aufrufe • vor 6 Monaten

DFlash speculative decoding on Apple Silicon Qwen3.5-9B bf16 · M5 Max · greedy exact match ▸ 85 tok/s, 3.3× at 1024 tokens (runtime) ▸ ~70 tok/s, 2.6× in the video (terminal I/O overhead) ▸ 80 tok/s, 3.1× at 2048 tokens (runtime) Currently working on: → Long context (speedup degrades past 4K tokens, KV cache growth) → Int4 quantized models (27B class) Built on MLX, no CUDA, single machine. Draft generates 16 tokens in parallel, target verifies in one forward pass. Will open source when ready.

DFlash speculative decoding on Apple Silicon Qwen3.5-9B bf16 · M5 Max · greedy exact match ▸ 85 tok/s, 3.3× at 1024 tokens (runtime) ▸ ~70 tok/s, 2.6× in the video (terminal I/O overhead) ▸ 80 tok/s, 3.1× at 2048 tokens (runtime) Currently working on: → Long context (speedup degrades past 4K tokens, KV cache growth) → Int4 quantized models (27B class) Built on MLX, no CUDA, single machine. Draft generates 16 tokens in parallel, target verifies in one forward pass. Will open source when ready.

bstn 👁️

36,942 Aufrufe • vor 2 Monaten

AMD has created the worlds most powerful AI APU The AMD Ryzen™ AI Max+ PRO 395 runs Nvidia's custom Llama 3.1 70B at 5.94 tkns/s Context: The average person reads at 5.2 tkns/s The 4090 runs the same model at 2.7 tkns/s HP is releasing it as a powerful mini PC.

AMD has created the worlds most powerful AI APU The AMD Ryzen™ AI Max+ PRO 395 runs Nvidia's custom Llama 3.1 70B at 5.94 tkns/s Context: The average person reads at 5.2 tkns/s The 4090 runs the same model at 2.7 tkns/s HP is releasing it as a powerful mini PC.

AMDGPU

42,665 Aufrufe • vor 1 Jahr

The new gpt-oss-120b from OpenAI is now the best general purpose model to run on a Framework Desktop! ~40 tok/s on the MXFP4 version in LM Studio on Fedora 42.

The new gpt-oss-120b from OpenAI is now the best general purpose model to run on a Framework Desktop! ~40 tok/s on the MXFP4 version in LM Studio on Fedora 42.

Framework

129,716 Aufrufe • vor 11 Monaten

Qwen 3.6 Plus Preview just dropped on OpenRouter for Free. > 1M token context. > $0 input. $0 output. Meanwhile: - Claude Opus 4.6 → $5 / $25 per 1M - GPT-5.4 → charges for 1M context Qwen just nuked the pricing model. Open source isn’t slowing down. It’s taking over.

Qwen 3.6 Plus Preview just dropped on OpenRouter for Free. > 1M token context. > $0 input. $0 output. Meanwhile: - Claude Opus 4.6 → $5 / $25 per 1M - GPT-5.4 → charges for 1M context Qwen just nuked the pricing model. Open source isn’t slowing down. It’s taking over.

WorldofAI

146,277 Aufrufe • vor 3 Monaten

Everyone's comparing the DGX Spark to a 5090 and calling it slow. I think that's the wrong comparison. I ran Qwen3.6 35B-A3B FP8 with the full 262K context window enabled (~96GB RAM) — something gaming GPUs can't really do. Results: 🟢No context: 51.3 tok/s, TTFT 110ms 🟣200K prefill: 34.6 tok/s, TTFT 85s (~2,341 tok/s prefill) Prefill is way faster than a Mac. And 35 tok/s deep into 200K context, on a model this strong, is genuinely usable. The Spark plays a different game.

Everyone's comparing the DGX Spark to a 5090 and calling it slow. I think that's the wrong comparison. I ran Qwen3.6 35B-A3B FP8 with the full 262K context window enabled (~96GB RAM) — something gaming GPUs can't really do. Results: 🟢No context: 51.3 tok/s, TTFT 110ms 🟣200K prefill: 34.6 tok/s, TTFT 85s (~2,341 tok/s prefill) Prefill is way faster than a Mac. And 35 tok/s deep into 200K context, on a model this strong, is genuinely usable. The Spark plays a different game.

stevibe

33,301 Aufrufe • vor 2 Monaten

Nemotron-3-Ultra running on 4x 6000s edits my latest demo video.. - 75 tok/s decode - 8x concurrency - 256k context - 899 tok/s prefill - 20k tok/s prefill cache - NVFP4 Setting it up to be my Hermes driver. It's good enough at most things and doesn't talk like a moron.

Nemotron-3-Ultra running on 4x 6000s edits my latest demo video.. - 75 tok/s decode - 8x concurrency - 256k context - 899 tok/s prefill - 20k tok/s prefill cache - NVFP4 Setting it up to be my Hermes driver. It's good enough at most things and doesn't talk like a moron.

0xSero

15,674 Aufrufe • vor 1 Monat

For a little bit of context as to how this top deck alone compares to Pad 1's deluge, the red line is Pad 1's max strength (pushing outward (not height)) from the middle of the launch mount. It is crazy how much power this plate has behind it. NSF - NASASpaceflight.com

For a little bit of context as to how this top deck alone compares to Pad 1's deluge, the red line is Pad 1's max strength (pushing outward (not height)) from the middle of the launch mount. It is crazy how much power this plate has behind it. NSF - NASASpaceflight.com

StarbaseTracking

20,038 Aufrufe • vor 4 Monaten

Mad Max on 14.2.1 is PATHETIC. Traffic on this road moves around 55 to 60. No issue with going 42 to 45 but stay out of the left lane! You can see a car pass me on the right. For context, 13.2.9 in Standard with max speed temporarily set to 60 would go 52 to 55. Perfect.

Mad Max on 14.2.1 is PATHETIC. Traffic on this road moves around 55 to 60. No issue with going 42 to 45 but stay out of the left lane! You can see a car pass me on the right. For context, 13.2.9 in Standard with max speed temporarily set to 60 would go 52 to 55. Perfect.

JS

36,812 Aufrufe • vor 6 Monaten

Pretty cool that with the new Qwen 2.5 models you can ask questions / generate using a reasonably sized code-base as context, all running on a laptop with mlx-lm. The 7B runs pretty fast on an M4 Max using the mlx-lm code base (~16k lines) as context:

Pretty cool that with the new Qwen 2.5 models you can ask questions / generate using a reasonably sized code-base as context, all running on a laptop with mlx-lm. The 7B runs pretty fast on an M4 Max using the mlx-lm code base (~16k lines) as context:

Awni Hannun

27,442 Aufrufe • vor 1 Jahr

Context that F1 didn't show you: Esteban overtook Max before Max overtook him, ruining both their laps. 📸: Formula 1 via f1mediafan #F1 #LasVegasGP

Context that F1 didn't show you: Esteban overtook Max before Max overtook him, ruining both their laps. 📸: Formula 1 via f1mediafan #F1 #LasVegasGP

MultiViewer

3,524,193 Aufrufe • vor 2 Jahren

Qwen3.5 runs quite well in mlx-lm. Awesome that we have a frontier-level hybrid model. The context gets longer but the inference speed and memory use barely change. Here's the Q4 generating a space invaders game on an M3 Ultra. Generated 4,120 tokens at 37.6 tok/s.

Qwen3.5 runs quite well in mlx-lm. Awesome that we have a frontier-level hybrid model. The context gets longer but the inference speed and memory use barely change. Here's the Q4 generating a space invaders game on an M3 Ultra. Generated 4,120 tokens at 37.6 tok/s.

Awni Hannun

48,673 Aufrufe • vor 4 Monaten