Video wird geladen...

Video konnte nicht geladen werden

Zur Startseite

NVIDIA just dropped Nemotron-3-Nano:4b — a tiny 2.8GB model. Guess whose hardware runs it the fastest? - RTX 4090: 226 tok/s - RTX 3090: 187 tok/s - Mac Studio M2 Ultra: 86 tok/s - Mac Mini M4: 25 tok/s Home court advantage is real. Also trying a new layout...

stevibe's profile picture

stevibe

27,064 subscribers

127,408 Aufrufe • vor 3 Monaten •via X (Twitter)

Nachrichten & PolitikWissenschaft & Technologie
LIVE

Anya Rossi• Live Now

Private livecam show

0 Kommentare

Keine Kommentare verfügbar

Kommentare vom Original-Post werden hier angezeigt

Ähnliche Videos

.AI at Meta's Llama 4 Scout ripping 42.27 tok / sec on M2 Ultra Mac Studio
0:11

.AI at Meta's Llama 4 Scout ripping 42.27 tok / sec on M2 Ultra Mac Studio

LM Studio

19,587 Aufrufe • vor 1 Jahr

Tested the new MacBook Pro M4 Pro vs. the Mac mini M1 in LM Studio, running Llama 3.2 3B 4-bit MLX. Results: M1 (8-Core) ------> 25 tok/sec M4 Pro (14-Core) -> 104 tok/sec 🤯 – 4x faster!
0:10

Tested the new MacBook Pro M4 Pro vs. the Mac mini M1 in LM Studio, running Llama 3.2 3B 4-bit MLX. Results: M1 (8-Core) ------> 25 tok/sec M4 Pro (14-Core) -> 104 tok/sec 🤯 – 4x faster!

01000010

111,446 Aufrufe • vor 1 Jahr

MiniMax M2.5 casually responding at ~50 tok/s with MLX (M3 Ultra). The model was released one hour ago 🥳
0:14

MiniMax M2.5 casually responding at ~50 tok/s with MLX (M3 Ultra). The model was released one hour ago 🥳

Pedro Cuenca

170,493 Aufrufe • vor 4 Monaten

M4 Mac Mini AI Cluster Uses EXO Labs with Thunderbolt 5 interconnect (80Gbps) to run LLMs distributed across 4 M4 Pro Mac Minis. The cluster is small (iPhone for reference). It’s running Nemotron 70B at 8 tok/sec and scales to Llama 405B (benchmarks soon).
0:19

M4 Mac Mini AI Cluster Uses EXO Labs with Thunderbolt 5 interconnect (80Gbps) to run LLMs distributed across 4 M4 Pro Mac Minis. The cluster is small (iPhone for reference). It’s running Nemotron 70B at 8 tok/sec and scales to Llama 405B (benchmarks soon).

Alex Cheema

3,516,399 Aufrufe • vor 1 Jahr

WhisperKit-0.7.0 is out! Single file inference is several times faster! The demo below is running distil-whisper large-v3 at 300 tok/s and transcribes 101 seconds of audio in 1 second on an M2 Ultra Mac Studio. Details 🧵 Code (MIT): Demo Audio Input: Demo App: (TestFlight update under review)
0:14

WhisperKit-0.7.0 is out! Single file inference is several times faster! The demo below is running distil-whisper large-v3 at 300 tok/s and transcribes 101 seconds of audio in 1 second on an M2 Ultra Mac Studio. Details 🧵 Code (MIT): Demo Audio Input: Demo App: (TestFlight update under review)

argmax

50,549 Aufrufe • vor 2 Jahren

10x speedup: OpenAI's privacy-filter on MLX vs CPU, side by side in your browser. 1,818 tok/s vs 179 tok/s. Real-time PII redaction across English, German, French. On-device. No cloud. Privacy-first. Which model should we port next?
0:15

10x speedup: OpenAI's privacy-filter on MLX vs CPU, side by side in your browser. 1,818 tok/s vs 179 tok/s. Real-time PII redaction across English, German, French. On-device. No cloud. Privacy-first. Which model should we port next?

Maziyar PANAHI

25,329 Aufrufe • vor 2 Monaten

Run Llama 3.1 1B offline at 60+ tok/s. Entirely on your phone. Siri can’t save you offline, but a local model just might.
0:14

Run Llama 3.1 1B offline at 60+ tok/s. Entirely on your phone. Siri can’t save you offline, but a local model just might.

Aaron Ng

154,576 Aufrufe • vor 1 Jahr

nvidia/Qwen3.6-35B-A3B-NVFP4 running in vLLM nightly on my Nvidia GB10 is actually insane 50 tok/s, 4 concurrent generations. total 200 tok/s. ideal for spawning subagents or working in parallel its tool calling behavior is very good as well. I will be giving it test drive on an openclaw instance, and keep you posted More details on NVIDIA forum:
0:12

nvidia/Qwen3.6-35B-A3B-NVFP4 running in vLLM nightly on my Nvidia GB10 is actually insane 50 tok/s, 4 concurrent generations. total 200 tok/s. ideal for spawning subagents or working in parallel its tool calling behavior is very good as well. I will be giving it test drive on an openclaw instance, and keep you posted More details on NVIDIA forum:

Onur Solmaz

27,584 Aufrufe • vor 22 Tagen

Making a bop folder on tik tok dm if interested (『S』 will be included)
0:15

Making a bop folder on tik tok dm if interested (『S』 will be included)

bopperless

37,158 Aufrufe • vor 6 Monaten

llama 3 70b beamed to my phone from my M1 Max ~7.6 tok/s with mlx. your own little gpt-4 at home
0:17

llama 3 70b beamed to my phone from my M1 Max ~7.6 tok/s with mlx. your own little gpt-4 at home

Aaron Ng

1,116,956 Aufrufe • vor 2 Jahren

Running DeepSeek R1 on my desk Uses EXO Labs with Thunderbolt 5 interconnect (80Gbps) to run the full (671B, 8-bit) DeepSeek R1 distributed across 2 M3 Ultra 512GB Mac Studios (1TB total Unified Memory). Runs at 11 tok/sec. Theoretical max is ~20 tok/sec.
0:14

Running DeepSeek R1 on my desk Uses EXO Labs with Thunderbolt 5 interconnect (80Gbps) to run the full (671B, 8-bit) DeepSeek R1 distributed across 2 M3 Ultra 512GB Mac Studios (1TB total Unified Memory). Runs at 11 tok/sec. Theoretical max is ~20 tok/sec.

Alex Cheema

992,246 Aufrufe • vor 1 Jahr

M4 Mac Mini AI cluster running Llama 3.3 70B Preliminary results show Gigabit ethernet switch (4 tok/sec) is slower than Thunderbolt-5 with EXO Labs Benchmarks including Thunderbolt-5 coming soonTM.
0:11

M4 Mac Mini AI cluster running Llama 3.3 70B Preliminary results show Gigabit ethernet switch (4 tok/sec) is slower than Thunderbolt-5 with EXO Labs Benchmarks including Thunderbolt-5 coming soonTM.

Alex Cheema

73,090 Aufrufe • vor 1 Jahr

EXO 1.0 at Apple NeurIPS booth. 4 x 512GB M3 Ultra Mac Studios clustered all-to-all with RDMA over Thunderbolt 5. Runs DeepSeek v3.2 (8-bit) at 25 tok/sec with tensor parallelism.
0:15

EXO 1.0 at Apple NeurIPS booth. 4 x 512GB M3 Ultra Mac Studios clustered all-to-all with RDMA over Thunderbolt 5. Runs DeepSeek v3.2 (8-bit) at 25 tok/sec with tensor parallelism.

EXO Labs

185,728 Aufrufe • vor 6 Monaten

Qwen3.6 26b is local AGI. Macbook pro m5 24gb ram. Backend: mlx-lm Tok/s: ~15
0:11

Qwen3.6 26b is local AGI. Macbook pro m5 24gb ram. Backend: mlx-lm Tok/s: ~15

Marco Franzon

69,138 Aufrufe • vor 21 Tagen

Gemma 4 12B QAT (dense) achieves 1000+ tokens/sec prefill on 8GB VRAM with 120k context Gemma 4 12B QAT (dense), TurboQuant (Without MTP), RTX 4060 8GB VRAM: Prefill: 1000+ tok/s (42% increase) Decode: 25+ tok/s (25% increase) Context: 120k (150% increase) prefill was 700 tok/sec and decode 20 tok/sec with only 48k context without turbo quant (older test with mtp link in the comments) llama.cpp TurboQuant flags: -m gemma-4-12B-it-qat-UD-Q4_K_XL.gguf -c 120000 --cache-type-k q8_0 --cache-type-v turbo3 -ngl 99 --port 8080 tested with a 27k prompt, 120k context loaded. -ngl 99 here isn't a typo, full 12B dense, every layer on GPU, on an 8GB card. that's the part worth sitting with. The model has vision, audio input, thinking/reasoning and fits your 8GB card. TurboQuant's KV cache savings are what free up the room to do that at 120k context. side by side with yesterday: 26B A4B MoE got 320+ tok/s prefill. this dense 12B is clearing 1000+ rig: RTX 4060 8GB · i7H · 16GB RAM same two flags as yesterday, different model size: --cache-type-k q8_0 --cache-type-v turbo3 thanks to TheTom/llama-cpp-turboquant, TurboQuant fork of llama.cpp by Tom Turney (Tom Turney) to make this work. unsloth's model quant huggingface and the llama.cpp fork github link in the comments Do you prefer a dense or a MoE for your 8GB card?
0:12

Gemma 4 12B QAT (dense) achieves 1000+ tokens/sec prefill on 8GB VRAM with 120k context Gemma 4 12B QAT (dense), TurboQuant (Without MTP), RTX 4060 8GB VRAM: Prefill: 1000+ tok/s (42% increase) Decode: 25+ tok/s (25% increase) Context: 120k (150% increase) prefill was 700 tok/sec and decode 20 tok/sec with only 48k context without turbo quant (older test with mtp link in the comments) llama.cpp TurboQuant flags: -m gemma-4-12B-it-qat-UD-Q4_K_XL.gguf -c 120000 --cache-type-k q8_0 --cache-type-v turbo3 -ngl 99 --port 8080 tested with a 27k prompt, 120k context loaded. -ngl 99 here isn't a typo, full 12B dense, every layer on GPU, on an 8GB card. that's the part worth sitting with. The model has vision, audio input, thinking/reasoning and fits your 8GB card. TurboQuant's KV cache savings are what free up the room to do that at 120k context. side by side with yesterday: 26B A4B MoE got 320+ tok/s prefill. this dense 12B is clearing 1000+ rig: RTX 4060 8GB · i7H · 16GB RAM same two flags as yesterday, different model size: --cache-type-k q8_0 --cache-type-v turbo3 thanks to TheTom/llama-cpp-turboquant, TurboQuant fork of llama.cpp by Tom Turney (Tom Turney) to make this work. unsloth's model quant huggingface and the llama.cpp fork github link in the comments Do you prefer a dense or a MoE for your 8GB card?

Alok

34,500 Aufrufe • vor 20 Tagen

SmallThinker 3B is one of the AI models I’d want backed up on my phone. Here’s it is reasoning at 12 tok/s on my phone. If you’re ever stuck offline, you’ll want a model that can think.
0:15

SmallThinker 3B is one of the AI models I’d want backed up on my phone. Here’s it is reasoning at 12 tok/s on my phone. If you’re ever stuck offline, you’ll want a model that can think.

Aaron Ng

35,858 Aufrufe • vor 1 Jahr

Running Kimi K2.5 on my desk. Runs at 24 tok/sec with 2 x 512GB M3 Ultra Mac Studios connected with Thunderbolt 5 (RDMA) using EXO Labs / MLX backend. Yes, it can run clawdbot.
0:18

Running Kimi K2.5 on my desk. Runs at 24 tok/sec with 2 x 512GB M3 Ultra Mac Studios connected with Thunderbolt 5 (RDMA) using EXO Labs / MLX backend. Yes, it can run clawdbot.

Alex Cheema

3,048,931 Aufrufe • vor 5 Monaten

New Dirk Tik Tok dropped today
0:14

New Dirk Tik Tok dropped today

Mavs Film Room 🐴🎥

130,235 Aufrufe • vor 10 Monaten

Qwen QwQ 32B fp16 on M4 Max and M2 Ultra powered by MLX! M2 Ultra - 10.2 toks/s M4 Max - 7.6 toks/s! "Create an amazing animation using p5js" o1-mini level local model! Note: use temp 0.7-0.75 for optimal results in coding. I did some tests and this
0:10

Qwen QwQ 32B fp16 on M4 Max and M2 Ultra powered by MLX! M2 Ultra - 10.2 toks/s M4 Max - 7.6 toks/s! "Create an amazing animation using p5js" o1-mini level local model! Note: use temp 0.7-0.75 for optimal results in coding. I did some tests and this

Ivan Fioravanti ᯅ

62,377 Aufrufe • vor 1 Jahr