Hikari∣LocalLLM⚡'s banner

Hikari∣LocalLLM⚡

@Hikari_07_jp • 19,211 subscribers

what happens if you actually own the silicon? For work inquiries, please send a direct message. https://t.co/TxUDcW6DEb

Shorts

GLM 5.3を推論してる時のノイズ撮ってみた！結構うるさいし、熱もすごい。 4x RTX PRO 6000 Max-Q GPU合計1200W

GLM 5.3を推論してる時のノイズ撮ってみた！結構うるさいし、熱もすごい。 4x RTX PRO 6000 Max-Q GPU合計1200W

46,015 Aufrufe

My dual RTX PRO 6000 setup is currently training a Draft model for Qwen 3.6 27B! 🔥 I'm taking the paper DeepSeek dropped on 6/26 and going for a super ambitious application to the 27B scale. Thanks to my homelab, I was able to dive straight in — I read the paper and immediately started experimenting. The amount I've learned has been insane: - How memory bandwidth bottlenecks speed and clever ways to hack around it - Methods to train the draft model and boost its accuracy - Mechanisms to reference tokens all the way back to the previous one to skyrocket draft acceptance rates - The impact of Attention vs. GateDeltaNet on speculative decoding performance and how to handle those differences - The unique approaches and trade-offs of MTP, Dflash, JetSpec, and DSpark I could go on forever, but just from speculative decoding alone I've learned so much. The 27B architecture feels way more DSpark-native than JetSpec, so once draft training finishes, I'm going all-in with DSpark! My goal is to beat existing speculative decoding speeds outright — no task-specific shortcuts or cheating, pure general improvement. If you're into this kind of research, I'd love to hear your thoughts, impressions, and any suggestions — please reply! 🚀

My dual RTX PRO 6000 setup is currently training a Draft model for Qwen 3.6 27B! 🔥 I'm taking the paper DeepSeek dropped on 6/26 and going for a super ambitious application to the 27B scale. Thanks to my homelab, I was able to dive straight in — I read the paper and immediately started experimenting. The amount I've learned has been insane: - How memory bandwidth bottlenecks speed and clever ways to hack around it - Methods to train the draft model and boost its accuracy - Mechanisms to reference tokens all the way back to the previous one to skyrocket draft acceptance rates - The impact of Attention vs. GateDeltaNet on speculative decoding performance and how to handle those differences - The unique approaches and trade-offs of MTP, Dflash, JetSpec, and DSpark I could go on forever, but just from speculative decoding alone I've learned so much. The 27B architecture feels way more DSpark-native than JetSpec, so once draft training finishes, I'm going all-in with DSpark! My goal is to beat existing speculative decoding speeds outright — no task-specific shortcuts or cheating, pure general improvement. If you're into this kind of research, I'd love to hear your thoughts, impressions, and any suggestions — please reply! 🚀

57,218 Aufrufe

Achieving 3,000 tokens/s with 16-parallel inference of Qwen3.6 35B-A3B NVFP4 on a single RTX PRO 6000 WorkStation Edition. I am using ParallelHue—a tool I am developing—to color-code the tokens generated simultaneously via speculative decoding.

Achieving 3,000 tokens/s with 16-parallel inference of Qwen3.6 35B-A3B NVFP4 on a single RTX PRO 6000 WorkStation Edition. I am using ParallelHue—a tool I am developing—to color-code the tokens generated simultaneously via speculative decoding.

27,607 Aufrufe

I'm looking forward to the release of 2-bit quantization for GLM-5.2. I'll try to see how much throughput I can squeeze out of my machine.

I'm looking forward to the release of 2-bit quantization for GLM-5.2. I'll try to see how much throughput I can squeeze out of my machine.

18,709 Aufrufe

Videos

Anya Rossi

sweetdream.ai

SweetDream.ai•Sponsored•Livecam

Watch Anya Live

Anya is streaming live right now! Join her private show and enjoy exclusive content.

Exclusive private shows

1.2k viewers online

Private Show

Join now for exclusive access

Free preview available • Premium content

MiniCPM-V 4.7 35B-A3B, running on an iPhone. A 35B MoE vision model with ~3B active, fully on-device with just 3 GB of memory. One photo of a weathered window, one question — and it actually read the photo 🧵 GitHub: HF:

MiniCPM-V 4.7 35B-A3B, running on an iPhone. A 35B MoE vision model with ~3B active, fully on-device with just 3 GB of memory. One photo of a weathered window, one question — and it actually read the photo 🧵 GitHub: HF:

Hikari∣LocalLLM⚡

262,795 Aufrufe • vor 24 Tagen

RTX PRO 6000 Max-Q 開封動画 #NVIDIA NVIDIA AI

RTX PRO 6000 Max-Q 開封動画 #NVIDIA NVIDIA AI

Hikari∣LocalLLM⚡

59,442 Aufrufe • vor 1 Monat

Finally!! Achieved 385 tok/s with GLM 5.2 NVFP4 on 8x RTX PRO 6000. It was buggy and incredibly slow at first. It took me over a week using agents to get this far.

Finally!! Achieved 385 tok/s with GLM 5.2 NVFP4 on 8x RTX PRO 6000. It was buggy and incredibly slow at first. It took me over a week using agents to get this far.

Hikari∣LocalLLM⚡

104,551 Aufrufe • vor 3 Monaten

リアルタイム翻訳付きXライブのために、完全ローカルなシステムを作りました！日本語で話しながら他言語の字幕を出してグローバルなローカルLLMについてのライブ配信をやっていきます🙌 応援してくれる方は拡散していただけると助かります。ローカルLLM盛り上げて行くぞー！

リアルタイム翻訳付きXライブのために、完全ローカルなシステムを作りました！日本語で話しながら他言語の字幕を出してグローバルなローカルLLMについてのライブ配信をやっていきます🙌 応援してくれる方は拡散していただけると助かります。ローカルLLM盛り上げて行くぞー！

Hikari∣LocalLLM⚡

20,688 Aufrufe • vor 29 Tagen

Keine weiteren Inhalte verfügbar