Hikari∣LocalLLM⚡'s banner

Hikari∣LocalLLM⚡

@Hikari_07_jp • 5,063 subscribers

Own the silicon. Hack the stack. Model hacks • NVFP4/SM120 • speculative decode. Measure → break → ship → improve. Never done.

Shorts

Achieving 3,000 tokens/s with 16-parallel inference of Qwen3.6 35B-A3B NVFP4 on a single RTX PRO 6000 WorkStation Edition. I am using ParallelHue—a tool I am developing—to color-code the tokens generated simultaneously via speculative decoding.

Achieving 3,000 tokens/s with 16-parallel inference of Qwen3.6 35B-A3B NVFP4 on a single RTX PRO 6000 WorkStation Edition. I am using ParallelHue—a tool I am developing—to color-code the tokens generated simultaneously via speculative decoding.

25,418 просмотров

My dual RTX PRO 6000 setup is currently training a Draft model for Qwen 3.6 27B! 🔥 I'm taking the paper DeepSeek dropped on 6/26 and going for a super ambitious application to the 27B scale. Thanks to my homelab, I was able to dive straight in — I read the paper and immediately started experimenting. The amount I've learned has been insane: - How memory bandwidth bottlenecks speed and clever ways to hack around it - Methods to train the draft model and boost its accuracy - Mechanisms to reference tokens all the way back to the previous one to skyrocket draft acceptance rates - The impact of Attention vs. GateDeltaNet on speculative decoding performance and how to handle those differences - The unique approaches and trade-offs of MTP, Dflash, JetSpec, and DSpark I could go on forever, but just from speculative decoding alone I've learned so much. The 27B architecture feels way more DSpark-native than JetSpec, so once draft training finishes, I'm going all-in with DSpark! My goal is to beat existing speculative decoding speeds outright — no task-specific shortcuts or cheating, pure general improvement. If you're into this kind of research, I'd love to hear your thoughts, impressions, and any suggestions — please reply! 🚀

My dual RTX PRO 6000 setup is currently training a Draft model for Qwen 3.6 27B! 🔥 I'm taking the paper DeepSeek dropped on 6/26 and going for a super ambitious application to the 27B scale. Thanks to my homelab, I was able to dive straight in — I read the paper and immediately started experimenting. The amount I've learned has been insane: - How memory bandwidth bottlenecks speed and clever ways to hack around it - Methods to train the draft model and boost its accuracy - Mechanisms to reference tokens all the way back to the previous one to skyrocket draft acceptance rates - The impact of Attention vs. GateDeltaNet on speculative decoding performance and how to handle those differences - The unique approaches and trade-offs of MTP, Dflash, JetSpec, and DSpark I could go on forever, but just from speculative decoding alone I've learned so much. The 27B architecture feels way more DSpark-native than JetSpec, so once draft training finishes, I'm going all-in with DSpark! My goal is to beat existing speculative decoding speeds outright — no task-specific shortcuts or cheating, pure general improvement. If you're into this kind of research, I'd love to hear your thoughts, impressions, and any suggestions — please reply! 🚀

56,798 просмотров

I'm looking forward to the release of 2-bit quantization for GLM-5.2. I'll try to see how much throughput I can squeeze out of my machine.

I'm looking forward to the release of 2-bit quantization for GLM-5.2. I'll try to see how much throughput I can squeeze out of my machine.

18,709 просмотров

Videos

Anya Rossi

sweetdream.ai

SweetDream.ai•Sponsored•Livecam

Watch Anya Live

Anya is streaming live right now! Join her private show and enjoy exclusive content.

Exclusive private shows

1.2k viewers online

Private Show

Join now for exclusive access

Free preview available • Premium content

Finally!! Achieved 385 tok/s with GLM 5.2 NVFP4 on 8x RTX PRO 6000. It was buggy and incredibly slow at first. It took me over a week using agents to get this far.

Finally!! Achieved 385 tok/s with GLM 5.2 NVFP4 on 8x RTX PRO 6000. It was buggy and incredibly slow at first. It took me over a week using agents to get this far.

Hikari∣LocalLLM⚡

63,382 просмотров • 2 дней назад

Больше нет контента для загрузки