Video yükleniyor...
Video Yüklenemedi
🚨HUGE. Mukesh Ambani plays another masterstroke, joins hands with BSNL 🤯 ~ Jio users in rural & remote regions can now connect via BSNL’s NETWORK through new ICR recharge plans 👏🏼 Airtel & VI were already bleeding. With BSNL–Jio link-up, disruption just got next level
421,082 görüntüleme • 9 ay önce •via X (Twitter)
0 Yorum
Yorum bulunmuyor
Orijinal gönderinin yorumları burada görünecek
![If you are running local LLMs without N-gram speculative decoding, you are wasting massive amounts of compute. Whether your AI is editing a document, outputting structured JSON, or rewriting boilerplate templates, a huge chunk of the text it generates is highly repetitive or already exists right there in the prompt. Standard decoding wastes expensive GPU compute cycles "re thinking" every single token. By adding one hidden flag in llama.cpp, you can instantly fast forward through the repetition. Zero draft models. Zero extra VRAM. And virtually zero compute overhead. Google Colab hands you an enterprise grade NVIDIA Tesla T4 GPU with 16GB of VRAM for free. It’s the perfect Ubuntu Linux sandbox to build a bleeding edge inference engine from scratch. Recently, I showed you how to double your local speeds using MTP (Multi Token Prediction). But MTP requires a secondary neural network draft model. That eats into your precious VRAM (slightly though) and burns extra compute for every guess it makes. N-gram Speculative Decoding gives you a massive speed boost for exactly 0 memory cost and minimal compute. And it's faster than MTP when it works. Here is how it actually works under the hood: Standard autoregressive decoding is slow because it predicts one token at a time. If you ask an agent to format a long JSON object or update one line in an HTML file, it runs heavy matrix multiplications to calculate the probability of every single bracket, space, and letter from scratch. N-gram changes the game. It acts as a lightweight caching system. Instead of running heavy neural network math to guess the next word, it uses a simple hash table. Whenever the LLM starts outputting a sequence of tokens that already exists anywhere in its context window, N-gram instantly recognizes the pattern. Because it is just doing lightning fast string matching, the compute cost is practically zero. It "fast forwards" through the text, drafting the boilerplate instantly from memory, and the main model just verifies it in parallel. Pure speed. Using quantized GGUFs from Unsloth via HuggingFace, I spun up DeepMind’s massive Gemma 4 26B A4B QAT MoE on a free Colab instance to test this. Just look at the raw benchmark data on code editing task: Without N-gram: [ Prompt: 638.6 t/s | Generation: 45.9 t/s ] With N-gram: [ Prompt: 601.9 t/s | Generation: 107.1 t/s ] Here is the exact llama.cpp CLI command to activate it. Notice we don't even need the --model-draft flag: ./llama-cli -m gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf -cnv -n 6000 -c 12000 -ngl 99 -fa on --spec-type ngram-mod Stop waiting for your GPU to re calculate words it already knows. I’ve built a free, interactive, cell by cell Google Colab notebook that lets you test this live in your browser. You can literally chat with the model and watch the text generation speed absolutely fly on the second turn when you ask it to edit a file. There are additional parameters for ngram-mod that you can tune once you get it working with the single flag. Link to the free Colab Notebook is in the comments below. It walks you through the entire stack: pulling pre built llama.cpp CUDA binaries for Linux, fetching GGUFs from HuggingFace, and spinning up the inference engine with ngram-mod from scratch. Let me know if you have already tried ngram-mod](https://image.24vids.com/tw-2077476686678151422/amplify_video_thumb/2077475852384960512/img/mC5h-t6vqd9H1r2j.jpg)
![Today I'm proud to announce that I've officially become an Aptos Advisor! Let's go back in time... I had my initial comeup years back through Facebook advertising, chasing a dream that required me to leave everything behind in France where I was studying at uni but learning nothing that wasn't already obvious to me. Doing whatever it takes - I washed windows and worked the night shift at a plastic factory to fund my first business arbitraging consumer goods across the world from my grandparents attic when I was 18. I remember hitting a plateau. Some days in the green, some days in the red - learning a lot but it wasn't enough to leave the nest. I got in touch with 2 guys who would alter the trajectory and change my life forever so much its hard to quantify, even now. I sent a cold Instagram DM to one of them at 3 in the morning, expecting no reply. I poured my heart out, explaining my current situation, problems, and solutions i've tried to no avail. However I was met with him giving me his personal phone number, the one only his wife and kids had. I wasn't asking them to "put me on", but they did anyways. To an extent, they saw themselves in me, becoming mentors. My parents confused, at an instant I moved to Puerto Rico, where they ran their global operations from at a huge scale unlike anyone else at the time, and I became the youngest person under act 20/22 on the island. I later reverse image searched their tax attorney, showing up to the office in San Juan. I remember when I first entered the door, he looked around me to see if my parents were coming behind me, since I was so young. These mentors of mine would show me the ropes of how to really scale any product or SKU to 9 figures and beyond. Not just consumer goods, but software and subscription models - they had the Midas Touch with just an AMEX and a dream, going band for band with ANY Fortune500 Facebook marketing spend with much larger teams. It was unreal seeing their backend, filming ads in their home, and getting introduced to heavy hitters and some of the biggest web2 influencers like Logan Paul, which later would be convinced to become neighbors on the island, moving to Puerto Rico before the Floyd Mayweather Jr. fight expecting a huge capital gains event. Fast forward a few years. I got into crypto starting in 2017, but really 2023. I remember hearing stories of my mentors being super early into ETH, fullporting with size in 2015, and telling me during the advertising era to "never fade Facebook", which stuck with me all these years - leading me to Aptos, being the premier ex-libra/diem team with a clear focus on money movement. The first time I sent stablecoins on Aptos it was just so obvious, this product is just better by a mile than anything else, reminiscent of the early days "Pre-meta" of Facebook advertising which created an infinite money generator with the right creative. It only became obvious to others years later, becoming a crowded trade. Aptos is early but not without problems, if we're being honest. The engineering is truly next level and unmatched, which the market is fading for now, but will soon be unignorable with my help going forward. With my advisory position, I look to do a few things exceptionally well: - Combine both of my worlds together, introducing my Facebook mentors (which are now leading WLFI) to the leadership of Aptos, such as avery.apt 🇺🇸 strengthening the relationship by working on [REDACTED] in the coming months. - Guerilla marketing campaigns to gain net new buyers of APT effectively, running enormous retail traffic to Aptos that sustainably converts new users over time at scale. - Fully realize the Aptos CTO (Community Takeover), or activist investing for you oldheads out there. Imagine what happens when the largest Facebook advertisers join hands with the best Facebook developers. This partnership is only just beginning. Chase Herro ... Zak Folkman ... It's time to run it back... Aptos looking good here.](https://image.24vids.com/tw-1973381615637135405/amplify_video_thumb/1973381199759314944/img/FW4gQoEdxZhmxGQV.jpg)
