Загрузка видео...

Не удалось загрузить видео

На главную

we sped up distributed inference by up to 5x with decentralized speculative decoding. many don't realize that AI models normally generate text one single word at a time, waiting for the network after every word. speculative decoding changes this by using a "guess & confirm" system, similar to autocomplete....

45,584 просмотров • 8 месяцев назад •via X (Twitter)

Комментарии: 0

Нет доступных комментариев

Здесь появятся комментарии из оригинального поста

Похожие видео

I told you to claim your free 16GB NVIDIA GPU for learning Local LLMs. Now I’m going to show you how to double its inference speed without touching the hardware. Google Colab gives you an enterprise grade NVIDIA Tesla T4 GPU for free, roughly 4 hours every single day. It is the absolute perfect sandbox for learning AI engineering, testing inference flags, and pushing massive context windows. The local AI timeline is moving way too fast. If you aren't using Multi Token Prediction (MTP) yet, you are leaving massive performance on the table. I just pushed DeepMind’s Gemma 4 26B to 64.9 t/s on this exact free tier. Let's look at the raw benchmark data running on an Ubuntu Linux environment with the latest compiled llama.cpp binaries and quantized GGUFs from Unsloth via HuggingFace: # Qwen 3.5 9B (Dense): Base: [ Prompt: 626.7 t/s | Generation: 21.0 t/s ] With MTP: [ Prompt: 539.1 t/s | Generation: 24.8 t/s ] # Gemma 4 26B QAT (MoE): Base: [ Prompt: 634.2 t/s | Generation: 48.3 t/s ] With MTP: [ Prompt: 572.1 t/s | Generation: 64.9 t/s ] If you are paying attention, this single Colab notebook reveals 3 massive observations about the current state of local LLMs: # 1. The MTP Speedup (Software Overclocking) Standard autoregressive decoding guesses one token at a time. MTP acts like a highly optimized, built in speculative decoder. It predicts multiple future tokens at once and the main model verifies them in parallel. The result? Zero accuracy loss and a massive throughput increase. Gemma jumped from 48 to 65 t/s just by flipping a flag. # 2. The MoE Paradox (Bigger is Faster) How does a 26B parameter model absolutely destroy a 9B model in raw speed on the exact same hardware? Architecture. Qwen 3.5 9B is a dense model. it activates all 9 billion parameters for every single token. Gemma 4 26B is a Mixture of Experts (MoE) model. It routes data efficiently, activating only 4B parameters per token. You get the reasoning capabilities of a 26B model with the compute cost of a 4B model. 3. Thinking Efficiency When I ran the exact same complex prompt on both models, the larger MoE spent significantly fewer "thinking" tokens to arrive at the correct answer. A smarter model doesn't just give better answers; it gets to the point faster, saving you compute cycles and preserving your context window. # Want to run this yourself? Here are the exact llama.cpp CLI commands. For Qwen (MTP is baked into the main model): ./llama-cli -m Qwen3.5-9B-UD-Q4_K_XL.gguf -p "Explain quantum computing." -n 2000 -c 8000 -ngl 99 -fa on --spec-type draft-mtp --spec-draft-n-max 4 --spec-draft-p-min 0.7 For Gemma (Using a separate lightweight draft model): ./llama-cli -m gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf --model-draft mtp-gemma-4-26B-A4B-it.gguf -p "Explain quantum computing." -n 2000 -c 8000 -ngl 99 -fa on --spec-type draft-mtp --spec-draft-n-max 4 --spec-draft-p-min 0.7 Stop waiting for a $3,000 rig. Boot up Colab, pull these models, and start building your stack. I’ve put together a completely free, cell by cell Google Colab notebook that automates this entire workflow so you can test it yourself in 5 minutes and learn. Link to the notebook is in the comments below. Experiemt with different MTP parameters, context windows and post your results in the comments.

Alok

170,442 просмотров • 2 месяцев назад

HydraFusion Explained. Part I: How does the Copilot engine know what to optimize for? Your prompt is evaluated across 4 dimensions: ➡ Does it require deep reasoning? (aka. reasoning depth) ➡ Is it a sophisticated problem? (aka. code generation complexity) ➡ Is it untangling a complicated mess? (aka. debugging difficulty) ➡ Is it dominated by tool-use? (aka. tool orchestration needs) Based on this evaluation, a HyDRA score is assigned to determine the capability profile your task needs the most and to establish a quality bar. Part II: How does it choose a model? Note: It doesn' t pick one model to handle the entire job e2e, (that's Auto mode). Instead, it selects 1 of 3 execution workflows and assigns the best model at different stages based on the HyDRA score: 1️⃣ Single ⚙️ How it works: A single model completes the task from start to finish. ⚖️ Rationale: The task comfortably meets the quality bar with one model. Multi-model orchestration would add latency and cost with no meaningful quality gain. 2️⃣ Cascade ⚙️ How it works: A lightweight, cost-efficient model generates the solution. This draft is evaluated against a quality gate and if it falls short of the quality bar, the entire task escalates to a stronger, frontier model. ⚖️ Rationale: Only bring in the big guns when there is concrete evidence that a lightweight model won't meet the quality threshold. 3️⃣ Critique ⚙️ How it works: A lightweight model drafts the initial code and tool interactions. An independent, read-only frontier model reviews that draft and provides feedback. The original lightweight model then performs any targeted revision(s) before the final response is sent to the user. ⚖️ Rationale: Writing code (output tokens) is expensive while reviewing code (input tokens) is cheap. Instead of incurring the cost of a powerhouse writing hundreds of lines from scratch, a cost-efficient model writes the first draft, and the frontier model just reviews it and points out fixes. HydraFusion is available in experimental preview on the GitHub Copilot CLI: /experimental on, /model and select Hydrafusion (Research Preview)

Julia Muiruri

12,687 просмотров • 11 дней назад

Many $IREN investors are waiting for an announcement from the company, to remove any doubts around Sweetwater 2 being included in batch zero baseload. At OnlyFrans, we prefer to follow the development facts, and as you can see in the time-lapse from the past 3 weeks, the progress is undeniable. Matching the rendering, the top-view from the satellite shows that the bulk substation earthworks are showing the typical IREN proportions. Roughly 2/3 of the footprint for the 345kV network, and the 1/3 smaller footprint above for the 138kV network. Further to the left, near the top of the image, we can see that the first signs for the two 300MW Primary substations are taking shape. I expect civil works to start in Q4, and after the stormwater works at Sweetwater 1 are done, I expect the main focus of the work to be stormwater management, in parallel with building out the bulk, and primary substations. Needless to say, Sweetwater 2 is batch zero baseload. Sweetwater 2 is an early example of the market moving before the rulebook did. In March 2025, IREN committed roughly $31m to secure 600MW with AEP — ~$51.7k/MW. Nearly a year later, Texas settled on a $50k/MW financial-security benchmark (under CIAC) for large loads entering Batch Zero. This means $IREN and AEP (the interconnecting TDSP for SW2) were already operating with the kind of financial commitment Texas would later require to separate real projects from speculative load requests. In other words: IREN was positioning its projects as real/financially committed rather than speculative well before Batch Zero forced everyone to do it. Think about it, if anyone would be compliant to the batch zero baseload requirements, wouldn't it be the company that had completed CIAC (Contribution in Aid of Construction) a year before it became mandatory for everyone else? I see a site progressing as if it's approved, and a connection progress that was completed before it became common knowledge. And somehow there are people that doubt that IREN will add 600MW of power next year? Now if you are interested to find out what else there is in batch zero baseload from IREN, you know where to go. Long $IREN

Frans Bakker

102,932 просмотров • 13 дней назад

This week is already so hot. 🔥 Massive release from Decart : Lucy 2.0 a World Editing Model running at 1080p, 30FPS in realtime. This is truly exciting, the era of real-time generative reality is here. We are moving from watching AI video to living inside AI video. A breakthrough model capable of transforming the visual world in real-time. Moving beyond offline rendering, Lucy 2.0 delivers high-fidelity 1080p video generation with near-zero latency. Lucy 2.0 literally "redraws" the entire world pixel-by-pixel, while you are watching it. e.g. If you want to be an anime character, it doesn't just put a mask on you. It turns your skin into anime skin, your hair into anime hair, and the lighting in your room into anime lighting. Lucy 2.0 is also trained to stop the generated video from slowly falling apart over time, so the same stream can run much longer without faces and details drifting. So why is this a "Massive Deal"? Traditional AI video-generation model takes a prompt, you wait 10–20 minutes, and the computer "bakes" a video for you. You couldn't touch it or change it while it was happening. But Lucy 2.0 works like a mirror. It happens in real-time (30 frames per second). There is no waiting. You move your hand, the AI character moves its hand instantly. The craziest part isn't the visuals; it's the physics. Usually, AI hallucinations are glitchy—hands merge into faces, walls melt. Lucy 2.0 understands how the world works without being told. It knows that if you take off a helmet, there is hair underneath. It knows that if you splash water, droplets fly. It learned "physics" just by watching millions of videos. The physical behavior you see emerges from learned visual dynamics, not from engineered geometry or explicit physics engines. Their official technical report explicitly states that the model does not use traditional 3D engines, depth maps, or wireframes. It is a "pure diffusion model."

Rohan Paul

12,761 просмотров • 7 месяцев назад

You don't understand... Higgsfield MCP + Claude just automated AI film making. Every single step you used to grind through to make an AI movie, you can now do 10x faster. Drop the script into Claude Opus 4.8 and say: "Here's my script. Break it into a full shotlist. Shot number, scene, shot type, camera move and the action in each frame." Now the whole film is mapped, shot by shot. - Pull your assets. Ask Claude: "From this shotlist, list every character, every location and every prop across the whole film." That's your build list. The stuff you would need to generate and give as references in next steps. - Build the character sheets. Higgsfield MCP is connected, so Claude has hands now to do stuff directly. It generates the images itself. Have the full body, back view and close up in the character sheet. One per character. Each sheet becomes the locked reference for that face. Same move for locations, generate the empty plate for each one before anyone steps into it. - Generate the frames. Feed Claude the references plus the shot and have it write and fire the Seedance 2.0 prompt. "Using the lead's character sheet and the alley plate, generate shot 4 in Seedance 2.0. Low angle, slow push-in, rain." Claude builds the prompt, calls Seedance 2.0 and the frame lands back in chat. Use a Seedance 2.0 skill to teach Claude how to prompt it properly. Now, there are 3 ways to make the shots. Pick one per scene. - Pure prompting. Fastest one. You describe the action in words and let Seedance interpret it. For consistency across a sequence, feed it a frame from the previous shot so the look carries. - Storyboarding. You hand it a panel and it matches that composition exactly. Way more control over how the shot is framed. The tradeoff is that it can introduce more cuts than you actually want. - Path Control System This is the latest technique Seedance 2.0 technique. Generate a still base plate of the scene. Draw a red line across it to mark the exact path of the movement, then describe what's happening. Seedance follows that line for the action. Also ask Claude to remove the red line when animating. This is the one for anything where motion has to land precisely. The output reads like real live action. - Lastly, generate every clip you need, then cut them together. Get it to Capcut for editing and audio design. And that's it. The pipeline that used to need a full crew and a studio can now run from one Claude chat. 2026 is gonna be wild

Rez Karim

10,951 просмотров • 3 месяцев назад

Batch Normalization by hand ✍️ ~ 7 steps walkthrough below Batch normalization is common practice for improving training and achieving faster convergence. It sounds simple. But it is often misunderstood. 🤔 Does batch normalization involve trainable parameters, tunable hyper-parameters, or both? 🤔 Is batch normalization applied to inputs, features, weights, biases, or outputs? 🤔 How is batch normalization different from layer normalization? So I drew and calculated one entirely by hand. Goal: normalize a mini-batch of 4 examples to mean 0 and variance 1, then let the network scale it back. = 1. Given = A mini-batch of 4 training examples, each with 3 features. = 2. Linear layer = Let us multiply by the weights and add the biases. Batch norm sits after this, which answers the second question: what gets normalized is features, not inputs, weights or biases. = 3. ReLU = We apply the activation, and -2 becomes 0. Negative values are suppressed before any statistic is taken. = 4. Batch statistics = Let us compute the sum, mean, variance and standard deviation, one row at a time. A row is a feature and the four columns are the four examples, so every number here measures one feature against the rest of the batch. That is the "batch" in batch normalization, and it is exactly what layer normalization does not do. The statistics are rounded to whole numbers, which is what keeps the rest of the page doable in pen. = 5. Shift to mean 0 = We subtract the mean, in green. The four values in each feature now average to zero. = 6. Scale to variance 1 = Let us divide by the standard deviation, in orange. Each feature now has variance one, whatever scale it arrived at. = 7. Scale and shift = We multiply by a linear transformation and pass the result on. The diagonal and the last column are trainable, so having just forced every feature to mean 0 and variance 1, we hand the network the means to undo it. The outputs: Mean of each feature = [2, 1, 2] Std dev of each feature = [1, 1, 2] To the next layer = [2, -2, 2, 0], [-3, 3, 6, -3], [2, 0, 1, 2] The answers: 🤔 Both. The scale and shift are trainable, the statistics are not. Epsilon and the momentum on the running statistics are the hyper-parameters, and one mini-batch by hand needs neither. 🤔 Features, after the linear layer, not inputs, weights or biases. 🤔 Batch norm measures across the batch, one feature at a time. Layer norm measures across the features, one example at a time. 💾 Save this post!

Tom Yeh

20,848 просмотров • 2 месяцев назад