Video yükleniyor...

Video Yüklenemedi

Bu video yüklenirken bir sorun oluştu. Bu geçici bir ağ sorunundan kaynaklanıyor olabilir veya video kullanılamıyor olabilir.

Ana Sayfaya Dön

The example below is using prompt-based speculative decoding. Specifically, ngram hashing is utilized to suggest drafts of up to 64 tokens. The hasher keeps track of ngrams in the observed contexts, so mostly effective for coding tasks. Here is another demo:

Georgi Gerganov

63,159 subscribers

29,890 görüntüleme • 6 ay önce •via X (Twitter)

Anya Rossi• Live Now

Private livecam show

0 Yorum

Yorum bulunmuyor

Orijinal gönderinin yorumları burada görünecek

Benzer Videolar

GPT-5.4 is here. Native computer-use capabilities. Up to 1M tokens of context in Codex and the API. Best-in-class agentic coding for complex tasks. Scalable tool search across larger ecosystems. More efficient reasoning for long, tool-heavy workflows.

GPT-5.4 is here. Native computer-use capabilities. Up to 1M tokens of context in Codex and the API. Best-in-class agentic coding for complex tasks. Scalable tool search across larger ecosystems. More efficient reasoning for long, tool-heavy workflows.

OpenAI Developers

1,076,232 görüntüleme • 7 ay önce

Sequential Monte Carlo speculative decoding from Makora keeps multiple draft tokens alive in parallel instead of rewinding failed matches.

Sequential Monte Carlo speculative decoding from Makora keeps multiple draft tokens alive in parallel instead of rewinding failed matches.

SemiAnalysis

24,418 görüntüleme • 4 ay önce

One of the BEST ways to perform RDLs or SLDLs is with the machine shown here This is called a GBSL Here is an example of EXCELLENT technique using it:

One of the BEST ways to perform RDLs or SLDLs is with the machine shown here This is called a GBSL Here is an example of EXCELLENT technique using it:

Dean Turner

22,171 görüntüleme • 3 gün önce

This inference engine runs LLMs 4x faster than llama.cpp: (Qwen3.5-9B at 92 tokens/s on 16GB Mac) I ran Qwen3.5 9B on a base M5 MacBook Pro using three local inference engines, one at a time. → llama.cpp reached 22.0 tokens/s → MLX reached 25.1 tokens/s → Uzu reached 92.1 tokens/s The video shows all three runs side by side. The first two results do not mean llama.cpp or MLX are poorly optimized. Both are approaching a physical limit of ordinary local decoding. Generating each new token normally requires another pass through the model. That pass streams roughly 5.2 GB of weights through memory. A base M5 provides about 153 GB/s of memory bandwidth. At one token per pass, the theoretical ceiling is roughly 153 ÷ 5.2, or 29 tokens/s, however fast the GPU computes. Uzu gets past that ceiling by optimizing speculative decoding for Apple Silicon. Speculative decoding uses a smaller drafting system to propose several future tokens. The full 9B model then checks those proposals together instead of generating every token through a separate pass. Uzu pushes this further in two ways. → It drafts up to 16 positions and uses Weaver to turn independent guesses into coherent sequences. → It verifies several possible sequences as a tree, then commits state only for the path the full model accepts. The 9B model remains in control of the final output. Uzu simply gives it more useful work to approve in each pass. In my run, every pass of the 9B model produced 7.5 output tokens on average. Ordinary decoding produces exactly 1. That is how Uzu reached 92.1 tokens/s on hardware whose one-token-per-pass ceiling is roughly 29. The rest of the engine is designed around this path too. Uzu ships its own quantized checkpoints, and on M5 chips its Metal kernels run verification through the GPU’s accelerated int8 matrix path. It is an open-source inference engine from Mirai, with bindings for Rust, Swift, Python, and TypeScript. The repository is here: (don't forget to star 🌟) I wrote the full breakdown of why local inference is memory-bound and how Uzu gets past that limit, with reproducible results and code. The article is quoted below.

This inference engine runs LLMs 4x faster than llama.cpp: (Qwen3.5-9B at 92 tokens/s on 16GB Mac) I ran Qwen3.5 9B on a base M5 MacBook Pro using three local inference engines, one at a time. → llama.cpp reached 22.0 tokens/s → MLX reached 25.1 tokens/s → Uzu reached 92.1 tokens/s The video shows all three runs side by side. The first two results do not mean llama.cpp or MLX are poorly optimized. Both are approaching a physical limit of ordinary local decoding. Generating each new token normally requires another pass through the model. That pass streams roughly 5.2 GB of weights through memory. A base M5 provides about 153 GB/s of memory bandwidth. At one token per pass, the theoretical ceiling is roughly 153 ÷ 5.2, or 29 tokens/s, however fast the GPU computes. Uzu gets past that ceiling by optimizing speculative decoding for Apple Silicon. Speculative decoding uses a smaller drafting system to propose several future tokens. The full 9B model then checks those proposals together instead of generating every token through a separate pass. Uzu pushes this further in two ways. → It drafts up to 16 positions and uses Weaver to turn independent guesses into coherent sequences. → It verifies several possible sequences as a tree, then commits state only for the path the full model accepts. The 9B model remains in control of the final output. Uzu simply gives it more useful work to approve in each pass. In my run, every pass of the 9B model produced 7.5 output tokens on average. Ordinary decoding produces exactly 1. That is how Uzu reached 92.1 tokens/s on hardware whose one-token-per-pass ceiling is roughly 29. The rest of the engine is designed around this path too. Uzu ships its own quantized checkpoints, and on M5 chips its Metal kernels run verification through the GPU’s accelerated int8 matrix path. It is an open-source inference engine from Mirai, with bindings for Rust, Swift, Python, and TypeScript. The repository is here: (don't forget to star 🌟) I wrote the full breakdown of why local inference is memory-bound and how Uzu gets past that limit, with reproducible results and code. The article is quoted below.

Akshay 🚀

81,359 görüntüleme • 16 gün önce

To the Pfizer lobby in India, here is the perfect example of #VishwaguruBharat.

To the Pfizer lobby in India, here is the perfect example of #VishwaguruBharat.

Anshul Saxena

370,101 görüntüleme • 3 yıl önce

In China, #Silver retail speculative sentiment is sub-zero. Social media personalities are making videos to trash silver. An example is the video below. 😭🤷👇

In China, #Silver retail speculative sentiment is sub-zero. Social media personalities are making videos to trash silver. An example is the video below. 😭🤷👇

Eric Yeung 👍🚀🌕

25,687 görüntüleme • 2 ay önce

Our first TPU megakernel for Kimi K3 reaches 709 tokens/s on low-concurrency decode, against 450 tokens/s for our GB200 baseline, both with DSpark speculative decoding. To our knowledge, this is the first TPU inference megakernel. The whole model runs in a single Pallas kernel, and without spec decoding it is roughly 1.4 to 2x the GB200 baseline at batch sizes 1 through 8. We are open sourcing it today. 1/2

Our first TPU megakernel for Kimi K3 reaches 709 tokens/s on low-concurrency decode, against 450 tokens/s for our GB200 baseline, both with DSpark speculative decoding. To our knowledge, this is the first TPU inference megakernel. The whole model runs in a single Pallas kernel, and without spec decoding it is roughly 1.4 to 2x the GB200 baseline at batch sizes 1 through 8. We are open sourcing it today. 1/2

Inferact

245,473 görüntüleme • 1 ay önce

The Tory Government's decision to reject all six of Leeds' levelling-up bids is another kick in the teeth for people in our area. Levelling-up has been yet another example of Tory favouritism and of the Tories looking after their own.

The Tory Government's decision to reject all six of Leeds' levelling-up bids is another kick in the teeth for people in our area. Levelling-up has been yet another example of Tory favouritism and of the Tories looking after their own.

Richard Burgon MP

56,921 görüntüleme • 3 yıl önce

this is the perfect example of using AI to hook attention

this is the perfect example of using AI to hook attention

ViralOps

40,076 görüntüleme • 28 gün önce

More experiments of this workflow: Generating quick 3d models using Lumas Genie 3D. A text prompt generates 4 models in 10 seconds. Then, using Lumas webviewer, I feed a screencapture of the model into Krea ai which details the model based on another prompt. Finally, Magnific AI is used to add the final finish. The Santa mechs are insanely refined😱 Check out the final "Renders" below. #3d #ai

More experiments of this workflow: Generating quick 3d models using Lumas Genie 3D. A text prompt generates 4 models in 10 seconds. Then, using Lumas webviewer, I feed a screencapture of the model into Krea ai which details the model based on another prompt. Finally, Magnific AI is used to add the final finish. The Santa mechs are insanely refined😱 Check out the final "Renders" below. #3d #ai

Martin Nebelong

88,148 görüntüleme • 2 yıl önce

The most underrated feature of ChatGPT is Scheduled Tasks In any chat window, you can ask ChatGPT to perform a scheduled search - the frequency & research prompt are fully customizable. This feature is a MUCH more refined version of setting up news alerts on Google News or any other RSS feed. I have a couple of scheduled tasks for sectors which I find most interesting (e.g. Defense, Semiconductors, Data Centers & SpaceTech in India). The prompt template I use is linked below ⤵️

The most underrated feature of ChatGPT is Scheduled Tasks In any chat window, you can ask ChatGPT to perform a scheduled search - the frequency & research prompt are fully customizable. This feature is a MUCH more refined version of setting up news alerts on Google News or any other RSS feed. I have a couple of scheduled tasks for sectors which I find most interesting (e.g. Defense, Semiconductors, Data Centers & SpaceTech in India). The prompt template I use is linked below ⤵️

Rahul Mathur

10,678 görüntüleme • 9 ay önce

Quick demo of how I use Pi to offload tasks to Codex CLI with the pi-interactive-shell extension. Codex is notoriously tricky to prompt well so Pi reads the Codex prompting guide to generate a tailored meta-prompt, then launches Codex CLI in an overlay. pi install npm:pi-interactive-shell

Quick demo of how I use Pi to offload tasks to Codex CLI with the pi-interactive-shell extension. Codex is notoriously tricky to prompt well so Pi reads the Codex prompting guide to generate a tailored meta-prompt, then launches Codex CLI in an overlay. pi install npm:pi-interactive-shell

Nico Bailon

36,088 görüntüleme • 8 ay önce

The Mother of Rigged NBA drafts With all this talk of Adam Silver rigging the draft to compensate the Mavs for their participation in the collusion that produced Luka to LeBron, here is the mother of all rigged drafts, the 1985 Knicks Patrick Ewing draft

The Mother of Rigged NBA drafts With all this talk of Adam Silver rigging the draft to compensate the Mavs for their participation in the collusion that produced Luka to LeBron, here is the mother of all rigged drafts, the 1985 Knicks Patrick Ewing draft

KAREN PAIGE

441,981 görüntüleme • 1 yıl önce

I think you need to understand how cool and good the Codex sub-agent feature is. For example, here is the perfect task for parallel agents: cleaning and refactoring components based on a skill. So I ask Codex to split the work to speed things up.

I think you need to understand how cool and good the Codex sub-agent feature is. For example, here is the perfect task for parallel agents: cleaning and refactoring components based on a skill. So I ask Codex to split the work to speed things up.

Thomas Ricouard

55,852 görüntüleme • 8 ay önce

Researchers found a way to make LLMs 8.5x faster! (without compromising accuracy) Speculative decoding is quite an effective way to address the single-token bottleneck in traditional LLM inference. A small "draft" model first generates the next several tokens, then the large model verifies all of them at once in a single forward pass. If a token at any position is wrong, you keep everything before it and restart from there. This never does worse than normal decoding. But current drafters in Speculative decoding still guess one token at a time. That makes the drafting step itself a bottleneck, capping real-world speedups at 2-3x. DFlash is a new technique that swaps the autoregressive drafter with a lightweight block diffusion model that guesses all tokens in one parallel shot. Drafting cost stays flat no matter how many tokens you speculate. On top of that, the drafter is conditioned on hidden features pulled from multiple layers of the target model and injected into every draft layer, so it makes significantly better guesses than a drafter working from scratch. In the side-by-side demo below, vanilla decoding runs at 48.5 tokens/sec. DFlash hits 415 tokens/sec on the same model, with zero quality loss. It's already integrated with vLLM, SGLang, and Transformers, with draft models on HuggingFace for several models like Qwen3, Qwen3.5, Llama 3.1, Kimi-K2.5, gpt-oss, and many more. I have shared the GitHub repo in the replies! KV caching is another must-know technique to boost LLM inference. I recently wrote an article about it. Read it below. 👉 Over to you: What use case are you working on that can benefit from this new technique?

Researchers found a way to make LLMs 8.5x faster! (without compromising accuracy) Speculative decoding is quite an effective way to address the single-token bottleneck in traditional LLM inference. A small "draft" model first generates the next several tokens, then the large model verifies all of them at once in a single forward pass. If a token at any position is wrong, you keep everything before it and restart from there. This never does worse than normal decoding. But current drafters in Speculative decoding still guess one token at a time. That makes the drafting step itself a bottleneck, capping real-world speedups at 2-3x. DFlash is a new technique that swaps the autoregressive drafter with a lightweight block diffusion model that guesses all tokens in one parallel shot. Drafting cost stays flat no matter how many tokens you speculate. On top of that, the drafter is conditioned on hidden features pulled from multiple layers of the target model and injected into every draft layer, so it makes significantly better guesses than a drafter working from scratch. In the side-by-side demo below, vanilla decoding runs at 48.5 tokens/sec. DFlash hits 415 tokens/sec on the same model, with zero quality loss. It's already integrated with vLLM, SGLang, and Transformers, with draft models on HuggingFace for several models like Qwen3, Qwen3.5, Llama 3.1, Kimi-K2.5, gpt-oss, and many more. I have shared the GitHub repo in the replies! KV caching is another must-know technique to boost LLM inference. I recently wrote an article about it. Read it below. 👉 Over to you: What use case are you working on that can benefit from this new technique?

Avi Chawla

158,352 görüntüleme • 5 ay önce

VP Harris: “There’s something about this whole Trump era that is really trying to suggest that the strength of a leader is based on who you beat down…The real strength of a leader is based on who you lift up.”

VP Harris: “There’s something about this whole Trump era that is really trying to suggest that the strength of a leader is based on who you beat down…The real strength of a leader is based on who you lift up.”

Republican Accountability

36,196 görüntüleme • 2 yıl önce

So the Trump Administration is allegedly using money for Ground based nuclear missiles to Retrofit Trump’s Qatar plane in the amount of $ 934 million dollars! This is why Trump wanted to keep the amount of retrofitting it quiet! 😡😡😡

So the Trump Administration is allegedly using money for Ground based nuclear missiles to Retrofit Trump’s Qatar plane in the amount of $ 934 million dollars! This is why Trump wanted to keep the amount of retrofitting it quiet! 😡😡😡

Suzie rizzio

45,448 görüntüleme • 1 yıl önce