正在加载视频...

视频加载失败

加载此视频时出现问题。这可能是由于临时网络问题，或视频可能不可用。

Paving the way for unlimited context windows. Introducing ZML /attnd: sparse logarithmic attention, on CPU, over UDP, faster than GPU.

Steeve Morin

9,588 subscribers

122,236 次观看 • 1 年前 •via X (Twitter)

科学技术新闻政治教育

Anya Rossi• Live Now

Private livecam show

39 条评论

Charles Foster 的头像

Charles Foster1 年前

@zml_ai How do you avoid computing the attention values that end up ~0 without computing the dot products between each query-key pair? Are you using some sort of approximation?

Steeve Morin 的头像

Steeve Morin1 年前

@zml_ai we model it as a graph

Charles Foster 的头像

Charles Foster1 年前

@zml_ai How does modeling it as a graph let you bypass the N × N for sequence length of N, unless you're making some kind of approximation?

Steeve Morin 的头像

Steeve Morin1 年前

@zml_ai Yes there is approximation in the sense that we don't compute values very close to zero. From the talk:

Charles Foster 的头像

Charles Foster1 年前

@zml_ai How? How do you know which attention values *will be* very close to zero without first computing the dot-products?

Charles Foster 的头像

Charles Foster1 年前

@zml_ai Maybe what you mean to say is that you compute the full N x N attention matrix, but aggregate the value vectors sparsely by skipping entries where the attention is ~0? That would turn one dense-dense matmul into a sparse-dense one, but wouldn’t make the whole thing log(N), right?

Jean-Baptiste Schiratti 的头像

Jean-Baptiste Schiratti1 年前

@zml_ai Loved the talk & demo 🤩 @zml_ai rocks 🔥🚀❤️

James 的头像

James1 年前

@zml_ai @TheAhmadOsman looks like a great local llama unlock

Javier Honduvilla Coto 的头像

Javier Honduvilla Coto1 年前

@zml_ai Great stuff!! Btw love the T-shirt 🤣

David Roldan 🇵🇷 的头像

David Roldan 🇵🇷1 年前

@zml_ai Democratizing AI!!! 👌🏼

Abhi 的头像

Abhi1 年前

@zml_ai Good work man, holy shit

Guillaume Wenzek 的头像

Guillaume Wenzek1 年前

@zml_ai Very excited to share this. Attention is fascinating. I've been toying with this algorithm idea for some time but it took some faith and crunch to actually make it work.

murat 🍥 的头像

murat 🍥1 年前

@zml_ai any performance degradation?

Steeve Morin 的头像

Steeve Morin1 年前

@zml_ai no

Andrew Huynh 的头像

Andrew Huynh1 年前

@zml_ai How to run it in my m2 laptop??

Saïd Aitmbarek 的头像

Saïd Aitmbarek1 年前

@zml_ai hell yeah, based optimization for cpu guys! will help democratize AI adoption to the immense pool of non gpu socs let's launch/spotlight you on whenever it helps guys

VibeCodeTeddy 的头像

VibeCodeTeddy1 年前

@zml_ai This is wild. Can't wait to see the practical applications of this!

Ak afridi 的头像

Ak afridi1 年前

@zml_ai Fascinating approach to solving the context window limitation. CPU-based sparse attention over UDP opening new possibilities for distributed AI processing. Looking forward to seeing how this scales in production environments.

Gary Basin 的头像

Gary Basin1 年前

@zml_ai very cool

Mykhailo Sorochuk 的头像

Mykhailo Sorochuk1 年前

@zml_ai This breakthrough could really democratize AI. Less hardware, more access. Excited to see how it develops!

Gloria Baeuerlein 的头像

Gloria Baeuerlein1 年前

@zml_ai 🔥🔥🔥

Eugene Cheah - AI builder @ 🇸🇬|🇺🇸 的头像

Eugene Cheah - AI builder @ 🇸🇬|🇺🇸1 年前

@zml_ai cool stuff - however you need benchmarks on basic evals on running with CPU vs GPU, to really get this across the crowd/finish line

James T 的头像

James T1 年前

@zml_ai Outstanding will try this out on my old macbook pro 2017

AI PlanetX 的头像

AI PlanetX1 年前

@zml_ai Exciting innovation in attention technology!

Alon Burg 的头像

Alon Burg1 年前

@RemiCadene @zml_ai Amazing demo! Kudos!

parody. Dean the Magic 8 Ball geeker 的头像

parody. Dean the Magic 8 Ball geeker1 年前

@ylecun @zml_ai @grok eli5

Andrew Huynh 的头像

Andrew Huynh1 年前

@zml_ai Cool

Pierre-Baptiste Borges 的头像

Pierre-Baptiste Borges1 年前

@zml_ai congrats!

Reza Rawassizadeh 的头像

Reza Rawassizadeh1 年前

@zml_ai Where can we read about sparse logarithm attention?

jdubbs 的头像

jdubbs1 年前

@zml_ai Loved your CRdo writeup/thread......Do you think this eventually hurts the opical guys like CIEN and GLW? Thanks

Cyril Mottier 的头像

Cyril Mottier1 年前

@zml_ai 💥

VibeEdge 的头像

VibeEdge1 年前

@zml_ai @steeve Cool breakthrough on Attention D sparse log attention on CPU via UDP outpacing GPUs? Game changer for AI scaling. Echoes Megalodon style infinite contexts without heavy hardware. Excited for the code!

pybankers 的头像

pybankers1 年前

@zml_ai What model size will I be able to run with 16gb ram cpu without any gpu with good speed?

Anthara Fairooz 的头像

Anthara Fairooz1 年前

@zml_ai Exciting innovation in AI! Great step forward.

Conor 的头像

Conor1 年前

@zml_ai Super cool, congrats

Giovanni P. 的头像

Giovanni P.1 年前

@zml_ai Very cool, bold to use zig 👍 But bazel 🤣 please no!

Steeve Morin 的头像

Steeve Morin1 年前

@zml_ai

iG3 - Edge AI 的头像

iG3 - Edge AI1 年前

@zml_ai 👀

gawen 的头像

gawen1 年前

@zml_ai Boua very cool ! 🔥

相关视频

Introducing Ziren (formerly zkMIPS) The most performant general-purpose zkVM on CPU, now scaled with GPU, distributed proving, and more. Build with confidence. Deploy without compromise 🟠 Full context: Technical updates:

Introducing Ziren (formerly zkMIPS) The most performant general-purpose zkVM on CPU, now scaled with GPU, distributed proving, and more. Build with confidence. Deploy without compromise 🟠 Full context: Technical updates:

ZKM

5,857,968 次观看 • 1 年前

$Introducing SubQ - a major breakthrough in LLM intelligence. It is the first model built on a fully sub-quadratic sparse-attention architecture (SSA), And the first frontier model with a 12 million token context window which is: - 52x faster than FlashAttention at 1MM tokens - Less than 5% the cost of Opus Transformer-based LLMs waste compute by processing every possible relationship between words (standard attention). Only a small fraction actually matter. Subquadratic finds and focuses only on the ones that do. That's nearly 1,000x less compute and a new way for LLMs to scale.$

Introducing SubQ - a major breakthrough in LLM intelligence. It is the first model built on a fully sub-quadratic sparse-attention architecture (SSA), And the first frontier model with a 12 million token context window which is: - 52x faster than FlashAttention at 1MM tokens - Less than 5% the cost of Opus Transformer-based LLMs waste compute by processing every possible relationship between words (standard attention). Only a small fraction actually matter. Subquadratic finds and focuses only on the ones that do. That's nearly 1,000x less compute and a new way for LLMs to scale.

Alexander Whedon

13,492,280 次观看 • 5 个月前

$INFQ GPU + CPU + QPU Frontier models are hitting saturation on tokens (data volume) and context windows (memory). $INFQ CTO walks through how they and $NVDA are working together to add a Quantum Processing Unit to the chain as the next inflection point.

$INFQ GPU + CPU + QPU Frontier models are hitting saturation on tokens (data volume) and context windows (memory). $INFQ CTO walks through how they and $NVDA are working together to add a Quantum Processing Unit to the chain as the next inflection point.

YeahDave

57,181 次观看 • 5 个月前

Recent KV cache compression research has focused on small models with dense attention. We scaled these techniques to GLM 5.3, a frontier sparse attention model. We adapted KVzip and Attention Matching, reducing KV cache size by 80% while retaining over 90% of full-context accuracy on the QuALITY benchmark.

Recent KV cache compression research has focused on small models with dense attention. We scaled these techniques to GLM 5.3, a frontier sparse attention model. We adapted KVzip and Attention Matching, reducing KV cache size by 80% while retaining over 90% of full-context accuracy on the QuALITY benchmark.

Ramp Labs

20,677 次观看 • 15 天前

Windows ML is now generally available! This milestone brings powerful, local AI capability to every Windows 11 device, harnessing the full power of the GPU, NPU, and CPU. Big thank you to the team and our silicon partners, AMD, Intel, NVIDIA, and Qualcomm, for their collaboration to unlock real-time, secure, and efficient AI experiences for developers on Windows. Let’s go 🚀!

Windows ML is now generally available! This milestone brings powerful, local AI capability to every Windows 11 device, harnessing the full power of the GPU, NPU, and CPU. Big thank you to the team and our silicon partners, AMD, Intel, NVIDIA, and Qualcomm, for their collaboration to unlock real-time, secure, and efficient AI experiences for developers on Windows. Let’s go 🚀!

Pavan Davuluri

11,663 次观看 • 1 年前

PowerInfer - a high-speed inference engine for deploying LLMs locally. Just came across this super interesting project on speeding up inference. It's not MoE but it's a simple approach that exploits the high locality in LLM inference to design a GPU-CPU hybrid inference engine. Hot-activated neurons are preloaded onto the GPU for fast access, while cold-activated neurons (the majority) are computed on the CPU. This approach significantly reduces GPU memory demands and CPU-GPU data transfer. It achieves an average token generation rate of 13.20 tokens/s, with a peak of 29.08 tokens/s, across various LLMs on a single NVIDIA RTX 4090 GPU. It's on only 18% lower than that achieved by a top-tier server-grade A100 GPU. It also significantly outperforms llama.cpp by up to 11.69x while retaining model accuracy. There is a lot more innovation around inference that's coming fast. Really encouraged by the study on sparse computation to enhance the computational efficiency of LLMs. It's now possible to use PowerInfer with Llama 2 and Faclon 40B. Mistral-7B support is coming soon!

PowerInfer - a high-speed inference engine for deploying LLMs locally. Just came across this super interesting project on speeding up inference. It's not MoE but it's a simple approach that exploits the high locality in LLM inference to design a GPU-CPU hybrid inference engine. Hot-activated neurons are preloaded onto the GPU for fast access, while cold-activated neurons (the majority) are computed on the CPU. This approach significantly reduces GPU memory demands and CPU-GPU data transfer. It achieves an average token generation rate of 13.20 tokens/s, with a peak of 29.08 tokens/s, across various LLMs on a single NVIDIA RTX 4090 GPU. It's on only 18% lower than that achieved by a top-tier server-grade A100 GPU. It also significantly outperforms llama.cpp by up to 11.69x while retaining model accuracy. There is a lot more innovation around inference that's coming fast. Really encouraged by the study on sparse computation to enhance the computational efficiency of LLMs. It's now possible to use PowerInfer with Llama 2 and Faclon 40B. Mistral-7B support is coming soon!

elvis

261,756 次观看 • 2 年前

Now introducing Replit Desktop, a seamless way to multitask and get the best of Replit right on your Desktop. Supported for both Windows and Mac.

Now introducing Replit Desktop, a seamless way to multitask and get the best of Replit right on your Desktop. Supported for both Windows and Mac.

Replit ⠕

83,988 次观看 • 3 个月前

“Greater Manchester productivity rose over three times faster than London.” Great interview by Lord Jim O’Neil on #R4Today. GM is leading the way to growth for the next Government. 👍🏻

“Greater Manchester productivity rose over three times faster than London.” Great interview by Lord Jim O’Neil on #R4Today. GM is leading the way to growth for the next Government. 👍🏻

Andy Burnham

34,798 次观看 • 2 年前

Today we release LFM2.5-Encoder-230M and LFM2.5-Encoder-350M: bidirectional encoders that stay fast at long context, even on CPU. > LFM2.5-Encoder-230M: about 3.7x faster than ModernBERT-base on CPU at 8,192 tokens. Under 30s per forward pass, versus over a minute and a half. > LFM2.5-Encoder-350M: 4th of 14 models on GLUE, SuperGLUE, and multilingual classification, behind only three larger models, one of them nearly 10x its size. 🧵

Today we release LFM2.5-Encoder-230M and LFM2.5-Encoder-350M: bidirectional encoders that stay fast at long context, even on CPU. > LFM2.5-Encoder-230M: about 3.7x faster than ModernBERT-base on CPU at 8,192 tokens. Under 30s per forward pass, versus over a minute and a half. > LFM2.5-Encoder-350M: 4th of 14 models on GLUE, SuperGLUE, and multilingual classification, behind only three larger models, one of them nearly 10x its size. 🧵

Liquid AI

199,659 次观看 • 2 个月前

Introducing HydraDB. The graph native context infrastructure for agents. Purpose built to deliver precise context & observability into why agents act the way they do. We've always believed graphs are the best way to manage AI context, but they've been too expensive to scale or impractical for storing full context. Until now. HydraDB combines in memory, NVMe, and object storage into a single graph layer, making context delivery faster, cheaper, and more precise. We want context delivery to be extremely fast, 1000x cheap, and highly precise. Give your agents a brain.

Introducing HydraDB. The graph native context infrastructure for agents. Purpose built to deliver precise context & observability into why agents act the way they do. We've always believed graphs are the best way to manage AI context, but they've been too expensive to scale or impractical for storing full context. Until now. HydraDB combines in memory, NVMe, and object storage into a single graph layer, making context delivery faster, cheaper, and more precise. We want context delivery to be extremely fast, 1000x cheap, and highly precise. Give your agents a brain.

Nishkarsh

2,302,368 次观看 • 4 个月前

No forced towers. No violation of rights. Myth vs Reality. Telecommunications Amendment Bill protects private property, ensures consent, and clears myths, while paving the way for faster, cheaper internet across Pakistan! 🇵🇰 Facts over propaganda. Always! 🙌🏼

No forced towers. No violation of rights. Myth vs Reality. Telecommunications Amendment Bill protects private property, ensures consent, and clears myths, while paving the way for faster, cheaper internet across Pakistan! 🇵🇰 Facts over propaganda. Always! 🙌🏼

Saad Kaiser 🇵🇰

19,069 次观看 • 3 个月前

Run Stable Audio 3 on ANY Computer with NO VRAM 1-click launcher for the official Stable Audio 3 gradio app. 1. Cross Platform: Works on ALL OS (Mac, Linux, Windows) out of the box 2. No GPU Needed: Even runs on CPU. 3. Ultra Fast: Generate a 2 minute clip in split seconds

Run Stable Audio 3 on ANY Computer with NO VRAM 1-click launcher for the official Stable Audio 3 gradio app. 1. Cross Platform: Works on ALL OS (Mac, Linux, Windows) out of the box 2. No GPU Needed: Even runs on CPU. 3. Ultra Fast: Generate a 2 minute clip in split seconds

cocktail peanut

72,933 次观看 • 4 个月前

PHALA PHALA SAGA | uMkhonto weSizwe Party MP Mzwanele Manyi says he expected the process paving the way for impeachment proceedings to move faster, but is now frustrated by the legal advice being given to the Subcommittee on the Review of Assembly Rules.

PHALA PHALA SAGA | uMkhonto weSizwe Party MP Mzwanele Manyi says he expected the process paving the way for impeachment proceedings to move faster, but is now frustrated by the legal advice being given to the Subcommittee on the Review of Assembly Rules.

SABC News

20,525 次观看 • 4 个月前

Paving the way for a transformed and seamless freight journey with the Rail-Over-Rail Flyover at the New Bhaupur - New Pt. Deen Dayal Upadhyaya Section on Eastern Dedicated Freight Corridor.

Paving the way for a transformed and seamless freight journey with the Rail-Over-Rail Flyover at the New Bhaupur - New Pt. Deen Dayal Upadhyaya Section on Eastern Dedicated Freight Corridor.

Ministry of Railways

38,262 次观看 • 1 年前

Building on the proven success of Stretch for automated container unloading, DHL Group is paving the way for the global deployment of more than 1,000 additional robots. Learn how DHL is accelerating their automation strategy:

Building on the proven success of Stretch for automated container unloading, DHL Group is paving the way for the global deployment of more than 1,000 additional robots. Learn how DHL is accelerating their automation strategy:

Boston Dynamics

54,125 次观看 • 1 年前

QVAC SDK 0.15.0 is live. This release adds multiple prompts batching, brings a native AMD GPU backend to the stack, moves more vision encoders onto mobile GPUs, and adds a second local coding-agent integration. Main highlights: - Prompt batching for the LLM addon. Batch multiple prompts into one job and process them concurrently, with each answer returned the moment its generation finishes. - Native AMD GPU backend. A first-class HIP/ROCm backend in @qvac/vla-ggml, auto-selected over Vulkan with clean fallback when ROCm is absent. - A second local coding agent. OpenClaw joins OpenCode for local, cloud-free agent workflows. AGENTS - OpenCode plugin update (@qvac/opencode-plugin). Aligned with the current SDK, CLI, and AI SDK provider packages. A fresh install runs OpenCode against managed local QVAC models out of the box, from the default qvac/qwen3.5-9b, with no manual qvac serve setup. - OpenClaw plugin (@qvac/openclaw-plugin). A second coding-agent integration alongside OpenCode. A fresh setup installs the plugin, creates a local qvac provider through onboarding, and runs a QVAC model through OpenClaw🦞's local service path. LANGUAGE MODELS - Prompt batching (LLM addon). Batch multiple prompts in one job and run them concurrently, each answer returns the moment its generation finishes, no waiting on the others. - Reasoning-context trimming on hybrid + recurrent models (@qvac/llm-llamacpp). remove_thinking_from_context now works beyond pure-attention models. Same JS API, no throw. VOICE AND SPEECH - Transcription (transcription-parakeet 0.9.0). More robust CPU fallback on GPU failure and a faster Vulkan backend on Pixel 9. - Text-to-speech features (tts-ggml 0.4.0). Adds LavaSR for noise removal and adjustable output frequency up to 48 kHz, plus Japanese via Chatterbox. - Text-to-speech fixes (tts-ggml 0.4.1). CPU fallback on GPU failure, a q8_0 KV crash fix on Metal with Chatterbox. VISION - Qwen3.5 vision encoder on GPU (Android). Image encoder moves onto the phone GPU, with a smarter tile-grid preprocessor and default image-token caps, for flagship Android: Vulkan on Mali (Pixel 9 Pro) and OpenCL on Adreno 830 (Galaxy S25). - Gemma-4 vision encoder on GPU (Android). Vision encoder runs on the phone GPU instead of CPU, same flagship Android targets. PLATFORM AND PERFORMANCE - AMD GPU backend (@qvac/vla-ggml). Native HIP/ROCm backend, auto-selected over Vulkan with clean fallback when ROCm is absent (Linux x64 only). Comes with ~23% faster than Vulkan, ~14% faster than PyTorch-ROCm, parity preserved. Unified code style. A cleaner, more consistent, easier-to-contribute codebase. Let's build. npm install @qvac/sdk

QVAC SDK 0.15.0 is live. This release adds multiple prompts batching, brings a native AMD GPU backend to the stack, moves more vision encoders onto mobile GPUs, and adds a second local coding-agent integration. Main highlights: - Prompt batching for the LLM addon. Batch multiple prompts into one job and process them concurrently, with each answer returned the moment its generation finishes. - Native AMD GPU backend. A first-class HIP/ROCm backend in @qvac/vla-ggml, auto-selected over Vulkan with clean fallback when ROCm is absent. - A second local coding agent. OpenClaw joins OpenCode for local, cloud-free agent workflows. AGENTS - OpenCode plugin update (@qvac/opencode-plugin). Aligned with the current SDK, CLI, and AI SDK provider packages. A fresh install runs OpenCode against managed local QVAC models out of the box, from the default qvac/qwen3.5-9b, with no manual qvac serve setup. - OpenClaw plugin (@qvac/openclaw-plugin). A second coding-agent integration alongside OpenCode. A fresh setup installs the plugin, creates a local qvac provider through onboarding, and runs a QVAC model through OpenClaw🦞's local service path. LANGUAGE MODELS - Prompt batching (LLM addon). Batch multiple prompts in one job and run them concurrently, each answer returns the moment its generation finishes, no waiting on the others. - Reasoning-context trimming on hybrid + recurrent models (@qvac/llm-llamacpp). remove_thinking_from_context now works beyond pure-attention models. Same JS API, no throw. VOICE AND SPEECH - Transcription (transcription-parakeet 0.9.0). More robust CPU fallback on GPU failure and a faster Vulkan backend on Pixel 9. - Text-to-speech features (tts-ggml 0.4.0). Adds LavaSR for noise removal and adjustable output frequency up to 48 kHz, plus Japanese via Chatterbox. - Text-to-speech fixes (tts-ggml 0.4.1). CPU fallback on GPU failure, a q8_0 KV crash fix on Metal with Chatterbox. VISION - Qwen3.5 vision encoder on GPU (Android). Image encoder moves onto the phone GPU, with a smarter tile-grid preprocessor and default image-token caps, for flagship Android: Vulkan on Mali (Pixel 9 Pro) and OpenCL on Adreno 830 (Galaxy S25). - Gemma-4 vision encoder on GPU (Android). Vision encoder runs on the phone GPU instead of CPU, same flagship Android targets. PLATFORM AND PERFORMANCE - AMD GPU backend (@qvac/vla-ggml). Native HIP/ROCm backend, auto-selected over Vulkan with clean fallback when ROCm is absent (Linux x64 only). Comes with ~23% faster than Vulkan, ~14% faster than PyTorch-ROCm, parity preserved. Unified code style. A cleaner, more consistent, easier-to-contribute codebase. Let's build. npm install @qvac/sdk

QVAC

29,260,217 次观看 • 3 个月前

locate-anything.cpp: native C++/ggml (ggml) inference for NVIDIA's LocateAnything-3B, open-vocabulary object detection / visual grounding, one of the neat detection VLMs out there, from the LocalAI team. Same detections as the official model, now running anywhere with no Python and faster, on CPU and GPU. Quantized GGUF on Hugging Face 🤗 Huge thanks to Georgi Gerganov for ggml and to NVIDIAAI for releasing LocateAnything! 🧵

locate-anything.cpp: native C++/ggml (ggml) inference for NVIDIA's LocateAnything-3B, open-vocabulary object detection / visual grounding, one of the neat detection VLMs out there, from the LocalAI team. Same detections as the official model, now running anywhere with no Python and faster, on CPU and GPU. Quantized GGUF on Hugging Face 🤗 Huge thanks to Georgi Gerganov for ggml and to NVIDIAAI for releasing LocateAnything! 🧵

Ettore Di Giacinto

16,770 次观看 • 4 个月前

We just killed Exa. Introducing the FASTEST search engine for agents on Earth. Your agent can now search the web 6× faster than Exa. Exa: 372ms P50 Us: 62ms P50 Made possible by OctenAI × Monid.

We just killed Exa. Introducing the FASTEST search engine for agents on Earth. Your agent can now search the web 6× faster than Exa. Exa: 372ms P50 Us: 62ms P50 Made possible by OctenAI × Monid.

Shengkun

278,890 次观看 • 2 个月前