Video wird geladen...

Video konnte nicht geladen werden

Beim Laden dieses Videos ist ein Problem aufgetreten. Dies könnte an einem vorübergehenden Netzwerkproblem liegen oder das Video ist möglicherweise nicht verfügbar.

Paving the way for unlimited context windows. Introducing ZML /attnd: sparse logarithmic attention, on CPU, over UDP, faster than GPU.

Steeve Morin

9,588 subscribers

122,236 Aufrufe • vor 1 Jahr •via X (Twitter)

Wissenschaft & Technologie Nachrichten & Politik Bildung

Anya Rossi• Live Now

Private livecam show

39 Kommentare

Profilbild von Charles Foster

Charles Fostervor 1 Jahr

@zml_ai How do you avoid computing the attention values that end up ~0 without computing the dot products between each query-key pair? Are you using some sort of approximation?

Profilbild von Steeve Morin

Steeve Morinvor 1 Jahr

@zml_ai we model it as a graph

Profilbild von Charles Foster

Charles Fostervor 1 Jahr

@zml_ai How does modeling it as a graph let you bypass the N × N for sequence length of N, unless you're making some kind of approximation?

Profilbild von Steeve Morin

Steeve Morinvor 1 Jahr

@zml_ai Yes there is approximation in the sense that we don't compute values very close to zero. From the talk:

Profilbild von Charles Foster

Charles Fostervor 1 Jahr

@zml_ai How? How do you know which attention values *will be* very close to zero without first computing the dot-products?

Profilbild von Charles Foster

Charles Fostervor 1 Jahr

@zml_ai Maybe what you mean to say is that you compute the full N x N attention matrix, but aggregate the value vectors sparsely by skipping entries where the attention is ~0? That would turn one dense-dense matmul into a sparse-dense one, but wouldn’t make the whole thing log(N), right?

Profilbild von Jean-Baptiste Schiratti

Jean-Baptiste Schirattivor 1 Jahr

@zml_ai Loved the talk & demo 🤩 @zml_ai rocks 🔥🚀❤️

Profilbild von James

Jamesvor 1 Jahr

@zml_ai @TheAhmadOsman looks like a great local llama unlock

Profilbild von Javier Honduvilla Coto

Javier Honduvilla Cotovor 1 Jahr

@zml_ai Great stuff!! Btw love the T-shirt 🤣

Profilbild von David Roldan 🇵🇷

David Roldan 🇵🇷vor 1 Jahr

@zml_ai Democratizing AI!!! 👌🏼

Profilbild von Abhi

Abhivor 1 Jahr

@zml_ai Good work man, holy shit

Profilbild von Guillaume Wenzek

Guillaume Wenzekvor 1 Jahr

@zml_ai Very excited to share this. Attention is fascinating. I've been toying with this algorithm idea for some time but it took some faith and crunch to actually make it work.

Profilbild von murat 🍥

murat 🍥vor 1 Jahr

@zml_ai any performance degradation?

Profilbild von Steeve Morin

Steeve Morinvor 1 Jahr

@zml_ai no

Profilbild von Andrew Huynh

Andrew Huynhvor 1 Jahr

@zml_ai How to run it in my m2 laptop??

Profilbild von Saïd Aitmbarek

Saïd Aitmbarekvor 1 Jahr

@zml_ai hell yeah, based optimization for cpu guys! will help democratize AI adoption to the immense pool of non gpu socs let's launch/spotlight you on whenever it helps guys

Profilbild von VibeCodeTeddy

VibeCodeTeddyvor 1 Jahr

@zml_ai This is wild. Can't wait to see the practical applications of this!

Profilbild von Ak afridi

Ak afridivor 1 Jahr

@zml_ai Fascinating approach to solving the context window limitation. CPU-based sparse attention over UDP opening new possibilities for distributed AI processing. Looking forward to seeing how this scales in production environments.

Profilbild von Gary Basin

Gary Basinvor 1 Jahr

@zml_ai very cool

Profilbild von Mykhailo Sorochuk

Mykhailo Sorochukvor 1 Jahr

@zml_ai This breakthrough could really democratize AI. Less hardware, more access. Excited to see how it develops!

Profilbild von Gloria Baeuerlein

Gloria Baeuerleinvor 1 Jahr

@zml_ai 🔥🔥🔥

Profilbild von Eugene Cheah - AI builder @ 🇸🇬|🇺🇸

Eugene Cheah - AI builder @ 🇸🇬|🇺🇸vor 1 Jahr

@zml_ai cool stuff - however you need benchmarks on basic evals on running with CPU vs GPU, to really get this across the crowd/finish line

Profilbild von James T

James Tvor 1 Jahr

@zml_ai Outstanding will try this out on my old macbook pro 2017

Profilbild von AI PlanetX

AI PlanetXvor 1 Jahr

@zml_ai Exciting innovation in attention technology!

Profilbild von Alon Burg

Alon Burgvor 1 Jahr

@RemiCadene @zml_ai Amazing demo! Kudos!

Profilbild von parody. Dean the Magic 8 Ball geeker

parody. Dean the Magic 8 Ball geekervor 1 Jahr

@ylecun @zml_ai @grok eli5

Profilbild von Andrew Huynh

Andrew Huynhvor 1 Jahr

@zml_ai Cool

Profilbild von Pierre-Baptiste Borges

Pierre-Baptiste Borgesvor 1 Jahr

@zml_ai congrats!

Profilbild von Reza Rawassizadeh

Reza Rawassizadehvor 1 Jahr

@zml_ai Where can we read about sparse logarithm attention?

Profilbild von jdubbs

jdubbsvor 1 Jahr

@zml_ai Loved your CRdo writeup/thread......Do you think this eventually hurts the opical guys like CIEN and GLW? Thanks

Profilbild von Cyril Mottier

Cyril Mottiervor 1 Jahr

@zml_ai 💥

Profilbild von VibeEdge

VibeEdgevor 1 Jahr

@zml_ai @steeve Cool breakthrough on Attention D sparse log attention on CPU via UDP outpacing GPUs? Game changer for AI scaling. Echoes Megalodon style infinite contexts without heavy hardware. Excited for the code!

Profilbild von pybankers

pybankersvor 1 Jahr

@zml_ai What model size will I be able to run with 16gb ram cpu without any gpu with good speed?

Profilbild von Anthara Fairooz

Anthara Fairoozvor 1 Jahr

@zml_ai Exciting innovation in AI! Great step forward.

Profilbild von Conor

Conorvor 1 Jahr

@zml_ai Super cool, congrats

Profilbild von Giovanni P.

Giovanni P.vor 1 Jahr

@zml_ai Very cool, bold to use zig 👍 But bazel 🤣 please no!

Profilbild von Steeve Morin

Steeve Morinvor 1 Jahr

@zml_ai

Profilbild von iG3 - Edge AI

iG3 - Edge AIvor 1 Jahr

@zml_ai 👀

Profilbild von gawen

gawenvor 1 Jahr

@zml_ai Boua very cool ! 🔥

Ähnliche Videos

Introducing Ziren (formerly zkMIPS) The most performant general-purpose zkVM on CPU, now scaled with GPU, distributed proving, and more. Build with confidence. Deploy without compromise 🟠 Full context: Technical updates:

Introducing Ziren (formerly zkMIPS) The most performant general-purpose zkVM on CPU, now scaled with GPU, distributed proving, and more. Build with confidence. Deploy without compromise 🟠 Full context: Technical updates:

ZKM

5,857,968 Aufrufe • vor 1 Jahr

$Introducing SubQ - a major breakthrough in LLM intelligence. It is the first model built on a fully sub-quadratic sparse-attention architecture (SSA), And the first frontier model with a 12 million token context window which is: - 52x faster than FlashAttention at 1MM tokens - Less than 5% the cost of Opus Transformer-based LLMs waste compute by processing every possible relationship between words (standard attention). Only a small fraction actually matter. Subquadratic finds and focuses only on the ones that do. That's nearly 1,000x less compute and a new way for LLMs to scale.$

Introducing SubQ - a major breakthrough in LLM intelligence. It is the first model built on a fully sub-quadratic sparse-attention architecture (SSA), And the first frontier model with a 12 million token context window which is: - 52x faster than FlashAttention at 1MM tokens - Less than 5% the cost of Opus Transformer-based LLMs waste compute by processing every possible relationship between words (standard attention). Only a small fraction actually matter. Subquadratic finds and focuses only on the ones that do. That's nearly 1,000x less compute and a new way for LLMs to scale.

Alexander Whedon

13,492,280 Aufrufe • vor 5 Monaten

$INFQ GPU + CPU + QPU Frontier models are hitting saturation on tokens (data volume) and context windows (memory). $INFQ CTO walks through how they and $NVDA are working together to add a Quantum Processing Unit to the chain as the next inflection point.

$INFQ GPU + CPU + QPU Frontier models are hitting saturation on tokens (data volume) and context windows (memory). $INFQ CTO walks through how they and $NVDA are working together to add a Quantum Processing Unit to the chain as the next inflection point.

YeahDave

57,181 Aufrufe • vor 5 Monaten

Recent KV cache compression research has focused on small models with dense attention. We scaled these techniques to GLM 5.3, a frontier sparse attention model. We adapted KVzip and Attention Matching, reducing KV cache size by 80% while retaining over 90% of full-context accuracy on the QuALITY benchmark.

Recent KV cache compression research has focused on small models with dense attention. We scaled these techniques to GLM 5.3, a frontier sparse attention model. We adapted KVzip and Attention Matching, reducing KV cache size by 80% while retaining over 90% of full-context accuracy on the QuALITY benchmark.

Ramp Labs

20,677 Aufrufe • vor 15 Tagen

Windows ML is now generally available! This milestone brings powerful, local AI capability to every Windows 11 device, harnessing the full power of the GPU, NPU, and CPU. Big thank you to the team and our silicon partners, AMD, Intel, NVIDIA, and Qualcomm, for their collaboration to unlock real-time, secure, and efficient AI experiences for developers on Windows. Let’s go 🚀!

Windows ML is now generally available! This milestone brings powerful, local AI capability to every Windows 11 device, harnessing the full power of the GPU, NPU, and CPU. Big thank you to the team and our silicon partners, AMD, Intel, NVIDIA, and Qualcomm, for their collaboration to unlock real-time, secure, and efficient AI experiences for developers on Windows. Let’s go 🚀!

Pavan Davuluri

11,663 Aufrufe • vor 1 Jahr

PowerInfer - a high-speed inference engine for deploying LLMs locally. Just came across this super interesting project on speeding up inference. It's not MoE but it's a simple approach that exploits the high locality in LLM inference to design a GPU-CPU hybrid inference engine. Hot-activated neurons are preloaded onto the GPU for fast access, while cold-activated neurons (the majority) are computed on the CPU. This approach significantly reduces GPU memory demands and CPU-GPU data transfer. It achieves an average token generation rate of 13.20 tokens/s, with a peak of 29.08 tokens/s, across various LLMs on a single NVIDIA RTX 4090 GPU. It's on only 18% lower than that achieved by a top-tier server-grade A100 GPU. It also significantly outperforms llama.cpp by up to 11.69x while retaining model accuracy. There is a lot more innovation around inference that's coming fast. Really encouraged by the study on sparse computation to enhance the computational efficiency of LLMs. It's now possible to use PowerInfer with Llama 2 and Faclon 40B. Mistral-7B support is coming soon!

PowerInfer - a high-speed inference engine for deploying LLMs locally. Just came across this super interesting project on speeding up inference. It's not MoE but it's a simple approach that exploits the high locality in LLM inference to design a GPU-CPU hybrid inference engine. Hot-activated neurons are preloaded onto the GPU for fast access, while cold-activated neurons (the majority) are computed on the CPU. This approach significantly reduces GPU memory demands and CPU-GPU data transfer. It achieves an average token generation rate of 13.20 tokens/s, with a peak of 29.08 tokens/s, across various LLMs on a single NVIDIA RTX 4090 GPU. It's on only 18% lower than that achieved by a top-tier server-grade A100 GPU. It also significantly outperforms llama.cpp by up to 11.69x while retaining model accuracy. There is a lot more innovation around inference that's coming fast. Really encouraged by the study on sparse computation to enhance the computational efficiency of LLMs. It's now possible to use PowerInfer with Llama 2 and Faclon 40B. Mistral-7B support is coming soon!

elvis

261,756 Aufrufe • vor 2 Jahren

Now introducing Replit Desktop, a seamless way to multitask and get the best of Replit right on your Desktop. Supported for both Windows and Mac.

Now introducing Replit Desktop, a seamless way to multitask and get the best of Replit right on your Desktop. Supported for both Windows and Mac.

Replit ⠕

83,988 Aufrufe • vor 3 Monaten

“Greater Manchester productivity rose over three times faster than London.” Great interview by Lord Jim O’Neil on #R4Today. GM is leading the way to growth for the next Government. 👍🏻

“Greater Manchester productivity rose over three times faster than London.” Great interview by Lord Jim O’Neil on #R4Today. GM is leading the way to growth for the next Government. 👍🏻

Andy Burnham

34,798 Aufrufe • vor 2 Jahren

Today we release LFM2.5-Encoder-230M and LFM2.5-Encoder-350M: bidirectional encoders that stay fast at long context, even on CPU. > LFM2.5-Encoder-230M: about 3.7x faster than ModernBERT-base on CPU at 8,192 tokens. Under 30s per forward pass, versus over a minute and a half. > LFM2.5-Encoder-350M: 4th of 14 models on GLUE, SuperGLUE, and multilingual classification, behind only three larger models, one of them nearly 10x its size. 🧵

Today we release LFM2.5-Encoder-230M and LFM2.5-Encoder-350M: bidirectional encoders that stay fast at long context, even on CPU. > LFM2.5-Encoder-230M: about 3.7x faster than ModernBERT-base on CPU at 8,192 tokens. Under 30s per forward pass, versus over a minute and a half. > LFM2.5-Encoder-350M: 4th of 14 models on GLUE, SuperGLUE, and multilingual classification, behind only three larger models, one of them nearly 10x its size. 🧵

Liquid AI

199,659 Aufrufe • vor 2 Monaten

Introducing HydraDB. The graph native context infrastructure for agents. Purpose built to deliver precise context & observability into why agents act the way they do. We've always believed graphs are the best way to manage AI context, but they've been too expensive to scale or impractical for storing full context. Until now. HydraDB combines in memory, NVMe, and object storage into a single graph layer, making context delivery faster, cheaper, and more precise. We want context delivery to be extremely fast, 1000x cheap, and highly precise. Give your agents a brain.

Introducing HydraDB. The graph native context infrastructure for agents. Purpose built to deliver precise context & observability into why agents act the way they do. We've always believed graphs are the best way to manage AI context, but they've been too expensive to scale or impractical for storing full context. Until now. HydraDB combines in memory, NVMe, and object storage into a single graph layer, making context delivery faster, cheaper, and more precise. We want context delivery to be extremely fast, 1000x cheap, and highly precise. Give your agents a brain.

Nishkarsh

2,302,368 Aufrufe • vor 4 Monaten

No forced towers. No violation of rights. Myth vs Reality. Telecommunications Amendment Bill protects private property, ensures consent, and clears myths, while paving the way for faster, cheaper internet across Pakistan! 🇵🇰 Facts over propaganda. Always! 🙌🏼

No forced towers. No violation of rights. Myth vs Reality. Telecommunications Amendment Bill protects private property, ensures consent, and clears myths, while paving the way for faster, cheaper internet across Pakistan! 🇵🇰 Facts over propaganda. Always! 🙌🏼

Saad Kaiser 🇵🇰

19,069 Aufrufe • vor 3 Monaten

Run Stable Audio 3 on ANY Computer with NO VRAM 1-click launcher for the official Stable Audio 3 gradio app. 1. Cross Platform: Works on ALL OS (Mac, Linux, Windows) out of the box 2. No GPU Needed: Even runs on CPU. 3. Ultra Fast: Generate a 2 minute clip in split seconds

Run Stable Audio 3 on ANY Computer with NO VRAM 1-click launcher for the official Stable Audio 3 gradio app. 1. Cross Platform: Works on ALL OS (Mac, Linux, Windows) out of the box 2. No GPU Needed: Even runs on CPU. 3. Ultra Fast: Generate a 2 minute clip in split seconds

cocktail peanut

72,933 Aufrufe • vor 4 Monaten

PHALA PHALA SAGA | uMkhonto weSizwe Party MP Mzwanele Manyi says he expected the process paving the way for impeachment proceedings to move faster, but is now frustrated by the legal advice being given to the Subcommittee on the Review of Assembly Rules.

PHALA PHALA SAGA | uMkhonto weSizwe Party MP Mzwanele Manyi says he expected the process paving the way for impeachment proceedings to move faster, but is now frustrated by the legal advice being given to the Subcommittee on the Review of Assembly Rules.

SABC News

20,525 Aufrufe • vor 4 Monaten

Paving the way for a transformed and seamless freight journey with the Rail-Over-Rail Flyover at the New Bhaupur - New Pt. Deen Dayal Upadhyaya Section on Eastern Dedicated Freight Corridor.

Paving the way for a transformed and seamless freight journey with the Rail-Over-Rail Flyover at the New Bhaupur - New Pt. Deen Dayal Upadhyaya Section on Eastern Dedicated Freight Corridor.

Ministry of Railways

38,262 Aufrufe • vor 1 Jahr

Building on the proven success of Stretch for automated container unloading, DHL Group is paving the way for the global deployment of more than 1,000 additional robots. Learn how DHL is accelerating their automation strategy:

Building on the proven success of Stretch for automated container unloading, DHL Group is paving the way for the global deployment of more than 1,000 additional robots. Learn how DHL is accelerating their automation strategy:

Boston Dynamics

54,125 Aufrufe • vor 1 Jahr

QVAC SDK 0.15.0 is live. This release adds multiple prompts batching, brings a native AMD GPU backend to the stack, moves more vision encoders onto mobile GPUs, and adds a second local coding-agent integration. Main highlights: - Prompt batching for the LLM addon. Batch multiple prompts into one job and process them concurrently, with each answer returned the moment its generation finishes. - Native AMD GPU backend. A first-class HIP/ROCm backend in @qvac/vla-ggml, auto-selected over Vulkan with clean fallback when ROCm is absent. - A second local coding agent. OpenClaw joins OpenCode for local, cloud-free agent workflows. AGENTS - OpenCode plugin update (@qvac/opencode-plugin). Aligned with the current SDK, CLI, and AI SDK provider packages. A fresh install runs OpenCode against managed local QVAC models out of the box, from the default qvac/qwen3.5-9b, with no manual qvac serve setup. - OpenClaw plugin (@qvac/openclaw-plugin). A second coding-agent integration alongside OpenCode. A fresh setup installs the plugin, creates a local qvac provider through onboarding, and runs a QVAC model through OpenClaw🦞's local service path. LANGUAGE MODELS - Prompt batching (LLM addon). Batch multiple prompts in one job and run them concurrently, each answer returns the moment its generation finishes, no waiting on the others. - Reasoning-context trimming on hybrid + recurrent models (@qvac/llm-llamacpp). remove_thinking_from_context now works beyond pure-attention models. Same JS API, no throw. VOICE AND SPEECH - Transcription (transcription-parakeet 0.9.0). More robust CPU fallback on GPU failure and a faster Vulkan backend on Pixel 9. - Text-to-speech features (tts-ggml 0.4.0). Adds LavaSR for noise removal and adjustable output frequency up to 48 kHz, plus Japanese via Chatterbox. - Text-to-speech fixes (tts-ggml 0.4.1). CPU fallback on GPU failure, a q8_0 KV crash fix on Metal with Chatterbox. VISION - Qwen3.5 vision encoder on GPU (Android). Image encoder moves onto the phone GPU, with a smarter tile-grid preprocessor and default image-token caps, for flagship Android: Vulkan on Mali (Pixel 9 Pro) and OpenCL on Adreno 830 (Galaxy S25). - Gemma-4 vision encoder on GPU (Android). Vision encoder runs on the phone GPU instead of CPU, same flagship Android targets. PLATFORM AND PERFORMANCE - AMD GPU backend (@qvac/vla-ggml). Native HIP/ROCm backend, auto-selected over Vulkan with clean fallback when ROCm is absent (Linux x64 only). Comes with ~23% faster than Vulkan, ~14% faster than PyTorch-ROCm, parity preserved. Unified code style. A cleaner, more consistent, easier-to-contribute codebase. Let's build. npm install @qvac/sdk

QVAC SDK 0.15.0 is live. This release adds multiple prompts batching, brings a native AMD GPU backend to the stack, moves more vision encoders onto mobile GPUs, and adds a second local coding-agent integration. Main highlights: - Prompt batching for the LLM addon. Batch multiple prompts into one job and process them concurrently, with each answer returned the moment its generation finishes. - Native AMD GPU backend. A first-class HIP/ROCm backend in @qvac/vla-ggml, auto-selected over Vulkan with clean fallback when ROCm is absent. - A second local coding agent. OpenClaw joins OpenCode for local, cloud-free agent workflows. AGENTS - OpenCode plugin update (@qvac/opencode-plugin). Aligned with the current SDK, CLI, and AI SDK provider packages. A fresh install runs OpenCode against managed local QVAC models out of the box, from the default qvac/qwen3.5-9b, with no manual qvac serve setup. - OpenClaw plugin (@qvac/openclaw-plugin). A second coding-agent integration alongside OpenCode. A fresh setup installs the plugin, creates a local qvac provider through onboarding, and runs a QVAC model through OpenClaw🦞's local service path. LANGUAGE MODELS - Prompt batching (LLM addon). Batch multiple prompts in one job and run them concurrently, each answer returns the moment its generation finishes, no waiting on the others. - Reasoning-context trimming on hybrid + recurrent models (@qvac/llm-llamacpp). remove_thinking_from_context now works beyond pure-attention models. Same JS API, no throw. VOICE AND SPEECH - Transcription (transcription-parakeet 0.9.0). More robust CPU fallback on GPU failure and a faster Vulkan backend on Pixel 9. - Text-to-speech features (tts-ggml 0.4.0). Adds LavaSR for noise removal and adjustable output frequency up to 48 kHz, plus Japanese via Chatterbox. - Text-to-speech fixes (tts-ggml 0.4.1). CPU fallback on GPU failure, a q8_0 KV crash fix on Metal with Chatterbox. VISION - Qwen3.5 vision encoder on GPU (Android). Image encoder moves onto the phone GPU, with a smarter tile-grid preprocessor and default image-token caps, for flagship Android: Vulkan on Mali (Pixel 9 Pro) and OpenCL on Adreno 830 (Galaxy S25). - Gemma-4 vision encoder on GPU (Android). Vision encoder runs on the phone GPU instead of CPU, same flagship Android targets. PLATFORM AND PERFORMANCE - AMD GPU backend (@qvac/vla-ggml). Native HIP/ROCm backend, auto-selected over Vulkan with clean fallback when ROCm is absent (Linux x64 only). Comes with ~23% faster than Vulkan, ~14% faster than PyTorch-ROCm, parity preserved. Unified code style. A cleaner, more consistent, easier-to-contribute codebase. Let's build. npm install @qvac/sdk

QVAC

29,260,217 Aufrufe • vor 3 Monaten

locate-anything.cpp: native C++/ggml (ggml) inference for NVIDIA's LocateAnything-3B, open-vocabulary object detection / visual grounding, one of the neat detection VLMs out there, from the LocalAI team. Same detections as the official model, now running anywhere with no Python and faster, on CPU and GPU. Quantized GGUF on Hugging Face 🤗 Huge thanks to Georgi Gerganov for ggml and to NVIDIAAI for releasing LocateAnything! 🧵

locate-anything.cpp: native C++/ggml (ggml) inference for NVIDIA's LocateAnything-3B, open-vocabulary object detection / visual grounding, one of the neat detection VLMs out there, from the LocalAI team. Same detections as the official model, now running anywhere with no Python and faster, on CPU and GPU. Quantized GGUF on Hugging Face 🤗 Huge thanks to Georgi Gerganov for ggml and to NVIDIAAI for releasing LocateAnything! 🧵

Ettore Di Giacinto

16,770 Aufrufe • vor 4 Monaten

We just killed Exa. Introducing the FASTEST search engine for agents on Earth. Your agent can now search the web 6× faster than Exa. Exa: 372ms P50 Us: 62ms P50 Made possible by OctenAI × Monid.

We just killed Exa. Introducing the FASTEST search engine for agents on Earth. Your agent can now search the web 6× faster than Exa. Exa: 372ms P50 Us: 62ms P50 Made possible by OctenAI × Monid.

Shengkun

278,890 Aufrufe • vor 2 Monaten