Video yükleniyor...

Video Yüklenemedi

Bu video yüklenirken bir sorun oluştu. Bu geçici bir ağ sorunundan kaynaklanıyor olabilir veya video kullanılamıyor olabilir.

Ana Sayfaya Dön

Paving the way for unlimited context windows. Introducing ZML /attnd: sparse logarithmic attention, on CPU, over UDP, faster than GPU.

Steeve Morin

9,588 subscribers

122,236 görüntüleme • 1 yıl önce •via X (Twitter)

Bilim & Teknoloji Haberler & Politika Eğitim

Anya Rossi• Live Now

Private livecam show

39 Yorum

Charles Foster profil fotoğrafı

Charles Foster1 yıl önce

@zml_ai How do you avoid computing the attention values that end up ~0 without computing the dot products between each query-key pair? Are you using some sort of approximation?

Steeve Morin profil fotoğrafı

Steeve Morin1 yıl önce

@zml_ai we model it as a graph

Charles Foster profil fotoğrafı

Charles Foster1 yıl önce

@zml_ai How does modeling it as a graph let you bypass the N × N for sequence length of N, unless you're making some kind of approximation?

Steeve Morin profil fotoğrafı

Steeve Morin1 yıl önce

@zml_ai Yes there is approximation in the sense that we don't compute values very close to zero. From the talk:

Charles Foster profil fotoğrafı

Charles Foster1 yıl önce

@zml_ai How? How do you know which attention values *will be* very close to zero without first computing the dot-products?

Charles Foster profil fotoğrafı

Charles Foster1 yıl önce

@zml_ai Maybe what you mean to say is that you compute the full N x N attention matrix, but aggregate the value vectors sparsely by skipping entries where the attention is ~0? That would turn one dense-dense matmul into a sparse-dense one, but wouldn’t make the whole thing log(N), right?

Jean-Baptiste Schiratti profil fotoğrafı

Jean-Baptiste Schiratti1 yıl önce

@zml_ai Loved the talk & demo 🤩 @zml_ai rocks 🔥🚀❤️

James profil fotoğrafı

James1 yıl önce

@zml_ai @TheAhmadOsman looks like a great local llama unlock

Javier Honduvilla Coto profil fotoğrafı

Javier Honduvilla Coto1 yıl önce

@zml_ai Great stuff!! Btw love the T-shirt 🤣

David Roldan 🇵🇷 profil fotoğrafı

David Roldan 🇵🇷1 yıl önce

@zml_ai Democratizing AI!!! 👌🏼

Abhi profil fotoğrafı

Abhi1 yıl önce

@zml_ai Good work man, holy shit

Guillaume Wenzek profil fotoğrafı

Guillaume Wenzek1 yıl önce

@zml_ai Very excited to share this. Attention is fascinating. I've been toying with this algorithm idea for some time but it took some faith and crunch to actually make it work.

murat 🍥 profil fotoğrafı

murat 🍥1 yıl önce

@zml_ai any performance degradation?

Steeve Morin profil fotoğrafı

Steeve Morin1 yıl önce

@zml_ai no

Andrew Huynh profil fotoğrafı

Andrew Huynh1 yıl önce

@zml_ai How to run it in my m2 laptop??

Saïd Aitmbarek profil fotoğrafı

Saïd Aitmbarek1 yıl önce

@zml_ai hell yeah, based optimization for cpu guys! will help democratize AI adoption to the immense pool of non gpu socs let's launch/spotlight you on whenever it helps guys

VibeCodeTeddy profil fotoğrafı

VibeCodeTeddy1 yıl önce

@zml_ai This is wild. Can't wait to see the practical applications of this!

Ak afridi profil fotoğrafı

Ak afridi1 yıl önce

@zml_ai Fascinating approach to solving the context window limitation. CPU-based sparse attention over UDP opening new possibilities for distributed AI processing. Looking forward to seeing how this scales in production environments.

Gary Basin profil fotoğrafı

Gary Basin1 yıl önce

@zml_ai very cool

Mykhailo Sorochuk profil fotoğrafı

Mykhailo Sorochuk1 yıl önce

@zml_ai This breakthrough could really democratize AI. Less hardware, more access. Excited to see how it develops!

Gloria Baeuerlein profil fotoğrafı

Gloria Baeuerlein1 yıl önce

@zml_ai 🔥🔥🔥

Eugene Cheah - AI builder @ 🇸🇬|🇺🇸 profil fotoğrafı

Eugene Cheah - AI builder @ 🇸🇬|🇺🇸1 yıl önce

@zml_ai cool stuff - however you need benchmarks on basic evals on running with CPU vs GPU, to really get this across the crowd/finish line

James T profil fotoğrafı

James T1 yıl önce

@zml_ai Outstanding will try this out on my old macbook pro 2017

AI PlanetX profil fotoğrafı

AI PlanetX1 yıl önce

@zml_ai Exciting innovation in attention technology!

Alon Burg profil fotoğrafı

Alon Burg1 yıl önce

@RemiCadene @zml_ai Amazing demo! Kudos!

parody. Dean the Magic 8 Ball geeker profil fotoğrafı

parody. Dean the Magic 8 Ball geeker1 yıl önce

@ylecun @zml_ai @grok eli5

Andrew Huynh profil fotoğrafı

Andrew Huynh1 yıl önce

@zml_ai Cool

Pierre-Baptiste Borges profil fotoğrafı

Pierre-Baptiste Borges1 yıl önce

@zml_ai congrats!

Reza Rawassizadeh profil fotoğrafı

Reza Rawassizadeh1 yıl önce

@zml_ai Where can we read about sparse logarithm attention?

jdubbs profil fotoğrafı

jdubbs1 yıl önce

@zml_ai Loved your CRdo writeup/thread......Do you think this eventually hurts the opical guys like CIEN and GLW? Thanks

Cyril Mottier profil fotoğrafı

Cyril Mottier1 yıl önce

@zml_ai 💥

VibeEdge profil fotoğrafı

VibeEdge1 yıl önce

@zml_ai @steeve Cool breakthrough on Attention D sparse log attention on CPU via UDP outpacing GPUs? Game changer for AI scaling. Echoes Megalodon style infinite contexts without heavy hardware. Excited for the code!

pybankers profil fotoğrafı

pybankers1 yıl önce

@zml_ai What model size will I be able to run with 16gb ram cpu without any gpu with good speed?

Anthara Fairooz profil fotoğrafı

Anthara Fairooz1 yıl önce

@zml_ai Exciting innovation in AI! Great step forward.

Conor profil fotoğrafı

Conor1 yıl önce

@zml_ai Super cool, congrats

Giovanni P. profil fotoğrafı

Giovanni P.1 yıl önce

@zml_ai Very cool, bold to use zig 👍 But bazel 🤣 please no!

Steeve Morin profil fotoğrafı

Steeve Morin1 yıl önce

@zml_ai

iG3 - Edge AI profil fotoğrafı

iG3 - Edge AI1 yıl önce

@zml_ai 👀

gawen profil fotoğrafı

gawen1 yıl önce

@zml_ai Boua very cool ! 🔥

Benzer Videolar

Introducing Ziren (formerly zkMIPS) The most performant general-purpose zkVM on CPU, now scaled with GPU, distributed proving, and more. Build with confidence. Deploy without compromise 🟠 Full context: Technical updates:

Introducing Ziren (formerly zkMIPS) The most performant general-purpose zkVM on CPU, now scaled with GPU, distributed proving, and more. Build with confidence. Deploy without compromise 🟠 Full context: Technical updates:

ZKM

5,857,968 görüntüleme • 1 yıl önce

$Introducing SubQ - a major breakthrough in LLM intelligence. It is the first model built on a fully sub-quadratic sparse-attention architecture (SSA), And the first frontier model with a 12 million token context window which is: - 52x faster than FlashAttention at 1MM tokens - Less than 5% the cost of Opus Transformer-based LLMs waste compute by processing every possible relationship between words (standard attention). Only a small fraction actually matter. Subquadratic finds and focuses only on the ones that do. That's nearly 1,000x less compute and a new way for LLMs to scale.$

Introducing SubQ - a major breakthrough in LLM intelligence. It is the first model built on a fully sub-quadratic sparse-attention architecture (SSA), And the first frontier model with a 12 million token context window which is: - 52x faster than FlashAttention at 1MM tokens - Less than 5% the cost of Opus Transformer-based LLMs waste compute by processing every possible relationship between words (standard attention). Only a small fraction actually matter. Subquadratic finds and focuses only on the ones that do. That's nearly 1,000x less compute and a new way for LLMs to scale.

Alexander Whedon

13,492,280 görüntüleme • 5 ay önce

$INFQ GPU + CPU + QPU Frontier models are hitting saturation on tokens (data volume) and context windows (memory). $INFQ CTO walks through how they and $NVDA are working together to add a Quantum Processing Unit to the chain as the next inflection point.

$INFQ GPU + CPU + QPU Frontier models are hitting saturation on tokens (data volume) and context windows (memory). $INFQ CTO walks through how they and $NVDA are working together to add a Quantum Processing Unit to the chain as the next inflection point.

YeahDave

57,181 görüntüleme • 5 ay önce

Recent KV cache compression research has focused on small models with dense attention. We scaled these techniques to GLM 5.3, a frontier sparse attention model. We adapted KVzip and Attention Matching, reducing KV cache size by 80% while retaining over 90% of full-context accuracy on the QuALITY benchmark.

Recent KV cache compression research has focused on small models with dense attention. We scaled these techniques to GLM 5.3, a frontier sparse attention model. We adapted KVzip and Attention Matching, reducing KV cache size by 80% while retaining over 90% of full-context accuracy on the QuALITY benchmark.

Ramp Labs

20,677 görüntüleme • 15 gün önce

Windows ML is now generally available! This milestone brings powerful, local AI capability to every Windows 11 device, harnessing the full power of the GPU, NPU, and CPU. Big thank you to the team and our silicon partners, AMD, Intel, NVIDIA, and Qualcomm, for their collaboration to unlock real-time, secure, and efficient AI experiences for developers on Windows. Let’s go 🚀!

Windows ML is now generally available! This milestone brings powerful, local AI capability to every Windows 11 device, harnessing the full power of the GPU, NPU, and CPU. Big thank you to the team and our silicon partners, AMD, Intel, NVIDIA, and Qualcomm, for their collaboration to unlock real-time, secure, and efficient AI experiences for developers on Windows. Let’s go 🚀!

Pavan Davuluri

11,663 görüntüleme • 1 yıl önce

PowerInfer - a high-speed inference engine for deploying LLMs locally. Just came across this super interesting project on speeding up inference. It's not MoE but it's a simple approach that exploits the high locality in LLM inference to design a GPU-CPU hybrid inference engine. Hot-activated neurons are preloaded onto the GPU for fast access, while cold-activated neurons (the majority) are computed on the CPU. This approach significantly reduces GPU memory demands and CPU-GPU data transfer. It achieves an average token generation rate of 13.20 tokens/s, with a peak of 29.08 tokens/s, across various LLMs on a single NVIDIA RTX 4090 GPU. It's on only 18% lower than that achieved by a top-tier server-grade A100 GPU. It also significantly outperforms llama.cpp by up to 11.69x while retaining model accuracy. There is a lot more innovation around inference that's coming fast. Really encouraged by the study on sparse computation to enhance the computational efficiency of LLMs. It's now possible to use PowerInfer with Llama 2 and Faclon 40B. Mistral-7B support is coming soon!

PowerInfer - a high-speed inference engine for deploying LLMs locally. Just came across this super interesting project on speeding up inference. It's not MoE but it's a simple approach that exploits the high locality in LLM inference to design a GPU-CPU hybrid inference engine. Hot-activated neurons are preloaded onto the GPU for fast access, while cold-activated neurons (the majority) are computed on the CPU. This approach significantly reduces GPU memory demands and CPU-GPU data transfer. It achieves an average token generation rate of 13.20 tokens/s, with a peak of 29.08 tokens/s, across various LLMs on a single NVIDIA RTX 4090 GPU. It's on only 18% lower than that achieved by a top-tier server-grade A100 GPU. It also significantly outperforms llama.cpp by up to 11.69x while retaining model accuracy. There is a lot more innovation around inference that's coming fast. Really encouraged by the study on sparse computation to enhance the computational efficiency of LLMs. It's now possible to use PowerInfer with Llama 2 and Faclon 40B. Mistral-7B support is coming soon!

elvis

261,756 görüntüleme • 2 yıl önce

Now introducing Replit Desktop, a seamless way to multitask and get the best of Replit right on your Desktop. Supported for both Windows and Mac.

Now introducing Replit Desktop, a seamless way to multitask and get the best of Replit right on your Desktop. Supported for both Windows and Mac.

Replit ⠕

83,988 görüntüleme • 3 ay önce

“Greater Manchester productivity rose over three times faster than London.” Great interview by Lord Jim O’Neil on #R4Today. GM is leading the way to growth for the next Government. 👍🏻

“Greater Manchester productivity rose over three times faster than London.” Great interview by Lord Jim O’Neil on #R4Today. GM is leading the way to growth for the next Government. 👍🏻

Andy Burnham

34,798 görüntüleme • 2 yıl önce

Today we release LFM2.5-Encoder-230M and LFM2.5-Encoder-350M: bidirectional encoders that stay fast at long context, even on CPU. > LFM2.5-Encoder-230M: about 3.7x faster than ModernBERT-base on CPU at 8,192 tokens. Under 30s per forward pass, versus over a minute and a half. > LFM2.5-Encoder-350M: 4th of 14 models on GLUE, SuperGLUE, and multilingual classification, behind only three larger models, one of them nearly 10x its size. 🧵

Today we release LFM2.5-Encoder-230M and LFM2.5-Encoder-350M: bidirectional encoders that stay fast at long context, even on CPU. > LFM2.5-Encoder-230M: about 3.7x faster than ModernBERT-base on CPU at 8,192 tokens. Under 30s per forward pass, versus over a minute and a half. > LFM2.5-Encoder-350M: 4th of 14 models on GLUE, SuperGLUE, and multilingual classification, behind only three larger models, one of them nearly 10x its size. 🧵

Liquid AI

199,659 görüntüleme • 2 ay önce

Introducing HydraDB. The graph native context infrastructure for agents. Purpose built to deliver precise context & observability into why agents act the way they do. We've always believed graphs are the best way to manage AI context, but they've been too expensive to scale or impractical for storing full context. Until now. HydraDB combines in memory, NVMe, and object storage into a single graph layer, making context delivery faster, cheaper, and more precise. We want context delivery to be extremely fast, 1000x cheap, and highly precise. Give your agents a brain.

Introducing HydraDB. The graph native context infrastructure for agents. Purpose built to deliver precise context & observability into why agents act the way they do. We've always believed graphs are the best way to manage AI context, but they've been too expensive to scale or impractical for storing full context. Until now. HydraDB combines in memory, NVMe, and object storage into a single graph layer, making context delivery faster, cheaper, and more precise. We want context delivery to be extremely fast, 1000x cheap, and highly precise. Give your agents a brain.

Nishkarsh

2,302,368 görüntüleme • 4 ay önce

No forced towers. No violation of rights. Myth vs Reality. Telecommunications Amendment Bill protects private property, ensures consent, and clears myths, while paving the way for faster, cheaper internet across Pakistan! 🇵🇰 Facts over propaganda. Always! 🙌🏼

No forced towers. No violation of rights. Myth vs Reality. Telecommunications Amendment Bill protects private property, ensures consent, and clears myths, while paving the way for faster, cheaper internet across Pakistan! 🇵🇰 Facts over propaganda. Always! 🙌🏼

Saad Kaiser 🇵🇰

19,069 görüntüleme • 3 ay önce

Run Stable Audio 3 on ANY Computer with NO VRAM 1-click launcher for the official Stable Audio 3 gradio app. 1. Cross Platform: Works on ALL OS (Mac, Linux, Windows) out of the box 2. No GPU Needed: Even runs on CPU. 3. Ultra Fast: Generate a 2 minute clip in split seconds

Run Stable Audio 3 on ANY Computer with NO VRAM 1-click launcher for the official Stable Audio 3 gradio app. 1. Cross Platform: Works on ALL OS (Mac, Linux, Windows) out of the box 2. No GPU Needed: Even runs on CPU. 3. Ultra Fast: Generate a 2 minute clip in split seconds

cocktail peanut

72,933 görüntüleme • 4 ay önce

PHALA PHALA SAGA | uMkhonto weSizwe Party MP Mzwanele Manyi says he expected the process paving the way for impeachment proceedings to move faster, but is now frustrated by the legal advice being given to the Subcommittee on the Review of Assembly Rules.

PHALA PHALA SAGA | uMkhonto weSizwe Party MP Mzwanele Manyi says he expected the process paving the way for impeachment proceedings to move faster, but is now frustrated by the legal advice being given to the Subcommittee on the Review of Assembly Rules.

SABC News

20,525 görüntüleme • 4 ay önce

Paving the way for a transformed and seamless freight journey with the Rail-Over-Rail Flyover at the New Bhaupur - New Pt. Deen Dayal Upadhyaya Section on Eastern Dedicated Freight Corridor.

Paving the way for a transformed and seamless freight journey with the Rail-Over-Rail Flyover at the New Bhaupur - New Pt. Deen Dayal Upadhyaya Section on Eastern Dedicated Freight Corridor.

Ministry of Railways

38,262 görüntüleme • 1 yıl önce

Building on the proven success of Stretch for automated container unloading, DHL Group is paving the way for the global deployment of more than 1,000 additional robots. Learn how DHL is accelerating their automation strategy:

Building on the proven success of Stretch for automated container unloading, DHL Group is paving the way for the global deployment of more than 1,000 additional robots. Learn how DHL is accelerating their automation strategy:

Boston Dynamics

54,125 görüntüleme • 1 yıl önce

QVAC SDK 0.15.0 is live. This release adds multiple prompts batching, brings a native AMD GPU backend to the stack, moves more vision encoders onto mobile GPUs, and adds a second local coding-agent integration. Main highlights: - Prompt batching for the LLM addon. Batch multiple prompts into one job and process them concurrently, with each answer returned the moment its generation finishes. - Native AMD GPU backend. A first-class HIP/ROCm backend in @qvac/vla-ggml, auto-selected over Vulkan with clean fallback when ROCm is absent. - A second local coding agent. OpenClaw joins OpenCode for local, cloud-free agent workflows. AGENTS - OpenCode plugin update (@qvac/opencode-plugin). Aligned with the current SDK, CLI, and AI SDK provider packages. A fresh install runs OpenCode against managed local QVAC models out of the box, from the default qvac/qwen3.5-9b, with no manual qvac serve setup. - OpenClaw plugin (@qvac/openclaw-plugin). A second coding-agent integration alongside OpenCode. A fresh setup installs the plugin, creates a local qvac provider through onboarding, and runs a QVAC model through OpenClaw🦞's local service path. LANGUAGE MODELS - Prompt batching (LLM addon). Batch multiple prompts in one job and run them concurrently, each answer returns the moment its generation finishes, no waiting on the others. - Reasoning-context trimming on hybrid + recurrent models (@qvac/llm-llamacpp). remove_thinking_from_context now works beyond pure-attention models. Same JS API, no throw. VOICE AND SPEECH - Transcription (transcription-parakeet 0.9.0). More robust CPU fallback on GPU failure and a faster Vulkan backend on Pixel 9. - Text-to-speech features (tts-ggml 0.4.0). Adds LavaSR for noise removal and adjustable output frequency up to 48 kHz, plus Japanese via Chatterbox. - Text-to-speech fixes (tts-ggml 0.4.1). CPU fallback on GPU failure, a q8_0 KV crash fix on Metal with Chatterbox. VISION - Qwen3.5 vision encoder on GPU (Android). Image encoder moves onto the phone GPU, with a smarter tile-grid preprocessor and default image-token caps, for flagship Android: Vulkan on Mali (Pixel 9 Pro) and OpenCL on Adreno 830 (Galaxy S25). - Gemma-4 vision encoder on GPU (Android). Vision encoder runs on the phone GPU instead of CPU, same flagship Android targets. PLATFORM AND PERFORMANCE - AMD GPU backend (@qvac/vla-ggml). Native HIP/ROCm backend, auto-selected over Vulkan with clean fallback when ROCm is absent (Linux x64 only). Comes with ~23% faster than Vulkan, ~14% faster than PyTorch-ROCm, parity preserved. Unified code style. A cleaner, more consistent, easier-to-contribute codebase. Let's build. npm install @qvac/sdk

QVAC SDK 0.15.0 is live. This release adds multiple prompts batching, brings a native AMD GPU backend to the stack, moves more vision encoders onto mobile GPUs, and adds a second local coding-agent integration. Main highlights: - Prompt batching for the LLM addon. Batch multiple prompts into one job and process them concurrently, with each answer returned the moment its generation finishes. - Native AMD GPU backend. A first-class HIP/ROCm backend in @qvac/vla-ggml, auto-selected over Vulkan with clean fallback when ROCm is absent. - A second local coding agent. OpenClaw joins OpenCode for local, cloud-free agent workflows. AGENTS - OpenCode plugin update (@qvac/opencode-plugin). Aligned with the current SDK, CLI, and AI SDK provider packages. A fresh install runs OpenCode against managed local QVAC models out of the box, from the default qvac/qwen3.5-9b, with no manual qvac serve setup. - OpenClaw plugin (@qvac/openclaw-plugin). A second coding-agent integration alongside OpenCode. A fresh setup installs the plugin, creates a local qvac provider through onboarding, and runs a QVAC model through OpenClaw🦞's local service path. LANGUAGE MODELS - Prompt batching (LLM addon). Batch multiple prompts in one job and run them concurrently, each answer returns the moment its generation finishes, no waiting on the others. - Reasoning-context trimming on hybrid + recurrent models (@qvac/llm-llamacpp). remove_thinking_from_context now works beyond pure-attention models. Same JS API, no throw. VOICE AND SPEECH - Transcription (transcription-parakeet 0.9.0). More robust CPU fallback on GPU failure and a faster Vulkan backend on Pixel 9. - Text-to-speech features (tts-ggml 0.4.0). Adds LavaSR for noise removal and adjustable output frequency up to 48 kHz, plus Japanese via Chatterbox. - Text-to-speech fixes (tts-ggml 0.4.1). CPU fallback on GPU failure, a q8_0 KV crash fix on Metal with Chatterbox. VISION - Qwen3.5 vision encoder on GPU (Android). Image encoder moves onto the phone GPU, with a smarter tile-grid preprocessor and default image-token caps, for flagship Android: Vulkan on Mali (Pixel 9 Pro) and OpenCL on Adreno 830 (Galaxy S25). - Gemma-4 vision encoder on GPU (Android). Vision encoder runs on the phone GPU instead of CPU, same flagship Android targets. PLATFORM AND PERFORMANCE - AMD GPU backend (@qvac/vla-ggml). Native HIP/ROCm backend, auto-selected over Vulkan with clean fallback when ROCm is absent (Linux x64 only). Comes with ~23% faster than Vulkan, ~14% faster than PyTorch-ROCm, parity preserved. Unified code style. A cleaner, more consistent, easier-to-contribute codebase. Let's build. npm install @qvac/sdk

QVAC

29,260,217 görüntüleme • 3 ay önce

locate-anything.cpp: native C++/ggml (ggml) inference for NVIDIA's LocateAnything-3B, open-vocabulary object detection / visual grounding, one of the neat detection VLMs out there, from the LocalAI team. Same detections as the official model, now running anywhere with no Python and faster, on CPU and GPU. Quantized GGUF on Hugging Face 🤗 Huge thanks to Georgi Gerganov for ggml and to NVIDIAAI for releasing LocateAnything! 🧵

locate-anything.cpp: native C++/ggml (ggml) inference for NVIDIA's LocateAnything-3B, open-vocabulary object detection / visual grounding, one of the neat detection VLMs out there, from the LocalAI team. Same detections as the official model, now running anywhere with no Python and faster, on CPU and GPU. Quantized GGUF on Hugging Face 🤗 Huge thanks to Georgi Gerganov for ggml and to NVIDIAAI for releasing LocateAnything! 🧵

Ettore Di Giacinto

16,770 görüntüleme • 4 ay önce

We just killed Exa. Introducing the FASTEST search engine for agents on Earth. Your agent can now search the web 6× faster than Exa. Exa: 372ms P50 Us: 62ms P50 Made possible by OctenAI × Monid.

We just killed Exa. Introducing the FASTEST search engine for agents on Earth. Your agent can now search the web 6× faster than Exa. Exa: 372ms P50 Us: 62ms P50 Made possible by OctenAI × Monid.

Shengkun

278,890 görüntüleme • 2 ay önce