Loading video...

Video Failed to Load

There was a problem loading this video. This could be due to a temporary network issue or the video might be unavailable.

Paving the way for unlimited context windows. Introducing ZML /attnd: sparse logarithmic attention, on CPU, over UDP, faster than GPU.

Steeve Morin

9,181 subscribers

122,107 views • 11 months ago •via X (Twitter)

Science & Technology News & Politics Education

Anya Rossi• Live Now

Private livecam show

0 Comments

No comments available

Comments from the original post will appear here

Related Videos

Introducing Ziren (formerly zkMIPS) The most performant general-purpose zkVM on CPU, now scaled with GPU, distributed proving, and more. Build with confidence. Deploy without compromise 🟠 Full context: Technical updates:

Introducing Ziren (formerly zkMIPS) The most performant general-purpose zkVM on CPU, now scaled with GPU, distributed proving, and more. Build with confidence. Deploy without compromise 🟠 Full context: Technical updates:

ZKM

5,857,968 views • 1 year ago

$Introducing SubQ - a major breakthrough in LLM intelligence. It is the first model built on a fully sub-quadratic sparse-attention architecture (SSA), And the first frontier model with a 12 million token context window which is: - 52x faster than FlashAttention at 1MM tokens - Less than 5% the cost of Opus Transformer-based LLMs waste compute by processing every possible relationship between words (standard attention). Only a small fraction actually matter. Subquadratic finds and focuses only on the ones that do. That's nearly 1,000x less compute and a new way for LLMs to scale.$

Introducing SubQ - a major breakthrough in LLM intelligence. It is the first model built on a fully sub-quadratic sparse-attention architecture (SSA), And the first frontier model with a 12 million token context window which is: - 52x faster than FlashAttention at 1MM tokens - Less than 5% the cost of Opus Transformer-based LLMs waste compute by processing every possible relationship between words (standard attention). Only a small fraction actually matter. Subquadratic finds and focuses only on the ones that do. That's nearly 1,000x less compute and a new way for LLMs to scale.

Alexander Whedon

13,474,896 views • 3 months ago

$INFQ GPU + CPU + QPU Frontier models are hitting saturation on tokens (data volume) and context windows (memory). $INFQ CTO walks through how they and $NVDA are working together to add a Quantum Processing Unit to the chain as the next inflection point.

$INFQ GPU + CPU + QPU Frontier models are hitting saturation on tokens (data volume) and context windows (memory). $INFQ CTO walks through how they and $NVDA are working together to add a Quantum Processing Unit to the chain as the next inflection point.

YeahDave

57,181 views • 3 months ago

Windows ML is now generally available! This milestone brings powerful, local AI capability to every Windows 11 device, harnessing the full power of the GPU, NPU, and CPU. Big thank you to the team and our silicon partners, AMD, Intel, NVIDIA, and Qualcomm, for their collaboration to unlock real-time, secure, and efficient AI experiences for developers on Windows. Let’s go 🚀!

Windows ML is now generally available! This milestone brings powerful, local AI capability to every Windows 11 device, harnessing the full power of the GPU, NPU, and CPU. Big thank you to the team and our silicon partners, AMD, Intel, NVIDIA, and Qualcomm, for their collaboration to unlock real-time, secure, and efficient AI experiences for developers on Windows. Let’s go 🚀!

Pavan Davuluri

11,663 views • 11 months ago

PowerInfer - a high-speed inference engine for deploying LLMs locally. Just came across this super interesting project on speeding up inference. It's not MoE but it's a simple approach that exploits the high locality in LLM inference to design a GPU-CPU hybrid inference engine. Hot-activated neurons are preloaded onto the GPU for fast access, while cold-activated neurons (the majority) are computed on the CPU. This approach significantly reduces GPU memory demands and CPU-GPU data transfer. It achieves an average token generation rate of 13.20 tokens/s, with a peak of 29.08 tokens/s, across various LLMs on a single NVIDIA RTX 4090 GPU. It's on only 18% lower than that achieved by a top-tier server-grade A100 GPU. It also significantly outperforms llama.cpp by up to 11.69x while retaining model accuracy. There is a lot more innovation around inference that's coming fast. Really encouraged by the study on sparse computation to enhance the computational efficiency of LLMs. It's now possible to use PowerInfer with Llama 2 and Faclon 40B. Mistral-7B support is coming soon!

PowerInfer - a high-speed inference engine for deploying LLMs locally. Just came across this super interesting project on speeding up inference. It's not MoE but it's a simple approach that exploits the high locality in LLM inference to design a GPU-CPU hybrid inference engine. Hot-activated neurons are preloaded onto the GPU for fast access, while cold-activated neurons (the majority) are computed on the CPU. This approach significantly reduces GPU memory demands and CPU-GPU data transfer. It achieves an average token generation rate of 13.20 tokens/s, with a peak of 29.08 tokens/s, across various LLMs on a single NVIDIA RTX 4090 GPU. It's on only 18% lower than that achieved by a top-tier server-grade A100 GPU. It also significantly outperforms llama.cpp by up to 11.69x while retaining model accuracy. There is a lot more innovation around inference that's coming fast. Really encouraged by the study on sparse computation to enhance the computational efficiency of LLMs. It's now possible to use PowerInfer with Llama 2 and Faclon 40B. Mistral-7B support is coming soon!

elvis

261,622 views • 2 years ago

Now introducing Replit Desktop, a seamless way to multitask and get the best of Replit right on your Desktop. Supported for both Windows and Mac.

Now introducing Replit Desktop, a seamless way to multitask and get the best of Replit right on your Desktop. Supported for both Windows and Mac.

Replit ⠕

82,858 views • 1 month ago

“Greater Manchester productivity rose over three times faster than London.” Great interview by Lord Jim O’Neil on #R4Today. GM is leading the way to growth for the next Government. 👍🏻

“Greater Manchester productivity rose over three times faster than London.” Great interview by Lord Jim O’Neil on #R4Today. GM is leading the way to growth for the next Government. 👍🏻

Andy Burnham

34,677 views • 2 years ago

Today we release LFM2.5-Encoder-230M and LFM2.5-Encoder-350M: bidirectional encoders that stay fast at long context, even on CPU. > LFM2.5-Encoder-230M: about 3.7x faster than ModernBERT-base on CPU at 8,192 tokens. Under 30s per forward pass, versus over a minute and a half. > LFM2.5-Encoder-350M: 4th of 14 models on GLUE, SuperGLUE, and multilingual classification, behind only three larger models, one of them nearly 10x its size. 🧵

Today we release LFM2.5-Encoder-230M and LFM2.5-Encoder-350M: bidirectional encoders that stay fast at long context, even on CPU. > LFM2.5-Encoder-230M: about 3.7x faster than ModernBERT-base on CPU at 8,192 tokens. Under 30s per forward pass, versus over a minute and a half. > LFM2.5-Encoder-350M: 4th of 14 models on GLUE, SuperGLUE, and multilingual classification, behind only three larger models, one of them nearly 10x its size. 🧵

Liquid AI

198,229 views • 28 days ago

Run Stable Audio 3 on ANY Computer with NO VRAM 1-click launcher for the official Stable Audio 3 gradio app. 1. Cross Platform: Works on ALL OS (Mac, Linux, Windows) out of the box 2. No GPU Needed: Even runs on CPU. 3. Ultra Fast: Generate a 2 minute clip in split seconds

Run Stable Audio 3 on ANY Computer with NO VRAM 1-click launcher for the official Stable Audio 3 gradio app. 1. Cross Platform: Works on ALL OS (Mac, Linux, Windows) out of the box 2. No GPU Needed: Even runs on CPU. 3. Ultra Fast: Generate a 2 minute clip in split seconds

cocktail peanut

72,933 views • 3 months ago

Introducing HydraDB. The graph native context infrastructure for agents. Purpose built to deliver precise context & observability into why agents act the way they do. We've always believed graphs are the best way to manage AI context, but they've been too expensive to scale or impractical for storing full context. Until now. HydraDB combines in memory, NVMe, and object storage into a single graph layer, making context delivery faster, cheaper, and more precise. We want context delivery to be extremely fast, 1000x cheap, and highly precise. Give your agents a brain.

Introducing HydraDB. The graph native context infrastructure for agents. Purpose built to deliver precise context & observability into why agents act the way they do. We've always believed graphs are the best way to manage AI context, but they've been too expensive to scale or impractical for storing full context. Until now. HydraDB combines in memory, NVMe, and object storage into a single graph layer, making context delivery faster, cheaper, and more precise. We want context delivery to be extremely fast, 1000x cheap, and highly precise. Give your agents a brain.

Nishkarsh

2,299,606 views • 2 months ago

No forced towers. No violation of rights. Myth vs Reality. Telecommunications Amendment Bill protects private property, ensures consent, and clears myths, while paving the way for faster, cheaper internet across Pakistan! 🇵🇰 Facts over propaganda. Always! 🙌🏼

No forced towers. No violation of rights. Myth vs Reality. Telecommunications Amendment Bill protects private property, ensures consent, and clears myths, while paving the way for faster, cheaper internet across Pakistan! 🇵🇰 Facts over propaganda. Always! 🙌🏼

Saad Kaiser 🇵🇰

19,040 views • 2 months ago

PHALA PHALA SAGA | uMkhonto weSizwe Party MP Mzwanele Manyi says he expected the process paving the way for impeachment proceedings to move faster, but is now frustrated by the legal advice being given to the Subcommittee on the Review of Assembly Rules.

PHALA PHALA SAGA | uMkhonto weSizwe Party MP Mzwanele Manyi says he expected the process paving the way for impeachment proceedings to move faster, but is now frustrated by the legal advice being given to the Subcommittee on the Review of Assembly Rules.

SABC News

20,525 views • 3 months ago

Paving the way for a transformed and seamless freight journey with the Rail-Over-Rail Flyover at the New Bhaupur - New Pt. Deen Dayal Upadhyaya Section on Eastern Dedicated Freight Corridor.

Paving the way for a transformed and seamless freight journey with the Rail-Over-Rail Flyover at the New Bhaupur - New Pt. Deen Dayal Upadhyaya Section on Eastern Dedicated Freight Corridor.

Ministry of Railways

38,262 views • 1 year ago

Building on the proven success of Stretch for automated container unloading, DHL Group is paving the way for the global deployment of more than 1,000 additional robots. Learn how DHL is accelerating their automation strategy:

Building on the proven success of Stretch for automated container unloading, DHL Group is paving the way for the global deployment of more than 1,000 additional robots. Learn how DHL is accelerating their automation strategy:

Boston Dynamics

53,995 views • 1 year ago

QVAC SDK 0.15.0 is live. This release adds multiple prompts batching, brings a native AMD GPU backend to the stack, moves more vision encoders onto mobile GPUs, and adds a second local coding-agent integration. Main highlights: - Prompt batching for the LLM addon. Batch multiple prompts into one job and process them concurrently, with each answer returned the moment its generation finishes. - Native AMD GPU backend. A first-class HIP/ROCm backend in @qvac/vla-ggml, auto-selected over Vulkan with clean fallback when ROCm is absent. - A second local coding agent. OpenClaw joins OpenCode for local, cloud-free agent workflows. AGENTS - OpenCode plugin update (@qvac/opencode-plugin). Aligned with the current SDK, CLI, and AI SDK provider packages. A fresh install runs OpenCode against managed local QVAC models out of the box, from the default qvac/qwen3.5-9b, with no manual qvac serve setup. - OpenClaw plugin (@qvac/openclaw-plugin). A second coding-agent integration alongside OpenCode. A fresh setup installs the plugin, creates a local qvac provider through onboarding, and runs a QVAC model through OpenClaw🦞's local service path. LANGUAGE MODELS - Prompt batching (LLM addon). Batch multiple prompts in one job and run them concurrently, each answer returns the moment its generation finishes, no waiting on the others. - Reasoning-context trimming on hybrid + recurrent models (@qvac/llm-llamacpp). remove_thinking_from_context now works beyond pure-attention models. Same JS API, no throw. VOICE AND SPEECH - Transcription (transcription-parakeet 0.9.0). More robust CPU fallback on GPU failure and a faster Vulkan backend on Pixel 9. - Text-to-speech features (tts-ggml 0.4.0). Adds LavaSR for noise removal and adjustable output frequency up to 48 kHz, plus Japanese via Chatterbox. - Text-to-speech fixes (tts-ggml 0.4.1). CPU fallback on GPU failure, a q8_0 KV crash fix on Metal with Chatterbox. VISION - Qwen3.5 vision encoder on GPU (Android). Image encoder moves onto the phone GPU, with a smarter tile-grid preprocessor and default image-token caps, for flagship Android: Vulkan on Mali (Pixel 9 Pro) and OpenCL on Adreno 830 (Galaxy S25). - Gemma-4 vision encoder on GPU (Android). Vision encoder runs on the phone GPU instead of CPU, same flagship Android targets. PLATFORM AND PERFORMANCE - AMD GPU backend (@qvac/vla-ggml). Native HIP/ROCm backend, auto-selected over Vulkan with clean fallback when ROCm is absent (Linux x64 only). Comes with ~23% faster than Vulkan, ~14% faster than PyTorch-ROCm, parity preserved. Unified code style. A cleaner, more consistent, easier-to-contribute codebase. Let's build. npm install @qvac/sdk

QVAC SDK 0.15.0 is live. This release adds multiple prompts batching, brings a native AMD GPU backend to the stack, moves more vision encoders onto mobile GPUs, and adds a second local coding-agent integration. Main highlights: - Prompt batching for the LLM addon. Batch multiple prompts into one job and process them concurrently, with each answer returned the moment its generation finishes. - Native AMD GPU backend. A first-class HIP/ROCm backend in @qvac/vla-ggml, auto-selected over Vulkan with clean fallback when ROCm is absent. - A second local coding agent. OpenClaw joins OpenCode for local, cloud-free agent workflows. AGENTS - OpenCode plugin update (@qvac/opencode-plugin). Aligned with the current SDK, CLI, and AI SDK provider packages. A fresh install runs OpenCode against managed local QVAC models out of the box, from the default qvac/qwen3.5-9b, with no manual qvac serve setup. - OpenClaw plugin (@qvac/openclaw-plugin). A second coding-agent integration alongside OpenCode. A fresh setup installs the plugin, creates a local qvac provider through onboarding, and runs a QVAC model through OpenClaw🦞's local service path. LANGUAGE MODELS - Prompt batching (LLM addon). Batch multiple prompts in one job and run them concurrently, each answer returns the moment its generation finishes, no waiting on the others. - Reasoning-context trimming on hybrid + recurrent models (@qvac/llm-llamacpp). remove_thinking_from_context now works beyond pure-attention models. Same JS API, no throw. VOICE AND SPEECH - Transcription (transcription-parakeet 0.9.0). More robust CPU fallback on GPU failure and a faster Vulkan backend on Pixel 9. - Text-to-speech features (tts-ggml 0.4.0). Adds LavaSR for noise removal and adjustable output frequency up to 48 kHz, plus Japanese via Chatterbox. - Text-to-speech fixes (tts-ggml 0.4.1). CPU fallback on GPU failure, a q8_0 KV crash fix on Metal with Chatterbox. VISION - Qwen3.5 vision encoder on GPU (Android). Image encoder moves onto the phone GPU, with a smarter tile-grid preprocessor and default image-token caps, for flagship Android: Vulkan on Mali (Pixel 9 Pro) and OpenCL on Adreno 830 (Galaxy S25). - Gemma-4 vision encoder on GPU (Android). Vision encoder runs on the phone GPU instead of CPU, same flagship Android targets. PLATFORM AND PERFORMANCE - AMD GPU backend (@qvac/vla-ggml). Native HIP/ROCm backend, auto-selected over Vulkan with clean fallback when ROCm is absent (Linux x64 only). Comes with ~23% faster than Vulkan, ~14% faster than PyTorch-ROCm, parity preserved. Unified code style. A cleaner, more consistent, easier-to-contribute codebase. Let's build. npm install @qvac/sdk

QVAC

29,260,217 views • 1 month ago

locate-anything.cpp: native C++/ggml (ggml) inference for NVIDIA's LocateAnything-3B, open-vocabulary object detection / visual grounding, one of the neat detection VLMs out there, from the LocalAI team. Same detections as the official model, now running anywhere with no Python and faster, on CPU and GPU. Quantized GGUF on Hugging Face 🤗 Huge thanks to Georgi Gerganov for ggml and to NVIDIAAI for releasing LocateAnything! 🧵

locate-anything.cpp: native C++/ggml (ggml) inference for NVIDIA's LocateAnything-3B, open-vocabulary object detection / visual grounding, one of the neat detection VLMs out there, from the LocalAI team. Same detections as the official model, now running anywhere with no Python and faster, on CPU and GPU. Quantized GGUF on Hugging Face 🤗 Huge thanks to Georgi Gerganov for ggml and to NVIDIAAI for releasing LocateAnything! 🧵

Ettore Di Giacinto

15,919 views • 2 months ago

We just killed Exa. Introducing the FASTEST search engine for agents on Earth. Your agent can now search the web 6× faster than Exa. Exa: 372ms P50 Us: 62ms P50 Made possible by OctenAI × Monid.

We just killed Exa. Introducing the FASTEST search engine for agents on Earth. Your agent can now search the web 6× faster than Exa. Exa: 372ms P50 Us: 62ms P50 Made possible by OctenAI × Monid.

Shengkun

278,890 views • 29 days ago