Video yükleniyor...

Video Yüklenemedi

Bu video yüklenirken bir sorun oluştu. Bu geçici bir ağ sorunundan kaynaklanıyor olabilir veya video kullanılamıyor olabilir.

Ana Sayfaya Dön

(1/5) FP4 hardware is here, but 4-bit attention still kills model quality, blocking true end-to-end FP4 serving. To fix that, we propose Attn-QAT, the first systematic study of quantization-aware training for attention. The result: FP4 attention quality is comparable to BF16 attention with 1.1x–1.5x higher throughput than SageAttention3 on... show more

Hao AI Lab

8,253 subscribers

38,252 görüntüleme • 6 ay önce •via X (Twitter)

Anya Rossi• Live Now

Private livecam show

12 Yorum

Hao AI Lab profil fotoğrafı

Hao AI Lab6 ay önce

(2/5)Naive QAT breaks when applied to FlashAttention kernels. We found two fixes are needed: 1. Store a small high-precision auxiliary output so the gradient computation stays mathematically consistent 2. Recompute attention probabilities in the backward pass using the same low precision as the forward pass These two changes stabilize 4-bit attention training.

Hao AI Lab profil fotoğrafı

Hao AI Lab6 ay önce

(3/5) Across both video diffusion models and language models, Attn-QAT recovers the quality drop of 4-bit attention without the extra outlier-mitigation heuristics. For continued pretraining, Attn-QAT recovers most of the quality loss caused by FP4 attention on Qwen3-14B and partially recovers it on Llama 3.1-70B. For supervised fine-tuning, Attn-QAT can be used as a drop-in replacement for BF16 attention. On Qwen3-14B, it achieves nearly identical downstream benchmark performance to BF16 attention. On Llama 3.1-70B, it remains close with a small gap. For randomly-selected example videos (generated by Wan-2.1-14B), we see that with Attn-QAT, FP4 attention produces videos comparable to BF16 attention, whereas SageAttention3 produces videos with artifacts.

Hao AI Lab profil fotoğrafı

Hao AI Lab6 ay önce

(4/5) Because the model learns to account for quantization error during QAT, inference needs no extra heuristics! No Q/K smoothing, no two-level quantization. Simpler kernel → faster inference compared to SageAttention3 on an RTX 5090.

Hao AI Lab profil fotoğrafı

Hao AI Lab6 ay önce

(5/5) On a B200, Blackwell's tensor cores are so fast that the softmax now becomes a bottleneck in addition to the GEMMs. Quantizing PV adds scale-factor overhead that piles onto the softmax warps. So we run NVFP4 QK + BF16 PV, with a careful TMEM overlap schedule to fit scale factors into an already-full pipeline. Result: 1801 TFLOPS and up to 1.39x over FlashAttention-4. 2x/4x faster exp on B300/Rubin should push this further, and end-to-end FP4 serving, once blocked by attention quality, is now within reach.

Lee Penkman profil fotoğrafı

Lee Penkman6 ay önce

dang i see this on PRs a lot :D tricky now theres all these agents to control what they are all doing, maybe they should set a githook or smth @grok whats best way to prevent this

Vipul kumar profil fotoğrafı

Vipul kumar6 ay önce

@gork is this true man

Vipul kumar profil fotoğrafı

Vipul kumar6 ay önce

@grok hi

vik profil fotoğrafı

vik6 ay önce

nice work

David Güera profil fotoğrafı

David Güera6 ay önce

Great work!

Cliff Lattner profil fotoğrafı

Cliff Lattner6 ay önce

@ye_combinator Did you try FP8? It should be the same performance uplift as FP4

homuraakemifan profil fotoğrafı

homuraakemifan6 ay önce

WOW!

Lee Penkman profil fotoğrafı

Lee Penkman6 ay önce

friends from @fal might like this :)

Benzer Videolar

🚀 Excited to release LongLive 2.0! 🎬 An end-to-end infrastructure for long video generation, with FP4 and parallelism at the core of both training and inference. ⚡45.7 FPS generation speed on 5B model⚡ ✨ LongLive 2.0 supports real-video training, few-step distillation, multi-shot training/inference, sequence-parallel acceleration, NVFP4 KV cache, and async VAE decoding deployment. 🧩 To our knowledge, this is the first open-source 4-bit long video generation infra that covers both training and inference. 🙌 Welcome to check it out, try it, and share feedback! 🔗 Code: 📰 Paper: 🎥 Demo: #LongVideoGeneration #VideoGeneration #Realtime #AIInfra #EfficientAI #FP4 #Parallel #NVIDIA

🚀 Excited to release LongLive 2.0! 🎬 An end-to-end infrastructure for long video generation, with FP4 and parallelism at the core of both training and inference. ⚡45.7 FPS generation speed on 5B model⚡ ✨ LongLive 2.0 supports real-video training, few-step distillation, multi-shot training/inference, sequence-parallel acceleration, NVFP4 KV cache, and async VAE decoding deployment. 🧩 To our knowledge, this is the first open-source 4-bit long video generation infra that covers both training and inference. 🙌 Welcome to check it out, try it, and share feedback! 🔗 Code: 📰 Paper: 🎥 Demo: #LongVideoGeneration #VideoGeneration #Realtime #AIInfra #EfficientAI #FP4 #Parallel #NVIDIA

Yukang Chen

59,918 görüntüleme • 4 ay önce

Open-source video generation is now faster than playback without compromising quality. Introducing Video Delta Net (VDN): hybrid attention for live text-to-video with near-lossless quality. VDN accelerates Minimax-H3 by 75 - 90 x, generating 14 seconds of 768p video in 11 seconds on 8× NVIDIA B200 GPUs. Checkpoints + training/inference code + Technical Blog ⬇️ (1/6)

Open-source video generation is now faster than playback without compromising quality. Introducing Video Delta Net (VDN): hybrid attention for live text-to-video with near-lossless quality. VDN accelerates Minimax-H3 by 75 - 90 x, generating 14 seconds of 768p video in 11 seconds on 8× NVIDIA B200 GPUs. Checkpoints + training/inference code + Technical Blog ⬇️ (1/6)

Haocheng Xi 🚗 COLM

442,506 görüntüleme • 1 ay önce

$Introducing SubQ - a major breakthrough in LLM intelligence. It is the first model built on a fully sub-quadratic sparse-attention architecture (SSA), And the first frontier model with a 12 million token context window which is: - 52x faster than FlashAttention at 1MM tokens - Less than 5% the cost of Opus Transformer-based LLMs waste compute by processing every possible relationship between words (standard attention). Only a small fraction actually matter. Subquadratic finds and focuses only on the ones that do. That's nearly 1,000x less compute and a new way for LLMs to scale.$

Introducing SubQ - a major breakthrough in LLM intelligence. It is the first model built on a fully sub-quadratic sparse-attention architecture (SSA), And the first frontier model with a 12 million token context window which is: - 52x faster than FlashAttention at 1MM tokens - Less than 5% the cost of Opus Transformer-based LLMs waste compute by processing every possible relationship between words (standard attention). Only a small fraction actually matter. Subquadratic finds and focuses only on the ones that do. That's nearly 1,000x less compute and a new way for LLMs to scale.

Alexander Whedon

13,490,366 görüntüleme • 5 ay önce

"When it comes to the jobs we create, we don’t just focus on the quantity, we pay more attention to the quality. So much so that the quality of a single job I create for one person is worth more than 50 jobs in other places." — Prof. Isa Ali Ibrahim, CON

"When it comes to the jobs we create, we don’t just focus on the quantity, we pay more attention to the quality. So much so that the quality of a single job I create for one person is worth more than 50 jobs in other places." — Prof. Isa Ali Ibrahim, CON

B_SARKY

18,893 görüntüleme • 4 ay önce

New short course: Attention in Transformers: Concepts and Code in PyTorch. Last week we released a course on how LLM transformers work. This week, go deeper and learn about the technical ideas behind the attention mechanism, and see how to code it in PyTorch. This course is built with Joshua Starmer, Founder and CEO of StatQuest. The attention mechanism was a breakthrough that led to transformers, the architecture powering large language models like ChatGPT. Transformers, introduced in the 2017 paper: "Attention is All You Need" by Viswani and others, took off because of its highly scalable design. In this course, you’ll learn how the attention mechanism, a key element of transformer-based LLMs, works and implement it in PyTorch. You'll develop deep intuition about building reliable, functional, and scalable AI applications. What you will do: - Understand the evolution of the attention mechanism, a key breakthrough that led to transformers. - Learn the relationships between word embeddings, positional embeddings, and attention. - Learn about the Query, Key, and Value matrices, and how to produce and use them in attention. - Walk through the math required to calculate self-attention and masked self-attention to learn why and how they work. - Understand the difference between self-attention and masked self-attention and how one is used in the encoder to build context-aware embeddings and the other is used in the decoder for generative outputs. - Learn the details of the encoder-decoder architecture, cross-attention, and multi-head attention and how they are all incorporated into a transformer. - Use PyTorch to code a class that implements self-attention, masked self-attention, and multi-head attention. There're lots of exciting technical details in this course. Please sign up here:

New short course: Attention in Transformers: Concepts and Code in PyTorch. Last week we released a course on how LLM transformers work. This week, go deeper and learn about the technical ideas behind the attention mechanism, and see how to code it in PyTorch. This course is built with Joshua Starmer, Founder and CEO of StatQuest. The attention mechanism was a breakthrough that led to transformers, the architecture powering large language models like ChatGPT. Transformers, introduced in the 2017 paper: "Attention is All You Need" by Viswani and others, took off because of its highly scalable design. In this course, you’ll learn how the attention mechanism, a key element of transformer-based LLMs, works and implement it in PyTorch. You'll develop deep intuition about building reliable, functional, and scalable AI applications. What you will do: - Understand the evolution of the attention mechanism, a key breakthrough that led to transformers. - Learn the relationships between word embeddings, positional embeddings, and attention. - Learn about the Query, Key, and Value matrices, and how to produce and use them in attention. - Walk through the math required to calculate self-attention and masked self-attention to learn why and how they work. - Understand the difference between self-attention and masked self-attention and how one is used in the encoder to build context-aware embeddings and the other is used in the decoder for generative outputs. - Learn the details of the encoder-decoder architecture, cross-attention, and multi-head attention and how they are all incorporated into a transformer. - Use PyTorch to code a class that implements self-attention, masked self-attention, and multi-head attention. There're lots of exciting technical details in this course. Please sign up here:

Andrew Ng

132,544 görüntüleme • 1 yıl önce

Why $SCHIFFY represents a new model for Web3 gaming... The idea is to turn attention into something with real underlying value instead of relying on an inflationary token that users have to believe is money The key shift: • Attention converts into RWA-backed assets • Value comes from the underlying stock exposure • The main challenge becomes sustaining attention and volume • Less reliance on inflationary "Ponzonomics" "The problem is sort of just moved, but I think it's a more tractable problem."

Why $SCHIFFY represents a new model for Web3 gaming... The idea is to turn attention into something with real underlying value instead of relying on an inflationary token that users have to believe is money The key shift: • Attention converts into RWA-backed assets • Value comes from the underlying stock exposure • The main challenge becomes sustaining attention and volume • Less reliance on inflationary "Ponzonomics" "The problem is sort of just moved, but I think it's a more tractable problem."

MCG

15,816 görüntüleme • 11 gün önce

.Glinert 🇺🇸 🏭 (Co-Founder & CEO of Sphere Semi) on how they're using AI to change how chips are designed: “CPUs and GPUs get all the attention, but analog is a third of the trillion-dollar chip industry and it’s critical for communications, warfare, and more.” “We think human chip design is coming to an end. In analog, it’s still done by hand and we’re putting that to an end.”

.Glinert 🇺🇸 🏭 (Co-Founder & CEO of Sphere Semi) on how they're using AI to change how chips are designed: “CPUs and GPUs get all the attention, but analog is a third of the trillion-dollar chip industry and it’s critical for communications, warfare, and more.” “We think human chip design is coming to an end. In analog, it’s still done by hand and we’re putting that to an end.”

TBPN

58,868 görüntüleme • 1 yıl önce

Zhilin at GTC: Introducing Attention Residuals Learning selective memory, rather than mechanically accumulating everything, is the beauty of attention. Many of you have probably read Attention Is All You Need, the 2017 Transformer paper that brought “human-like” attention into the model’s field of view. From that point on, models no longer simply read everything in a mechanical way. Instead, they began to develop a sense of what matters more and what matters less across the text, choosing to retain the more important information. Recently, Kimi applied this idea of attention to the temporal dimension, then rotated it 90 degrees into the model’s depth dimension. This allows the model to have attention not only over time, but also throughout the process of information transmission across layers—giving it a more intelligent way to understand and process information.

Zhilin at GTC: Introducing Attention Residuals Learning selective memory, rather than mechanically accumulating everything, is the beauty of attention. Many of you have probably read Attention Is All You Need, the 2017 Transformer paper that brought “human-like” attention into the model’s field of view. From that point on, models no longer simply read everything in a mechanical way. Instead, they began to develop a sense of what matters more and what matters less across the text, choosing to retain the more important information. Recently, Kimi applied this idea of attention to the temporal dimension, then rotated it 90 degrees into the model’s depth dimension. This allows the model to have attention not only over time, but also throughout the process of information transmission across layers—giving it a more intelligent way to understand and process information.

Kimi.ai

117,333 görüntüleme • 6 ay önce

FOUR 600W GPUS. 384GB OF GDDR7. AND DELL MADE THE INSTALL LOOK EASIER THAN BUILDING A GAMING PC. this is a Dell PowerEdge XE7740 receiving four NVIDIA RTX PRO 6000 Blackwell Server Edition GPUs. each card brings: 96GB GDDR7 ECC 24,064 CUDA cores 4 PFLOPS FP4 up to 600W but the most satisfying part is the cable management. every GPU power lead is already routed, labeled and held beside its assigned slot. drop in the card. lock the retention hardware. connect one 16-pin plug. no loose bundle hanging across the airflow path and no guessing which cable belongs where. the four-card build totals 384GB of GPU memory and 16 PFLOPS FP4. the 4U XE7740 supports up to eight of these GPUs, doubling that to 768GB and 32 PFLOPS FP4. honest line: the memory remains split between separate cards, and four GPUs can still pull up to 2.4kW. Dell made a rack-scale AI server this dense without turning the next GPU swap into cable surgery. bookmark & watch today ↓

FOUR 600W GPUS. 384GB OF GDDR7. AND DELL MADE THE INSTALL LOOK EASIER THAN BUILDING A GAMING PC. this is a Dell PowerEdge XE7740 receiving four NVIDIA RTX PRO 6000 Blackwell Server Edition GPUs. each card brings: 96GB GDDR7 ECC 24,064 CUDA cores 4 PFLOPS FP4 up to 600W but the most satisfying part is the cable management. every GPU power lead is already routed, labeled and held beside its assigned slot. drop in the card. lock the retention hardware. connect one 16-pin plug. no loose bundle hanging across the airflow path and no guessing which cable belongs where. the four-card build totals 384GB of GPU memory and 16 PFLOPS FP4. the 4U XE7740 supports up to eight of these GPUs, doubling that to 768GB and 32 PFLOPS FP4. honest line: the memory remains split between separate cards, and four GPUs can still pull up to 2.4kW. Dell made a rack-scale AI server this dense without turning the next GPU swap into cable surgery. bookmark & watch today ↓

Grimmer

26,693 görüntüleme • 2 ay önce

🎉 Congrats to MiniMax (official) on releasing MiniMax M3! Frontier coding and agentic capabilities, native image and video input, computer use, and a 1M-token context window, all in a single open model. At the heart of M3 is MSA, a new sparse attention architecture: instead of attending densely over the full KV cache, each query scores 128-token KV blocks and runs attention only over the top blocks. That is what makes 1M-token context practical to serve. M3 runs in vLLM with day-0 support, verified on NVIDIA and AMD hardware: ✨ MSA sparse attention with dedicated prefill and decode kernels ✨ 1M-token context serving with prefix caching and chunked prefill ✨ BF16 and MXFP8 checkpoints, with MoE backends for both Hopper and Blackwell ✨ Native multimodal input (image + video) ✨ Tool calling, reasoning parsing, and thinking-mode control for agent workloads Day-0 support like this is a true team effort. Grateful to the teams at MiniMax (official), NVIDIA AI, AI at AMD, and Inferact, and to the vLLM community for making it happen. 🙏 Deep dive into the implementation, kernel work, and deployment recipes: 🔗

🎉 Congrats to MiniMax (official) on releasing MiniMax M3! Frontier coding and agentic capabilities, native image and video input, computer use, and a 1M-token context window, all in a single open model. At the heart of M3 is MSA, a new sparse attention architecture: instead of attending densely over the full KV cache, each query scores 128-token KV blocks and runs attention only over the top blocks. That is what makes 1M-token context practical to serve. M3 runs in vLLM with day-0 support, verified on NVIDIA and AMD hardware: ✨ MSA sparse attention with dedicated prefill and decode kernels ✨ 1M-token context serving with prefix caching and chunked prefill ✨ BF16 and MXFP8 checkpoints, with MoE backends for both Hopper and Blackwell ✨ Native multimodal input (image + video) ✨ Tool calling, reasoning parsing, and thinking-mode control for agent workloads Day-0 support like this is a true team effort. Grateful to the teams at MiniMax (official), NVIDIA AI, AI at AMD, and Inferact, and to the vLLM community for making it happen. 🙏 Deep dive into the implementation, kernel work, and deployment recipes: 🔗

vLLM

40,306 görüntüleme • 3 ay önce

You need to know 4 things to run local AI on your own machine: 1) The model is the brain file. Gemma, Llama, Mistral, Qwen, Phi. 2) The warehouse is where you find the model. Hugging Face. 3) The software is what runs the model. LM Studio for the first try, Ollama when you build apps. 4) The workflow is the product you make from the other three. The size of the model decides the hardware: - 2B to 4B for phones and small tasks - 12B for the middle ground - 26B to 31B for a workstation Quantization compresses the model until it fits on a normal machine: - Q4 is easier to run - Q8 keeps more quality, but it needs more memory Q4 costs you a little quality but it saves you a $5,000 workstation. Start on the spare laptop from 2021. Open Hugging Face and read one model card slowly.

You need to know 4 things to run local AI on your own machine: 1) The model is the brain file. Gemma, Llama, Mistral, Qwen, Phi. 2) The warehouse is where you find the model. Hugging Face. 3) The software is what runs the model. LM Studio for the first try, Ollama when you build apps. 4) The workflow is the product you make from the other three. The size of the model decides the hardware: - 2B to 4B for phones and small tasks - 12B for the middle ground - 26B to 31B for a workstation Quantization compresses the model until it fits on a normal machine: - Q4 is easier to run - Q8 keeps more quality, but it needs more memory Q4 costs you a little quality but it saves you a $5,000 workstation. Start on the spare laptop from 2021. Open Hugging Face and read one model card slowly.

The Startup Ideas Podcast (SIP) 🧃

10,547 görüntüleme • 1 ay önce

.Noise video for those still code hunting. This is me shorting Opensea and Succint. Things to note: • The UX is intentionally simple, clean and fast. Gud. • You are trading attention, not token prices. Here that is represented by Kaito mindshare on crypto projects (infra and apps). • It is not an app restricted to web3 as attention is present (and desired) in all industries. If there is an accepted metric which tracks the attention of fashion brands you can put it in to an oracle and long/short seasonal trends. (are joggers coming back!?) Bonus note: The seamless flow is a look into a future where 7702 is implemented everywhere. No more approvals. While 7702 is already live on Mega that is not what is implemented here (wallet support needed first). It's a view into the future though.

.Noise video for those still code hunting. This is me shorting Opensea and Succint. Things to note: • The UX is intentionally simple, clean and fast. Gud. • You are trading attention, not token prices. Here that is represented by Kaito mindshare on crypto projects (infra and apps). • It is not an app restricted to web3 as attention is present (and desired) in all industries. If there is an accepted metric which tracks the attention of fashion brands you can put it in to an oracle and long/short seasonal trends. (are joggers coming back!?) Bonus note: The seamless flow is a look into a future where 7702 is implemented everywhere. No more approvals. While 7702 is already live on Mega that is not what is implemented here (wallet support needed first). It's a view into the future though.

bread.mega

48,127 görüntüleme • 1 yıl önce

1/5 Here's video #4 on The Art of Explanation. It looks at how effective communication and explanation is about substance - but it's about style too. Because we need people's attention if we're to get our message across. The other videos (and a link to pre-order) are below.

1/5 Here's video #4 on The Art of Explanation. It looks at how effective communication and explanation is about substance - but it's about style too. Because we need people's attention if we're to get our message across. The other videos (and a link to pre-order) are below.

Ros Atkins

189,319 görüntüleme • 3 yıl önce

[Self-Attention] by Hand ✍️ Self-attention is what enables LLMs to understand context. How does it work? This exercise demonstrates how to calculate a 6-3 attention head by hand. Note that if we have two instances of this, we get 6-6 attention (i.e., multi-head attention, n=2). -- 𝗚𝗼𝗮𝗹 -- Transform [6D Features 🟧] to [3D Attention Weighted Features 🟦] -- 𝗪𝗮𝗹𝗸𝘁𝗵𝗿𝗼𝘂𝗴𝗵 -- [1] Given ↳ A set of 4 feature vectors (6-D): x1,x2,x3,x4 [2] Query, Key, Value ↳ Multiply features x's with linear transformation matrices WQ, WK, and WV, to obtain query vectors (q1,q2,q3,q4), key vectors (k1,k2,k3,k4), and value vectors (v1,v2,v3,v4). ↳ "Self" refers to the fact that both queries and keys are derived from the same set of features. [3] 🟪 Prepare for MatMul ↳ Copy query vectors ↳ Copy the transpose of key vectors [4] 🟪 MatMul ↳ Multiply K^T and Q ↳ This is equivalent to taking dot product between every pair of query and key vectors. ↳ The purpose is to use dot product as an estimate of the "matching score" between every key-value pair. ↳ This estimate makes sense because dot product is the numerator of Cosine Similarity between two vectors. [5] 🟨 Scale ↳ Scale each element by the square root of dk, which is the dimension of key vectors (dk=3). ↳ The purpose is to normalize the impact of the dk on matching scores, even if we scale dk to 32, 64, or 128. ↳ To simplify hand calculation, we approximate [ □/sqrt(3) ] with [ floor(□/2) ]. [6] 🟩 Softmax: e^x ↳ Raise e to the power of the number in each cell ↳ To simplify hand calculation, we approximate e^□ with 3^□. [7] 🟩 Softmax: ∑ ↳ Sum across each column [8] 🟩 Softmax: 1 / sum ↳ For each column, divide each element by the column sum ↳ The purpose is normalize each column so that the numbers sum to 1. In other words, each column is a probability distribution of attention, and we have four of them. ↳ The result is the Attention Weight Matrix (A) (yellow) [9] 🟦 MatMul ↳ Multiply the value vectors (Vs) with the Attention Weight Matrix (A) ↳ The results are the attention weighted features Zs. ↳ They are fed to the position-wise feed forward network in the next layer.

[Self-Attention] by Hand ✍️ Self-attention is what enables LLMs to understand context. How does it work? This exercise demonstrates how to calculate a 6-3 attention head by hand. Note that if we have two instances of this, we get 6-6 attention (i.e., multi-head attention, n=2). -- 𝗚𝗼𝗮𝗹 -- Transform [6D Features 🟧] to [3D Attention Weighted Features 🟦] -- 𝗪𝗮𝗹𝗸𝘁𝗵𝗿𝗼𝘂𝗴𝗵 -- [1] Given ↳ A set of 4 feature vectors (6-D): x1,x2,x3,x4 [2] Query, Key, Value ↳ Multiply features x's with linear transformation matrices WQ, WK, and WV, to obtain query vectors (q1,q2,q3,q4), key vectors (k1,k2,k3,k4), and value vectors (v1,v2,v3,v4). ↳ "Self" refers to the fact that both queries and keys are derived from the same set of features. [3] 🟪 Prepare for MatMul ↳ Copy query vectors ↳ Copy the transpose of key vectors [4] 🟪 MatMul ↳ Multiply K^T and Q ↳ This is equivalent to taking dot product between every pair of query and key vectors. ↳ The purpose is to use dot product as an estimate of the "matching score" between every key-value pair. ↳ This estimate makes sense because dot product is the numerator of Cosine Similarity between two vectors. [5] 🟨 Scale ↳ Scale each element by the square root of dk, which is the dimension of key vectors (dk=3). ↳ The purpose is to normalize the impact of the dk on matching scores, even if we scale dk to 32, 64, or 128. ↳ To simplify hand calculation, we approximate [ □/sqrt(3) ] with [ floor(□/2) ]. [6] 🟩 Softmax: e^x ↳ Raise e to the power of the number in each cell ↳ To simplify hand calculation, we approximate e^□ with 3^□. [7] 🟩 Softmax: ∑ ↳ Sum across each column [8] 🟩 Softmax: 1 / sum ↳ For each column, divide each element by the column sum ↳ The purpose is normalize each column so that the numbers sum to 1. In other words, each column is a probability distribution of attention, and we have four of them. ↳ The result is the Attention Weight Matrix (A) (yellow) [9] 🟦 MatMul ↳ Multiply the value vectors (Vs) with the Attention Weight Matrix (A) ↳ The results are the attention weighted features Zs. ↳ They are fed to the position-wise feed forward network in the next layer.

Tom Yeh

101,225 görüntüleme • 2 yıl önce

🗓️ Release Week Recap Big week. mlx-audio and mlx-vlm are now among some of the fastest-growing OSS projects. Here’s what we shipped last week. Gemma 4 on Apple Silicon Two awesome releases by our partner Google DeepMind & Google Gemma : > Gemma 4 12B — their new dense, unified multimodal model. It uses an encoder free audio path and simplified vision encoder. > Gemma 4 QAT — quantization-aware training checkpoints, optimized to run locally on consumer GPUs and edge devices, compressing the model while preserving the quality you expect from Gemma 4. On the audio 🎧 side we added support for 15+ new TTS, ASR & VAD models, faster long-form transcription, and an expanded OpenAI-compatible audio server. All local on Apple Silicon. Huge thanks to every contributor and my co-maintainer Lucas Newman. 🙏🏽

🗓️ Release Week Recap Big week. mlx-audio and mlx-vlm are now among some of the fastest-growing OSS projects. Here’s what we shipped last week. Gemma 4 on Apple Silicon Two awesome releases by our partner Google DeepMind & Google Gemma : > Gemma 4 12B — their new dense, unified multimodal model. It uses an encoder free audio path and simplified vision encoder. > Gemma 4 QAT — quantization-aware training checkpoints, optimized to run locally on consumer GPUs and edge devices, compressing the model while preserving the quality you expect from Gemma 4. On the audio 🎧 side we added support for 15+ new TTS, ASR & VAD models, faster long-form transcription, and an expanded OpenAI-compatible audio server. All local on Apple Silicon. Huge thanks to every contributor and my co-maintainer Lucas Newman. 🙏🏽

Prince Canuma

11,810 görüntüleme • 4 ay önce

2 girls. 1 is human. 1 is AI. Who gets more sales? We’re about to enter a whole new world of marketing and content. I hope you’re paying attention! By the end of the year, AI will be 100% human-like. Right now it’s 70%. So the time to get started is now. The first 5 videos are free on

2 girls. 1 is human. 1 is AI. Who gets more sales? We’re about to enter a whole new world of marketing and content. I hope you’re paying attention! By the end of the year, AI will be 100% human-like. Right now it’s 70%. So the time to get started is now. The first 5 videos are free on

Nuseir Yassin

46,628 görüntüleme • 7 ay önce

Much respect to Pastor Mike Pastor Mike Behind Bars and his statement on the last Silver & Black Today 👊🏴‍☠️ This is a good way to get the attention of the organization that we fans are checking out. And sorry Pastor for my potty mouth at the end. #RaiderNation

Much respect to Pastor Mike Pastor Mike Behind Bars and his statement on the last Silver & Black Today 👊🏴‍☠️ This is a good way to get the attention of the organization that we fans are checking out. And sorry Pastor for my potty mouth at the end. #RaiderNation

Murf 🏴‍☠️

29,198 görüntüleme • 11 ay önce

🎥August 28 2026 1:02pm This Man was going through an overdose, and me and a few other people was able to revive him and give him another chance at life. Pay attention to the person with the green pants on at the end my next video is going to connect with that person.

🎥August 28 2026 1:02pm This Man was going through an overdose, and me and a few other people was able to revive him and give him another chance at life. Pay attention to the person with the green pants on at the end my next video is going to connect with that person.

jj smith

161,999 görüntüleme • 1 ay önce

Zcash is the kind of trade Ansem looks for before attention catches up Ansem’s Zcash thesis comes down to technology, people and attention. “Zcash has a lot of the same features as Bitcoin, but it’s quantum-resistant and it’s private.” “When a ton of smart people work on a really hard technological problem, solve it, and the only thing missing is attention, those are typically the best trades.” “Zcash should benefit from Bitcoin doing well, and it should be at a much higher ratio to Bitcoin than it currently is.” ZEC shares much of Bitcoin’s monetary DNA while adding privacy and stronger preparation for quantum risk. Ansem also points to the overlap between early Bitcoin believers and people who recognized Zcash early. The trade becomes especially interesting when strong technical work already exists while market attention still lags behind it. That gap between fundamentals and attention is exactly the type of asymmetry Ansem looks for. He also views $ZEC through its relationship with Bitcoin. A stronger BTC should create a favorable backdrop for Zcash, while Ansem believes the ZEC/BTC ratio still has significant room to re-rate higher. The technology is built. The smart money is paying attention. Ansem is betting the ZEC/BTC ratio eventually catches up. w/ Ansem 🐂🀄️, mert, Zcash 🛡️

Zcash is the kind of trade Ansem looks for before attention catches up Ansem’s Zcash thesis comes down to technology, people and attention. “Zcash has a lot of the same features as Bitcoin, but it’s quantum-resistant and it’s private.” “When a ton of smart people work on a really hard technological problem, solve it, and the only thing missing is attention, those are typically the best trades.” “Zcash should benefit from Bitcoin doing well, and it should be at a much higher ratio to Bitcoin than it currently is.” ZEC shares much of Bitcoin’s monetary DNA while adding privacy and stronger preparation for quantum risk. Ansem also points to the overlap between early Bitcoin believers and people who recognized Zcash early. The trade becomes especially interesting when strong technical work already exists while market attention still lags behind it. That gap between fundamentals and attention is exactly the type of asymmetry Ansem looks for. He also views $ZEC through its relationship with Bitcoin. A stronger BTC should create a favorable backdrop for Zcash, while Ansem believes the ZEC/BTC ratio still has significant room to re-rate higher. The technology is built. The smart money is paying attention. Ansem is betting the ZEC/BTC ratio eventually catches up. w/ Ansem 🐂🀄️, mert, Zcash 🛡️

Kaiz 🐂🀄️

27,841 görüntüleme • 1 ay önce

Ex-NVIDIA engineer who built Unsloth explained RL, kernels, reasoning, quantization, and agents in 2 hours 42 minutes - better than $5000 fine-tuning bootcamps. pick the base model -> write triton kernels for 2x faster fine-tune -> quantize to 4-bit -> run GRPO/DPO -> ship a reasoning model on your single GPU. That loop is why Unsloth is the default way to fine-tune Llama, Qwen, Gemma, and Phi on hardware you already own. Unsloth + Triton kernels + 4-bit quantization + GRPO/DPO + single-GPU fine-tuning - that's the stack. Watch and save it, then fine-tune your first model tonight.

Ex-NVIDIA engineer who built Unsloth explained RL, kernels, reasoning, quantization, and agents in 2 hours 42 minutes - better than $5000 fine-tuning bootcamps. pick the base model -> write triton kernels for 2x faster fine-tune -> quantize to 4-bit -> run GRPO/DPO -> ship a reasoning model on your single GPU. That loop is why Unsloth is the default way to fine-tune Llama, Qwen, Gemma, and Phi on hardware you already own. Unsloth + Triton kernels + 4-bit quantization + GRPO/DPO + single-GPU fine-tuning - that's the stack. Watch and save it, then fine-tune your first model tonight.

h100envy

497,258 görüntüleme • 3 ay önce