正在加载视频...

视频加载失败

加载此视频时出现问题。这可能是由于临时网络问题，或视频可能不可用。

the encoder

Kat ⊷ the Poet Engineer

84,664 subscribers

103,750 次观看 • 8 个月前 •via X (Twitter)

艺术科学技术

Anya Rossi• Live Now

Private livecam show

0 条评论

暂无评论

原始帖子的评论将显示在这里

相关视频

Crazy how transformers work! It’s the encoder part

Crazy how transformers work! It’s the encoder part

Dev Khant

166,681 次观看 • 1 年前

Training Wan 2.1-1.3B to use Qwen3-VL-2B text encoder. Doing 33% text only, 33% VL only, and 33% both. Just pretraining the two text input linear layers for now. It is amazing how quickly a model can adapt to a different text encoder. This is 25,750 steps, BS of 10.

Training Wan 2.1-1.3B to use Qwen3-VL-2B text encoder. Doing 33% text only, 33% VL only, and 33% both. Just pretraining the two text input linear layers for now. It is amazing how quickly a model can adapt to a different text encoder. This is 25,750 steps, BS of 10.

Ostris

26,469 次观看 • 2 个月前

i just ran Google's brand new Unsloth Gemma4 12B dense GGUF on my RTX 4060 using llama.cpp + CUDA 13.2 21 tokens per second. on a budget consumer GPU. locally. no API. no cloud. no subscription. and the benchmarks are absolutely cooked # first let's talk architecture because this is genuinely different every multimodal model you've used has a frozen vision encoder + frozen audio encoder + LLM backbone glued together Gemma 4 12B is different it's a single decoder only transformer. that's it. vision? raw 48×48 pixel patches → one matmul → projected directly into the LLM audio? raw 16kHz signal sliced into 40ms frames → linear projection → same LLM input space no encoder tax. no latency penalty. no fragmented memory to put the encoder savings in perspective: old Gemma 4 26B approach: - 550M param vision encoder (frozen) - 300M param audio encoder (frozen) - LLM backbone Gemma 4 12B: - 35M param vision embedder (a single matmul) - no audio encoder at all - LLM backbone handles EVERYTHING 550M → 35M for vision alone. that's a 15x reduction this is why the gemma-4-12b-it-Q4_K_M.gguf is just 6.6 GBs!!! and it has 256K native context context # Benchmarks: AIME 2026 (math olympiad): 77.5% GPQA Diamond (expert science): 78.8% LiveCodeBench v6 (real code): 72% Codeforces ELO: 1659 MMLU Pro: 77.2% MATH-Vision: 79.7% BigBench Extra Hard: 53% inference → llama.cpp, LM Studio, vLLM, SGLang llamacpp flags: -m "gemma-4-12b-it-Q4_K_M.gguf" -ngl 99 -c 8000 -v --port 8080 Available on huggingface now! Link below

Alok

281,007 次观看 • 3 个月前

Modern tech tends to focus on “more”. More data. More bandwidth. More compute. The best software engineering happens in the “less” sphere; codec engineering. Listen to this stream with 90% packet loss. The Opus Encoder team is cracked:

Modern tech tends to focus on “more”. More data. More bandwidth. More compute. The best software engineering happens in the “less” sphere; codec engineering. Listen to this stream with 90% packet loss. The Opus Encoder team is cracked:

LaurieWired

244,761 次观看 • 1 年前

4/ to achieve maximum memory efficiency, we quantize model weights to ~4-bit, getting the language model under 20GB with room for the kv cache, perception encoder, and drafter alongside it. a dflash drafter proposes blocks of tokens the main model verifies in parallel, so it stays responsive.

4/ to achieve maximum memory efficiency, we quantize model weights to ~4-bit, getting the language model under 20GB with room for the kv cache, perception encoder, and drafter alongside it. a dflash drafter proposes blocks of tokens the main model verifies in parallel, so it stays responsive.

Alexandr Wang

69,069 次观看 • 1 个月前

Autoencoder by hand✍️Excel~ I designed this exercise to show how an Encoder-Decoder network convert input to code and reconstruct input from code. It is annotated with equations, PyTorch, and graphs. 👇Join the 'AI Math' community. Download xlsx.

Autoencoder by hand✍️Excel~ I designed this exercise to show how an Encoder-Decoder network convert input to code and reconstruct input from code. It is annotated with equations, PyTorch, and graphs. 👇Join the 'AI Math' community. Download xlsx.

Tom Yeh

101,555 次观看 • 1 年前

Can a VLM see without a vision encoder? We trained one for $100, inspired by Gemma 4 12B. Latency on an M3 Pro MacBook: 112 ms -> 1.1 ms for the image path 30% lower end-to-end image+LLM The architecture is just: patchify the image -> linear projection with pos embeddings -> LLM Writeup:

Can a VLM see without a vision encoder? We trained one for $100, inspired by Gemma 4 12B. Latency on an M3 Pro MacBook: 112 ms -> 1.1 ms for the image path 30% lower end-to-end image+LLM The architecture is just: patchify the image -> linear projection with pos embeddings -> LLM Writeup:

Andi Marafioti

60,546 次观看 • 3 个月前

Autoencoder by hand✍️Excel~ I designed this exercise to show how an Encoder-Decoder network convert input to code and reconstruct input from code. It is annotated with equations, PyTorch, and graphs. I also made a medium version.👇Join the 'AI Math' community. Download xlsx.

Autoencoder by hand✍️Excel~ I designed this exercise to show how an Encoder-Decoder network convert input to code and reconstruct input from code. It is annotated with equations, PyTorch, and graphs. I also made a medium version.👇Join the 'AI Math' community. Download xlsx.

Tom Yeh

54,516 次观看 • 1 年前

Day 15 of learning AI bs from scratch > looked into seq2seq models > understood the encoder-decoder architecture > implemented a basic Seq2Seq model using RNNs > how the decoder generates outputs one step at a time > read couple more chapters of > understood the fixed-context bottleneck I updated the resource list blog, link below, it has more resources organized now. Fell asleep on the desk towards the morning, rweeting rn after waking up.

Day 15 of learning AI bs from scratch > looked into seq2seq models > understood the encoder-decoder architecture > implemented a basic Seq2Seq model using RNNs > how the decoder generates outputs one step at a time > read couple more chapters of > understood the fixed-context bottleneck I updated the resource list blog, link below, it has more resources organized now. Fell asleep on the desk towards the morning, rweeting rn after waking up.

Arham Amin

31,680 次观看 • 1 个月前

THE BEST visual explainer of how information propagates through a transformer. If you want to have more than intuition about how the Transformer architecture is ruling the LLM world - open-source project explains everything about LLM Transformer Models! - A great resource for anyone looking to gain a deeper understanding of how Transformer-based AI models like GPT work, including: - Self-attention mechanisms - Encoder-decoder architecture - Positional encoding - Multi-head attention

THE BEST visual explainer of how information propagates through a transformer. If you want to have more than intuition about how the Transformer architecture is ruling the LLM world - open-source project explains everything about LLM Transformer Models! - A great resource for anyone looking to gain a deeper understanding of how Transformer-based AI models like GPT work, including: - Self-attention mechanisms - Encoder-decoder architecture - Positional encoding - Multi-head attention

Rohan Paul

106,897 次观看 • 2 年前

Protein language models just got an upgrade. Meet Profluent-E1: a free, flexible, frontier protein sequence encoder. E1 is built with retrieval augmentation to learn from multiple sequences. Models trained over 4T tokens with only 150M-600M params, E1 is SOTA for zero-shot functional and unsupervised structural tasks. It raises the bar for protein representation learning and is freely available today.

Protein language models just got an upgrade. Meet Profluent-E1: a free, flexible, frontier protein sequence encoder. E1 is built with retrieval augmentation to learn from multiple sequences. Models trained over 4T tokens with only 150M-600M params, E1 is SOTA for zero-shot functional and unsupervised structural tasks. It raises the bar for protein representation learning and is freely available today.

Profluent

222,106 次观看 • 10 个月前

In my class, I teach the autoencoder by asking everyone to stand up. 🙆 Stretch your arms out wide. Imagine you are holding a heavy textbook (like Introduction to Algorithms by Prof. Cormen), the whole thing, every page. Now bring your hands slowly together until they almost touch your neck. The bottle "neck." The final exam is tomorrow and you are allowed one cheat sheet: whatever you can scribble on your palm. All nine hundred pages have to survive the squeeze. That is the encoder. Now imagine you sit in the exam. Push your arms back out to where they started. You try to rebuild the textbook from your palm notes. That is the decoder. Of course you cannot get every page back. What you get back is what mattered enough to write down, and the gap between the two is the loss the network is trying to shrink. I call it AI by Arms 🙆. It gets a laugh, and then it gets remembered. Goal: squeeze four numbers down to two, then rebuild the original four from them. 1. Given Let us start with four training examples: X1, X2, X3, X4. 2. Auto (copy to targets) We copy the training examples straight into the targets. That is the whole trick behind the name: "auto" is Greek for "self", and the data is its own label. 3. Encoder, layer 1 Let us multiply the inputs by the weights, add the biases, and apply ReLU. Negative values get crossed out and become zero. 4. Encoder, layer 2 (the bottleneck) We do it again, and now the four dimensions have become two. This layer is called the bottleneck, because everything has to fit through it. 5. Decoder, layer 1 Let us go back the other way: multiply, add, ReLU. This time there are no negatives to cross out. 6. Decoder, layer 2 We multiply once more and get the outputs Y. This is the decoder's attempt to rebuild the four original numbers from the two it was given. 7. MSE loss gradients Let us compare Y with the targets Y'. The gradient is 2 x (Y - Y'): subtract, then double. Those gradients kick off backpropagation, and the weights start to learn. Your entire education is all about encoding and decoding!

In my class, I teach the autoencoder by asking everyone to stand up. 🙆 Stretch your arms out wide. Imagine you are holding a heavy textbook (like Introduction to Algorithms by Prof. Cormen), the whole thing, every page. Now bring your hands slowly together until they almost touch your neck. The bottle "neck." The final exam is tomorrow and you are allowed one cheat sheet: whatever you can scribble on your palm. All nine hundred pages have to survive the squeeze. That is the encoder. Now imagine you sit in the exam. Push your arms back out to where they started. You try to rebuild the textbook from your palm notes. That is the decoder. Of course you cannot get every page back. What you get back is what mattered enough to write down, and the gap between the two is the loss the network is trying to shrink. I call it AI by Arms 🙆. It gets a laugh, and then it gets remembered. Goal: squeeze four numbers down to two, then rebuild the original four from them. 1. Given Let us start with four training examples: X1, X2, X3, X4. 2. Auto (copy to targets) We copy the training examples straight into the targets. That is the whole trick behind the name: "auto" is Greek for "self", and the data is its own label. 3. Encoder, layer 1 Let us multiply the inputs by the weights, add the biases, and apply ReLU. Negative values get crossed out and become zero. 4. Encoder, layer 2 (the bottleneck) We do it again, and now the four dimensions have become two. This layer is called the bottleneck, because everything has to fit through it. 5. Decoder, layer 1 Let us go back the other way: multiply, add, ReLU. This time there are no negatives to cross out. 6. Decoder, layer 2 We multiply once more and get the outputs Y. This is the decoder's attempt to rebuild the four original numbers from the two it was given. 7. MSE loss gradients Let us compare Y with the targets Y'. The gradient is 2 x (Y - Y'): subtract, then double. Those gradients kick off backpropagation, and the weights start to learn. Your entire education is all about encoding and decoding!

Tom Yeh

22,703 次观看 • 1 个月前

🏆 We're thrilled to announce that Meta FAIR’s Brain & AI team won 1st place at the prestigious Algonauts 2025 brain modeling competition. Their 1B parameter model, TRIBE (Trimodal Brain Encoder), is the first deep neural network trained to predict brain responses to stimuli across multiple modalities, cortical areas, and individuals. The approach combines pretrained representations of several foundational models from Meta – text (Llama 3.2), audio (Wav2Vec2-BERT from Seamless) and video (V-JEPA 2) – to predict a very large amount (80 hours per subject) of spatio-temporal fMRI brain responses to movies acquired by the Courtois NeuroMod project Download the code: Read the paper: Learn about the challenge: Download the data:

🏆 We're thrilled to announce that Meta FAIR’s Brain & AI team won 1st place at the prestigious Algonauts 2025 brain modeling competition. Their 1B parameter model, TRIBE (Trimodal Brain Encoder), is the first deep neural network trained to predict brain responses to stimuli across multiple modalities, cortical areas, and individuals. The approach combines pretrained representations of several foundational models from Meta – text (Llama 3.2), audio (Wav2Vec2-BERT from Seamless) and video (V-JEPA 2) – to predict a very large amount (80 hours per subject) of spatio-temporal fMRI brain responses to movies acquired by the Courtois NeuroMod project Download the code: Read the paper: Learn about the challenge: Download the data:

AI at Meta

1,094,196 次观看 • 1 年前

Vector Database by Hand ✍️ Vector databases are revolutionizing how we search and analyze complex data. They have become the backbone of Retrieval Augmented Generation (#RAG). How do vector databases work? [1] Given ↳ A dataset of three sentences, each has 3 words (or tokens) ↳ In practice, a dataset may contain millions or billions of sentences. The max number of tokens may be tens of thousands (e.g., 32,768 mistral-7b). Process "how are you" [2] 🟨 Word Embeddings ↳ For each word, look up corresponding word embedding vector from a table of 22 vectors, where 22 is the vocabulary size. ↳ In practice, the vocabulary size can be tens of thousands. The word embedding dimensions are in the thousands (e.g., 1024, 4096) [3] 🟩 Encoding ↳ Feed the sequence of word embeddings to an encoder to obtain a sequence of feature vectors, one per word. ↳ Here, the encoder is a simple one layer perceptron (linear layer + ReLU) ↳ In practice, the encoder is a transformer or one of its many variants. [4] 🟩 Mean Pooling ↳ Merge the sequence of feature vectors into a single vector using "mean pooling" which is to average across the columns. ↳ The result is a single vector. We often call it "text embeddings" or "sentence embeddings." ↳ Other pooling techniques are possible, such as CLS. But mean pooling is the most common. [5] 🟦 Indexing ↳ Reduce the dimensions of the text embedding vector by a projection matrix. The reduction rate is 50% (4->2). ↳ In practice, the values in this projection matrix is much more random. ↳ The purpose is similar to that of hashing, which is to obtain a short representation to allow faster comparison and retrieval. ↳ The resulting dimension-reduced index vector is saved in the vector storage. [6] Process "who are you" ↳ Repeat [2]-[5] [7] Process "who am I" ↳ Repeat [2]-[5] Now we have indexed our dataset in the vector database. [8] 🟥 Query: "am I you" ↳ Repeat [2]-[5] ↳ The result is a 2-d query vector. [9] 🟥 Dot Products ↳ Take dot product between the query vector and database vectors. They are all 2-d. ↳ The purpose is to use dot product to estimate similarity. ↳ By transposing the query vector, this step becomes a matrix multiplication. [10] 🟥 Nearest Neighbor ↳ Find the largest dot product by linear scan. ↳ The sentence with the highest dot product is "who am I" ↳ In practice, because scanning billions of vectors is slow, we use an Approximate Nearest Neighbor (ANN) algorithm like the Hierarchical Navigable Small Worlds (HNSW).

Vector Database by Hand ✍️ Vector databases are revolutionizing how we search and analyze complex data. They have become the backbone of Retrieval Augmented Generation (#RAG). How do vector databases work? [1] Given ↳ A dataset of three sentences, each has 3 words (or tokens) ↳ In practice, a dataset may contain millions or billions of sentences. The max number of tokens may be tens of thousands (e.g., 32,768 mistral-7b). Process "how are you" [2] 🟨 Word Embeddings ↳ For each word, look up corresponding word embedding vector from a table of 22 vectors, where 22 is the vocabulary size. ↳ In practice, the vocabulary size can be tens of thousands. The word embedding dimensions are in the thousands (e.g., 1024, 4096) [3] 🟩 Encoding ↳ Feed the sequence of word embeddings to an encoder to obtain a sequence of feature vectors, one per word. ↳ Here, the encoder is a simple one layer perceptron (linear layer + ReLU) ↳ In practice, the encoder is a transformer or one of its many variants. [4] 🟩 Mean Pooling ↳ Merge the sequence of feature vectors into a single vector using "mean pooling" which is to average across the columns. ↳ The result is a single vector. We often call it "text embeddings" or "sentence embeddings." ↳ Other pooling techniques are possible, such as CLS. But mean pooling is the most common. [5] 🟦 Indexing ↳ Reduce the dimensions of the text embedding vector by a projection matrix. The reduction rate is 50% (4->2). ↳ In practice, the values in this projection matrix is much more random. ↳ The purpose is similar to that of hashing, which is to obtain a short representation to allow faster comparison and retrieval. ↳ The resulting dimension-reduced index vector is saved in the vector storage. [6] Process "who are you" ↳ Repeat [2]-[5] [7] Process "who am I" ↳ Repeat [2]-[5] Now we have indexed our dataset in the vector database. [8] 🟥 Query: "am I you" ↳ Repeat [2]-[5] ↳ The result is a 2-d query vector. [9] 🟥 Dot Products ↳ Take dot product between the query vector and database vectors. They are all 2-d. ↳ The purpose is to use dot product to estimate similarity. ↳ By transposing the query vector, this step becomes a matrix multiplication. [10] 🟥 Nearest Neighbor ↳ Find the largest dot product by linear scan. ↳ The sentence with the highest dot product is "who am I" ↳ In practice, because scanning billions of vectors is slow, we use an Approximate Nearest Neighbor (ANN) algorithm like the Hierarchical Navigable Small Worlds (HNSW).

Tom Yeh

192,022 次观看 • 2 年前

Most imitation learning policies break when the camera moves or the robot changes. NOT THIS ONE 👇 [📍 Bookmark for later ] A new 3D scene representation encoder, tackles this by enabling zero-shot generalization to unseen embodiments and viewpoints… And it works with any IL algorithm. The trick? •Use a 2D foundation model to extract semantic features •Lift them into 3D space for localization (not semantics) •Condition the IL policy on this spatially grounded vector Across 93 simulated and 6 real tasks, Adapt3R: ✅ Maintains IL performance on LIBERO & MimicGen benchmarks ✅ Outperforms DP3 and 3D Diffuser Actor in most settings ✅ Holds >80% success on LIBERO even with large camera rotations Thanks for sharing this, Animesh Garg & Albert Wilcox! 📍Paper: Website: Code:

Most imitation learning policies break when the camera moves or the robot changes. NOT THIS ONE 👇 [📍 Bookmark for later ] A new 3D scene representation encoder, tackles this by enabling zero-shot generalization to unseen embodiments and viewpoints… And it works with any IL algorithm. The trick? •Use a 2D foundation model to extract semantic features •Lift them into 3D space for localization (not semantics) •Condition the IL policy on this spatially grounded vector Across 93 simulated and 6 real tasks, Adapt3R: ✅ Maintains IL performance on LIBERO & MimicGen benchmarks ✅ Outperforms DP3 and 3D Diffuser Actor in most settings ✅ Holds >80% success on LIBERO even with large camera rotations Thanks for sharing this, Animesh Garg & Albert Wilcox! 📍Paper: Website: Code:

Ilir Aliu

12,178 次观看 • 1 年前

NEW WORLD MODEL: Yann LeCun's team is back with an efficient model! This project involves Yann LeCun, Lukas Kuhn, Lucas Maes, Quentin Le Lidec, and Randall Balestriero. A couple definitions first: - DINO: self-DIstillation with NO labels. A self-supervised image model (Meta, 2021) where a student network learns to match a teacher (an EMA copy of itself) across two crops of the same image, with no labels and no negatives. - SIGReg: a regularizer that prevents embedding collapse by forcing the embeddings to match an isotropic Gaussian, tested with a normality test (Epps–Pulley) on many random 1-D projections instead of in full dimension. LeVJEPA is a self-supervised video pretraining method, released with open code, weights, and checkpoints. It learns a video representation by pushing the embeddings of global and local crops of the same clip together (an invariance loss), while a regularizer called SIGReg forces the embeddings toward an isotropic Gaussian to provably prevent representation collapse. Unlike V-JEPA and V-JEPA 2 it uses a single shared encoder with a projector and no target network, no predictor and no stop-gradient. It drops 95% of tokens per view, uses block-causal attention (each frame attends only to past frames), and has a single loss weight. It is evaluated purely as a representation learner via frozen probing on ImageNet-1K, Something-Something-v2 and Kinetics-400, not on any robot. What I find interesting, is that V-JEPA and V-JEPA 2 need an EMA target encoder, stop-gradients and a capacity-limited predictor to avoid collapse; LeVJEPA drops all of it for one shared encoder plus projector, preventing collapse instead with the SIGReg regularizer under a provable guarantee and a single hyperparameter. The "P" (predictor) in JEPA is effectively gone. LeVJEPA is also less compute intensive: - 5.6x to 20.8x lower total pretraining compute than V-JEPA 2 - 7.6 points higher on ImageNet-1K at matched FLOPs - trains at batch size 128 within 8GB where V-JEPA 2 saturates at batch size 28 Also worth mentioning: ImageNet-1K accuracy rises monotonically with the token-drop rate, from 33.9% at rho = 0 to 47.6% at rho = 0.95. The aggressive dropping is actually doing regularization work. On the JEPA-versus-DINO debate: - it loses to DINOv2 by 3.1 points on ImageNet-1K (appearance, static) - but wins on Something-Something-v2 by nearly 2x (motion, temporal) - and beats V-JEPA 2 by 1.9 points on ViT-L at 5.6x lower cost. -> optimized for temporal and motion understanding per compute dollar.

NEW WORLD MODEL: Yann LeCun's team is back with an efficient model! This project involves Yann LeCun, Lukas Kuhn, Lucas Maes, Quentin Le Lidec, and Randall Balestriero. A couple definitions first: - DINO: self-DIstillation with NO labels. A self-supervised image model (Meta, 2021) where a student network learns to match a teacher (an EMA copy of itself) across two crops of the same image, with no labels and no negatives. - SIGReg: a regularizer that prevents embedding collapse by forcing the embeddings to match an isotropic Gaussian, tested with a normality test (Epps–Pulley) on many random 1-D projections instead of in full dimension. LeVJEPA is a self-supervised video pretraining method, released with open code, weights, and checkpoints. It learns a video representation by pushing the embeddings of global and local crops of the same clip together (an invariance loss), while a regularizer called SIGReg forces the embeddings toward an isotropic Gaussian to provably prevent representation collapse. Unlike V-JEPA and V-JEPA 2 it uses a single shared encoder with a projector and no target network, no predictor and no stop-gradient. It drops 95% of tokens per view, uses block-causal attention (each frame attends only to past frames), and has a single loss weight. It is evaluated purely as a representation learner via frozen probing on ImageNet-1K, Something-Something-v2 and Kinetics-400, not on any robot. What I find interesting, is that V-JEPA and V-JEPA 2 need an EMA target encoder, stop-gradients and a capacity-limited predictor to avoid collapse; LeVJEPA drops all of it for one shared encoder plus projector, preventing collapse instead with the SIGReg regularizer under a provable guarantee and a single hyperparameter. The "P" (predictor) in JEPA is effectively gone. LeVJEPA is also less compute intensive: - 5.6x to 20.8x lower total pretraining compute than V-JEPA 2 - 7.6 points higher on ImageNet-1K at matched FLOPs - trains at batch size 128 within 8GB where V-JEPA 2 saturates at batch size 28 Also worth mentioning: ImageNet-1K accuracy rises monotonically with the token-drop rate, from 33.9% at rho = 0 to 47.6% at rho = 0.95. The aggressive dropping is actually doing regularization work. On the JEPA-versus-DINO debate: - it loses to DINOv2 by 3.1 points on ImageNet-1K (appearance, static) - but wins on Something-Something-v2 by nearly 2x (motion, temporal) - and beats V-JEPA 2 by 1.9 points on ViT-L at 5.6x lower cost. -> optimized for temporal and motion understanding per compute dollar.

Léo

90,267 次观看 • 16 天前

Testing the new Gemma 4 12B (QAT) vision and OCR capabilities locally with LM Studio. # The setup: - GPU: NVIDIA RTX 4060 (8GB VRAM) - CPU: Intel i7 - Runner: LM Studio - Config: 32k context, 38 layers offloaded, Flash Attention enabled - Speed: ~14 tokens/sec decode throughput # The test: I gave it a screenshot of Google AI Studio. Prompt: "clone this. give me a single html file" # The result: A solid one shot replication. It successfully mapped out the layout, recognized the UI text, and structured the divs correctly, with only minor differences from the original. Results available at the end of the video. Quite capable for a 12B model running on budget consumer hardware. A gpu that costs only $300. # Why the architecture under the hood is notable: Unlike traditional models that rely on heavy, separate vision and audio encoders, Gemma 4 12B uses a unified, encoder free architecture. It bypasses separate multi stage encoders. Uses a 35M parameter vision embedder to project raw 48x48 pixel patches directly to the LLM hidden dimension. Local multimodal development is becoming highly accessible on standard hardware. If you've spun up Gemma 4 12B locally, what setup are you using and what kind of throughput are you seeing?

Testing the new Gemma 4 12B (QAT) vision and OCR capabilities locally with LM Studio. # The setup: - GPU: NVIDIA RTX 4060 (8GB VRAM) - CPU: Intel i7 - Runner: LM Studio - Config: 32k context, 38 layers offloaded, Flash Attention enabled - Speed: ~14 tokens/sec decode throughput # The test: I gave it a screenshot of Google AI Studio. Prompt: "clone this. give me a single html file" # The result: A solid one shot replication. It successfully mapped out the layout, recognized the UI text, and structured the divs correctly, with only minor differences from the original. Results available at the end of the video. Quite capable for a 12B model running on budget consumer hardware. A gpu that costs only $300. # Why the architecture under the hood is notable: Unlike traditional models that rely on heavy, separate vision and audio encoders, Gemma 4 12B uses a unified, encoder free architecture. It bypasses separate multi stage encoders. Uses a 35M parameter vision embedder to project raw 48x48 pixel patches directly to the LLM hidden dimension. Local multimodal development is becoming highly accessible on standard hardware. If you've spun up Gemma 4 12B locally, what setup are you using and what kind of throughput are you seeing?

Alok

25,717 次观看 • 3 个月前