正在加载视频...

视频加载失败

返回首页

THE BEST visual explainer of how information propagates through a transformer. If you want to have more than intuition about how the Transformer architecture is ruling the LLM world - open-source project explains everything about LLM Transformer Models! - A great resource for anyone looking to gain a deeper...

rohanpaul_ai's profile picture

Rohan Paul

150,798 subscribers

106,897 次观看 • 1 年前 •via X (Twitter)

科学技术新闻政治教育
LIVE

Anya Rossi• Live Now

Private livecam show

0 条评论

暂无评论

原始帖子的评论将显示在这里

相关视频

Wow. This is one of the best interactive sites I’ve seen for learning how LLMs work! 🔥 It starts w/ a clear intro and guides you through every core component: from Embedding, Layer Norm, and Self-Attention to MLPs, Transformer blocks, Softmax, and Output layers. link in 🧵↓
0:18

Wow. This is one of the best interactive sites I’ve seen for learning how LLMs work! 🔥 It starts w/ a clear intro and guides you through every core component: from Embedding, Layer Norm, and Self-Attention to MLPs, Transformer blocks, Softmax, and Output layers. link in 🧵↓

Charly Wargnier

52,092 次观看 • 1 年前

Transformer by hand ✍️ ~ 6 steps walkthrough below Open the hood of a transformer and the parts list is overwhelming: embeddings, positional encoding, attention weighting, self-attention, cross-attention, multi-head attention, layer norm, skip connections, softmax, linear, Nx, shifted right, query, key, value, masking. Which of those actually make the car run? Two of them. Attention weighting and the feed-forward network. Everything else is an enhancement to make it run faster and longer, which is how we got from a car to a truck, and to the word "large" in large language model. So I drew and calculated those two parts entirely by hand. Goal: push five features through one transformer block, filling in every cell yourself. 1. Given Five positions of input features, arriving from the previous block. 2. Attention matrix Let us feed all five features to a query-key module (QK) and read back an attention weight matrix, A. The details of that module are a post of their own. 3. Attention weighting We multiply the input features by A to get the attention weighted features, Z. Still five positions. The effect is to combine features *across positions*, horizontally: X1 becomes X1 + X2, X2 becomes X2 + X3, and so on. 4. First layer Let us feed all five weighted features into the first layer of the FFN. Multiply by the weights and biases. This time the combining happens *across feature dimensions*, vertically, and each feature grows from 3 numbers to 4. Note that every position goes through the same weight matrix. That is what "position-wise" means. 5. ReLU We cross out the negatives. They become zeros. 6. Second layer Let us bring it back down: 4 dimensions to 3. The output feeds the next block, which has a completely separate set of parameters, and the whole thing runs again. You have just calculated a transformer block by hand. ✍️ The takeaway: the two parts are doing two different jobs, and neither one alone is enough. Attention mixes *across positions*, so a feature can see its neighbours. The FFN mixes *across feature dimensions*, so each position can think about itself. Horizontal, then vertical. Then that pattern repeats N times, each block with its own separate set of weights. That is the Nx from the list up top, and that is what makes the transformer run. 💾 Save this post! #AIbyHand #Transformers #DeepLearning
0:12

Transformer by hand ✍️ ~ 6 steps walkthrough below Open the hood of a transformer and the parts list is overwhelming: embeddings, positional encoding, attention weighting, self-attention, cross-attention, multi-head attention, layer norm, skip connections, softmax, linear, Nx, shifted right, query, key, value, masking. Which of those actually make the car run? Two of them. Attention weighting and the feed-forward network. Everything else is an enhancement to make it run faster and longer, which is how we got from a car to a truck, and to the word "large" in large language model. So I drew and calculated those two parts entirely by hand. Goal: push five features through one transformer block, filling in every cell yourself. 1. Given Five positions of input features, arriving from the previous block. 2. Attention matrix Let us feed all five features to a query-key module (QK) and read back an attention weight matrix, A. The details of that module are a post of their own. 3. Attention weighting We multiply the input features by A to get the attention weighted features, Z. Still five positions. The effect is to combine features *across positions*, horizontally: X1 becomes X1 + X2, X2 becomes X2 + X3, and so on. 4. First layer Let us feed all five weighted features into the first layer of the FFN. Multiply by the weights and biases. This time the combining happens *across feature dimensions*, vertically, and each feature grows from 3 numbers to 4. Note that every position goes through the same weight matrix. That is what "position-wise" means. 5. ReLU We cross out the negatives. They become zeros. 6. Second layer Let us bring it back down: 4 dimensions to 3. The output feeds the next block, which has a completely separate set of parameters, and the whole thing runs again. You have just calculated a transformer block by hand. ✍️ The takeaway: the two parts are doing two different jobs, and neither one alone is enough. Attention mixes *across positions*, so a feature can see its neighbours. The FFN mixes *across feature dimensions*, so each position can think about itself. Horizontal, then vertical. Then that pattern repeats N times, each block with its own separate set of weights. That is the Nx from the list up top, and that is what makes the transformer run. 💾 Save this post! #AIbyHand #Transformers #DeepLearning

Tom Yeh

25,883 次观看 • 18 天前

Part 2/30 of the LLM Series: RoPE (Rotary Position Embedding) How does a transformer know the difference between - "the dog bit the man" and "the man bit the dog"? The words are almost identical, but the meaning changes completely. RoPE encodes position as rotation, allowing transformers to understand relative order through geometry. Read more:
0:16

Part 2/30 of the LLM Series: RoPE (Rotary Position Embedding) How does a transformer know the difference between - "the dog bit the man" and "the man bit the dog"? The words are almost identical, but the meaning changes completely. RoPE encodes position as rotation, allowing transformers to understand relative order through geometry. Read more:

TensorTonic

11,766 次观看 • 2 个月前

🇯🇵 A JAPANESE LAB SAYS IT MATCHED FABLE 5 LEVELS. While the world braced for China to crack the frontier, a Tokyo lab did something stranger. Sakana AI, co-founded by one of the authors of the original Transformer paper, launched Fugu today. It's not a bigger model. It's a conductor. One API that quietly assembles a team of other models, splits the work, verifies it, and hands you one answer. Sakana claims its top tier matches Anthropic's Fable 5 and Mythos.
0:16

🇯🇵 A JAPANESE LAB SAYS IT MATCHED FABLE 5 LEVELS. While the world braced for China to crack the frontier, a Tokyo lab did something stranger. Sakana AI, co-founded by one of the authors of the original Transformer paper, launched Fugu today. It's not a bigger model. It's a conductor. One API that quietly assembles a team of other models, splits the work, verifies it, and hands you one answer. Sakana claims its top tier matches Anthropic's Fable 5 and Mythos.

CryptoGoos

27,449 次观看 • 1 个月前

What if aircraft were built like open-source software? Project Caribou is an ambitious open-source effort to build a 200 kg-class heavy-lift hexacopter that anyone can study, improve, and contribute to. The long-term goal is to accelerate the development of larger open-source aircraft and eVTOL platforms. The project targets a ~200 kg maximum takeoff weight with an ~100 kg payload, uses an 18S power architecture, and is built around an ArduPilot-based flight stack. The repository includes CAD models, PCB designs, documentation, and engineering workflows that are evolving in the open. If you are interested in drones, aerospace, robotics, or embedded systems, this is one of the most exciting open-source engineering projects to follow.
0:15

What if aircraft were built like open-source software? Project Caribou is an ambitious open-source effort to build a 200 kg-class heavy-lift hexacopter that anyone can study, improve, and contribute to. The long-term goal is to accelerate the development of larger open-source aircraft and eVTOL platforms. The project targets a ~200 kg maximum takeoff weight with an ~100 kg payload, uses an 18S power architecture, and is built around an ArduPilot-based flight stack. The repository includes CAD models, PCB designs, documentation, and engineering workflows that are evolving in the open. If you are interested in drones, aerospace, robotics, or embedded systems, this is one of the most exciting open-source engineering projects to follow.

Praveen Kumar Verma

32,163 次观看 • 2 天前

I’m launching my first open-source project. If you are building an app like sage by buildspace It’s a generative UI framework for AI applications. This is the demo we created at the ai tinkerers hackathon this Saturday. I am looking for users & contributors. Drop a comment below with how you want to use it.
0:18

I’m launching my first open-source project. If you are building an app like sage by buildspace It’s a generative UI framework for AI applications. This is the demo we created at the ai tinkerers hackathon this Saturday. I am looking for users & contributors. Drop a comment below with how you want to use it.

Michael Magán

14,631 次观看 • 2 年前

i just ran Google's brand new Unsloth Gemma4 12B dense GGUF on my RTX 4060 using llama.cpp + CUDA 13.2 21 tokens per second. on a budget consumer GPU. locally. no API. no cloud. no subscription. and the benchmarks are absolutely cooked # first let's talk architecture because this is genuinely different every multimodal model you've used has a frozen vision encoder + frozen audio encoder + LLM backbone glued together Gemma 4 12B is different it's a single decoder only transformer. that's it. vision? raw 48×48 pixel patches → one matmul → projected directly into the LLM audio? raw 16kHz signal sliced into 40ms frames → linear projection → same LLM input space no encoder tax. no latency penalty. no fragmented memory to put the encoder savings in perspective: old Gemma 4 26B approach: - 550M param vision encoder (frozen) - 300M param audio encoder (frozen) - LLM backbone Gemma 4 12B: - 35M param vision embedder (a single matmul) - no audio encoder at all - LLM backbone handles EVERYTHING 550M → 35M for vision alone. that's a 15x reduction this is why the gemma-4-12b-it-Q4_K_M.gguf is just 6.6 GBs!!! and it has 256K native context context # Benchmarks: AIME 2026 (math olympiad): 77.5% GPQA Diamond (expert science): 78.8% LiveCodeBench v6 (real code): 72% Codeforces ELO: 1659 MMLU Pro: 77.2% MATH-Vision: 79.7% BigBench Extra Hard: 53% inference → llama.cpp, LM Studio, vLLM, SGLang llamacpp flags: -m "gemma-4-12b-it-Q4_K_M.gguf" -ngl 99 -c 8000 -v --port 8080 Available on huggingface now! Link below
0:18

i just ran Google's brand new Unsloth Gemma4 12B dense GGUF on my RTX 4060 using llama.cpp + CUDA 13.2 21 tokens per second. on a budget consumer GPU. locally. no API. no cloud. no subscription. and the benchmarks are absolutely cooked # first let's talk architecture because this is genuinely different every multimodal model you've used has a frozen vision encoder + frozen audio encoder + LLM backbone glued together Gemma 4 12B is different it's a single decoder only transformer. that's it. vision? raw 48×48 pixel patches → one matmul → projected directly into the LLM audio? raw 16kHz signal sliced into 40ms frames → linear projection → same LLM input space no encoder tax. no latency penalty. no fragmented memory to put the encoder savings in perspective: old Gemma 4 26B approach: - 550M param vision encoder (frozen) - 300M param audio encoder (frozen) - LLM backbone Gemma 4 12B: - 35M param vision embedder (a single matmul) - no audio encoder at all - LLM backbone handles EVERYTHING 550M → 35M for vision alone. that's a 15x reduction this is why the gemma-4-12b-it-Q4_K_M.gguf is just 6.6 GBs!!! and it has 256K native context context # Benchmarks: AIME 2026 (math olympiad): 77.5% GPQA Diamond (expert science): 78.8% LiveCodeBench v6 (real code): 72% Codeforces ELO: 1659 MMLU Pro: 77.2% MATH-Vision: 79.7% BigBench Extra Hard: 53% inference → llama.cpp, LM Studio, vLLM, SGLang llamacpp flags: -m "gemma-4-12b-it-Q4_K_M.gguf" -ngl 99 -c 8000 -v --port 8080 Available on huggingface now! Link below

Alok

280,177 次观看 • 2 个月前

The cost of AI video was never really about the model. It was about how many models you had to pay for. MiniMax H3 understands text, image, audio and video together, then generates the final clip with sound in one pass. Topview is one of the first platforms in the world to carry it, at 30% of Seedance 2.0's price. It's a great time to be creating :
0:15

The cost of AI video was never really about the model. It was about how many models you had to pay for. MiniMax H3 understands text, image, audio and video together, then generates the final clip with sound in one pass. Topview is one of the first platforms in the world to carry it, at 30% of Seedance 2.0's price. It's a great time to be creating :

AI Frontliner

13,218 次观看 • 4 天前

Can industrial AI do more than just observe? Nicolas Cerisier, Vice President of 3DEXPERIENCE Platform R&D at Dassault Systèmes explains how the shift from SaaS to "Agent-as-a-Service" is introducing "Industry World Models" that understand the scientific laws of physics and engineering to power autonomous virtual companions.
0:18

Can industrial AI do more than just observe? Nicolas Cerisier, Vice President of 3DEXPERIENCE Platform R&D at Dassault Systèmes explains how the shift from SaaS to "Agent-as-a-Service" is introducing "Industry World Models" that understand the scientific laws of physics and engineering to power autonomous virtual companions.

NVIDIA

25,743 次观看 • 3 个月前

Republicans like Nancy Mace insist that Liam should be using the female restroom. Really? A few questions: - Why? - How does this make sense to anyone beyond a kindergarten-level understanding of gender? - Have you actually thought this through, or is this just about scoring cheap political points? The point of posts like this is simple: Complex, multi-faceted issues can’t be reduced to oversimplified, black-and-white answers. But hey, I guess nuance isn’t for everyone.
0:11

Republicans like Nancy Mace insist that Liam should be using the female restroom. Really? A few questions: - Why? - How does this make sense to anyone beyond a kindergarten-level understanding of gender? - Have you actually thought this through, or is this just about scoring cheap political points? The point of posts like this is simple: Complex, multi-faceted issues can’t be reduced to oversimplified, black-and-white answers. But hey, I guess nuance isn’t for everyone.

Brian Krassenstein

621,330 次观看 • 1 年前

Tired of manual prompt tweaking? Watch the latest Llama tutorial on how to optimize your existing GPT or other LLM prompts for Llama with `llama-prompt-ops`, the open-source Python library! In this video, Partner Engineer Justin Lee demonstrates installation, project setup, migrating your first prompt, and analyzing performance gains. Watch now to discover: ✨ Why systematic prompt optimization is crucial for migrating from GPT to Llama. 💻 A live code walkthrough of `llama-prompt-ops` in action for a customer service classification task. Watch the full tutorial here:
0:10

Tired of manual prompt tweaking? Watch the latest Llama tutorial on how to optimize your existing GPT or other LLM prompts for Llama with `llama-prompt-ops`, the open-source Python library! In this video, Partner Engineer Justin Lee demonstrates installation, project setup, migrating your first prompt, and analyzing performance gains. Watch now to discover: ✨ Why systematic prompt optimization is crucial for migrating from GPT to Llama. 💻 A live code walkthrough of `llama-prompt-ops` in action for a customer service classification task. Watch the full tutorial here:

Meta for Developers

38,047 次观看 • 1 年前

What if you kept asking an LLM to "make it better"? In some recent work at FAIR, we investigate how we can efficiently use RL to fine-tune LLMs to iteratively self-improve on their previous solutions at inference-time. Training for iterated self-improvement can be costly. The naive approach to training for K self-improvement steps leads to K times the number of rollout steps per episode. We introduce Exploratory Iteration (ExIt), an RL-based automatic curriculum method that bootstraps diverse training distributions of self-improvement tasks by upcycling the LLM's own responses at previous turns as the starting points for both self-improvement and *self-divergence.* In order to decide what task to train on next, the curriculum prioritizes sampling of partial turn histories that led to higher return variance in its GRPO group (a learnability score that comes for free). This automatic curriculum over the bootstrapped task space teaches the model how to perform iterated self-improvement while only ever training the model on single-step self-improvement tasks. We look at ExIt's impact in both single-turn (contest math problems) and multi-turn (BFCLv3 multi-turn tasks), as well as MLE-bench, where the LLM is run in a search scaffold to produce solutions to real Kaggle competitions. Across these eval settings, we find ExIt produces models with greater capacity for inference-time self-improvement compared to GRPO. Notably, ExIt models can self-improve on test tasks for many more steps than the typical solution depth encountered during training, including a 22% improvement in MLE-bench performance compared to GRPO.
0:10

What if you kept asking an LLM to "make it better"? In some recent work at FAIR, we investigate how we can efficiently use RL to fine-tune LLMs to iteratively self-improve on their previous solutions at inference-time. Training for iterated self-improvement can be costly. The naive approach to training for K self-improvement steps leads to K times the number of rollout steps per episode. We introduce Exploratory Iteration (ExIt), an RL-based automatic curriculum method that bootstraps diverse training distributions of self-improvement tasks by upcycling the LLM's own responses at previous turns as the starting points for both self-improvement and *self-divergence.* In order to decide what task to train on next, the curriculum prioritizes sampling of partial turn histories that led to higher return variance in its GRPO group (a learnability score that comes for free). This automatic curriculum over the bootstrapped task space teaches the model how to perform iterated self-improvement while only ever training the model on single-step self-improvement tasks. We look at ExIt's impact in both single-turn (contest math problems) and multi-turn (BFCLv3 multi-turn tasks), as well as MLE-bench, where the LLM is run in a search scaffold to produce solutions to real Kaggle competitions. Across these eval settings, we find ExIt produces models with greater capacity for inference-time self-improvement compared to GRPO. Notably, ExIt models can self-improve on test tasks for many more steps than the typical solution depth encountered during training, including a 22% improvement in MLE-bench performance compared to GRPO.

Minqi Jiang

41,099 次观看 • 11 个月前

Create a 3D model from a single image, set of images or a text prompt in < 1 minute 😮‍💨 This new AI paper called CAT3D shows us that it’ll keep getting easier to produce 3D models from 2D images — whether it’s a sparser real world 3D scan (a few photos instead of hundreds) or your favorite 2D image generator like Midjourney (just an image). How does this magic work? “This architecture is similar to video diffusion models, but with camera pose embeddings for each image instead of time embeddings. The generated views are passed into a robust 3D reconstruction pipeline to create the 3D representation (Zip-NeRF or 3DGS)”
0:18

Create a 3D model from a single image, set of images or a text prompt in < 1 minute 😮‍💨 This new AI paper called CAT3D shows us that it’ll keep getting easier to produce 3D models from 2D images — whether it’s a sparser real world 3D scan (a few photos instead of hundreds) or your favorite 2D image generator like Midjourney (just an image). How does this magic work? “This architecture is similar to video diffusion models, but with camera pose embeddings for each image instead of time embeddings. The generated views are passed into a robust 3D reconstruction pipeline to create the 3D representation (Zip-NeRF or 3DGS)”

Bilawal Sidhu

92,792 次观看 • 2 年前

LLM Artifacts Connected to Andrej Karpathy's LLM Knowledge base idea, I've been building out a fun way to generate dynamic artifacts from these knowledge bases with the goal of discovering and revealing meaningful and deeper insights. LLM KBs are hard to consume for humans, as I think they are more built for agents. So the question is, what form would be useful for humans to take actions and make important decisions? That's what I am trying to figure out with these artifacts. The artifact example shows a pulse on HN discussions around AI-related stories. The insights can go deeper, of course, but this is already super fun and thought-provoking, like some of my favorite podcasts. The format and depth matter a lot. The aggregation skills of agents are outstanding if you tune the prompts and skill carefully. I built this artifact generator in a few minutes through an agent skill, but I feel like there are so many ways that LLM-generated information can be used and consumed. Like generating deeper insights and analysis, and things that are just not feasible for humans today. The generated artifact (including its data and design) serves as reusable templates or can be updated in real-time via auomations, which is something I am also working on. It is truly an insane way to monitor and track information. Better than a newsletter. Better than newspapers. There is something about this that gets me really excited about the future of AI agents for knowledge generation and discovery. Lots of hidden gems everywhere just waiting to be discovered and acted on if the information is presented correctly. This is not perfect. The format, style/prose can be improved, but this is easy to customize via skill. You can personalize it to your liking. I feel like these dynamic artifacts are going to emerge as a strong new medium to stay on the cutting edge of things, both for agents and humans. My target is research, of course. This was just a basic example. Besides animation, I am also targeting other components like voice, videos, images, slides, etc. This space is full of opportunities to explore. Skill for this coming soon.
0:18

LLM Artifacts Connected to Andrej Karpathy's LLM Knowledge base idea, I've been building out a fun way to generate dynamic artifacts from these knowledge bases with the goal of discovering and revealing meaningful and deeper insights. LLM KBs are hard to consume for humans, as I think they are more built for agents. So the question is, what form would be useful for humans to take actions and make important decisions? That's what I am trying to figure out with these artifacts. The artifact example shows a pulse on HN discussions around AI-related stories. The insights can go deeper, of course, but this is already super fun and thought-provoking, like some of my favorite podcasts. The format and depth matter a lot. The aggregation skills of agents are outstanding if you tune the prompts and skill carefully. I built this artifact generator in a few minutes through an agent skill, but I feel like there are so many ways that LLM-generated information can be used and consumed. Like generating deeper insights and analysis, and things that are just not feasible for humans today. The generated artifact (including its data and design) serves as reusable templates or can be updated in real-time via auomations, which is something I am also working on. It is truly an insane way to monitor and track information. Better than a newsletter. Better than newspapers. There is something about this that gets me really excited about the future of AI agents for knowledge generation and discovery. Lots of hidden gems everywhere just waiting to be discovered and acted on if the information is presented correctly. This is not perfect. The format, style/prose can be improved, but this is easy to customize via skill. You can personalize it to your liking. I feel like these dynamic artifacts are going to emerge as a strong new medium to stay on the cutting edge of things, both for agents and humans. My target is research, of course. This was just a basic example. Besides animation, I am also targeting other components like voice, videos, images, slides, etc. This space is full of opportunities to explore. Skill for this coming soon.

elvis

31,242 次观看 • 3 个月前

Again, in any other presidency this by itself would be impeachable. Trump tells a story of how he forced the Navy to “put a little more spirit in the hull” of a battleship. “I want that ship to look gorgeous… Forget about stealth…“ Endangering our military personnel for “great looking equipment” is a bizarre admission of deliberate negligence at best, if not borderline treason. “The new battleship that we have, which I've seen and helped design, I put a little more spirit in the hull, a little more, give me a little bit more hull. I want that ship to look gorgeous, you know. Forget about stealth, they say, "Oh, it's stealth." They design ugly equipment for stealth, well, we designed a great looking equipment, also stealth.”
0:18

Again, in any other presidency this by itself would be impeachable. Trump tells a story of how he forced the Navy to “put a little more spirit in the hull” of a battleship. “I want that ship to look gorgeous… Forget about stealth…“ Endangering our military personnel for “great looking equipment” is a bizarre admission of deliberate negligence at best, if not borderline treason. “The new battleship that we have, which I've seen and helped design, I put a little more spirit in the hull, a little more, give me a little bit more hull. I want that ship to look gorgeous, you know. Forget about stealth, they say, "Oh, it's stealth." They design ugly equipment for stealth, well, we designed a great looking equipment, also stealth.”

Jim Stewartson, Decelerationist 🇨🇦🇺🇦🇺🇸

154,578 次观看 • 5 个月前

🇨🇳 Another great Chinese Model, OmniHuman-1.5 from ByteDance Turns 1 image plus a voice track into expressive avatar video by pairing a System 1 and System 2 inspired planner with a Diffusion Transformer, Produces coherent motion for over 1 minute with moving camera and multi character scenes. Most avatar models move to the beat of the audio but miss meaning, so gestures feel generic and emotions feel shallow. The fix here is a Multimodal LLM planner that listens to the speech and drafts a structured plan describing intent, emotions, beats, and high level actions, which gives the motion engine clear semantic targets instead of only rhythm. The motion engine is a Multimodal Diffusion Transformer that fuses the plan with audio, the single reference image, and optional text prompts, then synthesizes continuous body, face, and head motion that matches both words and tone. A key trick is a Pseudo Last Frame, a synthetic target that summarizes the next expected state, which stabilizes fusion across modalities and keeps motion consistent over long spans. From just 1 image and speech, the system outputs speaking avatars with synchronized lips, context aware gestures, and continuous camera movement, and it also supports multi character interactions without manual choreography. Reported results show strong lip sync accuracy, high video quality, natural motion, and close match to text prompts, and the same setup works on nonhuman characters too.
0:18

🇨🇳 Another great Chinese Model, OmniHuman-1.5 from ByteDance Turns 1 image plus a voice track into expressive avatar video by pairing a System 1 and System 2 inspired planner with a Diffusion Transformer, Produces coherent motion for over 1 minute with moving camera and multi character scenes. Most avatar models move to the beat of the audio but miss meaning, so gestures feel generic and emotions feel shallow. The fix here is a Multimodal LLM planner that listens to the speech and drafts a structured plan describing intent, emotions, beats, and high level actions, which gives the motion engine clear semantic targets instead of only rhythm. The motion engine is a Multimodal Diffusion Transformer that fuses the plan with audio, the single reference image, and optional text prompts, then synthesizes continuous body, face, and head motion that matches both words and tone. A key trick is a Pseudo Last Frame, a synthetic target that summarizes the next expected state, which stabilizes fusion across modalities and keeps motion consistent over long spans. From just 1 image and speech, the system outputs speaking avatars with synchronized lips, context aware gestures, and continuous camera movement, and it also supports multi character interactions without manual choreography. Reported results show strong lip sync accuracy, high video quality, natural motion, and close match to text prompts, and the same setup works on nonhuman characters too.

Rohan Paul

63,859 次观看 • 11 个月前

Testing the new Gemma 4 12B (QAT) vision and OCR capabilities locally with LM Studio. # The setup: - GPU: NVIDIA RTX 4060 (8GB VRAM) - CPU: Intel i7 - Runner: LM Studio - Config: 32k context, 38 layers offloaded, Flash Attention enabled - Speed: ~14 tokens/sec decode throughput # The test: I gave it a screenshot of Google AI Studio. Prompt: "clone this. give me a single html file" # The result: A solid one shot replication. It successfully mapped out the layout, recognized the UI text, and structured the divs correctly, with only minor differences from the original. Results available at the end of the video. Quite capable for a 12B model running on budget consumer hardware. A gpu that costs only $300. # Why the architecture under the hood is notable: Unlike traditional models that rely on heavy, separate vision and audio encoders, Gemma 4 12B uses a unified, encoder free architecture. It bypasses separate multi stage encoders. Uses a 35M parameter vision embedder to project raw 48x48 pixel patches directly to the LLM hidden dimension. Local multimodal development is becoming highly accessible on standard hardware. If you've spun up Gemma 4 12B locally, what setup are you using and what kind of throughput are you seeing?
0:16

Testing the new Gemma 4 12B (QAT) vision and OCR capabilities locally with LM Studio. # The setup: - GPU: NVIDIA RTX 4060 (8GB VRAM) - CPU: Intel i7 - Runner: LM Studio - Config: 32k context, 38 layers offloaded, Flash Attention enabled - Speed: ~14 tokens/sec decode throughput # The test: I gave it a screenshot of Google AI Studio. Prompt: "clone this. give me a single html file" # The result: A solid one shot replication. It successfully mapped out the layout, recognized the UI text, and structured the divs correctly, with only minor differences from the original. Results available at the end of the video. Quite capable for a 12B model running on budget consumer hardware. A gpu that costs only $300. # Why the architecture under the hood is notable: Unlike traditional models that rely on heavy, separate vision and audio encoders, Gemma 4 12B uses a unified, encoder free architecture. It bypasses separate multi stage encoders. Uses a 35M parameter vision embedder to project raw 48x48 pixel patches directly to the LLM hidden dimension. Local multimodal development is becoming highly accessible on standard hardware. If you've spun up Gemma 4 12B locally, what setup are you using and what kind of throughput are you seeing?

Alok

25,717 次观看 • 1 个月前

Qualia has been selected for the Google DeepMind Robotics Program. We train embodied models that put a robot on a real manual task and make it work, on the floor, not in a demo. Foundation models and reasoning are where robotics is heading, and doing that work alongside DeepMind, who are pushing this frontier, is exactly where we want to be. If you are a company looking to see how a new generation of robots can help your manual tasks, contact us at hello@qualiastudios.dev More soon
0:16

Qualia has been selected for the Google DeepMind Robotics Program. We train embodied models that put a robot on a real manual task and make it work, on the floor, not in a demo. Foundation models and reasoning are where robotics is heading, and doing that work alongside DeepMind, who are pushing this frontier, is exactly where we want to be. If you are a company looking to see how a new generation of robots can help your manual tasks, contact us at [email protected] More soon

Qualia

87,693 次观看 • 1 个月前