Loading video...

Video Failed to Load

Go Home

Reinforcement Learning (RL) has long been the dominant method for fine-tuning, powering many state-of-the-art LLMs. Methods like PPO and GRPO explore in action space. But can we instead explore directly in parameter space? YES we can. We propose a scalable framework for full-parameter fine-tuning using Evolution Strategies (ES). By...

yule_gan's profile picture

Yulu Gan

3,704 subscribers

414,920 views • 8 months ago •via X (Twitter)

Science & TechnologyEducation
LIVE

Anya Rossi• Live Now

Private livecam show

0 Comments

No comments available

Comments from the original post will appear here

Related Videos

🚀New paper out - We present Video-MSG (Multimodal Sketch Guidance), a novel planning-based training-free guidance method for T2V models, improving control of spatial layout and object trajectories. 🔧 Key idea: • Generate a Video Sketch — a spatio-temporal plan with background, foreground, and motion in the pixel space. • Encode this structure directly into the latent space of the diffusion model during generation, which does not require fine-tuning or additional memory during inference. 🧵
0:12

🚀New paper out - We present Video-MSG (Multimodal Sketch Guidance), a novel planning-based training-free guidance method for T2V models, improving control of spatial layout and object trajectories. 🔧 Key idea: • Generate a Video Sketch — a spatio-temporal plan with background, foreground, and motion in the pixel space. • Encode this structure directly into the latent space of the diffusion model during generation, which does not require fine-tuning or additional memory during inference. 🧵

Jialu Li

35,060 views • 1 year ago

Full Fine-tuning vs. Freezing Layers. Interact 👉 and == Full Fine-tuning == A real network has many — three layers in this example, billions of parameters in a production model. What does fine-tuning look like when you update all of them? That’s full fine-tuning: continue training every weight in the pretrained network on your new task. Every layer’s W gets its own ΔW. Nothing is frozen — every parameter is in play. Think of an MLP as a chain of prerequisites leading to an advanced course. Layer 1 might be Linear Algebra, layer 2 Probability, layer 3 Advanced Machine Learning — each one building on what came before. Fine-tuning is what happens during graduate study: the foundations are already there from undergrad, so you’re not re-learning. Full fine-tuning is reviewing every prerequisite to see what new topics have appeared and what discoveries the field has made since the last time you sat through them. Effective — but exhausting. This diagram shows the same three-layer MLP twice, side by side. On the left, the pretrained network runs on input X: three weight matrices W₁, W₂, W₃, each followed by a ReLU activation. Full fine-tuning gives the model the most freedom to specialize. Every parameter can move — and every parameter that can move must be stored. But not every prerequisite needs revisiting. The further you go back in the chain, the less the material has changed since pretraining — the linear-algebra basics under your computer-vision course are largely the same as they ever were. The next page does exactly that: freeze the prerequisites that haven’t moved, and only refresh the advanced one closest to your specialization. == Freezing Layers == Full fine-tuning reviewed every prerequisite — Linear Algebra, Probability, Advanced ML — to refresh each subject with the latest topics. Effective, but exhausting. Then you realize something. The prerequisites haven’t actually changed that much. Linear Algebra is still Linear Algebra; the matrix decompositions you learned still hold. Probability is still Probability; the distributions and Bayes’ rule haven’t moved. Almost all the new material — the new ideas, the recent discoveries — lives in the advanced layer at the top. That’s freezing layers: keep the prerequisite layers fixed at their pretrained state, and only update the advanced one. In the diagram below, W1​ and W2​ — the foundational prerequisites — stay frozen. Only W3​ — the layer closest to your task-specific output — gets a ΔW.
0:16

Full Fine-tuning vs. Freezing Layers. Interact 👉 and == Full Fine-tuning == A real network has many — three layers in this example, billions of parameters in a production model. What does fine-tuning look like when you update all of them? That’s full fine-tuning: continue training every weight in the pretrained network on your new task. Every layer’s W gets its own ΔW. Nothing is frozen — every parameter is in play. Think of an MLP as a chain of prerequisites leading to an advanced course. Layer 1 might be Linear Algebra, layer 2 Probability, layer 3 Advanced Machine Learning — each one building on what came before. Fine-tuning is what happens during graduate study: the foundations are already there from undergrad, so you’re not re-learning. Full fine-tuning is reviewing every prerequisite to see what new topics have appeared and what discoveries the field has made since the last time you sat through them. Effective — but exhausting. This diagram shows the same three-layer MLP twice, side by side. On the left, the pretrained network runs on input X: three weight matrices W₁, W₂, W₃, each followed by a ReLU activation. Full fine-tuning gives the model the most freedom to specialize. Every parameter can move — and every parameter that can move must be stored. But not every prerequisite needs revisiting. The further you go back in the chain, the less the material has changed since pretraining — the linear-algebra basics under your computer-vision course are largely the same as they ever were. The next page does exactly that: freeze the prerequisites that haven’t moved, and only refresh the advanced one closest to your specialization. == Freezing Layers == Full fine-tuning reviewed every prerequisite — Linear Algebra, Probability, Advanced ML — to refresh each subject with the latest topics. Effective, but exhausting. Then you realize something. The prerequisites haven’t actually changed that much. Linear Algebra is still Linear Algebra; the matrix decompositions you learned still hold. Probability is still Probability; the distributions and Bayes’ rule haven’t moved. Almost all the new material — the new ideas, the recent discoveries — lives in the advanced layer at the top. That’s freezing layers: keep the prerequisite layers fixed at their pretrained state, and only update the advanced one. In the diagram below, W1​ and W2​ — the foundational prerequisites — stay frozen. Only W3​ — the layer closest to your task-specific output — gets a ΔW.

Tom Yeh

27,225 views • 1 month ago

What if we could teach an AI to master the strategic game of 2048 through pure reinforcement learning? I did exactly that with "Agent 2048" - fine-tuning Qwen 7B model using GRPO to develop spatial reasoning and merge strategies with zero prior gameplay SFTdata! Thanks to Hugging Face and Unsloth AI for their easy to use implementation kalomaze and will brown you might like this :)
0:20

What if we could teach an AI to master the strategic game of 2048 through pure reinforcement learning? I did exactly that with "Agent 2048" - fine-tuning Qwen 7B model using GRPO to develop spatial reasoning and merge strategies with zero prior gameplay SFTdata! Thanks to Hugging Face and Unsloth AI for their easy to use implementation kalomaze and will brown you might like this :)

Hrishbh Dalal

30,851 views • 1 year ago

New project! Flow Policy Gradients for Robot Control tldr; a simple online RL recipe for training and fine-tuning flow policies for robots co-led w/ Hongsuk Benjamin Choi:
0:18

New project! Flow Policy Gradients for Robot Control tldr; a simple online RL recipe for training and fine-tuning flow policies for robots co-led w/ Hongsuk Benjamin Choi:

Brent Yi

73,900 views • 4 months ago

Fine tuning…the Ferrari…
0:13

Fine tuning…the Ferrari…

Stefan Johansson

23,708 views • 3 years ago

Fine-tuning the details
0:17

Fine-tuning the details

Washington Commanders

95,340 views • 1 year ago

fine tuning rotoscope ai video asset for the next 1/1
0:19

fine tuning rotoscope ai video asset for the next 1/1

神経色

18,861 views • 1 year ago

Model-Free Reinforcement Learning (MFRL) has been alluring, especially with supercharged compute with physics on GPU. However, the methods use 0-th order gradients, and are often not the best optimizers. Can we do better than PPO in continuous control for robotics? Turns out yes! 🥳 tl;dr: Faster, better RL than PPO in continuous control 💪 The answer lies in using more information from the simulation. We are juicing the simulation on GPU as it is, why not use it for gradients as well? This has been a driving question in a series of our works. We first studied this problem in ICLR 2022 paper on Short Horizon Actor Critic Naive gradient based methods are stuck in local minima and have exploding/vanishing gradients. SHAC solved this problem truncated rollouts and model based value estimation, where the model is Differentiable Sim. This boosted sample efficiency and wall-clock time immensely especially in high dimensional systems such as humanoids Yet, given enough compute PPO often caught up. Our follow up paper on on Adaptive Horizon Actor Critic at ICML 2024 discovers the cause and provides a fix. However, we find that even when given ground-truth dynamics, not all gradients are useful due to sample error. 1st-Order Model-Based Reinforcement Learning methods employing differentiable simulation provide gradients with reduced variance but are susceptible to bias in scenarios involving stiff dynamics, such as physical contact. We find that back-propagating through contact and long trajectories drastically reduces gradient accuracy. Using this insight, we propose AHAC to dynamically adapt its roll-out horizon to avoid differentiating through stiff contact. AHAC is a first-order model-based RL algorithm that learns high-dimensional tasks in minutes (wall clock) and outperforms PPO by 40%, even in the limit of data provided to PPO. This work is led by Ignat Georgiev alongside Krishnan Srinivasan, Jie Xu, Eric Heiden and ample assistance from warp team at NVIDIA Robotics (Miles Macklin)
0:10

Model-Free Reinforcement Learning (MFRL) has been alluring, especially with supercharged compute with physics on GPU. However, the methods use 0-th order gradients, and are often not the best optimizers. Can we do better than PPO in continuous control for robotics? Turns out yes! 🥳 tl;dr: Faster, better RL than PPO in continuous control 💪 The answer lies in using more information from the simulation. We are juicing the simulation on GPU as it is, why not use it for gradients as well? This has been a driving question in a series of our works. We first studied this problem in ICLR 2022 paper on Short Horizon Actor Critic Naive gradient based methods are stuck in local minima and have exploding/vanishing gradients. SHAC solved this problem truncated rollouts and model based value estimation, where the model is Differentiable Sim. This boosted sample efficiency and wall-clock time immensely especially in high dimensional systems such as humanoids Yet, given enough compute PPO often caught up. Our follow up paper on on Adaptive Horizon Actor Critic at ICML 2024 discovers the cause and provides a fix. However, we find that even when given ground-truth dynamics, not all gradients are useful due to sample error. 1st-Order Model-Based Reinforcement Learning methods employing differentiable simulation provide gradients with reduced variance but are susceptible to bias in scenarios involving stiff dynamics, such as physical contact. We find that back-propagating through contact and long trajectories drastically reduces gradient accuracy. Using this insight, we propose AHAC to dynamically adapt its roll-out horizon to avoid differentiating through stiff contact. AHAC is a first-order model-based RL algorithm that learns high-dimensional tasks in minutes (wall clock) and outperforms PPO by 40%, even in the limit of data provided to PPO. This work is led by Ignat Georgiev alongside Krishnan Srinivasan, Jie Xu, Eric Heiden and ample assistance from warp team at NVIDIA Robotics (Miles Macklin)

Animesh Garg

52,279 views • 2 years ago

Fine-tuning the strokes! 🎶 #WhistlePodu #Yellove
0:13

Fine-tuning the strokes! 🎶 #WhistlePodu #Yellove

Chennai Super Kings

58,903 views • 1 year ago

RF-DETR converges faster than other detectors RF-DETR-B after 10 epochs of fine-tuning vs. YOLO11m after 30 epochs of fine-tuning link:
0:19

RF-DETR converges faster than other detectors RF-DETR-B after 10 epochs of fine-tuning vs. YOLO11m after 30 epochs of fine-tuning link:

SkalskiP

30,515 views • 1 year ago

1/ Happy to share UniDisc - Unified Multimodal Discrete Diffusion – We train a 1.5 billion parameter transformer model from scratch on 250 million image/caption pairs using a **discrete diffusion objective**. Our model has all the benefits of diffusion models but now in multimodal space! - flexible compute-quality tradeoff, zero-shot inpainting and editing, better control via classifier-free guidance and lower latency! We open source everything - our code, weights and the training dataset.
0:10

1/ Happy to share UniDisc - Unified Multimodal Discrete Diffusion – We train a 1.5 billion parameter transformer model from scratch on 250 million image/caption pairs using a **discrete diffusion objective**. Our model has all the benefits of diffusion models but now in multimodal space! - flexible compute-quality tradeoff, zero-shot inpainting and editing, better control via classifier-free guidance and lower latency! We open source everything - our code, weights and the training dataset.

Mihir Prabhudesai

104,862 views • 1 year ago

Transformers & LLMs cheatsheets for Stanford's CME-295! Covering tokenization, self-attention, prompting, fine-tuning, LLM-as-a-judge, RAG, AI Agents, and reasoning models. 100% free and open-source.
0:15

Transformers & LLMs cheatsheets for Stanford's CME-295! Covering tokenization, self-attention, prompting, fine-tuning, LLM-as-a-judge, RAG, AI Agents, and reasoning models. 100% free and open-source.

Akshay 🚀

101,587 views • 1 year ago

Google released Gemma 3 270M, a new model for hyper-efficient local AI! We'll fine-tune this model and make it very smart at playing chess and predict the next move. Tech stack: - Unsloth AI for efficient fine-tuning. - Hugging Face transformers to run it locally. Let's go! 🚀
0:11

Google released Gemma 3 270M, a new model for hyper-efficient local AI! We'll fine-tune this model and make it very smart at playing chess and predict the next move. Tech stack: - Unsloth AI for efficient fine-tuning. - Hugging Face transformers to run it locally. Let's go! 🚀

Akshay 🚀

145,637 views • 10 months ago

SuiRWA is training its investment AI agents on extensive data, using neural networks for pattern recognition. They learn via supervised & unsupervised methods, with fine-tuning for market analysis, ensuring accuracy and minimizing bias. A revolution in investing might be here.
0:16

SuiRWA is training its investment AI agents on extensive data, using neural networks for pattern recognition. They learn via supervised & unsupervised methods, with fine-tuning for market analysis, ensuring accuracy and minimizing bias. A revolution in investing might be here.

Sui RWA

16,165 views • 1 year ago

Carlos: “I think we did some progress there. I feel a lot better with the car. Just fine tuning now.” Proud of him and the team efforts, but that miscommunication might cost them a penalty and honestly is frustrating
0:11

Carlos: “I think we did some progress there. I feel a lot better with the car. Just fine tuning now.” Proud of him and the team efforts, but that miscommunication might cost them a penalty and honestly is frustrating

ari

11,265 views • 1 year ago

EDGS: Eliminating Densification for Efficient Convergence of 3DGS Contributions: • We show that initial triangulation based on 2D correspondences can replace the incremental refinement process, fundamentally changing how 3DGS models allocate resources. • Our method reduces the path each Gaussian must travel in parameter space. Careful initialization not only accelerates convergence but also guides optimization toward a convergence point corresponding to lower reconstruction error and thus higher reconstruction quality. • Our approach outperforms both speed-optimized and quality-focused state-of-the-art models while using only half the splats of standard 3DGS. By improving initialization rather than altering the optimization process, this method is compatible with other 3DGS acceleration techniques, making it a flexible enhancement to existing models.
0:15

EDGS: Eliminating Densification for Efficient Convergence of 3DGS Contributions: • We show that initial triangulation based on 2D correspondences can replace the incremental refinement process, fundamentally changing how 3DGS models allocate resources. • Our method reduces the path each Gaussian must travel in parameter space. Careful initialization not only accelerates convergence but also guides optimization toward a convergence point corresponding to lower reconstruction error and thus higher reconstruction quality. • Our approach outperforms both speed-optimized and quality-focused state-of-the-art models while using only half the splats of standard 3DGS. By improving initialization rather than altering the optimization process, this method is compatible with other 3DGS acceleration techniques, making it a flexible enhancement to existing models.

MrNeRF

124,002 views • 1 year ago

The Sabotaging Practice of Over Supply and Sameness in the NFT Space. The current zeitgeist of the NFT space is that the same artists are doing the same kind of work five times a year, with project after project leaving a trail of disappointment and discontent among collectors and all of us watching in disbelief as huge resources are extracted from the space over work that feels like it could be left as an "artist study." I understand that you can do what you want with your money as collectors, but we are killing the whole space with this incestuous practice. No artist is that prolific to be able to do 5 collections of 100+ pieces each every year and actually deliver innovation and some kind of creative evolution. Of course, they can pretend play that the work has something new, but there is no precedent nor proof that that has ever happened in the speed that it happens in the NFT space. Again, people are free to through away their resources on whatever they want but with this way of doing things, we more and more are going to start seeing the consequences. Oh! There are consequences? Yes. Maybe unintended, but there are. Let's see. Let's start with the loss of belief in the NFT space as somewhere where emerging artists can come and find support for their experiments. Why even bother to bring experiments, innovation, and new ways to think of art on the blockchain if the same people have all the collectors hypnotized with their magical flutes? Why even try to come to a space where taking risks and challenging the status quo (the mission of art!!!) is overlooked? This makes the NFT space a social club and not a space for art. I guess it is fine, but IMO it is a recipe for disaster. New collectors stay away because the art will slowly but surely become stale and un-challenging. Why even bother to come and see what is happening here if you can't, as a collector, see new weird and up-and-coming artists? The amount of noise emitted by the same artists doing the same art over and over, drowns out any new voices. Again. A recipe for disaster. The NFT space is becoming a space of disappointment and doubt. We think that collections going to zero one after the other, over and over, is not damaging? I feel we are kidding ourselves. Disappointment piles up, and again, the people who will hurt are the emerging artists, the new blood, the ones who are willing to risk the most and, in return, put fire in this cold space of sameness. I love this space—don't get me wrong—it has changed my life, and I believe it has a ton of potential, but things need to change for it to become a beacon of light in art. But we need to support new voices. We need to support new ideas. The challenge is huge. I hope to contribute all I can to this change. I hope more and more see how exciting it is to go out and try to discover what else is out there and move this space forward. But again, I understand the leaps of faith needed, but if there is a space that is based on that, it's the NFT space...so there is hope. We will see. 📺by Boldtron
0:12

The Sabotaging Practice of Over Supply and Sameness in the NFT Space. The current zeitgeist of the NFT space is that the same artists are doing the same kind of work five times a year, with project after project leaving a trail of disappointment and discontent among collectors and all of us watching in disbelief as huge resources are extracted from the space over work that feels like it could be left as an "artist study." I understand that you can do what you want with your money as collectors, but we are killing the whole space with this incestuous practice. No artist is that prolific to be able to do 5 collections of 100+ pieces each every year and actually deliver innovation and some kind of creative evolution. Of course, they can pretend play that the work has something new, but there is no precedent nor proof that that has ever happened in the speed that it happens in the NFT space. Again, people are free to through away their resources on whatever they want but with this way of doing things, we more and more are going to start seeing the consequences. Oh! There are consequences? Yes. Maybe unintended, but there are. Let's see. Let's start with the loss of belief in the NFT space as somewhere where emerging artists can come and find support for their experiments. Why even bother to bring experiments, innovation, and new ways to think of art on the blockchain if the same people have all the collectors hypnotized with their magical flutes? Why even try to come to a space where taking risks and challenging the status quo (the mission of art!!!) is overlooked? This makes the NFT space a social club and not a space for art. I guess it is fine, but IMO it is a recipe for disaster. New collectors stay away because the art will slowly but surely become stale and un-challenging. Why even bother to come and see what is happening here if you can't, as a collector, see new weird and up-and-coming artists? The amount of noise emitted by the same artists doing the same art over and over, drowns out any new voices. Again. A recipe for disaster. The NFT space is becoming a space of disappointment and doubt. We think that collections going to zero one after the other, over and over, is not damaging? I feel we are kidding ourselves. Disappointment piles up, and again, the people who will hurt are the emerging artists, the new blood, the ones who are willing to risk the most and, in return, put fire in this cold space of sameness. I love this space—don't get me wrong—it has changed my life, and I believe it has a ton of potential, but things need to change for it to become a beacon of light in art. But we need to support new voices. We need to support new ideas. The challenge is huge. I hope to contribute all I can to this change. I hope more and more see how exciting it is to go out and try to discover what else is out there and move this space forward. But again, I understand the leaps of faith needed, but if there is a space that is based on that, it's the NFT space...so there is hope. We will see. 📺by Boldtron

alejandro cartagena

98,261 views • 2 years ago

WishList Update: We finished our implementation of the platform-native Scroll View component and can now use smooth scrolling and bounce events - awesome work Janic! ⛹️‍♂️ Just a little bit of fine-tuning and we'll be able to launch WishList this summer. Stay tuned! 👀🚀
0:18

WishList Update: We finished our implementation of the platform-native Scroll View component and can now use smooth scrolling and bounce events - awesome work Janic! ⛹️‍♂️ Just a little bit of fine-tuning and we'll be able to launch WishList this summer. Stay tuned! 👀🚀

Margelo

15,757 views • 2 years ago

Fine-tuning his skating skills with the master, Brian Orser! #TheCricketClub
0:17

Fine-tuning his skating skills with the master, Brian Orser! #TheCricketClub

Toronto Cricket Club

91,417 views • 2 years ago

Fine-tuning cave level's background assets, light and atmosphere ✨ #WishlistWednesday #pixelart #metroidvania #indiedev
0:12

Fine-tuning cave level's background assets, light and atmosphere ✨ #WishlistWednesday #pixelart #metroidvania #indiedev

Divinum | Play Demo on Steam

14,815 views • 7 months ago