Загрузка видео...

Не удалось загрузить видео

На главную

Reinforcement Learning (RL) has long been the dominant method for fine-tuning, powering many state-of-the-art LLMs. Methods like PPO and GRPO explore in action space. But can we instead explore directly in parameter space? YES we can. We propose a scalable framework for full-parameter fine-tuning using Evolution Strategies (ES). By...

yule_gan's profile picture

Yulu Gan

3,803 subscribers

415,936 просмотров • 1 год назад •via X (Twitter)

Наука и технологииОбразование
LIVE

Anya Rossi• Live Now

Private livecam show

Комментарии: 0

Нет доступных комментариев

Здесь появятся комментарии из оригинального поста

Похожие видео

🚀New paper out - We present Video-MSG (Multimodal Sketch Guidance), a novel planning-based training-free guidance method for T2V models, improving control of spatial layout and object trajectories. 🔧 Key idea: • Generate a Video Sketch — a spatio-temporal plan with background, foreground, and motion in the pixel space. • Encode this structure directly into the latent space of the diffusion model during generation, which does not require fine-tuning or additional memory during inference. 🧵
0:12

🚀New paper out - We present Video-MSG (Multimodal Sketch Guidance), a novel planning-based training-free guidance method for T2V models, improving control of spatial layout and object trajectories. 🔧 Key idea: • Generate a Video Sketch — a spatio-temporal plan with background, foreground, and motion in the pixel space. • Encode this structure directly into the latent space of the diffusion model during generation, which does not require fine-tuning or additional memory during inference. 🧵

Jialu Li

35,258 просмотров • 1 год назад

Full Fine-tuning vs. Freezing Layers. Interact 👉 and == Full Fine-tuning == A real network has many — three layers in this example, billions of parameters in a production model. What does fine-tuning look like when you update all of them? That’s full fine-tuning: continue training every weight in the pretrained network on your new task. Every layer’s W gets its own ΔW. Nothing is frozen — every parameter is in play. Think of an MLP as a chain of prerequisites leading to an advanced course. Layer 1 might be Linear Algebra, layer 2 Probability, layer 3 Advanced Machine Learning — each one building on what came before. Fine-tuning is what happens during graduate study: the foundations are already there from undergrad, so you’re not re-learning. Full fine-tuning is reviewing every prerequisite to see what new topics have appeared and what discoveries the field has made since the last time you sat through them. Effective — but exhausting. This diagram shows the same three-layer MLP twice, side by side. On the left, the pretrained network runs on input X: three weight matrices W₁, W₂, W₃, each followed by a ReLU activation. Full fine-tuning gives the model the most freedom to specialize. Every parameter can move — and every parameter that can move must be stored. But not every prerequisite needs revisiting. The further you go back in the chain, the less the material has changed since pretraining — the linear-algebra basics under your computer-vision course are largely the same as they ever were. The next page does exactly that: freeze the prerequisites that haven’t moved, and only refresh the advanced one closest to your specialization. == Freezing Layers == Full fine-tuning reviewed every prerequisite — Linear Algebra, Probability, Advanced ML — to refresh each subject with the latest topics. Effective, but exhausting. Then you realize something. The prerequisites haven’t actually changed that much. Linear Algebra is still Linear Algebra; the matrix decompositions you learned still hold. Probability is still Probability; the distributions and Bayes’ rule haven’t moved. Almost all the new material — the new ideas, the recent discoveries — lives in the advanced layer at the top. That’s freezing layers: keep the prerequisite layers fixed at their pretrained state, and only update the advanced one. In the diagram below, W1​ and W2​ — the foundational prerequisites — stay frozen. Only W3​ — the layer closest to your task-specific output — gets a ΔW.
0:16

Full Fine-tuning vs. Freezing Layers. Interact 👉 and == Full Fine-tuning == A real network has many — three layers in this example, billions of parameters in a production model. What does fine-tuning look like when you update all of them? That’s full fine-tuning: continue training every weight in the pretrained network on your new task. Every layer’s W gets its own ΔW. Nothing is frozen — every parameter is in play. Think of an MLP as a chain of prerequisites leading to an advanced course. Layer 1 might be Linear Algebra, layer 2 Probability, layer 3 Advanced Machine Learning — each one building on what came before. Fine-tuning is what happens during graduate study: the foundations are already there from undergrad, so you’re not re-learning. Full fine-tuning is reviewing every prerequisite to see what new topics have appeared and what discoveries the field has made since the last time you sat through them. Effective — but exhausting. This diagram shows the same three-layer MLP twice, side by side. On the left, the pretrained network runs on input X: three weight matrices W₁, W₂, W₃, each followed by a ReLU activation. Full fine-tuning gives the model the most freedom to specialize. Every parameter can move — and every parameter that can move must be stored. But not every prerequisite needs revisiting. The further you go back in the chain, the less the material has changed since pretraining — the linear-algebra basics under your computer-vision course are largely the same as they ever were. The next page does exactly that: freeze the prerequisites that haven’t moved, and only refresh the advanced one closest to your specialization. == Freezing Layers == Full fine-tuning reviewed every prerequisite — Linear Algebra, Probability, Advanced ML — to refresh each subject with the latest topics. Effective, but exhausting. Then you realize something. The prerequisites haven’t actually changed that much. Linear Algebra is still Linear Algebra; the matrix decompositions you learned still hold. Probability is still Probability; the distributions and Bayes’ rule haven’t moved. Almost all the new material — the new ideas, the recent discoveries — lives in the advanced layer at the top. That’s freezing layers: keep the prerequisite layers fixed at their pretrained state, and only update the advanced one. In the diagram below, W1​ and W2​ — the foundational prerequisites — stay frozen. Only W3​ — the layer closest to your task-specific output — gets a ΔW.

Tom Yeh

27,740 просмотров • 6 месяцев назад

What if we could teach an AI to master the strategic game of 2048 through pure reinforcement learning? I did exactly that with "Agent 2048" - fine-tuning Qwen 7B model using GRPO to develop spatial reasoning and merge strategies with zero prior gameplay SFTdata! Thanks to Hugging Face and Unsloth AI for their easy to use implementation kalomaze and will brown you might like this :)
0:20

What if we could teach an AI to master the strategic game of 2048 through pure reinforcement learning? I did exactly that with "Agent 2048" - fine-tuning Qwen 7B model using GRPO to develop spatial reasoning and merge strategies with zero prior gameplay SFTdata! Thanks to Hugging Face and Unsloth AI for their easy to use implementation kalomaze and will brown you might like this :)

Hrishbh Dalal

31,863 просмотров • 1 год назад

New project! Flow Policy Gradients for Robot Control tldr; a simple online RL recipe for training and fine-tuning flow policies for robots co-led w/ Hongsuk Benjamin Choi:
0:18

New project! Flow Policy Gradients for Robot Control tldr; a simple online RL recipe for training and fine-tuning flow policies for robots co-led w/ Hongsuk Benjamin Choi:

Brent Yi

94,426 просмотров • 8 месяцев назад

Fine tuning…the Ferrari…
0:13

Fine tuning…the Ferrari…

Stefan Johansson

23,710 просмотров • 3 лет назад

Fine-tuning the details
0:17

Fine-tuning the details

Washington Commanders

95,545 просмотров • 1 год назад

Won india's first AI-only Hackathon AI-Mayhem in Bangalore 🏆 48hrs full of fine-tuning and LLMs Zo House BLR thanks Harkirat Singh sorr #100xDevs 🏆🏆
0:10

Won india's first AI-only Hackathon AI-Mayhem in Bangalore 🏆 48hrs full of fine-tuning and LLMs Zo House BLR thanks Harkirat Singh sorr #100xDevs 🏆🏆

Piyush 100x

11,194 просмотров • 2 лет назад

Model-Free Reinforcement Learning (MFRL) has been alluring, especially with supercharged compute with physics on GPU. However, the methods use 0-th order gradients, and are often not the best optimizers. Can we do better than PPO in continuous control for robotics? Turns out yes! 🥳 tl;dr: Faster, better RL than PPO in continuous control 💪 The answer lies in using more information from the simulation. We are juicing the simulation on GPU as it is, why not use it for gradients as well? This has been a driving question in a series of our works. We first studied this problem in ICLR 2022 paper on Short Horizon Actor Critic Naive gradient based methods are stuck in local minima and have exploding/vanishing gradients. SHAC solved this problem truncated rollouts and model based value estimation, where the model is Differentiable Sim. This boosted sample efficiency and wall-clock time immensely especially in high dimensional systems such as humanoids Yet, given enough compute PPO often caught up. Our follow up paper on on Adaptive Horizon Actor Critic at ICML 2024 discovers the cause and provides a fix. However, we find that even when given ground-truth dynamics, not all gradients are useful due to sample error. 1st-Order Model-Based Reinforcement Learning methods employing differentiable simulation provide gradients with reduced variance but are susceptible to bias in scenarios involving stiff dynamics, such as physical contact. We find that back-propagating through contact and long trajectories drastically reduces gradient accuracy. Using this insight, we propose AHAC to dynamically adapt its roll-out horizon to avoid differentiating through stiff contact. AHAC is a first-order model-based RL algorithm that learns high-dimensional tasks in minutes (wall clock) and outperforms PPO by 40%, even in the limit of data provided to PPO. This work is led by Ignat Georgiev alongside Krishnan Srinivasan, Jie Xu, Eric Heiden and ample assistance from warp team at NVIDIA Robotics (Miles Macklin)
0:10

Model-Free Reinforcement Learning (MFRL) has been alluring, especially with supercharged compute with physics on GPU. However, the methods use 0-th order gradients, and are often not the best optimizers. Can we do better than PPO in continuous control for robotics? Turns out yes! 🥳 tl;dr: Faster, better RL than PPO in continuous control 💪 The answer lies in using more information from the simulation. We are juicing the simulation on GPU as it is, why not use it for gradients as well? This has been a driving question in a series of our works. We first studied this problem in ICLR 2022 paper on Short Horizon Actor Critic Naive gradient based methods are stuck in local minima and have exploding/vanishing gradients. SHAC solved this problem truncated rollouts and model based value estimation, where the model is Differentiable Sim. This boosted sample efficiency and wall-clock time immensely especially in high dimensional systems such as humanoids Yet, given enough compute PPO often caught up. Our follow up paper on on Adaptive Horizon Actor Critic at ICML 2024 discovers the cause and provides a fix. However, we find that even when given ground-truth dynamics, not all gradients are useful due to sample error. 1st-Order Model-Based Reinforcement Learning methods employing differentiable simulation provide gradients with reduced variance but are susceptible to bias in scenarios involving stiff dynamics, such as physical contact. We find that back-propagating through contact and long trajectories drastically reduces gradient accuracy. Using this insight, we propose AHAC to dynamically adapt its roll-out horizon to avoid differentiating through stiff contact. AHAC is a first-order model-based RL algorithm that learns high-dimensional tasks in minutes (wall clock) and outperforms PPO by 40%, even in the limit of data provided to PPO. This work is led by Ignat Georgiev alongside Krishnan Srinivasan, Jie Xu, Eric Heiden and ample assistance from warp team at NVIDIA Robotics (Miles Macklin)

Animesh Garg

52,308 просмотров • 2 лет назад

Grok 4.6 drops around August 7 with a 1.5 trillion parameter model and improved fine-tuning, per Elon Musk. Grok 4.7 follows a few weeks later at 2.1T, better in every benchmark but slightly slower to serve. xAI is stacking releases while Grok 4.5 already beats Opus 5 on price-performance in cybersecurity benchmarks.
0:20

Grok 4.6 drops around August 7 with a 1.5 trillion parameter model and improved fine-tuning, per Elon Musk. Grok 4.7 follows a few weeks later at 2.1T, better in every benchmark but slightly slower to serve. xAI is stacking releases while Grok 4.5 already beats Opus 5 on price-performance in cybersecurity benchmarks.

Digg

35,731 просмотров • 3 месяцев назад

fine tuning rotoscope ai video asset for the next 1/1
0:19

fine tuning rotoscope ai video asset for the next 1/1

神経色

18,915 просмотров • 1 год назад

Fine-tuning the strokes! 🎶 #WhistlePodu #Yellove
0:13

Fine-tuning the strokes! 🎶 #WhistlePodu #Yellove

Chennai Super Kings

58,904 просмотров • 1 год назад

I'm always looking for regions of parameter space where small mutations evoke a variety of behaviors.
0:12

I'm always looking for regions of parameter space where small mutations evoke a variety of behaviors.

oops_all_paperclips

139,377 просмотров • 2 месяцев назад

Fine tuning the camper for maximum RAAHHH 🦅🇺🇸 Connor Zilisch | Kampgrounds of America (KOA)
0:10

Fine tuning the camper for maximum RAAHHH 🦅🇺🇸 Connor Zilisch | Kampgrounds of America (KOA)

JR Motorsports

16,880 просмотров • 4 месяцев назад

Can a language model become SOTA in chemical synthesis? 🧪🔥 We put a 2.6B-parameter Liquid AI model into MMAI Gym. It now sets the state of the art in single-step retrosynthesis. 🚀 #insilicoSOTAFM
0:20

Can a language model become SOTA in chemical synthesis? 🧪🔥 We put a 2.6B-parameter Liquid AI model into MMAI Gym. It now sets the state of the art in single-step retrosynthesis. 🚀 #insilicoSOTAFM

Insilico Medicine

39,706 просмотров • 1 месяц назад

RF-DETR converges faster than other detectors RF-DETR-B after 10 epochs of fine-tuning vs. YOLO11m after 30 epochs of fine-tuning link:
0:19

RF-DETR converges faster than other detectors RF-DETR-B after 10 epochs of fine-tuning vs. YOLO11m after 30 epochs of fine-tuning link:

SkalskiP

30,515 просмотров • 1 год назад

1/ Happy to share UniDisc - Unified Multimodal Discrete Diffusion – We train a 1.5 billion parameter transformer model from scratch on 250 million image/caption pairs using a **discrete diffusion objective**. Our model has all the benefits of diffusion models but now in multimodal space! - flexible compute-quality tradeoff, zero-shot inpainting and editing, better control via classifier-free guidance and lower latency! We open source everything - our code, weights and the training dataset.
0:10

1/ Happy to share UniDisc - Unified Multimodal Discrete Diffusion – We train a 1.5 billion parameter transformer model from scratch on 250 million image/caption pairs using a **discrete diffusion objective**. Our model has all the benefits of diffusion models but now in multimodal space! - flexible compute-quality tradeoff, zero-shot inpainting and editing, better control via classifier-free guidance and lower latency! We open source everything - our code, weights and the training dataset.

Mihir Prabhudesai

105,076 просмотров • 1 год назад

Alex DeBrincat's been fine-tuning the details this offseason! 🐙 (🎥: IG/power_edge_pro)
0:10

Alex DeBrincat's been fine-tuning the details this offseason! 🐙 (🎥: IG/power_edge_pro)

NHL

258,208 просмотров • 2 месяцев назад

Transformers & LLMs cheatsheets for Stanford's CME-295! Covering tokenization, self-attention, prompting, fine-tuning, LLM-as-a-judge, RAG, AI Agents, and reasoning models. 100% free and open-source.
0:15

Transformers & LLMs cheatsheets for Stanford's CME-295! Covering tokenization, self-attention, prompting, fine-tuning, LLM-as-a-judge, RAG, AI Agents, and reasoning models. 100% free and open-source.

Akshay 🚀

101,701 просмотров • 1 год назад

SuiRWA is training its investment AI agents on extensive data, using neural networks for pattern recognition. They learn via supervised & unsupervised methods, with fine-tuning for market analysis, ensuring accuracy and minimizing bias. A revolution in investing might be here.
0:16

SuiRWA is training its investment AI agents on extensive data, using neural networks for pattern recognition. They learn via supervised & unsupervised methods, with fine-tuning for market analysis, ensuring accuracy and minimizing bias. A revolution in investing might be here.

Sui RWA

16,176 просмотров • 1 год назад

Google released Gemma 3 270M, a new model for hyper-efficient local AI! We'll fine-tune this model and make it very smart at playing chess and predict the next move. Tech stack: - Unsloth AI for efficient fine-tuning. - Hugging Face transformers to run it locally. Let's go! 🚀
0:11

Google released Gemma 3 270M, a new model for hyper-efficient local AI! We'll fine-tune this model and make it very smart at playing chess and predict the next move. Tech stack: - Unsloth AI for efficient fine-tuning. - Hugging Face transformers to run it locally. Let's go! 🚀

Akshay 🚀

145,984 просмотров • 1 год назад