1/ While most RL methods use shallow MLPs (~2–5... layers), we show that scaling up to 1000-layers for contrastive RL (CRL) can significantly boost performance, ranging from doubling performance to 50x on a diverse suite of robotic tasks. Webpage+Paper+Code:show more

Congratulations also to Patrick 👍 for his #ICLR paper... on Temporal Difference (TD) learning , in it We solve the decades-old puzzle of why TD can solve complex RL tasks that Gradient Descent cannot.show more

Population dynamics can provide insights into domains ranging from... health to environmental science. Here we introduce a geospatial foundation model (plus embeddings and code recipes) that could be employed for a variety of downstream tasks. →show more

New research from Databricks: LLMs Can Learn to Reason... via Off-Policy RL Optimal Advantage-based Policy Optimization with Lagged Inference policy (OAPL) shows you don’t need strict on-policy training to improve reasoning. It matches or beats Group Relative Policy Optimization (GRPO), stays stable with large policy lag, and uses ~3× fewer training generations. For Databricks customers, it’s a simpler, practical, and equally powerful approach to RL that Databricks is pioneering internally — and bringing directly to Databricks customers, so enterprises can improve agents using the same methods we use for our in-house agents, without complex infrastructure changes.show more

Introducing RL Environment Creator Skill Now any one can... create RL environments $ npx skills add adithya-s-k/RL_Envs_101 > You can create environments across multiple frameworks like OpenEnv, OpenReward, Verifiers, NemoGym ... > the repo has live working examples of environments that your coding agent can reference > The skill is design to first understand what type of model you are training and create an environment while keeping that in mind ps. There’s a lot more to building RL environments that can be used for training. One major aspect is the data, which this skill can’t directly solve. However, the skill will help with implementing tools, rewards, and other components of an RL environment, making it easier to go from idea to implementation quickly across different frameworks. Let me know if you’d be interested in a detailed, end-to-end blog/tutorial on building an environment and actually training a model for a useful use case.show more

RL is painfully slow 😭 — bottlenecked by super-long... CoT rollout. 🔭 Sparse attention should help, but naive sparse rollout hits a brutal efficiency–stability tradeoff: A tedious trial-and-error sparsity sweep for each dense policy is required before an actual RL run. 🐤Sparrow chirps no more pain! Introduce Sparrow: Sparse Rollout for stable and efficient long-context RL. Sparrow finds that: 💡As long as we keep the tail distribution mismatch throughout the sparse rollout above a critical threshold, the RL training will be stable. 💡Even cooler! Through comprehensive control studies of Qwen3-1.7B, 4B, 8B thinking models RL with 40K rollout max length, the critical threshold stays constant across model sizes. 💡Sparrow then finds the optimal dynamic sparse schedule to reach the threshold with minimal cost. 💡Sparrow's findings are empirically validated to generalize in Qwen3-14B, and hold on both Math and Coding RL. 🐤Sparrow empirically helps achieve 2.2× / 2.4× / 2.0× rollout speedup on Qwen3 1.7B / 4B / 8B thinking models, while keeping training stability over extended RL steps. We release the 🐤bird in the following formats. [1/n] Paper: Code: Blog:show more

🤖 Another zero-shot reward model is now in LeRobot:... ROBOMETER. A general-purpose, zero-shot video-language reward model from University of South Carolina, UT Dallas, Massachusetts Institute of Technology (MIT), University of Washington, Ai2, and NVIDIA that predicts frame-level task progress. Trained on 1M+ trajectories from 21 robot embodiments, generalizes zero-shot to unseen tasks, scenes, and robots. 2.4–4.5x better downstream success rates across online RL, offline RL, data filtering, failure detection, and data retrieval for IL. Project: Paper:show more

What if you kept asking an LLM to "make... it better"? In some recent work at FAIR, we investigate how we can efficiently use RL to fine-tune LLMs to iteratively self-improve on their previous solutions at inference-time. Training for iterated self-improvement can be costly. The naive approach to training for K self-improvement steps leads to K times the number of rollout steps per episode. We introduce Exploratory Iteration (ExIt), an RL-based automatic curriculum method that bootstraps diverse training distributions of self-improvement tasks by upcycling the LLM's own responses at previous turns as the starting points for both self-improvement and *self-divergence.* In order to decide what task to train on next, the curriculum prioritizes sampling of partial turn histories that led to higher return variance in its GRPO group (a learnability score that comes for free). This automatic curriculum over the bootstrapped task space teaches the model how to perform iterated self-improvement while only ever training the model on single-step self-improvement tasks. We look at ExIt's impact in both single-turn (contest math problems) and multi-turn (BFCLv3 multi-turn tasks), as well as MLE-bench, where the LLM is run in a search scaffold to produce solutions to real Kaggle competitions. Across these eval settings, we find ExIt produces models with greater capacity for inference-time self-improvement compared to GRPO. Notably, ExIt models can self-improve on test tasks for many more steps than the typical solution depth encountered during training, including a 22% improvement in MLE-bench performance compared to GRPO.show more

PHASE 3: #Node Rental Onboarding! 🔥 We’ve successfully completed... all setup operations, and our Nodes are primed and ready for use! After extensive testing of both our platform and hardware, we’ve fine-tuned each node to ensure peak performance for their designated tasks. ✅ Phase 1: Setup ✅ Phase 2: Testing ☑️ ➤ Phase 3: Launch ☑️ Phase 4: Scaling For this phase we are focussed on customer onboarding, as well as monitoring the performance and energy use of our Nodes for future refinement. Our #NodeRental service officially goes live: 🗓 Wed 16th Oct | 5 PM UTCshow more

(1/6) X-Humanoid 🤖: Scaling up data for Humanoid Robots.... We convert human daily activity videos (from Ego-Exo4D) into humanoid videos (i.e., Tesla Optimus) performing tasks like cooking or fixing a bike. This data can be potentially used to train robot policies and world models. 🔥 Project page: Paper link:show more

NVIDIA just released a very impressive text-to-video paper. Video... Latent Diffusion Models (Video LDMs) use a diffusion model in a compressed latent space to generate high-resolution videos. Here's a brief overview of how it works: 1. Pre-train image LDM on a dataset of images. 2. Turn the image LDM into a Video LDM by adding temporal layers to model video frames. 3. Fine-tune the Video LDM on encoded video sequences to create a video generator. 4. Temporally align diffusion model upsamplers to generate high-resolution videos. 5. Validate Video LDM on real driving videos of 512x1024 resolution, achieving state-of-the-art performance. 6. Apply the approach in creative content creation with text-to-video modeling. Paper: Project:show more

Dexter vs. Claude Code I ran tests overnight and... Dexter came out ahead on complex financial tasks that required deep research. Dexter won on: • speed (by 92%) • cost (by 26%) • correctness (by 31%) I use Claude Code often, so this was fun to see. A key challenge for CC is that it relies on web search for financial data. Most of what it finds comes from news sites, blogs, and other secondary sources. Dexter uses primary source data from Financial Datasets, so the performance gap makes sense. Plenty of room to improve on Dexter. The gap will only grow from here. Evals from vals. Report coming next.show more

🚀Thrilled to share what we’ve been building at TRI... over the past several months: our first Large Behavior Models (LBMs) are here! I’m proud to have been a core contributor to the multi-task policy learning and post-training efforts. At TRI, we’ve been researching how LBMs can help robots learn faster, better, and more efficiently. The key takeaways: ✅ We built an evaluation pipeline to benchmark LBM performance with real 𝐬𝐭𝐚𝐭𝐢𝐬𝐭𝐢𝐜𝐚𝐥 𝐜𝐨𝐧𝐟𝐢𝐝𝐞𝐧𝐜𝐞 ✅ Pre-training on hundreds of tasks makes models more robust—plus, we can teach new, complex tasks with 80% 𝐥𝐞𝐬𝐬 𝐝𝐚𝐭𝐚 ✅ The bigger and more diverse the pre-training, the better the results Check out our overview video, webpage and paper for more details: ✨ 🌎 📄 We hope this work helps move the field of robotics forward!show more

Haven't been to a conference in a while, really... excited to be at #NeurIPS2024! I'll be helping present 4 of our group's recent papers: 1. Overcoming the Sim-to-Real Gap: Leveraging Simulation to Learn to Explore for Real-World RL 2. Distributional Successor Features Enable Zero-Shot Policy Optimization 3. Learning to Cooperate with Humans using Generative Agents 4. Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learning Find more details on each paper and where to find us in this thread (1/6)show more

Microsoft presents Windows Agent Arena Evaluating Multi-Modal OS Agents... at Scale discuss: Large language models (LLMs) show remarkable potential to act as computer agents, enhancing human productivity and software accessibility in multi-modal tasks that require planning and reasoning. However, measuring agent performance in realistic environments remains a challenge since: (i) most benchmarks are limited to specific modalities or domains (e.g. text-only, web navigation, Q&A, coding) and (ii) full benchmark evaluations are slow (on order of magnitude of days) given the multi-step sequential nature of tasks. To address these challenges, we introduce the Windows Agent Arena: a reproducible, general environment focusing exclusively on the Windows operating system (OS) where agents can operate freely within a real Windows OS and use the same wide range of applications, tools, and web browsers available to human users when solving tasks. We adapt the OSWorld framework (Xie et al., 2024) to create 150+ diverse Windows tasks across representative domains that require agent abilities in planning, screen understanding, and tool usage. Our benchmark is scalable and can be seamlessly parallelized in Azure for a full benchmark evaluation in as little as 20 minutes. To demonstrate Windows Agent Arena's capabilities, we also introduce a new multi-modal agent, Navi. Our agent achieves a success rate of 19.5% in the Windows domain, compared to 74.5% performance of an unassisted human. Navi also demonstrates strong performance on another popular web-based benchmark, Mind2Web. We offer extensive quantitative and qualitative analysis of Navi's performance, and provide insights into the opportunities for future research in agent development and data generation using Windows Agent Arena.show more

DeCAF won the #ICML Test of Time Award 2024!... Big congrats to trevordarrell (my PhD advisor at MIT), and Jeff Donahue. 🎉 You may not heard of DeCAF, but it is everywhere! DeCAF stands for Deep Convolutional Activation Features. Published ten years ago, the DeCAF paper is a groundbreaking work that shows the activation features of the last few layers of a deep network contain useful features that can be "repurposed" for or "transferred to" many other tasks, not just the original task the network was trained for. I created this exercise to show where we can see DeCAF's influence in some of the most well-known architectures: AlexNet, ViT, U-Net, CLIP, and Latent Diffusion, to prove that DeCAF's "Test of Time Award" is well-deserved! Let's give a round of applause to DeCAF, the unsung hero of computer vision.show more

🔥 Nebius AI R&D is hiring AI Research Interns... for short, high-impact RL projects. Exclusive to X right now — no LinkedIn mass postings yet. In 2019, I was a fresh dental grad with 3 months of runway left, begging for an AI shot. I know the grind. We’re looking for sharp early-career folks (students, grads, career-switchers) to join us and work on: > Agent trajectories analysis at scale > Long-horizon tasks for coding agents > Pushing open RL environments > Any other data / RL env / eval project that will benefit open-source community What you get: 💰 Fully paid internship (3-6 month) 📦 100% open-source shipping 📄 Co-author research papers ⚡️ Access to Nebius compute infra 🌍 Remote-friendly (EU/US) or Amsterdam/London/other office. If you’ve done any cool AI/ML/RL stuff, dm me with your most impressive project + 1-sentence summary + cv Sharing appreciated!🤝show more

Say hello to Boojum 👋: zkSync Era’s new high-performance... proof system for radical decentralization. Boojum is an upgrade that will transition zkSync Era to a STARK-powered proof system, providing world-class performance on consumer-grade hardware. 💡 Learn more: TL;DR 👇 Boojum is the name of our Rust-based cryptographic library, which we use to implement the upgraded version of the ZK circuits for zkSync Era and the ZK Stack. The name Boojum was inspired by Lewis Carroll's poem "The Hunting of the Snark," where the Boojum represents the most fearsome kind of Snark. We intentionally designed zkSync Era in a way that cryptographic upgrades can be made without a regenesis, meaning that the Boojum upgrade won’t cause any user disruptions. Why Boojum❓ From day one, zkSync’s mission is to advance personal freedom for all — making digital self-ownership universally accessible by building a blockchain network that is trustless, secure, permissionless, affordable, easy to use, resilient and limitlessly scalable. Boojum plays an important role in advancing this mission by delivering: 1. World-class performance zkSync Era’s current SNARK-based proof system is effective today, but it won’t scale to the volume that we envision for hyperchains. zkSync Era’s sequencer can already process over 100 TPS; Boojum orders of magnitude improvements to performance complements this well. 2. Reduced hardware requirements for decentralization Our long-term goal is to enable user-powered, decentralized proof generation. Boojum represents a breakthrough in this direction — with the prover running on consumer-grade GPUs requiring only 16 GB GPU RAM. Boojum’s Journey to Mainnet 🚴🏽‍♀️ Boojum is now live on Mainnet, generating and verifying ‘shadow proofs’ today with real production data so that we can carefully test the system ahead of fully migrating. Today, we’re also open-sourcing the repo; if you’d like to take a look, you can find it here 👇 This is the first of a series of posts on Boojum. We will provide updates on our progress, including more details on implementation, security, and performance. Watch here for more, anon ∎show more

Deleting Soon INSTEAD OF WATCHING NETFLIX TONIGHT. Spend 1... hour reading the guide that i curated on how to make $10k/day from Glitchy (Performance Marketing Network). Performance marketing FULL COURSE that teaches you how to run any offers and promote it to tiktok efficiently. The people who read this tonight will wake up tomorrow with a new skill. Like+Retweet+comment "Money" to receive the guide (only up for 48 hours only)show more

We’re excited to introduce Text-to-LoRA: a Hypernetwork that generates... task-specific LLM adapters (LoRAs) based on a text description of the task. Catch our presentation at #ICML2025! Paper: Code: Biological systems are capable of rapid adaptation, given limited sensory cues. For example, our human visual system can quickly adapt and tune its light sensitivity to our surroundings. While modern LLMs exhibit a wide variety of capabilities and knowledge, they remain rigid when adding task-specific capabilities. Traditionally, customizing these models requires gathering large datasets and performing often expensive, time-consuming fine-tuning for specific applications. To bypass these limitations, Text-to-LoRA (T2L) meta-learns a “hypernetwork” that takes in a text description of a desired task, as a prompt, and generates a task-specific LoRA that performs well on the task. In our experiments, we show that T2L can encode hundreds of existing LoRA adapters. While the compression is lossy, T2L maintains the performance of task-specifically tuned LoRA adapters. We also show that T2L can even generalize to unseen tasks given a natural language description of the tasks. Importantly, Text-to-LoRA is parameter-efficient. It generates LoRAs in a single, inexpensive step, based solely on a simple text description of the task. This approach is a step towards dramatically lowering the technical and computational barriers, allowing non-technical users to specialize foundation models using plain language, rather than needing deep technical expertise or large compute resources.show more