Reinforcement learning should be able to improve upon behaviors... seen when training. In practice, RL agents often struggle to generalize to new long-horizon behaviors. Our new paper studies *horizon generalization*, the degree RL algorithms generalize to reaching distant goals. 1/show more

Haven't been to a conference in a while, really... excited to be at #NeurIPS2024! I'll be helping present 4 of our group's recent papers: 1. Overcoming the Sim-to-Real Gap: Leveraging Simulation to Learn to Explore for Real-World RL 2. Distributional Successor Features Enable Zero-Shot Policy Optimization 3. Learning to Cooperate with Humans using Generative Agents 4. Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learning Find more details on each paper and where to find us in this thread (1/6)show more

RL is painfully slow 😭 — bottlenecked by super-long... CoT rollout. 🔭 Sparse attention should help, but naive sparse rollout hits a brutal efficiency–stability tradeoff: A tedious trial-and-error sparsity sweep for each dense policy is required before an actual RL run. 🐤Sparrow chirps no more pain! Introduce Sparrow: Sparse Rollout for stable and efficient long-context RL. Sparrow finds that: 💡As long as we keep the tail distribution mismatch throughout the sparse rollout above a critical threshold, the RL training will be stable. 💡Even cooler! Through comprehensive control studies of Qwen3-1.7B, 4B, 8B thinking models RL with 40K rollout max length, the critical threshold stays constant across model sizes. 💡Sparrow then finds the optimal dynamic sparse schedule to reach the threshold with minimal cost. 💡Sparrow's findings are empirically validated to generalize in Qwen3-14B, and hold on both Math and Coding RL. 🐤Sparrow empirically helps achieve 2.2× / 2.4× / 2.0× rollout speedup on Qwen3 1.7B / 4B / 8B thinking models, while keeping training stability over extended RL steps. We release the 🐤bird in the following formats. [1/n] Paper: Code: Blog:show more

RL is back! But is it always the best... choice? In a new paper, we investigate under what circumstances neuroevolution outperforms RL in transfer learning tasks. See more details in the thread below 🧵 While NE performs best in simpler domains, it will be interesting to see if the lessons learned here can also be applied to more complex systems/tasks (LLMs?).show more

New research from Databricks: LLMs Can Learn to Reason... via Off-Policy RL Optimal Advantage-based Policy Optimization with Lagged Inference policy (OAPL) shows you don’t need strict on-policy training to improve reasoning. It matches or beats Group Relative Policy Optimization (GRPO), stays stable with large policy lag, and uses ~3× fewer training generations. For Databricks customers, it’s a simpler, practical, and equally powerful approach to RL that Databricks is pioneering internally — and bringing directly to Databricks customers, so enterprises can improve agents using the same methods we use for our in-house agents, without complex infrastructure changes.show more

Robora Sim: A PyBullet-Powered Environment for Learning Robotic Physical... Intelligence We are currently building our Robora simulation environment setup for our sim based learning, leveraging PyBullet, an industry-standard physics engine widely used in AI-driven robotics research and development. The environment is optimized with GPU-accelerated learning algorithms, enabling high-speed imitation learning and reinforcement learning within a safe and controlled virtual setup before shipping out to real world. This simulation platform allows our models to learn, adapt, and generalize across different robot morphologies, terrain types and task objectives - all before deployment to the real world. At it's core, the system combines a VLA-powered high-level planner with low-level motion control algorithms, working cohesively to produce emergent, physically intelligent behaviors. This synergy between simulation, learning, and real-world transfer marks a major step forward in our pursuit of adaptive and intelligent robotic systems. Through advanced domain randomization and synthetic data generation, the Robora Simulation Environment ensures that policies trained in simulation transfer effectively to real-world robots, minimizing the sim-to-real gap. Moreover, users will be able to test and integrate their own hardware kits within selected simulation environments in the Robora Dapp, ensuring seamless compatibility and safer real-world implementation.show more

🔬🤖Excited to share our new preprint on robust virtual... staining of key organelles! ✨ Virtual staining and segmentation of nuclei and cell membranes in phase images can significantly accelerate image-based screens and developmental studies. 🔍 Current virtual staining models often struggle with nuisance variations in label-free input and don't generalize well to new cell types. We’re releasing our training protocols that lead to robust models and enable few-shot generalization. 🔧 Stay tuned for our revamped codebase with weights and example data, coming soon! 👏🧫🐟 Huge shoutout to Ziwen Liu, Eduardo Hirata-Miyasaki, my team Chan Zuckerberg Biohub Network (Soorya, Johanna, Christian, Talon Chandler, Ivan Ivanov), and the teams of Manuel Leonetti, Carolina Arias, and Adrian Jacobo for this fun collaboration! #VirtualStaining #DL #CellBiology #Zebrafishshow more

We are excited to announce our strategic partnership with... Fraction AI for their IIO launch. Fraction AI is the first decentralized training platform for AI agents with millions of sessions, 150,000+ agents deployed, and over $1M in rewards distributed. On our platform, thousands of AI agents compete in specialized domains and continuously improve through Reinforcement Learning. Stay tuned for upcoming surprise announcements.show more

We're building the next phase of our long-horizon autonomy... study and we want your input to help shape it. What questions should we be asking? What behaviours would you want tested? Share your thoughts in the comments, or head to the link below to submit your input and stay notified when it goes live:show more

New work: The Value Axis 🎯 How do LLMs... choose which path to take mid-task? We find they internally track the chance of reaching their goal along a linear axis, akin to a value function in RL. We show it modulates confidence in math & coding and can be reshaped with DPO and SFT.show more

Model-Free Reinforcement Learning (MFRL) has been alluring, especially with... supercharged compute with physics on GPU. However, the methods use 0-th order gradients, and are often not the best optimizers. Can we do better than PPO in continuous control for robotics? Turns out yes! 🥳 tl;dr: Faster, better RL than PPO in continuous control 💪 The answer lies in using more information from the simulation. We are juicing the simulation on GPU as it is, why not use it for gradients as well? This has been a driving question in a series of our works. We first studied this problem in ICLR 2022 paper on Short Horizon Actor Critic Naive gradient based methods are stuck in local minima and have exploding/vanishing gradients. SHAC solved this problem truncated rollouts and model based value estimation, where the model is Differentiable Sim. This boosted sample efficiency and wall-clock time immensely especially in high dimensional systems such as humanoids Yet, given enough compute PPO often caught up. Our follow up paper on on Adaptive Horizon Actor Critic at ICML 2024 discovers the cause and provides a fix. However, we find that even when given ground-truth dynamics, not all gradients are useful due to sample error. 1st-Order Model-Based Reinforcement Learning methods employing differentiable simulation provide gradients with reduced variance but are susceptible to bias in scenarios involving stiff dynamics, such as physical contact. We find that back-propagating through contact and long trajectories drastically reduces gradient accuracy. Using this insight, we propose AHAC to dynamically adapt its roll-out horizon to avoid differentiating through stiff contact. AHAC is a first-order model-based RL algorithm that learns high-dimensional tasks in minutes (wall clock) and outperforms PPO by 40%, even in the limit of data provided to PPO. This work is led by Ignat Georgiev alongside Krishnan Srinivasan, Jie Xu, Eric Heiden and ample assistance from warp team at NVIDIA Robotics (Miles Macklin)show more

Newly reverted sister uses a book and a paper... to learn how to pray salah. reverts are the most determined and hard-working people when it comes to learning about islam. May Allah reward us for our efforts in this new spiritual journey 🙏 ☪️❤️show more

🔥 Nebius AI R&D is hiring AI Research Interns... for short, high-impact RL projects. Exclusive to X right now — no LinkedIn mass postings yet. In 2019, I was a fresh dental grad with 3 months of runway left, begging for an AI shot. I know the grind. We’re looking for sharp early-career folks (students, grads, career-switchers) to join us and work on: > Agent trajectories analysis at scale > Long-horizon tasks for coding agents > Pushing open RL environments > Any other data / RL env / eval project that will benefit open-source community What you get: 💰 Fully paid internship (3-6 month) 📦 100% open-source shipping 📄 Co-author research papers ⚡️ Access to Nebius compute infra 🌍 Remote-friendly (EU/US) or Amsterdam/London/other office. If you’ve done any cool AI/ML/RL stuff, dm me with your most impressive project + 1-sentence summary + cv Sharing appreciated!🤝show more

So many of our #GroveToken-era CEX listings have joined... us in the future that is #GroveCoin ! 💚 And according to @JohnGh87 it won't be long before there's a ❗️brand new❗️ top-tier listing for #GRV on the horizon. #GroveGreenArmy, let's see your guesses below! 👇👇show more

🚨How do LLMs acquire human values?🤔 We often point... to preference optimization. However, in our new work, we trace how and when model values shift during post-training and uncover surprising dynamics. We ask: How do data, algorithms, and their interaction shape model values?🧵show more

We’re kickstarting a brand-new season of the Open Metaverse... Podcast with a special guest! ⭐ …not to mention a brand-new host 😉: Introducing Raafeh Rahim (REKiTRaaf), our associate director of Investments and Strategic Partnerships. Market maker by trade, professional snacker by choice 🍕 Joining him is our group president Evan Auyang, who will be spilling the beans about our new direction for the year! 🦄Animoca Brands’ growth trajectory in 2025 🦄Joint stablecoin venture with Standard Chartered HK and #HKT 🦄Exciting new investments in our horizon 🦄 Key virtual asset trends to pay attention to Tune in now! 🍿show more

🌃 A strange pulse crossed the night sky Lightning... danced across the horizon A New purple sphere has emerged! 100 slots are now open to claim WL for our mint at: (Requirement: hold at least 1 Hyper in your wallet) ⚡RT + LIKE + Drop your EVM walletshow more

We're excited to unveil NRN Agents, a rebrand that... aligns our project identity with our token and strengthens our mission to power the future of AI-driven gaming. This mission requires collaboration, and starting this week, we will begin our expansion to become a multi-chain ecosystem. We are joining forces with leading gaming platforms and ecosystems to realize this vision. Stay tuned for more announcements to come. Why NRN Agents? NRN stands for NEURON, the fundamental unit of intelligence. Our AI agents function as the neural foundation of games, learning, adapting, and evolving within game worlds to deliver unparalleled engagement. NRN agent SDK enables advanced gaming agents powered by a proprietary machine learning infrastructure focused on behavioral learning. We've perfected the craft of gaming agent design, creating hyper-efficient agents that are performant and scalable—from casual to the most demanding games. Our SDK will seamlessly integrate into many platforms, tech stacks, and ecosystem – Any Game. Any Chain. More than just games, it's the path to AGI Gaming is our proving ground, but not our final destination. We're using games as a sandbox to accelerate the development of generalized intelligence—one that will create meaningful real-world impact. With the upcoming launch of [redacted] and a growing network of partners committed to the AGI vision, we're building an open-source innovation movement powered by an AI x gaming framework connected by $NRN. $NRN the token $NRN is a utility token that serves as the gateway to our growing ecosystem. It will power a diversified economy with multiple revenue streams and staking opportunities: Agent Deployment: NRN is the laboratory creating gaming agents that can be distributed through platforms and launchpads alike. The model is simple: More games integrate, more NRN agents get deployed, more monetization. Data Creation: NRN Reinforcement Learning (RL) enables token staking to create Data Capsules. Players contribute gameplay data into the Capsules, which are used train RL agents and reward participants (players & stakers). AI Arena: $NRN also continues to power AI Arena's in-game economy, a cult favorite of competitive diehards that features a skill-based wagering system. To our community who have supported us since 2021: thank you for being part of our journey—the next chapter will be the most exciting yet!show more

This woman is committed to finishing her degree, even... when she got pregnant and had a baby. She refused to take off and was even more determined to be able to provide a better life for her child. She often brings the baby to class because she doesn’t have a sitter. Her professor sees this and often will hold and comfort her baby for her so she can take notes in class or have a breather. Knowing the distractions young children can bring, is this appropriate or should young children not be allowed on college campuses?show more