😈 Today, we introduce WebGym, the largest-to-date open-source RL environment for web agent training that contains 300k tasks and a rollout framework optimized specifically for web environments' rollout speed. We reveal the effects of essential scaling directions we observe with WebGym. 1/n

WebGym contains 300k tasks. We source tasks from the 10 most recent and widely used benchmarks and training environments, like InSTA-v3, PAE-WebVoyager, BrowseComp, GAIA, and the Mind2Web series. 2/n

After we get the source tasks, we propose a rubric (a set of criteria) for each task (more accurately, "fact groups of a task"), and use compositions of these criteria to form new tasks that are easier than the original task. 3/n

These rubrics are not just for decomposing tasks. They can also be used for evaluation. We find that the accuracy (human alignment) of evaluation with all evaluator models improves significantly after we provide these rubrics for trajectory evaluation. 4/n

In the end, the WebGym task set includes 300k tasks, with a difficulty distribution biased towards easier tasks. We design the task set this way to help the model get faster signals for a wider range of tasks during RL, while encouraging it to focus more on exploitation for harder tasks. You're always welcome to sample only harder tasks and its decompositions with your favourite RL algorithm :) 5/n

This task set is so large! To our knowledge, no previous rollout framework was optimized specifically for web agents, and most web agent frameworks are synchronous rollout frameworks, either step-level or episode-level. In WebGym, we provide an asynchronous rollout framework. 6/n

With this framework, we’re able to achieve 4x-5x speedup compared to the naive synchronous rollout pipeline. Guess what, with the WebGym framework, you only need 128 CPUs to collect 1800 trajectories within 125 minutes! This rollout framework is written in a loosely-RL-framework-attached way so it can be easily integrated with any RL framework, like veRL, veRL-agent, slime, etc. 7/n

Now it’s time to cook the model! We start from the Qwen3-7B-VL model family. Yes, it’s VL because we firmly believe in the strong promise of vision for web agents. As we’re aiming to solve long-horizon tasks, it makes more sense to make it output some memory tokens at each step so it doesn’t forget what it did, which we learnt from the GLM series. We find it crucial for making RL successful on WebGym. 8/n

Another headache we had in our previous works, like Test-Time-Interaction and DigiRL, is that agents can be “stubborn” and repeat a wrong action forever. We found that filtering those “stubborn” steps out during training effectively solves this. We can solve this with even the simple REINFORCE algorithm! 9/n

As you may know, Qwen3-VL also has a Thinking variant. We have some explorations on this model using the same prompt as the Instruct model: the performance is higher at the beginning, but the gap drops quickly. The response length of a Thinking model (~2000 characters) is also doubling the response length of an Instruct model (~1000 chars), so we decided to focus on the Instruct model in this work. 10/n

Now that we’ve got a decent starting point, how do we leverage WebGym to scale the performance up? The first thing we can think of is to scale the breadth of the task set. We figure out that if we only sample from half of the domains in the task set, the performance improvement becomes marginal soon. 11/n

The second scaling dimension is the size of the sampled tasks. What if we sample the same tasks less frequently? We found that the performance scales even further. Below, “Only Easy” means that we only sample from easy tasks, which contain most of the different tasks. Remember the difficulty distribution of the task set we demonstrated above? 12/n

But only easy tasks is not enough! There are skills that the model need to learn from harder tasks, like learning to explore within the environment. Thus, we tried including harder tasks when sampling, and figured out that it improves the performance further. This is why we include hard tasks within the WebGym task set. 13/n

Finally, we also tried to reduce the per-episode step limit (the "Short Horizon" curve). Empirically, shortening the horizon improves both sample efficiency and final performance across all difficulty slices, increasing the peak success rate further. 14/n

There are many details we did not demonstrate in this post, such as details of the task set and evaluation pipeline, more fine-grained observations and analysis from the experiments, etc. Check out our paper, and I sincerely hope you can learn something from it! Website: (highly suggested for first-time audience, many animations and interactive experience) Paper: (much more details and takeaways, and more accurate descriptions than the website) HF Daily Paper: (come give a like if you like our work!) Code and Data are coming soon. 15/n

This work cannot happen without the strong support from @MSFTResearch and my collaborator Alexey Taymanov, and the invaluable suggestions and guidances from my mentors and advisors @spncrwhthd, @aviralku2, and Tong Zhang. 16/n

Subscribe to me if you would like to get the latest information on the release of the WebGym code & data! Code and data will be announced in a separate thread. You’re more than welcome to propose questions you have for the paper and website under this thread! n/n (The End.)

hey! fellow illini would love to chat, dm me :)

The async helps but the queue scheduling is doing heavy lifting here. That long-tail latency problem in browser sims is brutal - some rollouts take forever while GPUs just sit there waiting

Thanks for the comment. In the implementation, we set a timeout for each task - if it just gets stuck there it will get killed. When it begins to stuck, it does not block the GPU because other threads will utilize GPU. Only when (1) all threads are finished while few are still hanging there or (2) one thread takes unexpectedly long, should we intervene.

Got it!

this sounds super interesting. it’s amazing how much potential there is in training web agents. can't wait to see what people build with WebGym. hopefully platforms like First100 can leverage these advancements to streamline customer interactions even more.

dope RL env!

wow, 300k tasks is wild. love seeing the web get its own RL playground. curious to see what new agents will learn to dream up here.

It's quite fun and I did some preliminary experiments on this already: you just need to input the text instruction of doing something you like and make it a task in WebGym, then sample this heavily during RL training (while you should still sample other tasks), then you'll figure out that the performance on this task improves. You do need some existing tasks in the task set that are similar or at least adjacent to this task that you specify, though, to introduce diversity and avoid catastrophic forgetting. This is something that's already practiced in math and coding, but still not there in web. I will be looking forward to research like this upon WebGym.

Scaling tasks matters as much as scaling parameters. 300k real web style tasks is how you turn agents from demos into something dependable.

@grok explain in plain English with examples

Hey Jack! have a couple of questions I would love to ask. Could you open DM's?

Opened - also recommend to connect through email as I check DMs lazily

This is huge... Rollout speed is often the real bottleneck for web-agent RL not model capacity. Do you see diff scaling laws as task diversity increases (vs just more trajectories)?

You’re correct that rollout speed is the major headache in web RL. Friends near me who work on web RL are all complaining about it, and there’s one master student who initially worked with me on web agents who left this area because there’s no good frameworks and task sets in this area, which I felt so painful about. This was the original motivation of this work. On your question, yes, task diversity is important, both breadth (domain diversity) and depth (difficulty diversity). I recommend to check out the performance scaling part of my thread and maybe the website.

This is super helpful context. the breadth vs depth distinction makes the scaling story much clearer.

impressive

interesting