Pranjal Aggarwal's banner

Pranjal Aggarwal

@PranjalAggarw16 • 1,236 subscribers

PhD Student @LTIatCMU. Working on computer-use agents, RL envs, multi-agent RL. Prev: research scientist intern @AIatMeta FAIR, undergrad @IITD

Shorts

Claim: Computer-Use agents are embodied generalists Back in August, we used a single computer-use agent across 4 different embodied AI-related competitions at ECCV, and we ranked 1st, 2nd, 3rd, and 6th places. How? We constructed simple GUI software for each competition that connects to the benchmark tasks (e.g., using a keyboard to navigate inside a 3D house rendered in a web interface), and that's it. The agent itself stays the same. I have been meaning to write about it for a long time, but here it is now: Now to the position: we believe that computer-use agents are embodied generalists. Why? TLDR: We believe computer-use agents share striking similarities with models for embodied AI, and embodied tasks can be formulated as interactions with digital software. Once we have that, we can learn from what worked for computer use over the past 2 years, focus on digital scaling, and avoid mistakes or local minima in the grand scheme of things. I would urge anyone working in computer use or embodied AI to take a quick read of the paper, or at least go through the website. We do not think many people would agree now, and only time will tell. But we do think this is an important line of thought that everyone should consider. We look forward to feedback and constructive criticism!

Claim: Computer-Use agents are embodied generalists Back in August, we used a single computer-use agent across 4 different embodied AI-related competitions at ECCV, and we ranked 1st, 2nd, 3rd, and 6th places. How? We constructed simple GUI software for each competition that connects to the benchmark tasks (e.g., using a keyboard to navigate inside a 3D house rendered in a web interface), and that's it. The agent itself stays the same. I have been meaning to write about it for a long time, but here it is now: Now to the position: we believe that computer-use agents are embodied generalists. Why? TLDR: We believe computer-use agents share striking similarities with models for embodied AI, and embodied tasks can be formulated as interactions with digital software. Once we have that, we can learn from what worked for computer use over the past 2 years, focus on digital scaling, and avoid mistakes or local minima in the grand scheme of things. I would urge anyone working in computer use or embodied AI to take a quick read of the paper, or at least go through the website. We do not think many people would agree now, and only time will tell. But we do think this is an important line of thought that everyone should consider. We look forward to feedback and constructive criticism!

11,429 次观看

Videos

Anya Rossi

sweetdream.ai

SweetDream.ai•Sponsored•Livecam

Watch Anya Live

Anya is streaming live right now! Join her private show and enjoy exclusive content.

Exclusive private shows

1.2k viewers online

Private Show

Join now for exclusive access

Free preview available • Premium content

AI swarms are already showing promising (or dangerous) results in frontier labs. But how to implement them in the open community? And how do they scale? We built SwarmChat, an open-source API-based Slack for agents. Then we burnt half a trillion tokens, ran a series of swarm-scaling experiments, and tested various swarm structures on ProgramBench to study this question! Check out the blog here: We found a bunch of interesting results: 1. Swarms get the task done faster. For instance, 16-agent swarms reached the performance of a single agent 6x faster. 2. Swarm scaling is a third axis of scaling. The more agents, the better, and anyone can already start scaling with SwarmChat! 3. Different models communicate differently. GPT-6 is chatty and over-communicative, while Claude prefers to work in relative silence! 4. Different swarm structures barely have any benefit on aggregate. But on specific tasks, team structures give clear gains. 5. Bonus: SwarmChat has a Slack-like UI for us humans to monitor swarms! Makes monitoring a lot easier! We are already using SwarmChat for WIP projects, and we were able to catch cases of swarms breaking the 4th wall (sorry, more details are for a future post) Code: w/ Lawrence Jang , Jing Yu Koh , Daniel Fried, Russ Salakhutdinov, Sean Welleck, kevin , Nicolai Ouporov

AI swarms are already showing promising (or dangerous) results in frontier labs. But how to implement them in the open community? And how do they scale? We built SwarmChat, an open-source API-based Slack for agents. Then we burnt half a trillion tokens, ran a series of swarm-scaling experiments, and tested various swarm structures on ProgramBench to study this question! Check out the blog here: We found a bunch of interesting results: 1. Swarms get the task done faster. For instance, 16-agent swarms reached the performance of a single agent 6x faster. 2. Swarm scaling is a third axis of scaling. The more agents, the better, and anyone can already start scaling with SwarmChat! 3. Different models communicate differently. GPT-6 is chatty and over-communicative, while Claude prefers to work in relative silence! 4. Different swarm structures barely have any benefit on aggregate. But on specific tasks, team structures give clear gains. 5. Bonus: SwarmChat has a Slack-like UI for us humans to monitor swarms! Makes monitoring a lot easier! We are already using SwarmChat for WIP projects, and we were able to catch cases of swarms breaking the 4th wall (sorry, more details are for a future post) Code: w/ Lawrence Jang , Jing Yu Koh , Daniel Fried, Russ Salakhutdinov, Sean Welleck, kevin , Nicolai Ouporov

Pranjal Aggarwal

243,865 次观看 • 12 天前

Claim: Computer-Use agents are embodied generalists Back in August, we used a single computer-use agent across 4 different embodied AI-related competitions at ECCV, and we ranked 1st, 2nd, 3rd, and 6th places. How? We constructed simple GUI software for each competition that connects to the benchmark tasks (e.g., using a keyboard to navigate inside a 3D house rendered in a web interface), and that's it. The agent itself stays the same. I have been meaning to write about it for a long time, but here it is now: Now to the position: we believe that computer-use agents are embodied generalists. Why? TLDR: We believe computer-use agents share striking similarities with models for embodied AI, and embodied tasks can be formulated as interactions with digital software. Once we have that, we can learn from what worked for computer use over the past 2 years, focus on digital scaling, and avoid mistakes or local minima in the grand scheme of things. I would urge anyone working in computer use or embodied AI to take a quick read of the paper, or at least go through the website. We do not think many people would agree now, and only time will tell. But we do think this is an important line of thought that everyone should consider. We look forward to feedback and constructive criticism!

Claim: Computer-Use agents are embodied generalists Back in August, we used a single computer-use agent across 4 different embodied AI-related competitions at ECCV, and we ranked 1st, 2nd, 3rd, and 6th places. How? We constructed simple GUI software for each competition that connects to the benchmark tasks (e.g., using a keyboard to navigate inside a 3D house rendered in a web interface), and that's it. The agent itself stays the same. I have been meaning to write about it for a long time, but here it is now: Now to the position: we believe that computer-use agents are embodied generalists. Why? TLDR: We believe computer-use agents share striking similarities with models for embodied AI, and embodied tasks can be formulated as interactions with digital software. Once we have that, we can learn from what worked for computer use over the past 2 years, focus on digital scaling, and avoid mistakes or local minima in the grand scheme of things. I would urge anyone working in computer use or embodied AI to take a quick read of the paper, or at least go through the website. We do not think many people would agree now, and only time will tell. But we do think this is an important line of thought that everyone should consider. We look forward to feedback and constructive criticism!

Pranjal Aggarwal

11,429 次观看 • 4 天前

Introducing Gym-Anything for Robotics: Scaling realistic robot simulations. 200 RL environments across 25 simulations, including liquids, cloth, molten metal, soil, and even sound, in economically valuable domains! Best models get 0% And we show that frontier models can learn via ICL using different methods, and even significantly improve VLAs like π0.5! See video examples (sound on) 🧵

Introducing Gym-Anything for Robotics: Scaling realistic robot simulations. 200 RL environments across 25 simulations, including liquids, cloth, molten metal, soil, and even sound, in economically valuable domains! Best models get 0% And we show that frontier models can learn via ICL using different methods, and even significantly improve VLAs like π0.5! See video examples (sound on) 🧵

Pranjal Aggarwal

25,093 次观看 • 11 天前

没有更多内容可加载