Introducing UI-TARS-1.5, a vision-language model that beats OpenAI Operator and Claude 3.7 on GUI…
@TsingYoga5.6K followers
85,212 views
22 Comments
Comments from the original post on X (Twitter)

@TsingYoga
UI-TARS-1.5 achieves SOTA results on several GUI benchmarks, e.g., OSWorld, WindowsAgent Arena, Online Mind2web, Android World, and ScreenSpot-Pro. These results demonstrate UI-TARS's superiority on computer use, browser use, and phone use. Also, with the GUI Tool, UI-TARS almost matches GPT-4o with the search API
7 likes

@TsingYoga
Here's a demo from UI-TARS on GUI tasks~
5 likes

@TsingYoga
To further assess UI-TARS-1.5 in complex, open-ended environments, we tested it on Minecraft—a popular sandbox game well-suited for evaluating embodied intelligence. Unlike static GUI benchmarks, Minecraft requires real-time decision-making in a dynamic 3D space using visual input and low-level controls (mouse and keyboard), closely reflecting real-world computer use.
4 likes

@TsingYoga
TARS has amazing inference-time scaling ability. With more interaction rounds, TARS achieves far better performance in GUI tasks and Game tasks. The scaling curve surpasses both OpenAI CUA and Claude 3.7. We even observe performance gain when the interaction rounds are over 1000 steps.
3 likes

@TsingYoga
Gameplay represents a critical frontier for multimodal agents, serving as an ideal testing ground for evaluating complex reasoning, decision-making, and adaptability. Games demand intuitive, common-sense reasoning and strategic foresight, making them perfect benchmarks to test and showcase the advanced cognitive capabilities of multimodal agents. To evaluate UI-TARS-1.5's gameplay proficiency, we selected 14 diverse games from Each model was allowed up to 1,000 interaction steps per game to generate execution traces, repeated across multiple runs.
3 likes

@TsingYoga
Explore more interesting showcases of UI-TARS on
2 likes

@chrisbarber
42% on OSWorld is impressive!
2 likes

@TsingYoga
Thanks! Will be higher sooner!
1 like

@oran_ge
Impressive!
1 like

@trycua
soon as an agent loop in c/ua 👀
1 like

@TsingYoga
Sure it will be!
2 likes

@BerniceRob27993
nice work
1 like

@BleedingDev
How well can it play Pokemon? 🤔
1 like

@Oli82817545
looks really cool but when can we acess the larger 1.5 and will it be opensource too?

@TsingYoga
Sure! Soon will be
3 likes

@Oli82817545
nice really excited to try it great work
2 likes

@ajay_sreeram
I was trying 1.5 7b, it always tries to click few pixels above diagonally. Do we need to pass screen size somewhere from desktop app?

@chadhietala
Can you give details about deployment on vLLM? It seems like the model requires a min-version of it.

@thatboyweeks
plans to open source the full model?

@TsingYoga
Soon there will be~
1 like
