Skip navigation
Introducing UI-TARS-1.5, a vision-language model that beats OpenAI Operator and Claude 3.7 on GUI…
0:00 / 1:17

Introducing UI-TARS-1.5, a vision-language model that beats OpenAI Operator and Claude 3.7 on GUI…

Yujia Qin

@TsingYoga5.6K followers

Visit on X
205 likesDownloadOriginal post

85,212 views

Introducing UI-TARS-1.5, a vision-language model that beats OpenAI Operator and Claude 3.7 on GUI Agent and Game Agent tasks. We've open-sourced a small-size version model for research purposes, more details can be found in our blog. TARS learns solely from a screen, but generalizes beyond a screen! Blog: Model: App:

22 Comments

Comments from the original post on X (Twitter)

  • Yujia Qin@ICLR2025's profile picture

    @TsingYoga

    UI-TARS-1.5 achieves SOTA results on several GUI benchmarks, e.g., OSWorld, WindowsAgent Arena, Online Mind2web, Android World, and ScreenSpot-Pro. These results demonstrate UI-TARS's superiority on computer use, browser use, and phone use. Also, with the GUI Tool, UI-TARS almost matches GPT-4o with the search API

    7 likes

  • Yujia Qin@ICLR2025's profile picture

    @TsingYoga

    Here's a demo from UI-TARS on GUI tasks~

    5 likes

  • Yujia Qin@ICLR2025's profile picture

    @TsingYoga

    To further assess UI-TARS-1.5 in complex, open-ended environments, we tested it on Minecraft—a popular sandbox game well-suited for evaluating embodied intelligence. Unlike static GUI benchmarks, Minecraft requires real-time decision-making in a dynamic 3D space using visual input and low-level controls (mouse and keyboard), closely reflecting real-world computer use.

    4 likes

  • Yujia Qin@ICLR2025's profile picture

    @TsingYoga

    TARS has amazing inference-time scaling ability. With more interaction rounds, TARS achieves far better performance in GUI tasks and Game tasks. The scaling curve surpasses both OpenAI CUA and Claude 3.7. We even observe performance gain when the interaction rounds are over 1000 steps.

    3 likes

  • Yujia Qin@ICLR2025's profile picture

    @TsingYoga

    Gameplay represents a critical frontier for multimodal agents, serving as an ideal testing ground for evaluating complex reasoning, decision-making, and adaptability. Games demand intuitive, common-sense reasoning and strategic foresight, making them perfect benchmarks to test and showcase the advanced cognitive capabilities of multimodal agents. To evaluate UI-TARS-1.5's gameplay proficiency, we selected 14 diverse games from Each model was allowed up to 1,000 interaction steps per game to generate execution traces, repeated across multiple runs.

    3 likes

  • Yujia Qin@ICLR2025's profile picture

    @TsingYoga

    Explore more interesting showcases of UI-TARS on

    2 likes

  • Chris Barber's profile picture

    @chrisbarber

    42% on OSWorld is impressive!

    2 likes

  • Yujia Qin@ICLR2025's profile picture

    @TsingYoga

    Thanks! Will be higher sooner!

    1 like

  • orange.ai's profile picture

    @oran_ge

    Impressive!

    1 like

  • Cua's profile picture

    @trycua

    soon as an agent loop in c/ua 👀

    1 like

  • Yujia Qin@ICLR2025's profile picture

    @TsingYoga

    Sure it will be!

    2 likes

  • yanghan's profile picture

    @BerniceRob27993

    nice work

    1 like

  • Petr Glaser's profile picture

    @BleedingDev

    How well can it play Pokemon? 🤔

    1 like

  • Oli's profile picture

    @Oli82817545

    looks really cool but when can we acess the larger 1.5 and will it be opensource too?

  • Yujia Qin@ICLR2025's profile picture

    @TsingYoga

    Sure! Soon will be

    3 likes

  • Oli's profile picture

    @Oli82817545

    nice really excited to try it great work

    2 likes

  • Ajay Sreeram's profile picture

    @ajay_sreeram

    I was trying 1.5 7b, it always tries to click few pixels above diagonally. Do we need to pass screen size somewhere from desktop app?

  • chadhietala's profile picture

    @chadhietala

    Can you give details about deployment on vLLM? It seems like the model requires a min-version of it.

  • ☼░▒▅'s profile picture

    @thatboyweeks

    plans to open source the full model?

  • Yujia Qin@ICLR2025's profile picture

    @TsingYoga

    Soon there will be~

    1 like