Video yükleniyor...
Video Yüklenemedi
Working on multimodal instruction tuning and finding it hard to scale? Building Web/GUI agents but data is too narrow? Introducing 🚀MultiUI: 7.3M multimodal instructions from 1M webpage UIs, offering diverse data to boost text-rich visual understanding. Key takeaways: 🌟WebUI-trained models show major gains in visual web understanding and agent... show more
57,680 görüntüleme • 1 yıl önce •via X (Twitter)
10 Yorum

Overview of MultiUI, a 7M multimodal instruction-tuning dataset built from a diverse collection of Webpage UIs. The model UIX, trained on MultiUI, generalizes effectively to a broad range of unseen scenarios, including GUI understanding (web and mobile interfaces) and, surprisingly, non-GUI tasks such as document and chart understanding.

Methods: We generated examples by defining representative tasks and prompting text LLMs to create general instructions for various webpages. To enhance diversity, we used strategies like pairing with existing multimodal instructions as in-context examples.

Here is an example of different training tasks in our dataset.

We adopted a two-stage training strategy where we first train the LLaVA models on our 95% MultiUI samples and then we further train the model on a mix of general instructions (e.g., LLaVA) and a small portion (5%) of MultiUI samples. The stage training is mostly useful compared with simply merging the two training datasets. Scaling up different task samples generally leads to better performance.

Further ablation studies show that different tasks have mutual benefits to increase the different abilities of models.

Generating multimodal data with the help of text-based LLMs is an interesting research direction. Web and other potential sources with rendered text would be ideal platforms for exploring such methods. This project took a huge effort to finish in academia. We hope that open-sourcing our work could accelerate the advancement of open science in the field of multimodal LLMs and important downstream applications like multimodal GUI agents!

Nice work! We will also have one paper to be released in the coming week on UI understanding. 😄

Exceptionally well done!

nice work

nice work


