Writing high quality user interface (UI) code requires more than predicting the next token - it requires understanding unique UI code's unique syntax and assessing the rendered output's quality and relevance. Most LLMs struggle to do this.

[alt text: video shows an overview of training process. there is a visualization of a base LLM (starchat-beta) generating 100,000 SwiftUI programs. Only 10,000 compile and only 400 pass the vision-language model threshold. UIs after 5 iterations are much higher quality.

Human front-end devs constantly use compilers to check their code and visually assess their UIs. Similarly, UICoder incorporates tools like compilers and vision-language models, during its training to assess the syntactic correctness and visual relevance of its generated output.

By using these automated tools to score and filter an LLM's self-generated output (rejection sampling), we can construct a high-quality synthetic dataset for fine-tuning, without needing any human-authored examples! Training for multiple iters. improved key metrics.

If you are around @naaclmeeting @naacl this week, please stop by my poster Monday June 17th (tomorrow) at 11 am to learn more about our work. It's also my first time attending NAACL, would love to meet everyone in the community!

This work was done with my collaborators @zinosys, Alan Leung, @barik, @jeffbigham, and @jwnichls at @Apple

yes, this was originally submitted to (then rejected from) CHI 2024 but great to see other conferences valuing HCI research as well

Cool stuff!!

Thanks Jaidev!