Loading video...

Video Failed to Load

There was a problem loading this video. This could be due to a temporary network issue or the video might be unavailable.

Sketch2Code: Evaluating Vision-Language Models for Interactive Web Design Prototyping @RyanLi0802 Diyi Yang Paper: Project Page: Demo:

Yanzhe Zhang

1,100 subscribers

28,377 views • 1 year ago •via X (Twitter)

Anya Rossi• Live Now

Private livecam show

7 Comments

James Landay1 year ago

@RyanLi0802 @Diyi_Yang Just because you can recognize a sketch as a UI doesn't mean the best next step is to turn it into a "formal" UI prototype.

Yanzhe Zhang1 year ago

We collected 700+ sketches and studied two multi-turn interactions beyond direct generation: feedback-following and question-asking.

Yanzhe Zhang1 year ago

While user feedback is effective, the current models are not good at asking questions by themselves.

Yanzhe Zhang1 year ago

A user study with UI/UX experts reveals a significant preference for proactive question-asking over passive feedback reception. This highlights the need to develop more effective paradigms for multi-turn conversational agents.

Fission Lab Ai1 year ago

@RyanLi0802 @Diyi_Yang Such a cool project! At Fission AI Lab, we know how AI can transform web design. I'd love to hear more about your biggest challenges with Sketch2Code, @StevenyzZhang. Let's brainstorm solutions together!

Lyman1 year ago

@RyanLi0802 @Diyi_Yang nice work

GoatStack.AI1 year ago

@RyanLi0802 @Diyi_Yang AI Summary: The paper presents Sketch2Code, a novel framework that evaluates Vision-Language Models (VLMs) for automating the transformation of low-fidelity sketches into web UI prototypes. By collecting 731...

Related Videos

Check out our new paper that turns (text, sparse images, videos) => (dynamic 3D scenes)! I can't get over how cool the interactive demo is. Try it out for yourself on the project page:

Check out our new paper that turns (text, sparse images, videos) => (dynamic 3D scenes)! I can't get over how cool the interactive demo is. Try it out for yourself on the project page:

Aleksander Holynski

37,520 views • 1 year ago

VLAExplain — Interpreting Vision-Language-Action (VLA) Models VLAExplain is an interpretability toolkit designed to help users visually understand the inner workings of Vision-Language-Action (VLA) models. Currently, attention analysis is supported for both the pi05 and unifolm-vla models. For details, please check pi05 and UnifoLM-VLA readme files respectively. Demo of pi05 in action:

VLAExplain — Interpreting Vision-Language-Action (VLA) Models VLAExplain is an interpretability toolkit designed to help users visually understand the inner workings of Vision-Language-Action (VLA) models. Currently, attention analysis is supported for both the pi05 and unifolm-vla models. For details, please check pi05 and UnifoLM-VLA readme files respectively. Demo of pi05 in action:

Ryohei Sasaki@engineer

12,774 views • 5 months ago

🚀 We open-sourced LongLive — interactive, real-time long-video generation. 👥Generates video in real time as users enter text prompts. ⚡️20.7 FPS on a single H100,⏱️up to 240s per clip. 🎬Fine-tunes SOTA short-video models (e.g., Wan) into long-video generators. 🌍One step closer to World Models. All code for training & inference, model weights, demo page, and videos released! Paper: Code: Model: Demo Page: Introduction Video:

🚀 We open-sourced LongLive — interactive, real-time long-video generation. 👥Generates video in real time as users enter text prompts. ⚡️20.7 FPS on a single H100,⏱️up to 240s per clip. 🎬Fine-tunes SOTA short-video models (e.g., Wan) into long-video generators. 🌍One step closer to World Models. All code for training & inference, model weights, demo page, and videos released! Paper: Code: Model: Demo Page: Introduction Video:

Yukang Chen

11,943 views • 1 year ago

Humans learn and improve from failures. Similarly, foundation models adapt based on human feedback. Can we leverage this failure understanding to enhance robotics systems that use foundation models? Introducing AHA—a vision-language model for detecting and reasoning over failures in robotic manipulation. Project page: 🧵Thread👇 Aha!

Humans learn and improve from failures. Similarly, foundation models adapt based on human feedback. Can we leverage this failure understanding to enhance robotics systems that use foundation models? Introducing AHA—a vision-language model for detecting and reasoning over failures in robotic manipulation. Project page: 🧵Thread👇 Aha!

Jiafei Duan

48,830 views • 2 years ago

Yay, finally! Introducing Vision Banana🍌 from Google DeepMind, our unified model that outperforms SoTA specialist models on various vision tasks! By treating 2D/3D vision tasks as image generation, we unlock a new foundation for CV. Project page: (1/5)

Yay, finally! Introducing Vision Banana🍌 from Google DeepMind, our unified model that outperforms SoTA specialist models on various vision tasks! By treating 2D/3D vision tasks as image generation, we unlock a new foundation for CV. Project page: (1/5)

Songyou Peng

290,490 views • 5 months ago

👀Humans compare images by looking back and forth. Many open-weight VLMs encode each image independently, and defer comparison to the LM. We introduce SVE: Stateful Visual Encoders for Vision-Language Models, where the visual encoder itself becomes change-aware. 🌐Project: 📰Paper: 💻Code: 1/n

👀Humans compare images by looking back and forth. Many open-weight VLMs encode each image independently, and defer comparison to the LM. We introduce SVE: Stateful Visual Encoders for Vision-Language Models, where the visual encoder itself becomes change-aware. 🌐Project: 📰Paper: 💻Code: 1/n

Zirui "Colin" Wang

63,853 views • 4 months ago

we won the Google Build with AI prize track out of 200+ teams at Hack Canada this weekend! 🏆 we built an interactive sandbox for transit design. had such a blast working on both design and frontend for this project yipeee! Richard Li Evan Yang @christo28120856

we won the Google Build with AI prize track out of 200+ teams at Hack Canada this weekend! 🏆 we built an interactive sandbox for transit design. had such a blast working on both design and frontend for this project yipeee! Richard Li Evan Yang @christo28120856

Fiona Fang

98,033 views • 7 months ago

Excited to share our new work: StreamingVLM! 🚀 We tackle a major challenge for Vision-Language Models (VLMs): understanding infinite video streams in real-time without latency blowing up or running out of memory. Paper: Code:

Excited to share our new work: StreamingVLM! 🚀 We tackle a major challenge for Vision-Language Models (VLMs): understanding infinite video streams in real-time without latency blowing up or running out of memory. Paper: Code:

Guangxuan Xiao

92,141 views • 11 months ago

TidyBot: Personalized Robot Assistance with Large Language Models approach enables fast adaptation and achieves 91.2% accuracy on unseen objects in our benchmark dataset. We also demonstrate our approach on a real-world mobile manipulator called TidyBot, which successfully puts away 85.0% of objects in real-world test scenarios abs: project page: paper page:

TidyBot: Personalized Robot Assistance with Large Language Models approach enables fast adaptation and achieves 91.2% accuracy on unseen objects in our benchmark dataset. We also demonstrate our approach on a real-world mobile manipulator called TidyBot, which successfully puts away 85.0% of objects in real-world test scenarios abs: project page: paper page:

AK

326,009 views • 3 years ago

#NeurIPS23 Even wonder when Generative AI be useful for designing real-world robots. Excited to share our oral paper DiffuseBot! Diffusion models and Differentiable Physics are all you need! 😀😀😀😀 Project page:

#NeurIPS23 Even wonder when Generative AI be useful for designing real-world robots. Excited to share our oral paper DiffuseBot! Diffusion models and Differentiable Physics are all you need! 😀😀😀😀 Project page:

Chuang Gan

34,551 views • 2 years ago

When organizing a home, everyone has unique preferences for where things go. How can household robots learn your preferences from just a few examples? Introducing 𝗧𝗶𝗱𝘆𝗕𝗼𝘁: Personalized Robot Assistance with Large Language Models Project page:

When organizing a home, everyone has unique preferences for where things go. How can household robots learn your preferences from just a few examples? Introducing 𝗧𝗶𝗱𝘆𝗕𝗼𝘁: Personalized Robot Assistance with Large Language Models Project page:

Jimmy Wu

236,381 views • 3 years ago

Vision-language models can control robots, but what if the prompt is too complex for the robot to follow directly? We developed a way to get robots to “think through” complex instructions, feedback, and interjections. We call it the Hierarchical Interactive Robot (Hi Robot).

Vision-language models can control robots, but what if the prompt is too complex for the robot to follow directly? We developed a way to get robots to “think through” complex instructions, feedback, and interjections. We call it the Hierarchical Interactive Robot (Hi Robot).

Physical Intelligence

117,221 views • 1 year ago

Five frontier models. One simple prompt. No retries. No blender. Just a simple frontend web design. Fable 5.1 vs. GPT-6 Astra vs. DeepSeek v4.1 Flash vs. GLM 5.3 Flash vs. Qwen3.8 Flash Next Which model built the best looking luxury watch landing page? IMO Fable 5.1 and GLM 5.3 Flash look the best. LIVE DEMO:

Five frontier models. One simple prompt. No retries. No blender. Just a simple frontend web design. Fable 5.1 vs. GPT-6 Astra vs. DeepSeek v4.1 Flash vs. GLM 5.3 Flash vs. Qwen3.8 Flash Next Which model built the best looking luxury watch landing page? IMO Fable 5.1 and GLM 5.3 Flash look the best. LIVE DEMO:

Mia

16,501 views • 22 days ago

You might be wondering... How does Project Genie work?🤔 Great question. Project Genie is a prototype web app powered by several of our most advanced AI models, each bringing a unique capability to the equation. From Genie 3's ability to simulate the physics and interactions of any scenario, to Nano Banana's sketching and design, to Gemini's advanced world knowledge and reasoning, this combination results in Project Genie's unique, interactive, (and really cool) user experience. But don't just take our word for it, try it today at:

You might be wondering... How does Project Genie work?🤔 Great question. Project Genie is a prototype web app powered by several of our most advanced AI models, each bringing a unique capability to the equation. From Genie 3's ability to simulate the physics and interactions of any scenario, to Nano Banana's sketching and design, to Gemini's advanced world knowledge and reasoning, this combination results in Project Genie's unique, interactive, (and really cool) user experience. But don't just take our word for it, try it today at:

Google AI

136,689 views • 8 months ago

New open source project: Easel, a macOS workspace for AI-assisted product design and frontend iteration powered by the Codex SDK. Think of it as "Claude Design for Codex." It brings together a local project library, design-system setup, Codex chat, and a live web preview with a point-and-click inspector. Set up your design system however you like. Import one from Figma , bring your own, or have the agent generate it as a design.md. Then use it to build prototypes and slide decks. Codex-first, with Claude and other AI providers and local models planned for future releases. OpenAI Developers

New open source project: Easel, a macOS workspace for AI-assisted product design and frontend iteration powered by the Codex SDK. Think of it as "Claude Design for Codex." It brings together a local project library, design-system setup, Codex chat, and a live web preview with a point-and-click inspector. Set up your design system however you like. Import one from Figma , bring your own, or have the agent generate it as a design.md. Then use it to build prototypes and slide decks. Codex-first, with Claude and other AI providers and local models planned for future releases. OpenAI Developers

James Rochabrun

61,608 views • 3 months ago

Enjoy our recent brand and design project for the Databricks Developer Portal. 🔉 While many companies prefer to fully vibe-code their websites, I’m glad Databricks used vibe coding the right way - mainly for quick prototyping and communicating the overall page structure and messaging with our design team, rather than defining the final look. I’m really happy with how all our concepts and ideas, after passing through the prism of actual business needs, translated into a visual direction that feels mature and professional while still staying appealing to developers. Go and check it out for yourself at developers . databricks . com. Especially the interactive hero section, which is full of Easter eggs.

Alex Barashkov

18,552 views • 2 months ago

Very excited to share the best coding model we’ve ever built! Today we’re launching Gemini 2.5 Pro Preview 'I/O edition' with massively improved coding capabilities. Ranks no.1 on LMArena in Coding and no.1 on the WebDev Arena Leaderboard. It’s especially good at building interactive web apps - this demo shows how it can be helpful for prototyping ideas. Try it in Google Gemini, Vertex AI, and AI Studio Enjoy the pre-I/O goodies !

Very excited to share the best coding model we’ve ever built! Today we’re launching Gemini 2.5 Pro Preview 'I/O edition' with massively improved coding capabilities. Ranks no.1 on LMArena in Coding and no.1 on the WebDev Arena Leaderboard. It’s especially good at building interactive web apps - this demo shows how it can be helpful for prototyping ideas. Try it in Google Gemini, Vertex AI, and AI Studio Enjoy the pre-I/O goodies !

Demis Hassabis

996,705 views • 1 year ago

Introducing Vision Arena! Inspired by the awesome Chatbot Arena, we built a web demo on Hugging Face for testing Vision LMs (GPT-4V, Gemini, Llava, Qwen-VL, etc.). You can easily test two VLMs side by side and vote! It’s still a work-in-progress. Feedbacks are welcome! 🔗 Kudos to the main contributor Yujie Lu (Yujie Lu), and the team: Dongfu Jiang Wenhu Chen Ai2 ; Thanks to LMSYS Org’s great code & design!

Introducing Vision Arena! Inspired by the awesome Chatbot Arena, we built a web demo on Hugging Face for testing Vision LMs (GPT-4V, Gemini, Llava, Qwen-VL, etc.). You can easily test two VLMs side by side and vote! It’s still a work-in-progress. Feedbacks are welcome! 🔗 Kudos to the main contributor Yujie Lu (Yujie Lu), and the team: Dongfu Jiang Wenhu Chen Ai2 ; Thanks to LMSYS Org’s great code & design!

Bill Yuchen Lin

146,020 views • 2 years ago