Live visual descriptions can aid blind people in understanding their surroundings with autonomy and independence. In our #UIST2024 work, we present WorldScribe that generates automated visual descriptions that are adaptive to the users’ contexts in real-time, in the real world.

First, WorldScribe is adaptive to users' intent, i.e. prioritizing the most pertinent descriptions based on semantic relevance, or visual attributes based on customizations. E.g. users can specify to find specific objects, or ask for descriptions with more color and texture info.

Second, WorldScribe is adaptive to visual contexts, e.g. it provides consecutively succinct descriptions for dynamic visual scenes, while it presents longer and more detailed ones for stable settings. This strategy is particularly useful when describing scenes *live*.

Third, WorldScribe is also adaptive to sound contexts, e.g., increasing description volume in noisy environments, or pausing when conversations start. Such manipulations are related to our prior work on SoundShift in #DIS2024:

WorldScribe is powered by a suite of vision, language, and sound recognition models, and introduces a description generation pipeline with different VLMs that balances the tradeoffs between their richness and latency to support real-time usage.

Our user study with blind participants and subsequent pipeline evaluation show that WorldScribe can provide real-time and fairly accurate visual descriptions to facilitate environment understanding that is adaptive and customized to users' contexts.

However, there's still a lot more to do to make automated live descriptions truly context-aware and humanized, such as incorporating additional real-world knowledge like GPS and maps, adapting to users' changing intents, and embedding short- and long-term memory into the system.

WorldScribe is led by Ruei-Che Chang @RueiChe, who has been doing a series of work on multimodal real-world accessibility, e.g., OmniScribe, SoundShift, and WorldScribe. We will present and demo this work at #UIST2024. Check out the paper here:

Super cool work! Love how it focuses on inferring user intents and providing contextually-relevant information.

Thanks Toby! See ya in Pittsburgh!

good work