When organizing a home, everyone has unique preferences for where things go. How can household robots learn your preferences from just a few examples? Introducing 𝗧𝗶𝗱𝘆𝗕𝗼𝘁: Personalized Robot Assistance with Large Language Models Project page:

One of the key challenges in robotic household cleanup is deciding where each item goes. People's preferences can vary greatly depending on personal taste or cultural background. One person might want shirts in the drawer, another might want them on the shelf.

How can we infer these user preferences from only a handful of examples in a generalizable way? Our key insight: Summarization with LLMs is an effective way to achieve generalization in robotics

Let's see how this approach works in the household cleanup task. We first collect the preferences of a particular user by asking for a few examples of where different items should go:

Of course, a robot may encounter many objects that are not present in these examples. How would it know where to put these novel objects? Ideally we would generalize the examples into rules that are applicable to a broader range of objects:

This is where LLMs come in. With their remarkable summarization capabilities, LLMs can take in example object placements and adeptly summarize them into general preferences. Here is an example of how GPT-3 can do this:

GPT-3 can then use the summarized preferences to determine placements for novel objects:

Using action primitives and ViLD+CLIP for perception, we deploy this approach on a real-world mobile manipulator called TidyBot. Across 8 diverse test scenarios, TidyBot can put away 85% of objects.

To learn more, check out these links: Project page: Paper: Code:

This work was done @StanfordAILab with an amazing team: @contactrika, Adam Kan, @marionlepert, @andyzengtweets, @SongShuran, @leto__jean, Szymon Rusinkiewicz, Tom Funkhouser

@StanfordAILab @kichitinalopez Este era mi objetivo. Necesito un poco más de financiamiento.