Using tools is often regarded as the hallmark of intelligence. How can robots creatively use tools beyond standard functionalities and make impossible tasks possible? 🤖🗝️ We introduce RoboTool built on LLMs, enabling multiple creative tool-use behaviors, including tool selection, sequential tool use, and tool manufacturing!🧵👇🏼

Robotool could identify the activated environment- and embodiment-related constraints based on language descriptions. For example, it can extract implicit geometric concepts that affect task feasibility, such as the "gap" between the two sofas, even if no concept of "gap" is involved in the description!🤯

By discerning the critical concept, RoboTool enables discriminative tool-use behaviors -- using tools only when necessary -- showing more accurate grounding related to the environment and embodiment instead of being purely dominated by the prior knowledge in the LLMs.

RoboTool can solve long-horizon planning tasks with constraints through creative tool use. But how does it work? 🦾 RoboTool directly outputs executable codes by invoking a set of parameterized skills. It has four modules, with each handling one functionality and building on one LLM, including the Analyzer, Planner, Calculator, and Coder.

We build a creative tool-use benchmark containing six tasks and two robot morphologies to test different creative tool-use behaviors and show the generality of RoboTool. Adding more tasks with different morphologies and complexities in the future would be interesting! 📐

Joint work with amazing collaborators @huang_peide, Wenhao Yu (@Stacormed), Shiqi Liu, @XilunZhang1999, @yaru_niu, Tingnan Zhang, @xf1280, @JieTan42707141, @zhao__ding. Work partially done @GoogleDeepMind. #LLMs #Robotics

@Stacormed @XilunZhang1999 @yaru_niu @xf1280 @JieTan42707141 @zhao__ding @GoogleDeepMind 🌐 Website: 📄arXiv:

@mengdibellaxu Really cool work Mengdi! Excited to read your paper 😊

Thanks Dennis!

@mengdibellaxu Amazing!

@mengdibellaxu Amazing work!