Introducing 𝗥𝗼𝗯𝗼𝗣𝗶𝗮𝗻𝗶𝘀𝘁 🎹🤖, a new benchmark for high-dimensional robot control! Solving it requires mastering the piano with two anthropomorphic hands. This has been one year in the making, and I couldn’t be happier to release it today! Some highlights below:

First, if you enjoyed the video above, we have a live demo that runs MuJoCo in your browser using Javascript and Web Assembly! You can accompany the robot (drag the keys down), or be adversarial and tug at the fingers 🙃

Piano playing is a challenging task: it requires 1) precision in space and time, 2) coordinating 2 hands and 10 fingers, and 3) strategically planning key presses to facilitate future ones. Task success is also highly interpretable: does it sound good?

On an initial repertoire of 150 variable-difficulty songs, we investigate the performance envelope of a model-free (RL) and model-based (MPC) method and demonstrate that, while there is ample room for improvement, our policies are able to produce some compelling performances.

Overall, RoboPianist provides a rich and challenging environment with human-interpretable results, and is highly amenable to future expansion thanks to the abundance of song data freely available on the web (MIDI files and YouTube videos).

RoboPianist also provides exciting opportunities for other research directions, including in multi-task learning, zero-shot generalization, multimodal (sound, vision, touch) learning, and imitation learning!

This work wouldn’t have been possible without the support of my advisor @pabbeel, the amazing mentorship of @andyzengtweets and @peteflorence, and the most wonderful cast of collaborators @smithlaura1028 @yuvaltassa @taylorhowell @Sumeet_Robotics @xbpeng4 and Nimrod Gileadi.

To learn more about 𝗥𝗼𝗯𝗼𝗣𝗶𝗮𝗻𝗶𝘀𝘁, check out the links below: • 💻: • 📖: • 🎮: Spoiler alert: You can dig in today with `pip install robopianist` 🎹🤖

There are a plethora of videos on YouTube of hobbyist / research projects tackling robotic piano playing. I made a playlist here:

Piano playing robots aren't new! The WABOT, one of the first humanoid robots created in 1970, could talk and play an electronic organ:

The code for the web demo can be found here: