We’ve seen a gazillion startups using OpenAI APIs to do “co-pilot for X”. What’s next? Enter *physical* co-pilot! Here’s a compelling demo: you improvise by playing a “low resolution” piano, and the co-pilot compiles it real-time to Hi-Fi music! It unleashes our inner pianist.🧵

What’s behind the magic? “Piano Genie” is a discrete autoencoder architecture that uses LSTM to map piano notes to low-res controller buttons, then decode back to the piano space. It’s trained on 1400 virtuosic performances from the International Piano-e-Competition. 2/

Remarkably, this project was done in 2018, *before* the age of LLMs and co-pilot. That’s why it used LSTM. Brilliant work by @chrisdonahuey @iansimon @sedielem from @GoogleMagenta team. They recently did another super cool project called “SingSong” — your reverse Karaoke👇 3/

Piano Genie website: Cool web demo: Welcome to check out my past top threads here 🙌

@chrisdonahuey Sorry to be a negative voice here but that wasn’t really that impressive. That was taking Cmajor (notice no black keys were played) and adding a few rules to it to move up and down the registers systematically in the programmed sequence.

@chrisdonahuey I suppose the neural network learned the more obvious rules from data and couldn’t model the other intricacies. Thanks for the feedback!

@chrisdonahuey Musicians have been doing this stuff for decades, except their versions sound good

@chrisdonahuey no melody, no harmony, no modality, trash mindless key strokes on second look, one scale, sinus movement up and down octaves, no touch, no feelings any child can do this heck I even know children who can do way better

@chrisdonahuey Wii Music did something similar, probably without ML in 2008, and algorithmic composition has a long history. Pat Metheny built a whole tour around it :

@chrisdonahuey I’m unaware of this. Thanks for the pointer!

@chrisdonahuey why don’t you just hit the buttons on the piano