New side project: Samuel, a model I trained to mimic speech using Pink Trombone, a (pre-existing) silly vocal tract simulator. 🧵

From an ML perspective, it's a speech autoencoder where the decoder is Pink Trombone. It's a non-trivial task: Pink Trombone is non-differentiable, sample-by-sample, and if you optimize the model to just match the waveform, it's not intelligible. Write-up coming soon!

Try it: Credits to Neil Thapen for the original Pink Trombone and @ConcreteSciFi for the programmable version I used as the starting point.

This is awesome. But recording on Firefox under Linux seems a bit slow. If I can get it working I'll send a PR :)

Thank you, PRs with fixes welcome!

Found and fixed, I'll finish testing and send a PR shortly. Also, this always trips me up when creating from the API rather than with `gh repo create` - you aren't watching the repo so you won't get notifications on PRs or issues!

I'm not sure if this really is how vocal tract is supposed to move

omfg @repligate someone actually did it

couldn't this technically be used as an extremely efficient, human voice focused, audio encoder? the data to move the pink trombone correctly is probably tiny, this seems very intelligible and is more dynamic than just storing text and playing it back with TTS.

I wonder how well something like this would work as a super low bitrate voice codec.

Nice! I've also built speech simulator based on pink trombone You can control the intonation here manually I guess there can be switchable phoneme tables, based from your example, for TTS with accents.

i am proud of you i tried it but had alien voice problems you made it sound so good mine sounded awful yours is amazing i will finish my work using your work lol thank you

AAA III UUU EEE OOOOOOOOOOO

I LOVE THIS

Jesus Christ. Also, I want a version that does this with I have no idea why I want it, and I am exceedingly unlikely to make any music at all with it, but I still want it.

this is insane I love it

I played with pink trombone years ago and found it very hard to control. This is very impressive – it captured my singing quite well.

Very cool! How hard would it be to simulate the rest of the pharynx and train a model to sing with it?

what an amazing project I love it. who could have thought speech could be encoded in a pink tormbone lol 🎺

this is cool

Now to build a robot version and train it the same way

"Ni Ni Ni Niiii-" The internet has ruined me

@andrew_n_carr Copying this brb

Super interesting!

This is super cool, looking forward to the writeup! Pre-registering a guess: * train small frame-level neural proxy synth on random data * freeze proxy & train a controller on speech w/ frame-level ASR feature-matching & F0 losses * probably no need to post-train w/ real thing?

@ravarcheon look what you've done!

epic

Really cool project! Can't wait to hear more about the research behind it

Reminds me of Air: MoonSafari/Remember.

Really cool work

This is really cool! but it seems to struggle detecting higher pitches even though its capable of replicating them.

cool, I had the same idea to use pink trombone for something similar but the moment I was interested into, other 20 ideas spawned, happy to see someone doing this!! I think that pink trombone could be improved to reproduce better (more) sounds and this would improve the outcome, I will take a look at the repo, thanks for sharing!

this is lit it would also be interesting to transform the concept into a way to see how to pronounce certain words (showing where you should put your tongue)

awesome

That’s really awesome man. Keep up the good work!

This would also be a great starting point to improve mouth/speech animations in 3d movies.

Now train a video model and use classification controllers

very cool!