gpu poor? I wrote a simple distributed training module to train a vision model on three M1 macbooks. it's open sourced! i wanted to build a distributed training tool for M-series macs but I faced two hurdles: network latency is extremely high and my friends weren't happy with lending... out their macbooks during exam season LOL. i pivoted it to a federated learning module (where network costs are acceptable). it still needs more tweaks like removing the centralized datastore and compressing payloads but its 90% there :) so contributors are welcome federated learning is a hard problem to productionize... but its not that hard to understand. thanks to Javier Fernandez from Flower (hiring!) for giving me tips on optimizations. got questions? shoot away :)show more

wake up babe omkaar just dropped federated learning 🤩

😂😂😂

shout out @parthsareen for the code reviews

also shoutout @awnihannun for the mlx idea

Can't it be made decentralised (in terms of the aggregator) to eliminate the single point of failure

yeah i dont see why not, there are a few companies working on that idea like @PrimeIntellect

@PrimeIntellect Would love to see your approach

federated learning opens up entirely new use cases for ml as intelligence is democratized. keep data decentralized and train across networks.

Brilliant 🔥👏👏

That’s a very interesting concept!!

pro tip, post the git link - looks cool - might work on an mlx version this week if I have time

@TarianLovesArch Ah I actually did not make it clear that this supports mlx as well. its actually a very simple change from my repo, just import ml and change the file but retain the variables. I'll make a post on this later.

Makes sense yeah that’s what I figured - very cool

@TarianLovesArch i would still love contributions + optimizations!

Yeah I plan on digging in this week see what I can do (although I’m sure it’s 100% perfect)

@TarianLovesArch dawg between a full-time job and contracting, I only get a few hours a week to work on this loooool, im sure you will find many areas of optimization that I did not want to spend my time on

Loool i fucking feel that

have you considered @shog_AGI for some part of the datastorage?

@shog_AGI not really havent looked into them but i am fine w distributed in-device storage

Does this also work with non-macs ?

yup, just change device in from device("mps") to device("cpu")

this is sick!!

thank you!!

Bro cooked again 👨‍🍳🔥

!!!

Goat-Tier video

Soo going to milk my friends' macs for what they are worth with this now 😈

😈😈😈

I've been super interested in this idea for a while. Great work! I'm interested in helping if possible

ofc repo is completely free use, you can edit, redistribute, etc!

🔥

Can't you just use Ray Train?

ray train inspired me to build this lol, this was something I built in 12 hours for fun

Hi @omkizzy can you train a model with those three Mac if they are separated at a long distance?

yeah but the network costs would be astronomical, but my module here does support that

Okay,… So what if you increase The number of Mac’s will the “network cost” would decrease or that is not the factor at all?

in an ideal world, not a factor at all

Okay,… I hope it becomes a bigger thing man. And I will go through it on GitHub so I can understand more and if I will have some more questions I will ask you then.

ofc i welcome all questions! it already is a big thing, really cool companies like @flwrlabs are working on similar ideas!

@flwrlabs Yeah yeah , I just saw someone mention @flwlabs on your feed , it is so cool 😎 man

Awesome. This idea as crossed me multiple times. Happy to see it being implemented

@omkizzy you’re the hero we didn’t know about but need lol p.s. what are you up to May10-12?

im going to be in school, down to miss if you have something interesting going on 👀

cc: @parthsareen

@parthsareen @parthsareen come share extensible AI with the builders too!

Great video, once again!!

If you ever need more compute try @PrimeIntellect

@yikesawjeez 👀

@yikesawjeez they were kind enough to repost!

@yikesawjeez this project speaks for itself cooler release than llama3 tbh lfg

@yikesawjeez no way thats a huge compliment thank u!

@thepatch_kev we love distributed computing @BacalhauProject , @BigscienceW , @m_ryabinin , check it out. maybe swap notes/grant offers with our friend here ;)

@thepatch_kev @BacalhauProject @BigscienceW @m_ryabinin would love to chat w anyone interested!

Setting the standards for video quality! Great stuff here 👏

@AmritBuilds recorded on my iphone!

this is dope dude!

thank you!

The GPU poors shall be vindicated

@TiskTusk

Yoo, any tutorial on how u built it? Curious lol

it will take me some time to do that man, there is a lot going on lol

federated learning on macbooks is honestly genius. network latency is the real bottleneck but gradient compression + local sgd steps could help. m1 metal api >

Really cool project. Just made a PR to speed things up and for the learning to continue if one of the learners fails.

sick will check it out later!

So you expect a guy to have poor GPU but have 3 M1 Macbooks 🥴