正在加载视频...
视频加载失败
gpu poor? I wrote a simple distributed training module to train a vision model on three M1 macbooks. it's open sourced! i wanted to build a distributed training tool for M-series macs but I faced two hurdles: network latency is extremely high and my friends weren't happy with lending... show more
104,325 次观看 • 2 年前 •via X (Twitter)
65 条评论
wake up babe omkaar just dropped federated learning 🤩
😂😂😂
shout out @parthsareen for the code reviews
also shoutout @awnihannun for the mlx idea
Can't it be made decentralised (in terms of the aggregator) to eliminate the single point of failure
yeah i dont see why not, there are a few companies working on that idea like @PrimeIntellect
@PrimeIntellect Would love to see your approach
federated learning opens up entirely new use cases for ml as intelligence is democratized. keep data decentralized and train across networks.
Brilliant 🔥👏👏
That’s a very interesting concept!!
pro tip, post the git link - looks cool - might work on an mlx version this week if I have time
@TarianLovesArch Ah I actually did not make it clear that this supports mlx as well. its actually a very simple change from my repo, just import ml and change the file but retain the variables. I'll make a post on this later.
Makes sense yeah that’s what I figured - very cool
@TarianLovesArch i would still love contributions + optimizations!
Yeah I plan on digging in this week see what I can do (although I’m sure it’s 100% perfect)
@TarianLovesArch dawg between a full-time job and contracting, I only get a few hours a week to work on this loooool, im sure you will find many areas of optimization that I did not want to spend my time on
Loool i fucking feel that
have you considered @shog_AGI for some part of the datastorage?
@shog_AGI not really havent looked into them but i am fine w distributed in-device storage
Does this also work with non-macs ?
yup, just change device in from device("mps") to device("cpu")
this is sick!!
thank you!!
Bro cooked again 👨🍳🔥
!!!
Goat-Tier video
Soo going to milk my friends' macs for what they are worth with this now 😈
😈😈😈
I've been super interested in this idea for a while. Great work! I'm interested in helping if possible
ofc repo is completely free use, you can edit, redistribute, etc!
🔥
Can't you just use Ray Train?
ray train inspired me to build this lol, this was something I built in 12 hours for fun
Hi @omkizzy can you train a model with those three Mac if they are separated at a long distance?
yeah but the network costs would be astronomical, but my module here does support that
Okay,… So what if you increase The number of Mac’s will the “network cost” would decrease or that is not the factor at all?
in an ideal world, not a factor at all
Okay,… I hope it becomes a bigger thing man. And I will go through it on GitHub so I can understand more and if I will have some more questions I will ask you then.
ofc i welcome all questions! it already is a big thing, really cool companies like @flwrlabs are working on similar ideas!
@flwrlabs Yeah yeah , I just saw someone mention @flwlabs on your feed , it is so cool 😎 man
Awesome. This idea as crossed me multiple times. Happy to see it being implemented
@omkizzy you’re the hero we didn’t know about but need lol p.s. what are you up to May10-12?
im going to be in school, down to miss if you have something interesting going on 👀
cc: @parthsareen
@parthsareen @parthsareen come share extensible AI with the builders too!
Great video, once again!!
If you ever need more compute try @PrimeIntellect
@yikesawjeez 👀
@yikesawjeez they were kind enough to repost!
@yikesawjeez this project speaks for itself cooler release than llama3 tbh lfg
@yikesawjeez no way thats a huge compliment thank u!
@thepatch_kev we love distributed computing @BacalhauProject , @BigscienceW , @m_ryabinin , check it out. maybe swap notes/grant offers with our friend here ;)
@thepatch_kev @BacalhauProject @BigscienceW @m_ryabinin would love to chat w anyone interested!
Setting the standards for video quality! Great stuff here 👏
@AmritBuilds recorded on my iphone!
this is dope dude!
thank you!
The GPU poors shall be vindicated
@TiskTusk
Yoo, any tutorial on how u built it? Curious lol
it will take me some time to do that man, there is a lot going on lol
federated learning on macbooks is honestly genius. network latency is the real bottleneck but gradient compression + local sgd steps could help. m1 metal api >
Really cool project. Just made a PR to speed things up and for the learning to continue if one of the learners fails.
sick will check it out later!
So you expect a guy to have poor GPU but have 3 M1 Macbooks 🥴
![This is "Sequential Dexterity", a neural network that controls a robot arm to build legos given a manual 🤖 To do this task, the robot needs to chain together multiple skills (grasping, re-orienting, pushing, etc.) and execute without compounding error. I find some very simple but effective insights in the paper: - Skill chaining over a long sequence is hard. We need a way to "smooth it over". - Forward smoothing: train skill [i] to good performance, run it many times to generate possible "end states", and sample from these states to provide initialization for skill [i+1]. Then start training skill [i+1], and so on. - The problem is that skill [i] is unaware of [i+1], so it may produce bad end states that make life hard for [i+1]. - Backward smoothing: identify what are the good starting states for skill [i+1], then finetune the previous skill [i] towards those states by reinforcement learning. Website: Authors: Yuanpei Chen*, Chen Wang*, Li Fei-Fei, C. Karen Liu Paper: "Sequential Dexterity: Chaining Dexterous Policies for Long-Horizon Manipulation"](https://image.24vids.com/tw-1700207947350163765/ext_tw_video_thumb/1700199950930554880/pu/img/TlN7OycX8wZqe9zJ.jpg)
