Video wird geladen...

Video konnte nicht geladen werden

Zur Startseite

gpu poor? I wrote a simple distributed training module to train a vision model on three M1 macbooks. it's open sourced! i wanted to build a distributed training tool for M-series macs but I faced two hurdles: network latency is extremely high and my friends weren't happy with lending...

104,325 Aufrufe • vor 2 Jahren •via X (Twitter)

65 Kommentare

Profilbild von parth
parthvor 2 Jahren

wake up babe omkaar just dropped federated learning 🤩

Profilbild von omkaar
omkaarvor 2 Jahren

😂😂😂

Profilbild von omkaar
omkaarvor 2 Jahren

shout out @parthsareen for the code reviews

Profilbild von omkaar
omkaarvor 2 Jahren

also shoutout @awnihannun for the mlx idea

Profilbild von sid
sidvor 2 Jahren

Can't it be made decentralised (in terms of the aggregator) to eliminate the single point of failure

Profilbild von omkaar
omkaarvor 2 Jahren

yeah i dont see why not, there are a few companies working on that idea like @PrimeIntellect

Profilbild von sid
sidvor 2 Jahren

@PrimeIntellect Would love to see your approach

Profilbild von siddhartha
siddharthavor 2 Jahren

federated learning opens up entirely new use cases for ml as intelligence is democratized. keep data decentralized and train across networks.

Profilbild von Harry
Harryvor 2 Jahren

Brilliant 🔥👏👏

Profilbild von Yaya Soumah
Yaya Soumahvor 2 Jahren

That’s a very interesting concept!!

Profilbild von Mathieu Tuli
Mathieu Tulivor 2 Jahren

pro tip, post the git link - looks cool - might work on an mlx version this week if I have time

Profilbild von omkaar
omkaarvor 2 Jahren

@TarianLovesArch Ah I actually did not make it clear that this supports mlx as well. its actually a very simple change from my repo, just import ml and change the file but retain the variables. I'll make a post on this later.

Profilbild von Mathieu Tuli
Mathieu Tulivor 2 Jahren

Makes sense yeah that’s what I figured - very cool

Profilbild von omkaar
omkaarvor 2 Jahren

@TarianLovesArch i would still love contributions + optimizations!

Profilbild von Mathieu Tuli
Mathieu Tulivor 2 Jahren

Yeah I plan on digging in this week see what I can do (although I’m sure it’s 100% perfect)

Profilbild von omkaar
omkaarvor 2 Jahren

@TarianLovesArch dawg between a full-time job and contracting, I only get a few hours a week to work on this loooool, im sure you will find many areas of optimization that I did not want to spend my time on

Profilbild von Mathieu Tuli
Mathieu Tulivor 2 Jahren

Loool i fucking feel that

Profilbild von HDP
HDPvor 2 Jahren

have you considered @shog_AGI for some part of the datastorage?

Profilbild von omkaar
omkaarvor 2 Jahren

@shog_AGI not really havent looked into them but i am fine w distributed in-device storage

Profilbild von Atharva Pingale
Atharva Pingalevor 2 Jahren

Does this also work with non-macs ?

Profilbild von omkaar
omkaarvor 2 Jahren

yup, just change device in from device("mps") to device("cpu")

Profilbild von Silen Naihin
Silen Naihinvor 2 Jahren

this is sick!!

Profilbild von omkaar
omkaarvor 2 Jahren

thank you!!

Profilbild von JaysmooveV2
JaysmooveV2vor 2 Jahren

Bro cooked again 👨‍🍳🔥

Profilbild von omkaar
omkaarvor 2 Jahren

!!!

Profilbild von Bassel Al Omari
Bassel Al Omarivor 2 Jahren

Goat-Tier video

Profilbild von Virat Chauhan
Virat Chauhanvor 2 Jahren

Soo going to milk my friends' macs for what they are worth with this now 😈

Profilbild von omkaar
omkaarvor 2 Jahren

😈😈😈

Profilbild von eggo
eggovor 2 Jahren

I've been super interested in this idea for a while. Great work! I'm interested in helping if possible

Profilbild von omkaar
omkaarvor 2 Jahren

ofc repo is completely free use, you can edit, redistribute, etc!

Profilbild von NIK
NIKvor 2 Jahren

🔥

Profilbild von LA Bloke
LA Blokevor 2 Jahren

Can't you just use Ray Train?

Profilbild von omkaar
omkaarvor 2 Jahren

ray train inspired me to build this lol, this was something I built in 12 hours for fun

Profilbild von io ɳ
io ɳvor 2 Jahren

Hi @omkizzy can you train a model with those three Mac if they are separated at a long distance?

Profilbild von omkaar
omkaarvor 2 Jahren

yeah but the network costs would be astronomical, but my module here does support that

Profilbild von io ɳ
io ɳvor 2 Jahren

Okay,… So what if you increase The number of Mac’s will the “network cost” would decrease or that is not the factor at all?

Profilbild von omkaar
omkaarvor 2 Jahren

in an ideal world, not a factor at all

Profilbild von io ɳ
io ɳvor 2 Jahren

Okay,… I hope it becomes a bigger thing man. And I will go through it on GitHub so I can understand more and if I will have some more questions I will ask you then.

Profilbild von omkaar
omkaarvor 2 Jahren

ofc i welcome all questions! it already is a big thing, really cool companies like @flwrlabs are working on similar ideas!

Profilbild von io ɳ
io ɳvor 2 Jahren

@flwrlabs Yeah yeah , I just saw someone mention @flwlabs on your feed , it is so cool 😎 man

Profilbild von Ankur Shukla
Ankur Shuklavor 2 Jahren

Awesome. This idea as crossed me multiple times. Happy to see it being implemented

Profilbild von pooria
pooriavor 2 Jahren

@omkizzy you’re the hero we didn’t know about but need lol p.s. what are you up to May10-12?

Profilbild von omkaar
omkaarvor 2 Jahren

im going to be in school, down to miss if you have something interesting going on 👀

Profilbild von omkaar
omkaarvor 2 Jahren

cc: @parthsareen

Profilbild von pooria
pooriavor 2 Jahren

@parthsareen @parthsareen come share extensible AI with the builders too!

Profilbild von Javier Fernandez
Javier Fernandezvor 2 Jahren

Great video, once again!!

Profilbild von VA
VAvor 2 Jahren

If you ever need more compute try @PrimeIntellect

Profilbild von thecollabagepatch
thecollabagepatchvor 2 Jahren

@yikesawjeez 👀

Profilbild von omkaar
omkaarvor 2 Jahren

@yikesawjeez they were kind enough to repost!

Profilbild von thecollabagepatch
thecollabagepatchvor 2 Jahren

@yikesawjeez this project speaks for itself cooler release than llama3 tbh lfg

Profilbild von omkaar
omkaarvor 2 Jahren

@yikesawjeez no way thats a huge compliment thank u!

Profilbild von yikes (:D/acc)
yikes (:D/acc)vor 2 Jahren

@thepatch_kev we love distributed computing @BacalhauProject , @BigscienceW , @m_ryabinin , check it out. maybe swap notes/grant offers with our friend here ;)

Profilbild von omkaar
omkaarvor 2 Jahren

@thepatch_kev @BacalhauProject @BigscienceW @m_ryabinin would love to chat w anyone interested!

Profilbild von Amrit Chera
Amrit Cheravor 2 Jahren

Setting the standards for video quality! Great stuff here 👏

Profilbild von omkaar
omkaarvor 2 Jahren

@AmritBuilds recorded on my iphone!

Profilbild von Tony Kam
Tony Kamvor 2 Jahren

this is dope dude!

Profilbild von omkaar
omkaarvor 2 Jahren

thank you!

Profilbild von TiskTusk
TiskTuskvor 2 Jahren

The GPU poors shall be vindicated

Profilbild von μ-go
μ-govor 2 Jahren

@TiskTusk

Profilbild von Nag
Nagvor 2 Jahren

Yoo, any tutorial on how u built it? Curious lol

Profilbild von omkaar
omkaarvor 2 Jahren

it will take me some time to do that man, there is a lot going on lol

Profilbild von AK
AKvor 10 Monaten

federated learning on macbooks is honestly genius. network latency is the real bottleneck but gradient compression + local sgd steps could help. m1 metal api >

Profilbild von Chris 🇭🇷🇮🇪
Chris 🇭🇷🇮🇪vor 2 Jahren

Really cool project. Just made a PR to speed things up and for the learning to continue if one of the learners fails.

Profilbild von omkaar
omkaarvor 2 Jahren

sick will check it out later!

Profilbild von Ashwin 🇮🇳
Ashwin 🇮🇳vor 2 Jahren

So you expect a guy to have poor GPU but have 3 M1 Macbooks 🥴

Ähnliche Videos

Let's talk about training and how hard you need to push yourself at home before you go for any trials. I remember when I knew that I will be traveling for a trial, i stopped going out of my neighbourhood to play football and I just locked in on training for 2month straight. I will do a roadside jogging from my house to the pitch at Ashiribo and I will train for like 2hrs or more and that was my routine every week except Sundays to rest and I did that for 2 months. Victor McDonald assigned me a coach to train and get me ready and he was by far the hardest I have ever trained. Days where I will tie a tyre around my waist to run on the sand, I did a lot of running,strength training, speed work on the sand with tyre. People who see me train have no idea why I was k!lling myself to train that hard. I once trained with Raphael Edereho a guy who I was looking up at that time and I was not able to walk for 2 days after training with him. I trained like my life depends on it. And I stayed away from many things that will slow me down like, bad friends, drinking, sm0king, including girls and I was just locked in on my purpose to make it happen and it paid off. After 3 months that I left Nigeria the news broke out that I signed for Sölvesborg GIF and People were surprised to see it in the news paper and even my father got asked questions like how or when did I traveled. I thought I trained well enough but to my surprise I was no where near ready at all. I couldn't believe what I saw during the trainings I had to call one of my coaches back home and tell him how hard the trainings are for me. The new generation of young players don't like to train and they except someone to just come and say I will take you to trials. Many have failed not because they don't have the talent but because they don't train enough to meet the standards and social media has made them think otherwise. Your preparation is the most important thing you can do for sure yourself and just like Kobe Bryant said if you are well prepared for it you will not be afraid of failure.

Abiola Dauda

15,507 Aufrufe • vor 4 Monaten