Loading video...

Video Failed to Load

There was a problem loading this video. This could be due to a temporary network issue or the video might be unavailable.

gpu poor? I wrote a simple distributed training module to train a vision model on three M1 macbooks. it's open sourced! i wanted to build a distributed training tool for M-series macs but I faced two hurdles: network latency is extremely high and my friends weren't happy with lending... out their macbooks during exam season LOL. i pivoted it to a federated learning module (where network costs are acceptable). it still needs more tweaks like removing the centralized datastore and compressing payloads but its 90% there :) so contributors are welcome federated learning is a hard problem to productionize... but its not that hard to understand. thanks to Javier Fernandez from Flower (hiring!) for giving me tips on optimizations. got questions? shoot away :)show more

omkaar

4,337 subscribers

104,325 views • 2 years ago •via X (Twitter)

Anya Rossi• Live Now

Private livecam show

65 Comments

parth2 years ago

wake up babe omkaar just dropped federated learning 🤩

omkaar2 years ago

😂😂😂

omkaar2 years ago

shout out @parthsareen for the code reviews

omkaar2 years ago

also shoutout @awnihannun for the mlx idea

sid2 years ago

Can't it be made decentralised (in terms of the aggregator) to eliminate the single point of failure

omkaar2 years ago

yeah i dont see why not, there are a few companies working on that idea like @PrimeIntellect

sid2 years ago

@PrimeIntellect Would love to see your approach

siddhartha2 years ago

federated learning opens up entirely new use cases for ml as intelligence is democratized. keep data decentralized and train across networks.

Harry2 years ago

Brilliant 🔥👏👏

Yaya Soumah2 years ago

That’s a very interesting concept!!

Mathieu Tuli2 years ago

pro tip, post the git link - looks cool - might work on an mlx version this week if I have time

omkaar2 years ago

@TarianLovesArch Ah I actually did not make it clear that this supports mlx as well. its actually a very simple change from my repo, just import ml and change the file but retain the variables. I'll make a post on this later.

Mathieu Tuli2 years ago

Makes sense yeah that’s what I figured - very cool

omkaar2 years ago

@TarianLovesArch i would still love contributions + optimizations!

Mathieu Tuli2 years ago

Yeah I plan on digging in this week see what I can do (although I’m sure it’s 100% perfect)

omkaar2 years ago

@TarianLovesArch dawg between a full-time job and contracting, I only get a few hours a week to work on this loooool, im sure you will find many areas of optimization that I did not want to spend my time on

Mathieu Tuli2 years ago

Loool i fucking feel that

HDP2 years ago

have you considered @shog_AGI for some part of the datastorage?

omkaar2 years ago

@shog_AGI not really havent looked into them but i am fine w distributed in-device storage

Atharva Pingale2 years ago

Does this also work with non-macs ?

omkaar2 years ago

yup, just change device in from device("mps") to device("cpu")

Silen Naihin2 years ago

this is sick!!

omkaar2 years ago

thank you!!

JaysmooveV22 years ago

Bro cooked again 👨‍🍳🔥

omkaar2 years ago

!!!

Bassel Al Omari2 years ago

Goat-Tier video

Virat Chauhan2 years ago

Soo going to milk my friends' macs for what they are worth with this now 😈

omkaar2 years ago

😈😈😈

eggo2 years ago

I've been super interested in this idea for a while. Great work! I'm interested in helping if possible

omkaar2 years ago

ofc repo is completely free use, you can edit, redistribute, etc!

NIK2 years ago

🔥

LA Bloke2 years ago

Can't you just use Ray Train?

omkaar2 years ago

ray train inspired me to build this lol, this was something I built in 12 hours for fun

io ɳ2 years ago

Hi @omkizzy can you train a model with those three Mac if they are separated at a long distance?

omkaar2 years ago

yeah but the network costs would be astronomical, but my module here does support that

io ɳ2 years ago

Okay,… So what if you increase The number of Mac’s will the “network cost” would decrease or that is not the factor at all?

omkaar2 years ago

in an ideal world, not a factor at all

io ɳ2 years ago

Okay,… I hope it becomes a bigger thing man. And I will go through it on GitHub so I can understand more and if I will have some more questions I will ask you then.

omkaar2 years ago

ofc i welcome all questions! it already is a big thing, really cool companies like @flwrlabs are working on similar ideas!

io ɳ2 years ago

@flwrlabs Yeah yeah , I just saw someone mention @flwlabs on your feed , it is so cool 😎 man

Ankur Shukla2 years ago

Awesome. This idea as crossed me multiple times. Happy to see it being implemented

pooria2 years ago

@omkizzy you’re the hero we didn’t know about but need lol p.s. what are you up to May10-12?

omkaar2 years ago

im going to be in school, down to miss if you have something interesting going on 👀

omkaar2 years ago

cc: @parthsareen

pooria2 years ago

@parthsareen @parthsareen come share extensible AI with the builders too!

Javier Fernandez2 years ago

Great video, once again!!

VA2 years ago

If you ever need more compute try @PrimeIntellect

thecollabagepatch2 years ago

@yikesawjeez 👀

omkaar2 years ago

@yikesawjeez they were kind enough to repost!

thecollabagepatch2 years ago

@yikesawjeez this project speaks for itself cooler release than llama3 tbh lfg

omkaar2 years ago

@yikesawjeez no way thats a huge compliment thank u!

yikes (:D/acc)2 years ago

@thepatch_kev we love distributed computing @BacalhauProject , @BigscienceW , @m_ryabinin , check it out. maybe swap notes/grant offers with our friend here ;)

omkaar2 years ago

@thepatch_kev @BacalhauProject @BigscienceW @m_ryabinin would love to chat w anyone interested!

Amrit Chera2 years ago

Setting the standards for video quality! Great stuff here 👏

omkaar2 years ago

@AmritBuilds recorded on my iphone!

Tony Kam2 years ago

this is dope dude!

omkaar2 years ago

thank you!

TiskTusk2 years ago

The GPU poors shall be vindicated

μ-go2 years ago

@TiskTusk

Nag2 years ago

Yoo, any tutorial on how u built it? Curious lol

omkaar2 years ago

it will take me some time to do that man, there is a lot going on lol

AK10 months ago

federated learning on macbooks is honestly genius. network latency is the real bottleneck but gradient compression + local sgd steps could help. m1 metal api >

Chris 🇭🇷🇮🇪2 years ago

Really cool project. Just made a PR to speed things up and for the learning to continue if one of the learners fails.

omkaar2 years ago

sick will check it out later!

Ashwin 🇮🇳2 years ago

So you expect a guy to have poor GPU but have 3 M1 Macbooks 🥴

Related Videos

Why am I learning Chinese? This video mentions important points that also prompted me to learn Chinese. “Why can I speak a mouthful of glazed Chinese? Many friends are curious and ask me how I did it. I think the answer is very simple. When you develop a deep love for one thing, a country especially its culture, learning is no longer a burden but pleasure. I love China, its long history, rich culture, and blueberry-flavored food. This love has filled me with fun and motivation in the process of learning Chinese. Thank you for your support. I will continue to work hard to share more of my learning experience and love for China.”

Why am I learning Chinese? This video mentions important points that also prompted me to learn Chinese. “Why can I speak a mouthful of glazed Chinese? Many friends are curious and ask me how I did it. I think the answer is very simple. When you develop a deep love for one thing, a country especially its culture, learning is no longer a burden but pleasure. I love China, its long history, rich culture, and blueberry-flavored food. This love has filled me with fun and motivation in the process of learning Chinese. Thank you for your support. I will continue to work hard to share more of my learning experience and love for China.”

Johannes Maria

12,916 views • 2 years ago

This is "Sequential Dexterity", a neural network that controls a robot arm to build legos given a manual 🤖 To do this task, the robot needs to chain together multiple skills (grasping, re-orienting, pushing, etc.) and execute without compounding error. I find some very simple but effective insights in the paper: - Skill chaining over a long sequence is hard. We need a way to "smooth it over". - Forward smoothing: train skill [i] to good performance, run it many times to generate possible "end states", and sample from these states to provide initialization for skill [i+1]. Then start training skill [i+1], and so on. - The problem is that skill [i] is unaware of [i+1], so it may produce bad end states that make life hard for [i+1]. - Backward smoothing: identify what are the good starting states for skill [i+1], then finetune the previous skill [i] towards those states by reinforcement learning. Website: Authors: Yuanpei Chen*, Chen Wang*, Li Fei-Fei, C. Karen Liu Paper: "Sequential Dexterity: Chaining Dexterous Policies for Long-Horizon Manipulation"

This is "Sequential Dexterity", a neural network that controls a robot arm to build legos given a manual 🤖 To do this task, the robot needs to chain together multiple skills (grasping, re-orienting, pushing, etc.) and execute without compounding error. I find some very simple but effective insights in the paper: - Skill chaining over a long sequence is hard. We need a way to "smooth it over". - Forward smoothing: train skill [i] to good performance, run it many times to generate possible "end states", and sample from these states to provide initialization for skill [i+1]. Then start training skill [i+1], and so on. - The problem is that skill [i] is unaware of [i+1], so it may produce bad end states that make life hard for [i+1]. - Backward smoothing: identify what are the good starting states for skill [i+1], then finetune the previous skill [i] towards those states by reinforcement learning. Website: Authors: Yuanpei Chen, Chen Wang, Li Fei-Fei, C. Karen Liu Paper: "Sequential Dexterity: Chaining Dexterous Policies for Long-Horizon Manipulation"

Jim Fan

69,064 views • 3 years ago

The right has criticised Tommy Robinson for saying we wouldn’t have a problem with immigration if it weren't for Islam. I understand Tommy wants to prevent his non-white friends from being mistreated. He's trying to draw a clean distinction between Islam and everyone else, based on ideology. But that is a mistake. Immigration would still inflict crimes, economic costs, and cultural change on us, and cause white Brits to become a minority, even if there were no Islam in Britain. Islam is a problem, but it's not the only problem.

The right has criticised Tommy Robinson for saying we wouldn’t have a problem with immigration if it weren't for Islam. I understand Tommy wants to prevent his non-white friends from being mistreated. He's trying to draw a clean distinction between Islam and everyone else, based on ideology. But that is a mistake. Immigration would still inflict crimes, economic costs, and cultural change on us, and cause white Brits to become a minority, even if there were no Islam in Britain. Islam is a problem, but it's not the only problem.

Connor Tomlinson

71,425 views • 9 months ago

Steph Curry on what he feels he needs to accomplish at this point of his career: “I’m still chasing championships. I think that is the motivation. I feel like it’s realistic even though you’re not probably going to be on a team that is the perennial favorite going into any year. But we have, I think, a legit opportunity to make a run. That’s what keeps me going. It’s why I work so hard year in, year out. The offseasons are a grind to get ready for the marathon of a regular season. I still have fun, I still enjoy it. I still have a love for the game, but championships are really it.” #NBAAllStar

Steph Curry on what he feels he needs to accomplish at this point of his career: “I’m still chasing championships. I think that is the motivation. I feel like it’s realistic even though you’re not probably going to be on a team that is the perennial favorite going into any year. But we have, I think, a legit opportunity to make a run. That’s what keeps me going. It’s why I work so hard year in, year out. The offseasons are a grind to get ready for the marathon of a regular season. I still have fun, I still enjoy it. I still have a love for the game, but championships are really it.” #NBAAllStar

aly ✶

93,406 views • 8 months ago

Let's talk about training and how hard you need to push yourself at home before you go for any trials. I remember when I knew that I will be traveling for a trial, i stopped going out of my neighbourhood to play football and I just locked in on training for 2month straight. I will do a roadside jogging from my house to the pitch at Ashiribo and I will train for like 2hrs or more and that was my routine every week except Sundays to rest and I did that for 2 months. Victor McDonald assigned me a coach to train and get me ready and he was by far the hardest I have ever trained. Days where I will tie a tyre around my waist to run on the sand, I did a lot of running,strength training, speed work on the sand with tyre. People who see me train have no idea why I was k!lling myself to train that hard. I once trained with Raphael Edereho a guy who I was looking up at that time and I was not able to walk for 2 days after training with him. I trained like my life depends on it. And I stayed away from many things that will slow me down like, bad friends, drinking, sm0king, including girls and I was just locked in on my purpose to make it happen and it paid off. After 3 months that I left Nigeria the news broke out that I signed for Sölvesborg GIF and People were surprised to see it in the news paper and even my father got asked questions like how or when did I traveled. I thought I trained well enough but to my surprise I was no where near ready at all. I couldn't believe what I saw during the trainings I had to call one of my coaches back home and tell him how hard the trainings are for me. The new generation of young players don't like to train and they except someone to just come and say I will take you to trials. Many have failed not because they don't have the talent but because they don't train enough to meet the standards and social media has made them think otherwise. Your preparation is the most important thing you can do for sure yourself and just like Kobe Bryant said if you are well prepared for it you will not be afraid of failure.

Let's talk about training and how hard you need to push yourself at home before you go for any trials. I remember when I knew that I will be traveling for a trial, i stopped going out of my neighbourhood to play football and I just locked in on training for 2month straight. I will do a roadside jogging from my house to the pitch at Ashiribo and I will train for like 2hrs or more and that was my routine every week except Sundays to rest and I did that for 2 months. Victor McDonald assigned me a coach to train and get me ready and he was by far the hardest I have ever trained. Days where I will tie a tyre around my waist to run on the sand, I did a lot of running,strength training, speed work on the sand with tyre. People who see me train have no idea why I was k!lling myself to train that hard. I once trained with Raphael Edereho a guy who I was looking up at that time and I was not able to walk for 2 days after training with him. I trained like my life depends on it. And I stayed away from many things that will slow me down like, bad friends, drinking, sm0king, including girls and I was just locked in on my purpose to make it happen and it paid off. After 3 months that I left Nigeria the news broke out that I signed for Sölvesborg GIF and People were surprised to see it in the news paper and even my father got asked questions like how or when did I traveled. I thought I trained well enough but to my surprise I was no where near ready at all. I couldn't believe what I saw during the trainings I had to call one of my coaches back home and tell him how hard the trainings are for me. The new generation of young players don't like to train and they except someone to just come and say I will take you to trials. Many have failed not because they don't have the talent but because they don't train enough to meet the standards and social media has made them think otherwise. Your preparation is the most important thing you can do for sure yourself and just like Kobe Bryant said if you are well prepared for it you will not be afraid of failure.

Abiola Dauda

15,507 views • 4 months ago

X wouldn't like it, but I figured a way to get data for analytics without subscribe to their API I wanted to build this for myself to track get an overview of my activities. But thought other might benefit from it This is a browser extension, but not available on webstore yet If you want to try it out, leave a 👋 comment and I'll send it over

X wouldn't like it, but I figured a way to get data for analytics without subscribe to their API I wanted to build this for myself to track get an overview of my activities. But thought other might benefit from it This is a browser extension, but not available on webstore yet If you want to try it out, leave a 👋 comment and I'll send it over

Hieu 🚀

20,559 views • 2 years ago

So helped me to unlock the voice now it works! It's trained on my Lex Fridman podcast It's 99% perfect but the training data apparently still has some Lex "hummm" sounds in it that sometimes show up I think they call this "crosstalk" and it's hard to remove with AI so I think there's no other choice than to listen to the 1 hour of training data and remove those bits and re-train But then we have a good AI model to turn my book MAKE into an audio book without having to read it for hours/days/weeks

So helped me to unlock the voice now it works! It's trained on my Lex Fridman podcast It's 99% perfect but the training data apparently still has some Lex "hummm" sounds in it that sometimes show up I think they call this "crosstalk" and it's hard to remove with AI so I think there's no other choice than to listen to the 1 hour of training data and remove those bits and re-train But then we have a good AI model to turn my book MAKE into an audio book without having to read it for hours/days/weeks

@levelsio

169,787 views • 1 year ago

Lando: “In a way I wanted to keep number 4 because I'm just happy with that and it goes well with my logo. So that's how straightforward I am. But I will go with number 1. Because I have to. I have to. It could be my only opportunity in my life that I get to do such a thing. Not many people in the world get to do that kind of thing. And it's not for me. It's just for my mechanics. This is for my engineers. It's the fact that they get to run with the number 1 on the car. And I think it'll mean a lot more to them than it does to me as a number. And for me, that's a lot more important.”

naenia ¹ ⁶³

1,648,974 views • 10 months ago

⏰ I normally feed the dogs at 6.30am but couldn’t get out of bed until 9.30 today because of depression. 🙏 I could easily have got someone to cover and do it for me but wanted to force myself up. It was tough. 🐶 Anxiety too bad to deal with people today but the dogs and their happy tails helped immeasurably 🌴 Fresh air and doing something helps but as people with depression and anxiety know even a simple thing can seem monumentally hard. Feeding the dogs is the highlight of my day and I still struggled to get out and do it. Thinking of people who have to go to work, face hard social situations and can’t afford to not be in hard situations with their own anxiety / depression. 💤 I probably won’t get much more then this done today as I feel very rough but the dogs are fed and I feel like I achieved something even if I was moving like in quicksand

⏰ I normally feed the dogs at 6.30am but couldn’t get out of bed until 9.30 today because of depression. 🙏 I could easily have got someone to cover and do it for me but wanted to force myself up. It was tough. 🐶 Anxiety too bad to deal with people today but the dogs and their happy tails helped immeasurably 🌴 Fresh air and doing something helps but as people with depression and anxiety know even a simple thing can seem monumentally hard. Feeding the dogs is the highlight of my day and I still struggled to get out and do it. Thinking of people who have to go to work, face hard social situations and can’t afford to not be in hard situations with their own anxiety / depression. 💤 I probably won’t get much more then this done today as I feel very rough but the dogs are fed and I feel like I achieved something even if I was moving like in quicksand

Niall Harbison

1,388,473 views • 3 years ago

An unoptimized game from a gaming giant like GameFreak is not something I expected. I mean I understand when it's unoptimized for PC as there are various configs to test out but on Consoles? It rarely happens man, at least in my experience. Beast of Reincarnation will be a different beast after optimizations and patches.

An unoptimized game from a gaming giant like GameFreak is not something I expected. I mean I understand when it's unoptimized for PC as there are various configs to test out but on Consoles? It rarely happens man, at least in my experience. Beast of Reincarnation will be a different beast after optimizations and patches.

SpawnYaard 🎮

51,993 views • 2 months ago

Trying to figure out your holiday gift giving this year? I have a great suggestion for you. There are organizations like TisBest Philanthropy and DonorsChoose that will give a donation to your recipient’s favorite charity, on your behalf. I think gifts like these are more in keeping with what the holiday season is really about, so I wanted to pass it along.

Trying to figure out your holiday gift giving this year? I have a great suggestion for you. There are organizations like TisBest Philanthropy and DonorsChoose that will give a donation to your recipient’s favorite charity, on your behalf. I think gifts like these are more in keeping with what the holiday season is really about, so I wanted to pass it along.

Ray Dalio

49,710 views • 10 months ago

First Principles is back — and it's no longer a podcast. My goal is to explain hardware engineering in simple terms. But after ~25 episodes, it was clear that podcasting wasn't dense or visual enough… So I went to a nuclear reactor, pulled out a control rod, and got it all on film. The resulting video recently crossed 1 million views across platforms; it's been out almost two years, and it's still getting >1,000 views every day. Awesome format. Hard to pull off, equally hard to replicate. I was stoked and wanted to make more. But unfortunately for the show / very fortunately for my career & life, I got WAY more busy at Astranis at exactly that time... and creating fell by the wayside. But we are back, baby! Next video drops *tomorrow*, and I just shot two more last week. If you're building a hardware product that can teach a smart college freshman something about engineering — or want to nominate another startup for a feature — please reach out. Can't wait for tomorrow. Video goes live at 5am Pacific.

First Principles is back — and it's no longer a podcast. My goal is to explain hardware engineering in simple terms. But after ~25 episodes, it was clear that podcasting wasn't dense or visual enough… So I went to a nuclear reactor, pulled out a control rod, and got it all on film. The resulting video recently crossed 1 million views across platforms; it's been out almost two years, and it's still getting >1,000 views every day. Awesome format. Hard to pull off, equally hard to replicate. I was stoked and wanted to make more. But unfortunately for the show / very fortunately for my career & life, I got WAY more busy at Astranis at exactly that time... and creating fell by the wayside. But we are back, baby! Next video drops tomorrow, and I just shot two more last week. If you're building a hardware product that can teach a smart college freshman something about engineering — or want to nominate another startup for a feature — please reach out. Can't wait for tomorrow. Video goes live at 5am Pacific.

Christian Keil

26,946 views • 19 days ago

Jayson Tatum: “It might not seem like it, cause I’m back playing, but it was a very, very long time for me not to be doing what I love to do. “So, I can’t stress it enough, the fact that I get to put my uniform on and run out there with the team is a win for me...The amount of joy I’ve been able to find just being back out there and being out there with my teammates is all I think about.”

Jayson Tatum: “It might not seem like it, cause I’m back playing, but it was a very, very long time for me not to be doing what I love to do. “So, I can’t stress it enough, the fact that I get to put my uniform on and run out there with the team is a win for me...The amount of joy I’ve been able to find just being back out there and being out there with my teammates is all I think about.”

Bobby Krivitsky

113,424 views • 5 months ago

Demetrious Johnson reveals what it was like to train with Khamzat Chimaev: “Khamzat is working extremely hard—and I mean hard—almost to the point where I don’t miss it. The man is a machine… He’s so f***ing strong and big, man. Sometimes I forget I’m 5'3", but I have the heart of a f***ing giant. I’m like, ‘I can fight with him.’ Yeah, it didn’t work out that way. I’m gonna join wrestling practice… but I’m not worthy to be on that team. Everyone on that team is undefeated. I’m just here showing some love, bringing some good vibes, good energy—and that’s it.”

Demetrious Johnson reveals what it was like to train with Khamzat Chimaev: “Khamzat is working extremely hard—and I mean hard—almost to the point where I don’t miss it. The man is a machine… He’s so fing strong and big, man. Sometimes I forget I’m 5'3", but I have the heart of a fing giant. I’m like, ‘I can fight with him.’ Yeah, it didn’t work out that way. I’m gonna join wrestling practice… but I’m not worthy to be on that team. Everyone on that team is undefeated. I’m just here showing some love, bringing some good vibes, good energy—and that’s it.”

Ariel Helwani

289,695 views • 6 months ago

Greg Brockman on AGI arriving quietly and unevenly: "I think that AGI has turned out to be less of a point in time and more of this sort of fuzzy spectrum. And for me, Astra has really hit something that I'm like, 'Okay, I think this is pretty reasonable to call it AGI.'" "With its computer use capabilities, you really can ask it to do long-lived tasks, and it'll just do it. We've seen it run coherently for 24 hours to go accomplish tasks that I think are quite amazing and across a wide variety of domains." "It still is jagged... It's the first time it's not slop writing. But it's not great writing. There are a number of areas where I feel like we just need to polish it a little bit, and it would be fantastic. It's just not quite there." "What people are finding is that it is so capable across such a wide variety of tasks, that it is accelerative, it is empowering, and it's something that I think we've never really seen a model that's been a jump like this." Greg Brockman

Greg Brockman on AGI arriving quietly and unevenly: "I think that AGI has turned out to be less of a point in time and more of this sort of fuzzy spectrum. And for me, Astra has really hit something that I'm like, 'Okay, I think this is pretty reasonable to call it AGI.'" "With its computer use capabilities, you really can ask it to do long-lived tasks, and it'll just do it. We've seen it run coherently for 24 hours to go accomplish tasks that I think are quite amazing and across a wide variety of domains." "It still is jagged... It's the first time it's not slop writing. But it's not great writing. There are a number of areas where I feel like we just need to polish it a little bit, and it would be fantastic. It's just not quite there." "What people are finding is that it is so capable across such a wide variety of tasks, that it is accelerative, it is empowering, and it's something that I think we've never really seen a model that's been a jump like this." Greg Brockman

a16z

62,728 views • 1 month ago