Harrison Kinsley
@Sentdex • 110,166 subscribers
gpus and tractors. Neural networks from Scratch book: https://t.co/hyMkWyUP7R https://t.co/8WGZRkUGsn
Shorts
Videos
One question that's been on my mind for years now is: could we use regular multimodal LLMs not necessarily trained for robotics to do the high level robotics intelligence part that VLAs and WAMs attempt to do? The latest explosion of powerful opensource multi-modal LLMs has, IMO, begun to make this possible due both to intelligence and speed. This is GLM 5.3 Flash, which has vision understanding, but isn't meant to be a VLA/VLM/WAM/robotics model at all, controlling an XGO mini wheeled robot quadruped with an arm & gripper. GLM 5.3F simply has access to the robot's high level SDK for controlling movement, arm joints, open/close gripper...etc. It analyzes the frames from the camera and makes adjustments all on its own to solve the task. Nothing was trained here, nothing fine-tuned for this task. Z AI did not make this model for robots and tbh I think they're surprised this works when I talk to them about it! This also works quite well with DSV4F + a vision capable model like Qwen 3.8 27B. I havent tried JUST Qwen 3.8 27B, but I'm sure it works too. I like the "logic" to be a model that's as fast as possible (but still intelligent). There's also an experimental vision version of DSV4F, I'm confident that'll work too and might even be better bc the full loop might be the fastest of all with this model. An obvious question you might wonder is: well why not use VLA or VLM? The hard part about robotics isn't object detection, that's long solved. This also isn't a solution for gait/locomotion...yet, but I actually don't think this is far away either and I've done some experimentation with LLMs in this space in the past and it does show promise. It might actually already be here for quadrupeds, since you dont need super fast IMU readings to maintain balance. I've also tried many of the larger, more generalist, VLAs that you should be able to use with popular robots and tbh there are just so many edge cases that make things hard and not work. You gotta get the camera, lighting, task, everything *just right* or the demo fails. This is for the actual hard part in robotics right now: intelligence, logic, and planning for all the ways the real world just simply isn't perfect. I've trained VLAs. They're super finicky and you're always running into sim2real issues, especially around the camera. You also have to build the whole training pipeline in a simulator, and, if everything does work, you still just have a robot that does this 1 single thing after weeks of work. If you use teleop, this overcomes the "2real" problem, but now you need to painstakingly collect teleop data, and it's only good at that specific task and that particular robot. There is a growing set of egocentric training data for "general purpose" VLAs and world action models (for humanoid form factors), but I'm really starting to wonder: Why? I think we might just sidestep this whole area of research entirely. I didn't need any training data or special environment to work with this quadruped and arm to do the task I was after. This particular quadruped and arm doesn't even exist in the wild yet really, it's a demo build from a company launching it on kickstarter, so it's not like this robot's data exists in the LLM to any real extent. I think this is cool as heck that this works and I am interested to see just how far I can push it. Also this marks the first time that I've finally got a generalist solution to a task I've been trying to solve ever since I became a dad of twins: pick up toys off the ground. This is a big day!
Harrison Kinsley47,164 views • 2 days ago
People often ask how did the Unitree robots get so good all of a sudden. It wasn't all of a sudden, and it's because they ship their hardware and open source their SDKs. Arguably these robots are nearly useless out of the box, but you have full dev control of them. Because of that, the hardware has become a very popular R&D platform with an ecosystem around it and the Unitree G1 is undoubtedly an order of magnitude better than it could ever be at this point if Unitree was instead just doing quiet internal dev of both the hardware and software. Too many hardware companies for really cool products that seek to be community-driven (robots, AR glasses...etc) desire to make a profitable walled garden and their greed just ends up walling out developers and their product gets outpaced by the G1s of the world.
Harrison Kinsley816,468 views • 11 months ago
May be a bit too long for X, but: There's been a situation in AI.
Harrison Kinsley83,902 views • 2 months ago
GLM 5.3 is surprisingly very good @ 3D representation & modeling, despite being blind Plugged it in to our Lucky Robots mujoco asset gen system, asked for a Kubota tractor with an articulating front loader and bucket w/ a rotary cutter 2nd image shows all the composite parts
Harrison Kinsley14,242 views • 23 days ago
just cooked up a new sprinter policy, do we attempt sim2real?
Harrison Kinsley58,159 views • 10 months ago
picking something up off the floor w/ a humanoid is more challenging than a backflip
Harrison Kinsley27,182 views • 6 months ago
This is a vertically integrated end to end deep neural network performing forward pass inference real-time, controlling individual actuator's torque output for bidpedal gait generation in adverse, GPS denied, envs. ok its standard PPO rl trained in mjlab, strapped to a tractor.
Harrison Kinsley34,278 views • 8 months ago
In a world of PPO everything for reinforcement learning, I've been tinkering with SAC for training a quadruped gait. This gait is trained purely on CPU (training on one of the Dell GB10s) on a single environment. Training any particular run is obviously slower than PPO on an RTX Pro 6000 with 8092 envs, if you already know the exact hyperparams/rwd function for your PPO algo... but, if we're honest with ourselves, then we know we usually spend days tuning our PPO algo and fighting it to do what we want. In contrast, SAC has kind of been a breath of fresh air, very amenable to changing the reward function to tune behavior. So far, my first attempts to tune things have consistently just worked immediately rather than 15 different variations of reward hacking only to find previous tuned behaviors got lost in the process. There is also FastSAC, which I've not yet tried, but can speed things up potentially and introduce scale back into the equation. My main painpoint in getting SAC to work for gait was actually getting it to learn to step. It seems as though SAC is not as good as PPO at significant exploration on its own. I ended up starting with a sinusoidal gait (basically just a rule to make legs swing) as training wheels then blended it out through training as phase 1, then began working on smoothing things out after this. I think if we look at end to end dev time rather than any particular run that finally managed to work, SAC may actually be the "faster" algorithm to train. Quadruped gaits are inherently easier than bipedal and maybe there are areas where SAC falls short, but I'll definitely be spending more time with SAC.
Harrison Kinsley26,758 views • 6 months ago
Ladies and gentlemen, we have our first successful sim2real transfer on Geoff the G1!
Harrison Kinsley29,072 views • 9 months ago
after many failed attempts to make a single model that walks and crouches with all the attributes I wanted, I ended up just splitting into 2 models with basic control logic to swap between them. Could just keep adding "skills" as models and tweak those specific models as needed
Harrison Kinsley19,271 views • 7 months ago
alright, finally we're almost ready to ship our new policy to the world
Harrison Kinsley20,759 views • 10 months ago
