Loading video...

Video Failed to Load

Go Home

Excited to share our latest work: “Bio-Inspired Plastic Neural Networks for Zero-Shot Out-of-Distribution Generalization in Complex Animal-Inspired Robots” 🪲🦎 We show that Hebbian learning outperforms LSTM-based adaptation for real-world transfer. It even works without domain randomization! It can handle: ✅ Uneven terrain ✅ Morphological damage ✅ Sim-to-real gaps

14,390 views • 1 year ago •via X (Twitter)

13 Comments

Sebastian Risi's profile picture
Sebastian Risi1 year ago

Current ML methods are powerful but brittle. Most fail when conditions change — sim-to-real gaps, uneven terrain, or even minor morphological damage can cause catastrophic failures. Domain randomization is a way to increase robustness but can be expensive to train.

Sebastian Risi's profile picture
Sebastian Risi1 year ago

Inspired by biological brains, we use Hebbian plasticity where synaptic weights adapt in real-time based on neural activity. This allows dynamic adaptation, even in unseen environments and seems to not crucially depend on domain randomization. We tested this approach on two challenging, high-DOF robots: 🪲 An 18-DOF dung beetle-inspired robot 🦎 A 16-DOF gecko-like robot Both showed zero-shot sim-to-real transfer, adapting instantly without retraining.

Sebastian Risi's profile picture
Sebastian Risi1 year ago

What type of solution did the Hebbian network find? PCA of weight dynamics revealed limit cycle attractors, suggesting rhythmic motor control emerges from plasticity + sensory feedback. A limit cycle attractor can be observed in the weights of the ANN with Hebbian updates (left). In contrast, non-optimized Hebbian rules cannot achieve a limit cycle attractor and the weights were continually updated toward a single direction in the weight space (right).

Sebastian Risi's profile picture
Sebastian Risi1 year ago

We believe this work highlights the power of bio-inspired learning mechanisms in robotics. Our Hebbian networks are simple, lightweight, and robust and they generalize without requiring architectural complexity or training tricks. Paper: Project page with videos: A great collaboration with @BinggwongLeung , @worasuchad, @JoachimWinther. and @PManoonpong.

kyuxu's profile picture
kyuxu1 year ago

really cool. has hebbian learning been explored in combination with topology evolving networks, e.g. NEAT, or is the parameter search space on a fixed architecture already difficult to optimize?

Sebastian Risi's profile picture
Sebastian Risi1 year ago

It has actually! Here is a review article of that field: Although the article is now a few years old so maybe time for an update.

Abe's profile picture
Abe1 year ago

Zero shot is terrifying

Anesu Tembo's profile picture
Anesu Tembo1 year ago

Great work!!!

kira's profile picture
kira1 year ago

@drmichaellevin Grow up from your toys If you ready 🙊

Qiqi Duan (段琦琦)'s profile picture
Qiqi Duan (段琦琦)1 year ago

Interesting!

matty's profile picture
matty1 year ago

Is there a reason why these bio inspired approaches don’t get applied to language? It’s always robots automata etc ..

تطوير الالعاب - Ludology's profile picture
تطوير الالعاب - Ludology1 year ago

Compared to SNN ?

VistaShares ETFs's profile picture
VistaShares ETFs1 year ago

From semiconductors to data centers, AIS targets the critical components behind AI's exponential growth. Capture potential returns from this transformative technology sector.

Related Videos

I’m thrilled to announce that we just released GraspGen, a multi-year project we have been cooking at NVIDIA Robotics 🚀 GraspGen: A Diffusion-Based Framework for 6-DOF Grasping Grasping is a foundational challenge in robotics 🤖 — whether for industrial picking or general-purpose humanoids. VLA + real data collection is all the rage now but is expensive and scales poorly for this task. For every new gripper and/or scene, you’ll have to recollect the dataset in this paradigm for the best perf. 💡Key Idea: Since grasping is such a well-defined task in simulation - why can’t we just scale synthetic data generation and train a generative model for grasping? By embracing modularity and standardized grasp formats, we can make this a turnkey technology that works zero-shot for multiple settings. GraspGen is a modular framework for diffusion-based 6-DOF grasp generation that scales across embodiment types, observability conditions, clutter, task complexity. Key Features: ✅ Multi-embodiment support: suction, parallel-jaw, and multi-fingered grippers ✅ Generalization to partial + complete 3D point clouds ✅ Generalization to single-objects + cluttered scenes ✅ Modular design uses other robotics modules and foundation models (SAM2, cuRobo, FoundationStereo, FoundationPose). This allows GraspGen to focus on only one thing - grasp generation ✅ Training recipe: grasp discriminator is trained with On-Generator data from the diffusion model - so that it learns to correct the mistakes (if any) of the diffusion generator ✅ Real-time performance (~20 Hz) before any GPU acceleration; low memory footprint 📊 Results: • SOTA on the FetchBench [Han et al. CoRL 2024] benchmark • Zero-shot sim-to-real transfer on unknown objects and cluttered scenes • Dataset of 53M simulated grasps across 8K objects from Objaverse 📄 arXiv: 🌐 Website: 💻 Code: A huge thank you to everyone involved in this journey — excited to see what the community builds on top of it! Joint work with Clemens Eppner , Balakumar Sundaralingam , Yu-Wei, Jun Yamada Wentao Yuan and other collaborators #robotics #diffusionmodels #physicalAI #simtoreal

Adithya Murali

24,106 views • 1 year ago