We then trained an artificial neural network on real fly behavior from recorded videos 🎥 and let it control the model in MuJoCo. This enables it to learn how to move the virtual insect in the most realistic way.

To create it, we turned to MuJoCo, our open-source physics simulator – created for robotics and biomechanics – and added features such as: ▪️simulating fluid forces on the flapping wings, enabling flight ▪️adhesion actuators – mimicking the gripping force of insect feet

We built an AI model to simulate how a fruit fly walks, flies and behaves – in partnership with @HHMIJanelia. 🪰 Our computerized insect replicates realistic motion, and can even use its eyes to control its actions. Here’s how we developed it – and what it means for science. 🧵

Here, the fly shows accurate movement along complex natural flight trajectories. Watch how it’s instructed to follow a real path marked by the blue dots. 🔵🪰🔵

So why build these kinds of models? We believe they could help scientists better understand how the brain, body and environment drive specific behaviors in an animal – finding connections that labs can't always measure. 🥼

We’ve already applied this approach to multiple organisms – a virtual rodent, and now a fruit fly. So what comes next for neuroscientists? The zebrafish – a widely studied creature which shares 70% of its protein-coding genes with humans. 🐠 Find out more in @Nature ↓

To benefit the wider research community, we’ve also open-sourced this model with @HHMIJanelia. Here’s the code ↓

the flies aren't real

If I saw a fly moving like that I would immediately think something is wrong with it. Then I would probably swat it.

Waiting for world simulation.

would it be possible for next step to simulate the full brain activity of the fruit fly ? i think they have some models that have full 3d mapping of each neurons synapses ?

You should consider doing this for zebrafish larvae and not adults. especially at 5 days post fertilisation, and at high fps to focus on fast escape responses. Many behavioral assays based lon this are under development in the EU for use as toxicology assays!

I'm currently exploring mujoco, this is so interesting. I'm highly motivated! 😎🔥

Curious to see how it would move the fly without knowing what is "realistic" and just based on what is possible.

Why? Why does it need to do this; for a fruit fly - survive long enough; to reproduce and do so as efficiently as possible in terms of energy and time. For the artificial neural net and it's simulated body, why would it need to do what it does?

Can Machine Learning beat the market? Check out this post on my free Substack where I share code and commentary for an XGBoost model and a Random Forest model that both deliver powerful performances.