Loading video...
Video Failed to Load
1/ 🤖 New paper hot off the press 🐜 Automated behavioural analysis using machine learning is great, but still needs lots of effort to curate training data. What if a computer could generate that data for you? Say hello to replicAnt #OpenSource
54,807 views • 2 years ago •via X (Twitter)
10 Comments
🎮1b/ replicAnt : Unreal Engine 5 meets ecology 🐞 replicAnt is a pipeline that takes 3D animal models and places them in complex procedurally generated scenes in @UnrealEngine. Generate large volumes of annotated training data with the click of a mouse!
2/ 🌍 Open-Source & User-Friendly replicAnt is available to all, designed to run on your laptop, aiming to reduce the effort required to make your neural networks robust. All you need to get started is included here ( and we guide you through each step.
3/ 📦 What You Need: - 3D model of your study animal - replicAnt Unreal Project - Data parser for your network That's it! We provide examples for these essentials so you can hit the ground running. All 3D animal models in our paper were created with scAnt
4/ 🎨 We look at bugs, but you don’t have to! Whether you use a 3D model you've created or one you've downloaded (we provide examples!), we guide you through the model preparation process to bring your digital twin to life.
5/ 🦵 Rigging for success Assign virtual bones and joints to your model for realistic poses. This critical step ensures that your synthetic data is as realistic as possible. #DigitalTwins
6/ 🚀 Setup & Generate Load your 3D model into replicAnt, configure the data generation process, and watch as your datasets are produced, annotation included! #DataGen #MachineLearning
7/ 🔄 From replicAnt to AI Our data parsers convert the generated data into formats tailored for popular deep learning tools like SLEAP (@talmop), @DeepLabCut, @DeepPoseKit, YOLO, and Mask R-CNN. The networks below have never seen a real ant or termite!
8/🎓Training AI The output from replicAnt feeds into the training process, ensuring that your network learns from highly diverse datasets which can be used on their own or refined with hand-annotated examples. Here, we first trained on synthetic data and refined with real images:
9/🧪 Empower Your Research We have shown that for some applications, synthetic data from replicAnt is enough for robust performance across divers recording conditions (in the lab & in the field) and for others it drastically reduces the amount of hand-annotated data needed.
10/ 🌟 Conclusion: replicAnt @NatureComms has made our work easier and allowed us to train models that are able to generalise to new recording conditions with much less effort. For now, it’s all bugs (the good kind) but quadrupeds and co. are coming soon. What will you generate?
!["One of the very confusing things about the models right now: how to reconcile the fact that they are doing so well on evals. And you look at the evals and you go, 'Those are pretty hard evals.' But the economic impact seems to be dramatically behind. There is [a possible] explanation. Back when people were doing pre-training, the question of what data to train on was answered, because that answer was everything. So you don't have to think if it's going to be this data or that data. When people do RL training, they say, 'Okay, we want to have this kind of RL training for this thing and that kind of RL training for that thing.' You say, 'Hey, I would love our model to do really well when we release it. I want the evals to look great. What would be RL training that could help on this task?' If you combine this with generalization of the models actually being inadequate, that has the potential to explain a lot of what we are seeing, this disconnect between eval performance and actual real-world performance"](https://image.24vids.com/tw-1993450075616690474/amplify_video_thumb/1993449024045932544/img/W-1TjOXcXeiUbHaH.jpg)
