Excellent new fine-grained tracking from DeepMind: TAPIR: Tracking Any Point with per-frame Initialization and temporal Refinement arxiv: project: tldr: TapNet for localization then PIPs-style refinement; outperforms everything!

TAPIR not only outperforms PIPs and TAP-Net by a wide margin, but also beats the concurrent (and beautiful) "Tracking Everything Everywhere All at Once" (aka Omnimotion). This shows the power of (1) a well-designed model and (2) large-scale training on synthetic data.

I’m very excited to show this to my wife who is an equine vet. There we every expensive systems that can help find lameness in a horse’s gait, but most of them just tape cards on the body and watch for asymmetry. Super cool!

@samjstudios

This would be amazing to analyse opponents in sports 👌

🤔 That will help a lot in sports! To detect anomaly trajectories, muscles... Imagine in a martial arts combat plus eye tracking!

May I ask how this supersedes or complement classical methods of optical flow analysis in computer vision, which have been around for decades and use a bajillion times less parameters? Like, with OpenCV? Honest question, because if there is a value added, I'd sure like to know.

The hope here is to track through occlusions, which flow cannot do. Notice the rhino example, where the trajectories follow the rhino behind the tree.

I should say it's a good step to teach AI to sense the motion, and based on that, AI predict where that object is moving. The above feature will allow driverless cars to have better awareness.

@bayraitt Look great! Everything else seemed to be already spot-on except the rotating wheels of the car seemed to cause problems. I guess that's expected because the wheels look nearly identical again after rotating just 1/5 or 1/6 or 1/7 of a 360° rotation.

@AlexHHChan