Loading video...
Video Failed to Load
New AI research from Meta – CoTracker3 Simpler and Better Point Tracking by Pseudo-Labelling Real Videos. More details ➡️ Demo on Hugging Face ➡️ Building on our previous work on CoTracker, this new model demonstrates impressive tracking results where points can be tracked for a long time even when... show more
218,857 views • 1 year ago •via X (Twitter)
10 Comments
@huggingface
The paper introduces a new point tracking model called CoTracker3 that builds upon recent point tracking models like PIPs, TAPIR, and CoTracker. Point tracking is an important task in video analysis for applications like 3D reconstruction and video editing. The CoTracker3 model, when trained only on synthetic data, already outperforms state-of-the-art trackers on several benchmarks. When further fine-tuned on just 15,000 real-world unlabeled videos using the proposed protocol, it significantly surpasses the performance of BootsTAPIR, which was trained on 15 million real videos. CoTracker3 also shows better handling of occluded points compared to other models. full paper:
@huggingface love how meta keeps open-sourcing these research
@huggingface Your things @Mbounge_
@huggingface That’s awesome!
@huggingface Exciting development! Meta's new CoTracker3 could revolutionize point tracking in real videos.
@huggingface This is wild.
@huggingface This new AI model from Meta sounds impressive! CoTracker3 could really push forward developments in point tracking technology
@huggingface When are you going to release co-tracker under an Open Source/OSI license? It is the 3rd release under Creative Commons.
@huggingface wow!
![There is a beautiful story that just happened in AI so let me share it for a lighter tone weekend post among all the doom stories in our AI field this week. It’s a story of people on three continents building and sharing in the open a new small efficient and state-of-the-art AI model. It started a couple of months ago when a new team in the AI scene released their first model from their headquarters in Paris (France): Mistral 7B. Impressive model, small and very strong performances in the benchmarks, better than all previous models of this size. And open source! So you could build on top of it. Lewis in Bern (Switzerland) and Ed (in Lyon, in the South of France) both from the H4 team, a team of researchers in model fine-tuning and alignment were talking about it over a coffee, in one of these gatherings that often happen at Hugging Face to break the distance between people (literal distance as HF is a remote company). What about fine-tuning it using this new DPO method that a research team from Stanford in California just posted on Arxiv, says one? Hey, that’s a great idea, replies the other. We've just build a great code base (with Nathan, Nazneen, Costa, Younes and all the H4 team and TRL community) let's use it! The next day they start diving in the datasets openly shared on the HF hub and stumble upon two interesting large and good quality fine-tuning datasets recently open-sourced by OpenBMB, a Chinese team from Tsinghua: UltraFeedback and UltraChat. A few rounds of training experiments confirm the intuition, the resulting model is super strong, by far the strongest they have ever seen in their benchmarks from Berkeley and Stanford (LMSYS and Alpaca). Join Clementine, the big boss of the open evaluation leaderboard. Her deep dive into the model capabilities confirms the results: impressive performance. But the H4 team also hosts a famous faculty member, Pr. Sasha Rush, Associate Professor at Cornell University in his daytime, hacker at HF in his nighttime. Joining the conversation, he proposes to quickly draft a research paper to organize and share all the details with the community. A few days later, the model, called Zephyr (a wind like Mistral), paper, and all details are shared with the world. Quickly other companies, everywhere in the world starts to use it. LlamaIndex, a famous data framework and community, shares how the model blew their expectations on real-life use-case benchmarks, while researchers and practitioners discuss the paper and work on the Hugging Face hub. All this happened in just a few weeks catalyzed by open access to knowledge, models, research, and datasets released all over the world (Europe, California, China) and by the idea that people can build upon one another work in AI to bring real-world value with efficient and open models. Stories like this are numerous everywhere around us and make me really proud of the AI community and see how we can build amazingly useful things together. [the video is just me reading this Friday post hahah]](https://image.24vids.com/tw-1720503998518640703/ext_tw_video_thumb/1720503027273023490/pu/img/VOLi_zPnZlb1nuWB.jpg)
