Loading video...

Video Failed to Load

There was a problem loading this video. This could be due to a temporary network issue or the video might be unavailable.

🚀 Introducing MegaFlow: Zero-Shot Large Displacement Optical Flow! 🔥 Code, models, and demo — all available now! 🎯 Large displacement motion estimation (optical flow, point tracking) has been a long-standing challenge. MegaFlow proposes a simple solution by leveraging pre-trained vision priors. Combined with a global matching formulation, we tackle... show more

Haofei Xu

1,152 subscribers

32,143 views • 2 months ago •via X (Twitter)

Anya Rossi• Live Now

Private livecam show

0 Comments

No comments available

Comments from the original post will appear here

Related Videos

Check out our #PAMI paper with code "Dense Continuous-Time Optical Flow from Event Cameras," where we show how to regress *continuous-time* trajectories of every pixel from event cameras alone or events plus frames! The key idea is to iteratively estimate per-pixel polynomials using a recurrent lookup and update scheme. Paper: Code: DOI: We present a method for estimating dense continuous-time optical flow from event data. Traditional dense optical flow methods compute the pixel displacement between two images. Due to missing information, these approaches cannot recover the pixel trajectories in the blind time between two images. We show that it is possible to compute per-pixel, continuous-time optical flow using events from an event camera. Events provide temporally fine-grained information about movement in pixel space due to their asynchronous nature and microsecond response time. We leverage these benefits to predict pixel trajectories densely in continuous time via parameterized Bézier curves. To achieve this, we build a neural network with strong inductive biases for this task: First, we build multiple sequential correlation volumes in time using event data. Second, we use Bézier curves to index these correlation volumes at multiple timestamps along the trajectory. Third, we use the retrieved correlation to update the Bézier curve representations iteratively. Our method can optionally include image pairs to boost performance further. To train and evaluate our model, we introduce a synthetic dataset (MultiFlow) that features moving objects and ground truth trajectories for every pixel. Our quantitative experiments suggest that our method successfully predicts pixel trajectories in continuous time and is competitive in the traditional two-view pixel displacement metric on MultiFlow and DSEC-Flow. Open source code and datasets are released to the public. Kudos to Mathias Gehrig Manasi Muglikar

Check out our #PAMI paper with code "Dense Continuous-Time Optical Flow from Event Cameras," where we show how to regress continuous-time trajectories of every pixel from event cameras alone or events plus frames! The key idea is to iteratively estimate per-pixel polynomials using a recurrent lookup and update scheme. Paper: Code: DOI: We present a method for estimating dense continuous-time optical flow from event data. Traditional dense optical flow methods compute the pixel displacement between two images. Due to missing information, these approaches cannot recover the pixel trajectories in the blind time between two images. We show that it is possible to compute per-pixel, continuous-time optical flow using events from an event camera. Events provide temporally fine-grained information about movement in pixel space due to their asynchronous nature and microsecond response time. We leverage these benefits to predict pixel trajectories densely in continuous time via parameterized Bézier curves. To achieve this, we build a neural network with strong inductive biases for this task: First, we build multiple sequential correlation volumes in time using event data. Second, we use Bézier curves to index these correlation volumes at multiple timestamps along the trajectory. Third, we use the retrieved correlation to update the Bézier curve representations iteratively. Our method can optionally include image pairs to boost performance further. To train and evaluate our model, we introduce a synthetic dataset (MultiFlow) that features moving objects and ground truth trajectories for every pixel. Our quantitative experiments suggest that our method successfully predicts pixel trajectories in continuous time and is competitive in the traditional two-view pixel displacement metric on MultiFlow and DSEC-Flow. Open source code and datasets are released to the public. Kudos to Mathias Gehrig Manasi Muglikar

Davide Scaramuzza

12,637 views • 2 years ago

$Optical flow, fluid dynamics, refraction with lots and lots of render targets in WebGL #threejs #webgl$

Optical flow, fluid dynamics, refraction with lots and lots of render targets in WebGL #threejs #webgl

Aj

657,528 views • 1 year ago

Real-time visual effects, running 100% locally in your browser! 🚀 I created a demo where you can just drop your video and watch it analyze feature points and optical flow for some cool visual effects! Have fun 😊 Demo: #ComputerVision #VFX

Real-time visual effects, running 100% locally in your browser! 🚀 I created a demo where you can just drop your video and watch it analyze feature points and optical flow for some cool visual effects! Have fun 😊 Demo: #ComputerVision #VFX

Alexandre Devaux

33,368 views • 10 months ago

PointOdyssey: A Large-Scale Synthetic Dataset for Long-Term Point Tracking paper page: introduce PointOdyssey, a large-scale synthetic dataset, and data generation framework, for the training and evaluation of long-term fine-grained tracking algorithms. Our goal is to advance the state-of-the-art by placing emphasis on long videos with naturalistic motion. Toward the goal of naturalism, we animate deformable characters using real-world motion capture data, we build 3D scenes to match the motion capture environments, and we render camera viewpoints using trajectories mined via structure-from-motion on real videos. We create combinatorial diversity by randomizing character appearance, motion profiles, materials, lighting, 3D assets, and atmospheric effects. Our dataset currently includes 104 videos, averaging 2,000 frames long, with orders of magnitude more correspondence annotations than prior work. We show that existing methods can be trained from scratch in our dataset and outperform the published variants. Finally, we introduce modifications to the PIPs point tracking method, greatly widening its temporal receptive field, which improves its performance on PointOdyssey as well as on two real-world benchmarks.

PointOdyssey: A Large-Scale Synthetic Dataset for Long-Term Point Tracking paper page: introduce PointOdyssey, a large-scale synthetic dataset, and data generation framework, for the training and evaluation of long-term fine-grained tracking algorithms. Our goal is to advance the state-of-the-art by placing emphasis on long videos with naturalistic motion. Toward the goal of naturalism, we animate deformable characters using real-world motion capture data, we build 3D scenes to match the motion capture environments, and we render camera viewpoints using trajectories mined via structure-from-motion on real videos. We create combinatorial diversity by randomizing character appearance, motion profiles, materials, lighting, 3D assets, and atmospheric effects. Our dataset currently includes 104 videos, averaging 2,000 frames long, with orders of magnitude more correspondence annotations than prior work. We show that existing methods can be trained from scratch in our dataset and outperform the published variants. Finally, we introduce modifications to the PIPs point tracking method, greatly widening its temporal receptive field, which improves its performance on PointOdyssey as well as on two real-world benchmarks.

AK

122,525 views • 2 years ago

Can we synthesize 3D human-scene interactions without learning from any 3D data? Yes! Check out Lei Li's GenZI, a novel zero-shot approach to generating 3D interactions by distilling priors from large vision-language models.

Can we synthesize 3D human-scene interactions without learning from any 3D data? Yes! Check out Lei Li's GenZI, a novel zero-shot approach to generating 3D interactions by distilling priors from large vision-language models.

Angela Dai

106,850 views • 2 years ago

New AI research from Meta – CoTracker3 Simpler and Better Point Tracking by Pseudo-Labelling Real Videos. More details ➡️ Demo on Hugging Face ➡️ Building on our previous work on CoTracker, this new model demonstrates impressive tracking results where points can be tracked for a long time even when they're occluded or leave the field of view. CoTracker3 achieves state-of-the-art, outperforming all recent point tracking approaches on standard benchmarks — often by a substantial margin. We've released the research paper, code and a demo on Hugging Face — along with models available under an A-NC license to support further research in this space.

New AI research from Meta – CoTracker3 Simpler and Better Point Tracking by Pseudo-Labelling Real Videos. More details ➡️ Demo on Hugging Face ➡️ Building on our previous work on CoTracker, this new model demonstrates impressive tracking results where points can be tracked for a long time even when they're occluded or leave the field of view. CoTracker3 achieves state-of-the-art, outperforming all recent point tracking approaches on standard benchmarks — often by a substantial margin. We've released the research paper, code and a demo on Hugging Face — along with models available under an A-NC license to support further research in this space.

AI at Meta

218,898 views • 1 year ago

It's been a pleasure to work on this with Gerard Pons-Moll I think we found a really effective formulation for training large-scale strong pose and shape models. Here are some more qualitative results on tough, in-the-wild YouTube dance videos.

It's been a pleasure to work on this with Gerard Pons-Moll I think we found a really effective formulation for training large-scale strong pose and shape models. Here are some more qualitative results on tough, in-the-wild YouTube dance videos.

István Sárándi

40,536 views • 1 year ago

🔥Diffusion Masters Transparent Objects in Videos! We introduce #DKT, a foundation model that repurposes video diffusion for zero-shot depth and normal estimation on Transparent and Reflective Objects with Superior Temporal Consistency Demo: Code:

🔥Diffusion Masters Transparent Objects in Videos! We introduce #DKT, a foundation model that repurposes video diffusion for zero-shot depth and normal estimation on Transparent and Reflective Objects with Superior Temporal Consistency Demo: Code:

Chongjie(CJ) Ye

27,942 views • 5 months ago

Filmmaker Öner S. Biberkökü shares his "screenshot strategy" for building a full narrative from a single starting point during this In the Flow interview. Thanks Oner for the partnership! "At the end of the shot, you can take a screenshot, give it again [to Flow], and you now have a new image of that character in a different situation... So I think, with one image, you can stay in the flow, and make a story." Watch his breakdown:

Filmmaker Öner S. Biberkökü shares his "screenshot strategy" for building a full narrative from a single starting point during this In the Flow interview. Thanks Oner for the partnership! "At the end of the shot, you can take a screenshot, give it again [to Flow], and you now have a new image of that character in a different situation... So I think, with one image, you can stay in the flow, and make a story." Watch his breakdown:

Google Flow

34,772 views • 3 months ago

OPTICAL AIDS CAUSE 28 DAY MONTHS? NCS does not recommend that we use optical aids for a moonsighting system in Britain for many reasons. The primary reason is that we are seeking a unified solution, and unity is in the sunnah - local, naked eye sightings. Not everybody believes that optical aids are permissible, but naked eye sightings are wholly uncontroversial. Another issue is that optical aids would lead to more and more examples of the moon being “seen” a day early, and out of phase, on the 28th of a lunar month. This has ALREADY HAPPENED to those who use optical aids in the UK - yet has never happened to naked eye moonsighters for over 100 months. Whilst there are procedures from the sunnah of what to do if you were to see a moon on the 28th, we cannot really justify asking people to end their fast a day early on the 28th because of a moon which is invisible to the eye and only visible through optical aids. Hence our stance. Remember - we ourselves use optical aids - and had them on our livestream. We are experienced in them, and it is from our experience we share with you that we have found that it would leave to problems. In any case why are you in a rush to end Ramadan, using technology, and away from the sunnah?

OPTICAL AIDS CAUSE 28 DAY MONTHS? NCS does not recommend that we use optical aids for a moonsighting system in Britain for many reasons. The primary reason is that we are seeking a unified solution, and unity is in the sunnah - local, naked eye sightings. Not everybody believes that optical aids are permissible, but naked eye sightings are wholly uncontroversial. Another issue is that optical aids would lead to more and more examples of the moon being “seen” a day early, and out of phase, on the 28th of a lunar month. This has ALREADY HAPPENED to those who use optical aids in the UK - yet has never happened to naked eye moonsighters for over 100 months. Whilst there are procedures from the sunnah of what to do if you were to see a moon on the 28th, we cannot really justify asking people to end their fast a day early on the 28th because of a moon which is invisible to the eye and only visible through optical aids. Hence our stance. Remember - we ourselves use optical aids - and had them on our livestream. We are experienced in them, and it is from our experience we share with you that we have found that it would leave to problems. In any case why are you in a rush to end Ramadan, using technology, and away from the sunnah?

New Crescent Society

18,343 views • 2 months ago

Huawei has received nine awards at the Lightwave Innovation Reviews, held during the Optical Fiber Communication Conference and Exposition. The awards span multiple categories and are a testament to Huawei's leadership in optical network technologies.

Huawei has received nine awards at the Lightwave Innovation Reviews, held during the Optical Fiber Communication Conference and Exposition. The awards span multiple categories and are a testament to Huawei's leadership in optical network technologies.

Huawei

14,161 views • 2 months ago

TikTok presents Depth Anything Unleashing the Power of Large-Scale Unlabeled Data paper page: demo: Depth Anything is trained on 1.5M labeled images and 62M+ unlabeled images jointly, providing the most capable Monocular Depth Estimation (MDE) foundation models with the following features: zero-shot relative depth estimation, better than MiDaS v3.1 (BEiTL-512) zero-shot metric depth estimation, better than ZoeDepth optimal in-domain fine-tuning and evaluation on NYUv2 and KITTI

TikTok presents Depth Anything Unleashing the Power of Large-Scale Unlabeled Data paper page: demo: Depth Anything is trained on 1.5M labeled images and 62M+ unlabeled images jointly, providing the most capable Monocular Depth Estimation (MDE) foundation models with the following features: zero-shot relative depth estimation, better than MiDaS v3.1 (BEiTL-512) zero-shot metric depth estimation, better than ZoeDepth optimal in-domain fine-tuning and evaluation on NYUv2 and KITTI

AK

600,018 views • 2 years ago

Introducing 0x Cross-Chain API ⛓️ One integration to unify routing, bridging, execution, and tracking in a single flow. Ship cross-chain swaps faster. Operate with less overhead. Now in private beta 👇

Introducing 0x Cross-Chain API ⛓️ One integration to unify routing, bridging, execution, and tracking in a single flow. Ship cross-chain swaps faster. Operate with less overhead. Now in private beta 👇

0x

24,168 views • 3 months ago

The NFL is close to implementing an optical tracking solution to measure first downs 👏

The NFL is close to implementing an optical tracking solution to measure first downs 👏

Joe Pompliano

123,551 views • 1 year ago

MotionGPT: Human Motion as a Foreign Language paper page: Though the advancement of pre-trained large language models unfolds, the exploration of building a unified model for language and other multi-modal data, such as motion, remains challenging and untouched so far. Fortunately, human motion displays a semantic coupling akin to human language, often perceived as a form of body language. By fusing language data with large-scale motion models, motion-language pre-training that can enhance the performance of motion-related tasks becomes feasible. Driven by this insight, we propose MotionGPT, a unified, versatile, and user-friendly motion-language model to handle multiple motion-relevant tasks. Specifically, we employ the discrete vector quantization for human motion and transfer 3D motion into motion tokens, similar to the generation process of word tokens. Building upon this "motion vocabulary", we perform language modeling on both motion and text in a unified manner, treating human motion as a specific language. Moreover, inspired by prompt learning, we pre-train MotionGPT with a mixture of motion-language data and fine-tune it on prompt-based question-and-answer tasks. Extensive experiments demonstrate that MotionGPT achieves state-of-the-art performances on multiple motion tasks including text-driven motion generation, motion captioning, motion prediction, and motion in-between.

MotionGPT: Human Motion as a Foreign Language paper page: Though the advancement of pre-trained large language models unfolds, the exploration of building a unified model for language and other multi-modal data, such as motion, remains challenging and untouched so far. Fortunately, human motion displays a semantic coupling akin to human language, often perceived as a form of body language. By fusing language data with large-scale motion models, motion-language pre-training that can enhance the performance of motion-related tasks becomes feasible. Driven by this insight, we propose MotionGPT, a unified, versatile, and user-friendly motion-language model to handle multiple motion-relevant tasks. Specifically, we employ the discrete vector quantization for human motion and transfer 3D motion into motion tokens, similar to the generation process of word tokens. Building upon this "motion vocabulary", we perform language modeling on both motion and text in a unified manner, treating human motion as a specific language. Moreover, inspired by prompt learning, we pre-train MotionGPT with a mixture of motion-language data and fine-tune it on prompt-based question-and-answer tasks. Extensive experiments demonstrate that MotionGPT achieves state-of-the-art performances on multiple motion tasks including text-driven motion generation, motion captioning, motion prediction, and motion in-between.

AK

125,311 views • 3 years ago

The holy grail of robotics is to be able to perform previously-unseen, out-of-distribution manipulation tasks “zero shot” in a new environment. NovaFlow proposes an approach which (1) generates a video, (2) computes predicted flow — how points move through the scene — and (3) uses this flow as an objective to generate a motion. Using this procedure, NovaFlow generates motions in unseen scenes, for unseen tasks, and can transfer across embodiments. To learn more, we are joined by Hongyu Li and Jiahui Fu from RAI. Watch Episode #63 of RoboPapers with Chris Paxton and Michael Cho - Rbt/Acc now to learn more!

The holy grail of robotics is to be able to perform previously-unseen, out-of-distribution manipulation tasks “zero shot” in a new environment. NovaFlow proposes an approach which (1) generates a video, (2) computes predicted flow — how points move through the scene — and (3) uses this flow as an objective to generate a motion. Using this procedure, NovaFlow generates motions in unseen scenes, for unseen tasks, and can transfer across embodiments. To learn more, we are joined by Hongyu Li and Jiahui Fu from RAI. Watch Episode #63 of RoboPapers with Chris Paxton and Michael Cho - Rbt/Acc now to learn more!

RoboPapers

21,161 views • 3 months ago

Introducing Meta Perception Encoder: a vision encoder setting new standards in image & video tasks. It excels in zero-shot classification & retrieval, surpassing existing models. Learn more about Meta Perception Encoder, read the research paper, and download the code and dataset

Introducing Meta Perception Encoder: a vision encoder setting new standards in image & video tasks. It excels in zero-shot classification & retrieval, surpassing existing models. Learn more about Meta Perception Encoder, read the research paper, and download the code and dataset

AI at Meta

74,392 views • 1 year ago

With Nano Banana Pro in Flow, you have the control to design and curate the setting, refine the details, and visualize the camera angles. Get the shot just as you pictured it. See it in action.👇

With Nano Banana Pro in Flow, you have the control to design and curate the setting, refine the details, and visualize the camera angles. Get the shot just as you pictured it. See it in action.👇

FlowbyGoogle

19,697 views • 5 months ago

Introducing Flow Purification for Framer ! Show your transformation process in action - inputs flow in, core processes, polished outputs emerge. Fully customizable timing and animations for each stage. Check It Out👇

Introducing Flow Purification for Framer ! Show your transformation process in action - inputs flow in, core processes, polished outputs emerge. Fully customizable timing and animations for each stage. Check It Out👇

Nabeel

22,616 views • 6 months ago

NVIDIA AI Open-Sources ViPE (Video Pose Engine): A Powerful and Versatile 3D Video Annotation Tool for Spatial AI ViPE integrates bundle adjustment with dense optical flow, sparse keypoint tracking, and metric depth priors to estimate camera intrinsics, poses, and dense depth maps at 3–5 FPS on a single GPU. It significantly improves over prior uncalibrated pose estimation methods, achieving 18% and 50% error reduction on TUM and KITTI benchmarks, respectively, and shows robustness to dynamic scenes and diverse camera models. Beyond the method, the NVIDIA team also released a large-scale dataset comprising ~100K real-world internet videos, 1M AI-generated videos, and 2K panoramic videos (≈96M frames) annotated with metric depth and poses. This dataset and engine aim to accelerate training for spatial AI tasks such as 3D reconstruction, video generation, and robotics.... full analysis: paper: codes: NVIDIA NVIDIA AI NVIDIAnewsroom NVIDIA Robotics NVIDIA AIDev NVIDIAdeveloper

NVIDIA AI Open-Sources ViPE (Video Pose Engine): A Powerful and Versatile 3D Video Annotation Tool for Spatial AI ViPE integrates bundle adjustment with dense optical flow, sparse keypoint tracking, and metric depth priors to estimate camera intrinsics, poses, and dense depth maps at 3–5 FPS on a single GPU. It significantly improves over prior uncalibrated pose estimation methods, achieving 18% and 50% error reduction on TUM and KITTI benchmarks, respectively, and shows robustness to dynamic scenes and diverse camera models. Beyond the method, the NVIDIA team also released a large-scale dataset comprising ~100K real-world internet videos, 1M AI-generated videos, and 2K panoramic videos (≈96M frames) annotated with metric depth and poses. This dataset and engine aim to accelerate training for spatial AI tasks such as 3D reconstruction, video generation, and robotics.... full analysis: paper: codes: NVIDIA NVIDIA AI NVIDIAnewsroom NVIDIA Robotics NVIDIA AIDev NVIDIAdeveloper

Marktechpost AI Dev News ⚡

217,453 views • 9 months ago