正在加载视频...

视频加载失败

加载此视频时出现问题。这可能是由于临时网络问题，或视频可能不可用。

🚀 Introducing MegaFlow: Zero-Shot Large Displacement Optical Flow! 🔥 Code, models, and demo — all available now! 🎯 Large displacement motion estimation (optical flow, point tracking) has been a long-standing challenge. MegaFlow proposes a simple solution by leveraging pre-trained vision priors. Combined with a global matching formulation, we tackle... show more

Haofei Xu

1,402 subscribers

32,340 次观看 • 7 个月前 •via X (Twitter)

Anya Rossi• Live Now

Private livecam show

10 条评论

Haofei Xu 的头像

Haofei Xu7 个月前

🌐 Project: 📄 Paper: 💻 Code: This project was done by our truly exceptional Master's student @dingxi39482 (an incoming PhD student at ETH and MPI), together with amazing @FangjinhuaWang and @mapo1 !

Haofei Xu 的头像

Haofei Xu7 个月前

A little backstory: we developed GMFlow (CVPR 2022, and UniMatch (TPAMI 2023, about 4 years ago. It's incredibly exciting to see these old ideas shine again, empowered by modern architectures and tools. It’s equally rewarding to watch young students grow, and I'm so happy to be able to support their research journey!

Taro Bushidō 的头像

Taro Bushidō7 个月前

Finally, zero-shot large displacement that generalizes. The missing piece for robust world models. No domain gaps, no retraining.

Felix Wimbauer 的头像

Felix Wimbauer7 个月前

GMFlow rules. I am sure this one will too!

Haofei Xu 的头像

Haofei Xu7 个月前

Haha thanks Felix!

hamuz 的头像

hamuz7 个月前

No way, what a crazy coincidence! I actually started working on a similar project last month, and I’m using GH200 too! The only difference is I’m aiming Mamba-based approach. Your work looks awesome, I’m definitely going to dive deeper into it. Amazing job!

Haofei Xu 的头像

Haofei Xu7 个月前

Wow thank you so much! Great minds think alike haha. Your approach sounds super exciting, and I'm looking forward to seeing what you build with it! Best of luck with your project!

Kentskooking 的头像

Kentskooking6 个月前

@Machinedelusion This is so cool! Thank you!

Haotian Xue 的头像

Haotian Xue7 个月前

How fast is it?

Haofei Xu 的头像

Haofei Xu7 个月前

Hi, it runs at ~0.3s on an RTX 4090 at 540 × 960 resolution.

相关视频

expanding on my WebGPU motion tracking effect work this time with optical flow to visualize the motion of pixels as directional vectors in real time :)

expanding on my WebGPU motion tracking effect work this time with optical flow to visualize the motion of pixels as directional vectors in real time :)

Maxime

19,931 次观看 • 4 个月前

Meta just announced FlowVid Taming Imperfect Optical Flows for Consistent Video-to-Video Synthesis paper page: Diffusion models have transformed the image-to-image (I2I) synthesis and are now permeating into videos. However, the advancement of video-to-video (V2V) synthesis has been hampered by the challenge of maintaining temporal consistency across video frames. This paper proposes a consistent V2V synthesis framework by jointly leveraging spatial conditions and temporal optical flow clues within the source video. Contrary to prior methods that strictly adhere to optical flow, our approach harnesses its benefits while handling the imperfection in flow estimation. We encode the optical flow via warping from the first frame and serve it as a supplementary reference in the diffusion model. This enables our model for video synthesis by editing the first frame with any prevalent I2I models and then propagating edits to successive frames. Our V2V model, FlowVid, demonstrates remarkable properties: (1) Flexibility: FlowVid works seamlessly with existing I2I models, facilitating various modifications, including stylization, object swaps, and local edits. (2) Efficiency: Generation of a 4-second video with 30 FPS and 512x512 resolution takes only 1.5 minutes, which is 3.1x, 7.2x, and 10.5x faster than CoDeF, Rerender, and TokenFlow, respectively. (3) High-quality: In user studies, our FlowVid is preferred 45.7% of the time, outperforming CoDeF (3.5%), Rerender (10.2%), and TokenFlow (40.4%).

Meta just announced FlowVid Taming Imperfect Optical Flows for Consistent Video-to-Video Synthesis paper page: Diffusion models have transformed the image-to-image (I2I) synthesis and are now permeating into videos. However, the advancement of video-to-video (V2V) synthesis has been hampered by the challenge of maintaining temporal consistency across video frames. This paper proposes a consistent V2V synthesis framework by jointly leveraging spatial conditions and temporal optical flow clues within the source video. Contrary to prior methods that strictly adhere to optical flow, our approach harnesses its benefits while handling the imperfection in flow estimation. We encode the optical flow via warping from the first frame and serve it as a supplementary reference in the diffusion model. This enables our model for video synthesis by editing the first frame with any prevalent I2I models and then propagating edits to successive frames. Our V2V model, FlowVid, demonstrates remarkable properties: (1) Flexibility: FlowVid works seamlessly with existing I2I models, facilitating various modifications, including stylization, object swaps, and local edits. (2) Efficiency: Generation of a 4-second video with 30 FPS and 512x512 resolution takes only 1.5 minutes, which is 3.1x, 7.2x, and 10.5x faster than CoDeF, Rerender, and TokenFlow, respectively. (3) High-quality: In user studies, our FlowVid is preferred 45.7% of the time, outperforming CoDeF (3.5%), Rerender (10.2%), and TokenFlow (40.4%).

AK

123,771 次观看 • 2 年前

Tracking Everything Everywhere All at Once paper page: present a new test-time optimization method for estimating dense and long-range motion from a video sequence. Prior optical flow or particle video tracking algorithms typically operate within limited temporal windows, struggling to track through occlusions and maintain global consistency of estimated motion trajectories. We propose a complete and globally consistent motion representation, dubbed OmniMotion, that allows for accurate, full-length motion estimation of every pixel in a video. OmniMotion represents a video using a quasi-3D canonical volume and performs pixel-wise tracking via bijections between local and canonical space. This representation allows us to ensure global consistency, track through occlusions, and model any combination of camera and object motion. Extensive evaluations on the TAP-Vid benchmark and real-world footage show that our approach outperforms prior state-of-the-art methods by a large margin both quantitatively and qualitatively.

Tracking Everything Everywhere All at Once paper page: present a new test-time optimization method for estimating dense and long-range motion from a video sequence. Prior optical flow or particle video tracking algorithms typically operate within limited temporal windows, struggling to track through occlusions and maintain global consistency of estimated motion trajectories. We propose a complete and globally consistent motion representation, dubbed OmniMotion, that allows for accurate, full-length motion estimation of every pixel in a video. OmniMotion represents a video using a quasi-3D canonical volume and performs pixel-wise tracking via bijections between local and canonical space. This representation allows us to ensure global consistency, track through occlusions, and model any combination of camera and object motion. Extensive evaluations on the TAP-Vid benchmark and real-world footage show that our approach outperforms prior state-of-the-art methods by a large margin both quantitatively and qualitatively.

AK

280,570 次观看 • 3 年前

Check out our #PAMI paper with code "Dense Continuous-Time Optical Flow from Event Cameras," where we show how to regress *continuous-time* trajectories of every pixel from event cameras alone or events plus frames! The key idea is to iteratively estimate per-pixel polynomials using a recurrent lookup and update scheme. Paper: Code: DOI: We present a method for estimating dense continuous-time optical flow from event data. Traditional dense optical flow methods compute the pixel displacement between two images. Due to missing information, these approaches cannot recover the pixel trajectories in the blind time between two images. We show that it is possible to compute per-pixel, continuous-time optical flow using events from an event camera. Events provide temporally fine-grained information about movement in pixel space due to their asynchronous nature and microsecond response time. We leverage these benefits to predict pixel trajectories densely in continuous time via parameterized Bézier curves. To achieve this, we build a neural network with strong inductive biases for this task: First, we build multiple sequential correlation volumes in time using event data. Second, we use Bézier curves to index these correlation volumes at multiple timestamps along the trajectory. Third, we use the retrieved correlation to update the Bézier curve representations iteratively. Our method can optionally include image pairs to boost performance further. To train and evaluate our model, we introduce a synthetic dataset (MultiFlow) that features moving objects and ground truth trajectories for every pixel. Our quantitative experiments suggest that our method successfully predicts pixel trajectories in continuous time and is competitive in the traditional two-view pixel displacement metric on MultiFlow and DSEC-Flow. Open source code and datasets are released to the public. Kudos to Mathias Gehrig Manasi Muglikar

Check out our #PAMI paper with code "Dense Continuous-Time Optical Flow from Event Cameras," where we show how to regress continuous-time trajectories of every pixel from event cameras alone or events plus frames! The key idea is to iteratively estimate per-pixel polynomials using a recurrent lookup and update scheme. Paper: Code: DOI: We present a method for estimating dense continuous-time optical flow from event data. Traditional dense optical flow methods compute the pixel displacement between two images. Due to missing information, these approaches cannot recover the pixel trajectories in the blind time between two images. We show that it is possible to compute per-pixel, continuous-time optical flow using events from an event camera. Events provide temporally fine-grained information about movement in pixel space due to their asynchronous nature and microsecond response time. We leverage these benefits to predict pixel trajectories densely in continuous time via parameterized Bézier curves. To achieve this, we build a neural network with strong inductive biases for this task: First, we build multiple sequential correlation volumes in time using event data. Second, we use Bézier curves to index these correlation volumes at multiple timestamps along the trajectory. Third, we use the retrieved correlation to update the Bézier curve representations iteratively. Our method can optionally include image pairs to boost performance further. To train and evaluate our model, we introduce a synthetic dataset (MultiFlow) that features moving objects and ground truth trajectories for every pixel. Our quantitative experiments suggest that our method successfully predicts pixel trajectories in continuous time and is competitive in the traditional two-view pixel displacement metric on MultiFlow and DSEC-Flow. Open source code and datasets are released to the public. Kudos to Mathias Gehrig Manasi Muglikar

Davide Scaramuzza

12,692 次观看 • 2 年前

$Optical flow, fluid dynamics, refraction with lots and lots of render targets in WebGL #threejs #webgl$

Optical flow, fluid dynamics, refraction with lots and lots of render targets in WebGL #threejs #webgl

Aj

658,810 次观看 • 2 年前

PointOdyssey: A Large-Scale Synthetic Dataset for Long-Term Point Tracking paper page: introduce PointOdyssey, a large-scale synthetic dataset, and data generation framework, for the training and evaluation of long-term fine-grained tracking algorithms. Our goal is to advance the state-of-the-art by placing emphasis on long videos with naturalistic motion. Toward the goal of naturalism, we animate deformable characters using real-world motion capture data, we build 3D scenes to match the motion capture environments, and we render camera viewpoints using trajectories mined via structure-from-motion on real videos. We create combinatorial diversity by randomizing character appearance, motion profiles, materials, lighting, 3D assets, and atmospheric effects. Our dataset currently includes 104 videos, averaging 2,000 frames long, with orders of magnitude more correspondence annotations than prior work. We show that existing methods can be trained from scratch in our dataset and outperform the published variants. Finally, we introduce modifications to the PIPs point tracking method, greatly widening its temporal receptive field, which improves its performance on PointOdyssey as well as on two real-world benchmarks.

PointOdyssey: A Large-Scale Synthetic Dataset for Long-Term Point Tracking paper page: introduce PointOdyssey, a large-scale synthetic dataset, and data generation framework, for the training and evaluation of long-term fine-grained tracking algorithms. Our goal is to advance the state-of-the-art by placing emphasis on long videos with naturalistic motion. Toward the goal of naturalism, we animate deformable characters using real-world motion capture data, we build 3D scenes to match the motion capture environments, and we render camera viewpoints using trajectories mined via structure-from-motion on real videos. We create combinatorial diversity by randomizing character appearance, motion profiles, materials, lighting, 3D assets, and atmospheric effects. Our dataset currently includes 104 videos, averaging 2,000 frames long, with orders of magnitude more correspondence annotations than prior work. We show that existing methods can be trained from scratch in our dataset and outperform the published variants. Finally, we introduce modifications to the PIPs point tracking method, greatly widening its temporal receptive field, which improves its performance on PointOdyssey as well as on two real-world benchmarks.

AK

122,567 次观看 • 3 年前

Real-time visual effects, running 100% locally in your browser! 🚀 I created a demo where you can just drop your video and watch it analyze feature points and optical flow for some cool visual effects! Have fun 😊 Demo: #ComputerVision #VFX

Real-time visual effects, running 100% locally in your browser! 🚀 I created a demo where you can just drop your video and watch it analyze feature points and optical flow for some cool visual effects! Have fun 😊 Demo: #ComputerVision #VFX

Alexandre Devaux

33,705 次观看 • 1 年前

New AI research from Meta – CoTracker3 Simpler and Better Point Tracking by Pseudo-Labelling Real Videos. More details ➡️ Demo on Hugging Face ➡️ Building on our previous work on CoTracker, this new model demonstrates impressive tracking results where points can be tracked for a long time even when they're occluded or leave the field of view. CoTracker3 achieves state-of-the-art, outperforming all recent point tracking approaches on standard benchmarks — often by a substantial margin. We've released the research paper, code and a demo on Hugging Face — along with models available under an A-NC license to support further research in this space.

New AI research from Meta – CoTracker3 Simpler and Better Point Tracking by Pseudo-Labelling Real Videos. More details ➡️ Demo on Hugging Face ➡️ Building on our previous work on CoTracker, this new model demonstrates impressive tracking results where points can be tracked for a long time even when they're occluded or leave the field of view. CoTracker3 achieves state-of-the-art, outperforming all recent point tracking approaches on standard benchmarks — often by a substantial margin. We've released the research paper, code and a demo on Hugging Face — along with models available under an A-NC license to support further research in this space.

AI at Meta

219,050 次观看 • 2 年前

Can we synthesize 3D human-scene interactions without learning from any 3D data? Yes! Check out Lei Li's GenZI, a novel zero-shot approach to generating 3D interactions by distilling priors from large vision-language models.

Can we synthesize 3D human-scene interactions without learning from any 3D data? Yes! Check out Lei Li's GenZI, a novel zero-shot approach to generating 3D interactions by distilling priors from large vision-language models.

Angela Dai

107,013 次观看 • 2 年前

There was a demo at AWE of a motion sickness reduction technique called Reverse Optical Flow that I had never heard of before. It seemed to work quite better than any vignetting solution. It works by giving players a clear indication of any movement that's about to occur. It's kinda like "watching the road" when you're starting to feel motion sick in a car and anticipating turns.

There was a demo at AWE of a motion sickness reduction technique called Reverse Optical Flow that I had never heard of before. It seemed to work quite better than any vignetting solution. It works by giving players a clear indication of any movement that's about to occur. It's kinda like "watching the road" when you're starting to feel motion sick in a car and anticipating turns.

Flat2VR Modding

35,912 次观看 • 2 年前

Introducing PointZero—a 3D world model pre-trained without robots. Dexterous manipulation requires understanding diverse 3D dynamics, but current approaches rely on expensive robot data. So how can we pre-train a 3D dynamics model without it? PointZero introduces a simple idea: learning to complete 3D point tracks yields transferable 3D dynamics! Our model achieves SOTA results on: 🔥 Zero-shot 3D dynamics 🔥 Action-conditioned 3D dynamics (post-training) 🔥 Imitation learning (post-training) Details and links 👇 1/6

Introducing PointZero—a 3D world model pre-trained without robots. Dexterous manipulation requires understanding diverse 3D dynamics, but current approaches rely on expensive robot data. So how can we pre-train a 3D dynamics model without it? PointZero introduces a simple idea: learning to complete 3D point tracks yields transferable 3D dynamics! Our model achieves SOTA results on: 🔥 Zero-shot 3D dynamics 🔥 Action-conditioned 3D dynamics (post-training) 🔥 Imitation learning (post-training) Details and links 👇 1/6

Bardienus Duisterhof

28,984 次观看 • 1 个月前

It's been a pleasure to work on this with Gerard Pons-Moll I think we found a really effective formulation for training large-scale strong pose and shape models. Here are some more qualitative results on tough, in-the-wild YouTube dance videos.

It's been a pleasure to work on this with Gerard Pons-Moll I think we found a really effective formulation for training large-scale strong pose and shape models. Here are some more qualitative results on tough, in-the-wild YouTube dance videos.

István Sárándi

40,650 次观看 • 2 年前

🔥Diffusion Masters Transparent Objects in Videos! We introduce #DKT, a foundation model that repurposes video diffusion for zero-shot depth and normal estimation on Transparent and Reflective Objects with Superior Temporal Consistency Demo: Code:

🔥Diffusion Masters Transparent Objects in Videos! We introduce #DKT, a foundation model that repurposes video diffusion for zero-shot depth and normal estimation on Transparent and Reflective Objects with Superior Temporal Consistency Demo: Code:

Chongjie Ye

28,181 次观看 • 10 个月前

OPTICAL AIDS CAUSE 28 DAY MONTHS? NCS does not recommend that we use optical aids for a moonsighting system in Britain for many reasons. The primary reason is that we are seeking a unified solution, and unity is in the sunnah - local, naked eye sightings. Not everybody believes that optical aids are permissible, but naked eye sightings are wholly uncontroversial. Another issue is that optical aids would lead to more and more examples of the moon being “seen” a day early, and out of phase, on the 28th of a lunar month. This has ALREADY HAPPENED to those who use optical aids in the UK - yet has never happened to naked eye moonsighters for over 100 months. Whilst there are procedures from the sunnah of what to do if you were to see a moon on the 28th, we cannot really justify asking people to end their fast a day early on the 28th because of a moon which is invisible to the eye and only visible through optical aids. Hence our stance. Remember - we ourselves use optical aids - and had them on our livestream. We are experienced in them, and it is from our experience we share with you that we have found that it would leave to problems. In any case why are you in a rush to end Ramadan, using technology, and away from the sunnah?

OPTICAL AIDS CAUSE 28 DAY MONTHS? NCS does not recommend that we use optical aids for a moonsighting system in Britain for many reasons. The primary reason is that we are seeking a unified solution, and unity is in the sunnah - local, naked eye sightings. Not everybody believes that optical aids are permissible, but naked eye sightings are wholly uncontroversial. Another issue is that optical aids would lead to more and more examples of the moon being “seen” a day early, and out of phase, on the 28th of a lunar month. This has ALREADY HAPPENED to those who use optical aids in the UK - yet has never happened to naked eye moonsighters for over 100 months. Whilst there are procedures from the sunnah of what to do if you were to see a moon on the 28th, we cannot really justify asking people to end their fast a day early on the 28th because of a moon which is invisible to the eye and only visible through optical aids. Hence our stance. Remember - we ourselves use optical aids - and had them on our livestream. We are experienced in them, and it is from our experience we share with you that we have found that it would leave to problems. In any case why are you in a rush to end Ramadan, using technology, and away from the sunnah?

New Crescent Society

18,343 次观看 • 7 个月前

Huawei has received nine awards at the Lightwave Innovation Reviews, held during the Optical Fiber Communication Conference and Exposition. The awards span multiple categories and are a testament to Huawei's leadership in optical network technologies.

Huawei has received nine awards at the Lightwave Innovation Reviews, held during the Optical Fiber Communication Conference and Exposition. The awards span multiple categories and are a testament to Huawei's leadership in optical network technologies.

Huawei

14,161 次观看 • 7 个月前

Filmmaker Öner S. Biberkökü shares his "screenshot strategy" for building a full narrative from a single starting point during this In the Flow interview. Thanks Oner for the partnership! "At the end of the shot, you can take a screenshot, give it again [to Flow], and you now have a new image of that character in a different situation... So I think, with one image, you can stay in the flow, and make a story." Watch his breakdown:

Filmmaker Öner S. Biberkökü shares his "screenshot strategy" for building a full narrative from a single starting point during this In the Flow interview. Thanks Oner for the partnership! "At the end of the shot, you can take a screenshot, give it again [to Flow], and you now have a new image of that character in a different situation... So I think, with one image, you can stay in the flow, and make a story." Watch his breakdown:

Google Flow

43,255 次观看 • 8 个月前

TikTok presents Depth Anything Unleashing the Power of Large-Scale Unlabeled Data paper page: demo: Depth Anything is trained on 1.5M labeled images and 62M+ unlabeled images jointly, providing the most capable Monocular Depth Estimation (MDE) foundation models with the following features: zero-shot relative depth estimation, better than MiDaS v3.1 (BEiTL-512) zero-shot metric depth estimation, better than ZoeDepth optimal in-domain fine-tuning and evaluation on NYUv2 and KITTI

TikTok presents Depth Anything Unleashing the Power of Large-Scale Unlabeled Data paper page: demo: Depth Anything is trained on 1.5M labeled images and 62M+ unlabeled images jointly, providing the most capable Monocular Depth Estimation (MDE) foundation models with the following features: zero-shot relative depth estimation, better than MiDaS v3.1 (BEiTL-512) zero-shot metric depth estimation, better than ZoeDepth optimal in-domain fine-tuning and evaluation on NYUv2 and KITTI

AK

600,399 次观看 • 2 年前

Introducing 0x Cross-Chain API ⛓️ One integration to unify routing, bridging, execution, and tracking in a single flow. Ship cross-chain swaps faster. Operate with less overhead. Now in private beta 👇

Introducing 0x Cross-Chain API ⛓️ One integration to unify routing, bridging, execution, and tracking in a single flow. Ship cross-chain swaps faster. Operate with less overhead. Now in private beta 👇

0x

24,324 次观看 • 8 个月前

The NFL is close to implementing an optical tracking solution to measure first downs 👏

The NFL is close to implementing an optical tracking solution to measure first downs 👏

Joe Pompliano

123,552 次观看 • 1 年前

MotionGPT: Human Motion as a Foreign Language paper page: Though the advancement of pre-trained large language models unfolds, the exploration of building a unified model for language and other multi-modal data, such as motion, remains challenging and untouched so far. Fortunately, human motion displays a semantic coupling akin to human language, often perceived as a form of body language. By fusing language data with large-scale motion models, motion-language pre-training that can enhance the performance of motion-related tasks becomes feasible. Driven by this insight, we propose MotionGPT, a unified, versatile, and user-friendly motion-language model to handle multiple motion-relevant tasks. Specifically, we employ the discrete vector quantization for human motion and transfer 3D motion into motion tokens, similar to the generation process of word tokens. Building upon this "motion vocabulary", we perform language modeling on both motion and text in a unified manner, treating human motion as a specific language. Moreover, inspired by prompt learning, we pre-train MotionGPT with a mixture of motion-language data and fine-tune it on prompt-based question-and-answer tasks. Extensive experiments demonstrate that MotionGPT achieves state-of-the-art performances on multiple motion tasks including text-driven motion generation, motion captioning, motion prediction, and motion in-between.

MotionGPT: Human Motion as a Foreign Language paper page: Though the advancement of pre-trained large language models unfolds, the exploration of building a unified model for language and other multi-modal data, such as motion, remains challenging and untouched so far. Fortunately, human motion displays a semantic coupling akin to human language, often perceived as a form of body language. By fusing language data with large-scale motion models, motion-language pre-training that can enhance the performance of motion-related tasks becomes feasible. Driven by this insight, we propose MotionGPT, a unified, versatile, and user-friendly motion-language model to handle multiple motion-relevant tasks. Specifically, we employ the discrete vector quantization for human motion and transfer 3D motion into motion tokens, similar to the generation process of word tokens. Building upon this "motion vocabulary", we perform language modeling on both motion and text in a unified manner, treating human motion as a specific language. Moreover, inspired by prompt learning, we pre-train MotionGPT with a mixture of motion-language data and fine-tune it on prompt-based question-and-answer tasks. Extensive experiments demonstrate that MotionGPT achieves state-of-the-art performances on multiple motion tasks including text-driven motion generation, motion captioning, motion prediction, and motion in-between.

AK

125,403 次观看 • 3 年前