Skip navigation
Long video generation usually results in context increasing/scaling during chunk/frame-wise rollout.
0:00 / 1:44

Long video generation usually results in context increasing/scaling during chunk/frame-wise rollout.

Ceyuan Yang

@CeyuanY2.3K followers

Visit on X
216 likesDownloadOriginal post

22,279 views

Long video generation usually results in context increasing/scaling during chunk/frame-wise rollout. Considering context scaling may require context selection, we thus introduce the idea of MoE into long context modelling and propose Mixture of Contexts. All previous context/memory is considered while the chosen ones are computed in a data-driven manner. You can easily enjoy 7x compute savings.

5 Comments

Comments from the original post on X (Twitter)

  • Ceyuan Yang's profile picture

    @CeyuanY

    Home page: ArXiv: Thank all coauthors for making this happen, especially to Shengqu @prime_cai. Somehow our work is also ROUTED by arxiv to graphics rather than CV :)

    15 likes

  • Wenhao Chai @ COLM's profile picture

    @wenhaocha1

    great work, any idea which is better, this or block sparse attn 🧐

    2 likes

  • Ceyuan Yang's profile picture

    @CeyuanY

    really hard to make an apple-to-apple fair comparison, in terms of quality and training/inference flops, mfu. In my mind, the dynamic part may be the key to sparse attn.

    1 like

  • Torus Automations's profile picture

    @torusautomation

    Very interesting.

  • Tsukuyomi's profile picture

    @0xtsk

    context scaling in video generation? sounds like a recipe for an existential crisis. let’s hope it doesn’t turn into a MoE nightmare where we’re all left in the dark, watching our lives unfold in fragmented clips. cheers to the future of AI, I guess.