I believe that StoryDiffusion has the potential to be Animatediff's complex motion sister-model!
POM
Visit on X@peterom16K followers
22,141 views
I believe that StoryDiffusion has the potential to be Animatediff's complex motion sister-model!
While AD is amazing for granular control, micro-motion and all kinds of abstract motion, it fails at complex realistic motion - walking, human movements, cars, etc.
StoryDiffusion seems very promising for this + also has characteristics that will likely make the community very receptive to it and likely to extend its capabilities:
2) Appealing base-model results - likely to get the community excited - feels like significantly better realistic motion than AD
2) Modular - their approach is built with a number of components that can be combined and taken apart - it works by generating consistent images, then animating them together - each of these stages can likely be upgraded, used and influenced in different ways.
3) Flexible - they demonstrate a bunch of different conditioning options
4) Likely easy on RAM - it's based on SD 1.5 + authors mention precautions to reduce RAM consumption
5) Built to plug into the existing ecosystem - e.g. the fact that it works with the SD1.5 ecosystem will give it a huge advantage!
While it's very early to say - e.g. the video model hasn't even been released yet! - it does seem very promising.
With 9 months of SD1.5/Animatediff-esque progress improving every element of it, I can see an an extremely extended version of this beating Sora + running for a fraction of the compute resources on a consumer GPU.
Together with Animatediff to drive the micro-motions and abstract stuff, it could produce be extraordinary/otherworldly/insane/beautiful stuff.
This is the first open video model I've been excited about since Animatediff - though cautiously optimistic!
Link here:
Comments
Comments from the original post on X (Twitter)
No comments yet
