NEW WORLD MODEL: Yann LeCun's team is back with an efficient model! This project involves Yann LeCun, Lukas Kuhn, Lucas Maes, Quentin Le Lidec, and Randall Balestriero. A couple definitions first: - DINO: self-DIstillation with NO labels. A self-supervised image model (Meta, 2021) where a student network learns to... match a teacher (an EMA copy of itself) across two crops of the same image, with no labels and no negatives. - SIGReg: a regularizer that prevents embedding collapse by forcing the embeddings to match an isotropic Gaussian, tested with a normality test (Epps–Pulley) on many random 1-D projections instead of in full dimension. LeVJEPA is a self-supervised video pretraining method, released with open code, weights, and checkpoints. It learns a video representation by pushing the embeddings of global and local crops of the same clip together (an invariance loss), while a regularizer called SIGReg forces the embeddings toward an isotropic Gaussian to provably prevent representation collapse. Unlike V-JEPA and V-JEPA 2 it uses a single shared encoder with a projector and no target network, no predictor and no stop-gradient. It drops 95% of tokens per view, uses block-causal attention (each frame attends only to past frames), and has a single loss weight. It is evaluated purely as a representation learner via frozen probing on ImageNet-1K, Something-Something-v2 and Kinetics-400, not on any robot. What I find interesting, is that V-JEPA and V-JEPA 2 need an EMA target encoder, stop-gradients and a capacity-limited predictor to avoid collapse; LeVJEPA drops all of it for one shared encoder plus projector, preventing collapse instead with the SIGReg regularizer under a provable guarantee and a single hyperparameter. The "P" (predictor) in JEPA is effectively gone. LeVJEPA is also less compute intensive: - 5.6x to 20.8x lower total pretraining compute than V-JEPA 2 - 7.6 points higher on ImageNet-1K at matched FLOPs - trains at batch size 128 within 8GB where V-JEPA 2 saturates at batch size 28 Also worth mentioning: ImageNet-1K accuracy rises monotonically with the token-drop rate, from 33.9% at rho = 0 to 47.6% at rho = 0.95. The aggressive dropping is actually doing regularization work. On the JEPA-versus-DINO debate: - it loses to DINOv2 by 3.1 points on ImageNet-1K (appearance, static) - but wins on Something-Something-v2 by nearly 2x (motion, temporal) - and beats V-JEPA 2 by 1.9 points on ViT-L at 5.6x lower cost. -> optimized for temporal and motion understanding per compute dollar.show more

Léo

1,982 subscribers

Share Download