Kostas Daniilidis
@KostasPenn • 5,076 subscribers
Ruth Yalom Stone Professor @Penn @PennEngineers @PennCIS @GRASPlab
Videos
VLA policies learn generalist robot behaviors from massive teleoperation datasets, hoping that the right behavior emerges. But they rarely use perception during training or inference: powerful foundation models of 3D geometry, semantics, or human motion are ignored. TimSong, Long Le, and our GRASP Laboratory team introduce Omniguide, based on a simple idea: Instead of retraining policies, let perception guide action at inference time. We express diverse guidance sources as attractive and repulsive energy fields in 3D space and inject their gradients into the generative process of VLA policies. This lets perception modules steer actions without retraining the policy: A modular path toward composable robot intelligence. Project: Paper:
Kostas Daniilidis47,379 views • 3 months ago
No more content to load