All of this is enabled by fleet scale auto-labelling. By using video data from multiple trips in the same location, we can reconstruct the entire scene

Tesla is building the foundation models for autonomous robots

Our multi-modal neural networks are already in customer vehicles—these networks take in arbitrary modalities such as camera videos, maps, navigation, IMU (Inertial Measurement Unit), GPS etc.

Tasks such as Occupancy prediction are already quite general in what they represent—in some ways, they are ontology-free & simply predict the probability that some 3D position is occupied. Such occupancy can be used for collision avoidance by any robot.

In addition, we’re building off state-of-the-art generative modeling techniques—enabling us to predict possible outcomes given past observations, in a jointly consistent manner across multiple camera views

These imagined futures can be action-conditioned to produce different outcomes. For example, the videos below are generated entirely by the neural network by simply using different prompts

These models will learn from a huge set of extremely diverse data from the Tesla fleet

And will be trained on enormous amounts of compute

These video foundation models will serve as the brain of both the car & Optimus robot 🚘🧠🤖 Join the Tesla AI team to build the future of robotics! →

Wow... high-res nerfs? 🤔

Wow 👌

Amazing capability to reconstruct real world video for simulations. 👍.

How is this different than 3d hd maps. Does it save somewhere locally or globally?

Amazing technology! I truly believe data analysis helps fuel innovation ✨ Excited to see how @Tesla_AI's auto-labelling can improve AI recognition & make our roads safer. #innovation #sustainability #hundrx