🚨 Thought-to-Video with Stable Diffusion just released. An era has come where what you think in your head becomes a video. Here's everything you need to know: 🧵🧠

Incredibly exciting, and yet scary to think of this. On the left are the videos watched by the human subjects, on the right are the videos produced with the data deciphered from their brains.

People have been working on recreating human vision from brain activity using AI, which helps us understand how our minds work. While there has been a lot of success in recreating still images from brain scans, there hasn't been as much progress with video.

A new tool called Mind-Video uses fMRI data from the brain to recreate video images. main: github:

You can see the similarity between Ground truth Videos and Reconstructed Videos below: 1.

2.

3.

4.

The process includes two stages: Stage A uses Sparse-Coding Masked Brain Modeling (SC-MBM) on a large brain scan dataset. Stage B applies a Double-Conditioned Latent Diffusion Model (DC-LDM) to generate images from brain recordings.

Using this method, they achieved an average accuracy of 85% on task. Where the computer had to understand the video, and a score of 0.19 in a measure of visual quality, which was 45% better than the best results so far.

Video mapping your thoughts and even dreams will be a first and important step to something much bigger 🤔

If you found this helpful or interesting. Please retweet the tweet below to share it with others: