We’re very happy to share our latest study: ‘Brain decoding: toward real-time reconstruction of visual perception’ led by Yohann Benchetrit & @HubertBanville - paper: - blog: - summary: ⬇️

1/9 How the brain represents its surrounding worlds remains largely unknown. Since the early 2000s, machine learning has been used to solve this issue and trained to decode brain activity.

2/9 With modern AI, decoding of fMRI has greatly improved. Here, we extend this approach to MEG, a neuroimaging technique with a much high temporal resolution.

3/9 For this, we rely on the MEG-decoding architecture we recently released ( adapt it to visual perception, and add an image decoder on top.

4/9 We then train this pipeline on the MEG responses elicited in a rapid serial visual presentation of natural images provided by the THINGS initiative led by @martin_hebart, in which participants watch 0.5 s images every ~1.5s.

5/9 We compare the alignment between MEG signals and a variety of image encoders, when exposed to the same images.

6/9 We also compare a variety of design choices and finally use this Image - MEG alignment to condition the generation of plausible images. Here are some results with growing-window decoders: (Note the incorrect caption, each image lasts 500ms)

7/9 Overall the results are not as precise as what can be obtained with the relatively slow but spatially-precise 7T fMRI. Here are some examples obtained with a similar pipeline, trained tested on the NSD dataset, released by Thomas Naselaris and Kendrick Kay:

8/9 Nevertheless, the results preserve a remarkably high level of semantic features. E.g. the pandas is reconstructed as a black and white bear, the cheetah leads to a black-spotted mammals etc. We quite frankly did not expect that to be possible with MEG.

9/9 Overall, this approach leads to the exciting possibility of understanding the real-time unfolding of visual representations in the human brain.

We’re infinitely thankful to @AIatMeta, @ENS_ULM, the THINGS initiative and, more generally, the open-source and neuroscience communities for making this work possible.