Scientists have successfully reconstructed videos purely from the brain activity of mice, showing what the mice were seeing, in a new study led by University College London (UCL) researchers.
The findings, published in eLife, could help shed light on the intricate workings of how the brain processes visual information and open new avenues for exploring how different species perceive the world.
Over recent years, there has been a growing interest in understanding exactly how the human brain interprets signals from the eye. Images and movies have been played to people in fMRI machines and researchers around the world have tried to decode the brain's representations of visual information on a pixel level.
The new study builds on this approach but instead used single-cell recordings in mice, which offer the potential to provide a more precise measure of the brain's representations. This technique has enabled the team to create high-quality reconstructions of videos played to mice, based solely on the neural activity in the visual cortex.
Lead author Dr Joel Bauer (Sainsbury Wellcome Centre at UCL) said: "We wanted to have a better way of investigating how the brain interprets what we see. The current methods of understanding what specific groups of neurons are representing are not very generalisable to situations which haven't been specifically tested for. And so, we wanted to develop a method that can capture what is being represented in the brain and compare that to reality."
By looking for deviations between brain representations and reality, the new method could help researchers understand how specific visual cues shape neural representations.
Dr Bauer and colleagues used a dynamic neural encoding model, developed by another team for the 2023 Sensorium Competition, which predicts activity of individual neurons (brain cells) based on movies mice were being shown, while also accounting for the mouse's own movements and pupil diameter.
Using the same dataset, the UCL team further refined this model by calculating the difference between the predicted activity of neurons if a mouse had seen a blank screen, and the actual activity of the neurons (measured using a microscopic imaging technique that detects which individual brain cells are firing based on localised boosts in calcium levels). This allowed the scientists to gradually update the pixels of the blank movie through an algorithm and refine the model so that the output video closely resembled the video presented to the mouse.
Once the model was sufficiently trained, the researchers were then able to construct a 10-second movie based on a mouse's neural activity alone, measured as the mouse watched a video that had not been used previously in training the model.
Dr Bauer added: "Using this approach, we were able to achieve high-quality reconstructions of 10-second video clips. The accuracy of the reconstructions improved with the inclusion of data from more individual neurons, demonstrating the importance of comprehensive neural data."
To quantify the reliability of the reconstructions, the team used pixel correlation – correlating each pixel of the movie between the original version and the reconstructed version. They found minimal differences in the timing of the two videos, but they plan to focus on improving the resolution and coverage of visual reconstructions. This will involve getting data that can give higher resolution reconstructions and larger coverage of the visual scene.
Next, the team plan to use the technique to uncover new insights into the brain's visual processing capabilities. Specifically, they are interested in understanding how visual representations in the brain can deviate from what's actually in front of our eyes.
Dr Bauer concluded: "We don't have a perfect representation of the world in our heads. The visual processing pipeline skews and warps our representation in a way that modifies information. This deviation between reality and representations in the brain is not necessarily an error but a feature, reflecting how our minds interpret and augment sensory information. We want to explore how this happens in the brain."
