How do humans and monkeys find meaning in visual scenes? New in JNeurosci, researchers led by Vinay Raghavan, from the City College of New York, advanced understanding by assessing eye movements together with brain activity as humans and macaques watched full-length movies or short video clips.
The researchers discovered that new semantic—or conceptual—information during full-length movies influenced neural activity in attention and scene-processing brain areas. Deeper brain recordings taken from a separate set of human participants and two female macaques showed early, rapid brain activity in attention and scene-processing brain areas when short videos presented new semantic information, but only in between eye fixations while taking in a scene. What might that mean? Raghavan explains that the involvement of brain areas beyond visual regions suggests there is a higher order—or more abstract—mental system that may be involved in predicting what is about to happen between glimpses to quickly create meaning.
According to the researchers, this work suggests that visual processing is a continuous process that involves comparing semantic information from separate glimpses of visuals to understand scenes as they unfold. The research team emphasizes the relevance of this work to artificial intelligence (AI) models. Says Raghavan, "It's possible the learning strategy in this study trained neural networks and the primate brain to produce a 'semantic novelty signal.' The fact that this semantic novelty signal is encoded in the brain suggests that [we can implement this learning strategy] to train new vision models in AI."
