Artificial neural networks were originally inspired by the human brain, but they are still far less efficient at processing information. One reason the human brain is so efficient is its ability to focus only on the most relevant information and allocate cognitive effort based on the task. As artificial intelligence (AI) systems increasingly analyze multiple types of data, such as text, video, audio, and images, this ability to focus on the most important information is becoming increasingly important for reducing computational time and resource use. Most video frames contain little useful information, yet conventional AI systems still analyze them all, increasing computational cost and sometimes introducing noise that can reduce prediction accuracy.
A research team, led by Professor Shogo Okada from the Japan Advanced Institute of Science and Technology (JAIST), Japan, along with his Doctoral Student Hung Le from JAIST, has developed an AI model that identifies the key moments in a video, reducing the processing time for a 2-minute clip, from 52 seconds to 18 seconds. The findings of the study were made available online on July 11, 2026, and will be published in Volume 137 of the journal Information Fusion on January 01, 2027.
Doctoral Candidate Le, who is the first author of the paper, compares the model to the way humans naturally pay attention during a conversation: "Humans do not constantly keep their eyes on their conversation partner during a conversation. They first notice changes in sound and only direct their gaze at the specific moments they perceive as important."
The model, called EMF-dVAE (Efficient Multimodal Fusion with a discrete Variational Autoencoder), consists of two main components: a discrete variational autoencoder (dVAE) and a multimodal fusion (MF) network.
During training, the dVAE is given partially corrupted visual data, where the accompanying audio is first used to identify which visual segments should be masked. The model learns to reconstruct the missing visual information and, through this process, learns which visual regions are most informative for the task. Once training is complete, the dVAE identifies only the most important visual segments during inference. These selected visual features are then combined with the audio and language information by the MF network to produce the final prediction.
Since the model only analyzes a fraction of the video during inference, it requires much less computation while maintaining high accuracy. When tested on the ETS-Interview dataset, which contains 1,891 2-minute job interview videos from 260 participants, EMF-dVAE achieved state-of-the-art performance while using only 15.42% of the available visual features. It also reduced the processing time per video by approximately 65%, from 52 seconds to 18 seconds. Remarkably, by ignoring nearly 85% of the visual data, the model became both faster and more accurate, as redundant video frames can obscure informative signals while increasing computational cost.
"Much like the human brain, which focuses its attention on the most relevant moments, the AI automatically adjusts how much video data it analyzes for each clip, allowing it to allocate computational resources more efficiently," says Prof. Okada.
By making video analysis more efficient, the framework could enable AI-powered video interview coaches and communication-training tools that provide users with affordable, real-time feedback at near-expert levels. Because the framework processes only the most informative visual information, it also has the potential to reduce computational cost and energy consumption, supporting the development of more sustainable AI systems.
"Video is becoming the dominant form of data, and AI that must watch everything will not scale—economically or environmentally. Within 5–10 years, AI that budgets its attention the way humans do could make multimodal assistants, interview coaches, tutoring systems, and communication-support robots affordable and responsive on everyday devices," says Prof. Okada.
Reference
DOI: https://doi.org/10.1016/j.inffus.2026.104613
About Japan Advanced Institute of Science and Technology, Japan
Founded in 1990 in Ishikawa prefecture, the Japan Advanced Institute of Science and Technology (JAIST) was the first independent national graduate university that has its own campus in Japan. Now, after 30 years of steady progress, JAIST has become one of Japan's top-ranking universities. JAIST strives to foster capable leaders with a state-of-the-art education system where diversity is key; about 40% of its alumni are international students. The university has a unique style of graduate education based on a carefully designed coursework-oriented curriculum to ensure that its students have a solid foundation on which to carry out cutting-edge research. JAIST also works closely both with local and overseas communities by promoting industry–academia collaborative research.
Website: https://www.jaist.ac.jp/english/
About Professor Shogo Okada from Japan Advanced Institute of Science and Technology, Japan
Dr. Shogo Okada is a Professor at the AI Wisdom Research Area, Japan Advanced Institute of Science and Technology (JAIST), where he leads the Social Signal and Multimodal Interaction Research Group. He earned his Doctor of Engineering degree in Computer Science from the Tokyo Institute of Technology in 2008. His research focuses on artificial intelligence, multimodal interaction modeling, and human dynamics analysis using machine learning, pattern recognition, and data mining techniques. His work explores computational approaches for analyzing human behavior and interactions, contributing to the development of AI systems that can better understand complex human-centered data.
