Optical see-through augmented reality (AR) glasses are designed to overlay digital information onto the real world, but improving the realism of virtual objects often comes at a cost. Existing occlusion technologies physically block part of the incoming light, reducing the brightness of the real environment and making everyday tasks more challenging. Overcoming this balance between virtual realism and real-world visibility is a key hurdle for next-generation AR systems.
Addressing this challenge, a research team including Assistant Professor Xiaodan Hu, the key contributor to the project, from Shibaura Institute of Technology, Japan, in collaboration with Dr. Yan Zhang and Professor Xubo Yang from Shanghai Jiao Tong University, China, and Professor Kiyoshi Kiyokawa from the Nara Institute of Science and Technology, Japan, developed a perception-driven display strategy that jointly considers human visual perception and real-world lighting conditions. Instead of relying solely on hardware improvements, the researchers introduced a mask balancing method that dynamically adjusts the visibility of real and virtual scenes according to what users can actually perceive. This paper was made available online on April 8, 2026, and published in Volume 32, Issue 5 of IEEE Transactions on Visualization and Computer Graphics (TVCG), one of the leading journals in visualization, virtual and augmented reality, and computer graphics, on May 1, 2026.
The proposed method combines a polarized component that supports pixel-level occlusion with another polarized component that bypasses the optical system and preserves the natural brightness of the real world. By adjusting the cross-angle between a polarizing beam splitter and a linear polarizer, the system dynamically blends these two views. Real-time eye-tracking and scene analysis estimate the visibility of both the environment and virtual objects, allowing the display to continuously optimize the balance between them.
To establish perceptual thresholds, the researchers conducted a series of user studies. Experiments involving 12 participants quantified how much contrast was required for users to recognize textures in virtual objects, while another study with 24 participants evaluated the perception of lighting effects. The team then integrated these findings into a dynamic balancing strategy and validated it with a benchtop prototype. A final user study with 12 participants demonstrated that the system improved real-world visibility while maintaining a convincing appearance for virtual content across different illumination conditions.
"Our research demonstrates that human visual perception can be used to dynamically balance the visibility of the real world and virtual content," said Prof. Hu. "By adapting the display according to what users can actually perceive, our method improves real-world visibility while preserving the appearance of virtual objects under different lighting conditions."
The study also highlights a broader shift in AR display design. Rather than optimizing optical hardware alone, the researchers show that understanding how people perceive visual information can lead to more effective display control strategies. This perception-driven approach could help future AR glasses present virtual objects that blend more naturally into the real world while maintaining user safety and comfort.
Potential applications extend well beyond consumer electronics. Future optical see-through AR systems could support industrial maintenance, medical assistance, education, navigation, and remote collaboration, where users must simultaneously monitor their surroundings and interact with digital information. By preserving both real-world awareness and virtual image quality, the technology could make AR devices more practical for demanding real-world environments.
"In our previous research titled ' Perception-driven soft-edge occlusion for optical see-through head-mounted displays ,' we found that human perception of AR displays can differ significantly from predictions based solely on optical measurements," said Prof. Hu. "This inspired us to explore AR displays from a perceptual perspective because future AR glasses should be designed not only according to optical performance but also according to how humans actually perceive visual information."
Overall, the researchers believe that integrating perception science with display engineering may help overcome one of the most persistent barriers to widespread AR adoption. By leveraging human visual characteristics instead of depending entirely on hardware improvements, future systems may achieve more natural, photorealistic, and user-friendly AR experiences.
Reference
Title of original paper: Mask Balancing: Perception-Driven Dynamic Visibility Enhancement for Occlusion-Capable Optical See-Through Head-Mounted Displays
Journal: IEEE Transactions on Visualization and Computer Graphics
DOI: https://doi.org/10.1109/TVCG.2026.3679903
About Shibaura Institute of Technology (SIT), Japan
Shibaura Institute of Technology (SIT) is a private university with campuses in Tokyo and Saitama. Since the establishment of its predecessor, Tokyo Higher School of Industry and Commerce, in 1927, it has maintained "learning through practice" as its philosophy in the education of engineers. SIT was the only private science and engineering university selected for the Top Global University Project sponsored by the Ministry of Education, Culture, Sports, Science and Technology and had received support from the ministry for 10 years starting from the 2014 academic year. Its motto, "Nurturing engineers who learn from society and contribute to society," reflects its mission of fostering scientists and engineers who can contribute to the sustainable growth of the world by exposing their over 9,500 students to culturally diverse environments, where they learn to cope, collaborate, and relate with fellow students from around the world. Website: https://www.shibaura-it.ac.jp/en/
About Assistant Professor Xiaodan Hu
Xiaodan Hu works as an Assistant Professor at Shibaura Institute of Technology in Japan, where she directs the Augmented Imaging and Displays (AID) Laboratory . Before this, she worked as a postdoctoral researcher at Graz University of Technology in Austria. She also serves as a commissioned instructor at the Cybernetics and Reality Engineering Lab (CARE Lab) at the Nara Institute of Science and Technology in Japan, where she received her Ph.D. and M.Sc. degrees in Information Science under the supervision of Professor Kiyoshi Kiyokawa. Her research focuses on occlusion-capable optical see-through head-mounted displays, vision augmentation, and visual perception.
Funding Information
This work was funded by the Shanghai Pujiang Program (grant number: 23PJ1406800).
