An artificial intelligence-based projection makeup system from Science Tokyo lets users describe a mood or style in their own words and instantly see matching makeup colors on their faces. The technology learns each person's preferences in real time and displays results under realistic lighting that reflects individual skin tone and texture, making it more true to life than traditional virtual makeup apps that project effects onto two-dimensional displays.
Realistic Makeup Exploration via Voice Input and Face Projection
Finding the right makeup color is an important part of the user experience when shopping for cosmetics. Virtual makeup technologies, which typically use augmented reality to overlay makeup effects through a smartphone or tablet, have made experimenting easier. However, choosing the right colors from hundreds of options can still feel overwhelming, and the results often appear artificial on a flat screen, failing to mimic how makeup appears on real skin under natural light.
Now, researchers from Institute of Science Tokyo (Science Tokyo), Japan, have created a new system that turns a user's spoken impressions that describe a mood or theme, with phrases like 'Sakura in spring,' directly into personalized makeup colors. This method combines an image generation artificial intelligence (AI) model with a projection system that lets users see the makeup simulated on their actual faces in real life, while the generated colors are refined using real cosmetic color distributions.
The study was led by graduate student Kemeng Zhang, graduate student Hao-Lun Peng, and Associate Professor Yoshihiro Watanabe from the Department of Information and Communications Engineering, Science Tokyo. The results were published online in the International Journal of Human-Computer Interaction on January 21, 2026.
"Users can easily explore preferred makeup colors from a large number of combinations through interactive optimization using impression words and projection-based makeup. This can help non-expert users efficiently find satisfying results in the vast space of color combinations," says Watanabe.
In this impression-guided text-to-makeup-color model, users simply describe the vibe they want, and the system translates it into makeup color suggestions. Users are encouraged to imagine scenes, objects, or moods and describe them naturally, using phrases such as 'night rose' or 'autumn forest with warm sunlight.' The AI then generates a reference image representing that impression and produces five suggested color themes for the cheeks, eyeshadow, and lips.
These colors are projected directly onto the user's face using a high-speed dynamic projection mapping setup. This system uses a high-speed projector and camera to ensure that the makeup remains correctly aligned with the user's face even as they move. The system tracks the user's facial features in real time, and users can view the results in a mirror.
This system accounts for how different skin tones and lip colors reflect light, making the simulation more realistic than viewing makeup on a two-dimensional screen. As users select their preferred options from the projected results, the system updates its suggestions using an optimization method that gradually learns their preferences.
"With such a system, users can simply describe their desired impression of makeup colors in natural language and observe the effects in the mirror," says Watanabe.
The system showed strong performance in user evaluations. In an online survey of a hundred participants, users reported that the system produced appropriate color suggestions for their impression texts. In a hands-on study with fifteen users comparing the system to a manual color adjustment tool, participants found the impression-guided method faster and more intuitive. Many said they enjoyed being able to quickly try a wide range of makeup styles, and some discovered appealing color combinations they would not have chosen on their own.
The system was also rated highly by experts from the cosmetics industry, who noted its potential for both everyday users and professionals, such as exploring makeup ideas for themed fashion shows or supporting early-stage product development by quickly generating color concepts from abstract design ideas.
This makeup generation and recommendation system highlights the growing role of AI in creative fields, helping consumers explore makeup styles they enjoy and offering beauty professionals new ways to develop and test ideas.
Reference
- Authors:
- Kemeng Zhang, Hao-Lun Peng, and Yoshihiro Watanabe*
*Corresponding author
- Title:
- Impression-Guided Interactive Personalized Color Exploration Framework for Dynamic Projection Mapping Makeup
- Journal:
- International Journal of Human-Computer Interaction
- Affiliations:
- Department of Information and Communications Engineering, Institute of Science Tokyo, Japan
