New system could improve autonomous driving, machine vision and smartphone facial recognition
WASHINGTON - Researchers report a new compact low-cost hyperspectral projector system that provides both depth information and hyperspectral images. The new system could be useful for autonomous driving systems, machine vision in industrial robotics, agricultural monitoring of crops, and monitoring material surfaces for wear and corrosion.
"Our work enables fast 3D hyperspectral imaging in an efficient and low-cost manner," said lead author Yibo Xu, who earned her Ph.D. from Rice University. "This could one day allow the sensors used for facial recognition on smartphones to be used as hyperspectral 3D scanners, which would improve color accuracy and increase the security of face classification."
Caption: Researchers have developed a new compact low-cost hyperspectral projector system that provides both depth information and hyperspectral images using a monochrome camera as the sensor. They used the new system to analyze a scene that contained a small red and green ramp alongside red and green candies of different sizes. Although the green color on the ramp and the green coating of the candy look quite the same to the human eye and a RGB camera, the system was able to clearly distinguish the two materials from the reconstructed spectra as well as measure the correct heights for all the objects.
Credit: Yibo Xu, Rice University
Hyperspectral imagers detect dozens to hundreds of colors, or wavelengths, instead of the three detected by normal cameras. Combining this with 3D imaging is useful for perceiving and understanding real-world scenes and objects. Previous hyperspectral 3D imaging systems have required a complicated, high-cost hardware design and came with a long acquisition and reconstruction time.
In The Optical Society (OSA) journal Optics Express, the researchers detail their new simple design for a hyperspectral stripe projector and demonstrate that it allows the use of a monochrome camera to simultaneously capture depth information and also distinguish colors that appear visually similar.
"The combination of 3D spatial and spectrally specific material information is quite powerful," said research team member Kevin F. Kelly, Ph.D. "It can be used for analyzing cultural heritage objects and pieces of art, monitoring plants and agriculture for signs of nutrient deficiencies or disease, aiding industrial robot systems in sorting and assembly, and expanding current autonomous driving systems to better identify the roadway, other vehicles and potential hazards."
Optimizing for speed and simplicity
Most hyperspectral 3D imaging systems measure the spectral content of a scene using a hyperspectral camera. In the new work, the researchers redesigned the hardware and developed new software to allow the use of a monochrome camera to capture 4D information (3D spatial and spectral information) from a scene at once.
Caption: Lead author Xu with her research advisor Kelly from Rice University (pictured) developed the new hyperspectral projector system along with Anthony Giljum.
Credit: Yibo Xu, Rice University
A traditional digital projector uses a color wheel with just a handful of colors and is not suitable for encoding the spectral information. The researchers used a different approach that creates hyperspectral stripe patterns that can each be programmed to have an arbitrary spectrum. This allows simultaneous 3D spatial and spectral encoding while only requiring a monochrome camera to capture the images.
The projector creates stripes by using a diffraction grating to split white light from a lamp into its different color components. Each color can then be subdivided into finer wavelengths and focused onto an array of tiny, programmable mirrors called digital micromirror devices (DMD). The unique optical layout that guides light through the system makes it simple, efficient and compact. The researchers also developed new algorithms to reconstruct the collected images into a hyperspectral, 3D visualization of the scene.
"Other systems typically require two or more gratings and multiple DMDs or light modulators," said Xu. "This not only makes them larger and more expensive but also means a brighter light source is needed. Our system achieves its compact form factor by requiring only a single DMD and a single diffraction grating."
Capturing 3D color detail
The researchers used their new system to analyze a scene that contained a small red and green ramp alongside red and green candies of different sizes. Although the green color on the ramp and the green coating of the candy look quite the same to the human eye and an RGB camera, the system was able to clearly distinguish the two materials from the reconstructed spectra as well as measure the correct heights for all the objects.
"By having an easy way to perform controlled hyperspectral depth imaging, researchers will be able to more easily identify the chemical compounds that make up objects of interest," said Kelly. "This could also be useful for a variety of applications from medical diagnostics to monitoring fresh produce for damage and contamination during sorting and delivery."
The researchers are already working on the next-generation design, which will have a more refined optical system and improved reconstruction algorithms. They are also building variations that will operate beyond the visible into infrared portions of the electromagnetic spectrum.
Paper: Y. Xu, A. Giljum, K. F. Kelly, A Hyperspectral projector for simultaneous 3D spatial and hyperspectral imaging via structured illumination, Opt. Express, 28, 20, 29740-29755 (2020). DOI: https://doi.org/10.1364/OE.402812.
