Prof. Stanley Chan achieves new theoretical breakthrough in high dynamic range imaging
Stanley Chan, associate professor of electrical and computer engineering and statistics, and Abhiram Gnanasambandam, a PhD student in ECE, have found a new way to break through the high dynamic range imaging problem using single-photon image sensors.
Most cell phone cameras today have high dynamic range (HDR) imaging capability. When the scene is too bright or too dark, the HDR mode will take a stack of frames at different exposures and then use an image processing algorithm to fuse them. The bottleneck, however, is that the CMOS image sensors (CIS) we use today are not very good for extreme lighting conditions. For example, when the scene is extremely dark, the shot noise and read noise will override the signal. Also, capturing a stack of frames is time-consuming.
To break through the challenge, Chan and Gnanasambandam studied a new type of image sensor called the Quanta Image Sensor (QIS), a sensor proposed in 2005 by E. Fossum at Dartmouth College. In Fossum’s original design, the QIS was anticipated to operate at very high speed but the bit-depth of a pixel is low. It was hypothesized that the speed and bit-depth trade-off can potentially offer a breakthrough in HDR imaging. However, the theory had never been formally studied as the actual sensor was not made until very recently.
Chan and Gnanasambandam made two contributions to this open problem. First, they established the theoretical limit of the sensor. The results they derived are exact, meaning the equations do not involve any ad-hoc approximations or numerical calculation. According to their theory, the trade-off offered by QIS can lead to a substantially better dynamic range compared to a conventional CIS.
“People have been hypothesizing that QIS would offer better high dynamic range than a conventional CIS because of the speed and lower noise,” says Gnanasambandam, the student author of the paper. “However, no one has been able to quantify this. With our analysis and experiments, I am glad that we can finally prove it.”
The second contribution Chan and Gnansambandam made was a provably optimal reconstruction method. Despite the nonlinearity of the problem, they were able to obtain a very elegant expression that is reminiscent of the traditional linear methods for HDR imaging --- now with quadratic weights. This result leads to a new reconstruction algorithm that is provably optimal.
“This paper opens the door to a wide range of potential applications, including autonomous vehicles, surveillance, and medical imaging,” says Chan. “As long as you have a scene that contains very bright and very dark regions, QIS can easily outperform a CIS. I am very confident that in the near future we can see a new generation of cameras.”
The project is supported by the United States National Science Foundation. The authors acknowledge Prof Eric Fossum at Dartmouth College for offering many great advices on this project. They thank Dr. Jiaju Ma of GigaJot Tech for allowing them to use the QIS camera for experiments. The paper is accepted for publication in IEEE Transactions on Computational Imaging, where the manuscript is available on arXiv. The paper is presented in the 2019 International Image Sensor Workshop, June 2019.
Reference:
Abhiram Gnanasambandam, Stanley H. Chan, “HDR Imaging with Quanta Image Sensors: Theoretical Limits and Optimal Reconstruction”, IEEE Transactions on Computational Imaging.