Quanta Image Sensor

HDR Imaging with Quanta Image Sensors: Theoretical Limits and Optimal Reconstruction

IEEE Transactions on Computational Imaging, 2020
Abhiram Gnanasambandam and Stanley H. Chan
Manuscript: https://arxiv.org/abs/2011.03614

High dynamic range (HDR) imaging is one of the biggest achievements in modern photography. Traditional solutions to HDR imaging are designed for and applied to CMOS image sensors (CIS). However, the mainstream one-micron CIS cameras today generally have a high read noise and low frame-rate. These, in turn, limit the acquisition speed and quality, making the cameras slow in the HDR mode. In this paper, we propose a new computational photography technique for HDR imaging. Recognizing the limitations of CIS, we use the Quanta Image Sensor (QIS) to trade the spatial-temporal resolution with bit-depth. QIS is a single-photon image sensor that has comparable pixel pitch to CIS but substantially lower dark current and read noise. We provide a complete theoretical characterization of the sensor in the context of HDR imaging, by proving the fundamental limits in the dynamic range that QIS can offer and the trade-offs with noise and speed. In addition, we derive an optimal reconstruction algorithm for single-bit and multi-bit QIS. Our algorithm is theoretically optimal for emph{all} linear reconstruction schemes based on exposure bracketing. Experimental results confirm the validity of the theory and algorithm, based on synthetic and real QIS data.

Dynamic Low-light Imaging with Quanta Image Sensors

European Conference on Computer Vision (ECCV), 2020.
Manuscript: https://arxiv.org/abs/2007.08614
Yiheng Chi, Abhiram Gnanasambandam, Vladlen Koltun, and Stanley H. Chan

Imaging in low light is difficult because the number of photons arriving at the sensor is low. Imaging dynamic scenes in low-light environments is even more difficult because as the scene moves, pixels in adjacent frames need to be aligned before they can be denoised. Conventional CMOS image sensors (CIS) are at a particular disadvantage in dynamic low-light settings because the exposure cannot be too short lest the read noise overwhelms the signal. We propose a solution using Quanta Image Sensors (QIS) and present a new image reconstruction algorithm. QIS are single-photon image sensors with photon counting capabilities. Studies over the past decade have confirmed the effectiveness of QIS for low-light imaging but reconstruction algorithms for dynamic scenes in low light remain an open problem. We fill the gap by proposing a student-teacher training protocol that transfers knowledge from a motion teacher and a denoising teacher to a student network. We show that dynamic scenes can be reconstructed from a burst of frames at a photon level of 1 photon per pixel per frame. Experimental results confirm the advantages of the proposed method compared to existing methods.

Image Classification in the Dark using Quanta Image Sensors

European Conference on Computer Vision (ECCV), 2020
Manuscript: https://arxiv.org/abs/2006.02026
Abhiram Gnanasambandam, and Stanley H. Chan

Abstract:
State-of-the-art image classifiers are trained and tested using well-illuminated images. These images are typically captured by CMOS image sensors with at least tens of photons per pixel. However, in dark environments when the photon flux is low, image classification becomes difficult because the measured signal is suppressed by noise. In this paper, we present a new low-light image classification solution using Quanta Image Sensors (QIS). QIS are a new type of image sensors that possess photon counting ability without compromising on pixel size and spatial resolution. Numerous studies over the past decade have demonstrated the feasibility of QIS for low-light imaging, but their usage for image classification has not been studied. This paper fills the gap by presenting a student-teacher learning scheme which allows us to classify the noisy QIS raw data. We show that with student-teacher learning, we are able to achieve image classification at a photon level of one photon per pixel or lower. Experimental results verify the effectiveness of the proposed method compared to existing solutions.

Color Filter Array Designs

CFA 

Quanta image sensor (QIS) is envisioned to be the next generation image sensor after CCD and CMOS. In this paper, we discuss how to design color filter arrays for QIS. Designing color filter arrays for small pixels such as QIS is challenging because maximizing the light efficiency while suppressing aliasing and crosstalk are conflicting tasks. We present an optimization-based framework which unifies several mainstream color filter array design methodologies and offers greater generality and flexibility. Compared to existing methods, the new framework can simultaneously handle luminance sensitivity, chrominance sensitivity, cross-talk, anti-aliasing, manufacturability and orthogonality. Extensive experimental comparisons demonstrate the effectiveness and generality of the framework.

Publication:

  1. Omar A. Elgendy, and Stanley H. Chan, ‘‘Color filter arrays design for quanta image sensor’’, IEEE Trans. Computational Imaging, 2019.

High Dynamic Range Imaging

HDR 

An algorithmic solution is proposed for reconstructing high dynamic range (HDR) images from single-bit and multi-bit Quanta Image Sensor (QIS). Given a space-time cubicle of the QIS data, the algorithm partitions the frames into groups of different exposures. After summation and denoising, the resulting frames are combined to form the HDR image. The combination weights are determined according to a new theoretical result showing how signal to noise changes with the exposure. The new method is compared with conventional CMOS-based HDR image reconstruction methods on both synthetic and real QIS data.

Publication:

Abhiram Gnanasambandam, Jiaju Ma, and Stanley H. Chan, ‘‘High dynamic range imaging with quanta image sensor’’, International Image Sensor Workshop (IISW), Snowbird, Utah, Jun. 2019. Paper R23. (Oral)

Color Imaging

colorQIS 

Quanta Image Sensor (QIS) is a single-photon detector designed for extremely low light imaging conditions. Majority of the existing QIS prototypes are monochrome based on single-photon avalanche diodes (SPAD). Passive color imaging has not been demonstrated with single-photon detectors due to the intrinsic difficulty of shrinking the pixel size and increasing the spatial resolution while maintaining acceptable intra-pixel cross-talk. In this paper, we present image reconstruction of the first color QIS with a resolution of 1024-by-1024 pixels, supporting both single-bit and multi-bit photon counting capability. Our color image reconstruction is enabled by a customized joint demosaicing-denoising algorithm, leveraging truncated Poisson statistics and variance stabilizing transforms. Experimental results of the new sensor and algorithm demonstrate superior color imaging performance for very low-light conditions with a mean exposure of as low as a few photons per pixel in both real and simulated images.

Publication:

  1. Abhiram Gnanasambandam, Omar A. Elgendy, Jiaju Ma, and Stanley H. Chan, ‘‘Megapixel photon-counting color imaging using quanta image sensor’’, OSA Optics Express, vol. 27, no. 12, pp. 17298-17310, Jun. 2019.

Multi-bit QIS

multibit 

Publication:

Jiaju Ma, Yu-Wing Chung, Abhiram Gnanasambandam, Stanley H. Chan, and Saleh Masoodian, ‘‘Photon-counting imaging with multi-bit quanta image sensor’’, International Image Sensor Workshop (IISW), Snowbird, Utah, Jun. 2019. Paper R19. (Oral)

Image Reconstruction using Deep Convolutional Networks

QISNet 

Quanta Image Sensor (QIS) is a single-photon image sensor that oversamples the light field to generate binary measurements. Its single-photon sensitivity makes it an ideal candidate for the next generation image sensor after CMOS. However, image reconstruction of the sensor remains a challenging issue. Existing image reconstruction algorithms are largely based on optimization. In this paper, we present the first deep neural network approach for QIS image reconstruction. Our deep neural network takes the binary bit stream of QIS as input, learns the nonlinear transformation and denoising simultaneously. Experimental results show that the proposed network produces significantly better reconstruction results compared to existing methods.

Publication:

  1. Joon Hee Choi, Omar A. Elgendy and Stanley H. Chan, ‘‘Image reconstruction for Quanta Image Sensors using deep neural networks’’, IEEE ICASSP, pp. 6543–6547, Calgary, Canada, Apr. 2018.

Optimal Threshold

threshold 

Quanta Image Sensor (QIS) has been envisioned as a candidate solution for next generation image sensors. We provide two new contributions to the signal processing aspects of QIS. First, we develop an image reconstruction algorithm to recover the underlying images from the QIS data, which is a massive array of binarized Poisson random variables. The new algorithm supersedes existing methods by enabling arbitrary threshold level. Second, we present a threshold design scheme to adaptively update the threshold level for optimal image reconstruction. We discuss the existence of a phase transition in determining the optimal threshold. Experimental results on tone-mapped high dynamic range images validates the effectiveness of the threshold scheme and the image reconstruction algorithm.

MATLAB Implementation (1.1MB)

Publication:

  1. Omar A. Elgendy and Stanley H. Chan, ‘‘Optimal Threshold Design for Quanta Image Sensor’’, IEEE Trans. Computational Imaging, vol. 4, no. 1, Mar. 2018, pp. 99-111.

  1. Omar Elgendy and Stanley H. Chan, ‘‘Image reconstruction and threshold design for quanta image sensors’’, IEEE ICIP, pp.978-982, Phoenix, Arizona, Sep. 2016. (ICIP 2016 Best Paper Award)

Non-Iterative Image Reconstruction

transform_denoise 

Quanta image sensor (QIS) is a class of single photon imaging devices that measure light intensity using oversampled binary observations. Because of the stochastic nature of the photon arrivals, data acquired by QIS is a massive stream of random binary bits. The goal of image reconstruction is to recover the underlying image from these bits. In this paper, we present a non-iterative image reconstruction algorithm for QIS. Unlike existing reconstruction methods that formulate the problem from an optimization perspective, the new algorithm directly recovers the images through a pair of nonlinear transformations and an off-the-shelf image denoising algorithm. By skipping the usual optimization procedure, we achieve orders of magnitude improvement in speed and even better image reconstruction quality. We validate the new algorithm on synthetic datasets as well as real videos collected by 1-bit SPAD cameras.

Publication:

  1. Stanley H. Chan, Omar Elgendy and Xiran Wang, ‘‘Images from bits: Non-iterative image reconstruction for quanta image sensors’’, MDPI Sensors Special Issue on Photon-Counting Image Sensors, vol. 16, no. 11, paper 1961, pp.1-21, Nov. 2016.

ADMM Image Reconstruction

qis 

Recent advances in materials, devices and fabrication technologies have motivated a strong momentum in developing solid-state sensors that can detect individual photons in space and time. It has been envisioned that such sensors can eventually achieve very high spatial resolutions as well as high frame rates. In this paper, we present an efficient algorithm to reconstruct images from the massive binary bit-streams generated by these sensors. Based on the concept of alternating direction method of multipliers (ADMM), we transform the computationally intensive optimization problem into a sequence of subproblems, each of which has efficient implementations in the form of polyphase-domain filtering or pixel-wise nonlinear mappings. Moreover, we reformulate the original maximum likelihood estimation as maximum a posterior estimation by introducing a total variation prior. Numerical results demonstrate the strong performance of the proposed method, which achieves several dB's of improvement in PSNR and requires a shorter runtime as compared to standard gradient-based approaches.

Publication:

  1. Stanley H. Chan, and Yue M. Lu, ‘‘Efficient image reconstruction for giga-pixel quanta image sensors’’, IEEE GlobalSIP, pp. 312-316, Altanta, Georgia, Dec. 2014,