Quanta Image Sensor

Since the invention of CMOS imaging sensors in the early 90s, pixel pitch has been continually decreasing, approaching the diffraction limit for most imaging systems. While pushing for smaller pixels may still yield some modest gains in a near future, a fundamentally new generation of imaging sensors should be developed. The goal of this project is to study an emerging class of solid-state imaging devices, collectively known as quanta image sensors (QIS), that achieve single- photon sensitivity with high spatial and temporal resolution. The key scope of the project is to design efficient image reconstruction algorithms to “decode” the data acquired by the QIS which is a stream of massive one-bit stochastic signals resulting from the arriving photons. Resolving this technological challenge will remove one of the most critical roadblock for hardware design, software development, and performance analysis of the QIS.

Deep Learning for QIS Image Reconstruction


Quanta Image Sensor (QIS) is a single-photon image sensor that oversamples the light field to generate binary measurements. Its single-photon sensitivity makes it an ideal candidate for the next generation image sensor after CMOS. However, image reconstruction of the sensor remains a challenging issue. Existing image reconstruction algorithms are largely based on optimization. In this paper, we present the first deep neural network approach for QIS image reconstruction. Our deep neural network takes the binary bit stream of QIS as input, learns the nonlinear transformation and denoising simultaneously. Experimental results show that the proposed network produces significantly better reconstruction results compared to existing methods.


  1. Joon Hee Choi, Omar A. Elgendy and Stanley H. Chan, ‘‘Image reconstruction for Quanta Image Sensors using deep neural networks’’, IEEE ICASSP, pp. 6543–6547, Calgary, Canada, Apr. 2018.

Optimal Threshold Design for QIS


Quanta Image Sensor (QIS) has been envisioned as a candidate solution for next generation image sensors. We provide two new contributions to the signal processing aspects of QIS. First, we develop an image reconstruction algorithm to recover the underlying images from the QIS data, which is a massive array of binarized Poisson random variables. The new algorithm supersedes existing methods by enabling arbitrary threshold level. Second, we present a threshold design scheme to adaptively update the threshold level for optimal image reconstruction. We discuss the existence of a phase transition in determining the optimal threshold. Experimental results on tone-mapped high dynamic range images validates the effectiveness of the threshold scheme and the image reconstruction algorithm.

MATLAB Implementation (1.1MB)


  1. Omar A. Elgendy and Stanley H. Chan, ‘‘Optimal Threshold Design for Quanta Image Sensor’’, IEEE Trans. Computational Imaging, vol. 4, no. 1, Mar. 2018, pp. 99-111.

  1. Omar Elgendy and Stanley H. Chan, ‘‘Image reconstruction and threshold design for quanta image sensors’’, IEEE ICIP, pp.978-982, Phoenix, Arizona, Sep. 2016. (ICIP 2016 Best Paper Award)

Non-Iterative Image Reconstruction for QIS


Quanta image sensor (QIS) is a class of single photon imaging devices that measure light intensity using oversampled binary observations. Because of the stochastic nature of the photon arrivals, data acquired by QIS is a massive stream of random binary bits. The goal of image reconstruction is to recover the underlying image from these bits. In this paper, we present a non-iterative image reconstruction algorithm for QIS. Unlike existing reconstruction methods that formulate the problem from an optimization perspective, the new algorithm directly recovers the images through a pair of nonlinear transformations and an off-the-shelf image denoising algorithm. By skipping the usual optimization procedure, we achieve orders of magnitude improvement in speed and even better image reconstruction quality. We validate the new algorithm on synthetic datasets as well as real videos collected by 1-bit SPAD cameras.


  1. Stanley H. Chan, Omar Elgendy and Xiran Wang, ‘‘Images from bits: Non-iterative image reconstruction for quanta image sensors’’, MDPI Sensors Special Issue on Photon-Counting Image Sensors, vol. 16, no. 11, paper 1961, pp.1-21, Nov. 2016.

ADMM Image Reconstruction for QIS


Recent advances in materials, devices and fabrication technologies have motivated a strong momentum in developing solid-state sensors that can detect individual photons in space and time. It has been envisioned that such sensors can eventually achieve very high spatial resolutions as well as high frame rates. In this paper, we present an efficient algorithm to reconstruct images from the massive binary bit-streams generated by these sensors. Based on the concept of alternating direction method of multipliers (ADMM), we transform the computationally intensive optimization problem into a sequence of subproblems, each of which has efficient implementations in the form of polyphase-domain filtering or pixel-wise nonlinear mappings. Moreover, we reformulate the original maximum likelihood estimation as maximum a posterior estimation by introducing a total variation prior. Numerical results demonstrate the strong performance of the proposed method, which achieves several dB's of improvement in PSNR and requires a shorter runtime as compared to standard gradient-based approaches.


  1. Stanley H. Chan, and Yue M. Lu, ‘‘Efficient image reconstruction for giga-pixel quanta image sensors’’, IEEE GlobalSIP, pp. 312-316, Altanta, Georgia, Dec. 2014,