Photon-Limited ImagingElectronic Imaging 2022 Short CourseSignal Processing for Photon-Limited Imaging Photon-Limited Non-Blind Deblurring Using Algorithm UnrollingYash Sanghvi, Abhiram Gnanasambandam, Stanley H. Chan Image deblurring in photon-limited conditions is ubiquitous in a variety of low-light applications such as photography, microscopy and astronomy. However, the presence of photon shot noise due to low-illumination and/or short exposure makes the deblurring task substantially more challenging than the conventional deblurring problems. In this paper we present an algorithm unrolling approach for the photon-limited deblurring problem by unrolling a Plug-and-Play algorithm for a fixed number of iterations. By introducing a three-operator splitting formation of the Plug-and-Play framework, we obtain a series of differentiable steps which allows the fixed iteration unrolled network to be trained end-to-end. The proposed algorithm demonstrates significantly better image recovery compared to existing state-of-the-art deblurring approaches. We also present a new photon-limited deblurring dataset for evaluating the performance of algorithms. Photon-Limited Object Detection using Non-local Feature Matching and Knowledge DistillationIEEE International Conference on Computer Vision Workshop, 2021 Robust object detection under photon-limited conditions is crucial for applications such as night vision, surveillance, and microscopy, where the number of photons per pixel is low due to a dark environment and/or a short integration time. While the mainstream “low-light” image enhancement methods have produced promising results that improve the image contrast between the foreground and background through advanced coloring techniques, the more challenging problem of mitigating the photon shot noise inherited from the random Poisson process remains open. In this paper, we present a photon-limited object detection framework by adding two ideas to state-of-the-art object detectors: 1) a space-time non-local module that leverages the spatial-temporal information across an image sequence in the feature space, and 2) knowledge distillation in the form of student-teacher learning to improve the robustness of the detector’s feature extractor against noise. Experiments are conducted to demonstrate the improved performance of the proposed method in comparison with state-of-the-art baselines. When integrated with the latest photon counting devices, the algorithm achieves more than 50% mean average precision at a photon level of 1 photon per pixel. Low-Light Demosaicking and Denoising for Small Pixels Using Learned Frequency SelectionIEEE Transactions on Computational Imaging, 2021 Low-light imaging is a challenging task because of the excessive photon shot noise. Color imaging in low-light is even more difficult because one needs to demosaick and denoise simultaneously. Existing demosaicking algorithms are mostly designed for well-illuminated scenarios, which fail to work with low-light. Recognizing the recent development of small pixels and low read noise image sensors, we propose a learning-based joint demosaicking and denoising algorithm for low-light color imaging. Our method combines the classical theory of color filter arrays and modern deep learning. We use an explicit carrier to demodulate the color from the input Bayer pattern image. We integrate trainable filters into the demodulation scheme to improve flexibility. We introduce a guided filtering module to transfer knowledge from the luma channel to the chroma channels, thus offering substantially more reliable denoising. Extensive experiments are performed to evaluate the performance of the proposed method, using both synthetic datasets and real data. Results indicate that the proposed method offers consistently better performance over the current state-of-the-art, across several standard evaluation metrics. HDR Imaging with Quanta Image Sensors: Theoretical Limits and Optimal ReconstructionIEEE Transactions on Computational Imaging, 2020 An earlier version of the paper is presented at International Image Sensor Workshop (IISW) 2019. High dynamic range (HDR) imaging is one of the biggest achievements in modern
photography. Traditional solutions to HDR imaging are designed for and
applied to CMOS image sensors (CIS). However, the mainstream one-micron CIS
cameras today generally have a high read noise and low frame-rate. These, in
turn, limit the acquisition speed and quality, making the cameras slow in the
HDR mode. In this paper, we propose a new computational photography technique
for HDR imaging. Recognizing the limitations of CIS, we use the Quanta Image
Sensor (QIS) to trade the spatial-temporal resolution with bit-depth. QIS is
a single-photon image sensor that has comparable pixel pitch to CIS but
substantially lower dark current and read noise. We provide a complete
theoretical characterization of the sensor in the context of HDR imaging, by
proving the fundamental limits in the dynamic range that QIS can offer and
the trade-offs with noise and speed. In addition, we derive an optimal
reconstruction algorithm for single-bit and multi-bit QIS. Our algorithm is
theoretically optimal for emph{all} linear reconstruction schemes based on
exposure bracketing. Experimental results confirm the validity of the theory
and algorithm, based on synthetic and real QIS data. Dynamic Low-light Imaging with Quanta Image SensorsEuropean Conference on Computer Vision (ECCV), 2020. Imaging in low light is difficult because the number of photons arriving at
the sensor is low. Imaging dynamic scenes in low-light environments is even
more difficult because as the scene moves, pixels in adjacent frames need to
be aligned before they can be denoised. Conventional CMOS image sensors (CIS)
are at a particular disadvantage in dynamic low-light settings because the
exposure cannot be too short lest the read noise overwhelms the signal. We
propose a solution using Quanta Image Sensors (QIS) and present a new image
reconstruction algorithm. QIS are single-photon image sensors with photon
counting capabilities. Studies over the past decade have confirmed the
effectiveness of QIS for low-light imaging but reconstruction algorithms for
dynamic scenes in low light remain an open problem. We fill the gap by
proposing a student-teacher training protocol that transfers knowledge from a
motion teacher and a denoising teacher to a student network. We show that
dynamic scenes can be reconstructed from a burst of frames at a photon level
of 1 photon per pixel per frame. Experimental results confirm the advantages
of the proposed method compared to existing methods. Image Classification in the Dark using Quanta Image SensorsEuropean Conference on Computer Vision (ECCV), 2020 Abstract: Color Filter Arrays Design for Quanta Image SensorsIEEE Trans. Computational Imaging, 2019. Quanta image sensor (QIS) is envisioned to be the next generation image
sensor after CCD and CMOS. In this paper, we discuss how to design color
filter arrays for QIS. Designing color filter arrays for small pixels such as
QIS is challenging because maximizing the light efficiency while suppressing
aliasing and crosstalk are conflicting tasks. We present an
optimization-based framework which unifies several mainstream color filter
array design methodologies and offers greater generality and flexibility.
Compared to existing methods, the new framework can simultaneously handle
luminance sensitivity, chrominance sensitivity, cross-talk, anti-aliasing,
manufacturability and orthogonality. Extensive experimental comparisons
demonstrate the effectiveness and generality of the framework. Megapixel photon-counting color imaging using quanta image sensorOSA Optics Express 2019 Quanta Image Sensor (QIS) is a single-photon detector designed for extremely
low light imaging conditions. Majority of the existing QIS prototypes are
monochrome based on single-photon avalanche diodes (SPAD). Passive color
imaging has not been demonstrated with single-photon detectors due to the
intrinsic difficulty of shrinking the pixel size and increasing the spatial
resolution while maintaining acceptable intra-pixel cross-talk. In this
paper, we present image reconstruction of the first color QIS with a
resolution of 1024-by-1024 pixels, supporting both single-bit and multi-bit
photon counting capability. Our color image reconstruction is enabled by a
customized joint demosaicing-denoising algorithm, leveraging truncated
Poisson statistics and variance stabilizing transforms. Experimental results
of the new sensor and algorithm demonstrate superior color imaging
performance for very low-light conditions with a mean exposure of as low as a
few photons per pixel in both real and simulated images. Photon-counting Imaging with Multi-bit Quanta Image SensorsInternational Image Sensor Workshop (IISW), 2019 Image Reconstruction for Quanta Image Sensors using Deep Neural NetworksIEEE ICASSP, 2019 Quanta Image Sensor (QIS) is a single-photon image sensor that oversamples
the light field to generate binary measurements. Its single-photon
sensitivity makes it an ideal candidate for the next generation image sensor
after CMOS. However, image reconstruction of the sensor remains a challenging
issue. Existing image reconstruction algorithms are largely based on
optimization. In this paper, we present the first deep neural network
approach for QIS image reconstruction. Our deep neural network takes the
binary bit stream of QIS as input, learns the nonlinear transformation and
denoising simultaneously. Experimental results show that the proposed network
produces significantly better reconstruction results compared to existing
methods. Optimal Threshold Design for Quanta Image Sensors(ICIP 2016 Best Paper Award) IEEE Trans. Computational Imaging, 2018 Also presented in IEEE ICIP 2016 MATLAB Implementation (1.1MB) Quanta Image Sensor (QIS) has been envisioned as a candidate solution for next generation image sensors. We provide two
new contributions to the signal processing aspects of QIS. First, we develop an image reconstruction algorithm to
recover the underlying images from the QIS data, which is a massive array of binarized Poisson random variables. The
new algorithm supersedes existing methods by enabling arbitrary threshold level. Second, we present a threshold design
scheme to adaptively update the threshold level for optimal image reconstruction. We discuss the existence of a phase
transition in determining the optimal threshold. Experimental results on tone-mapped high dynamic range images
validates the effectiveness of the threshold scheme and the image reconstruction algorithm. Non-Iterative Image Reconstruction
ADMM Image Reconstruction
|