ReconfigISP: RECONFIGURABLE CAMERA IMAGE PROCESSING PIPELINE
ICCV 2021
Paper
Abstract
Image Signal Processor (ISP) is a crucial component in digital cameras that transforms sensor signals into images for us to perceive and understand. Existing ISP designs always adopt a fixed architecture, e.g., several sequential modules connected in a rigid order. Such a fixed ISP architecture may be suboptimal for real-world applications, where camera sensors, scenes and tasks are diverse. In this study, we propose a novel Reconfigurable ISP (ReconfigISP) whose architecture and parameters can be automatically tailored to specific data and tasks. In particular, we implement several ISP modules, and enable backpropagation for each module by training a differentiable proxy, hence allowing us to leverage the popular differentiable neural architecture search and effectively search for the optimal ISP architecture. A proxy tuning mechanism is adopted to maintain the accuracy of proxy networks in all cases. Extensive experiments conducted on image restoration and object detection, with different sensors, light conditions and efficiency constraints, validate the effectiveness of ReconfigISP. Only hundreds of parameters need tuning for every task.
The optimal ISP pipeline for different tasks is different. For example, the accuracy of color is important for restoration, but it is not that important in object detection.
Differentiable
ISP Modules
We implement 22 common ISP algorithms, including denoising, demosaicing, white balancing, global tone mapping, and gamma correction. There are both traditional and deep approaches. To make all the algorithms differentiable, we use a proxy network to learn the behaviour of the non-differentiable algorithm. In the learning process, the input images and the parameters of the algorithm are randomly sampled.
The Framework of
ReconfigISP
We leverage DARTS for ISP architecture search. As the inputs of different tasks are diverse, the proxy network cannot always resemble the original algorithm. To address this challenge, we propose a proxy tuning approach, fine-tuning the proxy networks using the intermediate images during training. All the algorithms are fully connected at the beginning. The algorithms with low performance are gradually pruned. The best algorithm at each step is selected to construct an ISP. The parameters of these algorithms are further fine-tuned to get the final ISP pipeline.
Image
Restoration
Qualitative results of image restoration on the SID dataset. Compared to the Sony pipeline, our method yields cleaner results with less noise, and the exposure looks more natural. It is worth noting that our method achieves such compelling performance with only hundreds of parameters being tuned.
Efficiency
Constraints
Our method can search for the optimal ISP under different efficiency constraints, with loss multiplied by a function of latency. It can accelerate the ISP pipeline by choosing faster denoising algorithms.
ReconfigISP-Fast achieves 2 times speedup compared to ReconfigISP, at a slight expense of performance. ReconfigISP-Faster achieves up to 50 times speedup compared to ReconfigISP. See the paper for details.
Object
Detection
We collect a dataset using OnePlus smartphone for object detection. ReconfigISP outperforms other baselines without choosing any white-balancing algorithms.
Tips for
ISP Design
Paper
Citation
@InProceedings{yu2021reconfigisp,
title={ReconfigISP: Reconfigurable Camera Image Processing Pipeline},
author={Yu, Ke and Li, Zexian and Peng, Yue and Loy, Chen Change and Gu, Jinwei},
booktitle={ICCV},
year={2021}
}
