×

Issues with common assumptions about the camera pipeline and their impact in HDR imaging from multiple exposures. (English) Zbl 1434.68627

Summary: Multiple-exposure approaches for high dynamic range (HDR) image generation share a set of building assumptions: that color channels are independent and that the camera response function (CRF) remains constant while changing the exposure. The first contribution of this paper is to highlight how these assumptions, which were correct for film photography, do not hold in general for digital cameras. As a consequence, results of multiexposure HDR methods are less accurate, and when tone-mapped they often present problems like hue shifts and color artifacts. The second contribution is to propose a method to stabilize the CRF while coupling all color channels, which can be applied to both static and dynamic scenes, and yield artifact-free results that are more accurate than those obtained with state-of-the-art methods according to several image metrics.

MSC:

68U10 Computing methodologies for image processing
62H35 Image analysis in multivariate analysis
65D18 Numerical aspects of computer graphics, image analysis, and computational geometry
94A08 Image processing (compression, reconstruction, etc.) in information and communication theory
PDFBibTeX XMLCite
Full Text: DOI

References:

[1] F. Banterle, A. Artusi, K. Debattista, and A. Chalmers, Advanced High Dynamic Range Imaging: Theory and Practice, A. K. Peters, Ltd., Natick, MA, 2011.
[2] M. Bertalmío, Image Processing for Cinema, Chapman & Hall/CRC Mathematical and Computational Imaging Sciences Series, CRC Press, Taylor & Francis, Boca Raon, FL, 2014. · Zbl 1285.68002
[3] S. Bianco, A. Bruna, F. Naccari, and R. Schettini, Color space transformations for digital photography exploiting information about the illuminant estimation process, J. Opt. Soc. Amer., 29 (2012), pp. 374-384.
[4] M. Chang, H. Feng, Z. Xu, and Q. Li, Robust ghost-free multiexposure fusion for dynamic scenes, J. Electron. Imaging, 27 (2018), 3.
[5] P. E. Debevec and J. Malik, Recovering high dynamic range radiance maps from photographs, in Proceedings of the 24th Annual Conference on Computer Graphics and Interactive Techniques, SIGGRAPH, New York, 1997, ACM Press/Addison-Wesley, New York, pp. 369-378.
[6] F. Drago, K. Myszkowski, T. Annen, and N. Chiba, Adaptive logarithmic mapping for displaying high contrast scenes, in Computer Graphics Forum, Wiley Online Library, New York, 2003, pp. 419-426.
[7] M. D. Fairchild, The HDR photographic survey, Color and Imaging Conference, 2007 (2007), pp. 233-238.
[8] S. Ferradans, M. Bertalmío, E. Provenzi, and V. Caselles, An analysis of visual adaptation and contrast perception for tone mapping, IEEE Trans. Pattern Anal. Mach. Intell., 33 (2011), pp. 2002-2012.
[9] R. Gil Rodríguez, J. Vazquez-Corral, and M. Bertalmío, The intrinsic error of exposure fusion for HDR imaging, and a way to reduce it, in Proceedings of the British Machine Vision Conference (BMVC), BMVA Press, Swansea, UK, 2015, 126.
[10] M. D. Grossberg and S. K. Nayar, High dynamic range from multiple images: Which exposures to combine, in Proceedings International Conference on Computer Vision Workshop (ICCVW) on Color and Photometric Methods in Computer Vision, Nice, France, 2003, IEEE, Piscataway, NJ.
[11] B. Guthier, S. Kopf, M. Wichtlhuber, and W. Effelsberg, Parallel implementation of a real-time high dynamic range video system, Integr. Computer Aided Eng., 21 (2014), pp. 189-202.
[12] P. Hanhart, M. Bernardo, M. Pereira, A. Pinheiro, and T. Ebrahimi, Benchmarking of objective quality metrics for HDR image quality assessment, EURASIP J. Image Video Process., 2015 (2015).
[13] J. Hu, O. Gallo, K. Pulli, and X. Sun, HDR deghosting: How to deal with saturation?, in IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Washington, DC, 2013, IEEE, Piscataway, NJ, pp. 1163-1170.
[14] S. B. Kang, M. Uyttendaele, S. Winder, and R. Szeliski, High dynamic range video, ACM Trans. Graphics, 22 (2003), pp. 319-325.
[15] S. J. Kim and M. Pollefeys, Robust radiometric calibration and vignetting correction, IEEE Trans. Pattern Anal. Mach. Intell., 30 (2008), pp. 562-576.
[16] J. Kronander, S. Gustavson, and J. Unger, Real-time HDR video reconstruction for multi-sensor systems, in ACM SIGGRAPH Posters, ACM, New York, 2012, 65.
[17] C. Lee, Y. Li, and V. Monga, Ghost-free high dynamic range imaging via rank minimization, IEEE Signal Process. Lett., 21 (2014), pp. 1045-1049.
[18] J.-Y. Lee, Y. Matsushita, B. Shi, I. S. Kweon, and K. Ikeuchi, Radiometric calibration by rank minimization, IEEE Trans. Pattern Anal. Mach. Intell., 35 (2013), pp. 144-156.
[19] I. Lissner, J. Preiss, P. Urban, M. S. Lichtenauer, and P. Zolliker, Image-difference prediction: From grayscale to color, IEEE Trans. Image Process., 22 (2013), pp. 435-446. · Zbl 1373.94254
[20] D. G. Lowe, Object recognition from local scale-invariant features, in IEEE International Conference on Computer Vision (ICCV), Washington, DC, 1999, IEEE, Piscataway, NJ, Vol. 2, pp. 1150-1157.
[21] K. Ma, H. Li, H. Yong, Z. Wang, D. Meng, and L. Zhang, Robust multi-exposure image fusion: A structural patch decomposition approach, IEEE Trans. Image Process., 26 (2017), pp. 2519-2532. · Zbl 1409.94433
[22] Z. Mai, H. Mansour, R. Mantiuk, P. Nasiopoulos, R. K. Ward, and W. Heidrich, Optimizing a tone curve for backward-compatible high dynamic range image and video compression, IEEE Trans. Image Process., 20 (2011), pp. 1558-1571. · Zbl 1372.94171
[23] S. Mann, Comparametric equations with practical applications in quantigraphic image processing, IEEE Trans. Image Process., 9 (2000), pp. 1389-1406. · Zbl 0968.68177
[24] S. Mann and R. W. Picard, On being “undigital” with digital cameras: Extending dynamic range by combining differently exposed pictures, in Proceedings of IS&T, Springfield, VA, 1995, pp. 442-448.
[25] R. Mantiuk, S. Daly, and L. Kerofsky, Display adaptive tone mapping, in ACM Trans. Graphics, Vol. 27, ACM, New York, 2008, 68.
[26] R. Mantiuk, K. J. Kim, A. G. Rempel, and W. Heidrich, HDR-VDP-\(2\): A calibrated visual metric for visibility and quality predictions in all luminance conditions, ACM Trans. Graphics, 30 (2011), 40.
[27] I. Merianos and N. Mitianoudis, Multiple-exposure image fusion for HDR image synthesis using learned analysis transformations, J. Imaging, 5 (2019), 32.
[28] T. Mitsunaga and S. Nayar, Radiometric self calibration, in IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Vol. 1, 1999, Ft. Collins, CO, IEEE, Piscataway, NJ, pp. 374-380.
[29] T.-H. Oh, J.-Y. Lee, Y.-W. Tai, and I. S. Kweon, Robust high dynamic range imaging by rank minimization, IEEE Trans. Pattern Anal. Mach. Intell., 37 (2015), pp. 1219-1232.
[30] M. A. Robertson, S. Borman, and R. L. Stevenson, Estimation-theoretic approach to dynamic range enhancement using multiple exposures, J. Electron. Imaging, 12 (2003), pp. 219-228.
[31] P. Sen and C. Aguerrebere, Practical high dynamic range imaging of everyday scenes: Photographing the world as we see it with our own eyes, IEEE Signal Process. Mag., 33 (2016), pp. 36-44.
[32] P. Sen, N. K. Kalantari, M. Yaesoubi, S. Darabi, D. B. Goldman, and E. Shechtman, Robust patch-based HDR reconstruction of dynamic scenes, ACM Trans. Graphics, 31 (2012), 203.
[33] G. Sharma, W. Wu, and E. N. Dalal, The CIEDE \(2000\) color-difference formula: Implementation notes, supplementary test data, and mathematical observations, Color Res. Appl., 30 (2005), pp. 21-30.
[34] M. D. Tocci, C. Kiser, N. Tocci, and P. Sen, A versatile HDR video production system, ACM Trans. Graphics, 30 (2011), 41.
[35] Y. Tsin, V. Ramesh, and T. Kanade, Statistical calibration of CCD imaging process, in IEEE International Conference on Computer Vision (ICCV), Vol. 1, Vancouver, Canada, 2001, IEEE, Piscataway, NJ, pp. 480-487.
[36] J. Vazquez-Corral and M. Bertalmío, Color stabilization along time and across shots of the same scene, for one or several cameras of unknown specifications, IEEE Trans. Image Process., 23 (2014), pp. 4564-4575. · Zbl 1374.94378
[37] Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, Image quality assessment: From error visibility to structural similarity, IEEE Trans. Image Process., 13 (2004), pp. 600-612.
[38] D. Zhang and X. Wu, Color demosaicking via directional linear minimum mean square-error estimation, IEEE Trans. Image Process., 14 (2005), pp. 2167-2178.
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. In some cases that data have been complemented/enhanced by data from zbMATH Open. This attempts to reflect the references listed in the original paper as accurately as possible without claiming completeness or a perfect matching.