Review on Image Inpainting using Intelligence Mining Techniques

doi:10.37015/AUDT.2023.230007

Abstract

Abstract:

Objective Inpainting is a technique for fixing or removing undesired areas of an image.

Methods In present scenario, image plays a vital role in every aspect such as business images, satellite images, and medical images and so on.

Results and Conclusion This paper presents a comprehensive review of past traditional image inpainting methods and the present state-of-the-art deep learning methods and also detailed the strengths and weaknesses of each to provide new insights in the field.

Key words: Image inpainting; Deep learning; CNN; Wavelet transformations

V. Merin Shobi, MCA, MPhil , ME, F. Ramesh Dhanaseelan, MSc, MTech , PhD. Review on Image Inpainting using Intelligence Mining Techniques. Advanced Ultrasound in Diagnosis and Therapy, 2023, 7(4): 366-372.

Figures/Tables 2

Figure 1

Table 1

References 29

[1]	Omar Elharrouss, Noor Almaadeed, Somaya Al-Maadeed, Younes Akbari. Image inpainting: A review. Neural Processing Letters 2020; 51:2007-2028. doi: 10.1007/s11063-019-10163-0
[2]	Ozturk, Şaban. Image inpainting based compact hash code learning using modified U-Net. 2020 4th International Symposium on Multidisciplinary Studies and Innovative Technologies (ISMSIT) IEEE 2020
[3]	Ning Wang, Sihan Ma, Jingyuan Li, Yipeng Zhang, Lefei Zhang. Multistage attention network for image inpainting. Pattern Recognition 2020; 106:107448. doi: 10.1016/j.patcog.2020.107448
[4]	Qianna Wang, Yi Chen, Nan Zhang, Yanhui Gu. Medical image inpainting with edge and structure priors. Measurement 2021; 185:110027. doi: 10.1016/j.measurement.2021.110027
[5]	Prutsch, Alexander, Antonio Pepe, Jan Egger. Design and development of a web-based tool for inpainting of dissected aortae in angiography images. 2020. arXiv preprint arXiv:2005.02760.
[6]	Ziheng Li, Ailong Cai, Linyuan Wang, Wenkun Zhang, Chao Tang, Lei Li, et al. Promising generative adversarial network based sinogram inpainting method for ultra-limited-angle computed tomography imaging. Sensors 2019; 19:3941. doi: 10.3390/s19183941
[7]	Minh-Trieu Tran, Soo-Hyung Kim, Hyung-Jeong Yang, Guee-Sang Lee. Multi-task learning for medical image inpainting based on organ boundary awareness. Applied Sciences 2021; 11:4247. doi: 10.3390/app11094247
[8]	Angel Torrado-Carvajal, Daniel Albrecht S, Jeungchan Lee, Ovidiu Andronesi C, Eva-Maria Ratai, Vitaly Napadow, et al. Inpainting as a technique for estimation of missing voxels in brain imaging. Annals of biomedical engineering 2021; 49:345-353. doi: 10.1007/s10439-020-02556-3
[9]	Siqi Zhang, Lei Wang, Jie Zhang, Ling Gu, Xiran Jiang, Xiaoyue Zhai, et al. Consecutive context perceive generative adversarial networks for serial sections inpainting. IEEE Access 2020; 8:190417-190430. doi: 10.1109/Access.6287639
[10]	Yaqiong Chai, Botian Xu, Kangning Zhang, Natasha Lepore, John Wood. MRI restoration using edge-guided adversarial learning. IEEE Access 2020; 8:83858-83870. doi: 10.1109/access.2020.2992204 pmid: 33747672
[11]	Ding, Ding, Sundaresh Ram, Jeffrey Rodríguez J. Image inpainting using nonlocal texture matching and nonlinear filtering. IEEE Transactions on Image Processing 2018; 28:1705-1719. doi: 10.1109/TIP.2018.2880681
[12]	Xiuhong Yang, Baolong Guo, Zhaolin Xiao, Wei Liang. Improved structure tensor for fine-grained texture inpainting. Signal Processing: Image Communication 2019; 73:84-95. doi: 10.1016/j.image.2018.02.006
[13]	Bertalmío M, Sapiro G, Caselles V, Ballester C. Image Inpainting. Proceedings of the 27th annual conference on Computer graphics and interactive techniques-SIGGRAPH '00. Proceedings of SIGGRAPH 2000, New Orleans, USA July 2000:417-424.
[14]	Anupama Sanjay Awati, Meenakshi Patil R. Object removal using modified directional median filtering for digital image inpainting. IJSTM 2015; 4:2394-1537.
[15]	Chen M, Liu Z. EDBGAN: Image inpainting via an edge-aware dual branch generative adversarial network in IEEE Signal Processing Letters 2021; 28:842-846.
[16]	Chen Y, Hu H. An improved method for semantic image inpainting with GANs: progressive inpainting. Neural Processing Letters 2019; 49:1355-1367. doi: 10.1007/s11063-018-9877-6
[17]	Chuanxia Zheng, Tat-Jen Cham, Jianfei Cai. Pluralistic image completion. CVPR 2019; 6:1438-1447.
[18]	Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, Alexei Efros A. Context encoders: feature learning by inpainting. in:Proc. CVPR 2016; 3:2536-2544.
[19]	Lahiri A, Jain AK, Agrawal S, Mitra P, Biswas PK. Prior guided GAN based semantic inpainting. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) 2020
[20]	Ms. Pooja Jain, An efficient algorithm for the exemplar based image inpainting. IJCEM 2015;2. ISSN:23489510.
[21]	Karim Armanious, Youssef Mecky, Sergios Gatidis, Bin Yang. Adversarial inpainting of medical image modalities. ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) IEEE 2019.
[22]	Huaming Liu, Xuehui Bi, Guanming Lu, Weilan Wang. Exemplar-based image inpainting with multi-resolution information and the graph cut technique. IEEE Access 2019; 7:101641-101657. doi: 10.1109/ACCESS.2019.2931064
[23]	Jingyuan Li, Fengxiang He, Lefei Zhang, Bo Du, Dacheng Tao. Progressive reconstruction of visual structure for image inpainting. Proceedings of the IEEE/CVF International Conference on Computer Vision 2019.
[24]	Jingyuan Li, Ning Wang, Lefei Zhang, Bo Du, Dacheng Tao. Recurrent feature reasoning for image inpainting. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition 2020.
[25]	Siqi Zhang, Lei Wang, Jie Zhang, Ling Gu, Xiran Jiang, Xiaoyue Zhai, et al. Consecutive context perceive generative adversarial networks for serial sections inpainting. IEEE Access 2020; 8:190417-190430. doi: 10.1109/Access.6287639
[26]	Bao Nguyen, Adam Feldman, Sarath Bethapudi, Andrew Jennings, Chris Willcocks G. Unsupervised region-based anomaly detection in brain mri with adversarial image inpainting. 2021 IEEE 18th International Symposium on Biomedical Imaging (ISBI). IEEE 2021.
[27]	Siavelis, Panagiotis-Rikarnto, Nefeli Lamprinou, Emmanouil Psarakis Z. An improved GAN semantic image inpainting. International Conference on Advanced Concepts for Intelligent Vision Systems. Springer, Cham 2020.
[28]	Maayan Frid-Adar, Idit Diamant, Eyal Klang, Michal Amitai, Jacob Goldberger, Hayit Greenspan. GAN-based synthetic medical image augmentation for increased CNN performance in liver lesion classification. Neurocomputing 2018; 321:321-331. doi: 10.1016/j.neucom.2018.09.013
[29]	Karim Armanious, Chenming Jiang, Sherif Abdulatif, Thomas Küstner, Sergios Gatidis, Bin Yang. Unsupervised medical image translation using cycle-MedGAN. 2019 27th European Signal Processing Conference (EUSIPCO). IEEE 2019.

Inpainting techniques	Advantages	Disadvantages
Image Restoration using Inpainting	Inpainted images with good resolution	More time is required for the computation
Texture Synthesis based Image Inpainting	Perform well in approximating textures	Difficulty in handling natural images
Hybrid Exemplar-Based Image Inpainting Algorithm	Produced good result for all digital images	If more than one pixel patch with same priority found then ambiguity is occurred
Digital Image Inpainting Using Patch Priority Based Method	Remove the large object and blur region	Not able to recover the satellite images
Modified Image Exemplar-Based Inpainting	Patch filling provides Speed efficiency, texture synthesis accuracy and propagation with maximum accuracy in linear structure	Slow, improvements needed in performance and video Inpainting
Object Removal Using Modified Directional Median Filtering For Digital Image Inpainting	Fast and simple algorithm is defined. The image recovered closely resemble with original image	Cannot used for non-homogenous images
Exemplar Based Image Inpainting	Patch founding is efficient to recover image	Extra computation time is required to find the patch and because of that, more time required for completion of image
Wavelet Transform based Inpainting	Utilizes inter and intra scale dependency to maintain image structure and texture quality using Wavelet Transform	Mask for regions are defined manually
Semi-Automatic Inpainting	Simple dynamic programming can be used to derive the optimal answer	Take minutes to hours to complete depending on the size of the Inpainting area
Fast Inpainting Technique	Takes less time to inpaint an image	Results in blur effect in image