Email this article IMAGE INPAINTING FOR ARBITRARY HOLES USING CUSTUMIZED RESIDUAL BLOCK ARCHITECTURE WITH PARTIAL CONVOLUTIONS
About this article
Received: 11/09/19 Revised: 18/09/19 Published: 03/10/19Authors
1. Le Dinh Nghiep
, Hong Duc University2. Pham Viet Binh, University of Information and Communication Technology - TNU
3. Do Nang Toan, Institute of Information Technology - VNU
4. Hoang Van Thi, Thanh Hoa Department of Education and Training
Abstract
Recently, learning-based algorithms for image inpainting achieve remarkable progress dealing with squared or regular holes. However, they still fail to generate plausible textures inside damaged area because there lacks surrounding information. In this paper, motivated by the residual learning algorithm which aims to learn the missing information in corrupted regions, thus facilitating feature integration and texture prediction we propose Residual Partial Convolution network (RBPConv) based on encoder and decoder U-net architecture to maintain texture while filling not only regular regions but also random holes. Both qualitative and quantitative experimental demonstrate that our model can deal with the corrupted regions of arbitrary shapes and performs favorably against previous state-of-the-art methods.
Keywords
Full Text:
PDF (Tiếng Việt)References
[1]. Bertalmio, M., Vese, L., Sapiro, G. and Osher, S., "Simultaneous structure and texture image inpainting," IEEE transactions on image processing, Vol. 12, No. 8, pp. 882-889, 2003.
[2]. Liu, D., Sun, X., Wu, F., Li, S., and Zhang, Y., "Image compression with edge-based inpainting," IEEE Transactions on Circuits and Systems for Video Technology, Vol. 17, No. 10, pp. 1273-1287, 2007.
[3]. Criminisi, A., Perez, P., and Toyama, K., "Object removal by exemplar-based inpainting," IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Vol. 2, pp. 721-728, 2003.
[4]. Drori, I., Cohen-Or, D., and Yeshurun, H., "Fragment-based image completion," TOG, Vol. 22, No. 3, pp. 303-312, 2003.
[5]. N. Komodakis, "Image completion using global optimization," CVPR, pp. 442–452, 2006.
[6]. Barnes, C., Shechtman, E., Finkelstein, A., Goldman, D. B., "Patchmatch: A randomized correspondence algorithm for structural image editing," ACM Transactions on Graphics-TOG, Vol. 28, No. 3, 2009.
[7]. Pathak, D., Krahenbuhl, P., Donahue, J., Darrell, T., Efros, A.A., "Context encoders: Feature learning by inpainting," Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2536–2544, 2016.
[8]. Yan, Z., Li, X., Li, M., Zuo, W., and Shan, S., "Shift-net: Image inpainting via deep feature rearrangement.," arXiv preprint arXiv:1801.09392, 2018.
[9]. Yang, C., Lu, X., Lin, Z., Shechtman, E., Wang, O., Li, H, "High-resolution image inpainting using multi-scale neural patch synthesis," The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Vol. 1, pp. 3, 2017.
[10]. Iizuka, S., Simo-Serra, E., Ishikawa, H., "Globally and locally consistent image completion," ACM Transactions on Graphics (TOG), Vol. 36, No. 4, 2017.
[11]. Yu, J., Lin, Z., Yang, J., Shen, X., Lu, X., Huang, T.S., "Generative image inpainting with contextual attention," arXiv preprint arXiv:1801.07892, 2018.
[12]. Sundaram, N., Brox, T., and Keutzer, K., "Dense point trajectories by gpu-accelerated large displacement optical flow," European conference on computer vision, pp. 438-451, 2010.
[13]. Liu, G., Reda, F. A., Shih, K. J., Wang, T.-C., Tao, A., and Catanzaro, B., "Image inpainting for irregular holes using partial convolutions," arXiv preprint arXiv:1804.07723, 2018.
[14]. Bertalmio, M., Sapiro, G., Caselles, V., and Ballester, C., "Image inpainting," Proceedings of the 27th annual conference on Computer graphics and interactive techniques. ACM Press/Addison-Wesley Publishing Co, p. 417–424, 2000.
[15]. Darabi, S., Shechtman, E., Barnes,C., Goldman, D. B., and Sen, P., "Image melding: Combining inconsistent images using patch-based synthesis," ACM Trans. Graph, 2012.
[16]. Huang, J., Kang, S. B., Ahuja, N. and Kopf, J., "Image completion using planar structure guidance," ACM Transactions on graphics (TOG), 2014.
[17]. Sun, J., Yuan, L., Jia, J., Shum, H., "Image completion with structure propagation," ACM Transactions on Graphics (ToG), pp. 861–868, 2005.
[18]. Xu, Z., and Sun, J., "Image inpainting by patch propagation using patch sparsity," IEEE transactions on image processing, pp. 1153–1165, 2010.
[19]. Liu, P., Qi, X., He, P., Li, Y., Lyu, M. R., and King, I., "Semantically consistent image completion with fine-grained details," arXiv preprint arXiv:1711.09345, 2017.
[20]. Yeh, R. A., Chen, C., Lim, T. Y., Schwing, A. G., HasegawaJohnson, M., and Do,M. N., "Semantic image inpainting with deep generative models," In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5485–5493, 2017.
[21]. Radford, A., Metz, L., and Chintala, S., "Unsupervised representation learning with deep convolutional generative adversarial networks," arXiv preprint arXiv:1511.06434, 2015.
[22]. Isola, P., Zhu, J., Zhou, T., and Efros, A. A., "Image-to-Image Translation with Conditional Adversarial Networks," Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1125-1134, 2017.
[23]. Nazeri, K., Eric, Ng., Joseph, T., Qureshi, F., and Ebrahimi, M., "EdgeConnect: Generative Image Inpainting with Adversarial Edge Learning," arXiv preprint arXiv:1901.00212, 2019.
[24]. Xiong, W., Lin, Z., Yang, J., Lu, X., Barnes, C., and Luo, J., "Foreground-aware Image Inpainting," arXiv preprint arXiv:1901.05945, 2019.
[25]. Huy V. V., Ngoc Q. K. D., and Pérez,P., "Structural Inpainting," Proceedings of the 26th ACM International Conference on Multimedia (MM ’18), pp. 1948–1956, 2018.
[26]. Zhang, H., Hu, Z., Luo, C., Zuo, W., and Wang, M., "Semantic Image Inpainting with Progressive Generative Networks," ACM Multimedia Conference on Multimedia Conference, pp. 1939–1947, 2018.
[27]. He, K., Zhang, X., Ren,S., and Sun, J., "Deep residual learning for image recognition," Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
[28]. Zeiler, M. D., and Fergus, R., "Visualizing and understanding convolutional networks," arXiv:1311.2901, 2013.
[29] Ronneberger, O., Fischer, P., and Brox, T., "U-net: Convolutional networks for biomedical image segmentation," International Conference on Medical image computing and computer-assisted intervention, pp. 234–241, 2015.
[30]. Johnson, J., Alahi, A., and Fei-Fei, L., "Perceptual losses for real-time style transfer and super-resolution," European Conference on Computer Vision, p. 694–711, 2016.
[31]. Mahajan, K. S., Vaidya, M. B., "Image in Painting Techniques: A survey," IOSR Journal of Computer Engineering, vol. 5, no. 4, pp. 45-49, 2012.
[32]. Kingma, D. P., Ba, J. L.: Adam, "A method for stochastic optimization," international conference on learning representations , 2015.
[33]. Zheng, C., Cham,T., and Cai, J., "Pluralistic Image Completion," CoRR abs/1903.04227, 2019.
[34]. Zhou, W., Bovik, A. C., Sheikh, H. R., and Simoncelli E. P., "Image Qualifty Assessment: From Error Visibility to Structural Similarity.," IEEE Transactions on Image Processing, vol. 13, no. 4, p. 600–612, 2004.
[35]. Gonzalez, R., and Wood, R. , "Digital Image Processing," Pearson Edn, 2009.Refbacks
- There are currently no refbacks.