Skip to main content Accessibility help
×
Home

Checkerboard artifacts free convolutional neural networks

  • Yusuke Sugawara (a1), Sayaka Shiota (a1) and Hitoshi Kiya (a1)

Abstract

It is well-known that a number of convolutional neural networks (CNNs) generate checkerboard artifacts in both of two processes: forward-propagation of upsampling layers and backpropagation of convolutional layers. A condition for avoiding the artifacts is proposed in this paper. So far, these artifacts have been studied mainly for linear multirate systems, but the conventional condition for avoiding them cannot be applied to CNNs due to the non-linearity of CNNs. We extend the avoidance condition for CNNs and apply the proposed structure to typical CNNs to confirm whether the novel structure is effective. Experimental results demonstrate that the proposed structure can perfectly avoid generating checkerboard artifacts while keeping the excellent properties that CNNs have.

  • View HTML
    • Send article to Kindle

      To send this article to your Kindle, first ensure no-reply@cambridge.org is added to your Approved Personal Document E-mail List under your Personal Document Settings on the Manage Your Content and Devices page of your Amazon account. Then enter the ‘name’ part of your Kindle email address below. Find out more about sending to your Kindle. Find out more about sending to your Kindle.

      Note you can select to send to either the @free.kindle.com or @kindle.com variations. ‘@free.kindle.com’ emails are free but can only be sent to your device when it is connected to wi-fi. ‘@kindle.com’ emails can be delivered even when you are not connected to wi-fi, but note that service fees apply.

      Find out more about the Kindle Personal Document Service.

      Checkerboard artifacts free convolutional neural networks
      Available formats
      ×

      Send article to Dropbox

      To send this article to your Dropbox account, please select one or more formats and confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your <service> account. Find out more about sending content to Dropbox.

      Checkerboard artifacts free convolutional neural networks
      Available formats
      ×

      Send article to Google Drive

      To send this article to your Google Drive account, please select one or more formats and confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your <service> account. Find out more about sending content to Google Drive.

      Checkerboard artifacts free convolutional neural networks
      Available formats
      ×

Copyright

This is an Open Access article, distributed under the terms of the Creative Commons Attribution licence (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted re-use, distribution, and reproduction in any medium, provided the original work is properly cited.

Corresponding author

Corresponding author: Hitoshi Kiya Email: kiya@tmu.ac.jp

References

Hide All
1Krizhevsky, A.; Sutskever, I.; Hinton, G.E.: ImageNet Classification with Deep Convolutional Neural Networks, in Proc. Advances in Neural Information Processing Systems (NIPS), 2012, 10971105.
2He, K.; Zhang, X.; Ren, S.; Sun, J.: Deep Residual Learning for Image Recognition, in Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2016, 770778.
3Huang, G.; Liu, Z.; Maaten, L.; van der Weinberger, K.Q.: Densely Connected Convolutional Networks, in Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2017, 22612269.
4Noh, H.; Hong, S.; Han, B.: Learning Deconvolution Network for Semantic Segmentation, in Proc. IEEE Int. Conf. on Computer Vision (ICCV), 2015, 15201528.
5Shelhamer, E.; Long, J.; Darrell, T.: Fully Convolutional Networks for Semantic Segmentation. IEEE Trans. Pattern Anal. Mach. Intell., 39(4) (2017), 640651.
6Dong, C.; Loy, C.C.; He, K.; Tang, X.: Image Super-Resolution Using Deep Convolutional Networks. IEEE Trans. Pattern Anal. Mach. Intell., 38(2) (2016), 295307.
7Ledig, C., et al. : Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network, in Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2017, 105114.
8Radford, A.; Metz, L.; Chintala, S.: Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks, in arXiv preprint arXiv:1511.06434, 2015.
9Odena, A.; Dumoulin, V.; Olah, C.: Deconvolution and Checkerboard Artifacts. Distill, (2016). [Online]. Available: http://distill.pub/2016/deconv-checkerboard.
10Zeiler, M.D.; Taylor, G.W.; Fergus, R.: Adaptive Deconvolutional Networks for Mid and High Level Feature Learning, in Proc. IEEE Int. Conf. on Computer Vision (ICCV). 2011, 20182025.
11Shi, W., et al. : Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network, in Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR). 2016, 18741883.
12Dosovitskiy, A., et al. : FlowNet: Learning Optical Flow with Convolutional Networks, in Proc. IEEE Int. Conf. on Computer Vision (ICCV). 2015, 27582766.
13Johnson, J.; Alahi, A.; Li, F.: Perceptual Losses for Real-Time Style Transfer and Super-Resolution, in Proc. European Conf. on Computer Vision (ECCV), Springer. 2016, 694711.
14Aitken, A.P.; Ledig, C.; Theis, L.; Caballero, J.; Wang, Z.; Shi, W.: Checkerboard Artifact Free Sub-Pixel Convolution: A Note on Sub-Pixel Convolution, Resize Convolution and Convolution Resize, in arXiv preprint arXiv:1707.02937, 2017.
15Gao, H.; Yuan, H.; Wang, Z.; Ji, S.: Pixel Deconvolutional Networks, in arXiv preprint arXiv:1705.06820, 2017.
16Wojna, Z., et al. : The Devil is in the Decoder, in Proc. British Machine Vision Conf. (BMVC), 2017.
17Harada, Y.; Muramatsu, S.; Kiya, H.: Multidimensional Multirate Filter without Checkerboard Effects, in Proc. European Signal Processing Conf. (EUSIPCO). 1998, 18811884.
18Tamura, T.; Kato, M.; Yoshida, T.; Nishihara, A.: Design of Checkerboard-Distortion-Free Multidimensional Multirate Filters. IEICE Trans. Fundamentals Electron. Commun. Computer Sci., E81-A(8) (1998), 15981606.
19Harada, Y.; Muramatu, S.; Kiya, H.: Multidimensional Multirate Filter and Filter Bank without Checkerboard Effect. IEICE Trans. Fundamentals Electron. Commun. Computer Sci., E81-A(8) (1998), 16071615.
20Iwai, H.; Iwahashi, M.; Kiya, H.: Methods for Avoiding the Checkerboard Distortion Caused by Finite Word Length Error in Multirate System. IEICE Trans. Fundamentals Electron. Commun. Computer Sci., E93-A(3) (2010), 631635.
21Sugawara, Y.; Shiota, S.; Kiya, H.: Super-Resolution using Convolutional Neural Networks without Any Checkerboard Artifacts, in Proc. IEEE Int. Conf. on Image Processing (ICIP). 2018, 6670.
22Vincent, D.; Francesco, V.: A guide to convolution arithmetic for deep learning, in arXiv preprint arXiv:1603.07285, 2016.
23Shi, W.; Caballero, J.; Theis, L.; Huszar, F.; Aitken, A.; Ledig, C.: Is the deconvolution layer the same as a convolutional layer?, Wangin, Z.: arXiv:1609.07009, 2016.
24Dong, C.; Loy, C.C.; He, K.; Tang, X.: Learning a Deep Convolutional Network for Image Super-Resolution, in Proc. European Conf. on Computer Vision (ECCV), Springer, 2014, 184–199.
25Kim, J.; Lee, J.K.; Lee, K.M.: Accurate Image Super-Resolution Using Very Deep Convolutional Networks, in Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR). 2016, 16461654.
26Kim, J.; Lee, J.K.; Lee, K.M.: Deeply-Recursive Convolutional Network for Image Super-Resolution, in Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR). 2016, 16371645.
27Tai, Y.; Yang, J.; Liu, X.: Image Super-Resolution via Deep Recursive Residual Network, in Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2017, 27902798.
28Dong, C.; Loy, C.C.; Tang, X.: Accelerating the Super-Resolution Convolutional Neural Network, in Proc. European Conf. on Computer Vision (ECCV), Springer, 2016, 391407.
29Lai, W.; Huang, J.; Ahuja, N.; Yang, M.: Deep Laplacian Pyramid Networks for Fast and Accurate Super-Resolution, in Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2017, 58355843.
30Sajjadi, M.S.M.; Schölkopf, B.; Hirsch, M.: EnhanceNet: Single Image Super-Resolution through Automated Texture Synthesis, in Proc. IEEE Int. Conf. on Computer Vision (ICCV), 2017, 44914500.
31Zhang, K.; Zuo, W.; Zhang, L.: Learning a Single Convolutional Super-Resolution Network for Multiple Degradations, in Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2018.
32Zhang, Y.; Li, K.; Li, K.; Wang, L.; Zhong, B; Fu, Y.: Learning a Deep Convolutional Network for Image Super-Resolution, in Proc. European Conf. on Computer Vision (ECCV), Springer, 2018.
33Sugawara, Y.; Shiota, S.; Kiya, H.: A Parallel Computation Algorithm for Super-Resolution Methods Using Convolutional Neural Networks, in Proc. Asia Pacific Signal and Information Processing Association (APSIPA) Annual Summit and Conf., 2017, 11691173.
34Zhang, Y.; Li, K.; Li, K.; Wang, L.; Zhong, B.; Fu, Y.: Image Super-Resolution Using Very Deep ResidualChannel Attention Networks, in Proc. European Conf. on Computer Vision (ECCV), 2018.
35Yang, J.; Wright, J.; Huang, T.S.; Ma, Y.: Image Super-Resolution via Sparse Representation. IEEE Trans. Image. Process., 19(11) (2010), 28612873.
36Bevilacqua, M.; Roumy, A.; Guillemot, C.; Alberi-Morel, M.L.: Low-Complexity Single-Image Super-Resolution based on Nonnegative Neighbor Embedding, in Proc. British Machine Vision Conf. (BMVC), 2012.
37Zeyde, R.; Elad, M.; Protter, M.: On Single Image Scale-Up Using Sparse-Representations, in Proc. Curves and Surfaces, 2010, 711730.
38Kingma, D.P.; Ba, J.: Adam: A Method for Stochastic Optimization, in Proc. Int. Conf. on Learning Representations (ICLR), 2015.
39He, K.; Zhang, X.; Ren, S.; Sun, J.: Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification, in Proc. IEEE Int. Conf. on Computer Vision (ICCV), 2015, 10261034.
40Abadi, M.; Agarwal, A.; Barham, P., et al. .: TensorFlow: Large-Scale Machine Learning on Heterogeneous Systems, 2015, software available from tensorflow.org. [Online]. Available: http://tensorflow.org/.
41Blau, Y.; Michaeli, T.: The Perception-Distortion Tradeoff, in Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2018, 62286237.
42Krizhevsky, A.; Hinton, G.E.: Learning Multiple Layers of Features from Tiny Images, Master's thesis, Department of Computer Science, University of Toronto, 2009.

Keywords

Checkerboard artifacts free convolutional neural networks

  • Yusuke Sugawara (a1), Sayaka Shiota (a1) and Hitoshi Kiya (a1)

Metrics

Altmetric attention score

Full text views

Total number of HTML views: 0
Total number of PDF views: 0 *
Loading metrics...

Abstract views

Total abstract views: 0 *
Loading metrics...

* Views captured on Cambridge Core between <date>. This data will be updated every 24 hours.

Usage data cannot currently be displayed