Deep Networks for Image-to-Image Translation with Mux and Demux Layers

  • Hanwen LiuEmail author
  • Pablo Navarrete Michelini
  • Dan Zhu
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11133)


Image processing methods using deep convolutional networks have achieved great successes on quantitative and qualitative assessments in many tasks, such as super–resolution, style transfer and enhancement. Most of these solutions use many layers, many filters and complex architectures. It is difficult to implement them on mobile devices, e.g. smart phones, because of the limited resources. Many applications need to deploy these methods on mobile devices. But it is difficult because of limited resources. In this paper we present a lightweight end–to–end deep learning approach for image enhancement. To improve the performance, we present mux layer and demux layers, which could perform up–sampling and down–sampling by shuffling the pixels without losing any information of feature maps. For further higher performance, denseblocks are used in the models. To ensure the consistency of the output and input, we use weighted L1 loss to increase PSNR. To improve image quality, we use adversarial loss, contextual loss and perceptual loss as parts of the objective functions during training. And NIQE is used for validation to get the best parameters for perceptual quality. Experiments show that, compared to the state–of–the–art, our method could improve both the quantitative and qualitative assessments, as well as the performance. With this system, we get the third place in PIRM Enhancement–On–Smartphones Challenge 2018 (PIRM–EoS Challenge 2018).


Mux layer Demux layer Image enhancement Deep learning 

Supplementary material

478826_1_En_10_MOESM1_ESM.pdf (30.3 mb)
Supplementary material 1 (pdf 31067 KB)


  1. 1.
    Johnson, J., Alahi, A., Fei-Fei, L.: Perceptual losses for real-time style transfer and super-resolution. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9906, pp. 694–711. Springer, Cham (2016). Scholar
  2. 2.
    Dong, C., Loy, C.C., He, K., Tang, X.: Learning a deep convolutional network for image super-resolution. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8692, pp. 184–199. Springer, Cham (2014). Scholar
  3. 3.
    Ignatov, A., Kobyshev, N., Vanhoey, K., Timofte, R., Gool, L.V.: DSLR-quality photos on mobile devices with deep convolutional networks. In: Proceedings of the IEEE International Conference on Computer Vision (2017)Google Scholar
  4. 4.
    Kim, J., Lee, J.K., Lee, K.M.: Accurate image super resolution using very deep convolutional networks. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 1646–1654 (2016)Google Scholar
  5. 5.
    Mao, X., Shen, C., Yang, Y.B.: Image restoration using very deep convolutional encoder-decoder networks with symmetric skip connections. In: Advances in Neural Information Processing System, vol. 29, pp. 2802–2810 (2016)Google Scholar
  6. 6.
    Shi, W., et al.: Real-time single image and video super-resolution using an efficient sub-pixel convolutional netual networks. In: IEEE Conference on Computer Vision and Pattern Recognition (2016)Google Scholar
  7. 7.
    Ledig, C., et al.: Photo-realistic single image super-resolution using a generative adversarial network. In: IEEE Conference on Computer Vision and Pattern Recognition (2017)Google Scholar
  8. 8.
    Cai, B., Xu, X., Jia, K., Qing, C., Tao, D.: Dehazenet: an end-to-end system for single image haze removal. IEEE Trans. Image Process. 25(11), 5287–5298 (2016)MathSciNetCrossRefGoogle Scholar
  9. 9.
    Hradis, M., Kotera, J., Zemcik, P., Sroubek, F.: Convolutional neural networks for direct text deblurring. In: Proceedings of BMVV 2015. The British Machine Vision Association and Society for Pattern Recognition (2015)Google Scholar
  10. 10.
    Ling, Z., Fan, G., Wang, Y., Lu, X.: Learning deep transmission network for single image dehazing. In: IEEE Conference on Computer Vision and Pattern Recognition (2016)Google Scholar
  11. 11.
    Ren, W., Liu, S., Zhang, H., Pan, J., Cao, X., Yang, M.-H.: Single image dehazing via multi-scale convolutional neural networks. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9906, pp. 154–169. Springer, Cham (2016). Scholar
  12. 12.
    Zhang, X., Wu, R.: Fast depth image denoising and enhancement using a deep convolutional network. In: IEEE International Conference on Acoustics, Speech and Signal Processing, pp. 2499–2503 (2016)Google Scholar
  13. 13.
    Svoboda, P., Hradis, M., Barina, D., Zemcik, P.: Compression artifacts removal using convolutional neural networks. CoRR, sbs,1605.00366 (2016)Google Scholar
  14. 14.
    Iandola, F.N., Han, S., Moskewicz, M.W., Ashraf, K., Dally, W.J., Keutzer, K.: Squeezenet: alexnet-level accuracy with 50x fewer parameters and \(<\)0.5 mb model size. arXiv preprint arXiv:1602.07360 (2016)
  15. 15.
    Howard, A.G., et al.: Mobilenets: efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861 (2017)
  16. 16.
    Zhang, X., Zhou, X., Lin, M., Sun, J.: Shufflenet: an extremely efficient convolutional neural network for mobile devices. arXiv preprint arXiv:1707.01083 (2017)
  17. 17.
    He, K., Zhang, X., Ren, S., Sun J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)Google Scholar
  18. 18.
    Huang, G., Liu, Z., Maaten, L.V.D.: Densely connected convolutional networks. In: IEEE Conference on Computer Vision and Pattern Recognition (2017)Google Scholar
  19. 19.
    Mechrez, R., Talmi, I., Manor, L.Z.: The contextual loss for image transformation with non aligned data. arXiv preprint arXiv:1803.02077 (2018)
  20. 20.
    Aly, H.A., Dubois, E.: Image up-sampling using totalvariation regularization with a new observation model. IEEE Trans. Image Process. 14(10), 1647–1659 (2005)CrossRefGoogle Scholar
  21. 21.
    Mittal, A., Soundararajan, R., Bovik, A.C.: Making a “completely blind” image quality analyzer. IEEE Sig. Process. Lett. 20(3), 209–212 (2013)CrossRefGoogle Scholar
  22. 22.
    Proakis, J.G., Manolakis, D.G.: Digital Signal Processing, Prentice Hall International Editions. Pearson Prentice Hall, Englewood Cliffs (2007)Google Scholar
  23. 23.
    Vaidyanathan, P.P.: Multirate Systems And Filter Banks. Prentice Hall, Englewood Cliffs (1993)zbMATHGoogle Scholar
  24. 24.
    Mallat, S.: A Wavelet Tour of Signal Processing. Academic Press, San Diego (1998)zbMATHGoogle Scholar
  25. 25.
    Navarrete, P., Liu, H.: Convolutional networks with muxout layers as multi-rate systems for image upscaling. CoRR abs/1705.07772 (2017)Google Scholar
  26. 26.
    Navarrete, P., Zhang, L., He, J.: Upscaling with deep convolutional networks and muxout layers. In: GPU Technology Conference 2016, Poster Session, San Jose, CA, USA, May 2016Google Scholar
  27. 27.
    Navarrete, P., Liu, H.: Upscaling beyond superresolution using a novel deep learning system. In: GPU Technology Conference, March 2017Google Scholar
  28. 28.
    Timofte, R., et al.: NTIRE 2018 challenge on single image super-resolution: methods and results. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, June 2018Google Scholar
  29. 29.
    Blau, Y., Michaeli, T.: The perception-distortion tradeoff. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018Google Scholar
  30. 30.
    Goodfellow, I., Pouget-Abadie, J., Mirza, M., et al.: Generative adversarial nets. In: NIPS (2014)Google Scholar
  31. 31.
    Qin, Z., Zhang, Z., Zhang, S., Yu, H., Peng, Y.: Merging and evolution: improving convolutional neural networks for mobile applications. arXiv preprint arXiv:0803.09127 (2018)
  32. 32.
    Liu, H., Navarrete, P., Zhu, D.: Arsty-GAN: a style transfer system with improved quality, diversity and performance. In: International Conference on Pattern Recognition (2018)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  • Hanwen Liu
    • 1
    Email author
  • Pablo Navarrete Michelini
    • 1
  • Dan Zhu
    • 1
  1. 1.BOE Technology Group Co., LTD.BeijingPeople’s Republic of China

Personalised recommendations