Skip to main content

Deep-trained illumination-robust precision positioning for real-time manipulation of embedded objects

The International Journal of Advanced Manufacturing Technology volume 111, pages 2259–2276 (2020)Cite this article

Abstract

Automated manipulation guided by a precision visual positioning system is realized based on a convolutional neural network (ConvNet) trained on coordinate-explicit target images. Whereas ConvNet has exhibited superior illumination adaptability, it is still possible that high-contrast illumination effects may reduce the success rate of visually guided manipulation in the factory. In this paper, an automated training data generation scheme is proposed to enhance the performance of the visual positioning ConvNet under high-contrast shadows cast by other objects. By collecting dozens of illumination patterns, illumination templates are created by the image-to-image translation GAN (pix2pix GAN). The templates are then applied to the basis photo of the target object to spawn multiple virtual images which vastly enrich the illumination diversity of the training set for the visual positioning ConvNet. Experiments were conducted to verify the effectiveness of the pix2pix GAN illumination generation module. Experimental results also showed that the positioning accuracy augmented by the illumination module reaches above 85%, while without the illumination-augmented training, the positioning accuracy is below 15%. Experiments on automated visual manipulation with a 5-DOF manipulator also confirmed the feasibility of adopting the proposed framework for real-time operations. With the illumination-augmented training, the manipulation success rate is above 90%; without it, the success rate is less than 40%. We provide the data used for training pix2pix GAN to generate the illumination templates at https://github.com/ntutindustry40/OneShot-II.

This is a preview of subscription content, access via your institution.

Access options

Buy single article

Instant access to the full article PDF.

USD 39.95

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19

References

  1. Li CHG, Chang YM (2019) Automated visual positioning and precision placement of a workpiece using deep learning. Int J Adv Manuf Technol 104:4527–4538

    Article  Google Scholar 

  2. Chang YM, Li CHG, Hong YF (2019) Real-time object coordinate detection and manipulator control using rigidly trained convolutional neural networks. In: 2019 IEEE international conference on automation science and engineering (CASE), pp 1347–1352

  3. Jain R, Kasturi R, Schunck BG (1995) Machine vision, McGraw-Hill

  4. Yoo H, Yang U, Sohn K (2013) Gradient-enhancing conversion for illumination-robust lane detection. IEEE Trans Intell Transp 14(3):1083–1094

    Article  Google Scholar 

  5. Agunbiade OY, Ngwira SM, Zuva T, Akanbi Y (2016) Improving ground detection for unmanned vehicle systems in environmental noise scenarios. Int J Adv Manuf Technol 84:2719–2727

    Article  Google Scholar 

  6. Martínez SS, García AS, Estévez EE, Ortega JG, García JG (2019) 3D object recognition for anthropomorphic robots performing tracking tasks. Int J Adv Manuf Technol 104:1403–1412

    Article  Google Scholar 

  7. Hsu Q, Ngo N, Ni R (2019) Development of a faster classification system for metal parts using machine vision under different lighting environments. Int J Adv Manuf Technol 100:3219–3235

    Article  Google Scholar 

  8. Krizhevsky A, Sutskever I, Hinton GE (2012) ImageNet classification with deep convolutional neural networks. In: Advances in neural information processing systems (NIPS), pp 1–9. http://papers.nips.cc/paper/4824-imagenet-classification-with-deep-convolutional-neural-networ. Accessed 10 Oct 2020

  9. Chen Y, Shen Y, Liu X, Zhong B (2015) 3D object tracking via image sets and depth-based occlusion detection. Signal Process ll2:146–153

    Article  Google Scholar 

  10. Girshick R, Donahue J, Darrell T, Malik J (2014) Rich feature hierarchies for accurate object detection and semantic segmentation. In: IEEE conference on computer vision and pattern recognition (CVPR), pp 580–587

  11. Sunderhauf N, Shirazi S, Dayoub F, Upcroft B, Milford M (2015) On the performance of ConvNet features for place recognition. In: IEEE international conference on intelligent robots and systems (IROS), pp 4297–4304. https://doi.org/10.1109/IROS.2015.7353986

  12. Girshick R (2015) Fast R-CNN. In: the IEEE international conference on computer vision (ICCV), pp 1440–1448

  13. Ren S, He K, Girshick R, Sun J (2015) Faster R-CNN: towards real-time object detection with region proposal networks. In: advances in neural information processing systems (NIPS), pp 1–9. http://papers.nips.cc/paper/5638-faster-r-cnn-towards-real-time-object-detection-with-region-proposal-networks. Accessed 10 Oct 2020

  14. Redmon J, Divvala S, Girshick R, Farhadi A (2016) You only look once: unified, real-time object detection. In: IEEE conference on computer vision and pattern recognition (CVPR), pp 779–788

  15. Oquab M, Bottou L, Laptev I, Sivic J (2015) Is object localization for free? -Weakly-supervised learning with convolutional neural networks. In: IEEE conference on computer vision and pattern recognition (CVPR), pp 685–694

  16. Goodfellow L, Pouget-Abadie J, Mirza M, Xu B, Warde-Farley D, Ozair S, Courville A, Bengio Y (2014) Generative adversarial networks. In: Advances in neural information processing systems (NIPS), pp 1–9, http://papers.nips.cc/paper/5423-generative-adversarial-nets. Accessed 10 Oct 2020

  17. Isola P, Zhu J, Zhou T, Efros AA (2017) Image-to-image translation with conditional adversarial networks. In: IEEE conference on computer vision and pattern recognition (CVPR), pp 1125–1134

  18. Huang Y, Chang Y, Li CHG (2019) Illumination-robust object coordinate detection by adopting pix2pix GAN for training image generation. In: International conference on technologies and applications of artificial intelligence (TAAI), pp 1–6. https://doi.org/10.1109/TAAI48200.2019.8959837

  19. Guo R, Dai Q, Hoiem D (2013) Paired regions for shadow detection and removal. IEEE T Pattern Anal 12:2956–2967

    Article  Google Scholar 

  20. Khan SH, Bennamoun M, Sohel F, Togneri R (2016) Automatic Shadow detection and removal from a single image. IEEE T Pattern Anal 3:431–446

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Graduate Institute of Manufacturing Technology, National Taipei University of Technology, Taipei, 10608, ROC, Taiwan

    Chih-Hung G. Li & Yi-Hao Huang

Authors
  1. Chih-Hung G. Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yi-Hao Huang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Chih-Hung G. Li.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Li, CH.G., Huang, YH. Deep-trained illumination-robust precision positioning for real-time manipulation of embedded objects. Int J Adv Manuf Technol 111, 2259–2276 (2020). https://doi.org/10.1007/s00170-020-06185-x

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00170-020-06185-x

Keywords

  • GAN
  • pix2pix
  • Convolutional neural network
  • Visual positioning
  • Illumination
  • Manipulation