Advertisement

Image Classification for Robotic Plastering with Convolutional Neural Network

  • Joshua BardEmail author
  • Ardavan Bidgoli
  • Wei Wei Chi
Conference paper

Abstract

Inspecting robotically fabricated objects to detect and classify discrepancies between virtual target models and as-built realities is one of the challenges that faces robotic fabrication. Industrial-grade computer vision methods have been widely used to detect manufacturing flaws in mass production lines. However, in mass-customization, a versatile and robust method should be flexible enough to ignore construction tolerances while detecting specified flaws in varied parts. This study aims to leverage recent developments in machine learning and convolutional neural networks to improve the resiliency and accuracy of surface inspections in architectural robotics. Under a supervised learning scenario, the authors compared two approaches: (1) transfer learning on a general purpose Convolutional Neural Network (CNN) image classifier, and (2) design and train a CNN from scratch to detect and categorize flaws in a robotic plastering workflow. Both CNNs were combined with conventional search methods to improve the accuracy and efficiency of the system. A web-based graphical user interface and a real-time video projection method were also developed to facilitate user interactions and control over the workflow.

Keywords

Architectural robotics Machine learning Convolutional neural networks Image classification 

References

  1. Amtsberg, F., Raspall, F., Trummer, A.: Digital-material feedback in architectural design. In: Ikeda, Y., Kaijima, S., Herr, C., Schnabel, M.A. (eds.) Proceedings of the 20th International Conference of the Association for Computer-Aided Architectural Design Research in Asia: Emerging Experience in Past, Present and Future of Digital Architecture, CAADRIA 2015, pp. 631–640. Daegu, South Korea (2015)Google Scholar
  2. Bard, J., Blackwood, D., Sekhar, S., Brian, N.: Reality is interface: two motion capture case studies of human–machine collaboration in high-skill domain. Int. J. Architectural Comput. 14(4), 398–408 (2016a)CrossRefGoogle Scholar
  3. Bard, J., Tursky, R., Jeffers, M.: RECONstruction. In: Reinhardt, D., Saunders, R., Burry, J. (eds.) Robotic Fabrication Architecture, Art and Design 2016, pp. 262–273. Springer, Switzerland (2016b)CrossRefGoogle Scholar
  4. Bidgoli, A., Cardoso Llach, D.: Towards a motion grammar for robotic stereotomy. In: Ikeda, Y., Kaijima, S., Herr, C., Schnabel, M.A. (eds.) Proceedings of the 20th International Conference of the Association for Computer-Aided Architectural Design Research in Asia: Emerging Experience in Past, Present and Future of Digital Architecture, CAADRIA 2015, pp. 723–732. Daegu, South Korea (2015)Google Scholar
  5. Dawson-Haggerty, M. (n.d.): Open ABB Driver. https://github.com/robotics/open_abb. Accessed 5 Oct 2017
  6. Donahue, J., Jia, Y., Vinyals, O., Hoffman, J., Zhang, N., Tzeng, E., Darrell, T.: DeCAF: a deep convolutional activation feature for generic visual recognition. In: Proceedings of the 31st International Conference on Machine Learning, PMLR, vol. 32(1), pp. 647–655. ACM, New York (2014)Google Scholar
  7. Egmont-Petersen, M., de Ridder, D., Handels, H.: Image processing with neural networks—a review. Pattern Recogn. 35(10), 2279–2301 (2002)CrossRefGoogle Scholar
  8. Esteva, A., Kuprel, B., Novoa, R.A., Ko, J., Swetter, S.M., Blau, H.M., Thrun, S.: Dermatologist-level classification of skin cancer with deep neural networks. Nature 542(7639), 115–118 (2017)CrossRefGoogle Scholar
  9. Image retraining Tutorial (n.d.). https://www.tensorflow.org/tutorials/image_retraining#training. Accessed 20 Nov 2017
  10. Inception v3 (2018). https://www.kaggle.com/pytorch/inceptionv3. Accessed 20 Feb 2018
  11. Krizhevsky, A., Sutskever, I., Hinton, G.E.: ImageNet classification with deep convolutional neural networks. In: Proceedings of the 25th International Conference on Neural Information Processing Systems, NIPS 2012, vol. 1, pp. 1097–1105. Lake Tahoe, CA (2012)Google Scholar
  12. LeCun, Y., Boser, B., Denker, J.S., Henderson, D., Howard, R.E., Hubbard, W., Jackel, L.D.: Backpropagation applied to handwritten zip code recognition. Neural Comput. 1(4), 541–551 (1989)CrossRefGoogle Scholar
  13. Oquab, M., Bottou, L., Laptev, I., Sivic, J.: Learning and transferring mid-level image representations using convolutional neural networks. In: Proceedings of the 2014 IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2014), pp. 1717–1724. Columbus, OH (2014).  https://doi.org/10.1109/CVPR.2014.222
  14. Pal, N.R., Pal, S.K.: A review on image segmentation techniques. Pattern Recogn. 26(9), 1277–1294 (1993)CrossRefGoogle Scholar
  15. Rocchini, C., Cignoni, P., Montani, C., Pingi, P., Scopigno, R.: A low cost 3D scanner based on structured light. Comput. Graph. Forum 20(3), 299–308 (2001)CrossRefGoogle Scholar
  16. Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Bernstein, M.: ImageNet large scale visual recognition challenge. Int. J. Comput. Vis. 115(3), 211–252 (2015)MathSciNetCrossRefGoogle Scholar
  17. Sharif Razavian, A., Azizpour, H., Sullivan, J., Carlsson, S.: CNN features off-the-shelf: an astounding baseline for recognition, In: Proceedings of the 2014 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPR-W 2014), pp. 806–813. Columbus, OH (2014).  https://doi.org/10.1109/CVPRW.2014.131
  18. Shin, H.-C., Roth, H.R., Gao, M., Lu, L., Xu, Z., Nogues, I., Summers, R.M.: Deep convolutional neural networks for computer-aided detection: CNN architectures, dataset characteristics and transfer learning. IEEE Trans. Med. Imaging 35(5), 1285–1298 (2016)CrossRefGoogle Scholar
  19. Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. arXiv Preprint arXiv:1409.1556 (2014)
  20. Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V, Rabinovich, A.: Going deeper with convolutions. In: Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2015), pp. 1–9. Boston, MA (2015)Google Scholar
  21. Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., Wojna, Z.: Rethinking the inception architecture for computer vision. In: Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2016), pp. 2818–2826. Las Vegas, NV (2016)Google Scholar
  22. Vasey, L., Maxwell, I., Pigram, D.: Adaptive part variation. In: McGee, W., Ponce de Leon, M. (eds.) Robotic Fabrication in Architecture, Art and Design 2014, pp. 291–304. Springer International Publishing Switzerland (2014)Google Scholar
  23. Zhang, L., Curless, B., Seitz, S.M.: Rapid shape acquisition using color structured light and multi-pass dynamic programming. In: Proceedings of First International Symposium on 3D Data Processing Visualization and Transmission (3DPVT 2002), pp. 24–36. Padova, Italy (2002)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  1. 1.Carnegie Mellon UniversityPittsburghUSA

Personalised recommendations