Abstract
Inspecting robotically fabricated objects to detect and classify discrepancies between virtual target models and as-built realities is one of the challenges that faces robotic fabrication. Industrial-grade computer vision methods have been widely used to detect manufacturing flaws in mass production lines. However, in mass-customization, a versatile and robust method should be flexible enough to ignore construction tolerances while detecting specified flaws in varied parts. This study aims to leverage recent developments in machine learning and convolutional neural networks to improve the resiliency and accuracy of surface inspections in architectural robotics. Under a supervised learning scenario, the authors compared two approaches: (1) transfer learning on a general purpose Convolutional Neural Network (CNN) image classifier, and (2) design and train a CNN from scratch to detect and categorize flaws in a robotic plastering workflow. Both CNNs were combined with conventional search methods to improve the accuracy and efficiency of the system. A web-based graphical user interface and a real-time video projection method were also developed to facilitate user interactions and control over the workflow.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
Code for this project are available on GitHub (https://github.com/Ardibid/RoboticPlasteringCNN).
- 2.
- 3.
AlexNet architecture might be confusing at the first sight since it has two parallel pipelines. However, the reason behind this dual pipeline is to train the model on two separate GPU simultaneously.
- 4.
Markups are simple user-defined drawings, i.e. circles and crosses, that can be used to communicate with the system.
- 5.
Although the authors first implemented Quad Tree search algorithm to compensate for the possible slow classification pipeline, the final model performance was good enough to provide near real-time experience. Accordingly, we opted for a grid search algorithm and avoided potential challenges that a Quad Tree search would introduce. The biggest drawback being its tendency to ignore small features in the initial steps of the search process when surveying large areas of the given image.
References
Amtsberg, F., Raspall, F., Trummer, A.: Digital-material feedback in architectural design. In: Ikeda, Y., Kaijima, S., Herr, C., Schnabel, M.A. (eds.) Proceedings of the 20th International Conference of the Association for Computer-Aided Architectural Design Research in Asia: Emerging Experience in Past, Present and Future of Digital Architecture, CAADRIA 2015, pp. 631–640. Daegu, South Korea (2015)
Bard, J., Blackwood, D., Sekhar, S., Brian, N.: Reality is interface: two motion capture case studies of human–machine collaboration in high-skill domain. Int. J. Architectural Comput. 14(4), 398–408 (2016a)
Bard, J., Tursky, R., Jeffers, M.: RECONstruction. In: Reinhardt, D., Saunders, R., Burry, J. (eds.) Robotic Fabrication Architecture, Art and Design 2016, pp. 262–273. Springer, Switzerland (2016b)
Bidgoli, A., Cardoso Llach, D.: Towards a motion grammar for robotic stereotomy. In: Ikeda, Y., Kaijima, S., Herr, C., Schnabel, M.A. (eds.) Proceedings of the 20th International Conference of the Association for Computer-Aided Architectural Design Research in Asia: Emerging Experience in Past, Present and Future of Digital Architecture, CAADRIA 2015, pp. 723–732. Daegu, South Korea (2015)
Dawson-Haggerty, M. (n.d.): Open ABB Driver. https://github.com/robotics/open_abb. Accessed 5 Oct 2017
Donahue, J., Jia, Y., Vinyals, O., Hoffman, J., Zhang, N., Tzeng, E., Darrell, T.: DeCAF: a deep convolutional activation feature for generic visual recognition. In: Proceedings of the 31st International Conference on Machine Learning, PMLR, vol. 32(1), pp. 647–655. ACM, New York (2014)
Egmont-Petersen, M., de Ridder, D., Handels, H.: Image processing with neural networks—a review. Pattern Recogn. 35(10), 2279–2301 (2002)
Esteva, A., Kuprel, B., Novoa, R.A., Ko, J., Swetter, S.M., Blau, H.M., Thrun, S.: Dermatologist-level classification of skin cancer with deep neural networks. Nature 542(7639), 115–118 (2017)
Image retraining Tutorial (n.d.). https://www.tensorflow.org/tutorials/image_retraining#training. Accessed 20 Nov 2017
Inception v3 (2018). https://www.kaggle.com/pytorch/inceptionv3. Accessed 20 Feb 2018
Krizhevsky, A., Sutskever, I., Hinton, G.E.: ImageNet classification with deep convolutional neural networks. In: Proceedings of the 25th International Conference on Neural Information Processing Systems, NIPS 2012, vol. 1, pp. 1097–1105. Lake Tahoe, CA (2012)
LeCun, Y., Boser, B., Denker, J.S., Henderson, D., Howard, R.E., Hubbard, W., Jackel, L.D.: Backpropagation applied to handwritten zip code recognition. Neural Comput. 1(4), 541–551 (1989)
Oquab, M., Bottou, L., Laptev, I., Sivic, J.: Learning and transferring mid-level image representations using convolutional neural networks. In: Proceedings of the 2014 IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2014), pp. 1717–1724. Columbus, OH (2014). https://doi.org/10.1109/CVPR.2014.222
Pal, N.R., Pal, S.K.: A review on image segmentation techniques. Pattern Recogn. 26(9), 1277–1294 (1993)
Rocchini, C., Cignoni, P., Montani, C., Pingi, P., Scopigno, R.: A low cost 3D scanner based on structured light. Comput. Graph. Forum 20(3), 299–308 (2001)
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Bernstein, M.: ImageNet large scale visual recognition challenge. Int. J. Comput. Vis. 115(3), 211–252 (2015)
Sharif Razavian, A., Azizpour, H., Sullivan, J., Carlsson, S.: CNN features off-the-shelf: an astounding baseline for recognition, In: Proceedings of the 2014 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPR-W 2014), pp. 806–813. Columbus, OH (2014). https://doi.org/10.1109/CVPRW.2014.131
Shin, H.-C., Roth, H.R., Gao, M., Lu, L., Xu, Z., Nogues, I., Summers, R.M.: Deep convolutional neural networks for computer-aided detection: CNN architectures, dataset characteristics and transfer learning. IEEE Trans. Med. Imaging 35(5), 1285–1298 (2016)
Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. arXiv Preprint arXiv:1409.1556 (2014)
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V, Rabinovich, A.: Going deeper with convolutions. In: Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2015), pp. 1–9. Boston, MA (2015)
Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., Wojna, Z.: Rethinking the inception architecture for computer vision. In: Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2016), pp. 2818–2826. Las Vegas, NV (2016)
Vasey, L., Maxwell, I., Pigram, D.: Adaptive part variation. In: McGee, W., Ponce de Leon, M. (eds.) Robotic Fabrication in Architecture, Art and Design 2014, pp. 291–304. Springer International Publishing Switzerland (2014)
Zhang, L., Curless, B., Seitz, S.M.: Rapid shape acquisition using color structured light and multi-pass dynamic programming. In: Proceedings of First International Symposium on 3D Data Processing Visualization and Transmission (3DPVT 2002), pp. 24–36. Padova, Italy (2002)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2019 Springer Nature Switzerland AG
About this paper
Cite this paper
Bard, J., Bidgoli, A., Chi, W.W. (2019). Image Classification for Robotic Plastering with Convolutional Neural Network. In: Willmann, J., Block, P., Hutter, M., Byrne, K., Schork, T. (eds) Robotic Fabrication in Architecture, Art and Design 2018. ROBARCH 2018. Springer, Cham. https://doi.org/10.1007/978-3-319-92294-2_1
Download citation
DOI: https://doi.org/10.1007/978-3-319-92294-2_1
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-92293-5
Online ISBN: 978-3-319-92294-2
eBook Packages: Intelligent Technologies and RoboticsIntelligent Technologies and Robotics (R0)