Abstract
In this paper we propose a robotic system for flower removal and pollination with the deep learning-based 3D perception module. The robotic system consists of a collaborative robot equipped with an RGB-D camera used in the detection pipeline. A semi-automated 3D based image annotation method (A3IA) is developed for the purpose of automatically generating the training data. The efficiency of the detection model trained on automatically annotated data is compared to that of the model trained on synthetic data. Experimental validation of the proposed methods is conducted and robustness and precision of both detection and positioning are reported.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
AliceVision: Meshroom: a 3D reconstruction software (2018). https://github.com/alicevision/meshroom
Barth, R., IJsselmuiden, J., Hemming, J., Van Henten, E.J.: Data synthesis methods for semantic segmentation in agriculture: a capsicum annuum dataset. Comput. Electron. Agric. 144, 284–296 (2018)
De Gregorio, D., Tonioni, A., Palli, G., Di Stefano, L.: Semiautomatic labeling for deep learning in robotics. IEEE Trans. Autom. Sci. Eng. 17(2), 611–620 (2020)
Di Cicco, M., Potena, C., Grisetti, G., Pretto, A.: Automatic model based dataset generation for fast and accurate crop and weeds detection. In: 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 5188–5195. IEEE (2017)
Dias, P.A., Tabb, A., Medeiros, H.: Apple flower detection using deep convolutional networks. Comput. Industry 99, 17–28 (2018)
Doersch, C., Zisserman, A.: Sim2real transfer learning for 3d human pose estimation: motion to the rescue. In: Wallach, H., Larochelle, H., Beygelzimer, A., d’Alché-Buc, F., Fox, E., Garnett, R. (eds.) Advances in Neural Information Processing Systems, vol. 32. Curran Associates, Inc. (2019)
He, L., Schupp, J.: Sensing and automation in pruning of apple trees: a review. Agronomy 8, 211 (2018)
Hess, R.: Blender Foundations: The Essential Guide to Learning Blender 2.6. Focal Press (2010)
Horton, R., Cano, E., Bulanon, D., Fallahi, E.: Peach flower monitoring using aerial multispectral imaging. J. Imaging 3(1) (2017)
Ilin, V., Kalinov, I., Karpyshev, P.A., Tsetserukou, D.: Deepscanner: a robotic system for automated 2d object dataset collection with annotations. arXiv:2108.02555 (2021)
Jia, W., Zhang, Y., Lian, J., Zheng, Y., Zhao, D., Li, C.: Apple harvesting robot under information technology: a review. Int. J. Adv. Robot. Syst. 17(3) (2020)
Kar, A., Prakash, A., Liu, M.Y., Cameracci, E., Yuan, J., Rusiniak, M., Acuna, D., Torralba, A., Fidler, S.: Meta-sim: learning to generate synthetic datasets. In: ICCV (2019)
Kiyokawa, T., Tomochika, K., Takamatsu, J., Ogasawara, T.: Fully automated annotation with noise-masked visual markers for deep-learning-based object detection. IEEE Robot. Autom. Lett. 4(2), 1972–1977 (2019)
Liu, W., Anguelov, D., Erhan, D., Szegedy, C., Reed, S., Fu, C.Y., Berg, A.: Ssd: Single shot multibox detector. In: Computer Vision—ECCV 2016, vol. 9905, pp. 21–37 (2016)
Maric, B., Polic, M., Tabak, T., Orsag, M.: Unsupervised optimization approach to in situ calibration of collaborative human-robot interaction tools. In: 2020 IEEE International Conference on Multisensor Fusion and Integration for Intelligent Systems (MFI), pp. 255–262. IEEE (2020)
Pashevich, A., Strudel, R., Kalevatykh, I., Laptev, I., Schmid, C.: Learning to augment synthetic images for sim2real policy transfer. In: 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 2651–2657 (2019)
Polic, M., Ivanovic, A., Maric, B., Arbanas, B., Tabak, J., Orsag, M.: Structured ecological cultivation with autonomous robots in indoor agriculture. In: 2021 16th International Conference on Telecommunications (ConTEL), pp. 189–195 (2021)
Polic, M., Tabak, J., Orsag, M.: Pepper to fall: a perception method for sweet pepper robotic harvesting. Intelligent Service Robotics (2021)
Polic, M., et al.: Strawberry flowers (2021). https://sites.google.com/view/specularia-pepper-picking/home/flower-picking. Accessed 2021-09-09
Ruiz-Sarmiento, J., Galindo, C., Gonzalez-Jimenez, J.: Olt: a toolkit for object labeling applied to robotic RGB-D datasets. In: 2015 European Conference on Mobile Robots (ECMR), pp. 1–6 (2015)
Strader, J., Yang, C., Gu, Y., Nguyen, J., Tatsch, C., Du, Y., Lassak, K., Buzzo, B., Watson, R., Cerbone, H., Ohi, N.: Flower interaction subsystem for a precision pollination robot. In: 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 5534–5541 (2019)
Stumpf, D., Krauß, S., Reis, G., Wasenmüller, O., Stricker, D.: Salt: a semi-automatic labeling tool for RGB-D video sequences. In: 16th International Conference on Computer Vision Theory and Applications, pp. 595–603 (2021)
Tobin, J., Fong, R., Ray, A., Schneider, J., Zaremba, W., Abbeel, P.: Domain randomization for transferring deep neural networks from simulation to the real world. In: 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 23–30. IEEE (2017)
Xiong, Y., Ge, Y., Grimstad, L., From, P.J.: An autonomous strawberry-harvesting robot: design, development, integration, and field evaluation. J. Field Robot. 37(2), 202–224 (2020)
Zhang, Q., Karkee, M., Tabb, A.: The use of agricultural robots in orchard management. In: Billingsley, J. (ed.) Robotics and Automation for Improving Agriculture. Burleigh DODDS Science Publishing (2019)
Acknowledgements
This work has been supported by Croatian Science Foundation under the project Specularia UIP-2017-05-4042 [17]. The work of doctoral student Jelena Vuletić has been supported in part by the “Young researchers’ career development project-training of doctoral students” of the Croatian Science Foundation funded by the European Union from the European Social Fund.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Vuletić, J., Polić, M., Orsag, M. (2023). Robotic Strawberry Flower Treatment Based on Deep-Learning Vision. In: Borja, P., Della Santina, C., Peternel, L., Torta, E. (eds) Human-Friendly Robotics 2022. HFR 2022. Springer Proceedings in Advanced Robotics, vol 26. Springer, Cham. https://doi.org/10.1007/978-3-031-22731-8_14
Download citation
DOI: https://doi.org/10.1007/978-3-031-22731-8_14
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-22730-1
Online ISBN: 978-3-031-22731-8
eBook Packages: Intelligent Technologies and RoboticsIntelligent Technologies and Robotics (R0)