Abstract
Robotic perception of filling mass estimation via multiple sensors and deep learning approaches is still an open problem due to the diverse pouring durations, small pixel ratio for target objects and complex pouring scenarios. In this paper, we propose a practical solution to tackle this challenging task via estimating filling level, filling type and container capacity simultaneously. The proposed method is inspired by how humans observe and understand the pouring process via the cooperation among multiple modalities, i.e., vision and audio. In a nutshell, our proposed method is divided into three folds to help the agent shape a rich understanding of the pouring procedure. First, the agent obtains the prior of container categories (i.e., cup, glass or box) through the object detection framework. Second, we integrate the audio features with the prior to make the agent learn a multi-modal feature space. Finally, the agent infers the distribution of both the container capacity and fluid properties. The experimental results show the effectiveness of the proposed method, which ranked as \(2^{nd}\) runner-up in the CORSMAL Challenge of Multi-modal Fusion and Learning For Robotics in ICPR 2020.
The work described in this paper was partially supported by grant from Guangdong-Hong Kong-Macau Greater Bay Area Center for Brain Science and Brain-Inspired Intelligence Fund (No. 20019009).
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Xompero, R.A., Sanchez-Matilla,R.M., Cavallaro, A.: CORSMAL Containers Manipulation (1.0) [Data set]. https://doi.org/10.17636/101CORSMAL1
Abdel-Hamid, O., Mohamed, A.R., Jiang, H., Deng, L., Penn, G., Yu, D.: Convolutional neural networks for speech recognition. IEEE/ACM Trans. Audio Speech Lang. Process. 22(10), 1533–1545 (2014)
Bae, H., et al.: Iros 2019 lifelong robotic vision: object recognition challenge [competitions]. IEEE Rob. Autom. Mag 27(2), 11–16 (2020)
Bhattacharyya, R., Floerkemeier, C., Sarma, S.: Rfid tag antenna based sensing: does your beverage glass need a refill? In: 2010 IEEE International Conference on RFID (IEEE RFID 2010), pp. 126–133. IEEE (2010)
Bochkovskiy, A., Wang, C.Y., Liao, H.Y.M.: Yolov4: optimal speed and accuracy of object detection. arXiv preprint arXiv:2004.10934 (2020)
Brandi, S., Kroemer, O., Peters, J.: Generalizing pouring actions between objects using warped parameters. In: 2014 IEEE-RAS International Conference on Humanoid Robots, pp. 616–621. IEEE (2014)
Clarke, S., Rhodes, T., Atkeson, C.G., Kroemer, O.: Learning audio feedback for estimating amount and flow of granular material. Proc. Mach. Learn. Res. 87 (2018)
Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: Imagenet: a large-scale hierarchical image database. In: 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 248–255. IEEE (2009)
Do, C., Burgard, W.: Accurate pouring with an autonomous robot using an RGB-D camera. In: Strand, M., Dillmann, R., Menegatti, E., Ghidoni, S. (eds.) IAS 2018. AISC, vol. 867, pp. 210–221. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-01370-7_17
Do, C., Schubert, T., Burgard, W.: A probabilistic approach to liquid level detection in cups using an RGB-D camera. In: 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 2075–2080. IEEE (2016)
Griffith, S., Sukhoy, V., Wegter, T., Stoytchev, A.: Object categorization in the sink: Learning behavior-grounded object categories with water. In: Proceedings of the 2012 ICRA Workshop on Semantic Perception, Mapping and Exploration. Citeseer (2012)
Gu, S., Holly, E., Lillicrap, T., Levine, S.: Deep reinforcement learning for robotic manipulation with asynchronous off-policy updates. In: 2017 IEEE International Conference on Robotics and Automation (ICRA), pp. 3389–3396. IEEE (2017)
Huang, Y., Sun, Y.: Learning to pour. In: 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 7005–7010. IEEE (2017)
Ikeno, S., Watanabe, R., Okazaki, R., Hachisu, T., Sato, M., Kajimoto, H.: Change in the amount poured as a result of vibration when pouring a liquid. In: Kajimoto, H., Ando, H., Kyung, K.-U. (eds.) Haptic Interaction. LNEE, vol. 277, pp. 7–11. Springer, Tokyo (2015). https://doi.org/10.1007/978-4-431-55690-9_2
Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)
LeCun, Y., Bengio, Y., Hinton, G.: Deep learning. Nature 521(7553), 436–444 (2015)
Lenz, I., Lee, H., Saxena, A.: Deep learning for detecting robotic grasps. Int. J. Rob. Res. 34(4–5), 705–724 (2015)
Liang, H., et al.: Making sense of audio vibration for liquid height estimation in robotic pouring. arXiv preprint arXiv:1903.00650 (2019)
Lin, T.Y., et al.: Microsoft COCO: common objects in context. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8693, pp. 740–755. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-10602-1_48
McFee, B., et al.: librosa: audio and music signal analysis in python. In: Proceedings of the 14th Python in Science Conference, vol. 8, pp. 18–25 (2015)
Muhlig, M., Gienger, M., Hellbach, S., Steil, J.J., Goerick, C.: Task-level imitation learning using variance-based movement optimization. In: 2009 IEEE International Conference on Robotics and Automation, pp. 1177–1184. IEEE (2009)
Nair, A., Bahl, S., Khazatsky, A., Pong, V., Berseth, G., Levine, S.: Contextual imagined goals for self-supervised robotic learning. In: Conference on Robot Learning, pp. 530–539. PMLR (2020)
Pastor, P., Hoffmann, H., Asfour, T., Schaal, S.: Learning and generalization of motor skills by learning from demonstration. In: 2009 IEEE International Conference on Robotics and Automation, pp. 763–768. IEEE (2009)
Paszke, A., et al.: Pytorch: an imperative style, high-performance deep learning library. In: Neural Information Processing Systems (NeurIPS), pp. 8024–8035 (2019)
Paulius, D., Huang, Y., Milton, R., Buchanan, W.D., Sam, J., Sun, Y.: Functional object-oriented network for manipulation learning. In: 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 2655–2662. IEEE (2016)
Paulius, D., Jelodar, A.B., Sun, Y.: Functional object-oriented network: Construction & expansion. In: 2018 IEEE International Conference on Robotics and Automation (ICRA), pp. 1–7. IEEE (2018)
Pierson, H.A., Gashler, M.S.: Deep learning in robotics: a review of recent research. Adv. Rob. 31(16), 821–835 (2017)
Pithadiya, K.J., Modi, C.K., Chauhan, J.D.: Selecting the most favourable edge detection technique for liquid level inspection in bottles. Int. J. Comput. Inf. Syst. Ind. Manag. Appl. (IJCISIM) ISSN, 2150–7988 (2011)
Redmon, J., Divvala, S., Girshick, R., Farhadi, A.: You only look once: unified, real-time object detection. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 779–788 (2016)
Rozo, L., Jiménez, P., Torras, C.: Force-based robot learning of pouring skills using parametric hidden Markov models. In: 9th International Workshop on Robot Motion and Control, pp. 227–232. IEEE (2013)
Saal, H.P., Ting, J.A., Vijayakumar, S.: Active estimation of object dynamics parameters with tactile sensors. In: 2010 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 916–921. IEEE (2010)
Sanchez-Matilla, R., et al.: Benchmark for human-to-robot handovers of unseen containers with unknown filling. IEEE Rob. Autom. Lett. 5(2), 1642–1649 (2020)
She, Q., et al.: Openloris-object: a robotic vision dataset and benchmark for lifelong deep learning. In: 2020 IEEE International Conference on Robotics and Automation (ICRA), pp. 4767–4773. IEEE (2020)
Shi, X., et al.: Are we ready for service robots? the openloris-scene datasets for lifelong slam. In: 2020 IEEE International Conference on Robotics and Automation (ICRA), pp. 3139–3145. IEEE (2020)
Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. In: ICLR, vol. 2015 (2015)
Yang, P.C., Sasaki, K., Suzuki, K., Kase, K., Sugano, S., Ogata, T.: Repeatable folding task by humanoid robot worker using deep learning. IEEE Rob. Autom. Lett 2(2), 397–403 (2016)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Liu, Q., Feng, F., Lan, C., Chan, R.H.M. (2021). VA2Mass: Towards the Fluid Filling Mass Estimation via Integration of Vision and Audio Learning. In: Del Bimbo, A., et al. Pattern Recognition. ICPR International Workshops and Challenges. ICPR 2021. Lecture Notes in Computer Science(), vol 12668. Springer, Cham. https://doi.org/10.1007/978-3-030-68793-9_33
Download citation
DOI: https://doi.org/10.1007/978-3-030-68793-9_33
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-68792-2
Online ISBN: 978-3-030-68793-9
eBook Packages: Computer ScienceComputer Science (R0)
