Autonomous Robots

, Volume 43, Issue 3, pp 611–629 | Cite as

Multimodal anomaly detection for assistive robots

  • Daehyung ParkEmail author
  • Hokeun Kim
  • Charles C. Kemp


Detecting when something unusual has happened could help assistive robots operate more safely and effectively around people. However, the variability associated with people and objects in human environments can make anomaly detection difficult. We previously introduced an algorithm that uses a hidden Markov model (HMM) with a log-likelihood detection threshold that varies based on execution progress. We now present an improved version of our previous algorithm (HMM-D) and introduce a new algorithm based on Gaussian process regression (HMM-GP). We also present a new and more thorough evaluation of 8 anomaly detection algorithms with force, sound, and kinematic signals collected from a robot closing microwave doors, latching a toolbox, scooping yogurt, and feeding yogurt to able-bodied participants. Overall, HMM-GP had the highest performance in terms of area under the curve for these real-world tasks, and multiple modalities improved performance with some anomalies being better detected with particular modalities. With synthetic anomalies, HMM-D exhibited shorter detection delays and outperformed HMM-GP with high-magnitude anomalies. In general, higher-magnitude synthetic anomalies tended to be detected more rapidly.


Multimodality Anomaly detection Assistive manipulation Execution monitoring 



We thank Youkeun Kim, Zackory Erickson, Ariel Kapusta, Chansu Kim, and Jane Chisholm for their assistance throughout this project. This work was supported in part by NSF Awards IIS-1150157, EFRI-1137229, and NIDILRR Grant 90RE5016-01-00 via RERC TechSAge. Dr. Kemp is a cofounder, a board member, an equity holder, and the CTO of Hello Robot, Inc., which is developing products related to this research. This research could affect his personal financial status. The terms of this arrangement have been reviewed and approved by Georgia Tech in accordance with its conflict of interest policies.


  1. Ando, S., Thanomphongphan, T., Hoshino, D., Seki, Y., & Suzuki, E. (2011). ACE: Anomaly clustering ensemble for multi-perspective anomaly detection in robot behaviors. In Proceedings of the international conference on data mining (pp. 1–12). SIAM.Google Scholar
  2. Angelov, V. P., Giglio, C., Guardiola, C., Lughofer, E., & Lujan, J. M. (2006). An approach to model-based fault detection in industrial measurement systems with application to engine test benches. Measurement Science and Technology, 17(7), 1809.CrossRefGoogle Scholar
  3. Argall, B. D. (2016). Modular and adaptive wheelchair automation. In M. A. Hsieh, O. Khatib, & V. Kumar (Eds.), Experimental robotics (pp. 835–848). Springer.Google Scholar
  4. Bishop, C. M. (2006). Pattern recognition and machine learning (information science and statistics). New York, NJ: Springer.zbMATHGoogle Scholar
  5. Bittencourt, A. C., Saarinen, K., Sander-Tavallaey, S., Gunnarsson, S., & Norrlöf, M. (2014). A data-driven approach to diagnostics of repetitive processes in the distribution domain-applications to gearbox diagnostics in industrial robots and rotating machines. Mechatronics, 24(8), 1032–1041.CrossRefGoogle Scholar
  6. Blank, S., Pfister, T., & Berns, K. (2011). Sensor failure detection capabilities in low-level fusion: a comparison between fuzzy voting and kalman filtering. In IEEE international conference on robotics and automation (ICRA) (pp 4974–4979). IEEE.Google Scholar
  7. Bouguerra, A., Karlsson, L., & Saffiotti, A. (2008). Monitoring the execution of robot plans using semantic knowledge. Robotics and Autonomous Systems, 56(11), 942–954.CrossRefGoogle Scholar
  8. Brambilla, D., Capisani, L. M., Ferrara, A., & Pisu, P. (2008). Fault detection for robot manipulators via second-order sliding modes. IEEE Transactions on Industrial Electronics, 55(11), 3954–3963. ISSN 0278-0046.
  9. Chandola, V., Banerjee, A., & Kumar, V. (2009). Anomaly detection. ACM Computing Surveys, 41(3), 1–58.,, ISSN 03600300.
  10. Chen, T. L., Ciocarlie, M., Cousins, S., Grice, P., Hawkins, K., Hsiao, et al. (2013). Robots for humanity: Using assistive robots to empower people with disabilities. IEEE Robotics & Automation Magazine, 20(1), 30–39.Google Scholar
  11. Chu, V., McMahon, I., Riano, L., McDonald, C. G., He, Q., Perez-Tejada, J. M., et al. (2013). Using robotic exploratory procedures to learn the meaning of haptic adjectives. In IEEE international conference on robotics and automation (ICRA) (pp. 3048–3055). IEEE.Google Scholar
  12. Ciocarlie, M., Hsiao, K., Leeper, A., & Gossow, D. (2012). Mobile manipulation through an assistive home robot. In IEEE/RSJ international conference on intelligent robots and systems. IEEE.Google Scholar
  13. Clifton, D. A., Hugueny, S., & Tarassenko, L. (2011). Novelty detection with multivariate extreme value statistics. Journal of Signal Processing Systems, 65(3), 371–389.CrossRefGoogle Scholar
  14. Copilusi, C., Kaur, M., & Ceccarelli, M. (2015a). Lab experiences with LARM clutched arm for assisting disabled people (pp. 603–611). Cham: Springer International Publishing. ISBN: 978-3-319-09411-3.
  15. Copilusi, C., Kaur, M., & Ceccarelli, M., (2015b). Lab experiences with larm clutched arm for assisting disabled people. In New trends in mechanism and machine science (pp. 603–611). Springer.Google Scholar
  16. Cressie, N. A. C. (1993). Statistics for spatial data. New York, CH: Wiley series in probability and mathematical statistics, Wiley. ISBN: 978-0-471-00255-0.
  17. Dames, P. M., Schwager, M., & Rus, D. (2016). Active magnetic anomaly detection using multiple micro aerial vehicles. IEEE Robotics and Automation Letters, 1(1), 153–160. ISSN: 2377-3766.
  18. Di Lello, E., Klotzbucher, M., De Laet, T., & Bruyninckx, H. (2013). Bayesian time-series models for continuous fault detection and recognition in industrial robotic tasks. In IEEE/RSJ international conference on intelligent robots and systems (IROS) (pp. 5827–5833). IEEE.Google Scholar
  19. Dua, S., & Du, X. (2011). Data mining and machine learning in cybersecurity. London: CRC Press.CrossRefzbMATHGoogle Scholar
  20. Eclipse Automation. (2016). Meet obi, a robot that helps disabled individuals eat unassisted. Accessed 15 July 2017.
  21. Fagogenis, G., Carolis, V. D., & Lane, D. M. (2016). Online fault detection and model adaptation for underwater vehicles in the case of thruster failures. In IEEE international conference on robotics and automation. IEEE.Google Scholar
  22. Fiala, M. (2005). Artag, a fiducial marker system using digital techniques. In IEEE computer society conference on computer vision and pattern recognition (Vol. 2, pp. 590–596). IEEE.Google Scholar
  23. Fujii, H., Yamashita, A., & Asama, H., (2016). Defect detection with estimation of material condition using ensemble learning for hammering test. In IEEE international conference on robotics and automation. IEEE.Google Scholar
  24. Fukunaga, K. (2013). Introduction to statistical pattern recognition. London: Academic press.zbMATHGoogle Scholar
  25. Gehring, J., Miao, Y., Metze, F., & Waibel, A. (2013). Extracting deep bottleneck features using stacked auto-encoders. In Proceedings of IEEE international conference on acoustics, speech and signal processing (ICASSP) (pp. 3377–3381). IEEE.Google Scholar
  26. Graf, B., Reiser, U., Hägele, M., Mauz, K., & Klein, P. (2009). Robotic home assistant care-o-bot® 3-product vision and innovation platform. In IEEE workshop on advanced robotics and its social impacts (ARSO) (pp. 139–144). IEEE.Google Scholar
  27. Haidu, A., Kohlsdorf, D., & Beetz, M., (2015) Learning action failure models from interactive physics-based simulations. In IEEE/RSJ international conference on intelligent robots and systems (IROS) (pp. 5370–5375). IEEE.Google Scholar
  28. Hawkins, K. P., Grice, P. M., Chen, T. L., King, C.-H., & Kemp, C. C. (2014) Assistive mobile manipulation for self-care tasks around the head. In IEEE symposium on computational intelligence in robotic rehabilitation and assistive technologies (pp. 16–25). IEEE.Google Scholar
  29. Hoffmann, H. (2007). Kernel PCA for novelty detection. Pattern recognition, 40(3), 863–874.CrossRefzbMATHGoogle Scholar
  30. Hung, Y.-X., Chiang, C.-Y., Hsu, S.J., & Chan, C.-T., (2010). Abnormality detection for improving elders daily life independent. In International conference on smart homes and health telematics (pp. 186–194). Springer.Google Scholar
  31. Jain, A., & Kemp, C. C. (2013). Improving robot manipulation with data-driven object-centric models of everyday forces. Autonomous Robots, 35(2–3), 143–159.CrossRefGoogle Scholar
  32. Jiménez Villarreal, J., & Ljungblad, S. (2011) Experience centred design for a robotic eating aid. In Proceedings of the 6th international conference on human–robot interaction (pp. 155–156). ACM.Google Scholar
  33. Kappler, D., Pastor, P., Kalakrishnan, M., Wuthrich, M., & Schaal, S. (2015). Data-driven online decision making for autonomous manipulation. In Proceedings of robotics: Science and systems.Google Scholar
  34. Kazemi, V., & Sullivan, J. (2014) One millisecond face alignment with an ensemble of regression trees. In IEEE conference on computer vision and pattern recognition (pp. 1867–1874).
  35. Kemp, C. C., Edsinger, A., & Torres-Jara, E. (2007). Challenges for robot manipulation in human environments [grand challenges of robotics]. IEEE Robotics & Automation Magazine, 14(1), 20–29.CrossRefGoogle Scholar
  36. Ketabdar, H., Vepa, J, Bengio, S., & Bourlard, H. (2006) Using more informative posterior probabilities for speech recognition. In IEEE international conference on acoustics speech and signal processing proceedings (Vol. 1, pp. 1). IEEE.Google Scholar
  37. Khan, S. S., Karg, M. E., Hoey, J., & Kulic, D. (2012). Towards the detection of unusual temporal events during activities using HMMs. In Proceedings of the ACM conference on ubiquitous computing (pp. 1075–1084). ACM.Google Scholar
  38. Kim, D. J., Wang, Z., Paperno, N., & Behal, A. (2014). System design and implementation of UCF-MANUS—an intelligent assistive robotic manipulator. IEEE/ASME transactions on mechatronics, 19(1), 225–237. ISSN: 1083-4435.
  39. King, C. H., Chen, T. L., Jain, A., & Kemp, C. C. (2010). Towards an assistive robot that autonomously performs bed baths for patient hygiene. In IEEE/RSJ international conference on intelligent robots and systems (pp. 319–324).
  40. King, D. E. (2009). Dlib-ml: A machine learning toolkit. Journal of Machine Learning Research, 10, 1755–1758.Google Scholar
  41. Lee, H., Hwang, B., & Cho, S. (2002). Analysis of novelty detection properties of autoassociative mlp. Journal of Korean Institute of Industrial Engineers, 28(2), 147–161.Google Scholar
  42. Leidner, D., Dietrich, A., Beetz, M., & Albu-Schäffer, A. (2016). Knowledge-enabled parameterization of whole-body control strategies for compliant service robots. Autonomous Robots, 40(3), 519–536. ISSN: 1573-7527.
  43. Lepora, N. F., Pearson, M. J., Mitchinson, B., Evans, M., Fox, C., Pipe, A., Gurney, K., & Prescott, T. J. (2010) Naive bayes novelty detection for a moving robot with whiskers. In IEEE international conference on robotics and biomimetics (ROBIO) (pp. 131–136). IEEE.Google Scholar
  44. Lühr, S., Venkatesh, S., West, G., & Bui, H. H. (2004). Explicit state duration HMM for abnormality detection in sequences of human activity. In Pacific rim international conference on artificial intelligence (pp. 983–984). Springer.Google Scholar
  45. Malhotra, P., Vig, L., Shroff, G., & Agarwal, P., (2015). Long short term memory networks for anomaly detection in time series. In 23rd European symposium on artificial neural networks, computational intelligence and machine learning (p. 89).Google Scholar
  46. Marcolino, F., & Wang, J. (2013). Detecting anomalies in humanoid joint trajectories. In IEEE/RSJ international conference on intelligent robots and systems (IROS) (pp. 2594–2599). IEEE.Google Scholar
  47. Markou, M., & Singh, S. (2003). Novelty detection: Areviewpart 1: statistical approaches. Signal Processing, 83(12), 2481–2497.CrossRefzbMATHGoogle Scholar
  48. Mathieu, B., Essid, S., Fillon, T., Prado, J., & Richard, G. (2010). Yaafe, an easy to use and efficient audio feature extraction software. In Proceedings of the 11th international conference on music information retrieval (ISMIR (pp. 441–446).Google Scholar
  49. Mealtime Partners. (2017). Specializing in assistive dining and drinking equipment. Accessed July 15, 2017.
  50. Mendoza, J. P., Veloso, M., & Simmons, R. (2014). Focused optimization for online detection of anomalous regions. In IEEE international conference on robotics and automation (ICRA) (pp. 3358–3363). IEEE.Google Scholar
  51. Morris, B. T., & Trivedi, M. M. (2008). Learning and classification of trajectories in dynamic scenes: A general framework for live video analysis. In IEEE fifth international conference on advanced video and signal based surveillance (AVSS) (pp. 154–161). IEEE.Google Scholar
  52. Nguyen, H., Anderson, C., Trevor, A., Jain, A., Xu, Z., & Kemp, C. C. (2008). El-e: An assistive robot that fetches objects from flat surfaces. In Robotic helpers, international conference on human–robot interaction.Google Scholar
  53. Niekum, S., Osentoski, S., Atkeson, C. G., & Barto, A. G., (2014). Learning articulation changepoint models from demonstration. In Robotics science and systems (RSS) workshop on learning plans with context from human signals.Google Scholar
  54. Ocak, H., & Loparo, K. A. (2005). HMM-based fault detection and diagnosis scheme for rolling element bearings. Journal of Vibration and Acoustics, 127(4), 299–306.CrossRefGoogle Scholar
  55. Ogorodnikova, O. (2008). Methodology of safety for a human robot interaction designing stage. In Conference on human system interactions (pp. 452–457). IEEE.Google Scholar
  56. Papageorgiou, X. S., Tzafestas, C. S., Maragos, P., Pavlakos, G., Chalvatzaki, G., Moustris, G., et al. (2014). Advances in intelligent mobility assistance robot integrating multimodal sensory processing. In International conference on universal access in human–computer interaction (pp. 692–703). Springer.Google Scholar
  57. Park, D., Erickson, Z., Bhattacharjee, T., & Kemp, C. C. (2016a). Multimodal execution monitoring for anomaly detection during robot manipulation. In IEEE international conference on robotics and automation. IEEE.Google Scholar
  58. Park, D., Kim, Y. K., Erickson, Z., & Kemp, C. C. (2016b). Towards assistive feeding with a general-purpose mobile manipulator. In IEEE international conference on robotics and automation-workshop on human–robot interfaces for enhanced physical interactions.Google Scholar
  59. Pastor, P., Kalakrishnan, M., Chitta, E., Theodorou, S., & Schaal, S. (2011). Skill learning and task outcome prediction for manipulation. In IEEE international conference on robotics and automation (ICRA) (pp. 3828–3834). IEEE.Google Scholar
  60. Patterson Medical. Meal buddy, (2017). Accessed: July 15, 2017.
  61. Perduca, V., & Nuel, G. (2013). Measuring the influence of observations in HMMs through the kullback-leibler distance. IEEE Signal Processing Letters, 20(2), 145–148.CrossRefGoogle Scholar
  62. Pettersson, O. (2005). Execution monitoring in robotics: A survey. Robotics and Autonomous Systems, 53(2), 73–88.MathSciNetCrossRefGoogle Scholar
  63. Pimentel, M. A. F., Clifton, L., David, A., & Tarassenko, L. (2014). A review of novelty detection. Signal Processing, 99, 215–249.CrossRefGoogle Scholar
  64. Rabiner, L. R. (1989). A tutorial on hidden markov models and selected applications in speech recognition. In Proceedings of the IEEE (pp. 257–286).Google Scholar
  65. Rasmussen, C. E. (2004). Gaussian processes in machine learning. In Advanced lectures on machine learning (pp. 63–71). Springer.Google Scholar
  66. Rasmussen, C. E., Kuss, M., et al. (2003). Gaussian processes in reinforcement learning. In Advances in neural information processing systems (Vol. 4, p. 1).Google Scholar
  67. Rodriguez, A., Bourne, D., Mason, M., Rossano, G. F., & Wang, J., (2010). Failure detection in assembly: Force signature analysis. In IEEE conference on automation science and engineering (CASE) (pp. 210–215). IEEE.Google Scholar
  68. Rodriguez, A., Mason, M. T., Srinivasa, S., Bernstein, M., & Zirbel, A. (2011). Abort and retry in grasping. In IEEE international conference on intelligent robots and systems (IROS).Google Scholar
  69. Sakaguchi, T., Yokoi, K., Ujiie, T., Tsunoo, S., & Wada, K. (2009). Design of common environmental information for door-closing tasks with various robots. International Journal of Robotics and Automation, 24(3), 203.Google Scholar
  70. Schliep, A., Rungsarityotin, W., & Georgi, B. (2004). General hidden markov model library.
  71. Schrer, S., Killmann, I., Frank, B., Vlker, M., Fiederer, L., Ball, T., & Burgard, W. (2015). An autonomous robotic assistant for drinking. In IEEE international conference on robotics and automation (ICRA) (pp. 6482–6487).
  72. Serdio, F., Lughofer, E., Pichler, K., Buchegger, T., & Efendic, H. (2014). Residual-based fault detection using soft computing techniques for condition monitoring at rolling mills. Information Sciences, 259, 304–320.CrossRefGoogle Scholar
  73. Silvrio, J., Rozo, L., Calinon, S., & Caldwell, D. G. (2015). Learning bimanual end-effector poses from demonstrations using task-parameterized dynamical systems. In 2015 IEEE/RSJ international conference on intelligent robots and systems (IROS) (pp. 464–470).
  74. Simanek, J., Kubelka, V., & Reinstein, M. (2015). Improving multi-modal data fusion by anomaly detection. Autonomous Robots, 39(2), 139–154.CrossRefGoogle Scholar
  75. Snelson, E., & Ghahramani, Z. (2006). Sparse gaussian processes using pseudo-inputs. Advances in Neural Information Processing Systems, 18, 1257.Google Scholar
  76. Sölch, M., Bayer, J., Ludersdorfer, M., & van der Smagt, P. (2016). Variational inference for on-line anomaly detection in high-dimensional time series. arXiv preprint arXiv:1602.07109.
  77. Song, W.-K., & Kim, J. (2012). Novel assistive robot for self-feeding. Rijeka: INTECH Open Access Publisher.CrossRefGoogle Scholar
  78. Suarez, A., Heredia, G., & Ollero, A., (2016). Cooperative sensor fault recovery in multi-UAV systems. In IEEE international conference on robotics and automation. IEEE.Google Scholar
  79. Suetani, H., Ideta, A. M., & Morimoto, J. (2011). Nonlinear structure of escape-times to falls for a passive dynamic walker on an irregular slope: Anomaly detection using multi-class support vector machine and latent state extraction by canonical correlation analysis. In IEEE/RSJ international conference on intelligent robots and systems (IROS) (pp. 2715–2722). IEEE.Google Scholar
  80. Sukhoy, V., Georgiev, V., Wegter, T., Sweidan, R., & Stoytchev, A., (2012). Learning to slide a magnetic card through a card reader. In IEEE international conference on robotics and automation (ICRA) (pp. 2398–2404). IEEE.Google Scholar
  81. Takahashi, Y., & Suzukawa, S. (2006). Easy human interface for severely handicapped persons and application for eating assist robot. In IEEE international conference on mechatronics (pp. 225–229).
  82. Topping, M. (2002). An overview of the development of handy 1, a rehabilitation robot to assist the severely disabled. Journal of Intelligent and Robotic Systems, 34(3), 253–263.CrossRefzbMATHGoogle Scholar
  83. Vasic, M., & Billard, A. (2013). Safety issues in human–robot interactions. In IEEE international conference on robotics and automation (ICRA) (pp. 197–204). IEEE.Google Scholar
  84. Vaswani, N., Roy-Chowdhury, A. K., & Chellappa, R. (2005). Shape activity: A continuous-state HMM for moving/deforming shapes with application to abnormal activity detection. IEEE Transactions on Image Processing, 14(10), 1603–1616.CrossRefGoogle Scholar
  85. Wakita, Y., Yoon, W. K., & Yamanobe, W. K., (2012). User evaluation to apply the robotic arm rapuda for an upper-limb disabilities patient’s daily life. In IEEE international conference on robotics and biomimetics (ROBIO) (pp. 1482–1487).
  86. Warrender, C., Forrest, S., & Pearlmutter, B. (1999). Detecting intrusions using system calls: Alternative data models. In Proceedings of the IEEE symposium on security and privacy (pp. 133–145). IEEE.Google Scholar
  87. Wiener, J. M., Raymond, W., Hanley, J., Clark, R., Nostrand, V., & Joan, F. (1990). Measuring the activities of daily living: Comparisons across national surveys. Journal of Gerontology, 45(6), S229–S237.CrossRefGoogle Scholar
  88. Williams, G., Baxter, R., He, H., Hawkins, S., & Gu, L. (2002). A comparative study of RNN for outlier detection in data mining. In Proceedings of IEEE international conference on data mining, ICDM (pp. 709–712). IEEE.Google Scholar
  89. Yamazaki, K., Ueda, R., Nozawa, S., Mori, Y., Maki, T., Hatao, N., et al. (2009). A demonstrative research for daily assistive robots on tasks of cleaning and tidying up rooms. In First international symposium on quality of life technology (June 2009).Google Scholar
  90. Yeung, D.-Y., & Ding, Y. (2003). Host-based intrusion detection using dynamic and static behavioral models. Pattern Recognition, 36(1), 229–243.CrossRefzbMATHGoogle Scholar
  91. Yu, D., & Seltzer, M L., (2011). Improved bottleneck features using pretrained deep neural networks. In INTERSPEECH (Vol. 237, p. 240).Google Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2018

Authors and Affiliations

  1. 1.Healthcare Robotics LabGeorgia Institute of TechnologyAtlantaUSA

Personalised recommendations