Skip to main content

Advertisement

SpringerLink
Log in

Multimodal anomaly detection for assistive robots

  • Published:
Autonomous Robots Aims and scope Submit manuscript

Abstract

Detecting when something unusual has happened could help assistive robots operate more safely and effectively around people. However, the variability associated with people and objects in human environments can make anomaly detection difficult. We previously introduced an algorithm that uses a hidden Markov model (HMM) with a log-likelihood detection threshold that varies based on execution progress. We now present an improved version of our previous algorithm (HMM-D) and introduce a new algorithm based on Gaussian process regression (HMM-GP). We also present a new and more thorough evaluation of 8 anomaly detection algorithms with force, sound, and kinematic signals collected from a robot closing microwave doors, latching a toolbox, scooping yogurt, and feeding yogurt to able-bodied participants. Overall, HMM-GP had the highest performance in terms of area under the curve for these real-world tasks, and multiple modalities improved performance with some anomalies being better detected with particular modalities. With synthetic anomalies, HMM-D exhibited shorter detection delays and outperformed HMM-GP with high-magnitude anomalies. In general, higher-magnitude synthetic anomalies tended to be detected more rapidly.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16

Similar content being viewed by others

Notes

  1. The state path of an HMM always starts from the first hidden state, \(\mathbf{z}^1\), setting \(\pi =\{1, 0,\ldots , 0\}\).

  2. The symbols, f, s, and k, in the parentheses represent force, sound, and kinematic modalities, respectively.

  3. Participants were 3 males and 5 females. Their age ranges from 19 to 35. They are either attending or have graduated college.

  4. Kinematic modality refers to the task-kinematic input measured by the encoder and vision sensors (i.e, relative distance or orientation between the PR2 and a target object or human).

References

  • Ando, S., Thanomphongphan, T., Hoshino, D., Seki, Y., & Suzuki, E. (2011). ACE: Anomaly clustering ensemble for multi-perspective anomaly detection in robot behaviors. In Proceedings of the international conference on data mining (pp. 1–12). SIAM.

  • Angelov, V. P., Giglio, C., Guardiola, C., Lughofer, E., & Lujan, J. M. (2006). An approach to model-based fault detection in industrial measurement systems with application to engine test benches. Measurement Science and Technology, 17(7), 1809.

    Article  Google Scholar 

  • Argall, B. D. (2016). Modular and adaptive wheelchair automation. In M. A. Hsieh, O. Khatib, & V. Kumar (Eds.), Experimental robotics (pp. 835–848). Springer.

  • Bishop, C. M. (2006). Pattern recognition and machine learning (information science and statistics). New York, NJ: Springer.

    MATH  Google Scholar 

  • Bittencourt, A. C., Saarinen, K., Sander-Tavallaey, S., Gunnarsson, S., & Norrlöf, M. (2014). A data-driven approach to diagnostics of repetitive processes in the distribution domain-applications to gearbox diagnostics in industrial robots and rotating machines. Mechatronics, 24(8), 1032–1041.

    Article  Google Scholar 

  • Blank, S., Pfister, T., & Berns, K. (2011). Sensor failure detection capabilities in low-level fusion: a comparison between fuzzy voting and kalman filtering. In IEEE international conference on robotics and automation (ICRA) (pp 4974–4979). IEEE.

  • Bouguerra, A., Karlsson, L., & Saffiotti, A. (2008). Monitoring the execution of robot plans using semantic knowledge. Robotics and Autonomous Systems, 56(11), 942–954.

    Article  Google Scholar 

  • Brambilla, D., Capisani, L. M., Ferrara, A., & Pisu, P. (2008). Fault detection for robot manipulators via second-order sliding modes. IEEE Transactions on Industrial Electronics, 55(11), 3954–3963. https://doi.org/10.1109/TIE.2008.2005932. ISSN 0278-0046.

  • Chandola, V., Banerjee, A., & Kumar, V. (2009). Anomaly detection. ACM Computing Surveys, 41(3), 1–58. https://doi.org/10.1145/1541880.1541882, http://dl.acm.org/citation.cfm?id=1541880.1541882, ISSN 03600300.

  • Chen, T. L., Ciocarlie, M., Cousins, S., Grice, P., Hawkins, K., Hsiao, et al. (2013). Robots for humanity: Using assistive robots to empower people with disabilities. IEEE Robotics & Automation Magazine, 20(1), 30–39.

  • Chu, V., McMahon, I., Riano, L., McDonald, C. G., He, Q., Perez-Tejada, J. M., et al. (2013). Using robotic exploratory procedures to learn the meaning of haptic adjectives. In IEEE international conference on robotics and automation (ICRA) (pp. 3048–3055). IEEE.

  • Ciocarlie, M., Hsiao, K., Leeper, A., & Gossow, D. (2012). Mobile manipulation through an assistive home robot. In IEEE/RSJ international conference on intelligent robots and systems. IEEE.

  • Clifton, D. A., Hugueny, S., & Tarassenko, L. (2011). Novelty detection with multivariate extreme value statistics. Journal of Signal Processing Systems, 65(3), 371–389.

    Article  Google Scholar 

  • Copilusi, C., Kaur, M., & Ceccarelli, M. (2015a). Lab experiences with LARM clutched arm for assisting disabled people (pp. 603–611). Cham: Springer International Publishing. https://doi.org/10.1007/978-3-319-09411-3_64. ISBN: 978-3-319-09411-3.

  • Copilusi, C., Kaur, M., & Ceccarelli, M., (2015b). Lab experiences with larm clutched arm for assisting disabled people. In New trends in mechanism and machine science (pp. 603–611). Springer.

  • Cressie, N. A. C. (1993). Statistics for spatial data. New York, CH: Wiley series in probability and mathematical statistics, Wiley. ISBN: 978-0-471-00255-0. http://opac.inria.fr/record=b1085822.

  • Dames, P. M., Schwager, M., & Rus, D. (2016). Active magnetic anomaly detection using multiple micro aerial vehicles. IEEE Robotics and Automation Letters, 1(1), 153–160. https://doi.org/10.1109/LRA.2015.2511444. ISSN: 2377-3766.

  • Di Lello, E., Klotzbucher, M., De Laet, T., & Bruyninckx, H. (2013). Bayesian time-series models for continuous fault detection and recognition in industrial robotic tasks. In IEEE/RSJ international conference on intelligent robots and systems (IROS) (pp. 5827–5833). IEEE.

  • Dua, S., & Du, X. (2011). Data mining and machine learning in cybersecurity. London: CRC Press.

    Book  MATH  Google Scholar 

  • Eclipse Automation. (2016). Meet obi, a robot that helps disabled individuals eat unassisted. https://meetobi.com/. Accessed 15 July 2017.

  • Fagogenis, G., Carolis, V. D., & Lane, D. M. (2016). Online fault detection and model adaptation for underwater vehicles in the case of thruster failures. In IEEE international conference on robotics and automation. IEEE.

  • Fiala, M. (2005). Artag, a fiducial marker system using digital techniques. In IEEE computer society conference on computer vision and pattern recognition (Vol. 2, pp. 590–596). IEEE.

  • Fujii, H., Yamashita, A., & Asama, H., (2016). Defect detection with estimation of material condition using ensemble learning for hammering test. In IEEE international conference on robotics and automation. IEEE.

  • Fukunaga, K. (2013). Introduction to statistical pattern recognition. London: Academic press.

    MATH  Google Scholar 

  • Gehring, J., Miao, Y., Metze, F., & Waibel, A. (2013). Extracting deep bottleneck features using stacked auto-encoders. In Proceedings of IEEE international conference on acoustics, speech and signal processing (ICASSP) (pp. 3377–3381). IEEE.

  • Graf, B., Reiser, U., Hägele, M., Mauz, K., & Klein, P. (2009). Robotic home assistant care-o-bot® 3-product vision and innovation platform. In IEEE workshop on advanced robotics and its social impacts (ARSO) (pp. 139–144). IEEE.

  • Haidu, A., Kohlsdorf, D., & Beetz, M., (2015) Learning action failure models from interactive physics-based simulations. In IEEE/RSJ international conference on intelligent robots and systems (IROS) (pp. 5370–5375). IEEE.

  • Hawkins, K. P., Grice, P. M., Chen, T. L., King, C.-H., & Kemp, C. C. (2014) Assistive mobile manipulation for self-care tasks around the head. In IEEE symposium on computational intelligence in robotic rehabilitation and assistive technologies (pp. 16–25). IEEE.

  • Hoffmann, H. (2007). Kernel PCA for novelty detection. Pattern recognition, 40(3), 863–874.

    Article  MATH  Google Scholar 

  • Hung, Y.-X., Chiang, C.-Y., Hsu, S.J., & Chan, C.-T., (2010). Abnormality detection for improving elders daily life independent. In International conference on smart homes and health telematics (pp. 186–194). Springer.

  • Jain, A., & Kemp, C. C. (2013). Improving robot manipulation with data-driven object-centric models of everyday forces. Autonomous Robots, 35(2–3), 143–159.

    Article  Google Scholar 

  • Jiménez Villarreal, J., & Ljungblad, S. (2011) Experience centred design for a robotic eating aid. In Proceedings of the 6th international conference on human–robot interaction (pp. 155–156). ACM.

  • Kappler, D., Pastor, P., Kalakrishnan, M., Wuthrich, M., & Schaal, S. (2015). Data-driven online decision making for autonomous manipulation. In Proceedings of robotics: Science and systems.

  • Kazemi, V., & Sullivan, J. (2014) One millisecond face alignment with an ensemble of regression trees. In IEEE conference on computer vision and pattern recognition (pp. 1867–1874). https://doi.org/10.1109/CVPR.2014.241.

  • Kemp, C. C., Edsinger, A., & Torres-Jara, E. (2007). Challenges for robot manipulation in human environments [grand challenges of robotics]. IEEE Robotics & Automation Magazine, 14(1), 20–29.

    Article  Google Scholar 

  • Ketabdar, H., Vepa, J, Bengio, S., & Bourlard, H. (2006) Using more informative posterior probabilities for speech recognition. In IEEE international conference on acoustics speech and signal processing proceedings (Vol. 1, pp. 1). IEEE.

  • Khan, S. S., Karg, M. E., Hoey, J., & Kulic, D. (2012). Towards the detection of unusual temporal events during activities using HMMs. In Proceedings of the ACM conference on ubiquitous computing (pp. 1075–1084). ACM.

  • Kim, D. J., Wang, Z., Paperno, N., & Behal, A. (2014). System design and implementation of UCF-MANUS—an intelligent assistive robotic manipulator. IEEE/ASME transactions on mechatronics, 19(1), 225–237. https://doi.org/10.1109/TMECH.2012.2226597. ISSN: 1083-4435.

  • King, C. H., Chen, T. L., Jain, A., & Kemp, C. C. (2010). Towards an assistive robot that autonomously performs bed baths for patient hygiene. In IEEE/RSJ international conference on intelligent robots and systems (pp. 319–324). https://doi.org/10.1109/IROS.2010.5649101.

  • King, D. E. (2009). Dlib-ml: A machine learning toolkit. Journal of Machine Learning Research, 10, 1755–1758.

    Google Scholar 

  • Lee, H., Hwang, B., & Cho, S. (2002). Analysis of novelty detection properties of autoassociative mlp. Journal of Korean Institute of Industrial Engineers, 28(2), 147–161.

    Google Scholar 

  • Leidner, D., Dietrich, A., Beetz, M., & Albu-Schäffer, A. (2016). Knowledge-enabled parameterization of whole-body control strategies for compliant service robots. Autonomous Robots, 40(3), 519–536. https://doi.org/10.1007/s10514-015-9523-3. ISSN: 1573-7527.

  • Lepora, N. F., Pearson, M. J., Mitchinson, B., Evans, M., Fox, C., Pipe, A., Gurney, K., & Prescott, T. J. (2010) Naive bayes novelty detection for a moving robot with whiskers. In IEEE international conference on robotics and biomimetics (ROBIO) (pp. 131–136). IEEE.

  • Lühr, S., Venkatesh, S., West, G., & Bui, H. H. (2004). Explicit state duration HMM for abnormality detection in sequences of human activity. In Pacific rim international conference on artificial intelligence (pp. 983–984). Springer.

  • Malhotra, P., Vig, L., Shroff, G., & Agarwal, P., (2015). Long short term memory networks for anomaly detection in time series. In 23rd European symposium on artificial neural networks, computational intelligence and machine learning (p. 89).

  • Marcolino, F., & Wang, J. (2013). Detecting anomalies in humanoid joint trajectories. In IEEE/RSJ international conference on intelligent robots and systems (IROS) (pp. 2594–2599). IEEE.

  • Markou, M., & Singh, S. (2003). Novelty detection: Areviewpart 1: statistical approaches. Signal Processing, 83(12), 2481–2497.

    Article  MATH  Google Scholar 

  • Mathieu, B., Essid, S., Fillon, T., Prado, J., & Richard, G. (2010). Yaafe, an easy to use and efficient audio feature extraction software. In Proceedings of the 11th international conference on music information retrieval (ISMIR (pp. 441–446).

  • Mealtime Partners. (2017). Specializing in assistive dining and drinking equipment. http://www.mealtimepartners.com/. Accessed July 15, 2017.

  • Mendoza, J. P., Veloso, M., & Simmons, R. (2014). Focused optimization for online detection of anomalous regions. In IEEE international conference on robotics and automation (ICRA) (pp. 3358–3363). IEEE.

  • Morris, B. T., & Trivedi, M. M. (2008). Learning and classification of trajectories in dynamic scenes: A general framework for live video analysis. In IEEE fifth international conference on advanced video and signal based surveillance (AVSS) (pp. 154–161). IEEE.

  • Nguyen, H., Anderson, C., Trevor, A., Jain, A., Xu, Z., & Kemp, C. C. (2008). El-e: An assistive robot that fetches objects from flat surfaces. In Robotic helpers, international conference on human–robot interaction.

  • Niekum, S., Osentoski, S., Atkeson, C. G., & Barto, A. G., (2014). Learning articulation changepoint models from demonstration. In Robotics science and systems (RSS) workshop on learning plans with context from human signals.

  • Ocak, H., & Loparo, K. A. (2005). HMM-based fault detection and diagnosis scheme for rolling element bearings. Journal of Vibration and Acoustics, 127(4), 299–306.

    Article  Google Scholar 

  • Ogorodnikova, O. (2008). Methodology of safety for a human robot interaction designing stage. In Conference on human system interactions (pp. 452–457). IEEE.

  • Papageorgiou, X. S., Tzafestas, C. S., Maragos, P., Pavlakos, G., Chalvatzaki, G., Moustris, G., et al. (2014). Advances in intelligent mobility assistance robot integrating multimodal sensory processing. In International conference on universal access in human–computer interaction (pp. 692–703). Springer.

  • Park, D., Erickson, Z., Bhattacharjee, T., & Kemp, C. C. (2016a). Multimodal execution monitoring for anomaly detection during robot manipulation. In IEEE international conference on robotics and automation. IEEE.

  • Park, D., Kim, Y. K., Erickson, Z., & Kemp, C. C. (2016b). Towards assistive feeding with a general-purpose mobile manipulator. In IEEE international conference on robotics and automation-workshop on human–robot interfaces for enhanced physical interactions.

  • Pastor, P., Kalakrishnan, M., Chitta, E., Theodorou, S., & Schaal, S. (2011). Skill learning and task outcome prediction for manipulation. In IEEE international conference on robotics and automation (ICRA) (pp. 3828–3834). IEEE.

  • Patterson Medical. Meal buddy, (2017). http://pattersonmedical.com/. Accessed: July 15, 2017.

  • Perduca, V., & Nuel, G. (2013). Measuring the influence of observations in HMMs through the kullback-leibler distance. IEEE Signal Processing Letters, 20(2), 145–148.

    Article  Google Scholar 

  • Pettersson, O. (2005). Execution monitoring in robotics: A survey. Robotics and Autonomous Systems, 53(2), 73–88.

    Article  MathSciNet  Google Scholar 

  • Pimentel, M. A. F., Clifton, L., David, A., & Tarassenko, L. (2014). A review of novelty detection. Signal Processing, 99, 215–249.

    Article  Google Scholar 

  • Rabiner, L. R. (1989). A tutorial on hidden markov models and selected applications in speech recognition. In Proceedings of the IEEE (pp. 257–286).

  • Rasmussen, C. E. (2004). Gaussian processes in machine learning. In Advanced lectures on machine learning (pp. 63–71). Springer.

  • Rasmussen, C. E., Kuss, M., et al. (2003). Gaussian processes in reinforcement learning. In Advances in neural information processing systems (Vol. 4, p. 1).

  • Rodriguez, A., Bourne, D., Mason, M., Rossano, G. F., & Wang, J., (2010). Failure detection in assembly: Force signature analysis. In IEEE conference on automation science and engineering (CASE) (pp. 210–215). IEEE.

  • Rodriguez, A., Mason, M. T., Srinivasa, S., Bernstein, M., & Zirbel, A. (2011). Abort and retry in grasping. In IEEE international conference on intelligent robots and systems (IROS).

  • Sakaguchi, T., Yokoi, K., Ujiie, T., Tsunoo, S., & Wada, K. (2009). Design of common environmental information for door-closing tasks with various robots. International Journal of Robotics and Automation, 24(3), 203.

    Google Scholar 

  • Schliep, A., Rungsarityotin, W., & Georgi, B. (2004). General hidden markov model library. http://www.ghmm.org/.

  • Schrer, S., Killmann, I., Frank, B., Vlker, M., Fiederer, L., Ball, T., & Burgard, W. (2015). An autonomous robotic assistant for drinking. In IEEE international conference on robotics and automation (ICRA) (pp. 6482–6487). https://doi.org/10.1109/ICRA.2015.7140110.

  • Serdio, F., Lughofer, E., Pichler, K., Buchegger, T., & Efendic, H. (2014). Residual-based fault detection using soft computing techniques for condition monitoring at rolling mills. Information Sciences, 259, 304–320.

    Article  Google Scholar 

  • Silvrio, J., Rozo, L., Calinon, S., & Caldwell, D. G. (2015). Learning bimanual end-effector poses from demonstrations using task-parameterized dynamical systems. In 2015 IEEE/RSJ international conference on intelligent robots and systems (IROS) (pp. 464–470). https://doi.org/10.1109/IROS.2015.7353413.

  • Simanek, J., Kubelka, V., & Reinstein, M. (2015). Improving multi-modal data fusion by anomaly detection. Autonomous Robots, 39(2), 139–154.

    Article  Google Scholar 

  • Snelson, E., & Ghahramani, Z. (2006). Sparse gaussian processes using pseudo-inputs. Advances in Neural Information Processing Systems, 18, 1257.

    Google Scholar 

  • Sölch, M., Bayer, J., Ludersdorfer, M., & van der Smagt, P. (2016). Variational inference for on-line anomaly detection in high-dimensional time series. arXiv preprint arXiv:1602.07109.

  • Song, W.-K., & Kim, J. (2012). Novel assistive robot for self-feeding. Rijeka: INTECH Open Access Publisher.

    Book  Google Scholar 

  • Suarez, A., Heredia, G., & Ollero, A., (2016). Cooperative sensor fault recovery in multi-UAV systems. In IEEE international conference on robotics and automation. IEEE.

  • Suetani, H., Ideta, A. M., & Morimoto, J. (2011). Nonlinear structure of escape-times to falls for a passive dynamic walker on an irregular slope: Anomaly detection using multi-class support vector machine and latent state extraction by canonical correlation analysis. In IEEE/RSJ international conference on intelligent robots and systems (IROS) (pp. 2715–2722). IEEE.

  • Sukhoy, V., Georgiev, V., Wegter, T., Sweidan, R., & Stoytchev, A., (2012). Learning to slide a magnetic card through a card reader. In IEEE international conference on robotics and automation (ICRA) (pp. 2398–2404). IEEE.

  • Takahashi, Y., & Suzukawa, S. (2006). Easy human interface for severely handicapped persons and application for eating assist robot. In IEEE international conference on mechatronics (pp. 225–229). https://doi.org/10.1109/ICMECH.2006.252529.

  • Topping, M. (2002). An overview of the development of handy 1, a rehabilitation robot to assist the severely disabled. Journal of Intelligent and Robotic Systems, 34(3), 253–263.

    Article  MATH  Google Scholar 

  • Vasic, M., & Billard, A. (2013). Safety issues in human–robot interactions. In IEEE international conference on robotics and automation (ICRA) (pp. 197–204). IEEE.

  • Vaswani, N., Roy-Chowdhury, A. K., & Chellappa, R. (2005). Shape activity: A continuous-state HMM for moving/deforming shapes with application to abnormal activity detection. IEEE Transactions on Image Processing, 14(10), 1603–1616.

    Article  Google Scholar 

  • Wakita, Y., Yoon, W. K., & Yamanobe, W. K., (2012). User evaluation to apply the robotic arm rapuda for an upper-limb disabilities patient’s daily life. In IEEE international conference on robotics and biomimetics (ROBIO) (pp. 1482–1487). https://doi.org/10.1109/ROBIO.2012.6491178.

  • Warrender, C., Forrest, S., & Pearlmutter, B. (1999). Detecting intrusions using system calls: Alternative data models. In Proceedings of the IEEE symposium on security and privacy (pp. 133–145). IEEE.

  • Wiener, J. M., Raymond, W., Hanley, J., Clark, R., Nostrand, V., & Joan, F. (1990). Measuring the activities of daily living: Comparisons across national surveys. Journal of Gerontology, 45(6), S229–S237.

    Article  Google Scholar 

  • Williams, G., Baxter, R., He, H., Hawkins, S., & Gu, L. (2002). A comparative study of RNN for outlier detection in data mining. In Proceedings of IEEE international conference on data mining, ICDM (pp. 709–712). IEEE.

  • Yamazaki, K., Ueda, R., Nozawa, S., Mori, Y., Maki, T., Hatao, N., et al. (2009). A demonstrative research for daily assistive robots on tasks of cleaning and tidying up rooms. In First international symposium on quality of life technology (June 2009).

  • Yeung, D.-Y., & Ding, Y. (2003). Host-based intrusion detection using dynamic and static behavioral models. Pattern Recognition, 36(1), 229–243.

    Article  MATH  Google Scholar 

  • Yu, D., & Seltzer, M L., (2011). Improved bottleneck features using pretrained deep neural networks. In INTERSPEECH (Vol. 237, p. 240).

Download references

Acknowledgements

We thank Youkeun Kim, Zackory Erickson, Ariel Kapusta, Chansu Kim, and Jane Chisholm for their assistance throughout this project. This work was supported in part by NSF Awards IIS-1150157, EFRI-1137229, and NIDILRR Grant 90RE5016-01-00 via RERC TechSAge. Dr. Kemp is a cofounder, a board member, an equity holder, and the CTO of Hello Robot, Inc., which is developing products related to this research. This research could affect his personal financial status. The terms of this arrangement have been reviewed and approved by Georgia Tech in accordance with its conflict of interest policies.

Author information

Authors and Affiliations

  1. Healthcare Robotics Lab, Georgia Institute of Technology, Atlanta, GA, USA

    Daehyung Park, Hokeun Kim & Charles C. Kemp

Authors
  1. Daehyung Park
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Hokeun Kim
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Charles C. Kemp
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Daehyung Park.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Park, D., Kim, H. & Kemp, C.C. Multimodal anomaly detection for assistive robots. Auton Robot 43, 611–629 (2019). https://doi.org/10.1007/s10514-018-9733-6

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10514-018-9733-6

Keywords

Search

Navigation