Pattern Analysis and Applications

, Volume 15, Issue 1, pp 87–111 | Cite as

Continuous activity recognition in a maintenance scenario: combining motion sensors and ultrasonic hands tracking

  • Georg OgrisEmail author
  • Paul Lukowicz
  • Thomas Stiefmeier
  • Gerhard Tröster
Industrial and Commercial Application


We describe the design and evaluation of pattern analysis methods for the recognition of maintenance-related activities. The presented work focuses on the spotting of subtle hand actions in a continuous stream of data based on a combination of body-mounted motion sensors and ultrasonic positioning. The spotting and recognition approach is based on three core ideas: (1) the use of location information to compensate for the ambiguity of hand motions, (2) the use of motion data to compensate for the slow sampling rate and unreliable signal of the low cost ultrasonic positioning system, and (3) an incremental, multistage spotting methodology. The proposed methods are evaluated in an elaborate bicycle repair experiment containing nearly 10 h of data from six subjects. The evaluation compares different strategies and system variants and shows that precision and recall rates around 90% can be achieved.


Contextual computing Hand tracking Location modeling Activity spotting Activity recognition 



This work was supported by the European Union under contract EC IP 004216.


  1. 1.
    Addlesee M, Curwen R, Hodges S, Newman J, Steggles P, Ward A, Hopper A (2001) Implementing a sentient computing system. IEEE Comput Mag 34(8):50–56Google Scholar
  2. 2.
    Amft O (2005) MARKER—a Matlab time-series labeling toolbox.∼oamft/projects/marker/
  3. 3.
    Amft O, Junker H, Tröster G (2005) Detection of eating and drinking arm gestures using inertial body-worn sensors. In: Proceedings of the 9th IEEE international symposium on wearable computers (ISWC’05). Osaka, Japan, pp 160–163. doi: 10.1109/ISWC.2005.17
  4. 4.
    Andrew AH, Anokwa Y, Koscher K, Lester J, Borriello G (2007) Context to make you more aware. In: ICDCS Workshops, p 49. doi: 10.1109/ICDCSW.2007.30
  5. 5.
    Antifakos S, Michahelles F, Schiele B (2002) Proactive instructions for furniture assembly. In: Proceedings of the 4th international conference on ubiquitous computing, Gothenborg, Sweden, pp 351–360Google Scholar
  6. 6.
    Arulampalam S, Maskell S, Gordon N, Clapp T (2001) A tutorial on particle filters for on-line non-linear/non-gaussian Bayesian tracking. IEEE Trans Signal Process 50:174–188CrossRefGoogle Scholar
  7. 7.
    Ashbrook D, Starner T (2003) Using GPS to learn significant locations and predict movement across multiple users. Pers Ubiquitous Comput 7(5):275–286. doi: 10.1007/s00779-003-0240-0 Google Scholar
  8. 8.
    Bannach D, Amft O, Lukowicz P (2008) Rapid prototyping of activity recognition applications. IEEE Pervasive Comput 7(2):22–31. doi: 10.1109/MPRV.2008.36 Google Scholar
  9. 9.
    Bannach D, Kunze K, Lukowicz P (2006) Distributed modular toolbox for multi-modal context recognition. In: Proceedings of 19th international conference on architecture of computing systems (ARCS 2006), Frankfurt/Main, Germany, pp 99–113. doi: 10.1007/11682127
  10. 10.
    Bao L, Intille SS (2004) Activity recognition from user-annotated acceleration data. In: Proceedings of the 2nd international conference on pervasive computing (Pervasive 2004), Linz, Austria, pp 1–17Google Scholar
  11. 11.
    Bilmes JA (1998) A gentle tutorial of the EM algorithm and its application to parameter estimation for gaussian mixture and hidden markov models. Tech. Rep. TR-97-021, Computer Science Division, Department of Electrical Engineering and Computer Science, U.C. BerkeleyGoogle Scholar
  12. 12.
    Brand M, Kettnaker V (2000) Discovery and segmentation of activities in video. IEEE Trans Pattern Anal Mach Intell 22(8):844–851. doi: 10.1109/34.868685 Google Scholar
  13. 13.
    Brashear H, Henderson V, Park KH, Hamilton H, Lee S, Starner T (2006) American sign language recognition in game development for deaf children. In: Proceedings of the 8th international ACM SIGACCESS conference on computers and accessibility (Assets 2006), Portland, OR, USA, pp 79–86Google Scholar
  14. 14.
    Brown RG, Hwang PY (1997) Introduction to random signals and applied Kalman filtering, 3rd edn. Wiley, New YorkGoogle Scholar
  15. 15.
    Cheung KW, So HC, Ma WK, Chan YT (2006) A constrained least squares approach to mobile positioning: algorithms and optimality. EURASIP J Appl Signal Process 2006:1–23Google Scholar
  16. 16.
    Dey A, Salber D, Abowd G (2001) A conceptual framework and a toolkit for supporting the rapid prototyping of context-aware applications. Hum Comput Interact Spec Issue Context Aware Comput 16(2–4):97–166Google Scholar
  17. 17.
    Elhay S, Golub GH, Kautsky J (1991) Updating and downdating of orthogonal polynomials with data fitting applications. SIAM J Matrix Anal Appl 12(2):327–353MathSciNetzbMATHCrossRefGoogle Scholar
  18. 18.
    Gellersen HW, Schmidt A, Beigl M (2002) Multi-sensor context-awareness in mobile devices and smart artifacts. MONET 7(5):341–351zbMATHGoogle Scholar
  19. 19.
    Gupta N, Hauser R (2007) Kalman filtering with equality and inequality state constraints. Tech. rep. Oxford University Computing Laboratory, Oxford, UKGoogle Scholar
  20. 20.
    Harter A, Hopper A, Steggles P, Ward A, Webster P (2002) The anatomy of a context-aware application. Wireless Netw 8(2–3):187–197zbMATHCrossRefGoogle Scholar
  21. 21.
    Hazas M, Kray C, Gellersen H, Agbota H, Kortuem G, Krohn A (2005) A relative positioning system for co-located mobile devices. In: Proceedings of the 3rd international conference on mobile systems, applications, and services (MobiSys 2005), Seattle, pp 177–190. doi: 10.1145/1067170.1067190
  22. 22.
    Jason I, Simon F (2004) Optimized wavelet hand pose estimation for american sign language recognition. In: Proceedings of the congress on evolutionary computation (CEC 2004), Portland, OR, pp 797–802Google Scholar
  23. 23.
    Junker H, Lukowicz P, Tröster G (2004) Continuous recognition of arm activities with wearable motion sensors. In: Proceedings of the 8th IEEE international symposium on wearable computers (ISWC’04), Arlington, VAGoogle Scholar
  24. 24.
    Krause A, Smailagic A, Siewiorek D (2006) Context-aware mobile computing: learning context-dependent personal preferences from a wearable sensor array. IEEE Trans Mobile Comput 5(2):113–127CrossRefGoogle Scholar
  25. 25.
    Kunze K, Wagner F, Kartal E, Kluge EM, Lukowicz P (2009) Does context matter? A quantitative evaluation in a real world maintenance scenario. In: Proceedings of 7th international conference on pervasive computing, Nara, Japan, pp 372–389Google Scholar
  26. 26.
    Lawo M, Herzog O, Witt H (2007) An industrial case study on wearable computing applications. In: Proceedings of the 9th ACM international conference on human computer interaction with mobile devices and services (MobileHCI 2007), Singapore, pp 448–451. doi: 10.1145/1377999.1378052
  27. 27.
    Lee C, Xu Y (1996) Online, interactive learning of gestures for human/robot interfaces. In: Proceedings of the IEEE international conference on robotics and automation (ICRA 1996), Minneapolis, MN, pp 2982–2987. doi: 10.1109/ROBOT.1996.509165
  28. 28.
    Lee HK, Kim JH (1998) Gesture spotting from continuous hand motion. Pattern Recognit Lett 19(5–6):513–520CrossRefGoogle Scholar
  29. 29.
    Lester J, Choudhury T, Borriello G (2006) A practical approach to recognizing physical activities. In: Proceedings of the 4th international conference on pervasive computing (Pervasive 2006), Dublin, pp 1–16Google Scholar
  30. 30.
    Li W, Jonsson M (2006) Providing user self-contained location information using mobile phones. In: Advances in pervasive computing, adjunct proceedings of pervasive 2006, pp 33–36Google Scholar
  31. 31.
    Lin J, Keogh E, Lonardi S, Chiu B (2003) A symbolic representation of time series, with implications for streaming algorithms. In: Proceedings of the 8th ACM SIGMOD workshop on research issues in data mining and knowledge discovery (DMKD 2003), San Diego, CAGoogle Scholar
  32. 32.
    Lin J, Wei EKL, Lonardi S (2007) Experiencing SAX: a novel symbolic representation of time series. Data Min Knowl Discov J 15(2):107–144MathSciNetCrossRefGoogle Scholar
  33. 33.
    Lukowicz P, Kirstein T, Tröster G (2004) Wearable systems for health care applications. Methods Inf Med 43(3):232–230Google Scholar
  34. 34.
    Lukowicz P, Timm-Giel A, Lawo M, Herzog O (2007) WearIT@work: toward real-world industrial wearable computing. IEEE Pervasive Comput 6(4):8–13CrossRefGoogle Scholar
  35. 35.
    Minnen D, Isbell C, Essa I, Starner T (2007) Detecting subdimensional motifs: An efficient algorithm for generalized multivariate pattern discovery. In: Proceedings of the 7th IEEE international conference on data mining (ICDM 2007), Omaha, NE, pp 601–606. doi: 10.1109/ICDM.2007.52
  36. 36.
    Mitra S, Acharya T (2007) Gesture recognition: A survey. IEEE Trans Syst Man Cybern C 37(3):311–324CrossRefGoogle Scholar
  37. 37.
    Müller M, Röder T, Clausen M (2005) Efficient content-based retrieval of motion capture data. In: Proceedings of the 35th international conference on computer graphics and interactive techniques (SIGGRAPH 2005), Los Angeles, CA, pp 677–685. doi: 10.1145/1073204.1073247
  38. 38.
    Naoum-Sawaya J, Slim M, Khawam S, Al-Alaoui MA (2006) Dynamic system design for american sign language recognition. In: Proceedings of the 2nd international symposium on communications, control, and signal processing (ISCCSP 2006), Marakesh, MoroccoGoogle Scholar
  39. 39.
    Ogris G (2009) Multi-modal on-body sensing of human activities. PhD thesis, University of Passau, PassauGoogle Scholar
  40. 40.
    Ogris G, Stiefmeier T, Junker H, Lukowicz P, Tröster G (2005) Using ultrasonic hand tracking to augment motion analysis based recognition of manipulative gestures. In: Proceedings of the 9th IEEE international symposium on wearable computers (ISWC’05), Osaka, Japan, pp 152–159. doi: 10.1109/ISWC.2005.54
  41. 41.
    Patterson DJ, Fox D, Kautz H, Philipose M (2005) Fine-grained activity recognition by aggregating abstract object usage. In: Proceedings of the 9th IEEE international symposium on wearable computers (ISWC’05), Osaka, Japan, pp 44–51. doi: 10.1109/ISWC.2005.22
  42. 42.
    Poppe R (2010) A survey on vision-based human action recognition. Image Vis Comput 28(6):976–990CrossRefGoogle Scholar
  43. 43.
    Priyantha NB (2005) The cricket indoor location system. PhD thesis, Massachusetts Institute of Technology, Cambridge, MAGoogle Scholar
  44. 44.
    Roetenberg D (2006) Inertial and magnetic sensing of human motion. PhD thesis, University of Twente, EnschedeGoogle Scholar
  45. 45.
    Roetenberg D, Veltink PH (2005) Camera-marker and inertial sensor fusion for improved motion tracking. In: Proceedings of the European Society of Movement Analysis for Adults and Children (ESMAC 2005), Barcelona, pp 51–52Google Scholar
  46. 46.
    Schölkopf B, Smola AJ (2002) Learning with kernels: support vector machines, regularization, optimization, and beyond (Adaptive computation and machine learning). MIT Press, CambridgeGoogle Scholar
  47. 47.
    Siewiorek D, Smailagic A, Starner T (2008) Application design for wearable computing. Synthesis lectures on mobile and pervasive computing. Morgan and Claypool PublishersGoogle Scholar
  48. 48.
    Smith A, Balakrishnan H, Goraczko M, Priyantha N (2004) Tracking moving devices with the Cricket location system. In: Proceedings of the 2nd international conference on mobile systems, applications, and services (MobiSys 2004), Boston, MA, pp 190–202. doi: 10.1145/990064.990088
  49. 49.
    Starner T, Schiele B, Pentland A (1998) Visual contextual awareness in wearable computing. In: Proceedings of the 2nd international symposium on wearable computers (ISWC’98), Pittsburgh, PA, pp 50–57. doi: 10.1109/ISWC.1998.729529
  50. 50.
    Starner T, Weaver J, Pentland A (1998) Real-time American sign language recognition using desk and wearable computer based video. IEEE Trans Pattern Anal Mach Intell 20(12):1371–1375. doi: 10.1109/34.735811
  51. 51.
    Stiefmeier T (2008) Real-time spotting of human activities in industrial environments. PhD thesis, ETH Zürich, ZurichGoogle Scholar
  52. 52.
    Stiefmeier T, Ogris G, Junker H, Lukowicz P, Tröster G (2006) Combining motion sensors and ultrasonic hands tracking for continuous activity recognition in a maintenance scenario. In: Proceedings of the 10th IEEE international symposium on wearable computers (ISWC’06), Montreux, pp 97–104. doi: 10.1109/ISWC.2006.286350
  53. 53.
    Stiefmeier T, Roggen D, Ogris G, Lukowicz P, Tröster G (2008) Wearable activity tracking in car manufacturing. IEEE Pervasive Comput Spec Issue Activity Based Comput 7(2):42–50Google Scholar
  54. 54.
    Stiefmeier T, Roggen D, Tröster G (2007) Fusion of string-matched templates for continuous activity recognition. In: Proceedings of the 11th IEEE international symposium on wearable computers (ISWC’07), Boston, MA, pp 41–44Google Scholar
  55. 55.
    Stiefmeier T, Roggen D, Tröster G (2007) Gestures are strings: Efficient online gesture spotting and classification using string matching. In: Proceedings of the 2nd international conference on body area networks (BodyNets 2007), Florence, ItalyGoogle Scholar
  56. 56.
    Ungarala S, Dolence E, Li K (2007) Constrained extended Kalman filter for nonlinear state estimation. In: Proceedings of the 8th international IFAC symposium on dynamics and control of process systems (DYCOPS 2007), Cancún, pp 63–68Google Scholar
  57. 57.
    Ward A (1998) Sensor-driven computing. PhD thesis, University of Cambridge, Cambridge, UKGoogle Scholar
  58. 58.
    Ward A, Jones A, Hopper A (1997) A new location technique for the active office. IEEE Personal Commun 4(5):42–47CrossRefGoogle Scholar
  59. 59.
    Ward JA (2006) Activity monitoring: continuous recognition and performance evaluation. PhD thesis, ETH Zürich, ZurichGoogle Scholar
  60. 60.
    Ward JA, Lukowicz P, Tröster G, Starner TE (2006) Activity recognition of assembly tasks using body-worn microphones and accelerometers. IEEE Trans Pattern Anal Mach Intell 28(10):1553–1567. doi: 10.1109/TPAMI.2006.197 Google Scholar
  61. 61.
    Westeyn T, Vadas K, Bian X, Starner T, Abowd G (2005) Recognizing mimicked autistic self-stimulatory behaviors using HMMs. In: Proceedings of the 9th IEEE international symposium on wearable computers (ISWC’05), pp 164–169Google Scholar
  62. 62.
    Wijnalda G, Pauws S, Vignoli F, Stuckenschmidt H (2005) A personalized music system for motivation in sport performance. IEEE Pervasive Comput 4(3):26–32CrossRefGoogle Scholar
  63. 63.
    Yamato J, Ohya J, Ishii K (1992) Recognizing human action in time-sequential images using Hidden Markov Models. In: Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR 1992), Champaign, IL, pp 379–385. doi: 10.1109/CVPR.1992.223161
  64. 64.
    Yoon HS, Soh J, Bae YJ, Yang HS (2001) Hand gesture recognition using combined features of location, angle and velocity. Pattern Recognit 34(7):1491–1501. doi: 10.1016/S0031-3203(00)00096-0
  65. 65.
    Zinnen A, Laerhoven KV, Schiele B (2007) Toward recognition of short and non-repetitive activities from wearable sensors. In: Proceedings of the European conference on ambient intelligence (AMI 2007), Darmstadt, pp 142–158Google Scholar
  66. 66.
    Zinnen A, Wojek C, Schiele B (2009) Multi activity recognition based on bodymodel-derived primitives. In: 4th international symposium on location and context awareness (LoCA 2009), TokyoGoogle Scholar

Copyright information

© Springer-Verlag London Limited 2011

Authors and Affiliations

  • Georg Ogris
    • 1
    Email author
  • Paul Lukowicz
    • 2
  • Thomas Stiefmeier
    • 3
  • Gerhard Tröster
    • 3
  1. 1.Department of Biomechanics, Kinesiology and Applied Computer ScienceUniversity of ViennaViennaAustria
  2. 2.Embedded Systems Lab. (ESL)University of PassauPassauGermany
  3. 3.Wearable Computing Lab. (WearLab)ETH ZürichZurichSwitzerland

Personalised recommendations