Skip to main content
Log in

Collegial activity learning between heterogeneous sensors

  • Regular Paper
  • Published:
Knowledge and Information Systems Aims and scope Submit manuscript

Abstract

Activity recognition algorithms have matured and become more ubiquitous in recent years. However, these algorithms are typically customized for a particular sensor platform. In this paper, we introduce PECO, a Personalized activity ECOsystem, that transfers learned activity information seamlessly between sensor platforms in real time so that any available sensor can continue to track activities without requiring its own extensive labeled training data. We introduce a multi-view transfer learning algorithm that facilitates this information handoff between sensor platforms and provide theoretical performance bounds for the algorithm. In addition, we empirically evaluate PECO using datasets that utilize heterogeneous sensor platforms to perform activity recognition. These results indicate that not only can activity recognition algorithms transfer important information to new sensor platforms, but any number of platforms can work together as colleagues to boost performance.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17

Similar content being viewed by others

References

  1. Solanas A, Patsakis C, Conti M, Vlachos I, Ramos V, Falcone F, Postolache O, Perez-Martinez PA, Di Pietro R, Perrea DN, Martinez-Balleste A (2014) Smart health: a context-aware health paradigm within smart cities. IEEE Commun Mag 52(8):74–81

    Article  Google Scholar 

  2. Aggarwal JK, Ryoo MS (2011) Human activity analysis: a review. ACM Comput Surv 43(3):1–47

    Article  Google Scholar 

  3. Chen L, Hoey J, Nugent CD, Cook DJ, Yu Z (2012) Sensor-based activity recognition. IEEE Trans Syst Man Cybern Part C Appl Rev 42(6):790–808

    Article  Google Scholar 

  4. Ke S-R, Thuc HLU, Lee Y-J, Hwang J-N, Yoo J-H, Choi K-H (2013) A review on video-based human activity recognition. Computers 2(2):88–131

    Article  Google Scholar 

  5. Bulling A, Blanke U, Schiele B (2014) A tutorial on human activity recognition using body-worn inertial sensors. ACM Comput Surv 46(3):107–140

    Article  Google Scholar 

  6. Reiss A, Stricker D, Hendeby G (2013) Towards robust activity recognition for everyday life: methods and evaluation. In: Pervasive Computing Technologies for Healthcare, pp. 25–32

  7. Vishwakarma S, Agrawal A (2013) A survey on activity recognition and behavior understanding in video surveillance. Vis Comput 29(10):983–1009

    Article  Google Scholar 

  8. Cook D (2012) Learning setting-generalized activity models for smart spaces. IEEE Intell Syst 27(1):32–38

    Article  Google Scholar 

  9. Hagras H, Doctor F, Lopez A, Callaghan V (2007) An incremental adaptive life long learning approach for type-2 fuzzy embedded agents in ambient intelligent environments. IEEE Trans Fuzzy Syst 15(1):41–55

    Article  Google Scholar 

  10. Munguia-Tapia E, Intille SS, Larson K (2004) Activity recognition in the home using simple and ubiquitous sensors. In: Pervasive, pp. 158–175

  11. van Kasteren T, Noulas A, Englebienne G, Krose B (2008) Accurate activity recognition in a home setting. In: ACM conference on ubiquitous computing, pp. 1–9

  12. He W, Guo Y, Gao C, Li X (2012) Recognition of human activities with wearable sensors. J Adv Signal Process 108:1–13

    Google Scholar 

  13. Junker H, Amft O, Lukowicz P, Groster G (2008) Gesture spotting with body-worn inertial sensors to detect user activities. Pattern Recognit 41:2010–2024

    Article  MATH  Google Scholar 

  14. Maurer U, Smailagic A, Siewiorek D, Deisher M (2006) Activity recognition and monitoring using multiple sensors on different body positions. In: Proceedings of the international workshop on wearable and implantable body sensor networks, pp. 113–116

  15. Bulling A, Ward JA, Gellersen H (2012) Multimodal recognition of reading activity in transit using body-worn sensors. ACM Trans Appl Percept 9(1):2:1–2:21

    Article  Google Scholar 

  16. Lara O, Labrador MA (2013) A survey on human activity recognition using wearable sensors. IEEE Commun Surv Tutor. 15(3):1192–1209

    Article  Google Scholar 

  17. Gu T, Chen S, Tao X, Lu J (2010) An unsupervised approach to activity recognition and segmentation based on object-use fingerprints. Data Knowl Eng 69(6):533–544

    Article  Google Scholar 

  18. Philipose M, Fishkin KP, Perkowitz M, Patterson DJ, Hahnel D, Fox D, Kautz H (2004) Inferring activities from interactions with objects. IEEE Pervasive Comput 3(4):50–57

    Article  Google Scholar 

  19. Buettner M, Prasad R, Philipose M, Wetherall D (2009) Recognizing daily activities with RFID-based sensors. In: International conference on ubiquitous computing, pp. 51–60

  20. Kwapisz J, Weiss G, Moore S (2010) Activity recognition using cell phone accelerometers. In: International workshop on knowledge discovery from sensor data, pp. 10–18

  21. Zhixian Y, Vigneshwaran S, Chakraborty D, Misra A, Aberer K (2012) Energy-efficient continuous activity recognition on mobile phones: an activity-adaptive approach. In: International symposium on wearable computers, pp. 17–24

  22. Moncrieff S, Venkatesh S, West G, Greenhill S (2007) Multi-modal emotive computing in a smart house environment. Pervasive Mob Comput 3(2):79–94

    Article  Google Scholar 

  23. Candamo J, Shreve M, Goldgof D, Sapper D, Kasturi R (2010) Understanding transit scenes: a survey on human behavior recognition algorithms. IEEE Trans Intell Transp Syst 11(1):206–224

    Article  Google Scholar 

  24. Gill T, Keller JM, Anderson DT, Luke RH (2011) A system for change detection and human recognition in voxel space using the Microsoft Kinect sensor. In: IEEE applied imagery pattern recognition workshop, pp. 1–8

  25. Malgireddy MR, Nwogu I, Govindaraju V (2013) Language-motivated approaches to action recognition. J Mach Learn Res 14:2189–2212

    MathSciNet  Google Scholar 

  26. Wang J, Liu Z, Wu Y (2013) Learning actionlet ensemble for 3D human action recognition. IEEE Trans Pattern Anal Mach Intell 36(5):914–927

    Article  Google Scholar 

  27. Bao L, Intille S (2004) Activity recognition from user annotated acceleration data. In: Pervasive, pp. 1–17

  28. Ravi N, Dandekar N, Mysore P, Littman ML (2005) Activity recognition from accelerometer data. In: Innovative applications of artificial intelligence, pp. 1541–1546

  29. Ward JA, Lukowicz P, Troster G, Starner TE (2006) Activity recognition of assembly tasks using body-worn microphones and accelerometers. IEEE Trans Pattern Anal Mach Intell 28(10):1553–1567

    Article  Google Scholar 

  30. Singla G, Cook DJ, Schmitter-Edgecombe M (2010) Recognizing independent and joint activities among multiple residents in smart environments. Ambient Intell Humaniz Comput J 1(1):57–63

    Article  Google Scholar 

  31. Lester J, Choudhury T, Kern N, Borriello G, Hannaford B (2005) A hybrid discriminative/generative approach for modeling human activities. In: International joint conference on artificial intelligence, pp. 766–772

  32. Amft O, Troster G (2009) On-body sensing solutions for automatic dietary monitoring. IEEE Pervasive Comput 8:62–70

    Article  Google Scholar 

  33. Blanke U, Schiele B, Kreil M, Lukowicz P, Sick B, Gruber T (2010) All for one or one for all? Combining heterogeneous features for activity spotting. In: IEEE international conference on pervasive computing and communications workshops, pp. 18–24

  34. Wang S, Pentney W, Popescu AM, Choudhury T, Philipose M (2007) Common sense based joint training of human activity recognizers. In: International joint conference on artificial intelligence, pp. 2237–2242

  35. Lester J, Choudhury T, Borriello G (2006) A practical approach to recognizing physical activities. In: International conference on pervasive computing, pp. 1–16

  36. Krishnan N, Cook DJ (2014) Activity recognition on streaming sensor data. Pervasive Mob Comput 10:138–154

    Article  Google Scholar 

  37. Arnold A, Nallapati R, Cohen WW (2007) A comparative study of methods for transductive transfer learning. In: International conference on data mining

  38. Elkan C (2011) The foundations of cost-sensitive learning. In: International joint conference on artificial intelligence, pp. 973–978

  39. Thrun S (1996) Explanation-based neural network learning: a lifelong learning approach. Kluwer Academic Publishers, Bostom, MA

  40. Thrun S, Pratt L (1998) Learning to learn. Kluwer Academic Publishers, Bostom, MA

  41. Vilalta R, Drissi Y (2002) A prospective view and survey of meta-learning. Artif Intell Rev 18:77–95

    Article  Google Scholar 

  42. Raina R, Battle A, Lee H, Packer B, Ng AY (2007) Self-taught learning: transfer learning from unlabeled data. In: International conference on machine learning, pp. 759–766

  43. Hachiya H, Sugiyama M, Ueda N (2012) Importance-weighted least-squares probabilistic classifier for covariate shift adaptation with application to human activity recognition. Neurocomputing 80:93–101

    Article  Google Scholar 

  44. Pan SJ, Yang Q (2010) A survey on transfer learning. IEEE Trans Knowl Data Eng 22(10):1345–1359

    Article  Google Scholar 

  45. Duan L, Xu D, Tsang I, Luo J (2012) Visual event recognition in videos by learning from web data. IEEE Trans Pattern Anal Mach Intell 34(9):1667–1680

    Article  Google Scholar 

  46. Wei B, Pal C (2011) Heterogeneous transfer learning with RBMs. In: AAAI conference on artificial intelligence, pp. 531–536

  47. Yang W, Wang Y, Mori G (2011) Learning transferable distance functions for human action recognition. In: Machine learning for vision-based motion analysis, pp. 349–370

  48. Feuz K, Cook DJ (2015) Transfer learning across feature-rich heterogeneous feature spaces via feature-space remapping. ACM Trans Intell Syst Technol 6(1):3

  49. Zhao Z, Chen Y, Liu J, Shen Z, Liu M (2011) Cross-people mobile-phone based activity recognition. In: International joint conference on artificial intelligence, pp. 2545–2550

  50. Cook DJ, Feuz K, Krishnan N (2013) Transfer learning for activity recognition: a survey. Knowl Inf Syst 36(3):537–556

    Article  Google Scholar 

  51. Ren Y, Wu Y, Ge Y (2014) A co-training algorithm for EEG classification with biomimetic pattern recognition and sparse representation. Neurocomputing 2(137):212–222

    Article  Google Scholar 

  52. Kurz M, Holzl G, Ferscha A, Calatroni A, Roggen D, Troster G (2011) Real-time transfer and evaluation of activity recognition capabilities in an opportunistic system. In: International conference on adaptive and self-adaptive systems and applications, pp. 73–78

  53. Hu DH, Yang Q (2011) Transfer learning for activity recognition via sensor mapping. In: International joint conference on artificial intelligence, pp. 1962–1967

  54. Daume H, Marcu D (2006) Domain adaptation for statistical classifiers. J Artif Intell Res 26(1):101–126

    MathSciNet  MATH  Google Scholar 

  55. Daume H, Kumar A, Saha A (2010) Co-regularization based semi-supervised domain adaptation. In: Advances in neural information processing systems, pp. 478–486

  56. Blitzer J, Dredze M, Pereira F (2007) Biographies, bollywood, boom-boxes and blenders: domain adaptation for sentiment classification. In: Annual meeting of the association for computational linguistics

  57. Zhong E, Fan W, Peng J, Zhang K, Ren J, Turaga D, Verscheure O (2009) Cross domain distribution adaptation via kernel mapping. In: ACM SIGKDD international conference on knowledge discovery and data mining, pp. 1027–1036

  58. Prettenhofer P, Stein B (2011) Cross-lingual adaptation using structural correspondence learning. ACM Trans Intell Syst Technol 3(1):13

    Article  Google Scholar 

  59. Zhou JT, Pan SJ, Tsang IW, Yan Y (2014) Hybrid heterogeneous transfer learning through deep learning. In: AAAI conference on artificial intelligence, pp. 2213–2219

  60. Shi X, Yu P (2012) Dimensionality reduction on heterogeneous feature space. In: International conference on data mining, pp. 635–644

  61. Yang Q, Chen Y, Xue G-R, Dai W, Yu Y (2009) Heterogeneous transfer learning for image clustering via the social web. In: Joint conference of the annual meeting of the ACL, pp. 1–9

  62. Kira Z (2010) Inter-robot transfer learning for perceptual classification. In: International conference on autonomous agents and multiagent systems, pp. 13–20

  63. Blum A, Mitchell T (1998) Combining labeled and unlabeled data with co-training. In: Annual conference on computational learning theory, pp. 92–100

  64. Sun S (2013) A survey of multi-view machine learning. Neural Comput Appl 23(7–8):2031–2038

    Article  Google Scholar 

  65. Wang C, Mahadevan S (2008) Manifold alignment using procrustes analysis. In: International conference on machine learning, pp. 1120–1127

  66. Pansiot J, Stoyanov D, McIlwraith D, Lo BPL, Yang GZ (2007) Ambient and wearable sensor fusion for activity recognition in healthcare monitoring systems. In: International workshop on wearable and implantable body sensor networks, pp. 208–212

  67. Ugolotti R, Sassi F, Mordonini M, Cagnoni S (2013) Multi-sensor system for detection and classification of human activities. J Ambient Intell Humaniz Comput 1(4):27–41

    Article  Google Scholar 

  68. Banos O, Damas M, Pomares H, Rojas I (2013) Activity recognition based on a multi-sensor meta-classifier. In: Advances in computational intelligence: international work conference on artificial neural networks, pp. 208–215

  69. Nigam K, Ghani R (2000) Analyzing the effectiveness and applicability of co-training. In: International conference on information and knowledge management, pp. 86–93

  70. Kakade SM, Foster DP (2007) Multi-view regression via canonical correlation analysis. In: Learning theory, Springer, pp. 82–96

  71. Wall ME, Rechtsteiner A, Rocha LM (2003) Singular value decomposition and principal component analysis. In: A practical approach to microarray data analysis, pp. 91–109

  72. Valiant LG (1984) A theory of the learnable. CACM 27(11):1134–1142

    Article  MATH  Google Scholar 

  73. Cook DJ, Crandall A, Thomas B, Krishnan N (2012) CASAS: a smart home in a box. IEEE Comput 46(7):62–69

    Article  Google Scholar 

  74. Pedregosa F, Varoquaux G, Framfort A, Michel V, Thirion B, Grisel O, Blondel M, Prettenhofer P, Weiss R, Dubourg V, Vanderplas J, Passos A, Cournapeau D, Brucher M, Perrot M, Duchesnay E (2011) Scikit-learn: machine learning in Python. J Mach Learn Res 12:2825–2830

    MathSciNet  MATH  Google Scholar 

  75. Roggen D, Frster K, Calatroni A, Trster G (2013) The adARC pattern analysis architecture for adaptive human activity recognition systems. J Ambient Intell Humaniz Comput 4:169–186

    Article  Google Scholar 

  76. Sagha H, Digumarti ST, del R. Millan J, Chavarriaga R, Calatroni A, Roggen D, Troster G (2011) Benchmarking classification techniques using the opportunity human activity dataset. In: IEEE international conference on systems, man, and cybernetics, pp. 36–40

  77. Sahaf Y, Krishnan N, Cook DJ (2011) Defining the complexity of an activity. In: Workshop AAAI on techniques and languages, activity context representation

Download references

Acknowledgements

The authors would like to thank Aaron Crandall and Biswaranjan Das for their help in collecting and processing Kinect data. Funding for research was provided by the National Science Foundation (DGE-0900781, IIS-1064628) and by the National Institutes of Health (R01EB015853).

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Diane J. Cook.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Feuz, K.D., Cook, D.J. Collegial activity learning between heterogeneous sensors. Knowl Inf Syst 53, 337–364 (2017). https://doi.org/10.1007/s10115-017-1043-3

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10115-017-1043-3

Keywords

Navigation