Object-Based Activity Recognition with Heterogeneous Sensors on Wrist

  • Takuya Maekawa
  • Yutaka Yanagisawa
  • Yasue Kishino
  • Katsuhiko Ishiguro
  • Koji Kamei
  • Yasushi Sakurai
  • Takeshi Okadome
Part of the Lecture Notes in Computer Science book series (LNCS, volume 6030)


This paper describes how we recognize activities of daily living (ADLs) with our designed sensor device, which is equipped with heterogeneous sensors such as a camera, a microphone, and an accelerometer and attached to a user’s wrist. Specifically, capturing a space around the user’s hand by employing the camera on the wrist mounted device enables us to recognize ADLs that involve the manual use of objects such as making tea or coffee and watering plant. Existing wearable sensor devices equipped only with a microphone and an accelerometer cannot recognize these ADLs without object embedded sensors. We also propose an ADL recognition method that takes privacy issues into account because the camera and microphone can capture aspects of a user’s private life. We confirmed experimentally that the incorporation of a camera could significantly improve the accuracy of ADL recognition.


Wearable sensors Recognizing daily activities Experiment 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Ahmad, F., Musilek, P.: A keystroke and pointer control input interface for wearable computers. In: Proc. PerCom 2006, pp. 2–11 (2006)Google Scholar
  2. 2.
    Bao, L., Intille, S.S.: Activity recognition from user-annotated acceleration data. In: Ferscha, A., Mattern, F. (eds.) PERVASIVE 2004. LNCS, vol. 3001, pp. 1–17. Springer, Heidelberg (2004)Google Scholar
  3. 3.
    Blum, M., Pentland, A.S., Troster, G.: Insense: Interest-based life logging. IEEE Multimedia 13(4), 40–48 (2006)CrossRefGoogle Scholar
  4. 4.
    Bouten, C.V., et al.: A triaxial accelerometer and portable data processing unit for the assessment of daily physical activity. IEEE Trans. on Bio-Medical Engineering 44(3), 136–147 (1997)CrossRefGoogle Scholar
  5. 5.
    Chen, J., Kam, A.H., Zhang, J., Liu, N., Shue, L.: Bathroom activity monitoring based on sound. In: Gellersen, H.-W., Want, R., Schmidt, A. (eds.) PERVASIVE 2005. LNCS, vol. 3468, pp. 47–61. Springer, Heidelberg (2005)Google Scholar
  6. 6.
    Clarkson, B., Mase, K., Pentland, A.: Recognizing user context via wearable sensors. In: Proc. ISWC 2000, pp. 69–75 (2000)Google Scholar
  7. 7.
    Comaniciu, D., Ramesh, V., Meer, P.: Kernel-based object tracking. IEEE Trans. on Pattern Analysis Machine Intelligence 25(5), 564–577 (2003)CrossRefGoogle Scholar
  8. 8.
    Cowling, M.: Non-speech environmental sound recognition system for autonomous surveillance. Ph.D. Thesis, Griffith University, Gold Coast Campus (2004)Google Scholar
  9. 9.
    Fitzpatrick, P., Kemp, C.C.: Shoes as a platform for vision. In: Proc. ISWC 2003, pp. 231–234 (2003)Google Scholar
  10. 10.
    Froehlich, J.E., Larson, E., Campbell, T., Haggerty, C., Fogarty, J., Patel, S.N.: HydroSense: Infrastructure-mediated single-point sensing of whole-home water activity. In: Proc. Ubicomp 2009, pp. 235–244 (2009)Google Scholar
  11. 11.
    Huynh, T., Schiele, B.: Towards less supervision in activity recognition from wearable sensors. In: Proc. ISWC 2006, pp. 3–10 (2006)Google Scholar
  12. 12.
    Intille, S.S., Tapia, E.M., Rondoni, J., Beaudin, J., Kukla, C., Agarwal, S., Bao, L., Larson, K.: Tools for studying behavior and technology in natural settings. In: Dey, A.K., Schmidt, A., McCarthy, J.F. (eds.) UbiComp 2003. LNCS, vol. 2864, pp. 157–174. Springer, Heidelberg (2003)Google Scholar
  13. 13.
    Jaakkola, T., Haussler, D.: Exploiting generative models in discriminative classifiers. In: Proc. Advances in Neural Information Processing Systems, vol. 11, pp. 487–493 (1999)Google Scholar
  14. 14.
    Kasteren, T.V., Noulas, A., Englebienne, G., Krose, B.: Accurate activity recognition in a home setting. In: Proc. UbiComp 2008, pp. 1–9 (2008)Google Scholar
  15. 15.
    Lester, J., Choudhury, T., Kern, N., Borriello, G., Hannaford, B.: A hybrid discriminative/generative approach for modeling human activities. In: Proc. IJCAI 2005, pp. 766–772 (2005)Google Scholar
  16. 16.
    Lester, J., Choudhury, T., Borriello, G.: A practical approach to recognizing physical activities. In: Fishkin, K.P., Schiele, B., Nixon, P., Quigley, A. (eds.) PERVASIVE 2006. LNCS, vol. 3968, pp. 1–16. Springer, Heidelberg (2006)CrossRefGoogle Scholar
  17. 17.
    Logan, B., Healey, J., Philipose, M., Tapia, E.M., Intille, S.S.: A long-term evaluation of sensing modalities for activity recognition. In: Krumm, J., Abowd, G.D., Seneviratne, A., Strang, T. (eds.) UbiComp 2007. LNCS, vol. 4717, pp. 483–500. Springer, Heidelberg (2007)CrossRefGoogle Scholar
  18. 18.
    Lowe, D.G.: Distinctive image features from scale–invariant keypoints. Int’l Journal on Computer Vision 60(2), 91–110 (2004)CrossRefGoogle Scholar
  19. 19.
    Lukowicz, P., Junker, H., et al.: WearNET: a distributed multi-sensor system for context aware wearables. In: Borriello, G., Holmquist, L.E. (eds.) UbiComp 2002. LNCS, vol. 2498, pp. 361–370. Springer, Heidelberg (2002)CrossRefGoogle Scholar
  20. 20.
    Lukowicz, P., Ward, J., Junker, H., Stager, M., Troster, G., Atrash, A., Starner, T.: Recognizing workshop activity using body worn microphones and accelerometers. In: Ferscha, A., Mattern, F. (eds.) PERVASIVE 2004. LNCS, vol. 3001, pp. 18–32. Springer, Heidelberg (2004)Google Scholar
  21. 21.
    Maekawa, T., Yanagisawa, Y., Kishino, Y., Kamei, K., Sakurai, Y., Okadome, T.: Object-blog system for environment-generated content. IEEE Pervasive Computing 7(4), 20–27 (2008)CrossRefGoogle Scholar
  22. 22.
    Mayol, W.W., Murray, D.W.: Wearable hand activity recognition for event summarization. In: Proc. ISWC 2005, pp. 122–129 (2005)Google Scholar
  23. 23.
    Mihailidis, A., Carmichael, B., Boger, J.: The use of computer vision in an intelligent environment to support aging-in-place, safety, and independence in the home. IEEE Trans. on Info. Tech. in BioMedicine 8(3), 238–247 (2004)CrossRefGoogle Scholar
  24. 24.
    Morikawa, S., Ito, K., Shibata, T.: A k-means VLSI processor and its application to autonomous area segmentation in images. IEIC Technical Report 106(342), 19–24 (2006)Google Scholar
  25. 25.
    Philipose, M., Fishkin, K.P., Perkowitz, M.: Inferring activities from interactions with objects. IEEE Pervasive Computing 3(4), 50–57 (2004)CrossRefGoogle Scholar
  26. 26.
    Raina, R., Shen, Y., Ng, A.Y., McCallum, A.: Classification with hybrid generative/discriminative models. In: Proc. Advances in Neural Information Processing Systems, vol. 16 (2003)Google Scholar
  27. 27.
    Schiele, B., James, L.C.: Object recognition using multidimensional receptive field histograms. In: Proc. European Conference on Computer Vision, pp. 610–619 (1996)Google Scholar
  28. 28.
    Shi, Y., Huang, Y., Minnen, D., Bobick, A., Essa, I.: Propagation networks for recognition of partially ordered sequential action. In: Proc. CVPR 2004, vol. 2, pp. 862–869 (2004)Google Scholar
  29. 29.
    Starner, T., Schiele, B., Pentland, A.: Visual contextual awareness in wearable computing. In: Proc. ISWC 1998, pp. 50–57 (1998)Google Scholar
  30. 30.
    Swain, M.J., Ballard, D.H.: Color indexing. Int’l Journal of Computer Vision 7, 11–32 (1991)CrossRefGoogle Scholar
  31. 31.
    Tapia, E.M., Intille, S.S., Larson, K.: Activity recognition in the home using simple and ubiquitous sensors. In: Ferscha, A., Mattern, F. (eds.) PERVASIVE 2004. LNCS, vol. 3001, pp. 158–175. Springer, Heidelberg (2004)Google Scholar
  32. 32.
    Tapia, E.M., Intille, S.S., Larson, K.: Portable wireless sensors for object usage sensing in the home: challenges and practicalities. In: Schiele, B., Dey, A.K., Gellersen, H., de Ruyter, B., Tscheligi, M., Wichert, R., Aarts, E., Buchmann, A. (eds.) AmI 2007. LNCS, vol. 4794, pp. 19–37. Springer, Heidelberg (2007)CrossRefGoogle Scholar
  33. 33.
    Welk, G., Differding, J.: The utility of the Digi-Walker step counter to assess daily physical activity patterns. Medicine & Science in Sports & Exercise 32(9), S481–S488 (2000)CrossRefGoogle Scholar
  34. 34.
    Witten, I.H., Frank, E.: Data Mining: Practical machine learning tools and techniques, 2nd edn. Morgan Kaufmann, San Francisco (2005)MATHGoogle Scholar
  35. 35.
    Wu, J., Osuntogun, A., et al.: A scalable approach to activity recognition based on object use. In: Proc. ICCV 2007, pp. 1–8 (2007)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2010

Authors and Affiliations

  • Takuya Maekawa
    • 1
  • Yutaka Yanagisawa
    • 1
  • Yasue Kishino
    • 1
  • Katsuhiko Ishiguro
    • 1
  • Koji Kamei
    • 1
  • Yasushi Sakurai
    • 1
  • Takeshi Okadome
    • 1
  1. 1.NTT Communication Science LaboratoriesKyotoJapan

Personalised recommendations