Intelligent Service Robotics

, Volume 1, Issue 3, pp 185–193 | Cite as

Beat gesture recognition and finger motion control of a piano playing robot for affective interaction of the elderly

  • Kwang-Hyun ParkEmail author
  • Sung-Hoon Jeong
  • Christopher Pelczar
  • Z. Zenn. Bien
Original Research Paper


This paper introduces a piano playing robot in views of smart house and assistive robot technology to care the affective states of the elderly. We address the current issues in this research area and propose a piano playing robot as a solution. For affective interaction based on music, we first present a beat gesture recognition method to synchronize the tempo of a robot playing a piano with the desired tempo of the user. To estimate the period of an unstructured beat gesture expressed by any part of a body or an object, we apply an optical flow method, and use the trajectories of the center of gravity and normalized central moments of moving objects in images. In addition, we also apply a motion control method by which robotic fingers are trained to follow a set of trajectories. Since the ability to track the trajectories influences the sound a piano generates, we adopt an iterative learning control method to reduce the tracking error.


Affective interaction Piano playing robot Beat gesture Motion control Iterative learning control 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Bien ZZ, Park KH, Jung JW and Do JH (2005). Intention reading is essential in human-friendly interfaces for the elderly and the handicapped. IEEE Trans Ind Electron 52(6): 1500–1505 CrossRefGoogle Scholar
  2. 2.
    Bondi P, Casalino G and Gambardella L (1988). On the iterative learning control theory for robotic manipulators. IEEE Trans Rob Autom 4(1): 14–22 CrossRefGoogle Scholar
  3. 3.
    Cohen C, Conway L (1996) Dynamical system representation, generation, and recognition of basic oscillatory motion gestures. In: Proceedings of the 2nd international conference on automatic face and gesture recognition, pp 60–65Google Scholar
  4. 4.
    Cutler R and Davis L (2000). Robust real-time periodic motion detection, analysis and applications. IEEE Trans Pattern Anal Mach Intell 22(8): 781–796 CrossRefGoogle Scholar
  5. 5.
    Davis J, Bobick A, Richards W (2000) Categorical representation and recognition of oscillatory motion patterns. In: Proceedings of the IEEE conference on computer vision and pattern recognition, vol 1, pp 628–635Google Scholar
  6. 6.
    Fawcett J, Scheftner W, Fogg L, Clark D, Young M, Hedeker D and Gibbons R (1990). Time-related predictors of suicide in major affective disorder. Am J Psychiatry 147: 1189–1194 Google Scholar
  7. 7.
    Filova V, Solina F and Lenarcic J (1998). Automatic reconstruction of 3d human arm motion from a monocular image sequence. Mach Vis Appl 10: 223–231 CrossRefGoogle Scholar
  8. 8.
    Kato I, Ohteru S, Shirai K, Narita S, Sugano S, Matsushima T, Kobayashi T and Fujisawa E (1987). The robot musician ‘wabot-2’. Robotics 3(2): 143–155 CrossRefGoogle Scholar
  9. 9.
    Lee HE, Kim YM, Park KH and Bien ZZ (2005). Development of a steward robot for intelligent sweet home. Int J Hum-friendly Welf Rob Syst 6(4): 57–64 Google Scholar
  10. 10.
    Liu F, Picard R (1998) Finding periodicity in space and time. In: Proceedings of the 6th international conference on computer vision, pp 376–383Google Scholar
  11. 11.
    Lucas B, Kanade T (1981) An iterative image registration technique with an application to stereo vision. In: Proceedings of the DARPA image understanding workshop, pp 121–130Google Scholar
  12. 12.
    Luukinen H, Laippala P and Huikuri H (2003). Depressive symptoms and the risk of sudden cardiac death among the elderly. Eur Heart J 24(22): 2021–2026 CrossRefGoogle Scholar
  13. 13.
    Moore K (1993). Iterative learning control for deterministic systems. Springer, Heidelberg zbMATHGoogle Scholar
  14. 14.
    Oh J, Choi Y, Park J and Zheng Y (2004). Complete coverage navigation of cleaning robots using triangular-cell-based map. IEEE Trans Ind Electron 51(3): 718–726 CrossRefGoogle Scholar
  15. 15.
    Palacin J, Salse J, Valganon I and Clua X (2004). Building a mobile robot for a floor-cleaning operation in domestic environments. IEEE Trans Instrum Meas 53(5): 1418–1424 CrossRefGoogle Scholar
  16. 16.
    Paul R (1982). Robot manipulators: mathematics, programming and control. MIT Press, Cambridge, p 198 Google Scholar
  17. 17.
    Polana R, Nelson R (1993) Detecting activities. In: Proceedings of the IEEE computer society conference on computer vision and pattern recognition, pp 2–7Google Scholar
  18. 18.
    Seitz S, Dyer C (1994) Affine invariant detection of periodic motion. In: Proceedings of IEEE computer society conference on computer vision and pattern recognition, pp 970–975Google Scholar
  19. 19.
    Sugano S, Kato I (1987) Wabot-2: autonomous robot with dexterous finger-arm. In: Proceedings of the IEEE international conference on robotics and automation, pp 90–97Google Scholar
  20. 20.
    Tsai P, Shah M, Keiter K and Kasparis T (1994). Cyclic motion detection for motion based recognition. Pattern Recognit 27(12): 1591–1603 CrossRefGoogle Scholar
  21. 21.
    Watanabe K, Kosaka H (1995) Evaluation of keyboard switches based on kansei (human sensitivity) information. In: Proceedings of the 4th IEEE international workshop on robot and human communication, Tokyo, pp 65–70Google Scholar
  22. 22.
    Yoshikawa T (1990). Foundations of robotics, analysis and control chap 3. MIT Press, Cambridge Google Scholar

Copyright information

© Springer-Verlag 2008

Authors and Affiliations

  • Kwang-Hyun Park
    • 1
    Email author
  • Sung-Hoon Jeong
    • 1
  • Christopher Pelczar
    • 2
  • Z. Zenn. Bien
    • 1
  1. 1.Department of Electrical Engineering and Computer ScienceKorea Advanced Institute of Science and TechnologyDaejeonRepublic of Korea
  2. 2.Institute of AutomationUniversity of BremenBremenGermany

Personalised recommendations