Experiential Media Systems – The Biofeedback Project

  • Yinpeng Chen
  • Hari Sundaram
  • Thanassis Rikakis
  • Todd Ingalls
  • Loren Olson
  • Jiping He
Part of the Signals and Communication Technology book series (SCT)


Experiential media systems refer to real-time, physically grounded multimedia systems in which the user is both the producer and consumer of meaning. These systems require embodied interaction on part of the user to gain new knowledge. In this chapter, we have presented our efforts to develop a real-time, multimodal biofeedback system for stroke patients. It is a highly specialized experiential media system where the knowledge that is imparted refers to a functional task — the ability to reach and grasp an object. There are several key ideas in this chapter: we show how to derive critical motion features using a biomechanical model for the reaching functional task. Then we determine the formal progression of the feedback and its relationship to action. We show how to map movement parameters into auditory and visual parameters in real-time. We develop novel validation metrics for spatial accuracy, opening, flow, and consistency. Our real-world experiments with unimpaired subjects show that we are able to communicate key aspects of motion through feedback. Importantly, they demonstrate that the messages encoded in the feedback can be parsed by the unimpaired subjects.


Biofeedback, analysis, action-Feedback Coupling, validation 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    G. D. Abowd, E. D. Mynatt and T. Rodden (2002). The human experience [of ubiquitous computing]. IEEE Pervasive Computing 1(1): 48–57.CrossRefGoogle Scholar
  2. 2.
    . V. Basmajian (1989). Biofeedback : principles and practice for clinicians. Williams & Wilkins 0683003569 Baltimore.Google Scholar
  3. 3.
    R. A. Brooks (1991). Intelligence Without Reason, International Joint Conference on Articial Intelligence, pp. 569–595, Aug. 1991, Sydney, Australia.Google Scholar
  4. 4.
    R. A. Brooks (1991). Intelligence without representation. Artificial Intelligence 47(1–3): 139–159.CrossRefGoogle Scholar
  5. 5.
    R. A. Brooks, M. Coen, D. Dang, J. Debonet, J. Kramer, T. Lozano-Perez, J. Mellor, P. Pook, C. Stauffer, L. Stein, M. Torrance and M. Wessler (1997). The Intelligent Room Project, Proceedings of the Second International Cognitive Technology Conference (CT'97), Aug. 1997, Aizu, Japan.Google Scholar
  6. 6.
    Y. Chen, H. Huang, W. Xu, R. Wallis, H. Sundaram, T. Rikakis, J. He, T. Ingalls and L. Olson (2006). The Design Of A Real-Time, Multimodal Biofeedback System For Stroke Patient Rehabilitation, SIG ACM Multimedia, Oct. 2006, Santa Barbara, CA.CrossRefGoogle Scholar
  7. 7.
    Y. Chen, H. Huang, W. Xu, R. I. Wallis, H. Sundaram, T. Rikakis, T. Ingalls, L. Olson and J. He (2006). The design of a real-time, multimodal biofeedback system for stroke patient rehabilitation, Proc. of the 14th annual ACM international conference on Multimedia, 763–772, Oct. 2006, Santa Barbara, CA, USA.Google Scholar
  8. 8.
    Y. Chen, W. Xu, H. Sundaram, T. Rikakis and S.-M. Liu (2007). Media Adaptation Framework in Biofeedback System for Stroke Patient Rehabilitation, Proceedings of the 15th annual ACM international conference on Multimedia, ACM Press, Sep. 2007, Augsburg, Germany.Google Scholar
  9. 9.
    M. C. Cirstea, A. B. Mitnitski, A. G. Feldman and M. F. Levin (2003). Interjoint coordination dynamics during reaching in stroke. Experimental Brain Research 151(3): 289–300.CrossRefGoogle Scholar
  10. 10.
    M. L. Dombovy (2004). Understanding stroke recovery and rehabilitation: current and emerging approaches. Current Neurology and Neuroscience Reports 2004 4(1): 31–35.Google Scholar
  11. 11.
    P. Dourish (2001). Where the action is : the foundations of embodied interaction. MIT Press 0262041960 (alk. paper) Cambridge, Mass. ; London.Google Scholar
  12. 12.
    E. Dursun, N. Dursun and D. Alican (2004). Effects of biofeedback treatment on gait in children with cerebral palsy. Disability and Rehabilitation 26(2): 116–120.CrossRefGoogle Scholar
  13. 13.
    J. Gallichio and P. Kluding (2004). Virtual Reality in Stroke Rehabilitation: Review of the Emerging Research. Physical Therapy Reviews 9(4): 207–212.CrossRefGoogle Scholar
  14. 14.
    C. Ghez, T. Rikakis, R. L. Dubois and P. Cook (2000). An Auditory display system for aiding interjoint coordination, Proc. International Conference on Auditory Display, Apr. 2000, Atlanta, GA.Google Scholar
  15. 15.
    J. Gray (2003). What next ? : A dozen information-technology research goals. Journal of the ACM 50(1): 41–57.CrossRefGoogle Scholar
  16. 16.
    G. E. Gresham, P. W. Duncan and W. B. E. A. Stason (1996). Post-Stroke Rehabilitation/Clinical Practive Guideline. Aspen Publishers, Inc. 30-010-00 Gaithersburg, Maryland.Google Scholar
  17. 17.
    D. J. Grout and C. V. Palisca (2001). A history of western music. Norton 0393975274 New York.Google Scholar
  18. 18.
    H. Woldag, G. Waldmann, G. Heuschkel and H. Hummelsheim (2003). Is the repetitive training of complex hand and arm movements beneficial for motor recovery in stroke patients? Clinical Rehabilitation 2003 Nov 17(7): 723–730.CrossRefGoogle Scholar
  19. 19.
    X. He, W.-Y. Ma, O. King, M. Li and H. Zhang (2003). Learning and Inferring a Semantic Space from User's Relevance Feedback for Image Retrieval. IEEE Transactions on Circuits and Systems for Video Technology.Google Scholar
  20. 20.
    E. R. Hilgard and G. H. Bower (1975). Recent developments. Theories of learning(eds). Englewood Cliffs, N.J.,, Prentice-Hall: 550–605.Google Scholar
  21. 21.
    M. Holden and T. Dyar (2002). Virtual environment traing: a new tool for neurorehabilitation. Neurology Report 26(2): 62–72.Google Scholar
  22. 22.
    M. Holden, E. Todorov, J. Callahan and E. Bizzi (1999). Virtual environment training imporves motor performance in two patients with stroke: case report. Neurology Report 23(2): 57–67.Google Scholar
  23. 23.
    J. Hollan, E. Hutchins, D. Kirsh and A. Sutcliffe (2000). Distributed cognition: toward a new foundation for human-computer interaction research On the effective use and reuse of HCI knowledge. ACM Transaction Computing-Human Interaction 7(2): 174–196.CrossRefGoogle Scholar
  24. 24.
    E. Hutchins (1995). Cognition in the wild. MIT Press 0262082314 Cambridge, Mass.Google Scholar
  25. 25.
    H. Ishii and B. Ullmer (1997). Tangible bits: towards seamless interfaces between people, bits and atoms, Proceedings of the SIGCHI conference on Human factors in computing systems, ACM Press, 234–241,Google Scholar
  26. 26.
    H. Ishii, C. Wisneski, S. Brave, A. Dahley, M. Gorbet, B. Ullmer and P. Yarin (1998). ambientROOM: integrating ambient media with architectural space, CHI 98 conference summary on Human factors in computing systems, ACM Press, 173–174,Google Scholar
  27. 27.
    D. Jack, R. Boian, A. S. Merians, M. Tremaine, G. C. Burdea, S. V. Adamovich, M. Recce and H. Poizner (2001). Virtual reality-enhanced stroke rehabilitation. IEEE Transactions on Neural Systems and Rehabilitation Engineering 9: 308–318.CrossRefGoogle Scholar
  28. 28.
    R. V. Kenyon, J. Leigh and E. A. Keshner (2004). Considerations for the future development of virtual technology as a rehabilitation tool. J Neuroengineering Rehabilitation 1(1): 13.CrossRefGoogle Scholar
  29. 29.
    D. Kirsh (1995). The intelligent use of space. Artificial Intelligence 73(1–2): 31–68.CrossRefGoogle Scholar
  30. 30.
    Y.-F. Ma and H.-J. Zhang (2003). Contrast-based image attention analysis by using fuzzy growing, Proceedings of the eleventh ACM international conference on Multimedia, 1-58113-722-2, ACM Press, 374–381, Nov. 2003., Berkeley, CA, USA.Google Scholar
  31. 31.
    A. Mazalek, G. Davenport and H. Ishii (2002). Tangible viewpoints: a physical approach to multimedia stories, Proceedings of the tenth ACM international conference on Multimedia, ACM Press, 153--160,Google Scholar
  32. 32.
    J. Moreland and M. A. Thomson (1994). Efficacy of electromyographic biofeedback compared with conventional physical therapy for upper-extremity function in patients following stroke: a research overview and meta-analysis. Phys Ther 74(6): 534–543; discussion 544–537.Google Scholar
  33. 33.
    M. T. Schultheis and A. A. Rizzo (2001). The application of virtual reality technology for rehabilitation. Rehabilitation Psychology 46: 296–311.CrossRefGoogle Scholar
  34. 34.
    Y. Sun, H. Zhang, L. Zhang and M. Li (2002). A System for Home Photo Management and Processing, Proceedings of the 10th ACM international conference on Multimedia, pp. 81–82, Dec. 2002, Juan Les-Pins, France.Google Scholar
  35. 35.
    H. Sundaram and S.-F. Chang (2000). Determining Computable Scenes in Films and their Structures using Audio-Visual Memory Models, Proc. Of ACM International Conference on Multimedia 2000, pp. 95–104, Nov. 2000, Los Angeles, CA, USA.Google Scholar
  36. 36.
    G. Theocharous, K. Murphy and L. P. Kaelbling (2003). Representing hierarchical POMDPs as DBNs for multi-scale robot localization, Workshop on Reasoning about Uncertainty in Robotics, International Joint Conference on Artificial Intelligence, Acapulco, Mexico.Google Scholar
  37. 37.
    M. Tidwell, R. S. Johnston, D. Melville and T. A. Furness (1995). The virtual retinal display-a retinal scanning imaging system, Proceeding of Virtual Reality World' 95, 325–333, Heidelberg.Google Scholar
  38. 38.
    B. Ullmer and H. Ishii (2000). Emerging Frameworks for Tangible User Interfaces. IBM Systems Journal 39(3 & 4): pp. 915–931.CrossRefGoogle Scholar
  39. 39.
    J. P. Wann and J. D. Turnbull (1993). Motor skill learning in cerebral palsy: movement, action and computer-enhanced therapy. Baillieres Clinical Neurology 2(1): 15–28.Google Scholar
  40. 40.
    M. Weiser (1993). Some computer science issues in ubiquitous computing. Communication ACM 36(7): 75–84.CrossRefGoogle Scholar
  41. 41.
    D. White, K. Burdick, G. Fulk, J. Searleman and J. Carroll (2005). A virtual reality application for stroke patient rehabilitation, IEEE International Conference on Mechatronics & Automation Niagara Falls, July 2005, Canada.Google Scholar
  42. 42.
    S. L. Wolf, P. A. Catlin, S. Blanton, J. Edelman, N. Lehrer and D. Schroeder (1994). Overcoming limitations in elbow movement in the presence of antagonist hyperactivity. Physical Theraphy 74(9): 826–835.Google Scholar
  43. 43.
    S. H. You, S. H. Jang, Y. H. Kim, M. Hallett, S. H. Ahn, Y. H. Kwon, J. H. Kim and M. Y. Lee (2005). Virtual reality-induced cortical reorganization and associated locomotor recovery in chronic stroke: an experimenter-blind randomized study. Stroke 36(6): 1166–1171.CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC 2009

Authors and Affiliations

  • Yinpeng Chen
    • 1
  • Hari Sundaram
  • Thanassis Rikakis
  • Todd Ingalls
  • Loren Olson
  • Jiping He
  1. 1.Arts Media and Engineering, Arizona State UniversityTempeUSA

Personalised recommendations