Robotics and Virtual Reality: A Marriage of Two Diverse Streams of Science

  • Tauseef Gulrez
  • Manolya Kavakli
  • Alessandro Tognetti
Part of the Studies in Computational Intelligence book series (SCI, volume 96)

In an immersive computationally intelligent virtual reality (VR) environment, humans can interact with a virtual 3D scene and navigate a robotic device. The non-destructive nature of VR makes it an ideal testbed for many applications and a prime candidate for use in rehabilitation robotics simulation and patient training. We have developed a testbed for robot mediated neurorehabilitation therapy that combines the use of robotics, computationally intelligent virtual reality and haptic interfaces. We have employed the theories of neuroscience and rehabilitation to develop methods for the treatment of neurological injuries such as stroke, spinal cord injury, and traumatic brain injury. As a sensor input we have used two stateof-the-art technologies, depicting the two different approaches to solve the mobility loss problem. In our first experiment we have used a 52 piezoresistive sensor laden shirt as an input device to capture the residual signals arising from the patient's body. In our second experiment, we have used a precision position tracking (PPT) system to capture the same signals from the patient's upper body movement. The key challenge in both of these experiments was to accurately localise the movement of the object in reality and map its corresponding position in 3D VR. In this book chapter, we describe the basic theory of the development phase and of the operation of the complete system. We also present some preliminary results obtained from subjects using upper body postures to control the simulated wheelchair.


Virtual Reality Robotic Device Virtual Reality Environment Immersive Virtual Reality Rehabilitation Device 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Australian Bureau of Statistics (ABS). 1998 disability, ageing and carers, australia: Confidentialised unit record file. Technical paper. Canberra: ABS.1999.Google Scholar
  2. 2.
    Australian Bureau of Statistics (ABS). Population projections australia: 1999 to 2101. Canberra: ABS, 2000, (Catalogue No. 3222.0.).Google Scholar
  3. 3.
    Australian institute of health and welfare (AIHW). disability and ageing: Australian population patterns and implications. Canberra: AIHW. 2000, (AIHW Catalogue No. DIS 19.).Google Scholar
  4. 4.
    Coin 3d graphics library.
  5. 5.
    Elastosil lr3162.
  6. 6.
    World Health Organisation. World report on disability and rehabilitation. Concept Paper, World Health Organisation. 2006.Google Scholar
  7. 7.
    C. Baker, J.B. Tenenbaum, and R.R. Saxe. Bayesian models of human action understanding. Advances in Neural Information Processing Systems, 18, 2006.Google Scholar
  8. 8.
    G.A. Barnard and Thomas Bayes. Studies in the history of probability and statistics: Ix. thomas bayes’s essay towards solving a problem in the doctrine of chances. Biometrika, 45:293–315, 1958.zbMATHGoogle Scholar
  9. 9.
    T.V.P. Bliss and G.L. Collingridge. A synaptic model of memory: Long-term potentiation in the hippocampus. Nature, 361:31–39.Google Scholar
  10. 10.
    D.J. Brown, S.J. Kerr, and V. Bayon. The development of the virtual city: A user centered approach. In 2nd European Conference on Disability, Virtual Reality and Associated Techniques, Mount Billingen, Skovde, Sweden, September 1998.Google Scholar
  11. 11.
    A. Buckert-Donelson. Heads-up products: Virtual worlds ease dental patients. VR World, 3:9–16, 1995.Google Scholar
  12. 12.
    K. ByungMoon and T. Panagiotis. Controllers for unicycle-type wheeled robots: Some theoretical results and experimental validation. IEEE Transactions on Robotics and Automation, 18(3):294–307, 2002.CrossRefGoogle Scholar
  13. 13.
    S. Challa, T. Gulrez, Z. Chazcko, and T. Paranesha. Opportunistic information fusion: A new paradigm for next generation networked sensing systems. In 8th IEEE International Conference on Information Fusion, Philadelphia, USA, 2005.Google Scholar
  14. 14.
    C. Christiansen, B. Abreu, K. Ottenbacher, K. Huffman, B. Masel, and R. Culpepper. Task performance in virtual environments used for cognitive rehabilitation after traumatic brain injury. Archives of Physical Medicine and Rehabilitation, 79:888–892, 1998.CrossRefGoogle Scholar
  15. 15.
    M.E. Clynes and N.S. Kline. Cyborgs and space. Astronautics, American Rocket Society, 14:26–27, 1960.Google Scholar
  16. 16.
    M.A. Conditt and F.A. Mussa-Ivaldi. Central representation of time during motor learning. Philosophical Transcations of Royal Society of London, 96:11625–11630, 1999.Google Scholar
  17. 17.
    J.P. Donoghue, Connecting cortex to machines: Recent advances in brain interfaces. Nature Neuroscience Reviews, 5:1085–1088, 2002.CrossRefGoogle Scholar
  18. 18.
    L. Fehr, W.E. Langbein, and S.B. Skaar. Adequacy of power wheelchair control interfaces for persons with severe disabilities: A clinical survey. Journal of Rehabilitation Research and Development, 37:353–360, 2000.Google Scholar
  19. 19.
    C.C. Flynn and C.M. Clark. Rehabilitation technology: Assessment practices in vocational agencies. Assistive Technology, 7:111–118, 1995.Google Scholar
  20. 20.
    T.F. Freund and G. Buzski. Interneurons of the hippocampus. Hippocampus, 6:347–470, 1958.CrossRefGoogle Scholar
  21. 21.
    F. Gandolfo, F.A. Mussa-Ivaldi, and E. Bizzi. Motor learning by field approximation. Proceedings of National Academy of Sciences USA, 93:3843–3846, 1996.CrossRefGoogle Scholar
  22. 22.
    C.L. Giles, D. Cameron, and M. Crotty. Disability in older australians: Projections for 2006–2031. Medical Journal of Australia, 179:130–133, 2003.Google Scholar
  23. 23.
    T.L. Griffiths and J.B. Tenenbaum. Statistics and the bayesian mind. Significance, 3(3):130–133, 2006.CrossRefMathSciNetGoogle Scholar
  24. 24.
    T. Gulrez and S. Challa. Sensor relevance validation for autonomous mobile robot navigation. In IEEE Conference on Robotics Automation and Mechatronics (RAM), Bangkok, Thailand, June 7–9, 2006.Google Scholar
  25. 25.
    T. Gulrez, S. Challa, T. Yaqub, and J. Katupitiya. Relevant opportunistic information extraction scheduling in heterogeneous sensor networks. In 1st IEEE International Workshop on Computational Advances in Multi-Sensor Adaptive Processing, Mexico-City, 2005.Google Scholar
  26. 26.
    H.G. Hoffman, J.N. Doctor, D.R. Patterson, G.J. Carrougher, and T.A.I. Furness. Use of virtual reality for adjunctive treatment of adolescent burn pain during wound care: A case report. Pain, 85:305–309, 2000.CrossRefGoogle Scholar
  27. 27.
    N. Hogan, H. Krebs, J. Charnnarong, P. Srikrishna, and A. Sharon. Mit - vmanus a workstation for manual therapy and training ii. In SPIE Conf. Telemanipulator Technologies, pages 28–34, 1992.Google Scholar
  28. 28.
    A. Johnson, D. Sandin, G. Dawe, Z. Qiu, and D. Plepys. Developing the paris: Using the cave to prototype a new vr display. In Proceedings of IPT 2000, Ames, Iowa, USA, June 2000.Google Scholar
  29. 29.
    K. Kording and D. Wolpert. Bayesian integration in sensorimotor learning. Nature, 427:244–247, 2004.CrossRefGoogle Scholar
  30. 30.
    M. Kavakli and M. Lloyd. Spaceengine: A seamless simulation system for virtual presence in space. In Innovations in Intelligent Systems and Applications, IEEE Computational Intelligence Society, pages 231–233, Turkey, Yildiz Technical University, Istanbul, Turkey, 2005.Google Scholar
  31. 31.
    J. Keefe and L. Nadel. The Hippocampus as a Cognitive Map. Oxford University Press, New York, 1978.Google Scholar
  32. 32.
    K. Kording and D. Wolpert. Bayesian decision theory in sensorimotor control. Review – Trends in Cognitive science, 10:319–326, 2006.CrossRefGoogle Scholar
  33. 33.
    J.W. Krakauer and R. Shadmehr. Consolidation of motor memory. Review – Trends in Neuroscience, 29:58–64, 2006.CrossRefGoogle Scholar
  34. 34.
    A. Kubler. Brain computer communication: Unlocking the locked. Psychology Bulletin, 127:358–375, 2001.CrossRefGoogle Scholar
  35. 35.
    F. Lorussi, W. Rocchia, E.P. Scilingo, A. Tognetti, and D. De Rossi. Wearable redundant fabric-based sensors arrays for reconstruction of body segment posture. IEEE Sensors Journal, 4(6):807–818, 2004.CrossRefGoogle Scholar
  36. 36.
    F. Lorussi, E.P. Scilingo, M. Tesconi, A. Tognetti, and D. De Rossi. Strain sensing fabric for hand posture and gesture monitoring. IEEE Transactions on Information Technology in Biomedicine, 9(3):372–381, 2005.CrossRefGoogle Scholar
  37. 37.
    J.L. McClelland, B.L. McNaughton, and R.C. O’Reilly. Why there are complementary learning systems in the hippocampus and neocortex: Insights from the successes and failures of connectionist models of learning and memory. Psychlogical review, 102:419–457, 1995.CrossRefGoogle Scholar
  38. 38.
    Medical, Readiness, and Trainer-Team. Immersive virtual reality platform for medical training: A “killer-application”. In Medicine Meets Virtual Reality 2000, pages 207–213, Burke, Virginia, USA, 2000.Google Scholar
  39. 39.
    C. Mercier, K. Reilly, C. Vargas, A. Aballea, and A. Srigu. Mapping phantom movement representations in the motor cortex of amputees. Brain, 129: 2202–2210, 2006.CrossRefGoogle Scholar
  40. 40.
    F.A. Mussa-Ivaldi and E. Bizzi. Motor learning through the combination of primitives. Philosophical Transcations of Royal Society of London, 355: 1755–1769, 2000.CrossRefGoogle Scholar
  41. 41.
    F.A. Mussa-Ivaldi, A. Fishbach, T. Gulrez, A. Tognetti, and D. De, Rossi. Remapping the residual motor space of spinal-cord injured patients for the control of assistive devices. In Neuroscience 2006, Atlanta, GA, USA, October 14–18, 2006.Google Scholar
  42. 42.
    F.A. Mussa-Ivaldi, N. Hogan, and E. Bizzi. Neural, mechanical, and geometric factors subserving arm posture in humans. Journal of Neuroscience, 5:2732–2743, 1985.Google Scholar
  43. 43.
    F.A. Mussa-Ivaldi and L.E. Miller. Brain machine interfaces: Computational demands and clinical needs meet basic neuroscience. Review, Trends in Neuroscience, 26:329–334, 2003.CrossRefGoogle Scholar
  44. 44.
    F.A. Mussa-Ivaldi and S. Solla. Neural primitives for motion control. IEEE Journal of Oceanic Engineering, 29:640–650, 2004.CrossRefGoogle Scholar
  45. 45.
    M. Oshuga, F. Tatsuno, K. Shimono, K. Hirasawa, H. Oyama, and H. Okamura. Development of a bedside wellness system. Cyberpsychology and Behavior, 1:105–111, 1998.CrossRefGoogle Scholar
  46. 46.
    J. Patton and F. Mussa-Ivaldi. Robotic teaching by exploiting the nervous system’s adaptive mechanisms. In 7th International Conference on Rehabilitation Robotics (ICORR), Evry, France, 2001.Google Scholar
  47. 47.
    W. Penfield and T. Rasmussen. The Cerebral Cortex of Man: A Clinical Study of Localisation of Function. Macmillan, New York, 1950.Google Scholar
  48. 48.
    Wang Peng, Xu Feng, Ding Tianhuai, and Qin Yuanzhen. Time dependence of electrical resistivity under uniaxial pressures for carbon black/polymer composites. Journal of Materials Science, 39(15), 2004.Google Scholar
  49. 49.
    Wang Peng, Ding Tianhuai, Xu Feng, and Qin Yuanzhen. Piezoresistivity of conductive composites filled by carbon black particles. Acta Materlae Compositae Sinica, 21(6):34–38, 2004.Google Scholar
  50. 50.
    B.E. Pfingst. Neural Prostheses for Restoration of Sensory and Motor Function, J.K. Chapin, K.A. Moxon (eds.). CRC, Boca Raton, 2000.Google Scholar
  51. 51.
    R.G. Platts and M.H. Fraser. Assistive technology in the rehabilitation of patients with high spinal cord injury lesions. Paraplegia, 31:280–287, 1993.Google Scholar
  52. 52.
    M.W. Post, F.W. vanAsbeck, A.J. vanDijk, and A.J. Schrijvers. Spinal cord injury rehabilitation: 3 functional outcomes. Archives of Physical Medicine and Rehabilitation, 87:59–64, 1997.Google Scholar
  53. 53.
    L. Pugnetti, L. Mendozzi, E. Barbieri, F. Rose, and E. Attree. Nervous system correlates of virtual reality experience. In First European Conference on Disability, Virtual Reality and Associated Technology, pages 239–246, Maidenhead, UK: The University of Reading, July 1996.Google Scholar
  54. 54.
    G. Riva and L. Melis. Virtual reality for the treatment of body image distrubance. In Virtual Reality in Neuro-Psycho-Physiology, Amsterdam, 1997.Google Scholar
  55. 55.
    B.O. Rothbaum, L.F. Hodges, R. Alarcon, D. Ready, F. Shahar, K. Graap, J. Pair, P. Hebert, D. Gotz, B. Wills, and D. Baltzell. Virtual reality exposure therapy for ptsd vietnam veterans: A case study. Journal of Traumatic Stress, 12:263–272, 1999.CrossRefGoogle Scholar
  56. 56.
    B.O. Rothbaum, L.F. Hodges, and R. Kooper. Virtual reality exposure therapy. Journal of Psychotherapy Practice and Research, 6:291–296, 1997.Google Scholar
  57. 57.
    R.M. Satava. Medical virtual reality: The current status of the future. In The Medicine Meets Virtual Reality 4th Conference, pages 100–106, Berlin, Germany, Sept 1996.Google Scholar
  58. 58.
    R. Shadmehr and F.A. Mussa-Ivaldi. Adaptive representation of dynamics during learning of a motor task. Journal of Neuroscience, 14(5):3208–3224, 1994.Google Scholar
  59. 59.
    S. Takezawa, T. Gulrez, C.D. Herath, and W.M. Dissanayake. Environmental recognition for autonomous robot using slam. real time path planning with dynamical localised voronoi division. International Journal of Japan Society of Mechanical Engineering (JSME), 3:904–911, 2005.Google Scholar
  60. 60.
    M. Taya, W.J. Kim, and K. Ono. Piezoresistivity of a short fiber/elastomer matrix composite. Mechanics of Materials, 28(3):53–59, 1998.CrossRefGoogle Scholar
  61. 61.
    J.B. Tenenbaum and T.L. Griffiths. Generalization, similarity, and bayesian inference. Behavioral and Brain Sciences, 24:629–641, 2001.Google Scholar
  62. 62.
    A. Tognetti, F. Lorussi, R. Bartalesi, S. Quaglini, M. Tesconi, G. Zupone, and D. De Rossi. Wearable kinesthetic system for capturing and classifying upper limb gesture in post-stroke rehabilitation. Journal of NeuroEngineering and Rehabilitation, 2(8), 2005.Google Scholar
  63. 63.
    D.M. Wolpert, Z. Ghahramani, and M.I. Jordan. An internal model for sensorimotor integration. Science, 269:1880–1882, 1995.CrossRefGoogle Scholar
  64. 64.
    XiangWu Zhang, Yi Pan, Qiang Zheng, and XiaoSu Yi. Time dependence of piezoresistance for the conductor-filled polymer composites. Journal of Polymer Science, 38(21), 2000.Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2008

Authors and Affiliations

  • Tauseef Gulrez
    • 1
  • Manolya Kavakli
    • 1
  • Alessandro Tognetti
    • 2
  1. 1.Virtual Interactive Simulations of Reality (VISOR) Research Group, Department of Computing, Division of Information and Communication SciencesMacquarie University, SydneyAustralia
  2. 2.Interdepartmental Research Center “E. Piaggio”University of PisaItaly

Personalised recommendations