Skip to main content
SpringerLink
Log in

“Wear it”—Wearable Robotic Musicians

  • Chapter
  • First Online:
Robotic Musicianship

Part of the book series: Automation, Collaboration, & E-Services ((ACES,volume 8))

Abstract

Recent developments in wearable technology can help people with disabilities regain their lost capabilities, merging their biological body with robotic enhancements. Myoelectric prosthetic hands, for example, allow amputees to perform basic daily-life activities by sensing and analyzing electric activity from their residual limbs, which is then used to actuate a robotic hand. These new developments not only bring back lost functionalities, but can also provide humanly impossible capabilities, turning those who were considered disabled to become super-abled. The next frontier of Robotic Musicianship research at Georgia Tech focuses on the development of wearable robotic limbs that allow not only amputees, but able-bodied people as well, to play music like no human can, with virtuosity and speed that are humanly impossible. This chapter addresses the promises and challenges of the new frontier of wearable robotic musicians, from a Robotic Prosthetic Drumming Arm that contains a drumming stick with a “mind of its own”, to a “Third Arm” that augments able-bodied drummers, to the Skywalker Piano Hand that uses deep learning predictions from ultrasound muscle data to allow amputees to play the piano using dexterous and expressive finger gestures.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 139.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 179.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 179.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    http://www.robotis.com/xe/dynamixel_en.

  2. 2.

    https://cycling74.com/.

  3. 3.

    https://www.arduino.cc/en/Main/ArduinoBoardMega2560.

References

  1. Hugh, Herr, Graham Paul Whiteley, and Dudley Childress. 2003. Cyborg technology-biomimetic orthotic and prosthetic technology. SPIE Press, Bellingham, Washington.

    Google Scholar 

  2. Llorens-Bonilla, Baldin, Federico Parietti, and H. Harry Asada. 2012. Demonstration-based control of supernumerary robotic limbs. In 2012 IEEE/RSJ international conference on intelligent robots and systems (IROS), 3936–3942. IEEE.

    Google Scholar 

  3. Brian, Dellon, and Matsuoka Yoky. 2007. Prosthetics, exoskeletons, and rehabilitation [grand challenges of robotics]. Robotics & Automation Magazine, IEEE 14 (1): 30–34.

    Article  Google Scholar 

  4. Schirner, Gunar, Deniz Erdogmus, Kaushik Chowdhury, and Taskin Padir. 2013. The future of human-in-the-loop cyber-physical systems.

    Google Scholar 

  5. Li, Qinan, Weidong Chen, and Jingchuan Wang. 2011. Dynamic shared control for human-wheelchair cooperation. In 2011 IEEE international conference on robotics and automation (ICRA), 4278–4283. IEEE.

    Google Scholar 

  6. Nudehi, Shahin S., Ranjan Mukherjee, and Moji Ghodoussi. 2005. A shared-control approach to haptic interface design for minimally invasive telesurgical training. IEEE Transactions on Control Systems Technology 13 (4): 588–592.

    Article  Google Scholar 

  7. Gil, Weinberg, and Driscoll Scott. 2006. Toward robotic musicianship. Computer Music Journal 30 (4): 28–45.

    Article  Google Scholar 

  8. Abbink, David A., and M. Mulder. 2010. Neuromuscular analysis as a guideline in designing shared control. Advances in Haptics 109: 499–516.

    Google Scholar 

  9. Gentili, Rodolphe J., Hyuk Oh, Isabelle M. Shuggi, Ronald N. Goodman, Jeremy C. Rietschel, Bradley D. Hatfield, and James A. Reggia. 2013. Human-robotic collaborative intelligent control for reaching performance. In Foundations of augmented cognition, 666–675. Springer.

    Google Scholar 

  10. Kapur, Ajay, Michael Darling, Dimitri Diakopoulos, Jim W. Murphy, Jordan Hochenbaum, Owen Vallis, and Curtis Bahn. 2011. The machine orchestra: An ensemble of human laptop performers and robotic musical instruments. Computer Music Journal 35 (4): 49–63.

    Article  Google Scholar 

  11. Laura, Maes, Raes Godfried-Willem, and Rogers Troy. 2011. The man and machine robot orchestra at logos. Computer Music Journal 35 (4): 28–48.

    Article  Google Scholar 

  12. Cakmak, Maya, Crystal Chao, and Andrea L. Thomaz. 2010. Designing interactions for robot active learners. IEEE Transactions on Autonomous Mental Development 2 (2): 108–118.

    Article  Google Scholar 

  13. Hoffman, Guy, and Gil Weinberg. 2010. Shimon: An interactive improvisational robotic marimba player. In CHI’10 extended abstracts on human factors in computing systems, 3097–3102. ACM.

    Google Scholar 

  14. Cipriani, Christian, Franco Zaccone, Silvestro Micera, and Maria Chiara Carrozza. 2008. On the shared control of an emg-controlled prosthetic hand: analysis of user-prosthesis interaction. IEEE Transactions on Robotics 24 (1): 170–184.

    Article  Google Scholar 

  15. Chris, Lake, and Dodson Robert. 2006. Progressive upper limb prosthetics. Physical Medicine and Rehabilitation Clinics of North America 17 (1): 49–72.

    Article  Google Scholar 

  16. Cipriani, Christian, Marco Controzzi, and M. Chiara Carrozza. 2009. Progress towards the development of the smarthand transradial prosthesis. In IEEE international conference on rehabilitation robotics, 2009. ICORR 2009, 682–687. IEEE.

    Google Scholar 

  17. Kim, Hyun K., J. Biggs, David W. Schloerb, Jose M. Carmena, Mikhail A. Lebedev, Miguel A.L. Nicolelis, and Mandayam A. Srinivasan. 2006. Continuous shared control for stabilizing reaching and grasping with brain-machine interfaces. IEEE Transactions on Biomedical Engineering 53 (6): 1164–1173.

    Article  Google Scholar 

  18. Hochberg, Leigh R., Mijail D. Serruya, Gerhard M. Friehs, Jon A. Mukand, Maryam Saleh, Abraham H. Caplan, Almut Branner, David Chen, Richard D. Penn, and John P. Donoghue. 2006. Neuronal ensemble control of prosthetic devices by a human with tetraplegia. Nature 442 (7099): 164–171.

    Article  Google Scholar 

  19. Wu, Faye Y., and Harry Asada. 2014. Bio-artificial synergies for grasp posture control of supernumerary robotic fingers.

    Google Scholar 

  20. Davenport, Clark Clark Michael. 2013. Supernumerary robotic limbs: Biomechanical analysis and human-robot coordination training. PhD thesis, Massachusetts Institute of Technology.

    Google Scholar 

  21. Singer, Eric, Jeff Feddersen, Chad Redmon, and Bil Bowen. 2004. Lemur’s musical robots. In Proceedings of the 2004 conference on new interfaces for musical expression, 181–184. National University of Singapore.

    Google Scholar 

  22. Weinberg, Gil, and Scott Driscoll. 2006. Robot-human interaction with an anthropomorphic percussionist. In Proceedings of the SIGCHI conference on human factors in computing systems, 1229–1232. ACM.

    Google Scholar 

  23. Puckette, Miller S., Miller S. Puckette Ucsd, Theodore Apel, et al. 1998. Real-time audio analysis tools for Pd and MSP.

    Google Scholar 

  24. Carlson, Tom, and José del R Millán. 2013. Brain-controlled wheelchairs: A robotic architecture. IEEE Robotics and Automation Magazine 20 (EPFL-ARTICLE-181698): 65–73.

    Article  Google Scholar 

  25. Sun, Sisi, Trishul Mallikarjuna, and Gil Weinberg. Effect of visual cues in synchronization of rhythmic patterns.

    Google Scholar 

  26. Guy, Hoffman, and Weinberg Gil. 2011. Interactive improvisation with a robotic marimba player. Autonomous Robots 31 (2–3): 133–153.

    Google Scholar 

  27. Kapur, Ajay. 2005. A history of robotic musical instruments. In Proceedings of the international computer music conference, 21–28. Citeseer.

    Google Scholar 

  28. Logan-Greene, Richard. The music of Richard Johnson Logan-Greene. http://zownts.com. Accessed 5 Jan 2014.

  29. Ort, Teddy, Faye Wu, Nicholas C. Hensel, and H. Harry Asada. 2015. Supernumerary robotic fingers as a therapeutic device for hemiparetic patients. In ASME 2015 dynamic systems and control conference, V002T27A010–V002T27A010. American Society of Mechanical Engineers.

    Google Scholar 

  30. Dheeraj, Nimawat, and Jailiya Pawan Raj Singh. 2015. Requirement of wearable robots in current scenario. European Journal of Advances in Engineering and Technology 2 (2): 19–23.

    Google Scholar 

  31. Samer, Mohammed, Amirat Yacine, and Rifai Hala. 2012. Lower-limb movement assistance through wearable robots: State of the art and challenges. Advanced Robotics 26 (1–2): 1–22.

    Google Scholar 

  32. Gopura, R.A.R.C., D.S.V. Bandara, Kazuo Kiguchi, and George K.I. Mann. 2016. Developments in hardware systems of active upper-limb exoskeleton robots: A review. Robotics and Autonomous Systems 75: 203–220.

    Article  Google Scholar 

  33. Gálvez-Zúñiga, Miguel A., and Alejandro Aceves-López. 2016. A review on compliant joint mechanisms for lower limb exoskeletons. Journal of Robotics.

    Google Scholar 

  34. Gopura, R.A.R.C., Kazuo Kiguchi, and D.S.V. Bandara. 2011. A brief review on upper extremity robotic exoskeleton systems. In 2011 6th international conference on industrial and information systems, 346–351. IEEE.

    Google Scholar 

  35. Rocon, E., A.F. Ruiz, R. Raya, A. Schiele, J.L. Pons, J.M. Belda-Lois, R. Poveda, M.J. Vivas, and J.C. Moreno. 2008. Human-robot physical interaction. In Wearable robots: Biomechatronic exoskeletons, 127–163.

    Google Scholar 

  36. Allan Joshua Veale and Shane Quan Xie. 2016. Towards compliant and wearable robotic orthoses: A review of current and emerging actuator technologies. Medical Engineering & Physics 38 (4): 317–325.

    Article  Google Scholar 

  37. Lenzi, Tommaso, Nicola Vitiello, Stefano Marco Maria De Rossi, Alessandro Persichetti, Francesco Giovacchini, Stefano Roccella, Fabrizio Vecchi, and Maria Chiara Carrozza. 2011. Measuring human-robot interaction on wearable robots: A distributed approach. Mechatronics 21 (6): 1123–1131.

    Article  Google Scholar 

  38. Knight, James F., Chris Baber, Anthony Schwirtz, and Huw William Bristow. 2002. The comfort assessment of wearable computers. ISWC 2: 65–74.

    Google Scholar 

  39. Bodine, Kerry, and Francine Gemperle. 2003. Effects of functionality on perceived comfort of wearables. In Proceedings of seventh IEEE international symposium on wearable computers, 2003, 57–60. IEEE.

    Google Scholar 

  40. Demers, Louise, Rhoda Weiss-Lambrou, and Bernadette Ska. 2002. The quebec user evaluation of satisfaction with assistive technology (quest 2.0): An overview and recent progress. Technology and Disability 14 (3): 101–105.

    Google Scholar 

  41. Nigel Corlett, E., and R.P. Bishop. 1976. A technique for assessing postural discomfort. Ergonomics 19 (2): 175–182.

    Article  Google Scholar 

  42. Singer, Neil C., and Warren P. Seering. 1990. Preshaping command inputs to reduce system vibration. Journal of Dynamic Systems, Measurement, and Control 112 (1): 76–82.

    Article  Google Scholar 

  43. Zhai, Xiaolong, Beth Jelfs, Rosa H.M. Chan, and Chung Tin. 2016. Short latency hand movement classification based on surface emg spectrogram with PCA. In 2016 38th annual international conference of the IEEE engineering in medicine and biology society (EMBC), 327–330. IEEE.

    Google Scholar 

  44. Jun-Uk, Chu, Moon Inhyuk, Lee Yun-Jung, Kim Shin-Ki, and Mun Mu-Seong. 2007. A supervised feature-projection-based real-time emg pattern recognition for multifunction myoelectric hand control. IEEE/ASME Transactions on Mechatronics 12 (3): 282–290.

    Article  Google Scholar 

  45. Huang, Yonghong, Kevin B. Englehart, Bernard Hudgins, and Adrian D.C. Chan. 2005. A gaussian mixture model based classification scheme for myoelectric control of powered upper limb prostheses. IEEE Transactions on Biomedical Engineering 52 (11): 1801–1811.

    Article  Google Scholar 

  46. Silvia, Muceli, and Farina Dario. 2011. Simultaneous and proportional estimation of hand kinematics from emg during mirrored movements at multiple degrees-of-freedom. IEEE Transactions on Neural Systems and Rehabilitation Engineering 20 (3): 371–378.

    Google Scholar 

  47. Afshar, Pedram, and Yoky Matsuoka. 2004. Neural-based control of a robotic hand: Evidence for distinct muscle strategies. In Proceedings of IEEE international conference on robotics and automation, 2004. ICRA’04, vol. 5, 4633–4638. IEEE.

    Google Scholar 

  48. Hioki, Masaaki, and Haruhisa Kawasaki. 2012. Estimation of finger joint angles from sEMG using a neural network including time delay factor and recurrent structure. ISRN Rehabilitation 2012.

    Google Scholar 

  49. Shrirao, Nikhil A., Narender P. Reddy, and Durga R. Kosuri. 2009. Neural network committees for finger joint angle estimation from surface emg signals. Biomedical Engineering Online 8 (1): 2.

    Article  Google Scholar 

  50. Ngeo, Jimson G., Tomoya Tamei, and Tomohiro Shibata. 2014. Continuous and simultaneous estimation of finger kinematics using inputs from an emg-to-muscle activation model. Journal of Neuroengineering and Rehabilitation 11 (1): 122.

    Article  Google Scholar 

  51. Zheng, Yong-Ping, M.M.F. Chan, Jun Shi, Xin Chen, and Qing-Hua Huang. 2006. Sonomyography: Monitoring morphological changes of forearm muscles in actions with the feasibility for the control of powered prosthesis. Medical Engineering & Physics 28 (5): 405–415.

    Article  Google Scholar 

  52. Castellini, Claudio, and Georg Passig. 2011. Ultrasound image features of the wrist are linearly related to finger positions. In 2011 IEEE/RSJ international conference on intelligent robots and systems, 2108–2114. IEEE.

    Google Scholar 

  53. Castellini, Claudio and David Sierra González. 2013. Ultrasound imaging as a human-machine interface in a realistic scenario. In 2013 IEEE/RSJ international conference on intelligent robots and systems, 1486–1492. IEEE.

    Google Scholar 

  54. Vikram, Ravindra, and Castellini Claudio. 2014. A comparative analysis of three non-invasive human-machine interfaces for the disabled. Frontiers in Neurorobotics 8: 24.

    Google Scholar 

  55. Xin, Chen, Zheng Yong-Ping, Guo Jing-Yi, and Shi Jun. 2010. Sonomyography (smg) control for powered prosthetic hand: a study with normal subjects. Ultrasound in Medicine & Biology 36 (7): 1076–1088.

    Article  Google Scholar 

  56. Guo, Jing-Yi, Yong-Ping Zheng, Laurence P.J. Kenney, Audrey Bowen, David Howard, and Jiri J. Canderle. 2011. A comparative evaluation of sonomyography, electromyography, force, and wrist angle in a discrete tracking task. Ultrasound in Medicine & Biology 37 (6): 884–891.

    Article  Google Scholar 

  57. Li, Yuefeng, Keshi He, Xueli Sun, and Honghai Liu. 2016. Human-machine interface based on multi-channel single-element ultrasound transducers: A preliminary study. In 2016 IEEE 18th international conference on e-health networking, applications and services (Healthcom), 1–6. IEEE.

    Google Scholar 

  58. Sikdar, Siddhartha, Huzefa Rangwala, Emily B. Eastlake, Ira A. Hunt, Andrew J. Nelson, Jayanth Devanathan, Andrew Shin, and Joseph J. Pancrazio. 2013. Novel method for predicting dexterous individual finger movements by imaging muscle activity using a wearable ultrasonic system. IEEE Transactions on Neural Systems and Rehabilitation Engineering 22 (1): 69–76.

    Article  Google Scholar 

  59. Hariharan, Harishwaran, Nima Aklaghi, Clayton A. Baker, Huzefa Rangwala, Jana Kosecka, and Siddhartha Sikdar. 2016. Classification of motor intent in transradial amputees using sonomyography and spatio-temporal image analysis. In Medical imaging 2016: Ultrasonic imaging and tomography, vol. 9790, 97901Q. International Society for Optics and Photonics.

    Google Scholar 

  60. Gordon, Claire C., Thomas Churchill, Charles E. Clauser, Bruce Bradtmiller, John T. McConville, Ilse Tebbetts, and Robert A. Walker. 1989. Anthropometric survey of us army personnel: Summary statistics, interim report for 1988. Technical report, Anthropology Research Project Inc Yellow Springs OH.

    Google Scholar 

  61. Szegedy, Christian, Sergey Ioffe, Vincent Vanhoucke, and Alexander A. Alemi. 2017. Inception-v4, inception-resnet and the impact of residual connections on learning. In Thirty-First AAAI conference on artificial intelligence.

    Google Scholar 

  62. Khazendar, S., A. Sayasneh, H. Al-Assam, Du Helen, L. Jeroen Kaijser, Dirk Timmerman Ferrara, S. Jassim, and Tom Bourne. 2015. Automated characterisation of ultrasound images of ovarian tumours: The diagnostic accuracy of a support vector machine and image processing with a local binary pattern operator. Facts, Views & Vision in ObGyn 7 (1): 7.

    Google Scholar 

  63. Oquab, Maxime, Leon Bottou, Ivan Laptev, and Josef Sivic. 2014. Learning and transferring mid-level image representations using convolutional neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, 1717–1724.

    Google Scholar 

  64. Cireşan, Dan C., Ueli Meier, and Jürgen Schmidhuber. 2012. Transfer learning for latin and chinese characters with deep neural networks. In The 2012 international joint conference on neural networks (IJCNN), 1–6. IEEE.

    Google Scholar 

  65. Yosinski, Jason, Jeff Clune, Yoshua Bengio, and Hod Lipson. 2014. How transferable are features in deep neural networks? In Advances in neural information processing systems, 3320–3328.

    Google Scholar 

  66. Yu, Fisher, and Vladlen Koltun. 2015. Multi-scale context aggregation by dilated convolutions. arXiv:1511.07122.

Download references

Author information

Authors and Affiliations

  1. Georgia Institute of Technology, Atlanta, GA, USA

    Gil Weinberg

  2. Novato, CA, USA

    Mason Bretan

  3. Cornell University, Ithaca, NY, USA

    Guy Hoffman

  4. Atlanta, GA, USA

    Scott Driscoll

Authors
  1. Gil Weinberg
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Mason Bretan
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Guy Hoffman
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Scott Driscoll
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Gil Weinberg .

Rights and permissions

Reprints and permissions

Copyright information

© 2020 Springer Nature Switzerland AG

About this chapter

Check for updates. Verify currency and authenticity via CrossMark

Cite this chapter

Weinberg, G., Bretan, M., Hoffman, G., Driscoll, S. (2020). “Wear it”—Wearable Robotic Musicians. In: Robotic Musicianship. Automation, Collaboration, & E-Services, vol 8. Springer, Cham. https://doi.org/10.1007/978-3-030-38930-7_7

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-38930-7_7

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-38929-1

  • Online ISBN: 978-3-030-38930-7

  • eBook Packages: EngineeringEngineering (R0)

Publish with us

Policies and ethics

Search

Navigation