Skip to main content
SpringerLink
Log in

Design space exploration of hardware platforms for interactive low latency movement sonification

  • Original Paper
  • Published:
Journal on Multimodal User Interfaces Aims and scope Submit manuscript

Abstract

Objective real-time acoustic feedback of complex movements arises in applications ranging from assisted training in sport to medical rehabilitation. This so called interactive movement sonification enables continuous feedback. Multiple studies have shown the benefit of sonification in movement relearning. Ubiquitous feedback requires a mobile and wearable hardware platform independent from any application. Therefore, in this paper a design space exploration regarding the design objectives power dissipation and latency is performed. Especially in sports or medical rehabilitation a low-latency feedback is crucial. Furthermore, low power dissipation mobile devices enable the utilization of the sonification technique within new applications. The most computational intensive task within the data-flow is real-time interactive sound synthesis. In contrast to commercial processors, customizable processors enable execution time reduction of critical sections by integration of application specific instruction set extensions. This paper aims to show the benefits from processor customization, as well as the location of consumer electronics devices within the design space.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

Abbreviations

Ăngström:

Linux distribution tailored for mobile devices

API:

Application programming interface

ASIC:

Application specific integrated circuit

ASIP:

Application specific instruction set processor

CI:

Custom instruction

CODEC:

Coder–decoder

CPU:

Central processing unit

DSP:

Digital signal processor

FPGA:

Field-programmable gate array

FPU:

Floating-point unit

GPS:

Global positioning system

GPP:

General purpose processor

GUI:

Graphical user interface

IIR:

Infinite impulse response

IMU:

Inertial measurement unit

MAC:

Media access control

MTw:

Xsens wireless IMU

MTx:

Xsens tethered IMU

OS:

Operating System

PC:

Personal computer

RISC:

Reduced instruction set computer

SDK:

Software development kit

SDL:

Simple direct media layer

SoC:

System on a chip

SIMD:

Single instruction, multiple data

STK:

Sound synthesis toolkit

TI:

Texas instruments

TIE:

Tensilica instruction-set extension language

USB:

Universal serial bus

WLAN:

Wireless local area network

XBM:

Xsens IMU bus master device

References

  1. Heidenreich PA, Trogdon JG, Khavjou OA, Butler J, Dracup K, Ezekowitz MD, Finkelstein EA, Hong Y, Johnston SC, Khera A (2011) Forecasting the future of cardiovascular disease in the United States a policy statement from the American heart association. In: Circulation, Am. Heart Assoc., pp 933–944

  2. Wallis I, Ingalls T, Rikakis T, Olson L, Chen Y, Xu W, Sundaram H (2007) Real-time sonification of movement for an immersive stroke rehabilitation environment. In: Proceedings of the 13th international conference on auditory display, pp 26–29

  3. Duff M, Chen Y, Cheng L, Liu S-M, Blake P, Wolf SL, Rikakis T (2013) Adaptive mixed reality rehabilitation improves quality of reaching movements more than traditional reaching therapy following stroke. Neurorehabil Neural Repair 27(4):306–315

  4. Roerdink M, Lamoth CJ, Kwakkel G, van Wieringen PC, Beek PJ (2007) Gait coordination after stroke: benefits of acoustically paced treadmill walking. Phys Ther 87(8):1009–1022

    Article  Google Scholar 

  5. Chouhan S, Kumar S (2012) Comparing the effects of rhythmic auditory cueing and visual cueing in acute hemiparetic stroke. Int J Ther Rehabilitation 19(6):344–351

    Article  Google Scholar 

  6. Schaffert N, Mattes K, Effenberg A (2011) The sound of rowing stroke cycles as acoustic feedback. In: The 17th annual conference on auditory display

  7. Dubus G, Bresin R (2010) Sonification of sculler movements, development of preliminary methods. In: Proceedings of ISon 2010, 3rd interactive sonification workshop, pp 39-43

  8. Eriksson M, Bresin R (2010) Improving running mechanics by use of interactive sonification. In: Proceedings of the interaction sonification workshop (ISon) 2010, pp 95–98

  9. Levitin D, MacLean K, Mathews M, Chu L, Jensen E (2000) The perception of cross-modal simultaneity. Int J Comput Anticip Syst 323–329

  10. Scavone G, Cook P (2005) RtMidi, RtAudio, and a synthesis toolkit (STK) update. In: International computer music conference

  11. Xsens Technologies BV (2015). http://xsens.com/. Accessed 27 June 2015

  12. Shimmer Research (2015). http://shimmer-research.com/. Accessed 27 June 2015

  13. Intersense (2015). http://intersense.com/. Accessed 27 June 2015

  14. Brunetti F, Moreno J, Ruiz A, Rocon E, Pons J (2006) A new platform based on IEEE802.15.4 wireless inertial sensors for motion caption and assessment. In: Engineering in medcine and biology society, 2006. EMBS’06. 28th Annual international conference of the IEEE, pp 649–6500

  15. Zhuohua L, Zecca M, Sessa S, Bartolomeo L, Ishii H, Takanishi A (2011) Development of the wireless ultra-miniaturized inertial measurement unit WB-4: Preliminary performance evaluation. In: Engineering in medicine and biology society, EMBC, 2011 annual international conference of the IEEE, pp 6927–6930

  16. Comotti D, Ermidoro M, Galizzi M, Vitali A (2013) Development of a wireless low-power multi-sensor network for motion tracking applications. In: Body sensor networks (BSN), 2013 IEEE international conference on, pp 1-6

  17. Sonic Golf (2015) Golf training system utilizing auditory feedback. http://sonicgolf.com/. Accessed 25 April 2015

  18. BeSB GmbH (2015) Sofirow—Real-time sonification of rowing boat motion. http://accrow.de/. Accessed 21 Sept 2015

  19. RowingInMotion (2015) Rowing coach app using auditory feedback. http://rowinginmotion.com/. Accessed 21 Sept 2015

  20. Barrass S, Schaffert N, Barrass T (2010) Probing preferences between six designs of interactive sonifications for recreational sports, health and fitness. In: Human interaction with auditory displays—Proceedings of the interactive sonification workshop, pp 23–30

  21. Cesarini D, Schaffert N, Manganiello C, Mattes K, Avvenuti M (2013) A smartphone based sonification and telemetry platform for on-water rowing training. In: International conference on auditory display, pp 297–300

  22. Brückner H-P, Bartels C, Blume H (2011) PC-based real-time sonification of human motion captured by inertial sensors. In: The 17th annual international conference on auditory display (ICAD)

  23. Brückner H-P, Wielage M, Blume H (2012) Intuitive and interactive movement sonification on a heterogeneous RISC/DSP platform. In: The 18th international conference on auditory display, pp 75–82

  24. BeagleBoard Community (2015) Texas Instruments BeagleBone Black development plattform. http://beagleboard.org/Products/BeagleBone+Black/. Accessed 21 Sept 2015

  25. Blackberry (2015) Blackberry Z10 smartphone. http://blackberry.com/smartphones/blackberry-z10.html/. Accessed 31 Jan 2015

  26. Brückner H-P, Theimer W, Blume H (2014) Real-time low latency movement sonification in stroke rehabilitation based on a mobile platform. In: Consumer electronics (ICCE), 2014 IEEE international conference on, pp 242–243

  27. Altera corporation (2015) Altera Nios II softcore processor—overview. http://altera.com/devices/processor/nios2/ni2-index.html/. Accessed 28 March 2015

  28. Tenilica (2015) Xtensa LX4. http://tensilica.com/. Accessed 21 Sept 2015

Download references

Acknowledgments

The authors thank the student assistants Benjamin Krüger, Moritz Bente, Sebastian Henning and Jennifer Hundt for their contributions in porting the application on various processing platforms and analysis of execution times.

Author information

Authors and Affiliations

  1. Institute of Microelectronic Systems, Leibniz Universität Hannover, Appelstraße 4, 30167, Hannover, Germany

    Hans-Peter Brückner, Sebastian Lesse & Holger Blume

  2. BlackBerry, Bochum, Germany

    Wolfgang Theimer

Authors
  1. Hans-Peter Brückner
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Sebastian Lesse
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Wolfgang Theimer
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Holger Blume
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Hans-Peter Brückner.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Brückner, HP., Lesse, S., Theimer, W. et al. Design space exploration of hardware platforms for interactive low latency movement sonification. J Multimodal User Interfaces 10, 1–11 (2016). https://doi.org/10.1007/s12193-015-0199-y

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12193-015-0199-y

Keywords

Search

Navigation