“It’s like holding a human heart”: the design of Vital + Morph, a shape-changing interface for remote monitoring

AI & SOCIETY volume 33, pages599–619(2018)Cite this article

Abstract

Based on the concept of data physicalization, we developed Vital + Morph, an interactive surface for remote connection and awareness of clinical data. It enables users located in remote places to monitor and feel the vital signs measured from a hospitalized person through shape-change. We propose shape-changing interfaces as a way of making data physicalization a richer, intriguing and memorable experience that communicates complex information and insights about data. To demonstrate and validate our proposed concept, we developed an exploratory study about the design and its implications. For evaluating the social impact of shape-changing interfaces in the context of remote monitoring, we presented Vital + Morph in several Media Art festivals. We collected and analyzed the feedback from the visitors during the exhibitions, and discussed the possibilities of the proposed system. A preliminary evaluation shows how shape-changing displays are perceived by users, which establishes not only the potential benefits but also highlights the concerns that several users have raised. Through this study, we aim to contribute to the design of remote monitoring systems by providing a novel approach for displaying clinical data that consider the richness of the physical world. In today’s information-driven society, we should not just focus on how abstract data are collected and analyzed, but also on how it can be presented and incorporated into our daily lives.

This is a preview of subscription content, access via your institution.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17

Notes

  1. 1.

    https://artcom.de/en/project/kinetic-sculpture/.

  2. 2.

    https://www.microsoft.com/en-us/research/project/physical-charts/.

  3. 3.

    https://physionet.org/physiobank/.

References

  1. Barrass S (2016) Diagnosing blood pressure with acoustic sonification singing bowls. Int J Hum Comput Stud 85(C):68–71. doi:10.1016/j.ijhcs.2015.08.007

    Article  Google Scholar 

  2. Bitterman N (2006) Technologies and solutions for data display in the operating room. J Clin Monit Comput 20(3):165–173. doi:10.1007/s10877-006-9017-0

    Article  Google Scholar 

  3. Brown C, Hurst A (2012) VizTouch: automatically generated tactile visualizations of coordinate spaces. In: Proceedings of of TEI’12. doi:10.1145/2148131.2148160

  4. Cermack M (2006) Monitoring and telemedicine support in remote environments and in human space flight. Br J Anaesth 97(1):107–114. doi:10.1093/bja/ael132

    Article  Google Scholar 

  5. Coelho M, Zigelbaum J (2011) Shape-changing interfaces. Pers Ubi Comput 15(2):161–173. doi:10.1007/s00779-010-0311-y

    Article  Google Scholar 

  6. Drews FA, Westenskow DR (2016) The right picture is worth a thousand numbers: data displays in anesthesia. Hum Factors 48(1):59–71. doi:10.1518/001872006776412270

    Article  Google Scholar 

  7. Dubus G, Bresin R (2014) A systematic review of mapping strategies for the sonification of physical quantities. PLoS One 9:4. doi:10.1371/journal.pone.0096018

    Article  Google Scholar 

  8. Follmer S, Leithinger D, Olwal A, Hogge A, Ishii H (2013) inFORM: dynamic physical affordances and constraints through shape and object actuation. In: Proceedings of UIST’13. doi:10.1145/2501988.2502032

  9. Garnæs K, Grünberger O, Kjeldskov J, Skov MB (2007) Designing technologies for presence-in-absence: illustrating the cube and the picture frame. Pers Ubi Comput 11(5):403–408. doi:10.1007/s00779-006-0072-9

    Article  Google Scholar 

  10. Görges M, Staggers N (2008) Evaluations of physiological monitoring displays: a systematic review. J Clin Monit Comput. doi:10.1007/s10877-007-9106-8

    Article  Google Scholar 

  11. Greenberg S, Kuzuoka H (1999) Using digital but physical surrogates to mediate awareness, communication and privacy in media spaces. Pers Technol 3(4):182–198. doi:10.1007/BF01540552

    Article  Google Scholar 

  12. Hardy J, Weichel C, Taher F, Vidler J, Alexander J (2015) ShapeClip: towards rapid prototyping with shape-changing displays for designers. In: Proceedings of CHI’15. doi:10.1145/2702123.2702599

  13. Heiner JM, Hudson SE, Tanaka K (1999) The information percolator: ambient information display in a decorative object. In: Proceedings of UIST’99. doi:10.1145/320719.322595

  14. Hemmert F, Löwe M, Wohlauf A, Joost G (2013) Animate mobiles: proxemically reactive posture actuation as a means of relational interaction with mobile phones. In: Proceedings of TEI’12. doi:10.1145/2460625.2460669

  15. Iaconesi S, Persico O (2016) La cura. Codice Edizioni

  16. Ishii H, Ullmer B (1997) Tangible bits: towards seamless interfaces between people, bits and atoms. In: Proceedings of CHI’97. doi:10.1145/258549.258715

  17. Ishii H, Lakatos D, Bonanni L, Labrune JB (2012) Radical atoms: beyond tangible bits, toward transformable materials. Interactions 19(1):38–51. doi:10.1145/2065327.2065337

    Article  Google Scholar 

  18. Iwata H (2005) Art and technology in interface devices. In: Proceedings of VRST’05. doi:10.1145/1101616.1101617

  19. Iwata H, Yano H, Nakaizumi F, Kawamura R (2001) Project FEELEX: adding haptic surface to graphics. In: Proceedings of SIGGRAPH’01. doi:10.1145/383259.383314

  20. Iwata H, Yano H, Ono N (2005) Volflex. In: Emerging technologies SIGGRAPH’05. doi:10.1145/1187297.1187329

  21. Jansen Y, Dragicevic P, Fekete JD (2013) Evaluating the efficiency of physical visualizations. In: Proceedings of CHI’13. doi:10.1145/2470654.2481359

  22. Jansen Y et al (2015) Opportunities and challenges for data physicalization. In: Proceedings of CHI’15. doi:10.1145/2702123.2702180

  23. Janssen JH, Bailenson JN, Ijsselsteijn WA, Westerink JHDM (2010) Intimate heartbeats: opportunities for affective communication technology. IEEE Trans Affect Comput 1(2):72–80. doi:10.1109/T-AFFC.2010.13

    Article  Google Scholar 

  24. Johnson GJ, Ambrose PJ (2006) Neo-tribes: the power and potential of online communities in health care. Commun ACM 49(1):107–113. doi:10.1145/1107458.1107463

    Article  Google Scholar 

  25. Jovanon E, Starcevic D, Radivojevic V (2001) Perceptualization of biomedical data. In: Akay M, Marsh A (eds) Information technologies in medicine, vol I. Medical simulation and education. Wiley, New York, pp 189–204

    Google Scholar 

  26. Kaltenbrunner M, Jorda S, Geiger G, Alonso M (2006) The reacTable*: a collaborative musical instrument. In: Proceedings of IEEE WETICE’06. doi:10.1109/WETICE.2006.68

  27. Korhonen I, Parkka J, Van Gils M (2003) Health monitoring in the home of the future. IEEE Eng Med Biol 22(3):66–73. doi:10.1109/MEMB.2003.1213628

    Article  Google Scholar 

  28. Kusunoki DS, Aleksandra S, Zhang Z, Burd RS (2013) Understanding visual attention of teams in dynamic medical settings through vital signs monitor use. In: Proceedings of CSCW’13. doi:10.1145/2441776.2441836

  29. Leithinger D, Follmer S, Olwal A, Ishii H (2014) Physical telepresence: shape capture and display for embodied, computer-mediated remote collaboration. In: Proceedings of UIST’14. doi:10.1145/2642918.2647377

  30. Lindlbauer D, Grønbæk JE, Birk M, Halskov K, Alexa M, Müller J (2016) Combining shape-changing interfaces and spatial augmented reality enables extended object appearance. In: Proceedings of CHI’16. doi:10.1145/2858036.2858457

  31. Lynn G (1999) Animated form. Princeton Architectural Press, New York

    Google Scholar 

  32. Maciejewski R, Choi S, Ebert DS, Tan HZ (2005) Multi-modal perceptualization of volumetric data and its application to molecular docking. In: Proceedings of world haptics conference. doi:10.1109/WHC.2005.97

  33. Mankoff J, Dey AK, Hsieh G, Kientz J, Lederer S, Ames M (2003) Heuristic evaluation of ambient displays. In: Proceedings of CHI’03. doi:10.1145/642611.642642

  34. Manovich L (2008) Introduction to info-aesthetics. http://www.mariabuszek.com/kcai/PoMoSeminar/Readings/ManovichInfoAesthetics.pdf. Accessed 15 June 2017

    Google Scholar 

  35. McLanders M, Santomauro C, Tran J, Sanderson P (2014) Tactile displays of pulse oximetry in integrated and separated configurations. In: Proceedings of HFES’14. doi:10.1177/1541931214581158

    Article  Google Scholar 

  36. Nafus D (2016) The domestication of data: why embracing digital data means embracing bigger questions. In: Proceedings of EPIC’16. doi:10.1111/1559-8918.2016.01097

    Article  Google Scholar 

  37. Nakamichi D, Nishio S, Ishiguro H (2014) Training of telecommunication through teleoperated android “Telenoid” and its effect. In: Proceedings of IEEE RO-MAN. doi:10.1109/ROMAN.2014.6926396

  38. Nangalia V, Prytherch DR, Smith GB (2010) Health technology assessment review: remote monitoring of vital signs—current status and future challenges. Crit Care 14(5):233. doi:10.1186/cc9208

    Article  Google Scholar 

  39. Pangaro G, Maynes-Aminzade D, Ishii H (2002) The actuated workbench: computer-controlled actuation in tabletop tangible interfaces. In: Proceedings of UIST’02. doi:10.1145/571985.572011

  40. Park YW, Park J, Nam TJ (2015) The trial of Bendi in a coffeehouse: use of a shape-changing device for a tactile-visual phone conversation. In: Proceedings of CHI’15. doi:10.1145/2702123.2702326

  41. Paul C (2015) From immateriality to neomateriality: art and the conditions of digital materiality. In: Proceedings of ISEA 2015. http://isea2015.org/proceeding/submissions/ISEA2015_submission_154.pdf. Accessed 15 June 2017

  42. Poupyrev I, Nashida T, Okabe M (2007) Actuation and tangible user interfaces: the Vaucanson duck, robots, and shape displays. In: Proceedings of TEI’07. doi:10.1145/1226969.1227012

  43. Rasmussen MK, Pedersen EW, Petersen MG, Hornbæk K (2012) Shape-changing interfaces: a review of the design space and open research questions. In: Proceedings of CHI’12. doi:10.1145/2207676.2207781

  44. Schmitz M (2010) Concepts for life-like interactive objects. In: Proceedings of TEI’11. doi:10.1145/1935701.1935732

  45. Sekiguchi D, Inami M, Tachi S (2001) RobotPHONE: RUI for interpersonal communication. In: Proceedings of CHI EA’01. doi:10.1145/634067.634231

  46. Slovák P, Janssen J, Fitzpatrick G (2012) Understanding heart rate sharing: towards unpacking physiological space. In: Proceedings of CHI’12. doi:10.1145/2207676.2208526

  47. Sommerer C, Mignonneau L (2004) Mobile feelings-wireless communication of heartbeat and breath for mobile art. In: Proceedings of ICAT’04

  48. Sugiyama Y, Hirai S (2006) Crawling and jumping by a deformable robot. Int J Robot Res 25(5–6):603–620. doi:10.1177/0278364906065386

    Article  Google Scholar 

  49. Taher F et al (2017) Investigating the use of a dynamic physical bar chart for data exploration and presentation. IEEE Trans Vis Comput Gr 23(1):451–460. doi:10.1109/TVCG.2016.2598498

    Article  Google Scholar 

  50. Thompson D (1992) On growth and form. Cambridge University Press, Cambridge

    Google Scholar 

  51. Van de Moere A (2008) Beyond the tyranny of the pixel: exploring the physicality of information visualization. In: Proceedings of IV’08. doi:10.1109/IV.2008.84

  52. Vertegaal R, Poupyrev I (eds) (2008) Special issue: organic user interfaces. Commun ACM 51(6)

  53. Wakita A, Nakano A, Kobayashi N (2010) Programmable blobs: a rheologic interface for organic shape design. In: Proceedings of TEI’11. doi:10.1145/1935701.1935760

  54. Weiser M (1991) The computer for the 21st century. Sci Am 265(3):94–104

    Article  Google Scholar 

  55. Willett W, Jansen Y, Dragicevic P (2017) Embedded data representations. IEEE Trans Vis Comput Gr 23(1):461–470. doi:10.1109/TVCG.2016.2598608

    Article  Google Scholar 

  56. Wisneski C et al (1998) Ambient displays: turning architectural space into an interface between people and digital information. In: Streitz NA, Konomi S, Burkhardt HJ (eds) Cooperative buildings: integrating information, organization, and architecture. Cobuild 1998, Lecture Notes in Computer Science, vol 1370. Springer, New York, pp 22–32

    Google Scholar 

  57. Yao L et al (2015) bioLogic: natto cells as nanoactuators for shape changing interfaces. In: Proceedings of CHI’15. doi:10.1145/2702123.2702611

Download references

Acknowledgements

We would like to thank Kai Sasaki and Shori Kano for the help provided during the development of Vital + Morph. In addition, we want to thank Prof. Hideaki Kuzuoka, Masakazu Hirokawa, and Alessio Chierico for the fruitful discussions and comments provided during the development of this manuscript.

Author information

Affiliations

  1. Empowerment Informatics, School of Integrative and Global Majors, University of Tsukuba, Tennodai 1-1-1, Tsukuba, 305-8573, Japan

    Alberto Boem

  2. Faculty of Engineering, Information and Systems, University of Tsukuba, Tennodai 1-1-1, Tsukuba, 305-8573, Japan

    Hiroo Iwata

Authors
  1. Alberto Boem
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Hiroo Iwata
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Alberto Boem.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Boem, A., Iwata, H. “It’s like holding a human heart”: the design of Vital + Morph, a shape-changing interface for remote monitoring. AI & Soc 33, 599–619 (2018). https://doi.org/10.1007/s00146-017-0752-1

Download citation

Keywords

  • Shape-changing interfaces
  • Remote monitoring
  • Information technologies
  • Haptics
  • Vital signs