Skip to main content

Advertisement

SpringerLink
Log in

LAB-IN-A-BOX: semi-automatic tracking of activity in the medical office

  • Original Article
  • Published:
Personal and Ubiquitous Computing Aims and scope Submit manuscript

Abstract

Patient-centered healthcare and increased efficiency are major goals of modern medicine, and physician–patient interaction and communication are a cornerstone of clinical encounters. The introduction of the electronic health record (EHR) has been a key component in shaping not only organization, clinical workflow and ultimately physicians’ clinical decision making, but also patient–physician communication in the medical office. In order to inform the design of future EHR interfaces and assess their impact on patient-centered healthcare, designers and researchers must understand the multimodal nature of the complex physician–patient–EHR system interaction. However, characterizing multimodal activity is difficult and expensive, often requiring manual coding of hours of video data. We present our Lab-in-a-Box solution that enables the capture of multimodal activity in real-world settings. We focus here on the medical office where our Lab-in-a-Box system exploits a range of sensors to track computer-based activity, speech interaction, visual attention and body movements, and automatically synchronize and segment this data. The fusion of multiple sensors allows us to derive initial activity segmentation and to visualize it for further interactive analysis. By empowering researchers with cutting-edge data collection tools and accelerating analysis of multimodal activity in the medical office, our Lab-in-a-Box has the potential to uncover important insights and inform the next generation of Health IT systems.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

Similar content being viewed by others

Notes

  1. http://kinectforwindows.org.

  2. http://www.vicon.com.

  3. http://www.techsmith.com/morae.

  4. http://www.dev-audio.com.

  5. http://www.smivision.com/redm.

  6. Although we are not using any additional devices for sensing activity in the medical office, ChronoSense already supports data collection with LeapMotion for tracking finger movements (http://leapmotion.com), and the development version of the new Kinect for Windows v2. We are in the process of integrating also affordable eye trackers such as the EyeTribe (https://theeyetribe.com).

  7. This can be configured, but with standard laptop computers, an interval of 0.1 s (10 Hz) works best.

  8. http://www.vamp-plugins.org.

  9. In our ongoing studies we experienced several failures of the Morae software, as well as of the eye tracking, and continuous changes in the physical distance of the physician from the Kinect sometimes prevented continuous data collection.

  10. http://leapmotion.com.

  11. http://threegear.com.

References

  1. Als A (1997) The desk-top computer as a magic box: patterns of behaviour connected with the desk-top computer; gps’ and patients’ perceptions. Fam Pract 14(1):17–23

    Article  Google Scholar 

  2. Armijo D, McDonnell C, Werner K (2009) Electronic health record usability: evaluation and use case framework. Agency for Healthcare Research and Quality, U.S. Department of Health and Human Services, AHRQ Publication No. 09(10)-0091-1-EF, October 2009

  3. Bateson G (1972) Steps to an ecology of mind: collected essays in anthropology, psychiatry, evolution, and epistemology. University of Chicago Press, Chicago

    Google Scholar 

  4. Bouamrane MM, Luz S (2007) Meeting browsing. Multimed Syst 12(4–5):439–457

    Article  Google Scholar 

  5. Bouamrane MM, Mair FS (2013) A study of general practitioners’ perspectives on electronic medical records systems in NHSScotland. BMC Med Inform Decis Mak 13(1):58

    Article  Google Scholar 

  6. Calvitti A, Farber N, Chen Y, Zuest D, Liu L, Bell K, Gray B, Agha Z (2012) Temporal analysis of physicians’ EHR workflow during outpatient visits. In: Proceedings of healthcare informatics, imaging and systems biology (HISB), pp 140–140

  7. Chen Y, Ngo V, Harrison S, Duong V (2011) Unpacking exam-room computing: negotiating computer-use in patient-physician interactions. In: Proceedings of CHI’11, pp 3343–3352

  8. Chen Y, Ngo V, Harrison S, Duong V (2010) Documenting transitional information in EMR. In: Proceedings of CHI’10, pp 1787–1796

  9. Clark RA, Pua YH, Fortin K, Ritchie C, Webster KE, Denehy L, Bryant AL (2012) Validity of the Microsoft Kinect for assessment of postural control. Gait Posture 36(3):372–377

    Article  Google Scholar 

  10. Committee on Quality of Health Care in America (2001) Institute of Medicine (National Academy Press): crossing the quality chasm: a new health system for the twenty-first century

  11. Darmon D, Sauvant R, Staccini P, Letrilliart L (2014) Which functionalities are available in the electronic health record systems used by french general practitioners? an assessment study of 15 systems. Int J Med Inform 83(1):37–46

    Article  Google Scholar 

  12. Dixon S, Widmer G (2005) MATCH: a music alignment tool chest. In: Proceedings of ISMIR, pp 492–497

  13. Dutta T (2012) Evaluation of the kinect sensor for 3-D kinematic measurement in the workplace. Appl Ergon 43(4):645–649

    Article  Google Scholar 

  14. Hutchins E (2003) Cognitive ethnography. Plenary address at the 25th meeting of the Cognitive Science Society, Boston

  15. El-Kareh R, Gandhi TK, Poon EG, Newmark LP, Ungar J, Lipsitz S, Sequist TD (2009) Trends in primary care clinician perceptions of a new electronic health record. J Gen Intern Med 24(4):464–468

    Article  Google Scholar 

  16. Embi P, Yackel T, Logan J, Bowen J, Cooney T, Gorman P (2004) Impacts of computerized physician documentation in a teaching hospital: perceptions of faculty and resident physicians. J Am Med Inform Assoc 11(4):300–309

    Article  Google Scholar 

  17. Fouse A, Weibel N, Hutchins E, Hollan JD (2011) ChronoViz: a system for supporting navigation of time-coded data. In: Proceedings of CHI’11

  18. Frankel R, Altschuler A, George S (2005) Effects of exam-room computing on clinician–patient communication: a longitudinal qualitative study. J Gen Intern Med 20(8):677–682

    Article  Google Scholar 

  19. Gadd CS, Penrod LE (2000) Dichotomy between physicians’ and patients’ attitudes regarding EMR use during outpatient encounters. In: Proceedings of AMIA, p 275

  20. Gibbings-Isaac D, Iqbal M, Tahir MA, Kumarapeli P, de Lusignan S (2012) The pattern of silent time in the clinical consultation: an observational multichannel video study. Fam Pract 29(5):616–621

    Article  Google Scholar 

  21. Han J, Shao L, Xu D, Shotton J (2013) Enhanced computer vision with microsoft kinect sensor: a review. IEEE Trans Cybern 43(5):1318–1334

    Article  Google Scholar 

  22. Han Y, Carcillo J, Venkataraman S (2005) Unexpected increased mortality after implementation of a commercially sold computerized physician order entry system. Pediatrics 116(6):1506–12

    Article  Google Scholar 

  23. Hazlehurst B, Gorman P, McMullen C (2008) Distributed cognition: an alternative model of cognition for medical informatics. Int J Med Inform 77(4):226

    Article  Google Scholar 

  24. Heath C, Luff P (1996) Documents and professional practice: bad organizational reasons for good clinical records. In: Proceedings of CSCW’96, pp 354–363

  25. Hendrich A, Chow M, Skierczynski B, Lu Z (2008) A 36-hospital time and motion study: how do medical-surgical nurses spend their time? Perm. J. 12(3):25–34

    Article  Google Scholar 

  26. Hillestad R, Bigelow J, Bower A (2005) Can electronic medical record systems transform health care? Potential health benefits, savings, and costs. Health Aff (Millwood) 24(5):1103–1117

    Article  Google Scholar 

  27. Hollan J, Hutchins E, Kirsh D (2000) Distributed cognition: toward a new foundation for human–computer interaction research. ACM Trans Comput Hum Interact 7(2):174196

    Article  Google Scholar 

  28. Hollan JD, Hutchins EL (2009) Opportunities and challenges for augmented environments: a distributed cognition perspective. In: Lahlou S (ed) Designing user friendly augmented work environments: from meeting rooms to digital collaborative spaces. Springer, Berlin

    Google Scholar 

  29. Holroyd-Leduc JM, Lorenzetti D, Straus SE, Sykes L, Quan H (2011) The impact of the electronic medical record on structure, process, and outcomes within primary care: a systematic review of the evidence. J Am Med Inform Assoc 18(6):732–737

    Article  Google Scholar 

  30. Hutchins E (1995) Cognition in the wild. MIT Press, Cambridge

    Google Scholar 

  31. Jha AK, DesRoches CM, Campbell EG et al (2009) Use of electronic health records in US hospitals. N Engl J Med 360(16):1628–1638

    Article  Google Scholar 

  32. Kaplan B, Maxwell JA (2005) Qualitative research methods for evaluating computer information systems. In: Evaluating the organizational impact of healthcare information systems. Springer, Berlin, pp 30–55

  33. Koppel R, Metlay J, Cohen A (2005) Role of computerized physician order entry systems in facilitating medication errors. JAMA 293(10):1197–203

    Article  Google Scholar 

  34. Lau F, Price M, Boyd J, Partridge C, Bell H, Raworth R (2012) Impact of electronic medical record on physician practice in office settings: a systematic review. BMC Med Inform Dec Mak 12(1):10

    Article  Google Scholar 

  35. Lelievre S, Schultz K (2010) Does computer use in patient-physician encounters influence patient satisfaction? Can Fam Phys 56(1):e6–e12

    Google Scholar 

  36. Luz S (2012) The nonverbal structure of patient case discussions in multidisciplinary medical team meetings. ACM Trans Inf Syst (TOIS) 30(3):17

    Article  Google Scholar 

  37. Lyle J (2003) Stimulated recall: a report on its use in naturalistic research. Br Educ Res J 29(6):861–878

    Article  Google Scholar 

  38. Lyons J, Dixit R, Emmenegger C, Hill L, Weibel N, Hollan J (2013) Factors affecting physician–patient communication in the medical exam room. In: Proceedings of HCI international 2013, pp 187–191

  39. Makoul G, Curry R, Tang P (2001) The use of electronic medical records: communication patterns in outpatient encounters. J Am Med Inform Assoc 8(6):610–615

    Article  Google Scholar 

  40. Margalit R, Roter D, Dunevant M, Larson S, Reis S (2006) Electronic medical record use and physician-patient communication: an observational study of israeli primary care encounters. Patient Educ Couns 61(1):134–141

    Article  Google Scholar 

  41. McCowan I, Gatica-Perez D, Bengio S, Lathoud G, Barnard M, Zhang D (2005) Automatic analysis of multimodal group actions in meetings. IEEE Trans Pattern Anal Mach Intell 27(3):305–317

    Article  Google Scholar 

  42. Mollo V, Falzon P (2004) Auto-and allo-confrontation as tools for reflective activities. Appl Ergon 35(6):531–540

    Article  Google Scholar 

  43. Pearce C, Arnold M, Phillips C, Trumble S, Dwan K (2011) The patient and the computer in the primary care consultation. J Am Med Inform Assoc 18(2):138–142

    Article  Google Scholar 

  44. Poissant L, Pereira J, Tamblyn R, Kawasumi Y (2005) The impact of electronic health records on time efficiency of physicians and nurses: a systematic review. J Am Med Inform Assoc 12(5):505–516

    Article  Google Scholar 

  45. Rosenbloom S, Harrell FJ, Lehmann C, Schneider J, Spooner S, Johnson K (2006) Perceived increase in mortality after process and policy changes implemented with computerized physician order entry. Pediatrics 117(4):1452–6

    Article  Google Scholar 

  46. Shachak A, Reis S (2009) The impact of electronic medical records on patient–doctor communication during consultation: a narrative literature review. J Eval Clin Pract 15(4):641–649

    Article  Google Scholar 

  47. US Department of Health and Human Services, Center for Medicare and Medicaid Services (CMS): Meaningful Use: EHR Incentives Program (2011)

  48. Virapongse A, Bates D, Shi P (2008) Electronic health records and malpractice claims in office practice. Arch Intern Med 168(21):2362–2367

    Article  Google Scholar 

  49. Wang S, Middleton B, Prosser L (2003) A cost-benefit analysis of electronic medical records in primary care. Am J Med 114(5):397–403

    Article  Google Scholar 

  50. Wattenberg M, Viégas F, Hollenbach K (2007) Visualizing activity on wikipedia with chromograms. In: Proceedings of Interact’07, pp 272–287

  51. Weibel N, Emmenegger C, Lyons J, Dixit R, Hill L, Hollan J (2013) Interpreter-mediated physician–patient communication: Opportunities for multimodal healthcare interfaces. In: Proceedings of PervasiveHealth 2013, pp 113–120

  52. Weir C, Nebeker J, Hicken B, Campo R, Drews F, Lebar B (2007) A cognitive task analysis of information management strategies in a computerized provider order entry environment. J Am Med Inform Assoc 14(1):65–75

    Article  Google Scholar 

  53. Zhang Z (2012) Microsoft Kinect sensor and its effect. IEEE MultiMedia 19(2):4–10

    Article  Google Scholar 

  54. Zheng K, Padman R, Johnson MP, Diamond HS (2009) An interface-driven analysis of user interactions with an electronic health records system. JAMIA 16(2):228–237

    Google Scholar 

Download references

Acknowledgments

This research was funded by AHRQ Grant 1 R01 HS021290-01A1, Zia Agha PI. We would like to thank all participants of our ongoing studies (physicians and patients) without which we could not collect the rich data that we used as a baseline for designing the Lab-in-a-Box. Many thanks also to our colleagues participating in the QUICK and PACE research who helped the development of our tools with their advices. Also thanks to Shimona Carvalho for working on an early version of ChronoSense and to Jenny Tsao for her work on expanding it. A special acknowledgment to Adam Fouse, the designer and developer of ChronoViz who made possible the tight integration of the many data stream collected with Lab-in-a-Box with the development of ChronoViz templates, and finally to Jim Hollan, Ed Hutchins and the DCog-HCI lab at UCSD who were instrumental in bootstrapping this line of research.

Author information

Authors and Affiliations

  1. University of California San Diego, 9500 Gilman Drive, La Jolla, CA, 92093, USA

    Nadir Weibel, Steven Rick, Colleen Emmenegger & Zia Agha

  2. Veteran Affairs Medical Research Foundation, HSRD VA San Diego Healthcare System, San Diego, CA, 92161, USA

    Nadir Weibel, Steven Rick, Shazia Ashfaq & Alan Calvitti

  3. West Health Institute, 10350 N Torrey Pines Rd, La Jolla, CA, 92037, USA

    Zia Agha

Authors
  1. Nadir Weibel
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Steven Rick
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Colleen Emmenegger
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Shazia Ashfaq
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Alan Calvitti
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Zia Agha
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Nadir Weibel.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Weibel, N., Rick, S., Emmenegger, C. et al. LAB-IN-A-BOX: semi-automatic tracking of activity in the medical office. Pers Ubiquit Comput 19, 317–334 (2015). https://doi.org/10.1007/s00779-014-0821-0

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00779-014-0821-0

Keywords

Search

Navigation