Skip to main content
SpringerLink
Log in

A Hybrid 2D/3D User Interface for Radiological Diagnosis

  • Published:
Journal of Digital Imaging Aims and scope Submit manuscript

Abstract

This paper presents a novel 2D/3D desktop virtual reality hybrid user interface for radiology that focuses on improving 3D manipulation required in some diagnostic tasks. An evaluation of our system revealed that our hybrid interface is more efficient for novice users and more accurate for both novice and experienced users when compared to traditional 2D only interfaces. This is a significant finding because it indicates, as the techniques mature, that hybrid interfaces can provide significant benefit to image evaluation. Our hybrid system combines a zSpace stereoscopic display with 2D displays, and mouse and keyboard input. It allows the use of 2D and 3D components interchangeably, or simultaneously. The system was evaluated against a 2D only interface with a user study that involved performing a scoliosis diagnosis task. There were two user groups: medical students and radiology residents. We found improvements in completion time for medical students, and in accuracy for both groups. In particular, the accuracy of medical students improved to match that of the residents.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

Similar content being viewed by others

References

  1. Chen M, Mountford SJ, Sellen A. A study in interactive 3-D rotation using 2-D control devices. ACM SIGGRAPH Computer Graphics 1988;22(4):121–129.

    Article  Google Scholar 

  2. Hinckley K, Tullio J, Pausch R, Proffitt D, Kassell N. Usability analysis of 3D rotation techniques. Proceedings of the 10th Annual ACM Symposium on User Interface Software and Technology. ACM; 1997. p. 1–10.

  3. Bowman D, Kruijff E, LaViola Jr JJ, Poupyrev IP: 3D User interfaces: theory and practice, CourseSmart eTextbook. Addison-Wesley, 2004.

  4. Graves MJ, Black RT, Lomas DJ. Constrained surface controllers for three-dimensional image data reformatting. Radiology 2009;252(1):218–224.

    Article  PubMed  Google Scholar 

  5. zSpace zSpace, inc. https://zspace.com/. Accessed: 2017-01-09.

  6. Emerson T, Prothero JD, Weghorst SJ: Medicine and virtual reality: a guide to the literature (medVR). Human Interface Technology Laboratory, 1994.

  7. Ayache N. Medical computer vision, virtual reality and robotics. Image and Vision Computing 1995;13(4): 295–313.

    Article  Google Scholar 

  8. Székely G, Satava RM. Virtual reality in medicine. BMJ: British Medical Journal 1999;319(7220):1305.

    Article  PubMed  PubMed Central  Google Scholar 

  9. Gallo L, Minutolo A, De Pietro G. A user interface for VR-ready 3D medical imaging by off-the-shelf input devices. Computers in Biology and Medicine 2010;40(3):350–358.

    Article  CAS  PubMed  Google Scholar 

  10. Hand C. A survey of 3D interaction techniques. Computer graphics forum, vol 16 no 5. Wiley; 1997. p. 269–281.

  11. Shoemake K. Arcball: a user interface for specifying three-dimensional orientation using a mouse. Graphics Interface; 1992. p. 151–156.

    Google Scholar 

  12. Henriksen K, Sporring J, Hornbæk K. Virtual trackballs revisited. IEEE Trans Visual Comput Graphics 2004;10(2):206–216.

    Article  Google Scholar 

  13. Bade R, Ritter F, Preim B. Usability comparison of mouse-based interaction techniques for predictable 3D rotation. International Symposium on Smart Graphics. Springer; 2005. p. 138–150.

  14. Hinckley K, Pausch R, Goble JC, Kassell NF. Passive real-world interface props for neurosurgical visualization. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems: Celebrating Interdependence. ACM; 1994. p. 452–458.

  15. Frohlich B, Plate J, Wind J, Wesche G, Gobel M. Cubic-mouse-based interaction in virtual environments. IEEE Comput Graphics Appl 2000;20(4):12–15.

    Article  Google Scholar 

  16. Gallo L, De Pietro G, Marra I. 3D Interaction with volumetric medical data: experiencing the wiimote. Proceedings of the 1st International Conference on Ambient Media and Systems. ICST (Institute for Computer Sciences, Social-Informatics and Telecommunications Engineering); 2008. p. 14.

  17. Mauser S, Burgert O. Touch-free, gesture-based control of medical devices and software based on the leap motion controller. Stud Health Technol Inform 2014;196:265–270.

    PubMed  Google Scholar 

  18. Gallo L, Placitelli AP, Ciampi M. Controller-free exploration of medical image data: Experiencing the Kinect. 2011 24th International Symposium on Computer-based medical systems (CBMS). IEEE; 2011. p. 1–6.

  19. Ruppert GCS, Reis LO, Amorim PHJ, de Moraes TF, da Silva JVL. Touchless gesture user interface for interactive image visualization in urological surgery. World journal of urology 2012;30(5):687–691.

    Article  PubMed  Google Scholar 

  20. Balakrishnan R, Baudel T, Kurtenbach G, Fitzmaurice G. The Rockin’Mouse: integral 3D manipulation on a plane. Proceedings of the ACM SIGCHI Conference on Human Factors in Computing Systems. ACM; 1997. p. 311–318.

  21. Dang NT, Tavanti M, Rankin I, Cooper M. A comparison of different input devices for a 3D environment. Int J Ind Ergon 2009;39(3):554–563.

    Article  Google Scholar 

  22. Zudilova-Seinstra EV, de Koning PJ, Suinesiaputra A, van Schooten BW, van der Geest RJ, Reiber JH, Sloot PM. Evaluation of 2D and 3D glove input applied to medical image analysis. Int J Hum Comput Stud 2010;68(6):355–369.

    Article  Google Scholar 

  23. Bérard F, Ip J, Benovoy M, El-Shimy D, Blum JR, Cooperstock JR. Did Minority Report get it wrong? Superiority of the mouse over 3D input devices in a 3D placement task. IFIP Conference on Human-Computer Interaction. Springer; 2009. p. 400–414.

  24. Wang G, McGuffin MJ, Bérard F, Cooperstock JR. Pop-up depth views for improving 3D target acquisition. Proceedings of Graphics Interface 2011. Canadian Human-Computer Communications Society; 2011. p. 41–48.

  25. Feiner S, Shamash A. Hybrid user interfaces: breeding virtually bigger interfaces for physically smaller computers. Proceedings of the 4th annual ACM symposium on User interface software and technology. ACM; 1991. p. 9–17.

  26. Fitzmaurice GW, Zhai S, Chignell MH. Virtual reality for palmtop computers. ACM Trans Inf Syst (TOIS) 1993;11(3):197–218.

    Article  Google Scholar 

  27. Angus IG, Sowizral HA. Embedding the 2D interaction metaphor in a real 3D virtual environment. IS&T/SPIE’s Symposium on Electronic Imaging: Science & Technology. International Society for Optics and Photonics; 1995. p. 282–293.

  28. Hachet M, Guitton P, Reuter P. The CAT for efficient 2D and 3D interaction as an alternative to mouse adaptations. Proceedings of the ACM Symposium on Virtual Reality Software and Technology. ACM; 2003. p. 225–112.

  29. Darken RP, Durost R. Mixed-dimension interaction in virtual environments. Proceedings of the ACM Symposium on Virtual Reality Software and Technology. ACM; 2005. p. 38–45.

  30. Wang J, Lindeman RW. Object impersonation: towards effective interaction in tablet-and HMD-based hybrid virtual environments. 2015 IEEE Virtual Reality (VR). IEEE; 2015. p. 111–118.

  31. Wloka M. Interacting with virtual reality. Virtual Prototyping. Springer; 1995. p. 199–212.

  32. Coninx K, Van Reeth F, Flerackers E. A hybrid 2D/3D user interface for immersive object modeling. Proceedings of Computer Graphics International, 1997. IEEE; 1997. p. 47–55.

  33. Rekimoto J. Pick-and-drop: a direct manipulation technique for multiple computer environments. Proceedings of the 10th Annual ACM Symposium on User Interface Software and Technology. ACM; 1997. p. 31–39.

  34. Ullmer B, Ishii H. The metaDESK: models and prototypes for tangible user interfaces. Proceedings of the 10th Annual ACM Symposium on User Interface Software and Technology. ACM; 1997. p. 223–232.

  35. Riva G. Virtual reality for health care: the status of research. Cyberpsychol Behav 2002;5(3):219–225.

    Article  PubMed  Google Scholar 

  36. Arvanitis TN. Virtual reality in medicine. Handbook of Research on Informatics in Healthcare and Biomedicine. IGI Global; 2006. p. 59–67.

  37. Pensieri C, Pennacchini M. Virtual Reality in medicine. Handbook on 3D3C Platforms. Springer; 2016. p. 353–401.

  38. Baumgärtner S, Ebert A, Deller M, Agne S. 2D meets 3D: a human-centered interface for visual data exploration. CHI’07 Extended Abstracts on Human Factors in Computing Systems. ACM; 2007. p. 2273–2278.

  39. Bornik A, Beichel R, Kruijff E, Reitinger B, Schmalstieg D. A hybrid user interface for manipulation of volumetric medical data. IEEE Symposium on 3D User Interfaces, 3DUI 2006. IEEE; 2006. p. 29–36.

  40. Teistler M, Breiman R, Lison T, Bott O, Pretschner D, Aziz A, Nowinski W. Simplifying the exploration of volumetric images: development of a 3D user interface for the radiologist’s workplace. J Digit Imaging 2008;21(1):2–12.

    Article  Google Scholar 

  41. Teistler M, Ampanozi G, Schweitzer W, Flach P, Thali M, Ebert L. Use of a low-cost three-dimensional gaming controller for forensic reconstruction of CT images. J Forensic Radiol Imaging 2016;7:10–13.

    Article  Google Scholar 

  42. Aamir R, Chernoglazov A, Bateman C, Butler A, Butler P, Anderson N, Bell S, Panta R, Healy J, Mohr J, et al. MARS Spectral molecular imaging of lamb tissue: data collection and image analysis. J Instrum 2014;9(02):P02005.

    Article  Google Scholar 

  43. Rajendran K, Walsh M, De Ruiter N, Chernoglazov A, Panta R, Butler A, Butler P, Bell S, Anderson N, Woodfield T, et al. Reducing beam hardening effects and metal artefacts in spectral CT using medipix3RX. J Instrum 2014;9(03):P03015.

    Article  Google Scholar 

  44. Rajendran K, Löbker C, Schon BS, Bateman CJ, Younis RA, de Ruiter NJ, Chernoglazov AI, Ramyar M, Hooper GJ, Butler AP, et al. Quantitative imaging of excised osteoarthritic cartilage using spectral CT. Eur Radiol 2017;27(1):384–392.

    Article  PubMed  Google Scholar 

  45. DCMTK OFFIS DICOM toolkit. http://dicom.offis.de/dcmtk.php.en. Accessed: 2017-01-12.

  46. Qt API Th Qt Company. https://www.qt.io/ui/. Accessed: 2017-01-12.

  47. OpenGL API Khronos Group. https://www.opengl.org/. Accessed: 2017-01-12.

  48. CUDA API NVIDIA Corporation. https://developer.nvidia.com/cuda-toolkit. Accessed: 2017-01-12.

  49. Inteleviewer Intelerad Medical Systems Incorporated. http://www.intelerad.com/en/products/inteleviewer/. Accessed: 2017-01-09.

  50. Lorensen WE, Cline HE. Marching cubes: A high resolution 3D surface construction algorithm. ACM Siggraph Computer Graphics. ACM; 1987. p. 163–169.

  51. Dai Y, Zheng J, Yang Y, Kuai D, Yang X: Volume-Rendering-Based Interactive 3D measurement for quantitative analysis of 3D medical images. Computational and mathematical methods in medicine, vol 2013, 2013.

  52. Preim B, Tietjen C, Spindler W, Peitgen HO. Integration of measurement tools in medical 3D visualizations. Proceedings of the Conference on Visualization’02. IEEE Computer Society; 2002. p. 21–28.

  53. Human Ethics Committee university of canterbury. http://www.canterbury.ac.nz/humanethics/. Accessed: 2016-06-01.

  54. Brant WE, Helms CA: Fundamentals of diagnostic radiology. Lippincott Williams & Wilkins, 2012.

  55. Drebin RA, Carpenter L, Hanrahan P. Volume rendering. ACM Siggraph Computer Graphics, vol 22 no 4. ACM; 1988. p. 65–74.

  56. Cox GM, Cochran W: Experimental designs. JSTOR, 1953.

  57. Brooke J, et al. SUS-a quick and dirty usability scale. Usability Evaluation in Industry 1996;189(194):4–7.

    Google Scholar 

  58. Hart SG, Staveland LE. Development of nasa-tlx (task load index): results of empirical and theoretical research. Adv Psychol 1988;52:139–183.

    Article  Google Scholar 

  59. Maxwell SE, Delaney HD. Designing experiments and analyzing data: a model comparison perspective. Psychology Press, vol 1, 2004.

  60. Hartung J, Knapp G, Sinha BK: Statistical meta-analysis with applications. Wiley, vol 738, 2011.

  61. Zudilova-Seinstra E, van Schooten B, Suinesiaputra A, van der Geest R, van Dijk B, Reiber J, Sloot P. Exploring individual user differences in the 2D/3D interaction with medical image data. Virtual Reality 2010;14(2):105–118.

    Article  Google Scholar 

  62. Bangor A, Kortum P, Miller J. Determining what individual SUS scores mean: adding an adjective rating scale. J Usability Stud 2009;4(3):114–123.

    Google Scholar 

  63. Patterson RE: Human factors of stereoscopic 3D displays. 1270 Springer, 2015.

  64. Frees S, Kessler GD. Precise and rapid interaction through scaled manipulation in immersive virtual environments. IEEE Proceedings of Virtual Reality, 2005. VR 2015. IEEE; 2005. p. 99–106.

  65. MARS Bioimaging Ltd. http://www.marsbioimaging.com/mars/. Accessed: 2017-01-09.

Download references

Acknowledgements

We would like to thank all the radiology residents from the Christchurch Hospital and fourth year medical students from the University of Otago who participated in the evaluation of our system. This work was funded by the MARS Bioimaging [65].

Author information

Authors and Affiliations

  1. University of Canterbury, Christchurch, New Zealand

    Veera Bhadra Harish Mandalika, Alexander I. Chernoglazov, Christoph Bartneck, Niels de Ruiter, Anthony P. H. Butler & Philip H. Butler

  2. University of South Australia, Adelaide, Australia

    Mark Billinghurst

  3. Division of Health Sciences, University of Otago, Christchurch, New Zealand

    Michael A. Hurrell, Niels de Ruiter & Anthony P. H. Butler

  4. MARS Bioimaging Ltd., Christchurch, New Zealand

    Veera Bhadra Harish Mandalika, Alexander I. Chernoglazov, Niels de Ruiter, Anthony P. H. Butler & Philip H. Butler

  5. HIT Lab NZ, Christchurch, New Zealand

    Veera Bhadra Harish Mandalika, Christoph Bartneck & Niels de Ruiter

Authors
  1. Veera Bhadra Harish Mandalika
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Alexander I. Chernoglazov
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Mark Billinghurst
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Christoph Bartneck
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Michael A. Hurrell
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Niels de Ruiter
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Anthony P. H. Butler
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Philip H. Butler
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Veera Bhadra Harish Mandalika.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Mandalika, V., Chernoglazov, A., Billinghurst, M. et al. A Hybrid 2D/3D User Interface for Radiological Diagnosis. J Digit Imaging 31, 56–73 (2018). https://doi.org/10.1007/s10278-017-0002-6

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10278-017-0002-6

Keywords

Search

Navigation