Journal of Digital Imaging

, Volume 31, Issue 1, pp 56–73 | Cite as

A Hybrid 2D/3D User Interface for Radiological Diagnosis

  • Veera Bhadra Harish MandalikaEmail author
  • Alexander I. Chernoglazov
  • Mark Billinghurst
  • Christoph Bartneck
  • Michael A. Hurrell
  • Niels de Ruiter
  • Anthony P. H. Butler
  • Philip H. Butler


This paper presents a novel 2D/3D desktop virtual reality hybrid user interface for radiology that focuses on improving 3D manipulation required in some diagnostic tasks. An evaluation of our system revealed that our hybrid interface is more efficient for novice users and more accurate for both novice and experienced users when compared to traditional 2D only interfaces. This is a significant finding because it indicates, as the techniques mature, that hybrid interfaces can provide significant benefit to image evaluation. Our hybrid system combines a zSpace stereoscopic display with 2D displays, and mouse and keyboard input. It allows the use of 2D and 3D components interchangeably, or simultaneously. The system was evaluated against a 2D only interface with a user study that involved performing a scoliosis diagnosis task. There were two user groups: medical students and radiology residents. We found improvements in completion time for medical students, and in accuracy for both groups. In particular, the accuracy of medical students improved to match that of the residents.


3D input Hybrid user interface Diagnostic radiology Medical visualization User interface 



We would like to thank all the radiology residents from the Christchurch Hospital and fourth year medical students from the University of Otago who participated in the evaluation of our system. This work was funded by the MARS Bioimaging [65].


  1. 1.
    Chen M, Mountford SJ, Sellen A. A study in interactive 3-D rotation using 2-D control devices. ACM SIGGRAPH Computer Graphics 1988;22(4):121–129.CrossRefGoogle Scholar
  2. 2.
    Hinckley K, Tullio J, Pausch R, Proffitt D, Kassell N. Usability analysis of 3D rotation techniques. Proceedings of the 10th Annual ACM Symposium on User Interface Software and Technology. ACM; 1997. p. 1–10.Google Scholar
  3. 3.
    Bowman D, Kruijff E, LaViola Jr JJ, Poupyrev IP: 3D User interfaces: theory and practice, CourseSmart eTextbook. Addison-Wesley, 2004.Google Scholar
  4. 4.
    Graves MJ, Black RT, Lomas DJ. Constrained surface controllers for three-dimensional image data reformatting. Radiology 2009;252(1):218–224.CrossRefPubMedGoogle Scholar
  5. 5.
    zSpace zSpace, inc. Accessed: 2017-01-09.
  6. 6.
    Emerson T, Prothero JD, Weghorst SJ: Medicine and virtual reality: a guide to the literature (medVR). Human Interface Technology Laboratory, 1994.Google Scholar
  7. 7.
    Ayache N. Medical computer vision, virtual reality and robotics. Image and Vision Computing 1995;13(4): 295–313.CrossRefGoogle Scholar
  8. 8.
    Székely G, Satava RM. Virtual reality in medicine. BMJ: British Medical Journal 1999;319(7220):1305.CrossRefPubMedPubMedCentralGoogle Scholar
  9. 9.
    Gallo L, Minutolo A, De Pietro G. A user interface for VR-ready 3D medical imaging by off-the-shelf input devices. Computers in Biology and Medicine 2010;40(3):350–358.CrossRefPubMedGoogle Scholar
  10. 10.
    Hand C. A survey of 3D interaction techniques. Computer graphics forum, vol 16 no 5. Wiley; 1997. p. 269–281.Google Scholar
  11. 11.
    Shoemake K. Arcball: a user interface for specifying three-dimensional orientation using a mouse. Graphics Interface; 1992. p. 151–156.Google Scholar
  12. 12.
    Henriksen K, Sporring J, Hornbæk K. Virtual trackballs revisited. IEEE Trans Visual Comput Graphics 2004;10(2):206–216.CrossRefGoogle Scholar
  13. 13.
    Bade R, Ritter F, Preim B. Usability comparison of mouse-based interaction techniques for predictable 3D rotation. International Symposium on Smart Graphics. Springer; 2005. p. 138–150.Google Scholar
  14. 14.
    Hinckley K, Pausch R, Goble JC, Kassell NF. Passive real-world interface props for neurosurgical visualization. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems: Celebrating Interdependence. ACM; 1994. p. 452–458.Google Scholar
  15. 15.
    Frohlich B, Plate J, Wind J, Wesche G, Gobel M. Cubic-mouse-based interaction in virtual environments. IEEE Comput Graphics Appl 2000;20(4):12–15.CrossRefGoogle Scholar
  16. 16.
    Gallo L, De Pietro G, Marra I. 3D Interaction with volumetric medical data: experiencing the wiimote. Proceedings of the 1st International Conference on Ambient Media and Systems. ICST (Institute for Computer Sciences, Social-Informatics and Telecommunications Engineering); 2008. p. 14.Google Scholar
  17. 17.
    Mauser S, Burgert O. Touch-free, gesture-based control of medical devices and software based on the leap motion controller. Stud Health Technol Inform 2014;196:265–270.PubMedGoogle Scholar
  18. 18.
    Gallo L, Placitelli AP, Ciampi M. Controller-free exploration of medical image data: Experiencing the Kinect. 2011 24th International Symposium on Computer-based medical systems (CBMS). IEEE; 2011. p. 1–6.Google Scholar
  19. 19.
    Ruppert GCS, Reis LO, Amorim PHJ, de Moraes TF, da Silva JVL. Touchless gesture user interface for interactive image visualization in urological surgery. World journal of urology 2012;30(5):687–691.CrossRefPubMedGoogle Scholar
  20. 20.
    Balakrishnan R, Baudel T, Kurtenbach G, Fitzmaurice G. The Rockin’Mouse: integral 3D manipulation on a plane. Proceedings of the ACM SIGCHI Conference on Human Factors in Computing Systems. ACM; 1997. p. 311–318.Google Scholar
  21. 21.
    Dang NT, Tavanti M, Rankin I, Cooper M. A comparison of different input devices for a 3D environment. Int J Ind Ergon 2009;39(3):554–563.CrossRefGoogle Scholar
  22. 22.
    Zudilova-Seinstra EV, de Koning PJ, Suinesiaputra A, van Schooten BW, van der Geest RJ, Reiber JH, Sloot PM. Evaluation of 2D and 3D glove input applied to medical image analysis. Int J Hum Comput Stud 2010;68(6):355–369.CrossRefGoogle Scholar
  23. 23.
    Bérard F, Ip J, Benovoy M, El-Shimy D, Blum JR, Cooperstock JR. Did Minority Report get it wrong? Superiority of the mouse over 3D input devices in a 3D placement task. IFIP Conference on Human-Computer Interaction. Springer; 2009. p. 400–414.Google Scholar
  24. 24.
    Wang G, McGuffin MJ, Bérard F, Cooperstock JR. Pop-up depth views for improving 3D target acquisition. Proceedings of Graphics Interface 2011. Canadian Human-Computer Communications Society; 2011. p. 41–48.Google Scholar
  25. 25.
    Feiner S, Shamash A. Hybrid user interfaces: breeding virtually bigger interfaces for physically smaller computers. Proceedings of the 4th annual ACM symposium on User interface software and technology. ACM; 1991. p. 9–17.Google Scholar
  26. 26.
    Fitzmaurice GW, Zhai S, Chignell MH. Virtual reality for palmtop computers. ACM Trans Inf Syst (TOIS) 1993;11(3):197–218.CrossRefGoogle Scholar
  27. 27.
    Angus IG, Sowizral HA. Embedding the 2D interaction metaphor in a real 3D virtual environment. IS&T/SPIE’s Symposium on Electronic Imaging: Science & Technology. International Society for Optics and Photonics; 1995. p. 282–293.Google Scholar
  28. 28.
    Hachet M, Guitton P, Reuter P. The CAT for efficient 2D and 3D interaction as an alternative to mouse adaptations. Proceedings of the ACM Symposium on Virtual Reality Software and Technology. ACM; 2003. p. 225–112.Google Scholar
  29. 29.
    Darken RP, Durost R. Mixed-dimension interaction in virtual environments. Proceedings of the ACM Symposium on Virtual Reality Software and Technology. ACM; 2005. p. 38–45.Google Scholar
  30. 30.
    Wang J, Lindeman RW. Object impersonation: towards effective interaction in tablet-and HMD-based hybrid virtual environments. 2015 IEEE Virtual Reality (VR). IEEE; 2015. p. 111–118.Google Scholar
  31. 31.
    Wloka M. Interacting with virtual reality. Virtual Prototyping. Springer; 1995. p. 199–212.Google Scholar
  32. 32.
    Coninx K, Van Reeth F, Flerackers E. A hybrid 2D/3D user interface for immersive object modeling. Proceedings of Computer Graphics International, 1997. IEEE; 1997. p. 47–55.Google Scholar
  33. 33.
    Rekimoto J. Pick-and-drop: a direct manipulation technique for multiple computer environments. Proceedings of the 10th Annual ACM Symposium on User Interface Software and Technology. ACM; 1997. p. 31–39.Google Scholar
  34. 34.
    Ullmer B, Ishii H. The metaDESK: models and prototypes for tangible user interfaces. Proceedings of the 10th Annual ACM Symposium on User Interface Software and Technology. ACM; 1997. p. 223–232.Google Scholar
  35. 35.
    Riva G. Virtual reality for health care: the status of research. Cyberpsychol Behav 2002;5(3):219–225.CrossRefPubMedGoogle Scholar
  36. 36.
    Arvanitis TN. Virtual reality in medicine. Handbook of Research on Informatics in Healthcare and Biomedicine. IGI Global; 2006. p. 59–67.Google Scholar
  37. 37.
    Pensieri C, Pennacchini M. Virtual Reality in medicine. Handbook on 3D3C Platforms. Springer; 2016. p. 353–401.Google Scholar
  38. 38.
    Baumgärtner S, Ebert A, Deller M, Agne S. 2D meets 3D: a human-centered interface for visual data exploration. CHI’07 Extended Abstracts on Human Factors in Computing Systems. ACM; 2007. p. 2273–2278.Google Scholar
  39. 39.
    Bornik A, Beichel R, Kruijff E, Reitinger B, Schmalstieg D. A hybrid user interface for manipulation of volumetric medical data. IEEE Symposium on 3D User Interfaces, 3DUI 2006. IEEE; 2006. p. 29–36.Google Scholar
  40. 40.
    Teistler M, Breiman R, Lison T, Bott O, Pretschner D, Aziz A, Nowinski W. Simplifying the exploration of volumetric images: development of a 3D user interface for the radiologist’s workplace. J Digit Imaging 2008;21(1):2–12.CrossRefGoogle Scholar
  41. 41.
    Teistler M, Ampanozi G, Schweitzer W, Flach P, Thali M, Ebert L. Use of a low-cost three-dimensional gaming controller for forensic reconstruction of CT images. J Forensic Radiol Imaging 2016;7:10–13.CrossRefGoogle Scholar
  42. 42.
    Aamir R, Chernoglazov A, Bateman C, Butler A, Butler P, Anderson N, Bell S, Panta R, Healy J, Mohr J, et al. MARS Spectral molecular imaging of lamb tissue: data collection and image analysis. J Instrum 2014;9(02):P02005.CrossRefGoogle Scholar
  43. 43.
    Rajendran K, Walsh M, De Ruiter N, Chernoglazov A, Panta R, Butler A, Butler P, Bell S, Anderson N, Woodfield T, et al. Reducing beam hardening effects and metal artefacts in spectral CT using medipix3RX. J Instrum 2014;9(03):P03015.CrossRefGoogle Scholar
  44. 44.
    Rajendran K, Löbker C, Schon BS, Bateman CJ, Younis RA, de Ruiter NJ, Chernoglazov AI, Ramyar M, Hooper GJ, Butler AP, et al. Quantitative imaging of excised osteoarthritic cartilage using spectral CT. Eur Radiol 2017;27(1):384–392.CrossRefPubMedGoogle Scholar
  45. 45.
    DCMTK OFFIS DICOM toolkit. Accessed: 2017-01-12.
  46. 46.
    Qt API Th Qt Company. Accessed: 2017-01-12.
  47. 47.
    OpenGL API Khronos Group. Accessed: 2017-01-12.
  48. 48.
    CUDA API NVIDIA Corporation. Accessed: 2017-01-12.
  49. 49.
    Inteleviewer Intelerad Medical Systems Incorporated. Accessed: 2017-01-09.
  50. 50.
    Lorensen WE, Cline HE. Marching cubes: A high resolution 3D surface construction algorithm. ACM Siggraph Computer Graphics. ACM; 1987. p. 163–169.Google Scholar
  51. 51.
    Dai Y, Zheng J, Yang Y, Kuai D, Yang X: Volume-Rendering-Based Interactive 3D measurement for quantitative analysis of 3D medical images. Computational and mathematical methods in medicine, vol 2013, 2013.Google Scholar
  52. 52.
    Preim B, Tietjen C, Spindler W, Peitgen HO. Integration of measurement tools in medical 3D visualizations. Proceedings of the Conference on Visualization’02. IEEE Computer Society; 2002. p. 21–28.Google Scholar
  53. 53.
    Human Ethics Committee university of canterbury. Accessed: 2016-06-01.
  54. 54.
    Brant WE, Helms CA: Fundamentals of diagnostic radiology. Lippincott Williams & Wilkins, 2012.Google Scholar
  55. 55.
    Drebin RA, Carpenter L, Hanrahan P. Volume rendering. ACM Siggraph Computer Graphics, vol 22 no 4. ACM; 1988. p. 65–74.Google Scholar
  56. 56.
    Cox GM, Cochran W: Experimental designs. JSTOR, 1953.Google Scholar
  57. 57.
    Brooke J, et al. SUS-a quick and dirty usability scale. Usability Evaluation in Industry 1996;189(194):4–7.Google Scholar
  58. 58.
    Hart SG, Staveland LE. Development of nasa-tlx (task load index): results of empirical and theoretical research. Adv Psychol 1988;52:139–183.CrossRefGoogle Scholar
  59. 59.
    Maxwell SE, Delaney HD. Designing experiments and analyzing data: a model comparison perspective. Psychology Press, vol 1, 2004.Google Scholar
  60. 60.
    Hartung J, Knapp G, Sinha BK: Statistical meta-analysis with applications. Wiley, vol 738, 2011.Google Scholar
  61. 61.
    Zudilova-Seinstra E, van Schooten B, Suinesiaputra A, van der Geest R, van Dijk B, Reiber J, Sloot P. Exploring individual user differences in the 2D/3D interaction with medical image data. Virtual Reality 2010;14(2):105–118.CrossRefGoogle Scholar
  62. 62.
    Bangor A, Kortum P, Miller J. Determining what individual SUS scores mean: adding an adjective rating scale. J Usability Stud 2009;4(3):114–123.Google Scholar
  63. 63.
    Patterson RE: Human factors of stereoscopic 3D displays. 1270 Springer, 2015.Google Scholar
  64. 64.
    Frees S, Kessler GD. Precise and rapid interaction through scaled manipulation in immersive virtual environments. IEEE Proceedings of Virtual Reality, 2005. VR 2015. IEEE; 2005. p. 99–106.Google Scholar
  65. 65.
    MARS Bioimaging Ltd. Accessed: 2017-01-09.

Copyright information

© Society for Imaging Informatics in Medicine 2017

Authors and Affiliations

  • Veera Bhadra Harish Mandalika
    • 1
    • 4
    • 5
    Email author
  • Alexander I. Chernoglazov
    • 1
    • 4
  • Mark Billinghurst
    • 2
  • Christoph Bartneck
    • 1
    • 5
  • Michael A. Hurrell
    • 3
  • Niels de Ruiter
    • 1
    • 3
    • 4
    • 5
  • Anthony P. H. Butler
    • 1
    • 3
    • 4
  • Philip H. Butler
    • 1
    • 4
  1. 1.University of CanterburyChristchurchNew Zealand
  2. 2.University of South AustraliaAdelaideAustralia
  3. 3.Division of Health SciencesUniversity of OtagoChristchurchNew Zealand
  4. 4.MARS Bioimaging Ltd.ChristchurchNew Zealand
  5. 5.HIT Lab NZChristchurchNew Zealand

Personalised recommendations