Annals of Biomedical Engineering

, Volume 46, Issue 12, pp 2177–2188 | Cite as

Simulating Developmental Cardiac Morphology in Virtual Reality Using a Deformable Image Registration Approach

  • Arash Abiri
  • Yichen Ding
  • Parinaz Abiri
  • René R. Sevag Packard
  • Vijay Vedula
  • Alison Marsden
  • C.-C. Jay Kuo
  • Tzung K. Hsiai


While virtual reality (VR) has potential in enhancing cardiovascular diagnosis and treatment, prerequisite labor-intensive image segmentation remains an obstacle for seamlessly simulating 4-dimensional (4-D, 3-D + time) imaging data in an immersive, physiological VR environment. We applied deformable image registration (DIR) in conjunction with 3-D reconstruction and VR implementation to recapitulate developmental cardiac contractile function from light-sheet fluorescence microscopy (LSFM). This method addressed inconsistencies that would arise from independent segmentations of time-dependent data, thereby enabling the creation of a VR environment that fluently simulates cardiac morphological changes. By analyzing myocardial deformation at high spatiotemporal resolution, we interfaced quantitative computations with 4-D VR. We demonstrated that our LSFM-captured images, followed by DIR, yielded average dice similarity coefficients of 0.92 ± 0.05 (n = 510) and 0.93 ± 0.06 (n = 240) when compared to ground truth images obtained from Otsu thresholding and manual segmentation, respectively. The resulting VR environment simulates a wide-angle zoomed-in view of motion in live embryonic zebrafish hearts, in which the cardiac chambers are undergoing structural deformation throughout the cardiac cycle. Thus, this technique allows for an interactive micro-scale VR visualization of developmental cardiac morphology to enable high resolution simulation for both basic and clinical science.


Medical simulation Light-sheet imaging Cardiology Image registration Dynamic imaging Surgical simulation 



This work was supported by the National Institutes of Health (5R01HL083015-10, 1R01HL118650, 1R01HL129727, 7R01HL111437) and the American Heart Association (Career Development Award 18CDA34110338, Scientist Development Grant 16SDG30910007).

Conflict of interest

The authors declare no competing financial interests.

Supplementary material (7 mb)
4-D VR Simulation of contracting embryonic zebrafish heart. Video of the 4-D VR simulation of a contracting embryonic zebrafish heart throughout an entire cardiac cycle. The VR scene was generated and visualized using the Unity engine. Supplementary material 1 (MOV 7180 kb)


  1. 1.
    Abiri, A., A. Tao, M. LaRocca, X. Guan, S. Askari, J. Bisley, E. Dutson, and W. Grundfest. Visual–perceptual mismatch in robotic surgery. Surg. Endosc. 31:3271–3278, 2016.CrossRefGoogle Scholar
  2. 2.
    Araki, T., N. Ikeda, N. Dey, S. Chakraborty, L. Saba, D. Kumar, E. Godia, X. Jiang, A. Gupta, P. Radeva, J. Laird, A. Nicolaides, and J. Suri. A comparative approach of four different image registration techniques for quantitative assessment of coronary artery calcium lesions using intravascular ultrasound. Comput. Methods Programs Biomed. 118:158–172, 2015.CrossRefGoogle Scholar
  3. 3.
    Brock, K. K. Results of a multi-institution deformable registration accuracy study (MIDRAS). Int. J. Radiat. Oncol. Biol. Phys. 76:583–596, 2010.CrossRefGoogle Scholar
  4. 4.
    Brock, K. K., L. A. Dawson, M. B. Sharpe, D. J. Moseley, and D. A. Jaffray. Feasibility of a novel deformable image registration technique to facilitate classification, targeting, and monitoring of tumor and normal tissue. Int. J. Radiat. Oncol. Biol. Phys. 64:1245–1254, 2006.CrossRefGoogle Scholar
  5. 5.
    Brock, K., M. Sharpe, L. Dawson, S. Kim, and D. Jaffray. Accuracy of finite element model-based multi-organ deformable image registration. Med. Phys. 32:1647–1659, 2005.CrossRefGoogle Scholar
  6. 6.
    Bronstein, A. M., M. M. Bronstein, R. Kimmel, M. Mahmoudi, and G. Sapiro. A Gromov-Hausdorff framework with diffusion geometry for topologically-robust non-rigid shape matching. Int. J. Comp. Vis. 89:266–286, 2010.CrossRefGoogle Scholar
  7. 7.
    Brown, L. A survey of image registration techniques. ACM Comput. Surv. 24:325–376, 1992.CrossRefGoogle Scholar
  8. 8.
    Chan, S., F. Conti, K. Salisbury, and N. Blevins. Virtual reality simulation in neurosurgery: technologies and evolution. Neurosurgery 72:A154–A164, 2013.CrossRefGoogle Scholar
  9. 9.
    Cuchet, E., J. Knoplioch, D. Dormont, and C. Marsault. Registration in neurosurgery and neuroradiotherapy applications. J. Image Guid. Surg. 1:198–207, 1995.CrossRefGoogle Scholar
  10. 10.
    Dallal, G. An analytic approximation to the distribution of Lilliefors’s test statistic for normality. Am. Stat. 40:294–296, 1986.Google Scholar
  11. 11.
    Ding, Y., A. Abiri, P. Abiri, S. Li, C.-C. Chang, K. I. Baek, J. J. Hsu, E. Sideris, Y. Li, J. Lee, T. Segura, T. P. Nguyen, A. Bui, and R. R. Sevag. Integrating light-sheet imaging with virtual reality to recapitulate developmental cardiac mechanics. JCI Insight 2:22, 2017.CrossRefGoogle Scholar
  12. 12.
    Ding, Y., J. Lee, J. J. Hsu, C. C. Chang, K. I. Baek, S. Ranjbarvaziri, R. Ardehali, R. R. S. Packard, and T. K. Hsiai. Light-sheet imaging to elucidate cardiovascular injury and repair. Curr. Cardiol. Rep. 20:35, 2018.CrossRefGoogle Scholar
  13. 13.
    Ding, Y., H. Xie, T. Peng, Y. Lu, D. Jin, J. Teng, Q. Ren, and P. Xi. Laser oblique scanning optical microscopy (LOSOM) for phase relief imaging. Opt. Express 20:14100–14108, 2012.CrossRefGoogle Scholar
  14. 14.
    Ding, Y., M. Zhang, J. Lang, J. Leng, Q. Ren, J. Yang, and C. Li. In vivo study of endometriosis in mice by photoacoustic microscopy. J. Biophotonics 8:94–101, 2015.CrossRefGoogle Scholar
  15. 15.
    Fei, P., J. Lee, R. Sevag Packard, K.-I. Sereti, H. Xu, J. Ma, Y. Ding, H. Kang, H. Chen, K. Sung, R. Kulkarni, R. Ardehali, J. Kuo, X. Xu, C.-M. Ho, and T. Hsiai. Cardiac light-sheet fluorescent microscopy for multi-scale and rapid imaging of architecture and function. Sci. Rep. 6:22489, 2016.CrossRefGoogle Scholar
  16. 16.
    Fei, P., J. Nie, J. Lee, Y. Ding, S. Li, Z. Yu, H. Zhang, M. Hagiwara, T. Yu, T. Segura, C.-M. Ho, D. Zhu, and T. K. Hsiai. Sub-voxel light-sheet microscopy for high-resolution, high-throughput volumetric imaging of large biomedical specimens. bioRxiv, 2018.
  17. 17.
    Freeborough, P. A., R. P. Woods, and N. C. Fox. Accurate registration of serial 3D MR brain images and its application to visualizing change in neurodegenerative disorders. J. Comput. Assist. Tomogr. 20:1012–1022, 1996.CrossRefGoogle Scholar
  18. 18.
    Fuchs, E., J. Jaffe, R. Long, and F. Azam. Thin laser light sheet microscope for microbial oceanography. Opt. Express 10:145–154, 2002.CrossRefGoogle Scholar
  19. 19.
    Gallagher, A. G., and C. U. Cates. Virtual reality training for the operating room and cardiac catheterisation laboratory. Lancet 364:1538–1540, 2004.CrossRefGoogle Scholar
  20. 20.
    Greenbaum, P. The lawnmower man. Film Video 9:58–62, 1992.Google Scholar
  21. 21.
    Gu, X., H. Pan, Y. Liang, R. Castillo, D. Yang, D. Choi, E. Castillo, A. Majumdar, T. Guerrero, and S. Jiang. Implementation and evaluation of various demons deformable image registration algorithms on a GPU. Phys. Med. Biol. 55:207–219, 2010.CrossRefGoogle Scholar
  22. 22.
    Guiraudon, G. M., D. L. Jones, D. Bainbridge, and T. M. Peters. Mitral valve implantation using off-pump closed beating intracardiac surgery: a feasibility study. Interact. Cardiovasc. Thorac. Surg. 6:603–607, 2007.CrossRefGoogle Scholar
  23. 23.
    Handels, H., and J. Ehrhardt. Medical image computing for computer-supported diagnostics and therapy. Methods Inf. Med. 48:11–17, 2009.CrossRefGoogle Scholar
  24. 24.
    Hill, D., P. Batchelor, M. Holden, and D. Hawkes. Medical image registration. Phys. Med. Biol. 46:R1, 2001.CrossRefGoogle Scholar
  25. 25.
    Holden, M., D. Hill, E. Denton, J. Jarosz, T. Cox, T. Rohlfing, J. Goodey, and D. Hawkes. Voxel similarity measures for 3-D serial MR brain image registration. IEEE Trans. Med. Imaging 19:94–102, 2000.CrossRefGoogle Scholar
  26. 26.
    Huisken, J., J. Swoger, F. Del Bene, J. Wittbrodt, and E. H. K. Stelzer. Optical sectioning deep inside live embryos by selective plane illumination microscopy. Science. 305:1007–1009, 2004.CrossRefGoogle Scholar
  27. 27.
    Hwang, S. S., H.-D. Kim, T. Y. Jang, J. Yoo, S. Kim, K. Paeng, and S. D. Kim. Image-based object reconstruction using run-length representation. Signal Proc. Image Commun. 51:1–12, 2017.CrossRefGoogle Scholar
  28. 28.
    Kanade, T., and P. J. Narayanan. Virtualized reality: perspectives on 4D digitization of dynamic events. IEEE Comp. Graph. Appl. 27:32–40, 2007.CrossRefGoogle Scholar
  29. 29.
    Kardell, M., M. Magnusson, M. Sandborg, G. Alm Carlsson, J. Jeuthe, and A. Malusek. Automatic segmentation of pelvis for brachytherapy of prostate. Radiat. Prot. Dosimetr. 169:398–404, 2016.CrossRefGoogle Scholar
  30. 30.
    Keller, P. J., A. D. Schmidt, J. Wittbrodt, and E. H. K. Stelzer. Reconstruction of zebrafish early embryonic development by scanned light sheet microscopy. Science. 322:1065–1069, 2008.CrossRefGoogle Scholar
  31. 31.
    King, F., J. Jayender, S. Bhagavatula, P. Shyn, S. Pieper, T. Kapur, A. Lasso, and G. Fichtinger. An Immersive Virtual Reality Environment for Diagnostic Imaging. J. Med. Robot. Res. 1:1640003-1–9, 2016.CrossRefGoogle Scholar
  32. 32.
    Lee, J., P. Fei, R. Sevag Packard, H. Kang, H. Xu, K. I. Baek, N. Jen, J. Chen, H. Yen, J. Kuo, N. Chi, C.-M. Ho, and T. Hsiai. 4-Dimensional light-sheet microscopy to elucidate shear stress modulation of cardiac trabeculation. J. Clin. Invest. 126:1679–1690, 2016.CrossRefGoogle Scholar
  33. 33.
    Lemole, G., P. Banerjee, C. Luciano, S. Neckrysh, and F. Charbel. Virtual reality in neurosurgical education. Neurosurgery 61:142–149, 2007.CrossRefGoogle Scholar
  34. 34.
    Li, G., D. Citrin, K. Camphausen, B. Mueller, C. Burman, B. Mychalczak, R. W. Miller, and Y. Song. Advances in 4D medical imaging and 4D radiation therapy. Technol. Cancer Res. Treat. 7:67–81, 2008.CrossRefGoogle Scholar
  35. 35.
    Li, Z., M. Wu, W. Zhou, and J. Yu. 4D human body correspondences from panoramic depth maps. 2018.Google Scholar
  36. 36.
    Lipman, Y., and T. Funkhouser. Möbius voting for surface correspondence. ACM T. Graph. 28:72, 2009.CrossRefGoogle Scholar
  37. 37.
    Litman, R., and A. M. Bronstein. Learning spectral descriptors for deformable shape correspondence. IEEE T. Patt. Anal. Mach. Intell. 36:171–180, 2014.CrossRefGoogle Scholar
  38. 38.
    Lorenzo-Valdés, M., G. I. Sanchez-Ortiz, R. Mohiaddin, and D. Rueckert. Atlas-based segmentation and tracking of 3D cardiac MR images using non-rigid registration BT—medical image computing and computer-assisted intervention—MICCAI 2002. Berlin: Springer, 2002.Google Scholar
  39. 39.
    Lu, W., P. J. Parikh, I. M. El Naqa, M. M. Nystrom, J. P. Hubenschmidt, S. H. Wahab, S. Mutic, A. K. Singh, G. E. Christensen, J. D. Bradley, and D. A. Low. Quantitation of the reconstruction quality of a four-dimensional computed tomography process for lung cancer patients. Med. Phys. 32:890–901, 2005.CrossRefGoogle Scholar
  40. 40.
    Lu, Y., K. Yang, K. Zhou, B. Pang, G. Wang, Y. Ding, Q. Zhang, H. Han, J. Tian, C. Li, and Q. Ren. An integrated Quad-modality molecular imaging system for small animals. J. Nucl. Med. 55:1375–1379, 2014.CrossRefGoogle Scholar
  41. 41.
    Mcinerney, T., and D. Terzopoulos. A dynamic finite element surface model for segmentation and tracking in multidimensional medical images with application to cardiac 4D image analysis. Comput. Med. Imaging Graph. 19:69–83, 1995.CrossRefGoogle Scholar
  42. 42.
    Metz, C. T., S. Klein, M. Schaap, T. van Walsum, and W. J. Niessen. Nonrigid registration of dynamic medical imaging data using nD + t B-splines and a groupwise optimization approach. Med. Image Anal. 15:238–249, 2011.CrossRefGoogle Scholar
  43. 43.
    Mitchell, S. C., J. G. Bosch, B. P. F. Lelieveldt, R. J. Van der Geest, J. H. C. Reiber, and M. Sonka. 3-D active appearance models: segmentation of cardiac MR and ultrasound images. IEEE T. Med. Imaging 21:1167–1178, 2002.CrossRefGoogle Scholar
  44. 44.
    Montagnat, J., and H. Delingette. 4D deformable models with temporal constraints: application to 4D cardiac image segmentation. Med. Image Anal. 9:87–100, 2005.CrossRefGoogle Scholar
  45. 45.
    Otsu, N. A threshold selection method from gray-level histograms. IEEE Trans. Syst. Man. Cybern. 9:62–66, 1979.CrossRefGoogle Scholar
  46. 46.
    Packard, R. R. S., K. I. Baek, T. Beebe, N. Jen, Y. DIng, F. Shi, P. Fei, B. J. Kang, P. H. Chen, J. Gau, M. Chen, J. Y. Tang, Y. H. Shih, Y. DIng, D. Li, X. Xu, and T. K. Hsiai. Automated segmentation of light-sheet fluorescent imaging to characterize experimental doxorubicin-induced cardiac injury and repair. Sci. Rep. 7:1–11, 2017.CrossRefGoogle Scholar
  47. 47.
    Peng, H., Z. Ruan, F. Long, J. H. Simpson, and E. W. Myers. V3D enables real-time 3D visualization and quantitative analysis of large-scale biological image data sets. Nat. Biotechnol. 28:348–353, 2010.CrossRefGoogle Scholar
  48. 48.
    Peng, H., J. Tang, H. Xiao, A. Bria, J. Zhou, V. Butler, Z. Zhou, P. T. Gonzalez-Bellido, S. W. Oh, J. Chen, A. Mitra, R. W. Tsien, H. Zeng, G. A. Ascoli, G. Iannello, M. Hawrylycz, E. Myers, and F. Long. Virtual finger boosts three-dimensional imaging and microsurgery as well as terabyte volume image visualization and analysis. Nat. Commun. 5:1–13, 2014.Google Scholar
  49. 49.
    Planchon, T., L. Gao, D. Milkie, M. Davidson, J. Galbraith, C. Galbraith, and E. Betzig. Rapid three-dimensional isotropic imaging of living cells using Bessel beam plane illumination. Nat. Methods 8:417–423, 2011.CrossRefGoogle Scholar
  50. 50.
    Pottmann, H., J. Wallner, Q.-X. Huang, and Y.-L. Yang. Integral invariants for robust geometry processing. Comp. Aided Geom. Des. 26:37–60, 2009.CrossRefGoogle Scholar
  51. 51.
    Power, R. M., and J. Huisken. A guide to light-sheet fluorescence microscopy for multiscale imaging. Nat. Meth. 14:360–373, 2017.CrossRefGoogle Scholar
  52. 52.
    Reznick, R., and H. MacRae. Teaching surgical skills—changes in the wind. N. Engl. J. Med. 355:2664–2669, 2006.CrossRefGoogle Scholar
  53. 53.
    Riva, G. Applications of virtual environments in medicine. Methods Inf. Med. 42:524–534, 2003.CrossRefGoogle Scholar
  54. 54.
    Shen, J.-K., B. Matuszewski, L.-K. Shark, A. Skalski, T. Zielinski, and C. Moore. Deformable image registration—a critical evaluation: demons, B-Spline FFD and spring mass system. 2008.Google Scholar
  55. 55.
    Smith, L. N., A. R. Farooq, M. L. Smith, I. E. Ivanov, and A. Orlando. Realistic and interactive high-resolution 4D environments for real-time surgeon and patient interaction. Int. J. Med. Robot. 13:e1761, 2017.CrossRefGoogle Scholar
  56. 56.
    Thirion, J. Image matching as a diffusion process: an analogy with Maxwell’s demons. Med. Image Anal. 2:243–260, 1998.CrossRefGoogle Scholar
  57. 57.
    Turinsky, A. L., and C. W. Sensen. On the way to building an integrated computational environment for the study of developmental patterns and genetic diseases. Int. J. Nanomed. 1:89, 2006.CrossRefGoogle Scholar
  58. 58.
    van der Meijden, O., and M. Schijven. The value of haptic feedback in conventional and robot-assisted minimal invasive surgery and virtual reality training: a current review. Surg. Endosc. 23:1180–1190, 2009.CrossRefGoogle Scholar
  59. 59.
    Vemuri, A. S., J. C.-H. Wu, K.-C. Liu, and H.-S. Wu. Deformable three-dimensional model architecture for interactive augmented reality in minimally invasive surgery. Surg. Endosc. 26:3655–3662, 2012.CrossRefGoogle Scholar
  60. 60.
    Verveer, P., J. Swoger, F. Pampaloni, K. Greger, M. Marcello, and E. Stelzer. High-resolution three-dimensional imaging of large specimens with light sheet-based microscopy. Nat. Methods 4:311–313, 2007.CrossRefGoogle Scholar
  61. 61.
    Weissleder, R., and M. Pittet. Imaging in the era of molecular oncology. Nature 452:580–589, 2008.CrossRefGoogle Scholar
  62. 62.
    Wierzbicki, M., M. Drangova, G. Guiraudon, and T. Peters. Validation of dynamic heart models obtained using non-linear registration for virtual reality training, planning, and guidance of minimally invasive cardiac surgeries. Med. Image Anal. 8:387–401, 2004.CrossRefGoogle Scholar
  63. 63.
    Yan, D. Adaptive radiotherapy: merging principle into clinical practice. Semin. Radiat. Oncol. 20:79–83, 2010.CrossRefGoogle Scholar
  64. 64.
    Yan, D., F. Vicini, J. Wong, and A. Martinez. Adaptive radiation therapy. Phys. Med. Biol. 43:123, 1997.CrossRefGoogle Scholar
  65. 65.
    Yang, J. C., C. H. Chen, and M. C. Jeng. Integrating video-capture virtual reality technology into a physically interactive learning environment for English learning. Comput. Educ. 55:1346–1356, 2010.CrossRefGoogle Scholar
  66. 66.
    Zou, K. H., S. K. Warfield, A. Bharatha, C. M. C. Tempany, M. R. Kaus, S. J. Haker, W. M. W. Iii, and F. A. Jolesz. Statistical validation of image segmentation quality based on a spatial overlap index. Sci. Rep. 11:178–189, 2006.Google Scholar

Copyright information

© Biomedical Engineering Society 2018

Authors and Affiliations

  • Arash Abiri
    • 1
    • 2
    • 4
  • Yichen Ding
    • 1
    • 2
  • Parinaz Abiri
    • 1
    • 2
  • René R. Sevag Packard
    • 2
  • Vijay Vedula
    • 5
  • Alison Marsden
    • 5
    • 6
    • 7
  • C.-C. Jay Kuo
    • 8
  • Tzung K. Hsiai
    • 1
    • 2
    • 3
  1. 1.Department of BioengineeringUniversity of CaliforniaLos AngelesUSA
  2. 2.Department of MedicineDavid Geffen School of Medicine at UCLALos AngelesUSA
  3. 3.Medical EngineeringCalifornia Institute of TechnologyPasadenaUSA
  4. 4.Department of Biomedical EngineeringUniversity of CaliforniaIrvineUSA
  5. 5.Department of Pediatrics (Cardiology)Stanford UniversityStanfordUSA
  6. 6.Department of BioengineeringStanford UniversityStanfordUSA
  7. 7.Institute for Computational and Mathematical EngineeringStanford UniversityStanfordUSA
  8. 8.Department of Electrical EngineeringUniversity of Southern CaliforniaLos AngelesUSA

Personalised recommendations