Advertisement

Mobile markerless augmented reality and its application in forensic medicine

We’re sorry, something doesn't seem to be working properly.

Please try refreshing the page. If that doesn't work, please contact support so we can address the problem.

Abstract

Purpose

During autopsy, forensic pathologists today mostly rely on visible indication, tactile perception and experience to determine the cause of death. Although computed tomography (CT) data is often available for the bodies under examination, these data are rarely used due to the lack of radiological workstations in the pathological suite. The data may prevent the forensic pathologist from damaging evidence by allowing him to associate, for example, external wounds to internal injuries. To facilitate this, we propose a new multimodal approach for intuitive visualization of forensic data and evaluate its feasibility.

Methods

   A range camera is mounted on a tablet computer and positioned in a way such that the camera simultaneously captures depth and color information of the body. A server estimates the camera pose based on surface registration of CT and depth data to allow for augmented reality visualization of the internal anatomy directly on the tablet. Additionally, projection of color information onto the CT surface is implemented.

Results

   We validated the system in a postmortem pilot study using fiducials attached to the skin for quantification of a mean target registration error of \(4.4 \pm 1.3\) mm.

Conclusions

   The system is mobile, markerless, intuitive and real-time capable with sufficient accuracy. It can support the forensic pathologist during autopsy with augmented reality and textured surfaces. Furthermore, the system enables multimodal documentation for presentation in court. Despite its preliminary prototype status, it has high potential due to its low price and simplicity.

This is a preview of subscription content, log in to check access.

Access options

Buy single article

Instant unlimited access to the full article PDF.

US$ 39.95

Price includes VAT for USA

Subscribe to journal

Immediate online access to all issues from 2019. Subscription will auto renew annually.

US$ 99

This is the net price. Taxes to be calculated in checkout.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

Notes

  1. 1.

    E.g., in the institute for Forensic Medicine and Traffic Medicine in Heidelberg, almost for every case a CT scan is acquired before autopsy.

  2. 2.

    http://www.mitk.org.

  3. 3.

    We provide design and 3D printing plan publicly at http://www.thingiverse.com/thing:343425.

  4. 4.

    http://www.microsoft.com/en-us/kinectforwindowsdev/.

  5. 5.

    http://www.openni.org.

  6. 6.

    Bronchoscopy Model Sick Boy http://www.gtsimulators.com.

  7. 7.

    http://structure.io.

  8. 8.

    http://www.xbox.com/en-US/xbox-one/get-the-facts?xr=shellnav.

References

  1. 1.

    Ampanozi G, Zimmermann D, Hatch GM, Ruder TD, Ross S, Flach PM, Thali MJ, Ebert LC (2012) Format preferences of district attorneys for post-mortem medical imaging reports: understandability, cost effectiveness, and suitability for the courtroom: A questionnaire based study. Legal Med 14(3):116–120

  2. 2.

    Batcher KE (1968) Sorting networks and their applications. In: Proceedings of the April 30-May 2, Spring Joint Computer Conference, pp 307–314

  3. 3.

    Bauer S, Seitel A, Hofmann H, Blum T, Wasza J, Balda M, Meinzer HP, Navab N, Hornegger J, Maier-Hein L (2013) Real-time range imaging in health care: a survey. In: Time-of-flight and depth imaging. Sensors, algorithms, and applications. Springer, Berlin, pp 228–254

  4. 4.

    Besl PJ, Neil DM (1992) A method for registration of 3-d shapes. IEEE Trans Pattern Anal Mach Intell 14(2):239–256

  5. 5.

    Bhatia A, Patel S, Pantol G, Wu YY, Plitnikas M, Hancock C (2013) Intra and inter-observer reliability of mobile tablet PACS viewer system versus standard PACS viewing station-diagnosis of acute central nervous system events. Open J Radiol 03(02):91–98

  6. 6.

    Blum T, Kleeberger V, Bichlmeier C, Navab N (2012) mirracle: an augmented reality magic mirror system for anatomy education. In: Virtual reality short papers and posters (VRW), IEEE, pp 115–116

  7. 7.

    Chetverikov D, Svirko D, Stepanov D, Krsek P (2002) The trimmed iterative closest point algorithm. In: Proceedings of the 16th international conference on pattern recognition, vol 3. IEEE, pp 545–548

  8. 8.

    Dabrowski JR, Munson EV (2001) Is 100 milliseconds too fast? In: CHI EA ’01 CHI ’01 Extended abstracts on human factors in computing systems. ACM Press, pp 317–318

  9. 9.

    Danilchenko A, Fitzpatrick JM (2011) General approach to first-order error prediction in rigid point registration. IEEE Trans Med Imaging 30(3):679–693

  10. 10.

    dos Santos TR, Seitel A, Kilgus T, Suwelack S, Wekerle AL, Kenngott H, Speidel S, Schlemmer HP, Meinzer HP, Heimann T, Maier-Hein L (2014) Pose-independent surface matching for intra-operative soft-tissue marker-less registration. Med Image Anal (accepted subject to minor revisions)

  11. 11.

    Ebert L, Hatch G, Thali M, Ross S (2013) Invisible touch–control of a DICOM viewer with finger gestures using the kinect depth camera. J Forensic Radiol Imaging 1(1):10–14

  12. 12.

    Ebert LC, Hatch G, Ampanozi G, Thali MJ, Ross S (2012) You can’t touch this: touch-free navigation through radiological images. Surg Innov 19(3):301–307

  13. 13.

    Ebert LC, Ptacek W, Breitbeck R, Fürst M, Kronreif G, Martinez RM, Thali M, Flach PM (2014) Virtobot 2.0: the future of automated surface documentation and CT-guided needle placement in forensic medicine. Forensic Sci Med Pathol 10(2):179–186

  14. 14.

    Ebert LC, Ptacek W, Naether S, Fürst M, Ross S, Buck U, Weber S, Thali M (2009) Virtobot: a multi-functional robotic system for 3D surface scanning and automatic post mortem biopsy. Int J Med Robot Comput Assist Surg 6:18–27

  15. 15.

    Glocker B, Izadi S, Shotton J, Criminisi A (2013) Real-time RGB-D camera relocalization. In: IEEE international symposium on mixed and augmented reality (ISMAR). IEEE, pp 173–179

  16. 16.

    Handa A, Newcombe RA, Angeli A, Davison AJ (2012) Real-time camera tracking: when is high frame-rate best? In: Computer vision-ECCV 2012. Springer, pp 222–235

  17. 17.

    Hartley R, Zisserman A (2003) Multiple view geometry in computer vision. Cambridge University Press, Cambridge

  18. 18.

    Horn BKP (1987) Closed-form solution of absolute orientation using unit quaternions. J Opt Soc Am A 4(4):629–642

  19. 19.

    Izadi S, Kim D, Hilliges O, Molyneaux D, Newcombe R, Kohli P, Shotton J, Hodges S, Freeman D, Davison A, Fritzgibbon A (2011) KinectFusion: real-time 3D reconstruction and interaction using a moving depth camera. In: Proceedings of the 24th annual ACM symposium on user interface software and technology. ACM, pp 559–568

  20. 20.

    John S, Poh ACC, Lim TCC, Chan EHY, Chong LR (2012) The iPad tablet computer for mobile on-call radiology diagnosis? Auditing discrepancy in CT and MRI reporting. J Digit Imaging 25(5):628–634

  21. 21.

    Khoshelham K, Elberink SO (2012) Accuracy and resolution of kinect depth data for indoor mapping applications. Sensors 12(12):1437–1454

  22. 22.

    Kilgus T, Franz A, Seitel A, März K, Bartha L, Fangerau M, Mersmann S, Groch A, Meinzer HP, Maier-Hein L (2012) Registration of partially overlapping surfaces for range image based augmented reality on mobile devices. In: SPIE medical imaging. International Society for Optics and Photonics, pp 83160T–83160T

  23. 23.

    Kilgus T, dos Santos TR, Seitel A, Yung K, Franz AM, Groch A, Wolf I, Meinzer HP, Maier-Hein L (2011) Generation of triangle meshes from time-of-flight data for surface registration. In: Bildverarbeitung für die Medizin 2011. Springer, pp 189–193

  24. 24.

    Kirk DB, Wen-mei WH (2012) Programming massively parallel processors: a hands-on approach. Newnes

  25. 25.

    Kramer J, Burrus N, Echtler F, Daniel HC, Parker M (2012) Hacking the Kinect. Springer, New York

  26. 26.

    Li H, Sumner RW, Pauly M (2008) Global correspondence optimization for non-rigid registration of depth scans. In: Eurographics symposium on geometry processing, vol 27. Wiley, pp 1421–1430

  27. 27.

    Lorensen WE, Cline HE (1987) Marching cubes: a high resolution 3d surface construction algorithm. In: ACM siggraph computer graphics, vol 21. ACM, pp 163–169

  28. 28.

    Maier-Hein L, Fangerau M, Meinzer HP, Seitel A (2013) Visualization of anatomical data by augmented reality. US Patent App. 13/891,310

  29. 29.

    Maier-Hein L, Franz AM, Fangerau M, Schmidt M, Seitel A, Mersmann S, Kilgus T, Groch A, Yung K, dos Santos TR, Meinzer HP (2011) Towards mobile augmented reality for on-patient visualization of medical images. In: Bildverarbeitung für die Medizin 2011, Informatik Aktuell. Springer, pp 389–393

  30. 30.

    Maier-Hein L, Franz AM, Santos TRD, Schmidt M, Fangerau M, Meinzer HP, Fitzpatrick JM (2012) Convergent iterative closest-point algorithm to accommodate anisotropic and inhomogenous localization error. IEEE Trans Pattern Anal Mach Intell 34(8):1520–1532

  31. 31.

    Mersmann S, Seitel A, Erz M, Jähne B, Nickel F, Mieth M, Mehrabi A, Maier-Hein L (2013) Calibration of time-of-flight cameras for accurate intraoperative surface reconstruction. Med Phys 40(8):082701

  32. 32.

    Müller M, Rassweiler MC, Klein J, Seitel A, Gondan M, Baumhauer M, Teber D, Rassweiler JJ, Meinzer HP, Maier-Hein L (2013) Mobile augmented reality for computer-assisted percutaneous nephrolithotomy. Int J Comput Assist Radiol Surg 8(4):663–675

  33. 33.

    Myronenko A, Song X (2010) Point set registration: coherent point drift. IEEE Trans Pattern Anal Mach Intell 32(12):2262–2275

  34. 34.

    Navab N, Traub J, Sielhorst T, Feuerstein M, Bichlmeier C (2007) Action- and workflow-driven augmented reality for computer-aided medical procedures. IEEE Comput Gr Appl 27(5):10–14

  35. 35.

    Nolden M, Zelzer S, Seitel A, Wald D, Müller M, Franz AM, Maleike D, Fangerau M, Baumhauer M, Maier-Hein L, Maier-Hein KH, Meinzer HP, Wolf I (2013) The medical imaging interaction toolkit: challenges and advances. Int J Comput Assist Radiol Surg 8(4):607–620

  36. 36.

    Qiu D, May S, Nüchter A (2009) GPU-accelerated nearest neighbor search for 3D registration. In: Proceedings of the 7th international conference on computer vision systems. Springer, pp 194–203

  37. 37.

    Roberts IS, Benamore RE, Benbow EW, Lee SH, Harris JN, Jackson A, Mallett S, Patankar T, Peebles C, Roobottom C, Trail ZC (2012) Post-mortem imaging as an alternative to autopsy in the diagnosis of adult deaths: a validation study. Lancet 379(9811):136–142

  38. 38.

    Rusu RB, Blodow N, Beetz M (2009) Fast point feature histograms (FPFH) for 3D registration. In: IEEE international conference on robotics and automation. IEEE, pp 3212–3217

  39. 39.

    Seeling P, Fitzek FH, Ertli G, Pulipaka A, Reisslein M (2010) Video network traffic and quality comparison of VP8 and H.264 SVC. In: Proceedings of the 3rd workshop on mobile video delivery. ACM, pp 33–37

  40. 40.

    Seitel A, Engel M, Sommer CM, Radeleff BA, Essert-Villard C, Baegert C, Fangerau M, Fritzsche KH, Yung K, Meinzer HP (2011) Computer-assisted trajectory planning for percutaneous needle insertions. Med Phys 38(6):3246–3259

  41. 41.

    Seitel A, Yung K, Mersmann S, Kilgus T, Groch A, dos Santos TR, Franz AM, Nolden M, Meinzer HP, Maier-Hein L (2012) MITK-ToF: range data within MITK. Int J Comput Assist Radiol Surg 7(1):87–96

  42. 42.

    Sugimoto M, Yasuda H, Koda K, Suzuki M, Yamazaki M, Tezuka T, Kosugi C, Higuchi R, Watayo Y, Yagawa Y, Uemura S, Tsuchiya H, Azuma T (2010) Image overlay navigation by markerless surface registration in gastrointestinal, hepatobiliary and pancreatic surgery. J Hepato Biliary Pancreat Sci 17(5):629–636

  43. 43.

    Thali MJ, Braun M, Wirth J, Vock P, Dirnhofer R (2003) 3D surface and body documentation in forensic medicine: 3-D/CAD photogrammetry merged with 3D radiological scanning. J Forensic Sci 48(6):1356–1365

  44. 44.

    Thali MJ, Dirnhofer R, Vock P (2009) The virtopsy approach: 3D optical and radiological scanning and reconstruction in forensic medicine. CRC Press, Boca Raton

  45. 45.

    Urschler M, Höller J, Bornik A, Paul T, Giretzlehner M, Bischof H, Yen K, Scheurer E (2014) Intuitive presentation of clinical forensic data using anonymous and person-specific 3d reference manikins. Forensic Sci Int 241:155–166

  46. 46.

    Valette S, Chassery JM, Prost R (2008) Generic remeshing of 3D triangular meshes with metric-dependent discrete voronoi diagrams. IEEE Trans Vis Comput Gr 14(2):369–381

  47. 47.

    Yang J, Li H, Jia Y (2013) Go-ICP: solving 3D registration efficiently and globally optimally. In: International conference on computer vision. IEEE, pp 1457–1464

  48. 48.

    Zhang Z (2000) A flexible new technique for camera calibration. IEEE Trans Pattern Anal Mach Intell 22(11):1330–1334

  49. 49.

    Zhu E, Masiello I, Hadadgar A, Zary N (2014) Augmented reality in healthcare education: an integrative review. PeerJ PrePrints 2:1–22

Download references

Acknowledgments

This study was performed as part of the Research Training Group 1126 Intelligente Chirurgie and project PD 15577, both funded by the German Research Foundation (DFG). We further thank all staff—the Division of Radiology of the German Cancer Research Center and the Institute for Anatomy and Cell Biology in Heidelberg—who supported the phantom and postmortem experiments. Thanks go to our intern, Simon Weiser, for assisting in postmortem experiments.

Author information

Correspondence to Thomas Kilgus.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (mp4 26146 KB)

Supplementary material 2 (mp4 14791 KB)

Supplementary material 3 (mp4 66602 KB)

Supplementary material 1 (mp4 26146 KB)

Supplementary material 2 (mp4 14791 KB)

Supplementary material 3 (mp4 66602 KB)

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Kilgus, T., Heim, E., Haase, S. et al. Mobile markerless augmented reality and its application in forensic medicine. Int J CARS 10, 573–586 (2015). https://doi.org/10.1007/s11548-014-1106-9

Download citation

Keywords

  • Mobile augmented reality
  • Forensic medicine
  • Range imaging
  • Kinect
  • Iterative closest-point algorithm
  • Mobile application
  • Surface documentation