Mobile markerless augmented reality and its application in forensic medicine

  • Thomas Kilgus
  • Eric Heim
  • Sven Haase
  • Sabine Prüfer
  • Michael Müller
  • Alexander Seitel
  • Markus Fangerau
  • Tamara Wiebe
  • Justin Iszatt
  • Heinz-Peter Schlemmer
  • Joachim Hornegger
  • Kathrin Yen
  • Lena Maier-Hein
Original Article

Abstract

Purpose

During autopsy, forensic pathologists today mostly rely on visible indication, tactile perception and experience to determine the cause of death. Although computed tomography (CT) data is often available for the bodies under examination, these data are rarely used due to the lack of radiological workstations in the pathological suite. The data may prevent the forensic pathologist from damaging evidence by allowing him to associate, for example, external wounds to internal injuries. To facilitate this, we propose a new multimodal approach for intuitive visualization of forensic data and evaluate its feasibility.

Methods

   A range camera is mounted on a tablet computer and positioned in a way such that the camera simultaneously captures depth and color information of the body. A server estimates the camera pose based on surface registration of CT and depth data to allow for augmented reality visualization of the internal anatomy directly on the tablet. Additionally, projection of color information onto the CT surface is implemented.

Results

   We validated the system in a postmortem pilot study using fiducials attached to the skin for quantification of a mean target registration error of \(4.4 \pm 1.3\) mm.

Conclusions

   The system is mobile, markerless, intuitive and real-time capable with sufficient accuracy. It can support the forensic pathologist during autopsy with augmented reality and textured surfaces. Furthermore, the system enables multimodal documentation for presentation in court. Despite its preliminary prototype status, it has high potential due to its low price and simplicity.

Keywords

Mobile augmented reality Forensic medicine Range imaging Kinect Iterative closest-point algorithm Mobile application Surface documentation 

Supplementary material

Supplementary material 1 (mp4 26146 KB)

Supplementary material 2 (mp4 14791 KB)

Supplementary material 3 (mp4 66602 KB)

References

  1. 1.
    Ampanozi G, Zimmermann D, Hatch GM, Ruder TD, Ross S, Flach PM, Thali MJ, Ebert LC (2012) Format preferences of district attorneys for post-mortem medical imaging reports: understandability, cost effectiveness, and suitability for the courtroom: A questionnaire based study. Legal Med 14(3):116–120CrossRefPubMedGoogle Scholar
  2. 2.
    Batcher KE (1968) Sorting networks and their applications. In: Proceedings of the April 30-May 2, Spring Joint Computer Conference, pp 307–314Google Scholar
  3. 3.
    Bauer S, Seitel A, Hofmann H, Blum T, Wasza J, Balda M, Meinzer HP, Navab N, Hornegger J, Maier-Hein L (2013) Real-time range imaging in health care: a survey. In: Time-of-flight and depth imaging. Sensors, algorithms, and applications. Springer, Berlin, pp 228–254Google Scholar
  4. 4.
    Besl PJ, Neil DM (1992) A method for registration of 3-d shapes. IEEE Trans Pattern Anal Mach Intell 14(2):239–256CrossRefGoogle Scholar
  5. 5.
    Bhatia A, Patel S, Pantol G, Wu YY, Plitnikas M, Hancock C (2013) Intra and inter-observer reliability of mobile tablet PACS viewer system versus standard PACS viewing station-diagnosis of acute central nervous system events. Open J Radiol 03(02):91–98CrossRefGoogle Scholar
  6. 6.
    Blum T, Kleeberger V, Bichlmeier C, Navab N (2012) mirracle: an augmented reality magic mirror system for anatomy education. In: Virtual reality short papers and posters (VRW), IEEE, pp 115–116Google Scholar
  7. 7.
    Chetverikov D, Svirko D, Stepanov D, Krsek P (2002) The trimmed iterative closest point algorithm. In: Proceedings of the 16th international conference on pattern recognition, vol 3. IEEE, pp 545–548Google Scholar
  8. 8.
    Dabrowski JR, Munson EV (2001) Is 100 milliseconds too fast? In: CHI EA ’01 CHI ’01 Extended abstracts on human factors in computing systems. ACM Press, pp 317–318Google Scholar
  9. 9.
    Danilchenko A, Fitzpatrick JM (2011) General approach to first-order error prediction in rigid point registration. IEEE Trans Med Imaging 30(3):679–693CrossRefPubMedGoogle Scholar
  10. 10.
    dos Santos TR, Seitel A, Kilgus T, Suwelack S, Wekerle AL, Kenngott H, Speidel S, Schlemmer HP, Meinzer HP, Heimann T, Maier-Hein L (2014) Pose-independent surface matching for intra-operative soft-tissue marker-less registration. Med Image Anal (accepted subject to minor revisions)Google Scholar
  11. 11.
    Ebert L, Hatch G, Thali M, Ross S (2013) Invisible touch–control of a DICOM viewer with finger gestures using the kinect depth camera. J Forensic Radiol Imaging 1(1):10–14CrossRefGoogle Scholar
  12. 12.
    Ebert LC, Hatch G, Ampanozi G, Thali MJ, Ross S (2012) You can’t touch this: touch-free navigation through radiological images. Surg Innov 19(3):301–307CrossRefPubMedGoogle Scholar
  13. 13.
    Ebert LC, Ptacek W, Breitbeck R, Fürst M, Kronreif G, Martinez RM, Thali M, Flach PM (2014) Virtobot 2.0: the future of automated surface documentation and CT-guided needle placement in forensic medicine. Forensic Sci Med Pathol 10(2):179–186CrossRefPubMedGoogle Scholar
  14. 14.
    Ebert LC, Ptacek W, Naether S, Fürst M, Ross S, Buck U, Weber S, Thali M (2009) Virtobot: a multi-functional robotic system for 3D surface scanning and automatic post mortem biopsy. Int J Med Robot Comput Assist Surg 6:18–27Google Scholar
  15. 15.
    Glocker B, Izadi S, Shotton J, Criminisi A (2013) Real-time RGB-D camera relocalization. In: IEEE international symposium on mixed and augmented reality (ISMAR). IEEE, pp 173–179Google Scholar
  16. 16.
    Handa A, Newcombe RA, Angeli A, Davison AJ (2012) Real-time camera tracking: when is high frame-rate best? In: Computer vision-ECCV 2012. Springer, pp 222–235Google Scholar
  17. 17.
    Hartley R, Zisserman A (2003) Multiple view geometry in computer vision. Cambridge University Press, CambridgeGoogle Scholar
  18. 18.
    Horn BKP (1987) Closed-form solution of absolute orientation using unit quaternions. J Opt Soc Am A 4(4):629–642CrossRefGoogle Scholar
  19. 19.
    Izadi S, Kim D, Hilliges O, Molyneaux D, Newcombe R, Kohli P, Shotton J, Hodges S, Freeman D, Davison A, Fritzgibbon A (2011) KinectFusion: real-time 3D reconstruction and interaction using a moving depth camera. In: Proceedings of the 24th annual ACM symposium on user interface software and technology. ACM, pp 559–568Google Scholar
  20. 20.
    John S, Poh ACC, Lim TCC, Chan EHY, Chong LR (2012) The iPad tablet computer for mobile on-call radiology diagnosis? Auditing discrepancy in CT and MRI reporting. J Digit Imaging 25(5):628–634CrossRefPubMedCentralPubMedGoogle Scholar
  21. 21.
    Khoshelham K, Elberink SO (2012) Accuracy and resolution of kinect depth data for indoor mapping applications. Sensors 12(12):1437–1454CrossRefPubMedCentralPubMedGoogle Scholar
  22. 22.
    Kilgus T, Franz A, Seitel A, März K, Bartha L, Fangerau M, Mersmann S, Groch A, Meinzer HP, Maier-Hein L (2012) Registration of partially overlapping surfaces for range image based augmented reality on mobile devices. In: SPIE medical imaging. International Society for Optics and Photonics, pp 83160T–83160TGoogle Scholar
  23. 23.
    Kilgus T, dos Santos TR, Seitel A, Yung K, Franz AM, Groch A, Wolf I, Meinzer HP, Maier-Hein L (2011) Generation of triangle meshes from time-of-flight data for surface registration. In: Bildverarbeitung für die Medizin 2011. Springer, pp 189–193Google Scholar
  24. 24.
    Kirk DB, Wen-mei WH (2012) Programming massively parallel processors: a hands-on approach. NewnesGoogle Scholar
  25. 25.
    Kramer J, Burrus N, Echtler F, Daniel HC, Parker M (2012) Hacking the Kinect. Springer, New YorkCrossRefGoogle Scholar
  26. 26.
    Li H, Sumner RW, Pauly M (2008) Global correspondence optimization for non-rigid registration of depth scans. In: Eurographics symposium on geometry processing, vol 27. Wiley, pp 1421–1430Google Scholar
  27. 27.
    Lorensen WE, Cline HE (1987) Marching cubes: a high resolution 3d surface construction algorithm. In: ACM siggraph computer graphics, vol 21. ACM, pp 163–169Google Scholar
  28. 28.
    Maier-Hein L, Fangerau M, Meinzer HP, Seitel A (2013) Visualization of anatomical data by augmented reality. US Patent App. 13/891,310Google Scholar
  29. 29.
    Maier-Hein L, Franz AM, Fangerau M, Schmidt M, Seitel A, Mersmann S, Kilgus T, Groch A, Yung K, dos Santos TR, Meinzer HP (2011) Towards mobile augmented reality for on-patient visualization of medical images. In: Bildverarbeitung für die Medizin 2011, Informatik Aktuell. Springer, pp 389–393Google Scholar
  30. 30.
    Maier-Hein L, Franz AM, Santos TRD, Schmidt M, Fangerau M, Meinzer HP, Fitzpatrick JM (2012) Convergent iterative closest-point algorithm to accommodate anisotropic and inhomogenous localization error. IEEE Trans Pattern Anal Mach Intell 34(8):1520–1532CrossRefPubMedGoogle Scholar
  31. 31.
    Mersmann S, Seitel A, Erz M, Jähne B, Nickel F, Mieth M, Mehrabi A, Maier-Hein L (2013) Calibration of time-of-flight cameras for accurate intraoperative surface reconstruction. Med Phys 40(8):082701Google Scholar
  32. 32.
    Müller M, Rassweiler MC, Klein J, Seitel A, Gondan M, Baumhauer M, Teber D, Rassweiler JJ, Meinzer HP, Maier-Hein L (2013) Mobile augmented reality for computer-assisted percutaneous nephrolithotomy. Int J Comput Assist Radiol Surg 8(4):663–675CrossRefPubMedGoogle Scholar
  33. 33.
    Myronenko A, Song X (2010) Point set registration: coherent point drift. IEEE Trans Pattern Anal Mach Intell 32(12):2262–2275CrossRefPubMedGoogle Scholar
  34. 34.
    Navab N, Traub J, Sielhorst T, Feuerstein M, Bichlmeier C (2007) Action- and workflow-driven augmented reality for computer-aided medical procedures. IEEE Comput Gr Appl 27(5):10–14CrossRefGoogle Scholar
  35. 35.
    Nolden M, Zelzer S, Seitel A, Wald D, Müller M, Franz AM, Maleike D, Fangerau M, Baumhauer M, Maier-Hein L, Maier-Hein KH, Meinzer HP, Wolf I (2013) The medical imaging interaction toolkit: challenges and advances. Int J Comput Assist Radiol Surg 8(4):607–620CrossRefPubMedGoogle Scholar
  36. 36.
    Qiu D, May S, Nüchter A (2009) GPU-accelerated nearest neighbor search for 3D registration. In: Proceedings of the 7th international conference on computer vision systems. Springer, pp 194–203Google Scholar
  37. 37.
    Roberts IS, Benamore RE, Benbow EW, Lee SH, Harris JN, Jackson A, Mallett S, Patankar T, Peebles C, Roobottom C, Trail ZC (2012) Post-mortem imaging as an alternative to autopsy in the diagnosis of adult deaths: a validation study. Lancet 379(9811):136–142CrossRefPubMedCentralPubMedGoogle Scholar
  38. 38.
    Rusu RB, Blodow N, Beetz M (2009) Fast point feature histograms (FPFH) for 3D registration. In: IEEE international conference on robotics and automation. IEEE, pp 3212–3217Google Scholar
  39. 39.
    Seeling P, Fitzek FH, Ertli G, Pulipaka A, Reisslein M (2010) Video network traffic and quality comparison of VP8 and H.264 SVC. In: Proceedings of the 3rd workshop on mobile video delivery. ACM, pp 33–37Google Scholar
  40. 40.
    Seitel A, Engel M, Sommer CM, Radeleff BA, Essert-Villard C, Baegert C, Fangerau M, Fritzsche KH, Yung K, Meinzer HP (2011) Computer-assisted trajectory planning for percutaneous needle insertions. Med Phys 38(6):3246–3259CrossRefPubMedGoogle Scholar
  41. 41.
    Seitel A, Yung K, Mersmann S, Kilgus T, Groch A, dos Santos TR, Franz AM, Nolden M, Meinzer HP, Maier-Hein L (2012) MITK-ToF: range data within MITK. Int J Comput Assist Radiol Surg 7(1):87–96 Google Scholar
  42. 42.
    Sugimoto M, Yasuda H, Koda K, Suzuki M, Yamazaki M, Tezuka T, Kosugi C, Higuchi R, Watayo Y, Yagawa Y, Uemura S, Tsuchiya H, Azuma T (2010) Image overlay navigation by markerless surface registration in gastrointestinal, hepatobiliary and pancreatic surgery. J Hepato Biliary Pancreat Sci 17(5):629–636Google Scholar
  43. 43.
    Thali MJ, Braun M, Wirth J, Vock P, Dirnhofer R (2003) 3D surface and body documentation in forensic medicine: 3-D/CAD photogrammetry merged with 3D radiological scanning. J Forensic Sci 48(6):1356–1365PubMedGoogle Scholar
  44. 44.
    Thali MJ, Dirnhofer R, Vock P (2009) The virtopsy approach: 3D optical and radiological scanning and reconstruction in forensic medicine. CRC Press, Boca RatonCrossRefGoogle Scholar
  45. 45.
    Urschler M, Höller J, Bornik A, Paul T, Giretzlehner M, Bischof H, Yen K, Scheurer E (2014) Intuitive presentation of clinical forensic data using anonymous and person-specific 3d reference manikins. Forensic Sci Int 241:155–166CrossRefPubMedGoogle Scholar
  46. 46.
    Valette S, Chassery JM, Prost R (2008) Generic remeshing of 3D triangular meshes with metric-dependent discrete voronoi diagrams. IEEE Trans Vis Comput Gr 14(2):369–381CrossRefGoogle Scholar
  47. 47.
    Yang J, Li H, Jia Y (2013) Go-ICP: solving 3D registration efficiently and globally optimally. In: International conference on computer vision. IEEE, pp 1457–1464Google Scholar
  48. 48.
    Zhang Z (2000) A flexible new technique for camera calibration. IEEE Trans Pattern Anal Mach Intell 22(11):1330–1334CrossRefGoogle Scholar
  49. 49.
    Zhu E, Masiello I, Hadadgar A, Zary N (2014) Augmented reality in healthcare education: an integrative review. PeerJ PrePrints 2:1–22Google Scholar

Copyright information

© CARS 2014

Authors and Affiliations

  • Thomas Kilgus
    • 1
  • Eric Heim
    • 1
  • Sven Haase
    • 5
  • Sabine Prüfer
    • 6
  • Michael Müller
    • 1
  • Alexander Seitel
    • 1
    • 2
  • Markus Fangerau
    • 3
  • Tamara Wiebe
    • 1
  • Justin Iszatt
    • 1
  • Heinz-Peter Schlemmer
    • 4
  • Joachim Hornegger
    • 5
  • Kathrin Yen
    • 6
  • Lena Maier-Hein
    • 1
  1. 1.Computer-Assisted InterventionsGerman Cancer Research Center (DKFZ)HeidelbergGermany
  2. 2.Robotics and Control LaboratoryUniversity of British ColumbiaVancouverCanada
  3. 3.Division of Medical and Biological InformaticDKFZHeidelbergGermany
  4. 4.Division of RadiologyDKFZHeidelbergGermany
  5. 5.Department of Computer Science, Pattern Recognition LabFriedrich-Alexander University Erlangen-NurembergErlangenGermany
  6. 6.Institute for Forensic Medicine and Traffic MedicineHeidelberg University HospitalHeidelbergGermany

Personalised recommendations