A Virtual Reality Framework for Multimodal Imagery for Vessels in Polar Regions

  • Scott SorensenEmail author
  • Abhishek Kolagunda
  • Andrew R. Mahoney
  • Daniel P. Zitterbart
  • Chandra Kambhamettu
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10133)


Maintaining total awareness when maneuvering an ice-breaking vessel is key to its safe operation. Camera systems are commonly used to augment the capabilities of those piloting the vessel, but rarely are these camera systems used beyond simple video feeds. To aid in visualization for decision making and operation, we present a scheme for combining multiple modalities of imagery into a cohesive Virtual Reality application which provides the user with an immersive, real scale, view of conditions around a research vessel operating in polar waters. The system incorporates imagery from a \(360^{\circ }\) Long-wave Infrared camera as well as an optical band stereo camera system. The Virtual Reality application allows the operator multiple natural ways of interacting with and observing the data, as well as provides a framework for further inputs and derived observations.


Video Stream Camera System Head Mount Display Structure From Motion Center Camera 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.



We would like to thank the crew and scientific party of the Polarstern ARK-XXVII/3 research cruise. This work is funded by NSF CDI Type I grant 1124664.


  1. 1.
    Chakravorty, T., Bilodeau, G., Granger, E.: Automatic image registration in infrared-visible videos using polygon vertices. CoRR, abs/1403.4232 (2014)Google Scholar
  2. 2.
    Chen, C., Ross, A.: Matching thermal to visible face images using hidden factor analysis in a cascaded subspace learning framework. Pattern Recogn. Lett. 72, 25–32 (2016). Special Issue on ICPR 2014 Awarded PapersCrossRefGoogle Scholar
  3. 3.
    Edwards, J.: Telepresence: virtual reality in the real world [special reports]. IEEE Signal Process. Mag. 28(6), 9–142 (2011)CrossRefGoogle Scholar
  4. 4.
    Ellmauthaler, A., da Silva, E.A.B., Pagliari, C.L., Gois, J.N., Neves, S.R.: A novel iterative calibration approach for thermal infrared cameras. In: 2013 20th IEEE International Conference on Image Processing (ICIP), pp. 2182–2186, September 2013Google Scholar
  5. 5.
    Gurusamy, K.S., Aggarwal, R., Palanivelu, L., Davidson, B.R.: Virtual reality training for surgical trainees in laparoscopic surgery. Cochrane Database Syst. Rev. (1) (2009). Article Number CD006575, doi: 10.1002/14651858.CD006575.pub2
  6. 6.
    Hilsenstein, V.: Surface reconstruction of water waves using thermographic stereo imaging (2005)Google Scholar
  7. 7.
    Hu, S., Choi, J., Chan, A.L., Schwartz, W.R.: Thermal-to-visible face recognition using partial least squares. J. Opt. Soc. Am. A 32(3), 431–442 (2015)CrossRefGoogle Scholar
  8. 8.
    Jung, H.H., Lyou, J.: Matching of thermal and color images with application to power distribution line fault detection. In: 2015 15th International Conference on Control, Automation and Systems (ICCAS), pp. 1389–1392, October 2015Google Scholar
  9. 9.
    Kadavasal, M.S., Oliver, J.H.: Sensor enhanced virtual reality teleoperation in dynamic environment. In: Virtual Reality Conference, VR 2007, pp. 297–298. IEEE, March 2007Google Scholar
  10. 10.
    Kipper, G., Rampolla, J.: Augmented Reality: An Emerging Technologies Guide to AR, 1st edn. Syngress Publishing, Rockland (2012)Google Scholar
  11. 11.
    Lowe, D.: Distinctive image features from scale-invariant keypoints. Int. J. Comput. Vis. 60, 91–110 (2003)CrossRefGoogle Scholar
  12. 12.
    Morina, N., Ijntema, H., Meyerbröker, K., Emmelkamp, P.M.: Can virtual reality exposure therapy gains be generalized to real-life? a meta-analysis of studies applying behavioral assessments. Behav. Res. Ther. 74, 18–24 (2015)CrossRefGoogle Scholar
  13. 13.
    Orghidan, R., Salvi, J., Gordan, M., Orza, B.: Camera calibration using two or three vanishing points. In: Proceedings of the Federated Conference on Computer Science and Information Systems (FedCSIS), Wrocław, Poland, pp. 9–12 (2012)Google Scholar
  14. 14.
    Rohith, M., Sorensen, S., Rhein, S., Kambhamettu, C.: Shape from stereo and shading by gradient constrained interpolation. In: 2013 20th IEEE International Conference on Image Processing (ICIP), pp. 2232–2236, September 2013Google Scholar
  15. 15.
    Rohith, M.V., Somanath, G., Kambhamettu, C., Geiger, C.A.: Stereo analysis of low textured regions with application towards sea-ice reconstruction. In: Arabnia, H.R., Schaefer, G. (eds.) IPCV, pp. 23–29. CSREA Press (2009)Google Scholar
  16. 16.
    Saponaro, P., Sorensen, S., Rhein, S., Kambhamettu, C.: Improving calibration of thermal stereo cameras using heated calibration board. In: 2015 IEEE International Conference on Image Processing (ICIP), pp. 4718–4722, September 2015Google Scholar
  17. 17.
    Sarfraz, M.S., Stiefelhagen, R.: Deep perceptual mapping for thermal to visible face recognition. CoRR, abs/1507.02879 (2015)Google Scholar
  18. 18.
    Sobel, I., Feldman, G.: A 3\(\times \)3 Isotropic Gradient Operator for Image Processing (1968). Never published but presented at a talk at the Stanford Artificial Intelligence ProjectGoogle Scholar
  19. 19.
    Torabi, A., Massé, G., Bilodeau, G.-A.: An iterative integrated framework for thermal-visible image registration, sensor fusion, and people tracking for video surveillance applications. Comput. Vis. Image Underst. 116(2), 210–221 (2012)CrossRefGoogle Scholar
  20. 20.
    Vidas, S., Lakemond, R., Denman, S., Fookes, C., Sridharan, S., Wark, T.: A mask-based approach for the geometric calibration of thermal-infrared cameras. IEEE Trans. Instrum. Meas. 61(6), 1625–1635 (2012)CrossRefGoogle Scholar
  21. 21.
    O. VR. Oculus best practices (2016, online)Google Scholar
  22. 22.
    Yang, R., Yang, W., Chen, Y., Wu, X.: Geometric calibration of IR camera using trinocular vision. J. Lightwave Technol. 29(24), 3797–3803 (2011)CrossRefGoogle Scholar
  23. 23.
    Yiu-Ming, H., Ng, R.: Acquisition of 3D surface temperature distribution of a car body. In: 2005 IEEE International Conference on Information Acquisition, 5 pages, June 2005Google Scholar
  24. 24.
    Young, A.T.: Distance to the horizon (2012)Google Scholar
  25. 25.
    Zhang, Z.: A flexible new technique for camera calibration. IEEE Trans. Pattern Anal. Mach. Intell. 22(11), 1330–1334 (2000)CrossRefGoogle Scholar
  26. 26.
    Zheng, Y., Peng, S.: A practical roadside camera calibration method based on least squares optimization. IEEE Trans. Intell. Transp. Syst. 15(2), 831–843 (2014)CrossRefGoogle Scholar
  27. 27.
    Zitterbart, D.P., Kindermann, L., Burkhardt, E., Boebel, O.: Automatic round-the-clock detection of whales for mitigation from underwater noise impacts. PLoS ONE 8(8), e71217 (2013)CrossRefGoogle Scholar

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  • Scott Sorensen
    • 1
    Email author
  • Abhishek Kolagunda
    • 1
  • Andrew R. Mahoney
    • 2
  • Daniel P. Zitterbart
    • 3
  • Chandra Kambhamettu
    • 1
  1. 1.University of DelawareNewarkUSA
  2. 2.University of Alaska FairbanksFairbanksUSA
  3. 3.Alfred Wegener Institute for Polar and Marine ResearchBremerhavenGermany

Personalised recommendations