Skip to main content
SpringerLink
Log in

Multi-screen dynamic viewpoint system for helicopter simulator

  • Original Paper
  • Published:
International Journal on Interactive Design and Manufacturing (IJIDeM) Aims and scope Submit manuscript

Abstract

Helicopter equipment plays a more and more important role in military and civil fields by virtue of its high maneuverability, takeoff and landing without terrain restrctions, hovering operation, so it is particularly necessary to train pilots’ operational skills for the wider use of helicopters. Nowadays, with the rapid development of virtual reality technology, the helicopter simulator technology has become a research hotspot in this filed. However, creating a driving environment with high sense of presence is the difficulty of simulator research. This paper proposed a multi-screen splicing technique based on dynamic viewpoint which added the driver’s head posture estimation. Firstly, the sensor technology was used to track the pilots’ head pose information in real time. Secondly, the rough estimation algorithm based on KALMAN enhancement and the precise estimation algorithm based on ICP of the driver’s pose was designed, then the position and angel of the viewpoint in the virtual scene are dynamically updated according to the pilots’ head pose. Moreover, the screen splicing method under dynamic viewpoint was also studied. The method proposed in this paper not only made the virtual scene change with the change of drivers’ head pose, but also provided the virtual scene from a large perspective, so as to the visual sense of the simulator was improved. The validity of the proposed algorithm was verified by four experiments in this paper.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18

Similar content being viewed by others

Data Availability

The data used to support the findings of this study are available from the corresponding author upon request.

Code availability

The code used to support the study are available from the corresponding author upon request.

References

  1. Walko, C., Peinecke, N.: Integration and use of an augmented reality display in a maritime helicopter simulator[J]. Opt. Eng. 59(4), 1 (2020)

    Article  Google Scholar 

  2. Wohlgenannt, I., Simons, A., Stieglitz, S.: Virtual reality[J]. Bus Inf Syst Eng 62, 455–461 (2020)

    Article  Google Scholar 

  3. Yasui, K., Kawai, R., Arakawa, T.: Immersive virtual reality supporting content for evaluating interface using oculus rift and leap motion[J]. Int. J. Innov. Comput. Inf. Control 15(2), 591–605 (2019)

    Google Scholar 

  4. Lenz, T.J., Kossyreva, E.A., Colella, M.R.: Helicopter emergency medical services utilization. Air Med. J. 38(4), 261–265 (2019)

    Article  Google Scholar 

  5. Dwight L, Randall. The helicopter as a meteorological tool[J]. Bull Am Meteor-ological Soc, (2018), 33 (10): 416–419

  6. Chen, R, Wen, C.Y, Lorente, S, Bejan, A. The evolution of helicopters[J]. J. Appl. Phys., (2016), 120 (1): 014901

  7. Chessa, M., Maiello, G., Borsari, A., Bex, P.J.: The perceptual quality of the oculus rift for immersive virtual reality. Hum. Comp. Interact. 34(1), 51–82 (2019)

    Article  Google Scholar 

  8. Parkin, S.: Oculus rift. Technol. Rev. 117(3), 50–52 (2014)

    MathSciNet  Google Scholar 

  9. PR Newswire. InstaVR Launches Single-Click Publishing to Oculus Quest and Rift S, Including World-Class Touch Interactivity. [J]. PR Newswire US, (2019).

  10. Palmisano, S., Mursic, R., Kim, J.: Vection and cybersickness generated by head-and-display motion in the Oculus Rift[J]. Displays 46, 1–8 (2017)

    Article  Google Scholar 

  11. Walko, C, Maibach, M.J. Flying a helicopter with the HoloLens as head-mounted display[C]// Virtual, Augmented, and Mixed Reality (XR) Technology for Multi-Domain Operations II. (2021)

  12. Morales, E.F., Murrieta-Cid, R., Becerra, I. et al. A survey on deep learning and deep reinforcement learning in robotics with a tutorial on deep reinforcement learning. Intel Serv Robotics. (2021)

  13. Dai, Y., Jin, T., Song, Y., Sun, S., Wu, C.: Convolutional neural network with spatial-variant convolution kernel[J]. Remote Sens 12, 2811 (2020)

    Article  Google Scholar 

  14. Mukherjee, S., Robertson, N.M.: Deep head pose: gaze-direction estimation in multimodal video[J]. IEEE Trans. Multimedia 17(11), 2094–2107 (2015)

    Article  Google Scholar 

  15. Ranjan, R., Patel, V.M., Chellappa, R.: Hyperface: a deep multi-task learning framework for face detection, landmark localization, pose estimation, and gender recognition[J]. IEEE Trans. Pattern Anal. Mach. Intell. 41(1), 121–135 (2019)

    Article  Google Scholar 

  16. Xue Sun, Hu., Liu, G.W., Zhou, Y.: Training effectiveness evaluation of helicopter emergency relief based on virtual simulation[J]. Chin. J. Aeronaut. 31(10), 36861 (2018)

    Google Scholar 

  17. Haofan, W., Zhenghua, C., Yi Z., Hybrid coarse-fine classification for head pose estimation[J]. Computer Vision and Pattern Recognition, (2019)

  18. Huang, Y., Da, F.: Registration algorithm for point cloud based on normalized cross-correlation[J]. IEEE Access 7, 137136–137146 (2019)

    Article  Google Scholar 

  19. Vergara, V.M., Abrol, A., Calhoun, V.D.: An average sliding window correlation method for dynamic functional connectivity[J]. Hum. Brain Mapp. 40, 2089–2103 (2019)

    Article  Google Scholar 

  20. Ding, S., Shi, Z., Tao, D., An, Bo.: Recent advances in support vector Machines[J]. Neurocomputing 211(26), 1–3 (2016)

    Google Scholar 

  21. Huang, J., Shao, X., Ichsler, H. Face pose discrimination using support vector machines (SVM). [C]// Proceedings. Fourteenth International Conference on Pattern Recognition (Cat. No. 98EX170). IEEE, (1998), 1: 154–156

  22. Zhang, Z.Y, Liu, M, et al. Head pose estimation in seminar room using multi view face detectors. [C]// International Evaluation Workshop on Classification of Events, Activities and Relationships. Springer, Berlin, Heidelberg, (2006): 299–304.

  23. Yue, J., Miao, Z., He, Y., Nianchun, D.: Loss architecture search for few-shot object recognition[J]. Complexity (2020). https://doi.org/10.1155/2020/1041962

    Article  Google Scholar 

  24. Hess, A.S., Hess, J.R.: Principal component analysis[J]. Transfusion 58(7), 1580–1582 (2018)

    Article  Google Scholar 

  25. Xie, X., Sun, S.: Multi-view support vector machines with the consensus and complementarity information. IEEE Trans. Knowl. Data Eng. 32(12), 2401–2413 (2020)

    Article  Google Scholar 

  26. Abate, A.F., Barra, P., Pero, C., Tucci, M.: Head pose estimation by regression algorithm[J]. Patt. Rec. Lett. 140, 179–185 (2020)

    Article  Google Scholar 

  27. Liu, X., Lu, H., Zhang, D.: Head Pose Estimation Based on Manifold Embedding and Distance Metric Learning[C]// Asian Conference on Computer Vision-accv. DBLP, (2010)

  28. You, O.T., Pae, D.S., Kim, S.H., et al.: Pattern matching for industrial object recognition using geometry-based vector mapping descriptors[J]. Pattern Anal. Applic. 21, 1167–1183 (2018)

    Article  MathSciNet  Google Scholar 

  29. Paul, D.P., Mukherjee, P., Das, A., Gangopadhyay, A.R.: Improved random forest for classification[J]. IEEE Trans. Image Proc. (2018). https://doi.org/10.1109/TIP.2018.2834830

    Article  MathSciNet  MATH  Google Scholar 

  30. Bracmann, E., Michel, F., Krull, A., Yang, M.Y., Gumhold, S., Rother, C.: Uncertainty-driven 6D pose estimation of objects and scenes from a single RGB image. [J]. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), (2016): 3364–3372

  31. Zhang, H., Zhan, H., Zhang, L., Xu, F., Ding, X.: A kalman filter-based kernelized correlation filter algorithm for pose measurement of a micro-robot[J]. Micromachines 12, 774 (2021)

    Article  Google Scholar 

  32. Kalman, R.E.: A new approach to linear filtering and prediction problems[J]. Trans. ASME J. Bas. Eng., (2009), 82: 12

  33. Li, P., Wang, R., Wang, Y., Tao, W.: Evaluation of the ICP algorithm in 3D point cloud registration[J]. IEEE Access 8, 68030–68048 (2020)

    Article  Google Scholar 

Download references

Funding

This research was funded by Hebei Natural Science Foundation, grant number E2019203431.

Author information

Authors and Affiliations

  1. School of Vehicles and Energy, Yanshan University, Qinhuangdao, 066004, China

    Tao Ni, Panhong Zhang & Yahui Zhao

  2. School of Mechanical Engineering, Yanshan University, Qinhuangdao, 066004, China

    Haojie Yang & Dingxuan Zhao

  3. Baidu (China) Limited Company, Shanghai, 100193, China

    Sheng Chen

Authors
  1. Tao Ni
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Panhong Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yahui Zhao
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Sheng Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Haojie Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Dingxuan Zhao
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

Conceptualization, PZ Data Curation, TN Formal Analysis, PZ Funding Acquisition, DZ Methodology, TN Project administration, DZ Software, YZ, SC Investigation, YZ, HY. All authors have read and agreed to the published version of the manuscript.

Corresponding author

Correspondence to Dingxuan Zhao.

Ethics declarations

Conflicts of interest

The authors declared that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Ni, T., Zhang, P., Zhao, Y. et al. Multi-screen dynamic viewpoint system for helicopter simulator. Int J Interact Des Manuf 16, 955–968 (2022). https://doi.org/10.1007/s12008-021-00814-9

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12008-021-00814-9

Keywords

Search

Navigation