Cognitive Human-Machine Interfaces and Interactions for Unmanned Aircraft

  • Yixiang Lim
  • Subramanian Ramasamy
  • Alessandro Gardi
  • Trevor Kistan
  • Roberto Sabatini


This paper presents the concept of Cognitive Human-Machine Interfaces and Interactions (CHMI2) for Unmanned Aircraft System (UAS) Ground Control Stations (GCS). CHMI2 represents a new approach to aviation human factors engineering that introduces adaptive functionalities in the design of operators’ command, control and display functions. A CHMI2 system assesses human cognitive states based on measurement of key psycho-physiological observables. The cognitive states are used to predict and enhance operator performance in the accomplishment of aviation tasks, with the objective of improving the efficiency and effectiveness of the overall human-machine teaming. The CHMI2 system presented in this paper employs a four-layer architecture comprising sensing, extraction, classification and adaptation functionalities. An overview of each layer is provided along with the layer’s metrics, algorithms and functions. Two relevant case studies are presented to illustrate the interactions between the different layers, and the conceptual design of the associated display formats is described. The results indicate that specific eye tracking variables provide discrimination between different modes of control. Furthermore, results indicate that the higher levels of automation supported by the CHMI2 are beneficial in Separation Assurance and Collision Avoidance (SA&CA) scenarios involving low-detectability obstacles and stringent time constraints to implement recovery manoeuvres. These preliminary results highlight that the introduction of CHMI2 functionalities in future UAS can significantly reduce reaction time and enhance operational effectiveness of unmanned aircraft response to collision and loss of separation events, as well as improve the overall safety and efficiency of operations.


Unmanned aircraft Ground control station Sense and avoid Human factors engineering Psycho-physiological sensing Human machine interfaces Human machine interactions Cognitive ergonomics 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Jiang, T., Geller, J., Ni, D., Collura, J.: Unmanned aircraft system traffic management: Concept of operation and system architecture, International Journal of Transportation Science and Technology (2017)Google Scholar
  2. 2.
    FAA: Integration of Civil Unmanned Aircraft Systems (UAS) in the National Airspace System (NAS) Roadmap, 1st edn. Federal Aviation Administration, Washington (2013)Google Scholar
  3. 3.
    Dalamagkidis, K., Valavanis, K.P., Piegl, L.A.: Thoughts and Recommendations on a UAS Integration Roadmap. In: On Integrating Unmanned Aircraft Systems into the National Airspace System, ed, pp. 161–191 (2012)Google Scholar
  4. 4.
    Gardi, A., Ramasamy, S., Kistan, T., Sabatini, R.: UAS in the terminal area: challenges and opportunities. Wiley (2016)Google Scholar
  5. 5.
    Viguria, A.: Autonomy architectures. In: Encyclopedia of aerospace engineering. Wiley (2016)Google Scholar
  6. 6.
    Liu, J., Gardi, A., Ramasamy, S., Lim, Y., Sabatini, R.: Cognitive pilot-aircraft interface for single-pilot operations. Knowl.-Based Syst. 112, 37–53 (2016)CrossRefGoogle Scholar
  7. 7.
    ICAO: Doc 10019: Manual on Remotely Piloted Aircraft Systems (RPAS), 1st edn. International Civil Aviation Organization, Montréal (2015)Google Scholar
  8. 8.
    Neville, K., Blickensderfer, B., Archer, J., Kaste, K., Luxion, S.P.: A Cognitive Work Analysis to Identify Human-Machine Interface Design Challenges Unique to Uninhabited Aircraft Systems. In: Proceedings of the Human Factors and Ergonomics Society Annual Meeting, pp. 418–422 (2012)Google Scholar
  9. 9.
    Ramasamy, S., Sabatini, R., Gardi, A.: Cooperative and non-cooperative sense-and-avoid in the CNS + A context: A unified methodology. In: 2016 International Conference on Unmanned Aircraft Systems (ICUAS), pp. 531–539 (2016)Google Scholar
  10. 10.
    Ramasamy, S., Sabatini, R., Gardi, A., Liu, J.: LIDAR obstacle warning and avoidance system for unmanned aerial vehicle sense-and-avoid, Aerospace Science and Technology (2016)Google Scholar
  11. 11.
    Lai, J., Ford, J.J., Mejias, L., Wainwright, A.L., O’Shea, P.J., Walker, R.A.: Field-of-view, detection range, and false alarm trade-offs in vision-based aircraft detection (2012)Google Scholar
  12. 12.
    Lai, J., Mejias, L., Ford, J.J.: Airborne vision-based collision-detection system. J. Field Robot. 28, 137–157 (2011)CrossRefMATHGoogle Scholar
  13. 13.
    Gevins, A., Leong, H., Du, R., Smith, M.E., Le, J., DuRousseau, D., et al.: Towards measurement of brain function in operational environments. Biol. Psychol. 40, 169–186 (1995)CrossRefGoogle Scholar
  14. 14.
    Glaholt, M.G.: Eye tracking in the cockpit: a review of the relationships between eye movements and the aviator’s cognitive state (2014)Google Scholar
  15. 15.
    Rubio, S., Díaz, E, Martín, J., Puente, J.M.: Evaluation of subjective mental workload: A comparison of SWAT, NASA-TLX, and workload profile methods. Appl. Psychol. 53, 61–86 (2004)CrossRefGoogle Scholar
  16. 16.
    Kramer, A.F.: Physiological Metrics of Mental Workload: A Review of Recent Progress. In: Multiple-Task Performance, Damos, D. London: Taylor & Francis, pp. 279–328 (1991)Google Scholar
  17. 17.
    Gevins, A., Smith, M.E.: Neurophysiological measures of cognitive workload during human-computer interaction. Theor. Issues Ergon. Sci. 4, 113–131 (2003)CrossRefGoogle Scholar
  18. 18.
    Roscoe, A.H.: Assessing pilot workload. Why measure heart rate, HRV and respiration? Biol. Psychol. 34, 259–287 (1992)CrossRefGoogle Scholar
  19. 19.
    Henelius, A., Hirvonen, K., Holm, A., Korpela, J., Muller, K.: Mental workload classification using heart rate metrics. In: 2009 Annual International Conference of the IEEE Engineering in Medicine and Biology Society, pp. 1836–1839 (2009)Google Scholar
  20. 20.
    Marshall, S.P.: The index of cognitive activity: Measuring cognitive workload. In: 2002. Proceedings of the 2002 IEEE 7th Conference on Human Factors and Power Plants, pp. 7–5-7-9 (2002)Google Scholar
  21. 21.
    Harris, Sr., R.L., Glover, B.J., Spady, Jr., A.A.: Analytical techniques of pilot scanning behavior and their application (1986)Google Scholar
  22. 22.
    Jacob, R., Karn, K.S.: Eye tracking in human-computer interaction and usability research: Ready to deliver the promises. Mind 2, 4 (2003)Google Scholar
  23. 23.
    Di Nocera, F., Camilli, M., Terenzi, M.: A random glance at the flight deck: Pilots’ scanning strategies and the real-time assessment of mental workload. J. Cogn. Eng. Decis. Making 1, 271–285 (2007)CrossRefGoogle Scholar
  24. 24.
    Salvucci, D.D., Goldberg, J.H.: Identifying fixations and saccades in eye-tracking protocols. In: Proceedings of the 2000 Symposium on Eye Tracking Research & Applications, pp. 71–78 (2000)Google Scholar
  25. 25.
    Gilland, J.: Driving, eye-tracking and visual entropy: Exploration of age and task effects: ProQuest (2008)Google Scholar
  26. 26.
    Kohlmorgen, J., Dornhege, G., Braun, M., Blankertz, B., Mller, K.-.R., Curio, G., et al.: Improving Human Performance in a Real Operating Environment Through Real-time Mental Workload Detection. In: Toward Brain-Computer Interfacing, Dornhege, G., Milln, J., Hinterberger, T., McFarland D. and Mller, K.-.R. Cambridge: MIT Press, pp. 409–422 (2007)Google Scholar
  27. 27.
    Stevens, R., Galloway, T., Berka, C.: Integrating EEG models of cognitive load with machine learning models of scientific problem solving. Augmented Cogn. Past Present Fut. 2, 55–65 (2006)Google Scholar
  28. 28.
    Ke, Y., Qi, H., Zhang, L., Chen, S., Jiao, X., Zhou, P., et al.: Towards an effective cross-task mental workload recognition model using electroencephalography based on feature selection and support vector machine regression. Int. J. Psychophysiol. 98, 157–166 (2015)CrossRefGoogle Scholar
  29. 29.
    Yin, Z., Zhang, J., Wang, R.: Neurophysiological Feature-Based Detection of Mental Workload by Ensemble Support Vector Machines. In: Advances in Cognitive Neurodynamics (V), ed, pp. 469–475. Springer (2016)Google Scholar
  30. 30.
    Wilson, G.F., Russell, C.A.: Real-time assessment of mental workload using psychophysiological measures and artificial neural networks. Human Factors J. Human Factors Ergon. Soc. 45, 635–644 (2003)CrossRefGoogle Scholar
  31. 31.
    Noel, J.B., Bauer, K.W., Lanning, J.W.: Improving pilot mental workload classification through feature exploitation and combination: a feasibility study. Comput. Oper. Res. 32, 2713–2730 (2005)CrossRefMATHGoogle Scholar
  32. 32.
    Yang, G., Lin, Y.: Using ECG signal to quantify mental workload based on wavelet transform and competitive neural network techniques (special issuebiosensors: data acquisition, processing and control). Biomed. Fuzzy Human Sci. Off. J. Biomed. Fuzzy Syst. Assoc. 14, 17–25 (2009)Google Scholar
  33. 33.
    Zarjam, P., Epps, J., Chen, F., Lovell, N.H.: Estimating cognitive workload using wavelet entropy-based features during an arithmetic task. Comput. Biol. Med. 43, 2186–2195 (2013)CrossRefGoogle Scholar
  34. 34.
    Honal, M., Schultz, T.: Determine task demand from brain activity. In: Biosignals (1), pp. 100–107 (2008)Google Scholar
  35. 35.
    Wang, Z., Hope, R.M., Wang, Z., Ji, Q., Gray, W.D.: Cross-subject workload classification with a hierarchical Bayes model. NeuroImage 59, 64–69 (2012)CrossRefGoogle Scholar
  36. 36.
    Besson, P., Bourdin, C., Bringoux, L., Dousset, E., Maïano, C., Marqueste, T., et al.: Effectiveness of physiological and psychological features to estimate helicopter pilots’ workload: A Bayesian network approach. IEEE Trans. Intell. Transp. Syst. 14, 1872–1881 (2013)CrossRefGoogle Scholar
  37. 37.
    Guhe, M., Gray, W.D., Schoelles, M.J., Liao, W., Zhu, Z., Ji, Q.: Non-intrusive measurement of workload in real-time. In: Proceedings of the Human Factors and Ergonomics Society Annual Meeting, pp. 1157–1161 (2005)Google Scholar
  38. 38.
    Chaouachi, M., Jraidi, I., Frasson, C.: Modeling mental workload using EEG features for intelligent systems. In: International Conference on User Modeling, Adaptation, and Personalization, pp. 50–61 (2011)Google Scholar
  39. 39.
    Sabatini, R., Moore, T., Hill, C.: A novel GNSS integrity augmentation system for civil and military aircraft. Int. J. Mech. Indust. Sci. Eng. 7, 1433–1449 (2013)Google Scholar
  40. 40.
    Mulder, M., Van Der Vaart, H.J.C.: An information-centered analysis of the tunnel-in-the-sky display, part: The interception of curved trajectories. Int. J. Aviat. Psychol. 16, 21–49 (2006)CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media B.V. 2017

Authors and Affiliations

  • Yixiang Lim
    • 1
  • Subramanian Ramasamy
    • 1
  • Alessandro Gardi
    • 1
  • Trevor Kistan
    • 1
    • 2
  • Roberto Sabatini
    • 1
  1. 1.School of Engineering - Aerospace Engineering and AviationRMIT UniversityMelbourneAustralia
  2. 2.THALES Australia, World Trade CentreMelbourneAustralia

Personalised recommendations