Skip to main content
View expanded cover

Advances in Human Factors in Robots and Unmanned Systems pp 291–303Cite as

Confidence-Based State Estimation: A Novel Tool for Test and Evaluation of Human-Systems

Part of the Advances in Intelligent Systems and Computing book series (AISC,volume 499)

Abstract

Test and evaluation (T&E) of complex human-in-the loop systems has been a challenge for system developers. Traditional methods for T&E rely on questionnaires given periodically in combination with task performance measures to quantify the effectiveness of a given system. This approach is inherently obtrusive and interferes with natural system interaction. Here, we propose a method to leverage unobtrusive wearable technology to create a system for continuously assessing human state. Previous efforts at this type of assessment have often failed to generalize beyond controlled laboratory environments due to increased variability in signal quality from both the wearable sensors and in human behavior. We propose a method to account for this variability using measures of confidence to create robust estimates of state capable of dynamically adapting to changes in behavior over time. We postulate that the confidence-based approach can provide high-resolution estimates of state that will augment T&E of complex systems.

Keywords

  • Test and evaluation
  • Human assessment
  • Confidence
  • Sensor fusion
  • State estimation

This is a preview of subscription content, access via your institution.

Chapter
USD   29.95
Price excludes VAT (USA)
  • DOI: 10.1007/978-3-319-41959-6_24
  • Chapter length: 13 pages
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
eBook
USD   189.00
Price excludes VAT (USA)
  • ISBN: 978-3-319-41959-6
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
Softcover Book
USD   249.99
Price excludes VAT (USA)
Learn about institutional subscriptions
Fig. 1
Fig. 2
Fig. 3

References

  1. Council, N.R.: Human-system integration in the system development process: a new look. The National Academies Press, Washington, DC (2007)

    Google Scholar 

  2. Stikic, M., Johnson, R.R., Levendowski, D.J., Popovic, D.P., Olmstead, R.E., Berka, C.: EEG-derived estimators of present and future cognitive performance. Front Hum. Neurosci. 5 (2011)

    Google Scholar 

  3. Shen, K.-Q., Li, X.-P., Ong, C.-J., Shao, S.-Y., Wilder-Smith, E.P.V.: EEG-based mental fatigue measurement using multi-class support vector machines with confidence estimate. Clin. Neurophysiol. 119, 1524–1533 (2008)

    CrossRef  Google Scholar 

  4. Lin, C.-T., Wu, R.-C., Jung, T.-P., Liang, S.-F., Huang, T.-Y.: Estimating driving performance based on EEG spectrum analysis. EURASIP J. Appl. Signal Process. 3165–3174 (2005)

    Google Scholar 

  5. Hosseini, S.A., Khalilzadeh, M.A., Changiz, S.: Emotional stress recognition system for affective computing based on bio-signals. J. Biol. Syst. 18, 101–114 (2010)

    CrossRef  Google Scholar 

  6. Hope, R.M., Wang, Z., Wang, Z., Ji, Q., Gray, W.D.: Workload classification across subjects using EEG. Proc. Hum. Factors Ergon. Soc. Annu. Meet. 55, 202–206 (2011)

    Google Scholar 

  7. Wilson, G.F.: An analysis of mental workload in pilots during flight using multiple psychophysiological measures. Int. J. Aviat. Psychol. 12, 3–18 (2002)

    CrossRef  Google Scholar 

  8. Kothe, C.A., Makeig, S.: Estimation of task workload from EEG data: new and current tools and perspectives. In: Presented at the Annual International Conference of the IEEE Engineering in Medicine and Biology Society 30 September 2011

    Google Scholar 

  9. Duta, M., Alford, C., Wilson, S., Tarassenko, L.: Neural network analysis of the mastoid EEG for the assessment of vigilance. Int. J. Hum-Comput. Interact. 17, 171–195 (2004)

    CrossRef  Google Scholar 

  10. Hord, D.J.: An EEG predictor of performance decrement in a vigilance task (1982)

    Google Scholar 

  11. St John, M., Risser, M.R., Kobus, D.A.: Toward a usable closed-loop attention management system: predicting vigilance from minimal contact head, eye, and EEG measures. Found. Augment Cogn 12–18 (2006)

    Google Scholar 

  12. Gerson, A.D., Parra, L.C., Sajda, P.: Cortically coupled computer vision for rapid image search. IEEE Trans. Neural Syst. Rehabil. Eng. 14, 174–179 (2006)

    Google Scholar 

  13. Marathe, A.R., Ries, A.J., McDowell, K.: Sliding HDCA: single-trial EEG classification to overcome and quantify temporal variability. IEEE Trans. Neural Syst. Rehabil. Eng. 22, 201–211 (2014)

    CrossRef  Google Scholar 

  14. Khaleghi, B., Khamis, A., Karray, F.O., Razavi, S.N.: Multisensor data fusion: a review of the state-of-the-art. Inf. Fusion. 14, 28–44 (2013)

    CrossRef  Google Scholar 

  15. Kwon, J., Lee, K.M.: Tracking by sampling trackers. In: IEEE International Conference on Computer Vision (ICCV 2011), IEEE, 1195–1202 (2011)

    Google Scholar 

  16. Wu, S., Bondugula, S., Luisier, F., Zhuang, X., Natarajan, P.: Zero-shot event detection using multi-modal fusion of weakly supervised concepts. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2665–2672 (2014)

    Google Scholar 

  17. Kim, T., Lee, H., Lee, K.: Optical flow via locally adaptive fusion of complementary data costs. In: Proceedings of the IEEE International Conference on Computer Vision, 3344–3351 (2013)

    Google Scholar 

  18. Liu, C., Yuen, J., Torralba, A.: Sift flow: dense correspondence across scenes and its applications. Pattern Anal. Mach. Intell. IEEE. Trans. 33, 978–994 (2011)

    Google Scholar 

  19. Dempster, A.P.: Upper and lower probabilities induced by a multivalued mapping. Ann. Math. Stat. 325–339 (1967)

    Google Scholar 

  20. Shafer, G., others: A mathematical theory of evidence. Princeton University Press Princeton (1976)

    Google Scholar 

  21. Lee, H., Kwon, H., Robinson, R.M., Nothwang, W. d, Marathe, A.M.: Dynamic belief fusion for object detection. ArXiv Prepr. ArXiv151103183. (2015)

    Google Scholar 

  22. Pascal, B., Krailsheimer, A.J.: Pensees: Translated with an introduction by AJ Krailsheimer. Penguin (1968)

    Google Scholar 

  23. Bernoulli, D.: Exposition of a new theory on the measurement of risk. Econom. J. .Econom. Soc. 23–36 (1954)

    Google Scholar 

  24. Lehmann, E.L.: Some principles of the theory of testing hypotheses. Springer (2012)

    Google Scholar 

  25. Olson, E., Strom, J., Goeddel, R., Morton, R., Ranganathan, P., Richardson, A.: Exploration and mapping with autonomous robot teams. Commun ACM 56, 62–70 (2013)

    CrossRef  Google Scholar 

  26. Tsiligkaridis, T., Sadler, B., Hero, A.: Collaborative 20 questions for target localization. IEEE Trans. Inf. Theory. 60, 2233–2252 (2014)

    MathSciNet  CrossRef  Google Scholar 

  27. Christensen, J.C., Estepp, J.R., Wilson, G.F., Russell, C.A.: The effects of day-to-day variability of physiological data on operator functional state classification. NeuroImage 59, 57–63 (2012)

    CrossRef  Google Scholar 

  28. Ratcliff, R., Philiastides, M.G., Sajda, P.: Quality of evidence for perceptual decision making is indexed by trial-to-trial variability of the EEG. Proc. Natl. Acad. Sci. 106, 6539–6544 (2009)

    CrossRef  Google Scholar 

  29. McDowell, K., Lin, C.-T., Oie, K.S., Jung, T.-P., Gordon, S., Whitaker, K.W., Li, S.-Y., Lu, S.-W., Hairston, W.D.: Real-world neuroimaging technologies. IEEE Access. 1, 131–149 (2013)

    CrossRef  Google Scholar 

  30. Parasuraman, R., Wickens, C.D.: Humans: still vital after all these years of automation. Hum. Factors J. Hum. Factors Ergon. Soc. 50, 511–520 (2008)

    Google Scholar 

  31. Fong, T., Thorpe, C., Baur, C.: Multi-robot remote driving with collaborative control. IEEE. Trans. Ind. Electron. 50, 699–704 (2003)

    CrossRef  Google Scholar 

  32. Fong, T., Thorpe, C., Baur, C.: Robot, asker of questions. Robot. Auton. Syst. 42, 235–243 (2003)

    CrossRef  MATH  Google Scholar 

  33. Hayati, S., Venkataraman, S.: Design and implementation of a robot control system with traded and shared control capability. In: IEEE International Conference on Robotics and Automation, IEEE 1310–1315 (1989)

    Google Scholar 

  34. Sellner, B., Simmons, R., Singh, S.: User modelling for principled sliding autonomy in human-robot teams. In: Multi-Robot Systems. From Swarms to Intelligent Automata Vol. III, pp. 197–208. Springer (2005)

    Google Scholar 

  35. Sajda, P., Pohlmeyer, E., Wang, J., Parra, L.C., Christoforou, C., Dmochowski, J., Hanna, B., Bahlmann, C., Singh, M.K., Chang, S.-F.: In a Blink of an eye and a switch of a transistor: cortically coupled computer vision. Proc. IEEE. 98, 462–478 (2010)

    CrossRef  Google Scholar 

  36. Huang, Y., Erdogmus, D., Mathan, S., Pavel, M.: A Fusion approach for image triage using single trial erp detection. In: 3rd International IEEE/EMBS Conference on Neural Engineering CNE ’07, 473–476 (2007)

    Google Scholar 

  37. Marathe, A.R., Lance, B.J., Nothwang, W., Metcalfe, J.S., McDowell, K.: Confidence metrics improve human-autonomy integration. In: Presented at the Human Robot Interaction, Bielefield, Germany 3 March 2014

    Google Scholar 

  38. Marathe, A.R., Ries, A.J., Lawhern, V.J., Lance, B.J., Touryan, J., McDowell, K., Cecotti, H.: The effect of target and non-target similarity on neural classification performance: a boost from confidence. Front. Neurosci. 9, 270 (2015)

    CrossRef  Google Scholar 

  39. Touryan, J., Apker, G., Kerick, S., Lance, B., Ries, A.J., McDowell, K.: Translation of EEG-based performance prediction models to rapid serial visual presentation tasks. In: Foundations of Augmented Cognition. 521–530. Springer (2013)

    Google Scholar 

  40. Oie, K.S., Gordon, S.M., McDowell, K.: The multi-aspect measurement approach: rationale, technologies, tools, and challenges for systems design. In: Martin, J., Lockett, J.I., Allender, L.E., Savage-Knepshield, P. (eds.) Designing soldier systems: current issues in human factors. Ashgate, Burlington, VT (2013)

    Google Scholar 

  41. Settles, B.: Active learning literature survey. Univ. Wis. Madison. 52, 11 (2010)

    Google Scholar 

  42. Zhu, X.: Semi-supervised learning literature survey (2005)

    Google Scholar 

  43. Joshi, A.J., Porikli, F., Papanikolopoulos, N.P.: Scalable active learning for multiclass image classification. Pattern Anal. Mach. Intell. IEEE. Trans. 34, 2259–2273 (2012)

    Google Scholar 

  44. Tong, S., Koller, D.: Support vector machine active learning with applications to text classification. J. Mach. Learn Res. 2, 45–66 (2002)

    MATH  Google Scholar 

  45. Marathe, A., Lawhern, V., Wu, D., Slayback, D., Lance, B.: Improved neural signal classification in a rapid serial visual presentation task using active learning. IEEE Trans. Neural Syst. Rehabil. Eng. 1–1 (2015)

    Google Scholar 

  46. Wu, D., Lance, B.J., Parsons, T.D.: Collaborative filtering for brain-computer interaction using transfer learning and active class selection. PLoS ONE. 8, e56624 (2013)

    CrossRef  Google Scholar 

  47. Wu, D., Lance, B., Lawhern, V.: Transfer learning and active transfer learning for reducing calibration data in single-trial classification of visually-evoked potentials. In: IEEE International Conference on Systems, Man and Cybernetics (SMC 2014), IEEE, 2801–2807 (2014)

    Google Scholar 

  48. Gordon, S.M., McDaniel, J.R., Metcalfe, J.S., Passaro, A.D.: Using behavioral information to contextualize BCI performance. In: Foundations of Augmented Cognition. 211–220, Springer (2015)

    Google Scholar 

  49. Metcalfe, J.S., Gordon, S.M., Passaro, A.D., Kellihan, B., Oie, K.S.: Towards a translational method for studying the influence of motivational and affective variables on performance during human-computer interactions. In: Foundations of Augmented Cognition. 63–72, Springer (2015)

    Google Scholar 

Download references

Acknowledgments

This project was supported by the Office of the Secretary of Defense ARPI program MIPR DWAM31168 and by US Army Research Laboratory’s Cognition and Neuroergonomics/Collaborative Technology Alliance #W911NF-10-2-0022. The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of the Army Research Laboratory or U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation herein.

Author information

Authors and Affiliations

  1. Human Research and Engineering Directorate, U.S. Army Research Laboratory, Aberdeen Proving Ground, Adelphi, MD, 21005, USA

    Amar R. Marathe, Jonathan R. McDaniel, Stephen M. Gordon & Kaleb McDowell

  2. DCS Corporation, Alexandria, VA, USA

    Jonathan R. McDaniel & Stephen M. Gordon

Authors
  1. Amar R. Marathe
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jonathan R. McDaniel
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Stephen M. Gordon
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Kaleb McDowell
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Amar R. Marathe .

Editor information

Editors and Affiliations

  1. U.S. Army Research Laboratory, Aberdeen Proving Ground, USA

    Pamela Savage-Knepshield

  2. U.S. Army Research Laboratory, Aberdeen Proving Ground, USA

    Jessie Chen

Rights and permissions

Reprints and Permissions

Copyright information

© 2017 Springer International Publishing Switzerland

About this paper

Cite this paper

Marathe, A.R., McDaniel, J.R., Gordon, S.M., McDowell, K. (2017). Confidence-Based State Estimation: A Novel Tool for Test and Evaluation of Human-Systems. In: Savage-Knepshield, P., Chen, J. (eds) Advances in Human Factors in Robots and Unmanned Systems. Advances in Intelligent Systems and Computing, vol 499. Springer, Cham. https://doi.org/10.1007/978-3-319-41959-6_24

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-41959-6_24

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-41958-9

  • Online ISBN: 978-3-319-41959-6

  • eBook Packages: EngineeringEngineering (R0)