Abstract
In this paper, we describe a framework for comparing and selecting inference enterprise models. An inference enterprise is an organizational entity that uses data, tools, people, and processes to make mission-focused inferences. Intuitively, organizations could organize around one of several inference enterprise models to make the same inference. To address the inference enterprise model selection problem, we combine multi-inference enterprise modeling, model-based validation, and statistical inference to rank order inference enterprise candidates. Inference enterprise multi-modeling affords us the opportunity to simulate representative data set to the organization’s mission. Model-based validation employs normative decision theory to score empirical results using a utility function, and statistical inference allows us to generalize candidate rank ordering. We demonstrate the framework described in this paper and compare expected utility-based rank ordering with rank ordering based on expected F1 score. Using generic performance metrics such as F1 potentially has adverse impacts to an organization’s mission.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Abbreviations
- IARPA:
-
Intelligence Advanced Research Projects Activity
- IE:
-
Inference enterprise
- IEM:
-
Inference enterprise model
- MBV:
-
Model-based validation
- NB:
-
Naïve Bayesian network classifier
- RF:
-
Random forest classifier
- SCITE:
-
Scientific Advances to Continuous Insider Threat Evaluation
- TN:
-
True negative count
- TP:
-
True positive count
- VM:
-
Voting machine
- vNM:
-
von Neumann-Morgenstern
References
Huang, E., Zaidi, A. K., & Laskey, K. B. (2018). Inference enterprise multimodeling for insider threat detection systems. In A. M. Madni, B. Boehm, R. G. Ghanem, D. Erwin, & M. J. Wheaton (Eds.), Disciplinary convergence in systems engineering research (pp. 175–186). Cham: Springer International Publishing.
Turcotte, M. J., Kent, A. D., & Hash, C. (2017). Unified host and network data set. arXiv preprint arXiv:170807518.
Frey, D. D., & Dym, C. L. (2006). Validation of design methods: lessons from medicine. Research in Engineering Design, 17, 45–57.
Frey, D., & Li, X. (2006). Model-based validation of design methods. In K. E. Lewis, W. Chen, & L. C. Schmidt (Eds.), Decision making in engineering design (pp. 315–323). New York: ASME Press.
Nelsen, R. B. (2007). An introduction to copulas. Berlin: Springer Science & Business Media.
Von Neumann, J., & Morgenstern, O. (2007). Theory of games and economic behavior (60th Anniversary ed). Princeton, NJ: Princeton University Press.
Frey, D., Herder, P., Wijnia, Y., Subrahmanian, E., Katsikopoulos, K., & Clausing, D. (2009). The Pugh Controlled Convergence method: model-based evaluation and implications for design theory. Research in Engineering Design, 20, 41–58.
Domingos, P. (1999). Metacost: A general method for making classifiers cost-sensitive. In Proceedings of the Fifth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (pp. 155–164). New York: ACM.
Elkan, C. (2001). The foundations of cost-sensitive learning. In International Joint Conference on Artificial Intelligence (pp. 973–978). Mahwah, NJ: Lawrence Erlbaum Associates Ltd.
Pedregosa, F., Varoquaux, G., Gramfort, A., et al. (2011). Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12, 2825–2830.
Sokolova, M., & Lapalme, G. (2009). A systematic analysis of performance measures for classification tasks. Information Processing & Management., 45, 427–437.
Hadar, J., & Russell, W. R. (1969). Rules for ordering uncertain prospects. The American Economic Review, 59(1), 25–34.
Goutte, C., & Gaussier, E. (2005). A probabilistic interpretation of precision, recall and F-score, with implication for evaluation. In European Conference on Information Retrieval (pp. 345–359). Berlin: Springer.
Acknowledgment
Research reported here was supported under IARPA contract 2016-16031400006. The content is solely the responsibility of the authors and does not necessarily represent the official views of the US government.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2019 Springer Nature Switzerland AG
About this paper
Cite this paper
Vermillion, S.D., Brown, D.P., Buede, D.M. (2019). Applying Model-Based Validation to Inference Enterprise System Architecture Selection. In: Adams, S., Beling, P., Lambert, J., Scherer, W., Fleming, C. (eds) Systems Engineering in Context. Springer, Cham. https://doi.org/10.1007/978-3-030-00114-8_28
Download citation
DOI: https://doi.org/10.1007/978-3-030-00114-8_28
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-00113-1
Online ISBN: 978-3-030-00114-8
eBook Packages: Intelligent Technologies and RoboticsIntelligent Technologies and Robotics (R0)