Validation Metrics: A Case for Pattern-Based Methods
This chapter discusses the issue of choosing the best computer model for simulating a real-world phenomenon through the process of validating the model’s output against the historical, real-world data. Four families of techniques are discussed that are used in the context of validation. One is based on the comparison of statistical summaries of the historical data and the model output. The second is used where the models and data are stochastic, and distributions of variables must be compared, and a metric is used to measure their closeness. After exploring the desirable properties of such a measure, the paper compares the third and fourth methods (from information theory) of measuring closeness of patterns, using an example from strategic market competition. The techniques can, however, be used for validating computer models in any domain.
KeywordsModel validation State Similarity Measure Area Validation Metric Generalized Hartley metric
I should like to thank Dan MacKinlay for his mention of the K-L information loss measure, Arthur Ramer for his mention of the Hartley or U-uncertainty metric and his suggestions, and Vessela Daskalova for her mention of the “cityblock” metric. The efforts of the editors of this volume and anonymous referees were very constructive, and have greatly improved this chapter’s presentation.
- Akaike, H. (1973). Information theory as an extension of the maximum likelihood principle. In B. N. Petrov, & F. Csaki (Eds.), Second International Symposium on Information Theory (pp. 267–281). Budapest: Akademiai Kiado.Google Scholar
- Fagiolo, G., Guerini, M., Lamperti, F., Moneta, A., & Roventini, A. (2019). Validation of agent-based models in economics and finance. pp. 763–787.Google Scholar
- Gilbert, N., & Troitzsch, K. G. (2005). Simulation for the social scientist (2nd ed.). Open University Press.Google Scholar
- Krause, E. F. (1986). Taxicab geometry: An adventure in non-euclidean geometry, New York: Dover. (First published by Addison-Wesley in 1975.)Google Scholar
- Liu Y., Chen W., Arendt P., & Huang H.-Z. (2010). Towards a better understanding of model validation metrics. In 13th AIAA/ISSMO Multidisciplinary Analysis Optimization Conference, Multidisciplinary Analysis Optimization Conferences.Google Scholar
- Mankin, J. B., O’Neill, R. V., Shugart, H. H., & Rust, B. W. (1977). The importance of validation in ecosystem analysis. In G. S. Innis (Ed.), New Directions in the Analysis of Ecological Systems, Part 1, Simulation Council Proceedings Series, Simulation Councils, La Jolla, California (Vol. 5, pp. 63–71). Reprinted. In H. H. Shugart & R. V. O’Neill (Eds.), Systems ecology (pp. 309–317). Hutchinson and Ross, Stroudsburg, Pennsylvania: Dowden.Google Scholar
- Marks, R. E. (2010). Comparing two sets of time-series: The state similarity measure. In V. A. Alexandria (Ed.), 2010 Joint Statistical Meetings Proceedings-Statistics: A Key to Innovation in a Data-centric World, Statistical Computing Section (pp. 539–551). American Statistical Association.Google Scholar
- Marks, R. E. (2013). Validation and model selection: Three similarity measures compared. Complexity Economics, 2(1), 41–61. http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.401.6982&rep=rep1&type=pdf.
- Marks, R. E. (2016). Monte Carlo. In D. Teece, & M. Augier (Eds.), The palgrave encyclopedia of strategic management. London: Palgrave.Google Scholar
- Marks, R. E., Midgley, D. F., & Cooper, L. G. (1995). Adaptive behavior in an oligopoly. In J. Biethahn, & V. Nissen (Eds.), Evolutionary algorithms in management applications (pp. 225–239). Berlin: Springer.Google Scholar
- Midgley, D. F., Marks, R. E., & Cooper, L. G. (1997). Breeding competitive strategies. Management Science, 43(3), 257–275.Google Scholar
- Midgley, D. F., Marks, R. E., & Kunchamwar, D. (2007). The building and assurance of agent-based models: An example and challenge to the field. Journal of Business Research, 60(8), 884–893. (Special Issue: Complexities in Markets).Google Scholar
- Ramer, A. (1989). Conditional possibility measures. International Journal of Cybernetics and Systems, 20, 233–247. Reprinted in D. Dubois, H. Prade, & R. R. Yager, (Eds.). (1993). Readings in fuzzy sets for intelligent systems (pp. 233–240). San Mateo, California: Morgan Kaufmann Publishers.Google Scholar
- Rényi, A. (1970). Probability theory. Amsterdam: North-Holland (Chapter 9, Introduction to information theory, pp. 540–616).Google Scholar