Abstract
The current study explores the validity of verbal probability assessments in a sequential and highly ambiguous task, that is, one in which it is virtually impossible to know or learn about the true probabilities of possible outcomes. Participants observed the pre-defined motion of an unmanned aerial vehicle (UAV), such that the participant’s success depended on the UAV reaching a target sector without being spotted by an opponent UAV. At several points in each trajectory, participants’ task was to evaluate the likelihood of reaching the target successfully. The study utilized a 2 × 2 independent-groups factorial design to examine the effect of probability incentivization (Brier vs none), in which participants receive payment based on the nearness of their predictions to actual outcomes, and informational reviews (present vs absent), in which participants engage in detailed discussion with the experimenter, regarding their assessments in seven previous trials before continuing, on probability assessment. A statistically significant main effect of Brier scoring was found, such that Brier based incentivization improved assessment accuracy. The effect of informational review and the interaction effect were not significant. All groups performed significantly better than random and uninformed performance. Outcomes from this study improve our understanding of the validity of online judgments made by operators of unmanned vehicles in strategic settings. It is concluded that non-expert probability assessments carry important information value even in ambiguous settings and even without incentives, and importantly, are further amenable to incentives and training.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Notes
These characteristics may make the information value relatively easy to disregard; however, it would be a mistake to disregard valid sources of information (the participants’ judgments) due to a lack of certainty about the underlying cues that produce this information value.
References
Arnold, V., Clark, N., Collier, P. A., Leech, S. A., & Sutton, S. G. (2006). The differential use and effect of knowledge-based system explanations in novice and expert judgment decisions. Mis Quarterly, 79–97.
Azar P, Chen J, & Micali, S. (2012). Crowdsourced Bayesian auctions. In Proceedings of the 3rd Innovations in Theoretical Computer Science Conference (ITCS ′12). ACM, New York, NY, USA, 236–248.
Bambini, D., Washburn, J., & Perkins, R. (2009). Outcomes of clinical simulation for novice nursing students: communication, confidence, clinical judgment. Nursing Education Perspectives, 30(2), 79–82.
Benedetti, R. (2010). Scoring rules for forecast verification. Monthly Weather Review, 138, 203–211.
Brier, G. W. (1950). Verification of forecasts expressed in terms of probability. Monthly Weather Review, 78(1), 1–3.
Brocker, J., & Smith, L. A. (2007). Scoring probabilistic forecast: The importance of being proper. Weather Forecasting, 22, 382–388.
Brümmer, N., & van Leeuwen, D. A. (2006). On calibration of language recognition scores. Proceedings of Odyssey Speaker and Language Recognition Workshop, ISCA, San Juan.
Camerer, C. F., & Johnson, E. J. (1997). 10 The process-performance paradox in expert judgment: How can experts know so much and predict so badly?. Research on judgment and decision making: Currents, connections, and controversies, 342.
Casati, B., & Wilson, L. J. (2007). A new spatial-scale decomposition of the Brier score: application to the verification of lightning probability forecasts. Monthly Weather Review, 135(9), 3052–3069.
Casillas-Olvera, G., & Bessler, D. A. (2006). Probability forecasting and central bank accountability. Journal of Policy Modeling, 28(2), 223–234.
Ceren, R., Doshi, P., Meisel, M., Goodie, A., & Hall, D. (2013). On modeling human learning in sequential games with delayed reinforcements. In Proceedings of the IEEE Systems, Man and Cybernetics Conference (SMC), Manchester, UK, pp. 3108–3113.
Craik, F. I., & Tulving, E. (1975). Depth of processing and the retention of words in episodic memory. Journal of Experimental Psychology: General, 104(3), 268.
DeGroot, M. H., & Fienberg, S. E. (1983). The comparison and evaluation of forecasters. The statistician, 12–22.
Deitch, E. L. (2001). Learning to land: A qualitative examination of pre-flight and in-flight decision-making processes in expert and novice aviators (Doctoral dissertation, Virginia Polytechnic Institute and State University).
Fantino, E. (1998). Behavior analysis and decision making. Journal of the Experimental Analysis of Behavior, 69(3), 355–364.
Fox, C. R., & Tversky, A. (1995). Ambiguity aversion and comparative ignorance. Quarterly Journal of Economics, 110(3), 585–603.
Gerds, T. A., Cai, T., & Schumacher, M. (2008). The Performance of Risk Prediction Models. Biomedical Journal, 50(4), 457–479.
Gneiting, T., & Raftery, A. E. (2007). Strictly proper scoring rules, prediction, and estimation. Journal of the American Statistical Association, 102(477), 359–378.
Good, I. J. (1952). Rational decisions. Journal of the Royal Statistical Society, 14A, 107–114.
Goodie, A. S. (1997). Direct experience is ecologically valid. Behavioral and Brain Sciences, 20, 777–778.
Goodie, A. S. (2001). Are scripts or deception necessary when repeated trials are used?: On the social context of psychological experiments. Behavioral and Brain Sciences, 24, 412.
Hertwig, R., & Ortmann, A. (2001). Experimental practices in economics: A methodological challenge for psychologists? Behavioral and Brain Sciences, 24, 383–451.
Jackson, J. L., Michon, J. A., Boonstra, H., De Jonge, D., & De Velde Harsenhorst, J. (1986). The effect of depth of processing on temporal judgment tasks. Acta Psychologica, 62(3), 199–210.
Jansen, C. J. M., & Pollmann, M. M. W. (2001). On round numbers: Pragmatic aspects of numerical expressions. Journal of Quantitative Linguistics, 8, 187–201.
Kee, F., Jenkins, J., McIlwaine, S., Patterson, C., Harper, S., & Shields, M. (2003). Fast and frugal models of clinical judgment in novice and expert physicians. Medical Decision Making, 23(4), 293–300.
Kleinjans, K. J., & Van Soest, A. (2010). Nonresponse and focal point answers to subjective probability questions (No. 5272). Discussion paper series//Forschungsinstitut zur Zukunft der Arbeit.
McLaughlin, F. (1980). Probability assessments and performance in business game simulations. Experimental Learning Enters the Eighties, 7, 121–122.
Meehl, P. E. (1954). Clinical versus statistical prediction: A theoretical analysis and a review of the evidence. Minneapolis: University of Minnesota Press.
Neter, J., Wasserman, W., & Kutner, M. H. (1996). Applied linear statistical models (Vol. 4). Chicago: Irwin.
Poses, R. M., Cebul, R. D., & Centor, R. M. (1988). Eualuating Physicians’ Probabilistic Judgments. Medical Decision Making, 8(4), 233–240.
Roulston, M. S., & Smith, L. A. (2002). Evaluating probabilistic forecasts using information theory. Monthly Weather Review, 130(6), 1653–1660.
Rowe, E. J. (1974). Depth of processing in a frequency judgment task. Journal of Verbal Learning and Verbal Behavior, 13(6), 638–643.
Tversky, A., & Kahneman, D. (1983). Extensional versus intuitive reasoning: The conjunction fallacy in probability judgment. Psychological Review, 90(4), 293.
Williams, R. J., & Connolly, D. (2006). Does learning about the mathematics of gambling change gambling behavior? Psychology of Addictive Behaviors, 20, 62–68.
Winkler, R. L., Muñoz, J., Cervera, J. L., Bernardo, J. M., Blattenberger, G., Kadane, J. B., Lindley, D. V., Murphy, A. H., Oliver, R. M., & Ríos-Insua, D. (1996). Scoring rules and the evaluation of probabilities. Test, 5(1), 1–60.
Yates, J. F. (1982). External correspondence: decompositions of the mean probability score. Organizational Behavior and Human Performance, 30(1), 132–156.
Acknowledgments
This research was supported by grant 55749NS from the Army Research Office.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Goodie, A.S., Meisel, M.K., Ceren, R. et al. Evaluating and Improving Probability Assessment in an Ambiguous, Sequential Environment. Curr Psychol 35, 667–673 (2016). https://doi.org/10.1007/s12144-015-9335-9
Published:
Issue Date:
DOI: https://doi.org/10.1007/s12144-015-9335-9