Skip to main content
SpringerLink
Log in

The Development and Evaluation of ELI, an Interactive Elicitation Technique for Subjective Probability Distributions

  • Conference paper
Item Banking: Interactive Testing and Self-Assessment

Part of the book series: NATO ASI Series ((NATO ASI F,volume 112))

Abstract

Uncertain knowledge about continuous quantities is usually formalized through Subjective Probability Distributions (SPD’s). Past research showed that the quality of SPD’s is rather poor and that poor SPD quality might originate from method-induced biases. It is quite possible that equipped with more appropriate elicitation tools, assessors will prove to be more competent probability estimators than research thus far suggested. The present paper describes the search for a new elicitation methodology. The most important feature of the new ELIcitation technique ELI is the direct realization of a proper scoring rule in a graphically oriented interactive Computer program. An evaluation study in which ELI was compared with other elicitation techniques showed that ELI Performance was superior.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

Similar content being viewed by others

References

  • Alpert, M.,& Raiffa, H. (1982) A progress report on the training of probability assessors.In D. Kahneman, P. Slovic, & A. Tversky (Eds.) Judgment under uncertainty: .Heuristics and biases (pp. 294–305). Cambridge: UniversityPress.

    Google Scholar 

  • Cooke, R.M.,Mendel, M., & Thijs, W. (1988) Calibration and Information in expert resolution; a classical approach. Automatica, 24(1),87–94.

    Article  Google Scholar 

  • Fischer,G.W. (1982) Scoring-rule feedback and the overconfidence Syndrome in subjective probability forecasting. OrganizationalBehavior and Human Performance, 29,352–369.

    Article  Google Scholar 

  • Hofstee,W.K.B. (1987, December) Overconfidence.(Available from Willem K.B. Hofstee, Department of Personality Psychology, Faculty ofBehavioral and Social Sciences, GroteKruisstraat 2/1, 9712 TS Groningen, The Netherlands.)

    Google Scholar 

  • Hogarth,R.M. (1975) Cognitive processes and the assessment of subjective probability distributions. Journal of the American StatisticalAssociation, 70,271–289.

    Article  MATH  Google Scholar 

  • Hogarth,R.M. (1980) Judgement and choice: The psychology ofdecision.Chichester: Wiley.

    Google Scholar 

  • Huber, G.P.(1974) Methods for quantifying subjective probabilities and multi-attribute Utilities.Decision Sciences, 5,430–458.

    Article  Google Scholar 

  • Kahneman,D., Slovic, P., & Tversky, A. (Eds.) (1982) Judgment under uncertainty: Heuristicsand biases.Cambridge:University Press.

    Google Scholar 

  • Koriat, A.,Lichtenstein, S., & Fischhoff, B. (1980) Reasons for confidence. Journalof ExperimentalPsychology: Human Learning and Memory,6(2), 107–118.

    Article  Google Scholar 

  • Lichtenstein,S., Fischhoff, B., & Phillips, L.D. (1982) Calibration of probabilities: The State of the art to 1980. In D. Kahneman, P. Slovic,& A. Tversky (Eds.) Judgment underuncertainty: Heuristics and biases(pp.306–334). Cambridge: University Press.

    Google Scholar 

  • Lourens,P.F. (1984) The formalization ofknowledge by specification of subjective probability distributions.Dissertation, University of Groningen, Groningen.

    Google Scholar 

  • Ludke, R.L.,Stauss, F.F., & Gustafson, D.H. (1977) Comparison of five methods for estimating subjective probability distributions. OrganizationalBehavior and Human Performance, 19,162–179.

    Article  Google Scholar 

  • Moskowitz, H.& Bullers, W.I. (1979) Modified PERT versus fractile assessments of subjective probability distributions. OrganizationalBehavior and HumanPerformance, 24,167–194.

    Article  Google Scholar 

  • Murphy,A.H., & Winkler, R.L. (1970) Scoring rules in probability assessment and evaluation.Acta Psychologica, 34,273–286.

    Article  Google Scholar 

  • Novick, M.R. & Jackson, P.H.(1974) Statistical methodsfor educational and psychologicalresearch.NewYork: McGraw-Hill.

    Google Scholar 

  • Pickhardt,R.C. & Wallace, J.B. (1977) A study of the Performance of subjective probabilityassessors. Decision Sciences, 5,347–363.

    Article  Google Scholar 

  • Ronis, D.L.& Yates, J.F. (1987) Components of probability judgment accuracy: individual consistency and effects of subject matter andassessment method. Organizational Behavior and HumanDecision Processes, 40,193–218.

    Article  Google Scholar 

  • Schaefer,R.E. (1976) The evaluation of individual and aggregated subjective probability distributions. Organizational Behavior andHuman Performance, 17, 199–210.

    Article  Google Scholar 

  • Schütt, K.-P.(1981) Wahrscheinlichkeitsschätzungen im Computer-Dialog [Probability estimates with Computer dialogue].Stuttgart: Poeschel Verlag.

    Google Scholar 

  • Seaver,D.A., Von Winterfeldt, D., & Edwards, W. (1978) Eliciting subjective probability distributions on continuous variables. OrganizationalBehavior and HumanPerformance, 21,379–391.

    Article  Google Scholar 

  • Spetzler, C.S.,&Staël von Holstein, C.-A.S. (1975)Probability encoding in decision analysis. Management Science,22(3), 340–358.

    Article  Google Scholar 

  • Stael vonHolstein, C.-A. S. (1970) Measurement of subjective probability. Acta .Psychologica,34,146–159.

    Article  Google Scholar 

  • Stael vonHolstein, C.-A. S. (1971) Two techniques for assessment of subjective probability distributions. Acta Psychologica, 35,478–494.

    Article  Google Scholar 

  • Terlouw, P.(1989) Subjective probability distributions a psychometric approach. DissertationUniversity of Groningen, Groningen.

    Google Scholar 

  • Van Lenthe,J. (in press) ELI: an interactive elicitation technique for subjective probability distributions. Organizational Behavior andHuman Decision Processes.

    Google Scholar 

  • Van Lenthe, J. (1992) Ablueprint of ELI, a new method for eliciting subjective probability distributions.Manuscript submitted for publication.

    Google Scholar 

  • Van Naerssen,R.F. (1962) A scale for the measurement of subjective probability. Acta Psychologica,20,159–166.

    Article  Google Scholar 

  • Van Steen,J.F.J., & Oortman Gerlings, P.D. (1988) Het gebruik van expertmeningen in veiligheidsstudies [The use of expert opinion insafety studies].Internal report, TU Delft/TNO.

    Google Scholar 

  • VonWinterfeldt, D., & Edwards, W. (1986) Decision analysis and behavioral research.Cambridge:University Press.

    Google Scholar 

  • Wallsten,T.S., & Budescu, D.V. (1983) Encoding subjective probabilities: A psychological and psychometric review. ManagementSciences, 29(2),51–173.

    Google Scholar 

  • Winkler,R.L. (1971) Probabilistic prediction: some experimental results. Journal ofthe AmericanStatistical Society,66(336), 675–685.

    Google Scholar 

  • Winkler,R.L. (1986) On “good probability appraisers”. In P. Goel & A.Zellner (Eds.)Bayesian inference and decision techniques(pp. 265–278). Amsterdam: Elsevier.

    Google Scholar 

  • Wright, G.,& Wisudha, A. (1982) Distribution of probability assessments for almanac and fiiture event questions. Scandinavian Journal ofPsychology, 23,219–224.

    Article  Google Scholar 

  • Yates, J.F., Zhu, Y., Ronis, D.L.,Wang, D.-F., Shinotsuka, H., & Toda, M. (1989) Probability judgment accuracy: China, Japan, and theUnited States. Organizational Behavior andHuman Decision Processes, 43,145–171.

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Statistics and Measurement Theory, Faculty of Behavioral and Social Sciences, University of Groningen, Grote Kruisstraat 2/1, 9712 TS, Groningen, The Netherlands

    Jelle van Lenthe

Authors
  1. Jelle van Lenthe
    View author publications

    You can also search for this author in PubMed Google Scholar

Editor information

Editors and Affiliations

  1. Service de Technologie de l’Education, Université de Liège au Sart Tilman Boulevard du Rectorat, 5 Bât, B32 Sart Tilman, B-4000, Liège, Belgium

    Dieudonnè A. Leclercq

  2. Graduate School of Education, University of California, 405 Hilgard Avenue, Los Angeles, CA, 90024-1321, USA

    James E. Bruno

Rights and permissions

Reprints and permissions

Copyright information

© 1993 Springer-Verlag Berlin Heidelberg

About this paper

Cite this paper

van Lenthe, J. (1993). The Development and Evaluation of ELI, an Interactive Elicitation Technique for Subjective Probability Distributions. In: Leclercq, D.A., Bruno, J.E. (eds) Item Banking: Interactive Testing and Self-Assessment. NATO ASI Series, vol 112. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-58033-8_12

Download citation

  • DOI: https://doi.org/10.1007/978-3-642-58033-8_12

  • Publisher Name: Springer, Berlin, Heidelberg

  • Print ISBN: 978-3-642-63444-4

  • Online ISBN: 978-3-642-58033-8

  • eBook Packages: Springer Book Archive

Publish with us

Policies and ethics

Search

Navigation