Talent Spotting in Crowd Prediction

Abstract

Who is good at prediction? Addressing this question is key to recruiting and cultivating accurate crowds and effectively aggregating their judgments. Recent research on superforecasting has demonstrated the importance of individual, persistent skill in crowd prediction. This chapter takes stock of skill identification measures in probability estimation tasks, and complements the review with original analyses, comparing such measures directly within the same dataset. We classify all measures in five broad categories: (1) accuracy-related measures, such as proper scores, model-based estimates of accuracy and excess volatility scores; (2) intersubjective measures, including proxy, surrogate and similarity scores; (3) forecasting behaviors, including activity, belief updating, extremity, coherence, and linguistic properties of rationales; (4) dispositional measures of fluid intelligence, cognitive reflection, numeracy, personality and thinking styles; and (5) measures of expertise, including demonstrated knowledge, confidence calibration, biographical, and self-rated expertise. Among non-accuracy-related measures, we report a median correlation coefficient with outcomes of r = 0.20. In the absence of accuracy data, we find that intersubjective and behavioral measures are most strongly correlated with forecasting accuracy. These results hold in a LASSO machine-learning model with automated variable selection. Two focal applications provide context for these assessments: long-term, existential risk prediction and corporate forecasting tournaments.

Appendix: Methodological Details of Selected Predictors

1.1 Item Response Theory Models

In forecasting, one such confounder is the timing in which forecasts are made. In forecasting tournaments, forecasters make many forecasts about the same problems at various time points. Those who forecast problems closer to their resolution date have an accuracy advantage which may be important to account for in assessing their talent level (for more detail, see Himmelstein et al., this issue). IRT models can be extended so that their diagnostic properties change relative to the time point at which a forecaster makes their forecast. One such model is given below (Himmelstein et al., 2021; Merkle et al., 2016).

$$ \boldsymbol{N}{\boldsymbol{B}}_{\boldsymbol{f},\boldsymbol{q},\boldsymbol{d}}={\boldsymbol{b}}_{\mathbf{0},\boldsymbol{q}}+\left({\boldsymbol{b}}_{\mathbf{1},\boldsymbol{q}}-{\boldsymbol{b}}_{\mathbf{0},\boldsymbol{q}}\right){\boldsymbol{e}}^{-{\boldsymbol{b}}_{\mathbf{2}}{\boldsymbol{t}}_{\boldsymbol{f},\boldsymbol{q},\boldsymbol{d}}}+{\boldsymbol{\lambda}}_{\boldsymbol{q}}{\boldsymbol{\theta}}_{\boldsymbol{f}}+{\boldsymbol{\epsilon}}_{\boldsymbol{f},\boldsymbol{q},\boldsymbol{d}} $$

The three b parameters represent how an item’s difficult changes as time passes: b_{0, q} represents an item’s maximum difficulty (as time to resolution goes to infinity), b_{1, q} an item’s minimum difficulty (immediately prior to resolution), and b₂ the shape of the curve between b_{0, q} and b_{1, q} based on how much time is remaining in the question at the time of the forecast (t_{f, q, d}). The other two parameters represent how well an item discriminates between forecasters of different skill levels (λ_q) and how skilled the individual forecasters are (θ_f). As the estimate of forecaster skill, talent spotters will typically be most interested in this θ_f parameter, which is conventionally scaled so that it is on a standard normal distribution, θ_f~N(0, 1), with scores of 0 indicating an average forecaster, −1 a forecaster that is 1 SD below average, and 1 a forecaster that is 1 SD above average.

One potential problem with this model is that, in some cases, the distribution of Brier Scores is not well behaved. This typically occurs in cases which have many binary questions, so that the Brier score is a direct function of the accuracy assigned to the correct option. In such cases, the distribution of Brier scores can be multi-modal, because forecasters will tend to input many extreme and round number probability estimates, such as 0, .5, and 1 (Bo et al., 2017; Budescu et al., 1988; Merkle et al., 2016; Wallsten et al., 1993). To accommodate such multi-modal distributions, one option is to discretize the distribution of Brier scores into bins and reconfigure the model into an ordinal response model. Such models, such as the graded response model (Samejima, 1969), have a long history in the IRT literature.

Merkle et al. (2016) and Bo et al. (2017) describe examples of ordinal IRT models for forecasting judgment. However, the former found that the continuous and ordinal versions of the model were highly correlated (r = .87) in their assessment of forecaster ability level, and that disagreements tended to be focused on poor performing forecasters (who tend to make large errors) than high performing forecasters.

1.2 Contribution Scores

To obtain contribution scores for individual forecasters, it is necessary to first define some aggregation method for all of their judgments for each question. The simplest, and most common form of aggregation would just be to obtain the mean of all probabilities for all events associated with a forecasting problem. The aggregate probability (AP) for each of the c events associated with a forecasting question across all forecasters would be

$$ \boldsymbol{A}{\boldsymbol{P}}_{\boldsymbol{q},\boldsymbol{c}}=\frac{\sum \limits_{\boldsymbol{f}=\mathbf{1}}^{\boldsymbol{F}}{\boldsymbol{p}}_{\boldsymbol{q},\boldsymbol{c},\boldsymbol{f}}}{\boldsymbol{F}} $$

And the aggregate Brier score (AB) would then be

$$ \boldsymbol{A}{\boldsymbol{B}}_{\boldsymbol{q}}=\sum \limits_{\boldsymbol{c}=\mathbf{1}}^{\boldsymbol{C}}{\left({\boldsymbol{AP}}_{\boldsymbol{q},\boldsymbol{c}}-{\boldsymbol{y}}_{\boldsymbol{q}}\right)}^{\mathbf{2}} $$

Based on this aggregation approach, defining the contribution of individual forecasters to the aggregate is algebraically straightforward. We can define AP_{q, c, − f} as the aggregate probability with an individual forecaster’s judgment removed as

$$ \boldsymbol{A}{\boldsymbol{P}}_{\boldsymbol{q},\boldsymbol{c},-\boldsymbol{f}}=\frac{\left(\boldsymbol{F}\right)\left(\boldsymbol{A}{\boldsymbol{P}}_{\boldsymbol{q},\boldsymbol{c}}\right)-{\boldsymbol{p}}_{\boldsymbol{q},\boldsymbol{c},\boldsymbol{f}}}{\boldsymbol{F}-\mathbf{1}} $$

And the aggregate Brier score with an individual forecaster’s judgment removed as

$$ A{B}_{q,-f}=\sum \limits_{c=1}^C{\left({AP}_{q,c,-f}-{y}_q\right)}^{2.} $$

Finally, we define a forecaster’s average contribution to the accuracy of the aggregate crowd forecasts as

$$ {\boldsymbol{C}}_{\boldsymbol{f}}=\frac{\sum \limits_{\boldsymbol{q}=\mathbf{1}}^{\boldsymbol{Q}}\boldsymbol{A}{\boldsymbol{B}}_{\boldsymbol{q}}-\boldsymbol{A}{\boldsymbol{B}}_{\boldsymbol{q},-\boldsymbol{f}}}{\boldsymbol{Q}} $$

C_f is a representation of how much information a forecaster brings to the table, on average, that is both unique and beneficial. It is possible that a forecaster ranked very highly on individual accuracy might be ranked lower in terms of their contribution, because their forecasts tended to be very similar to the forecasts of others, and so they did less to move the needle when averaged into the crowd.

Both weighting members of the crowd by average contribution scores, as well as selecting positive or high performing contributors, have been demonstrated to improve the aggregate crowd judgment (Budescu & Chen, 2015; Chen et al., 2016). The approach is especially appealing because it can be extended into a model that is dynamic, in that it is able to update contribution scores for each member of a crowd as more information about their performance comes available; it requires relatively little information about past performance to reliably estimate high performing contributors; and it is cost effective, in that is able to select a relatively small group of high performing contributors who can produce an aggregate judgment that matches or exceeds the judgment of larger crowds in terms of accuracy (Chen et al., 2016).

The advent of contribution assessment was initially designed with a particular goal in mind: to improve the aggregate wisdom of the crowd (Budescu & Chen, 2015; Chen et al., 2016). One might challenge as slightly as a slightly narrower goal than pure talent spotting. It is clearly an effective tool for maximizing crowd wisdom, but is it a valid tool for assessing expertise? The answer appears to be yes. Chen et al. (2016) not only studied contribution scores as an aggregation tool but tested how well contribution scores perform at selecting forecasters known to have a skill advantage through various manipulations known to benefit expertise, such as explicit training and interactive collaboration.

Table 6.6 Correlation matrix for measures in Study 2. Pearson correlation coefficients reported. Below-diagonal values are assessed in-sample, above diagonal values are calculated out-of-sample. Diagonal values highlighted in gray are cross-sample reliability coefficients. The bottom five measures are not question specific, so out-of-sample correlation coefficients or cross-sample reliability coefficients are not relevant