Bayesian Thurstonian models for ranking data using JAGS

Abstract

A Thurstonian model for ranking data assumes that observed rankings are consistent with those of a set of underlying continuous variables. This model is appealing since it renders ranking data amenable to familiar models for continuous response variables—namely, linear regression models. To date, however, the use of Thurstonian models for ranking data has been very rare in practice. One reason for this may be that inferences based on these models require specialized technical methods. These methods have been developed to address computational challenges involved in these models but are not easy to implement without considerable technical expertise and are not widely available in software packages. To address this limitation, we show that Bayesian Thurstonian models for ranking data can be very easily implemented with the JAGS software package. We provide JAGS model files for Thurstonian ranking models for general use, discuss their implementation, and illustrate their use in analyses.

Introduction

Consider data where subjects rank K choices. Ranking data can be represented by a set of K response variables: ${Y_{i1 }},{Y_{i2 }},\ldots,{Y_{iK }}$, where Y _ik represents the rank given by subject i to choice k. For example, Y _i1 = 2, Y _i2 = 3, and Y _i3 = 1 is a ranking of K = 3 choices by subject i where the third choice is ranked first, the first choice is ranked second, and the second choice is ranked third. The use of ranking data in research is attractive because ranks are relatively easy to elicit from respondents, cannot violate transitivity, and are not subject to idiosyncrasiesin scale use. Ranking data are particularly useful in applications where respondents are faced with evaluating several choices simultaneously, as in conjoint analysis. The challenge of ranking data lies in the statistical analysis of these multivariate discrete responses. Reviews of statistical models for ranking data are given by Critchlow, Flinger, and Verducci (1991) and Marden (1995). One particular family of models, known as order statistic models, assume that the observed ranking can be modeled on the basis of the order statistics of a sample from an underlying multivariate continuous distribution. The Thurstonian ranking model is one of the oldest and most well-known order statistic models.

Despite being a relatively simple and intuitive model, reported applications of Thurstonian ranking models have been rare, and many of these are only examples in methodological articles. Computational difficulties in implementing Thurstonian ranking models are one likely reason for their rarity. Thurstonian ranking models define the probability of an observed ranking as a (K − 1)-dimensional integral, which cannot be expressed in closed form. The crux of the problem of implementing these models is how to either approximate this integral numerically or circumvent its evaluation altogether. Much of the methodological research on Thurstonian ranking models has focused on this issue, and a variety of solutions have been proposed. But these methods require significant technical expertise to use, and since software using these methods is not widely available, analyses must often be programmed on a case-by-case basis. This article directly addresses this problem by showing that Bayesian Thurstonian ranking models can be very easily implemented in the free and open-source software package JAGS. We also provide generic JAGS model files that users can easily adapt for their own analyses.

The rest of this article is organized as follows. We briefly review Thurstonian ranking models, including discussions of the specification of covariates, model identification, and estimation methods. We then describe in detail the implementation of Bayesian Thurstonian ranking models in JAGS. We also provide two demonstrations of analyses using Thurstonian ranking models with JAGS, using real data sets. Model files and further details are provided in the appendices and as supplementary material.¹

Thurstonian ranking models

The Thurstonian ranking model was proposed as a scaling method for ranking data by Thurstone (1931), on the basis of his law of comparative judgment (Thurstone, 1927). This model assumes that observed frequencies of rankings are consistent with probabilities of the ranking of unobserved continuous response variables representing what might be thought of as the psychological perception of each choice. A Thurstonian ranking derives the probability of a given ranking on the basis of the distribution of a set of K unobserved or latent response variables, ${U_{i1 }},{U_{i2 }},\ldots,{U_{iK }}$, corresponding to the K ranked choices.² These latent response variables are assumed to have a multivariate normal distribution, ${N_K}\left( {{\mu_i},∑} \right)$. One of the strengths of Thurstonian ranking models is in the flexibility of the mean and covariance structure of the latent responses. To extend the model to include covariates, we index the mean vector by the observation and will later discuss how to parameterize the mean as a function of one or more covariates. An arbitrary covariance structure allows heteroscedasticity of and correlations among the latent responses. Other related order-statistic models, such as “exploded” logit models (Beggs, Cardell, & Hausman, 1981; Chapman & Staelin, 1982; Hausman & Ruud, 1987; Luce, 1959; Plackett, 1975; Punj & Staelin, 1978), assume independence of the latent responses (sometimes referred to as independence of irrelevant alternatives. or IIA, in the discrete choice literature), which can be too restrictive.

It is common to write Thurstonian ranking models in terms of the differences among the latent responses for identification reasons and to slightly simplify computations. Let ${Z_{ik }}={U_{ik }}-{U_{iK }}$ represent the difference between the latent response to the kth and Kth (i.e., last) choice. In matrix notation, this can be written as z _i = C u _i , where $u_i^{\prime }=\left( {{U_{i1 }},{U_{i2 }},\ldots,{U_{iK }}} \right)$, $z_{i}^{\prime } = ({{Z}_{{i1}}},{{Z}_{{i2}}}, \ldots ,{{Z}_{{i,K - 1}}})$, and

$$ {\mathbf{C}}=\left[ {\begin{array}{*{20}c} 1 & 0 & \cdots & 0 & {-1} \\ 0 & 1 & \cdots & 0 & {-1} \\ \vdots & \vdots & \ddots & \vdots & \vdots \\ 0 & 0 & \cdots & 1 & {-1} \\\end{array}} \right]. $$

(1)

The probability of a given rank order can be expressed as

$$ P\left( {{Y_{i1 }}={y_1},\;{Y_{i2 }}={y_2},\ldots,{Y_{iK }}={y_K}} \right)=\int \cdots \int {\phi \left( {z;{\mathbf{C}}{\mu_i},{\mathbf{C∑C}}\prime } \right)} I\left( {z\in {S_i}} \right)d{z_1}d{z_2}\ldots d{z_{K\;-\;1 }}, $$

(2)

where ϕ denotes the probability density function of a multivariate normal distribution with given mean vector and covariance matrix evaluated at $z\prime =\left( {{z_1},{z_2},\ldots,{z_{K\;-\;1 }}} \right)$, and S _i is the set of values ${z_1},{z_2},\ldots,{z_{K-1 }}$ such that, when combined with z _K = 0, has the same rank order as ${y_{i1 }},{y_{i2 }},\ldots,{y_{iK }}$. Note that $I\left( {z\in {S_i}} \right)=1$ if  $z\in {S_i}$, and zero otherwise, so that Eq. 2 computes the probability that the multivariate, normally distributed vector z _i with mean vector C μ _i and covariance matrix $\mathbf{C}∑C\prime$ falls within S _i to produce the observed rank order ${{y}}_i^{\prime }=\left( {{y_{i1 }},{y_{i2 }},\ldots,{y_{iK }}} \right)$.

Including covariates

Discussions of Thurstonian ranking models do not always include covariates. The mean and variance are often specified as being constant over observations, although some researchers have discussed letting the mean μ _i vary over observations, due to its dependence on covariates (e.g., Yu, 2000). This is not a major extension of the model, but it is worth discussion since, in practice, covariates are common and there are some important considerations in terms of how their effects are specified and identified. Including covariates effectively creates a kind of regression model for ranking data.

Because ranking data are multivariate in the sense that each observation consists of K ranks, ${Y_{i1 }},{Y_{i2 }},\ldots,{Y_{iK }}$, that vary both between and within choice sets, it is useful to distinguish between covariates that vary only between choice sets (i.e., over i but not over k) and those that vary within choice sets (i.e., over k). In the first case, the covariates might represent characteristics of respondents, conditions under which respondents made their rankings, and interactions thereof. Assume that the values of P covariates are collected in a vector $x_i^{\prime }=\left( {1,{x_{i1 }},{x_{i2 }},\ldots,{x_{iP }}} \right)$. Then it might be reasonable to model the mean vector $\mu_i^{\prime }=\left( {{\mu_{i1 }},{\mu_{i2 }},\ldots,{\mu_{iK }}} \right)$ using the linear model

$$ {\mu_{ik }}={\beta_{0k }}+{\beta_{1k }}{x_{i1 }}+{\beta_{2k }}{x_{i2 }}+\ldots +{\beta_{Pk }}{x_{iP }}. $$

(3)

A second case is one where all covariates vary within choice sets. This could occur if characteristics of the choices are used as covariates. Here, the covariates are collected into a vector $x_{ik}^{\prime }=\left( {{x_{i1k }},{x_{i2k }},\ldots,{x_{ikP }}} \right)$. A linear model for the mean in this case would be

$$ {\mu_{ik }}={\beta_1}{x_{i1k }}+{\beta_2}{x_{i2k }}+\ldots +{\beta_P}{x_{iPk }}. $$

(4)

Note that the differences between Eqs. 3 and 4 are whether the parameters or covariates vary over choices. A third situation is one where some covariates vary only between choice sets, whereas others vary within choice sets. Suppose that there are P _b covariates that vary only between choice sets and P _w covariates that vary within choice sets. Here, a mean structure can be created that combines Eqs. 3 and 4 so that

$$ \begin{array}{*{20}c} {{\mu_{ik }}=\beta_{0k}^{(b) }+\beta_{1k}^{(b) }x_{i1}^{(b) }+\beta_{2k}^{(b) }x_{i2}^{(b) }+\ldots +\beta_{{{P_b}k}}^{(b) }x_{{i{P_b}}}^{(b) }} \\ {+\beta_1^{(w) }x_{i1k}^{(w) }+\beta_2^{(w) }x_{i2k}^{(w) }+\ldots +\beta_{{{P_w}}}^{(w) }x_{{i{P_w}k}}^{(w) }.} \\\end{array}$$

(5)

Here, we use the superscripts (b) and (w) to distinguish between covariates and parameters corresponding to effects that vary between choice sets and within choice sets, respectively. Clearly the models in Eqs. 3 and 4 can be thought of as special cases of Eq. 5. Perhaps less obviously, with the use of dummy variables for each choice, Eqs. 3 and 5 can be written as Equation 4. But for simplicity and clarity, it is useful to consider special cases separately, rather than considering only the general but more complicated model. Similar discussions of parameterizations for models for choice data are given by, for example, Agresti (2002).

Parameter identification

The identification of Thurstonian models for ranking data is complicated by the fact that the probabilities defined in Eq. 2 depend only on the differences between latent responses and that the scale of these differences is arbitrary. On the basis of the mean structure in Eq. 3, the mean of ${Z_{ik }}={U_{ik }}-{U_{{i{k}^{\prime}}}}$ is

$$ {\mu_{ik }}-{\mu_{{i{k}^{\prime}}}}={\beta_{0k }}-{\beta_{{0{k}^{\prime}}}}+({\beta_{1k }}-{\beta_{{1{k}^{\prime}}}}){x_{i1 }}+({\beta_{2k }}-{\beta_{{2{k}^{\prime}}}}){x_{i2 }}+\ldots +({\beta_{Pk }}-{\beta_{{P{k}^{\prime}}}}){x_{iP }}. $$

(6)

Since the mean difference depends only on the difference between β _jk and β _jk′, we can fix, say, β _jK = 0 to resolve the indeterminacy. However, it should be noted that the choice of which β _jk should be fixed and the value at which it should be fixed are arbitrary. Thus, inferences are limited to the differences among the β _jk between choices (i.e., over k). The mean of Z _ik based on the mean structure in Eq. 4 is

$$ {\mu_{ik }}-{\mu_{{i{k}^{\prime}}}}={\beta_1}({x_{i1k }}-{x_{{i1{k}^{\prime}}}})+{\beta_2}({x_{i2k }}-{x_{{i2{k}^{\prime}}}})+\ldots +{\beta_P}({x_{iPk }}-{x_{{iP{k}^{\prime}}}}). $$

(7)

This shows that an implicit assumption of this parameterization is that x _ijk varies over the choices; otherwise, ${x_{ijk }}-{x_{{ij{k}^{\prime}}}}=0$ and β _j are not identified. Models with covariates that do not vary within choice sets should use the parameterization in Eq. 3 or 5. The identification constraints necessary for the mean structure in Eq. 5 follow naturally from those for Eqs. 3 and 4, discussed above.

The covariance matrix $\mathbf{∑}$ is identified only up to the covariance matrix of the differences, $\mathbf{C}∑C\prime$. Our recommendation is to limit inferences to this covariance matrix. Although there are various constraints that can be imposed on $\mathbf{∑}$ so that it is identified (see Yu, 2000), these constraints, while technically convenient, are not particularly meaningful. The scale of the latent responses is not set, meaning that the probability of a given rank order does not change if μ _i and $\mathbf{C}∑C\prime$ are multiplied by d and $\sqrt{d}$, respectively, for any d > 0. Our approach to resolving this indeterminacy is to fix the generalized variance of $\mathbf{C}∑C\prime$, which is defined as the determinant $v=\left| {\mathbf{C}∑C\prime } \right|$.³ We find this approach useful because it does not involve fixing individual arbitrary parameters.

Estimation methods

Perhaps the most significant obstacle to the use of Thurstonian ranking models in research is the computational complexity of the probability defined in Eq. 2. The (K − 1)-dimensional integral cannot be expressed in closed form. In cases with relatively few observed rank orders, few choices, and/or few distinct values of the covariate vector, numerical methods can be used to directly approximate the integral (e.g., Genz, 1992), but these methods can be computationally or numerically problematic in other situations. Other estimation approaches have been proposed to circumvent the integration in various ways. Several researchers have proposed limited-information methods using the ranking of pairs or triples of choices (Brady, 1989; Chan & Bentler, 1998; Maydeu-Olivares, 1999). Others have proposed full-information maximum likelihood methods using Monte Carlo expectation-maximization algorithms (Poon & Lu, 2009; Xu, Poon, & Lee, 2008). Yao and Böckenholt (1999) and Yu (2000) showed that Bayesian Thurstonian ranking models are amenable to Markov chain Monte Carlo (MCMC) algorithms by using data augmentation with Gibbs sampling (Gelfand & Smith, 1990; Tanner & Wong, 1987). Using this approach, the unobserved differences are specified as missing data in a complete-data likelihood function to create the Bayesian probability model:

$$ h\left( \theta \right)\prod\limits_{i\;=\;1}^N \phi \left( {{z_i};C{\mu_i},\mathbf{C}∑C\prime } \right)I\left( {{z_i}\in {S_i}} \right), $$

(8)

where h(θ) denotes a prior distribution for all unknown parameters (collected into the vector θ). Using MCMC, a sample of realizations from the posterior distribution can be simulated. Inferences concerning the marginal posterior distribution of θ, “integrating” over the unobserved responses $z\prime_i=\left( {{Z_{i1 }},{Z_{i2 }},\ldots,{Z_{i,K\;-\;1 }}} \right)$, can then be done using summary statistics of the sample without reference to z _i .

While these approaches have significantly lessened computational obstacles to the use of Thurstonian ranking models, they require significant expertise in their use. Progress has been made in making limited-information methods accessible using standard software packages for mean and covariance structure models (Brown & Maydeu-Olivares, 2012; Maydeu-Olivares & Böckenholt, 2005). But full-information likelihood-based methods have not yet been incorporated into widely available user-friendly software packages, and implementing them oneself requires significant expertise in programming algorithms for simulation-based inference. The main purpose of this article is to address this obstacle by showing how Bayesian Thurstonian ranking models can be implemented very easily in the software package JAGS.

Implementing Bayesian Thurstonian Ranking Models in JAGS

Just Another Gibbs Sampler, or JAGS (Plummer, 2003), is a free program licensed under the GNU General Public License. Compiled versions of JAGS are available for the Macintosh OS X and Microsoft Windows operating systems and several Linux distributions. The source code is also available. JAGS is similar to WinBUGS (Lunn, Spiegelhalter, Thomas, & Best, 2009) and OpenBUGS (Thomas, O’Hara, Ligges, & Sturtz, 2006), where “BUGS” stands for “Bayesian inference using Gibbs sampling,” and also PROC MCMC in SAS/STAT (SAS Institute Inc., 2008). With these programs, a user specifies a Bayesian probability model, and the program builds an MCMC algorithm to sample realizations from the posterior distribution. These programs exploit the fact that such algorithms can be broken down into a series of smaller steps for models that can be represented as directed acyclic graphs, which include many common statistical models. But the user needs only to specify a model in a simple syntax. The syntax used by JAGS, WinBUGS, and OpenBUGS is similar to that used in R (R Development Core Team, 2012). All this allows the user to focus on model specification and inference, and not the development and programming of the MCMC algorithm. For a good didactic treatment of Bayesian modeling and analysis using JAGS, see Kruschke (2011).

Here, we show how to use JAGS to conduct analyses using Thurstonian ranking models. It is convenient to use the data augmentation approach to specify the Bayesian probability model given in Eq. 8, since the probability in Eq. 2 is difficult to evaluate. Given S _i , the distribution of z _i is censored, in the sense that the value of z _i is unknown but is known to be in S _i so as to be consistent with the observed ranking, y _i . We refer to z _i as being rank-censored. Without conditioning on ${z_i}\in {S_i}$, the observed ranking, the distribution of z _i is multivariate normal, so that

$$ {z_i}\sim {N_{K\;-\;1 }}\left( {C{\mu_i},\mathbf{C}∑C\prime } \right). $$

(9)

The trick is to specify the rank-censoring mechanism. This can be done by using some of the built-in programming functions to define bounds for the “distribution” in JAGS. This functionality is intended for censored response variables, such as in survival models, but it can also be used to specify the censoring of ${Z_{i1 }},{Z_{i2 }},\ldots,{Z_{i,K\;-\;1 }}$ implied by an observed ranking. Formally, the rank-censoring implies that

$$ {Z_{ik }}\in \left\{ {\begin{array}{*{20}c} {\left( {{Z_{{i{y_{{i\left( {k\;+\;1} \right)}}}}}},\infty } \right),} \hfill & {{\mathrm{if}}\;{y_{ik }}=1,} \hfill \\ {\left( {{Z_{{i{y_{{i\left( {k\;+\;1} \right)}}}}}},{Z_{{i{y_{{i\left( {k\;-\;1} \right)}}}}}}} \right),} \hfill & {{\mathrm{if}\;1} < {y_{ik }} < K,} \hfill \\ {\left( {0,{Z_{{i{y_{{i\left( {k\;-\;1} \right)}}}}}}} \right),} \hfill & {{\mathrm{if}}\;{y_{ik }}=K,} \hfill \\\end{array}} \right. $$

(10)

where y _i(k) is the value of y _ik with rank k.⁴ For example, the ranking y _i1 = 2, y _i2 = 3, y _i3 = 1, and y _i4 = 4 implies that S _i is the region such that 0 < Z _i2 < Z _i1 < Z _i3 (recall that Z _iK = 0 by definition). This can be expressed as interval-censoring, since the ranking is consistent with

$$ {Z_{i2 }}\in (0,{Z_{i1 }}),\:{Z_{i1 }}\in ({Z_{i2 }},{Z_{i3 }}),\:\mathrm{and}\;{Z_{i3 }}\in (0,\infty ). $$

(11)

Thus, each Z _ik is censored by one or two of the other latent responses. While WinBUGS, OpenBUGS, and PROC MCMC in SAS/STAT allow specification of censored variables, these packages cannot be used for Thurstonian ranking models in general, since they limit the censoring to scalars that are (conditionally) independent, which limits the range of models that can be considered. JAGS allows censoring of variables with an arbitrary joint distribution, and so, among the packages that have been developed for MCMC-based Bayesian inference, JAGS is particularly suitable for Thurstonian ranking models.

Appendix 1 lists the JAGS model file for a Thurstonian ranking model with a mean structure as defined in Eq. 3. In the interest of space, Appendix 2 lists only changes to this file to allow for the mean structures defined in refeq:mumodelw and Eq. 5. Little or no modification of the file is necessary otherwise. This file is quite general in the sense that it can be applied to data with an arbitrary number of observations, choices, and covariates. The files do not require changes on the part of the user, since known variable values are passed to the program as variables during a JAGS session. We will, however, consider a few points that may be of interest to some users. The rank-censored multivariate distribution of z _i is defined on lines 15–23. The censoring mechanism given in Eq. 10 is defined on lines 17 and 18, which define the bounds for each element of z _i , line 19, which indicates how each element of z _i is censored on the basis of those bounds (the “variable” is a dummy variable indicating that the element is between the supplied bounds), and lines 22 and 23, which define the distribution of the elements of z _i prior to censoring (here, Z _iK = 0 by definition). The prior distribution of all β _jk parameters is specified on line 27 as the relatively noninformative distribution N(0,1000). The prior distribution for the inverse of $\mathbf{C}ΣC\prime$ is specified to be a Wishart distribution on line 36, with inverse scale matrix R = (K – 1)CC′ (defined on line 10) and degrees of freedom K − 1, which is also a relatively noninformative prior distribution.⁵ Of course, these prior distributions can be modified by the user. The two quantities that a user will likely be interested in are and , which are defined as

(12)

and

(13)

respectively, where

$$ d=\mathop{{\left( {\frac{v}{{|\mathbf{C}∑\mathbf{C}|}}} \right)}}\nolimits^{{{1 \left/ {v} \right.}}}, $$

(14)

to fix the generalized variance at v, which in our examples is v = K.⁶ Recall that when β _jk is interpreted, it is useful to consider differences or contrasts between choices of the form β _jk – β _jk′ since β _jK = 0 to identify the model, but the choice of value at which to fix β _jk is arbitrary. In cases where all covariates vary between choices as in Eq. 4, the identified parameters for the mean structure are

(15)

and in cases where some covariates vary within choice sets and some do not, the identified parameters for the mean structure are

To simplify notation, where necessary, we use an asterisk to denote an identified parameter based on the generalized variance, so that, for example, $\beta_{jk}^{*}={\beta_{jk }}\sqrt{d}$ and $\beta_j^{*}={\beta_j}\sqrt{d}$. The rest of the model file concerns only minor computations and variable definitions that will generally not vary over applications.

Again, we would like to emphasize that the JAGS model files are quite general. In practice, one needs only to consider whether covariates vary between choice sets, within choice sets, or both to choose the appropriate model file. The only other information that needs to be provided are the N by K array of rankings, the design matrix or matrices, the number of observations (N), the number of choices (K), and the number of covariates (P, or P _b and P _w ). A relatively efficient way to run JAGS is by using the rjags package (Plummer, 2011) in R. We show how to do this in the following examples. The JAGS model files, data, and R scripts for replicating these analyses are available as supplementary material.

Example: Relationship between ranking of compensation plans and cultural orientation

This example concerns data collected as part of a study on compensation preferences (Kuhn & Johnson, 2004). While it is sometimes claimed that most people prefer to be paid on the basis of individual performance, rather than team or organizational performance, and that most strongly prefer fixed pay (salaries) to variable pay, such as performance bonuses and profit sharing (see Rynes, Colbert, & Brown, 2002), other research indicates more complex relationships (Kuhn, 2009; Kuhn & Yockey, 2003). In particular, cross-cultural variation in preferences for reward allocation has been widely studied (see Fischer & Smith, 2003, for a review), and U.S. students with more individualistic orientations tend to report greater preferences for individual rather than team-based raises (Cable & Judge, 1994). The data analyzed here adopted a more nuanced approach to individual differences in value orientations as predictors of preferences among common pay schemes, using the Triandis and Gelfand (1998) model that crosses individualism and collectivism with power distance to describe four cultural value types: horizontal individualism (hi), horizontal collectivism (hc), vertical individualism (vi), and vertical collectivism (vc). In addition to completing an inventory developed by Triandis and Gelfand to assess these four value types, each subject in a sample of N = 116 also ranked K = 4 compensation plans in order of preference: individual bonus, team bonus, profit sharing, and seniority.

Let Y _i1, Y _i2, Y _i3, and Y _i4 denote the ranks of the individual bonus, team bonus, profit sharing, and seniority compensation plants, respectively, and let hi _i , hc _i , vi _i , and vc _i denote the standardized cultural orientation scores for the ith subject.⁷ To model the rankings of the compensation plans as a function of these scores, we modeled the mean latent response of the ith subject to the kth compensation plan as

$$ {\mu_{ik }}={\beta_{0k }}+{\beta_{1k }}\mathrm{h}{{\mathrm{i}}_i}+{\beta_{2k }}\mathrm{h}{{\mathrm{c}}_i}+{\beta_{3k }}\mathrm{v}{{\mathrm{i}}_i}+{\beta_{4k }}\mathrm{v}{{\mathrm{c}}_i}. $$

(16)

The parameters for the seniority-based compensation plan are fixed at zero (i.e., ${\beta_{0,4 }}={\beta_{1,4 }}={\beta_{2,4 }}={\beta_{3,4 }}={\beta_{4,4 }}=0$) to identify the model. Because this constraint is arbitrary, to interpret the parameters, it is useful to consider differences in the mean response between compensation plans:

$$ \begin{aligned} {\mu_{ik }}-{\mu_{{i{k}^{\prime}}}}={\beta_{0k }}-{\beta_{{0{k}^{\prime}}}} \\ & & +\:({\beta_{1k }}-{\beta_{{1{k}^{\prime}}}})\mathrm{h}{{\mathrm{i}}_i}+({\beta_{2k }}-{\beta_{{2{k}^{\prime}}}})\mathrm{h}{{\mathrm{c}}_i}+({\beta_{3k }}-{\beta_{{3{k}^{\prime}}}})\mathrm{v}{{\mathrm{i}}_i}+({\beta_{4k }}-{\beta_{{4{k}^{\prime}}}})\mathrm{v}{{\mathrm{c}}_i}. \\ \end{aligned} $$

Thus, differences of parameters between plans capture the effect of the covariates on the mean difference or relative preference for the plans. For example, to compare individual bonus (k = 1) to team bonus (k′ = 2), β _1k – β _1k′ indicates how the mean preference for individual bonus changes relative to that of the team bonus plan as the horizontal individualism score (hi _i ) changes. If ${\beta_{1k }}-{\beta_{{1{k}^{\prime}}}} > 0$, respondents tend to prefer individual over team bonus plans more as hi _i increases, whereas if ${\beta_{1k }}-{\beta_{{1{k}^{\prime}}}} < 0$, respondents tend to prefer individual over team bonus plans less as hi _i increases. The value of ${\beta_{1k }}-{\beta_{{1{k}^{\prime}}}}$ is the rate at which this relative preference increases or decreases per unit change in hi _i .

To conduct our analysis, we simulated realizations from the posterior distribution of the Bayesian Thurstonian ranking model given in Eq. 8 with the mean structure described above using JAGS from R using rjags. The R commands are given below.

The data are an N = 116 by K = 4 array of rankings (), and the corresponding four vectors of N = 116 cultural orientation scores (, , , and ). These are stored in a list object called , along with the values of N, P, and K. Initial values for the latent differences are generated by the function and stored in the array (the function is provided in Appendix 3). The command checks and compiles the model defined in the file . The listing of this file is given in Appendix 1. The command produces an initial sample to “burn in” the algorithm for 5,000 iterations. The command then simulates 10,000 realizations from the posterior distribution of and , as defined in Eqs. 12 and 13, respectively, which are saved in . Summaries (e.g., posterior means, quantiles and percentile ranks) of the posterior distribution of any parameter or functions thereof, such as those shown in the figures discussed below, can then be obtained by simply computing descriptive statistics of with any general-purpose statistical package. The adequacy of the burn-in period, convergence of the simulation, and the size of the simulated sample from the posterior distribution can be assessed using utilities in the R package coda (Plummer, Best, Cowles, & Vines, 2006).⁸

Figure 1 shows the posterior distributions of the identified parameter differences, $\beta_{jk}^{*}-\beta_{{j{k}^{\prime}}}^{*}$. Each box plot depicts the posterior distribution for the parameter difference for a given pair of compensation plans, k and k′. The box plots are constructed on the basis of the 2.5 %, 25 %, 50 %, 75 %, and 97.5 % percentiles. These percentiles are approximated by the sample percentiles from the posterior distribution of simulated realizations of $\beta_{jk}^{*}-\beta_{{j{k}^{\prime}}}^{*}$ generated by JAGS. The box plots also show 95 % credibility intervals, since they cover the middle 95 % of the distribution. These can be used to assess which differences are unambiguously positive or negative. The example mentioned earlier of the effect of horizontal individualism on the relative preference for individual versus team bonus is shown by the left-most box plot in the panel in the second row and the second column (and also the left-most box plot in the panel in the second row and the first column, which is the reverse of this difference). This shows that on the basis of this analysis, we can be relatively confident that increases in horizontal individualism result in a preference for individual bonus to increase, relative to that of team bonus. As another example, the apparent effect of vertical collectivism is to increase preferences for team bonus, in comparison with the individual bonus, as shown in the left-most box plot in the last row and first and second columns. Probabilities can also be used to assess the (un)certainty in these differences. For example, the posterior probability that the preference for individual versus team bonus increases with horizontal individualism is $P\left( {{\beta_{1,1 }}-{\beta_{1,2 }} > 0\left| y \right.} \right)\approx 0.99$. The posterior probability that preference for team bonus versus individual bonus increases with vertical collectivism is $P\left( {{\beta_{4,2 }}-{\beta_{4,1 }} > 0\left| y \right.} \right)\approx 0.99$. These posterior probabilities are approximated very easily by the proportion of observations of the difference in the simulated sample that exceed zero.

Fig. 1

Box plots for the posterior distributions of $\beta_{jk}^{*}-\beta_{{j{k}^{\prime}}}^{*}$. Panel columns correspond to the k′th compensation plan, whereas the abscissa within each panel corresponds to the kth compensation plan

In this example, the covariance structure $\mathbf{C}ΣC\prime$ is not of interest. However, inferences concerning the covariance structure can be made using the posterior distribution of the elements of the covariance matrix. For illustration, Fig. 2 shows box plots depicting the posterior distribution of each distinct element of the covariance structure. It is important to remember that these are the posterior distributions of the (co)variances of the differences ${Z_{ik }}={U_{ik }}-{U_{iK }}$, and not U _ik , since, as was discussed earlier, the covariance structure of the latter is not identified.

Fig. 2

Box plots for the posterior distributions of $\mathrm{Var}({Z_{ik }},{Z_{{i{k}^{\prime}}}})d$—that is, the elements of the identified covariance matrix ${\mathbf{C}ΣC\prime} d$

Example: Ranking of job candidates after revelation of résumé discrepancies

This second example concerns a data set from a study that examined the effect of an experimental manipulation of apparent deception on the rank ordering of hypothetical job applicants (Kuhn, Johnson, & Miller, in press). Subjects were presented with a job description and were asked to evaluate the résumés of three candidates (with the initials MC, AS, and SW), and then they were given background check reports for each candidate before recording their final rank ordering. All participants evaluated the same three résumés, but they were randomly assigned to one of six conditions for the background reports. In each experimental condition, the background reports revealed that one applicant had a serious discrepancy (not having completed a claimed college degree), one applicant had a less serious discrepancy (employment dates of a previous job somewhat different than claimed), and the third applicant had no discrepancies. As part of the study, N = 73 subjects were asked to rank the K = 3 candidates with respect to their suitability for the job.

Let Y _i1, Y _i2, and Y _i3 denote the ranks of the candidates MC, AS, and SW, respectively. Since all covariates vary within choice sets, we used the mean parameterization given in Eq. 4. The model is

$$ {\mu_{ik }}={\beta_1}\mathrm{m}{{\mathrm{c}}_{ik }}+{\beta_2}\mathrm{a}{{\mathrm{s}}_{ik }}+{\beta_3}\mathrm{date}{{\mathrm{s}}_{ik }}+{\beta_4}\mathrm{degre}{{\mathrm{e}}_{ik }}, $$

(17)

where mc _ik and as _ik are indicator variables for the candidates MC and AS, respectively, and dates _ik and degree _ik are indicator variables for the dates and degree discrepancy conditions, respectively. On the scale of the latent response variables, this is effectively a three by three two-way factorial model with main effects for the candidate (MC, AS, or SW) and the presence and type of résumé discrepancy (dates, degree, or none). Here, the question is how the rankings depend on the candidate and the discrepancy conditions.

The implementation of this model in JAGS is similar to that in the previous example. The commands are given below.

The listing for the model file is described in Appendix 2. With respect to the previous example, the main difference here is the design matrix. Because the covariates vary over choices as well, three design matrices are assembled into an N by P by K array . Each “slice” of the array along the third dimension is a choice-specific design matrix, called , , and . A design matrix must be specified for each choice, but they are passed to JAGS as one array.

Figure 3a shows the posterior distributions of $\beta_{i1}^{*}$ and $\beta_{i2}^{*}$, as well as $\beta_{i1}^{*}-\beta_{i2}^{*}$, which are the three pairwise contrasts among the three candidates. The posterior distributions show a clear mean preference for candidate MC over AS, SW over AS, and MC over SW. The posterior probabilities that these mean differences exceed zero are all at least .98. Figure 3b shows the posterior distributions of $\beta_{i3}^{*}$, $\beta_{i4}^{*}$, and $\beta_{i3}^{*}-\beta_{i4}^{*}$, which are the three pairwise contrasts among the three discrepancy conditions. There appears to be a negligible distinction in mean preferences between the date discrepancy and the no-discrepancy versions. However, a college degree discrepancy clearly made the candidate less preferable, on average, in comparison with those with a date discrepancy or no discrepancy. The posterior probabilities that these mean differences are in the stated direction are both greater than 0.99.

Fig. 3

Box plots for the posterior distributions of the contrasts $\beta_{i1}^{*}$, $\beta_{i2}^{*}$, and $\beta_{i1}^{*}-\beta_{i2}^{*}$ for the main effect of candidate, and $\beta_{i3}^{*}$, $\beta_{i4}^{*}$, and $\beta_{i3}^{*}-\beta_{i4}^{*}$ for the main effect of discrepancy condition

The model used here is only one possible model for the observed data. Other Thurstonian models could be specified (e.g., with different mean or covariance structures), or other models from outside the family of Thurstonian models could be used. One general approach to assessing the fit of a Bayesian probability model is to use posterior predictive checks (Gelman, Meng, & Stern, 1996; Meng, 1994; Rubin, 1984). These checks are based on comparing observed values of a chosen statistic, computed from the data, with the posterior predictive distribution of the statistic implied by the model. This effectively compares the observed value of the statistic to its predicted value, while taking into account the uncertainty in the model parameters. For ranking data, one simple and intuitive measure is the count of each rank order. It is relatively simple to generate a sample of observations from the posterior predictive distribution using JAGS. This can be done by adding the following lines between lines 23 and 24 in the model file in Appendix 1:

These lines simulate realizations of Z _ik () from the posterior predictive distribution and the ranking they imply (). Then the frequency of the six possible rankings in each simulated sample can be tabulated. Figure 4 shows the observed and posterior predictive distributions for the frequencies of the six possible rankings of the three candidates.

Fig. 4

Observed and posterior predictive distributions of ranking frequencies for each of the six possible rankings. The observed ranking frequencies are indicated by a point and vertical line. The number in the top-right corner of each panel is the proportion of observations from the posterior predictive distribution at or below the observed count

A model with poor fit may reveal observed counts that are outliers relative to the posterior predictive distribution of the counts.⁹ Note that in Fig. 4, none of the observed counts appear to be particularly unusual, so this check does not suggest that the specified model is a poor fit. If one or more of the observed counts appeared to be extreme outliers, it would be appropriate to consider modifying the model to produce a better fit or to abandon the model if such modifications were unsuccessful. It should be noted that other statistics can be specified for the checks. Yao and Böckenholt (1999) suggested using counts of partial rankings of all subsets of two, three, and/or four choices. This approach is particularly useful when the observed counts of all possible rankings are too sparse, such as when the number of choices is large relative to the sample size.

Discussion

Our goal with this article is to show that Bayesian Thurstonian ranking models can be very easily implemented using the free software package JAGS. To this end, we have provided general-purpose and easy-to-use JAGS model files, as well as detailed information and examples as to their use.¹⁰ It is our hope that our contribution will make Thurstone’s classic model for ranking data significantly more accessible for research.

The Thurstonian ranking model will hopefully become more widely available in statistical packages, following the increasing trend in available packages for multinomial probit models (a special case of Thurstonian ranking models in which only the first or top-ranked choice is given). MCMC algorithms for Bayesian multinomial probit models can be extended to model a full ranking of choices (Imai & van Dyk, 2005). But even with increasing availability of packages implementing the Thurstonian ranking model, we believe that there are benefits to the approach we described here using JAGS. In particular, the “open” nature of the JAGS model file allows a user to easily extend the model in ways that may not be feasible with other software. For example, the user could specify a mean structure that is nonlinear in the covariates or a specialized covariance structure. The JAGS model files we have provided may be used to develop other models as special cases of or extensions to a Thurstonian ranking model.

The focus here has been on modeling the mean structure of the latent responses. This is in contrast to others who have focused on modeling the covariance structure (e.g., Maydeu-Olivares, 1999; Maydeu-Olivares & Böckenholt, 2005).We have focused on only modeling the mean structure for simplicity, but we would argue that this is an important case for applied work, since in many applications, the effects of observed covariates as reflected in the mean structure are likely to be of interest. Furthermore, in cases where the covariance structure can be thought of as a consequence of additional latent variables, such as factor-analytic models, the covariance structure can be modeled alternatively by specifying these latent variables as unobserved covariates in the mean structure.

The data sets featured in the examples were from laboratory studies and might be considered relatively small in terms of both sample size and number of choices. We have found, however, that JAGS scales very well with both N and K, so larger data sets are not a significant computational burden. At most, we have found that the increase in computer time is approximately linear with the number of observations and choices.¹¹

Appendices

Appendix 1

The following listing is the JAGS model file for a Thurstonian ranking model using the mean structure in Eq. 3, where covariates vary between, but not within, choice sets. It is named in the examples.

Appendix 2

The listing below shows the changes to the listing in Appendix 1 to create a JAGS model file for the mean structure in Eq. 4, where all covariates vary within choice sets. This file is named .

The changes are simply to replace lines 20–40 in the listing in Appendix 1 with lines 20–33 above. For a model with some covariates that vary strictly between choice sets and some that vary within choice sets, the listing in Appendix 1 can be changed by replacing lines 20–40 with the lines given below.

The full file is called . Note that to implement this model, one must specify design matrices for both sets of covariates (i.e., between and within choice sets), as well as the number of covariates of each type. As an example, consider the second example where the mean structure could be written alternatively using the mean structure in Eq. 5 as

$$ {\mu_{ik }}=\beta_{0k}^{(b) }+\beta_1^{(w)}{{\mathrm{date}}{{{\mathrm{s}}}_{ik }}}+\beta_2^{({w})}\mathrm{degre}{{\mathrm{e}}_{ik }}. $$

(18)

Here, there are two covariates that vary within choice sets and no covariates that vary only between choice sets. The “intercept” parameters $\beta_{0k}^{(b) }$ are equivalent to the parameters for the main effect for candidate. Using the model file , the data would be defined as follows:

Appendix 3

The following R function creates initial values for ${Z_{i1 }},{Z_{i2 }},\ldots,{Z_{i,K\;-\;1 }}$ to be passed to JAGS.