Towards greater validity in Schwartz’s portrait values indicator using experimental research

Abstract

In Schwartz’s 21-item portrait values questionnaire (PVQ) each item consists of two statements which refer to a single underling value. In each of the 21 items the respondents are asked to give a single response to the two statements. Anecdotal evidence and cognitive response theory alert us to potential measurement error in this two-statements-single-response approach. Respondents might be influenced by different themes contained in the statements and their answers might not be comparable. This paper addresses the question: Do the responses to the 21 items in Schwartz’s PVQ (where each item contains two statements) differ significantly when two separate responses are allowed per item (in the split version) compared to when a single response is allowed (in the combined version)? In order to answer this research question we adopted an experimental design in a two-wave panel study. In the first wave we used Schwartz’s combined version of the PVQ. In the second wave we split the two statements and treated each statement as a separate item, thus requiring responses to each statement. Data was collected from Sociology classes at two universities: one in Austria (n = 52) and the other in South Africa (n = 61). We used statistical and non-statistical methods of analysis. The overall statistical assessment (z test) supports the split version although not all the various z test results unanimously concur. The non-statistical assessment does not support either the split version or the combined version. These mixed results necessitate further interrogation of the continued use of the combined version in the PVQ.

Appendices

Appendix 1

See Table 7.

Appendix 2

2.1 Generalisation of Stouffer’s combined test

Stouffer (1949 quoted in Wolf 1986, p. 20) proposed the following combined test for summarising the results of n independent samples⁶

$$ Z_{c} = \frac{Z}{\sqrt n }. $$

Stouffer’s combined test is based on the fact that the sum of standard normal distributed variables

$$ Z = \sum\limits_{1 = 1}^{n} {Z_{i} } $$

has a normal distribution with mean

$$ E(Z) = \sum\limits_{1 = 1}^{n} {E(z_{i} )} = \sum\limits_{1 = 1}^{n} {0 = 0} , $$

Because E(z _i ) = 0 for standard normal distributed variables.

The variance is

$$ VAR(Z)\,\sum\limits_{1 = 1}^{n} {VAR(z_{i} )} + \sum\limits_{i = 1}^{n} {\sum\limits_{\begin{subarray}{l} j = 1 \\ j \ne i \end{subarray} }^{n} {COV(z_{i} ,z_{j} )} = p + \sum\limits_{i = 1}^{n} {\sum\limits_{\begin{subarray}{l} j = 1 \\ j \ne 1 \end{subarray} }^{n} {COR(z_{i} ,z_{j} )} } } $$

because VAR(z _i ) = 1 and COV (z _i ,z _j ) = COR (z _i ,z _j ) for standard normal distributed variables.

Assuming a constant correlation COV(z _i ,z _j ) = φ, the formula for the variance simplifies to

$$ VAR(Z) = n + n \times (n - 1) \times \varphi . $$

The random variable Z can be transformed in standard normal distributed variable z _c be dividing Z with the standard deviation

$$ z_{c} = \frac{Z}{{\sqrt {VAR(Z)} }} = \frac{Z}{{\sqrt {n + n \times (n - 1) \times \varphi } }} $$

z _c is a generalisation of Stouffer’s formula. Stouffer assumes independent measures. This implies that the correlation φ = 0. The variance of Z becomes $VAR(Z) = n + n \times (n - 1) \times 0 = n$ and results in Stouffer’s original proposal for z _c

$$ z_{c} = \frac{Z}{\sqrt n } $$

In the case of perfect dependency (φ = 1), z _c is the average of the individual z statistics:

$$ z_{c} = \frac{Z}{{\sqrt {n^{2} } }} = \frac{Z}{n} $$

because $VAR(Z) = n + n \times (n - 1) \times 1 = n^{2} .$