Progress in Adaptive Web Surveys: Comparing Three Standard Strategies and Selecting the Best

Abstract

Progress indicators inform the participants of web surveys about their state of completion and play a role in motivating participants with a special impact on dropout and answer behaviour. Researchers and practitioners should be aware of this impact and, therefore, should select the right indicator for their surveys with care. In some cases, the calculation of the progress becomes, however, more difficult than expected, especially, in adaptive surveys (with branches). Previous work explains how to compute the progress in such cases based on different prediction strategies, although the quality of prediction of these strategies still varies for different surveys. In this revised paper of a conference paper, we demonstrate the challenges of finding the best strategy for progress computation by presenting a way to select the best strategy via the RMSE measure. We show the application of this method in experimental designs with data from two large real-world surveys and in a simulation study with over 10k surveys. The experiments compare three prediction strategies taking into account the minimum, average, and maximum number of items that participants have to answer by the end of the survey. Selecting the mean as strategy is usually a good choice. However, we found that there is no single best strategy for every case, indicating a high dependence on the structure of the survey to produce good predictions.

Appendix

Section 3 mentioned that the MdAE and RMdSE mostly result in almost the same values. In this appendix, we are going to investigate this fact.

Assume a vector of (real) values $v = \{v_1,\ldots ,v_n\}$, $n \ge 1$. The first step of computing the median $\tilde{v}$ is to order the values of v by size. By taking the MdAE, the ordering is performed on |v|. For RMdSE, the ordering is performed on $v^2$. Since for each two arbitrary (real) values $v_1$ and $v_2$ it holds true that $|v_1| \le |v_2| \iff v^2_1 \le v^2_2$, it is easy to confirm that the ordering of |v| is equal to the ordering of $v^2$. Let $(a_1,\ldots ,a_n)$ be the resulting order of the absolute and $(s_1,\ldots ,s_n)$ be the ordering for the squared values. The following is valid:

$$\begin{aligned} \forall {1 \le i \le n}:a_i = \sqrt{s_i} \end{aligned}$$

(2)

The second step of computing the median depends on the number of dimensions n of v. There are two cases:

• 1. n is odd. The median is the value on position n/2 of the ordering. It is $a_{{n}/{2}}$ for MdAE and $s_{{n}/{2}}$ for RMdSE. For RMdSE, we have to take the root median, i.e., RMdSE is $\sqrt{s_{{n}/{2}}}$. With Eq. 2 in mind, the MdAE and RMdSE are equal.

• 2. n is even. The median is half the sum of the values on positions n/2 and ${n}/{2} + 1$. It is ${1}/{2}(a_{{n}/{2}} + a_{{n}/{2} + 1})$ for MdAE and ${1}/{2}(s_{{n}/{2}} + s_{{n}/{2} + 1})$ for RMdSE. For RMdSE, we take again the rooted median, $\sqrt{{1}/{2}(s_{{n}/{2}} + s_{{n}/{2} + 1})}$. Actually, the MdAE and RMdSE are unequal. But on closer inspection, the value of ${1}/{2}(s_{{n}/{2}} + s_{{n}/{2} + 1})$ is always between $s_{{n}/{2}}$ and $s_{{n}/{2} + 1}$, and therefore, the RMdSE is always between $a_{{n}/{2}}$ and $a_{{n}/{2} + 1}$ with regard on Eq. 2.

Both cases result in the following facts:

• 1. MdAE and RMdSE are equal if n is odd.

• 2. MdAE and RMdSE are almost equal if n is even and the values on positions n/2 and ${n}/{2} + 1$ are close to each other.