Peer effects in computer assisted learning: evidence from a randomized experiment

Abstract

We conduct a large scale experiment to investigate peer effects in computer assisted learning (CAL). Identification relies on three levels of randomization. We find an average 0.17 standard deviation improvement in math scores among primary school students. This average effect is the same for students treated individually or in pairs, implying that peer effects double the learning benefit from a given equipment. Among paired students, poor performers benefit more from CAL when paired with good performers and vice versa. Average performers benefit equally irrespective of who they are paired with. This suggests that the treatment is dominated by knowledge exchange between peers. We also find that CAL treatment reduces the dispersion in math scores and that the beneficial effects of CAL can be strengthened if weak students are systematically paired with strong students.

Appendix: Balancedness and attrition

In column (5) of Table 1 we report regression coefficient of the baseline characteristic of one student on the baseline characteristic of the other. The estimated regression is of the form:

$$\begin{aligned} y_{it}=\beta _{0}+\beta _{2}y_{jt}+u_{it} \end{aligned}$$

(9)

This random assignment test is subject to exclusion bias: because a student cannot be his/her own peer, negative correlation between peer characteristics naturally arises under random assignment. Consequently, under the null hypothesis of random assignment estimated $\widehat{\beta } _{2}$ are not centered on 0 but on a negative number. Caeyers and Fafchamps (2016) derives the magnitude of the bias for groups and selection pools of fixed size and shows that the bias is particularly large when the randomly assigned group is small, e.g., in pairs.

We cannot use their formula directly because the size of the selection pools varies: class sizes are not constant. To circumvent this problem, we simulate the distribution of $\widehat{\beta }_{2}$ under the null using a so-called permutation method. This method also delivers a consistent p value for $\beta _{2}$ and thus offers a way of testing the null of random assignment. This method works as follows. The object is to calculate the distribution of $\widehat{\beta }_{2}$ under the null that $y_{it}$ and $y_{jt}$ are uncorrelated. To simulate $\widehat{\beta }_{2}$ under the null, we create counterfactual random matches and estimate (9). In practice, this is implemented by artificially scrambling the order of students within each class to reassign them into counterfactual random pairs. By construction these samples of paired observations satisfy the null of random assignment within classroom. We repeat this process 1000 times to obtain a close approximation of the distribution of $\widehat{\beta }_{2}$ under the null. We then compare the actual $\widehat{\beta }_{2}$ to this distribution to get its p value.

We present in Fig. 2 the simulated distribution of $\widehat{\beta }_{2}$ for baseline math scores under the null hypothesis of random assignment. These simulated $\widehat{\beta }_{2}$’s are centered around $-0.05$, with very few values at or above 0. As shown in the first line of column (5) in Table 1, the $\widehat{\beta }_{2}$ estimated from the sample $-0.03$. Comparing this number to the histogram of $\widehat{\beta }_{2}$ under the null reported in Fig. 2, we find that 27% of simulated coefficients are larger than $-0.03$. From this we conclude that the p value is 0.27: we cannot reject the null hypothesis of random assignment based on baseline math scores.

Fig. 2

Histogram of ${\hat{\hat{\beta }}} 2$ under the null for baseline math scores

In column (5) and (6) of Table 1 we report the coefficient estimates for other baseline characteristics as well as similarly calculated p values for the null hypothesis of random assignment by these characteristics. All p values are above the 10% level. From this we conclude that the random assignment of peers was implemented in a satisfactory manner.

Table 7 Comparisons of attrition between the CAL school students and control school students, students who were paired and who sat alone in CAL classes, and between students who were assigned to a high achieving or a low achieving peer

Attrition during the experiment is low. A total of 7536 sample students surveyed in the baseline participated in the endline survey. Only 4% of the students who took the baseline survey did not take the endine survey. Based on information provided by the schools, attrition is mainly due to illness, dropout, and transfers to schools outside of the town. In “Appendix” Table 7 we examine whether attrition is correlated with treatment. Column 1 shows that attrition rates do not differ statistically between CAL school students and control school students. Attrition is also not correlated with being paired or not (Table 7, column 2) or with being assigned to a high or low achieving peer Table 7, column 3).

Table 8 Balance between CAL school students and control school students, students who were paired and who sat alone in CAL classes, and between students who were assigned to a high achieving or a low achieving peer after attrition

As a final check, we repeat the balancedness tests of Table 1 using only the non-attriting sample. Results are shown in “Appendix” Table 8. The same conclusions hold: we cannot reject balance on all baseline characteristics for the first two treatments. We also repeat the permutation tests to check random peer assignment on baseline math scores. We obtain p values all above 0.1 and again fail to reject the random peer assignment hypothesis.