Beyond the Class Norm: Bullying Behavior of Popular Adolescents and its Relation to Peer Acceptance and Rejection

  • Jan Kornelis Dijkstra
  • Siegwart Lindenberg
  • René Veenstra
Open Access
Article

Abstract

This study examined to what extent bullying behavior of popular adolescents is responsible for whether bullying is more or less likely to be accepted or rejected by peers (popularity-norm effect) rather than the behavior of all peers (class norm). Specifically, the mean level of bullying by the whole class (class norm) was split into behavior of popular adolescents (popularity-norm) and behavior of non-popular adolescents (non-popularity-norm), and examined in its interaction with individual bullying on peer acceptance and peer rejection. The data stem from a peer-nominations subsample of TRAILS, a large population-based sample of adolescent boys and girls (N = 3312). The findings of multilevel regression analyses demonstrated that the negative impact of individual bullying on peer acceptance and the positive impact on peer rejection were particularly weakened by bullying by popular adolescents. These results place the class-norm effects found in previous person-group dissimilarity studies in a different light, suggesting that particularly bullying by popular adolescents is related to the social status attached to bullying.

Keywords

Bullying Popularity Behavioral group norms Acceptance Rejection 

Research on peer relations has demonstrated that acceptance or rejection of behaviors varies in accordance with the behavioral norm in the peer group or classroom (e.g., Boivin et al. 1995; Chang 2004; DeRosier et al. 1994; Jackson et al. 2006; Sentse et al. 2007; Stormshak et al. 1999). In most of these studies, researchers drew on the person-group dissimilarity model of Wright et al. (1986). This model postulates that negative social behaviors like aggression are more likely to lead to negative peer evaluations when these behaviors are not normative (infrequent) in the peer context. That is, children will be rejected when their behavior is dissimilar to that in their peer group context (they deviate from the group norm), but not if they display behavior similar to that of their peers.

These studies have, however, neglected differences in influence reflected by adolescents’ reputation-based status in the peer group. Adolescents with a high-status position among peers have more power and the ability to exert influence upon their peers directly or indirectly (Adler and Adler 1995, 1998; DeBruyn and Cillessen 2006; LaFontana and Cillessen 2002; Lease et al. 2002; Parkhurst and Hopmeyer 1998). In view of their influential position among peers, it seems reasonable to suggest that involvement of popular adolescents in certain behaviors might affect the class norm and, consequently, the acceptance and rejection of behavior.

In the present study, we examined the interplay between the class norm and status. The role of status was incorporated in the original person-group dissimilarity model to explain the diffuse relation of negative behavior (here: bullying) with peer acceptance and peer rejection. We would like to investigate whether it is not the class norm as expressed by the mean level of behavior of all peers but rather the mean level of behavior of popular adolescents that accounts for the behavioral norm effect. This implies that the class norm might actually be a popularity norm. In the following we elaborate on the way behavior of popular adolescents is related to the class-norm.

Hypotheses

The central question was; Why and in what way are the class norm and popularity norm related to how behavior is evaluated by peers? To answer this question, we used a goal-framing approach in which goals of adolescents were taken as a starting point (Lindenberg 2001, 2006). Goals are important for what people pay attention to, and the way people evaluate and judge the behavior and characteristics of others. The idea that goals govern people’s attention to and evaluation of behavior is directly applicable to peer relations (Dijkstra et al. 2007; Veenstra et al. 2007). That is, behaviors and characteristics that facilitate goal pursuit are liked, whereas goal-thwarting behaviors are disliked (Ferguson and Bargh 2004; Heidgerken et al. 2004, Newcomb et al. 1993; Rubin et al. 2006).

An important goal for adolescents is to establish relations with peers and to gain a sense of belonging within the peer group (Baumeister and Leary 1995; Berndt 1979, 1982; Coleman 1961; Corsaro and Eder 1990; Hartup 1992; Parker and Asher 1987; Rubin et al. 2006; Sullivan 1953). In the light of the importance of goals for cognitive and emotional processes, it is not surprising that being rejected has been found to have detrimental effects on the mental and social development of adolescents, and that it puts them at risk for later social and mental maladjustment (Gifford-Smith and Brownell 2003; Kupersmidt and Coie 1990; Oldehinkel et al. 2007; Parker and Asher 1987; Rubin et al. 2006).

The way bullying is linked to the goal pursuit of adolescents and, consequently, to peer acceptance and rejection is not straightforward. Adolescents strive not only for belongingness, but also for status (Buhrmester 1990; Jarvinen and Nichols 1996; Ojanen et al. 2005). Bullying is often negative for the victim. But for the perpetrator and the onlookers, who may even provide assistance or approval (Salmivalli et al. 1996), bullying may help to improve their own status. As a result, bullying can be expected to lose at least part of its negative evaluation, and may even add to being accepted.

This status effect is especially likely when popular peers are involved in bullying. For adolescents, imitating popular others is an attractive means of elevating their own status. Bullying has been found to be related to popularity both cross-sectionally (DeBruyn and Cillessen 2006; Dijkstra et al. 2008; Lease et al. 2002) and longitudinally (Cillessen and Borch 2006). Imitation of successful peers can lead to reflected glory, and enhances the person’s own status (Cialdini and Richardson 1980; Erdogan 1999). Moreover, imitation increases the chances of affiliation with popular adolescents and being part of the popular clique, which is considered attractive for adolescents (Adler and Adler 1995, 1998; Eder 1985; Merten 1997).

In view of these considerations, we expected that it would be particularly bullying by popular adolescents rather than the overall class norm that was related to the evaluation of bullying behavior by peers. This means that the class-norm effect found in previous studies is actually a popularity-norm effect. Thus, we hypothesized that individual bullying would be less negatively related to peer acceptance when popular adolescents were involved in bullying. Conversely, we expected that the positive effect of individual bullying on peer rejection would be weakened by bullying behavior of popular adolescents.

To test our hypothesis, we first examined whether the impact of bullying on acceptance and rejection by peers is influenced by the class norm of bullying (class-norm effect). Subsequently, this class norm is split into bullying behavior of popular adolescents (popularity-norm) and bullying behavior of non-popular adolescents (non-popularity-norm). We additionally investigated the extent to which the bullying behavior of popular adolescents as well as bullying behavior of non-popular adolescents affects acceptance and rejection of bullying behavior.

Method

Sample

In the current study, we used cross-sectional data from a larger cohort study, TRAILS (TRacking Adolescents’ Individual Lives Survey). TRAILS is a prospective cohort study of Dutch preadolescents who will be measured biennially until they are at least 25 years old. TRAILS is designed to chart and explain the development of mental health and social development from preadolescence into adulthood. The TRAILS target sample involved pre-adolescents living in five municipalities in the north of the Netherlands, including both urban and rural areas (De Winter et al. 2005). During the second wave of data collection, a subsample of peer nominations was collected from both TRAILS participants and their classmates. This subsample was used in the present study.

Peer nominations were assessed in classes with at least three regular TRAILS participants. Schools provided the names of classmates of TRAILS participants. All eligible students then received an information letter for themselves and their parents, in which they were asked to participate. If students or their parents wished to refrain from participation, they were requested to send a reply card within 10 days. In total, 98 students, of whom three regular TRAILS participants, refused to participate. Approximately two weeks after the information letter had been sent, a TRAILS staff member visited the selected school classes to assess the peer nominations. The assessment of the peer nominations lasted about 15 min and took place during regular lessons. Peer nominations were assessed in a total of 172 classes in 34 schools in the first year (72 school classes) and second year (100 school classes) of secondary education, which is comparable with the 7th and 8th grade in middle school in the United States. The school classes were almost equally divided among levels of education: low education (60 school classes), middle education (53 school classes), and high education (59 school classes). In total, 3,312 students (1,675 boys, 1,637 girls), including 1,007 regular TRAILS participants, filled out the questionnaire and nominated their classmates (mean age = 14.02, SD = 0.73). Each classroom had 7 to 30 participating pupils (M = 18.39; SD = 5.99). The subsample consisted of 87.3% Caucasian, 0.5% Turkish, 0.6% Moroccan, 1.7% Surinamese, 1.2% Antillean/Aruban, 2.5% Indonesian, and 4.1% other ethnic origin. For 2% of the participating students, information about their ethnic origin was unavailable. All measures in the present study were based on peer nominations from this subsample. Respondents could nominate an unlimited number of same-sex and cross-sex classmates in their responses to all questions.

Measures

Individual Characteristics

At the individual level, assessment of the dependent and independent variables was based on the number of nominations respondents received from classmates in response to the following questions: “Who do you like?” (Peer Acceptance), “Who do you dislike?” (Peer Rejection), and “Which classmates bully you?” (Bullying). After the total number of peer nominations was added, proportion scores were calculated. To facilitate the interpretation of the results of the regression analyses, scores were then standardized across the whole sample to mean 0 and SD 1 (cf. Aiken & West 1991). Comparison of peer acceptance, peer rejection, and bullying between boys and girls revealed that boys scored higher on both peer rejection, t(3310) = 5.89, p < 0.01, and bullying, t(3310) = 12.71, p < 0.01; no difference was found for peer acceptance, t(3310) = 1.45, p = 0.15. Correlational analyses showed that bullying was somewhat stronger correlated with both peer acceptance r(1675) = −0.18, p < 0.01, and peer rejection r(1675) = 0.42, p < 0.01, for boys than for girls; r(1637) = −0.11, p < 0.01, and r(1637) = 0.29, p < 0.01 (see Table 1).
Table 1

Correlations between Main Variables for Boys and Girls (N = 3,312)

 

Peer acceptance

Peer rejection

Bullying

Peer acceptance

−0.61

−0.11a

Peer rejection

−0.65

0.29a

Bullying

−0.18a

0.42a

Boys’ correlations are printed below the diagonal (N = 1675); girls’ correlations above the diagonal (N = 1637). All correlations: p < 0.01

aSignificant gender difference.

Class Norm, Popularity Norm, and Non-Popularity-Norm

The class level of bullying was based on the mean level of bullying in each class. It appeared that the class level of bullying in class varied between −0.42 and 6.10.1 Subsequently, the class norm was split into the class level of bullying by popular adolescents (popularity-norm) and bullying behavior by non-popular adolescents in the class (non-popularity-norm).

To assess both norms, we first made a selection of popular adolescents. Popularity was based on the number of nominations adolescents received from their classmates in response to the question, “Who do others want to be associated with?”. In most studies of popularity among adolescents, respondents are asked to nominate the most (and least) popular peers; this can cover many aspects, such as influence, dominance, having social power, attractiveness, and resource control (cf. LaFontana and Cillessen, 2002; Lease et al. 2002; Parkhurst and Hopmeyer 1998). Our measure was based on what adolescents presumably mean by saying that a person is popular, namely, that people want to be connected with the popular person, to be associated with that person, to “bask in reflected glory” (Cialdini and Richardson 1980). Moreover, we explicitly disentangled personal preferences for being associated with a person from reputation-based preferences by asking respondents to nominate people with whom others wanted to be connected. We believe that this yielded a reputation based measure of affiliation attractiveness or affiliative popularity. For ease of interpretation we will refer to this measure of affiliative popularity simply as ‘popularity.’

Empirical evidence for the interpretation of this measure comes from its correlation with other dimensions of peer status. Our measure of popularity correlated modestly with best friend nominations r(3312) = 0.27, and the conventional measures of social preference, r(3312) = 0.15, and social impact, r(3312) = 0.24, all p’s < 0.001. As expected, popularity correlated positively with likeability and visibility or impact. At the same time, none of the correlations exceeded the criterion for convergent validity, thus demonstrating sufficient discriminant validity of the popularity measure from the other dimensions of peer status.

Again, we computed proportion scores to account for differences in class size, and standardized the popularity score within the sample to mean 0 and SD 1. Those adolescents who scored one standard deviation above the mean were considered popular. This yielded 447 pupils (13.5%) labeled as popular. The number of popular adolescents within each school class varied from 0 to 12, with a mean level of 2.60 and a standard deviation of 2.25. In 31 (18.0%) school classes, none of the adolescents had a score one standard deviation above the mean. These classes were included in the analyses by assigning a score of zero for the variable popularity norm. The sexes were equally represented in the total number of 447 popular adolescents: 232 (7.0%) popular boys versus 215 (6.5%) popular girls, χ2 (1, N = 3312) = 0.37, p = 0.55.

Having selected the popular adolescents, we then calculated the mean level of bullying by these popular adolescents for each classroom, that is, the total sum of bullying by these popular adolescents divided by the number of popular adolescents in the class room. The mean level of bullying by popular adolescents varied between −0.42 and 5.54, with a mean level of 0.45 and a standard deviation of 1.06. For the non-popularity norm, we calculated the mean level of bullying by adolescents who were not labeled as popular. The mean level of bullying by other adolescents varied between −0.42 and 8.75, with a mean level of 0.02 and a standard deviation of 0.80.

The correlation between class level of bullying and class level of bullying by non-popular adolescents was r(172) = 0.92, p < 0.01, whereas the correlation between class level of bullying and class level of bullying by popular adolescents was r(172) = 0.63, p < 0.01. Further, the class level of bullying by non-popular adolescents and popular adolescents correlated r(172) = 0.49, p < 0.01.

Statistical Analyses

To answer the questions that guided this study, we conducted multilevel regression analyses using MlwiN 2.00 (Rasbash et al. 2000). Using multilevel analyses enabled us to control for the violation of non-independence of observations caused by the nested structure of the data, that is, individuals (level 1) within classrooms (level 2; Snijders and Bosker 1999). In this study, the dependent variables peer acceptance and peer rejection, as well as the independent variables gender and bullying, were at the individual level. The class norm of bullying (class level of bullying), non-popularity norm (class level of bullying by non-popular adolescents), and the popularity norm (class level of bullying by popular adolescents) were at the class level. Furthermore, we needed to specify cross-level interactions (between the individual level and class level) to examine the relations of these norms with peer acceptance and rejection. Cross-level interactions were assessed by multiplying individual bullying by class-level bullying, level of bullying by non-popular adolescents, and level of bullying by popular adolescents, respectively. Although we did not expect a priori gender differences, gender interactions were included in the analyses for all explanatory variables.

The multilevel analyses were conducted in different steps (see Table 2 and Table 3). First, we examined the effect of individual bullying on peer acceptance and rejection, while controlling for gender (model 1). Second, we examined to what extent the effect of bullying was moderated by the class level of bullying (model 2). In step 3 and 4, we looked at the moderation-effect of the non-popularity-norm on individual bullying (model 3) and the moderation-effect of the popularity-norm on individual bullying (model 4), respectively. In the final model, we examined the moderation effect of both the non-popularity-norm and the popularity-norm at the same time (model 5). By examining the moderating effects of these two norms simultaneously, we were able to see which norm ultimately influenced the evaluation of individual bullying behavior by peers. In Table 2 and Table 3, the results of the multilevel analyses are presented for peer acceptance and rejection.
Table 2

Results of Multilevel Regression Analyses for Peer Acceptance (N = 3312)

 

Model 1

Model 2

Model 3

Model 4

Model 5

Individual bullying

Classroom norm bullying

Non-popularity-norm bullying

Popularity-norm bullying

Both non-popularity -norm and popularity-norm bullying

Predictor

b

SE

t

b

SE

t

b

SE

t

b

SE

t

b

SE

t

Gender (1 = boy)

0.016

0.031

0.52

0.013

0.031

0.42

0.019

0.031

0.61

−0.021

0.034

0.62

−0.014

0.035

0.40

Bullying

−0.130

0.037

3.51**

−0.076

0.042

1.81+

−0.094

0.040

2.35*

−0.116

0.040

2.90**

−0.102

0.042

2.43**

Bullying × Gender

0.009

0.039

0.23

−0.087

0.047

1.85+

−0.048

0.045

1.07

−0.054

0.042

1.29

−0.071

0.046

1.54

CN Bullying

   

−0.289

0.079

3.66**

         

CN Bullying × Gender

   

0.192

0.072

2.67**

         

Bullying × CN Bullying

   

0.022

0.012

1.83+

         

NPN Bullying

      

−0.220

0.068

3.24**

   

−0.197

0.076

2.59**

NPN Bullying × Gender

      

0.109

0.072

1.51

   

0.064

0.076

0.84

Bullying × NPN Bullying

      

0.012

0.011

1.09

   

−0.002

0.012

0.17

PN Bullying

         

−0.121

0.049

2.47*

−0.057

0.055

1.04

PN Bullying × Gender

         

0.099

0.035

2.83**

0.083

0.036

2.31*

Bullying × PN Bullying

         

0.027

0.010

2.70**

0.031

0.012

2.58**

Deviance

8,481

8,460

8,468

8,461

8,454

Decrease in Deviance

53 (df = 3)**

21 (df = 3)**

13 (df = 3)**

20 (df = 3)**

14 (df = 3)**

7 (df = 3)+

Note. Decrease in deviance indicates whether or not the model fits the data better than the former model. The decrease in deviance has approximately a chi-square distribution with the degrees of freedom equal to the difference in the number of parameters of the models. The decreases in deviance of models 2, 3, and 4 are based on a comparison with model 1, whereas for model 5 the decrease in deviance is based on a comparison with model 3 and 4, respectively.

CN Class norm; NPN non-popularity norm; PN popularity norm

+p < 0.10; *p < 0.05; **p < 0.01

Table 3

Results of Multilevel Regression Analyses for Peer Rejection (N = 3312)

 

Model 1

Model 2

Model 3

Model 4

Model 5

 

Individual bullying

Class-norm bullying

Non-popularity-norm bullying

Popularity-norm bullying

Both non-popularity-norm and popularity-norm bullying

Predictor

b

SE

t

b

SE

t

b

SE

T

b

SE

t

b

SE

t

Gender (1 = boy)

0.034

0.033

1.03

0.040

0.033

1.21

0.028

0.033

0.85

0.094

0.036

2.61**

0.072

0.037

1.95*

Bullying

0.422

0.038

11.10**

0.326

0.043

7.58**

0.335

0.041

8.17**

0.393

0.042

9.36**

0.341

0.044

7.75**

Bullying × Gender

−0.068

0.042

1.62

0.105

0.049

2.14**

0.069

0.047

1.47

0.043

0.045

0.96

0.104

0.048

2.17**

CN Bullying

   

0.390

0.060

6.50**

         

CN Bullying * Gender

   

−0.319

0.074

4.31**

         

Bullying * CN Bullying

   

−0.046

0.011

4.18**

         

NPN Bullying

      

0.340

0.051

6.67**

   

0.268

0.055

4.87**

NPN Bullying × Gender

      

−0.254

0.071

3.58**

   

−0.200

0.075

2.67**

Bullying × NPN Bullying

      

−0.029

0.009

3.22**

   

−0.007

0.011

0.64

PN Bullying

         

0.203

0.034

5.97**

0.134

0.037

3.62**

PN Bullying × Gender

         

−0.163

0.037

4.41**

−0.117

0.039

3.00**

Bullying × PN Bullying

         

−0.046

0.009

5.11**

−0.044

0.012

3.67**

Deviance

8,771

8,709

8,714

8,709

8,685

Decrease in Deviance

444 (df = 31)**

62 (df = 3)**

57 (df = 3)**

62 (df = 3)**

29 (df = 3)**

24 (df = 3)**

Note. Decrease in deviance indicates whether or not the model fits the data better than the former model. The decrease in deviance has approximately a chi-square distribution with the degrees of freedom equal to the difference in the number of parameters of the models. The decreases in deviance of models 2, 3, and 4 are based on a comparison with model 1, whereas for model 5 the decrease in deviance is based on a comparison with model 3 and 4, respectively.

NPN Non-popularity norm, PN popularity norm

*p < 0.05; **p < 0.01

Results

Peer Acceptance

Bullying was negatively related to peer acceptance, b = −0.13, t(3310) = 3.51, p < 0.01. We then tested for the class-norm effect by including the cross-level interaction of individual bullying with class-level bullying in the model (model 2). A marginally significant class-norm effect was found (b = 0.02, t(3310) = 1.83, p < 0.10). In a similar way, we tested whether the individual effect of individual bullying was affected by the class level of bullying of non-popular adolescents. This appeared not to be the case. The non-popularity norm did not influence the effect of individual bullying on peer acceptance. By contrast, bullying behavior of popular adolescents weakened the negative effect of individual bullying on peer acceptance (b = 0.03, t(3310) = 2.70, p < 0.01) (model 4).

When we examined the non-popularity-norm effect and popularity-norm effect simultaneously, only the popularity-norm effect was significant (b = 0.03, t(3310) = 2.67, p < 0.01) (model 5). These results indicate that it is the popularity norm rather than the non-popularity-norm that moderates the acceptance of bullying behavior by peers.

To interpret the interaction-effect of individual bullying with bullying behavior of popular adolescents (model 5), we wrote out multiple equations based on the parsimonious model (not presented here), alternating the values of the main effects (one standard deviation below and above the mean) and holding all other variables in the models to their sample means. This allowed us to draw Fig. 1. It appeared that the more popular adolescents were involved in bullying, the less negatively individual bullying was related to peer acceptance (b = −0.19, t(3310) = 6.82, p < 0.01 for a low level of bullying by popular adolescents versus b = −0.13, t(3310) = 7.06, p < 0.01 for a high level of bullying by popular adolescents). Involvement of popular adolescents in bullying weakened the negative effect of individual bullying on peer acceptance.
Fig. 1

Graphical presentation of the interaction of individual level of bullying, and class level of bullying by popular adolescents in relation to peer acceptance

To determine whether or not the model fit the data better than did the previous model, we calculated the decrease in deviance. The decrease in deviance has approximately a chi-square distribution, with the degrees of freedom equal to the difference in the number of parameters of the models. A significant decrease in deviance indicates a significant improvement of the model. The three models in which the class-norm (model 2), non-popularity-norm (model 3), and popularity-norm (model 4) were tested separately improved significantly compared to the model with only the individual effect of bullying (model 1). Further, it appeared that the model with both the non-popularity norm and the popularity-norm (model 5) proved to fit the data better than model 3 (χ2 = 14, df = 3, p < 0.05) and marginally also model 4 (χ2 = 7, df = 3, p = 0.07).

Peer Rejection

We focused next on the effect of bullying on peer rejection (Table 3). As expected, bullying was positively related to peer rejection, b = 0.42, t(3310) = 11.10, p < 0.01. Then, we examined whether the effect of individual bullying was further moderated by the class-norm, non-popularity-norm, and the popularity-norm, respectively. It appeared that all norms influenced the relation between bullying and peer rejection separately. That is, we found a significant interaction-effect between individual bullying and the class norm (b = −0.05, t(3310) = 4.18, p < 0.01) (model 2), the non-popularity norm (b = −0.03, t(3310) = 3.22, p < 0.01; model 3) and the popularity-norm (b = −0.05, t(3310) = 5.11, p < 0.01; model 4).

To see whether the popularity-norm or non-popularity-norm, ultimately moderates the relation between individual bullying and peer rejection, both norms were tested in model 5 simultaneously. It was found that the non-popularity-norm effect disappeared (b = −0.01, t(3310) = 0.64, p = 0.26), when tested with the popularity norm at the same time (b = −0.04, t(3310) = 3.67, p < 0.01; model 5).

Thus, it appeared that when the class norm was split into bullying behavior of popular and non-popular adolescents, bullying of popular adolescents ultimately influenced the way individual bullying was linked to peer rejection. To examine this interaction of individual bullying with the popularity-norm (model 5), we again wrote out multiple equations based on the parsimonious model (not presented here). This is depicted in Fig. 2.
Fig. 2

Graphical presentation of the interaction of individual level of bullying, and class level of bullying by popular adolescents in relation to peer rejection

As can be seen, individual bullying was more strongly related to peer rejection when bullying by popular adolescents was low (b = 0.44, t(3310) = 9.84, p < 0.01) than when bullying by popular adolescents was high (b = 0.33, t(3310) = 8.00, p < 0.01).

We again tested the improvement of the model using the decrease in deviance. It is shown in Table 3 that each model improved significantly. The final model with the both the non-popularity-norm and the popularity-norm also fit the data better than did the model that included the effects of both norms separately (χ2 = 29, df = 3, p < 0.01 and χ2 = 24, df = 3, p < 0.01).

Discussion

The starting point for this study was the finding that context matters for the evaluation of behavior by peers. Prior research findings have suggested that the class norm is an important contextual factor for explaining the somewhat ambiguous relations of behaviors with peer acceptance and rejection (e.g., Boivin et al. 1995; Chang 2004; DeRosier et al. 1994; Jackson et al. 2006; Sentse et al. 2007). However, it might not be the class norm as such that accounts for the different associations between peer status and behavior across different classes (class-norm effect); it may be mostly the behavior of popular adolescents that is responsible for whether behavior is more or less likely to be accepted or rejected by peers (popularity-norm effect). We came to these predictions after modifying the person-group dissimilarity model of Wright and his colleagues (1986) using goal-framing theory (Lindenberg 2001, 2006), by specifying the bullying behavior of popular adolescents.

We argued that it is not the behavior of all members of a class that influences the evaluation of behaviors by peers, but rather the involvement of high-status peers in behavior. The usefulness of negative behavior as means for goal-achievement reduces its negative connotations. Thus, when status is pursued by popular adolescents through bullying, this activity loses some of its negative connotations because it is instrumental for achieving a valued goal. This may even lead to the imitation of the use of such a successful means. In that sense, involvement of popular adolescents in bullying seems to take the sharp edges off negative behaviors. This also adds to our understanding of the paradox raised by Salmivalli and Voeten (2004) that the majority of children disapprove of bullying, but simultaneously have active roles in the bullying process.

We examined the effects of individual bullying on peer acceptance and rejection in relation to bullying behavior of the whole class (class norm) as well as to bullying behavior of popular adolescents (popularity norm) and bullying behavior of the non-popular adolescents in the class (non-popularity-norm). First of all, it appeared that individual bullying was negatively related to peer acceptance, whereas it was positively associated with peer rejection. Beyond these main effects of individual bullying, our results suggest that particularly bullying behavior of popular adolescents had an additional impact on the way bullying was related to peer acceptance and rejection.

We found that the initially, significant class norm effect on peer acceptance could be attributed to the bullying behavior of popular adolescents. That is, when the class norm was further specified by bullying behavior of non-popular adolescents and popular adolescents, the negative effect of individual bullying on peer acceptance was only weakened by the popularity-norm. No effect was found for the non-popularity-norm. Apparently, it is not the behavior of the complete class that affects the evaluation of behaviors by peers, but particularly behavior of popular adolescents that sets the norm within the class.

For peer rejection, all three norms were, when tested separately, initially related to the way bullying was evaluated by peers. However, when the popularity-norm and the non-popularity norm were examined simultaneously, the non-popularity norm was crowded out by the popularity-norm. That is, the effect of individual bullying on peer rejection was moderated by bullying behavior of popular adolescents. Specifically, we found that the positive effect of individual bullying on peer rejection was weakened in classes with high levels of bullying by popular adolescents. Similar to peer acceptance, these results emphasize that the norm in the class is not so much the outcome of behavior of all peers rather than the behavior of popular adolescents.

Although it was found in earlier studies that girls’ behavior is more dependent on the context (Salmivalli et al. 1998; Salmivalli and Voeten 2004), we did not find large gender differences in this particular study on adolescents. One reason might be our measure of bullying. Information on bullying is usually drawn from reputation-based measures in the classroom, which facilitates assessment of more overt forms of bullying, such as hitting and kicking, that are more common among boys than girls (Lagerspetz et al. 1988). Hence, these reputation-based measures might overestimate gender differences in bullying. In the present study, however, bullying was measured at the dyadic level, which enabled us to capture the more covert types of bullying, too. Gender differences might not have shown up for this reason.

The findings of our study place the class-norm effects found in previous studies in a different light by suggesting that especially the involvement of popular adolescents plays an important role in the acceptance and rejection of behavior by peers. Despite these effects of the class and popularity norm, individual levels of bullying are consistently negatively related to peer acceptance and positively related to peer rejection (see also Figs. 1 and 2). This also suggests that copying the bullying behavior of popular children does not have a strong salutary effect on unpopular children. Previous research has also shown that popularity among peers is rooted in the combination of antisocial behavior with positive features (Dijkstra et al. 2008; Vaillancourt and Hymel 2006). Therefore, imitation of negative behavior of popular adolescents without having positive features that reduce the negative effects of these behaviors on peer relations is likely to be a counterproductive strategy for adolescents to increase their status among peers.

Moreover, our study clearly revealed that bullying behavior of popular adolescents goes with less peer acceptance and more peer rejection in school classes. As can be seen in Figs. 1 and 2, classes with high levels of bullying by popular adolescents, are lower in peer acceptance and higher in peer rejection. This suggests that although involvement of popular adolescents in bullying takes the sharp edges of this negative behavior for the bullies, it harms the establishment of positive relationships among peers within the class as a whole.

The practical implications of this study are rather straightforward. We conclude that the view of bullying held by peers depends on the involvement of popular adolescents in bullying. This implies that behavioral changes in the popular clique of a school class might cause behavioral changes in the classroom as a whole. To this end, targeting interventions on these popular adolescents and making them aware of their influence could reduce bullying in school classes. Note that in this study, the variables were standardized across the sample for statistical reasons. Standardization within classroom would remove variability between classes, because for all classes the mean is set to zero. As a consequence, we would not be able to calculate reliable estimates for our multilevel model. For intervention purposes, however, identification of popular adolescents and particularly popular bullies, asks for standardization within the class.

In addition, it is important to make everyone in the school class aware of its role in the bullying process, from bully to bystander (cf. Salmivalli et al. 1996). However, to find out in more detail the role of adolescents in bullying in their classes asks for detailed network information about relations with peers, the content of these relations, the type of bullying behavior and so on. Gaining such information might profit from data collections that take explicitly the dyadic nature of the relations into account by asking not ‘who is a bully’, but by ‘whom are you bullied’ (see also Veenstra et al. 2007). In a similar way, information can be obtained about victimization, and other roles within the bullying process. Such data provide more accurate information that can be helpful to develop interventions aimed at bullying as a group process.

Despite that this study clearly demonstrated the involvement of popular adolescents in bullying, popular adolescents have also been characterized by prosocial behavior (LaFontana and Cillessen 2002; Lease, Kennedy and Axelrod 2002; Luthar and McMahon 1996; Parkhurst and Hopmeyer 1998; Rodkin et al. 2000). Rather than focusing on ‘bad’ behavior, promoting ‘good’ behavior might help as well, that is, creating opportunities for prosocial behavior, such as working in subgroups. This does not only foster positive exchange between peers, but could also provide opportunities to increase acceptance among classmates and protect against social exclusion (see Newcomb et al. 1993; Skinner et al. 2002).

Limitations and Strengths

This study has some limitations that should be considered. First, the data used in the study were cross-sectional, so that causal statements are not appropriate. However, in view of the aim of the study, that is, to examine the extent to which bullying was differently evaluated by peers dependent on the class norm or the involvement of popular peers in bullying, this limitation does not jeopardize our results. Still, we need to consider an alternative explanation that popular adolescents do not set the norm, but behave according to the class norm, which, in turn, leads to their popularity. In that sense, popular adolescents do not set the norm but follow it. Although longitudinal data are necessary to test this alternative approach, results of our study make this explanation rather implausible. We found that the mean level of bullying behavior of popular adolescents was much higher than the overall level in the class. This seems to suggest that the class level is to a large extent based on bullying behavior of popular adolescents. Thus, without the contribution of popular adolescents to bullying the class level would be much lower. The idea that popular adolescents follow the norm would not imply such large differences in level of bullying, but more similar levels.

Second, the data used in our study stem from peer nominations only, which might lead to problems regarding shared method variance. For the study of peer relations, however, the use of peer nominations is probably one of the most valid and reliable methods of disentangling the association between behaviors and peer relations (Bukowski et al. 1993; Bukowski and Hoza 1989).

Third, our study was built upon the differences between popular adolescents and non-popular adolescents. Although it could be argued that these decisions were rather arbitrary, it appeared that the number of popular adolescents was rather low. As a consequence, we think we included only those adolescents who were very popular. This reduced the number of so-called “false positives”, that is, the “unjustified” inclusion of some as popular adolescents. We feel it is safe to say, therefore, that the status of adolescents influences the evaluation, judgment, and appreciation of behavior by peers.

The important lesson from these findings is that the way behavior is judged and evaluated by peers is particularly dependent on who is involved rather than the overall level of behavior within the classroom. The findings show that the norm in the class is to a large extent set by popular adolescents. Future researchers might do well to further address the role of popular adolescents in how peers value, judge, and evaluate negative behaviors.

Footnotes

  1. 1.

    One class (N = 10) was extraordinarily high in bullying (z-score of 6.10). When we calculated the mean score of bullying based on the individual proportion scores (running form 0 to 1), this class had an average level of bullying of.36, which was exceptionally high compared to the rest of the sample (N = 3302; M = 0.02 and SD = 0.05). This class was also low in peer acceptance (M = 0.28 and SD = 0.11 compared to M = 0.55 with SD = 0.20 for the rest of the sample) and high in peer rejection (M = 0.50 with SD = 0.12 compared to M = 0.11 with SD = 0.13). Additional analyses without this class did not lead to noteworthy changes in the outcomes of the analyses. Hence, the class was not removed from the sample.

Notes

Acknowledgments

This research is part of the TRacking Adolescents’ Individual Lives Survey (TRAILS). Participating centers of TRAILS include various departments of the University Medical Center and University of Groningen, the Erasmus University Medical Center Rotterdam, the University of Utrecht, the Radboud Medical Center Nijmegen, and the Trimbos Institute, all in the Netherlands. Principal investigators are prof. dr. J. Ormel (University Medical Center Groningen) and prof. dr. F.C. Verhulst (Erasmus University Medical Center). TRAILS has been financially supported by various grants from the Netherlands Organization for Scientific Research NWO (Medical Research Council program grant GB-MW 940-38-011; ZonMW Brainpower grant 100-001-004; ZonMw Risk Behavior and Dependence grants 60-60600-98-018 and 60-60600-97-118; ZonMw Culture and Health grant 261-98-710; Social Sciences Council medium-sized investment grants GB-MaGW 480-01-006 and GB-MaGW 480-07-001; Social Sciences Council project grants GB-MaGW 457-03-018, GB-MaGW 452-04-314, an GB-MaGW 452-06-004; NWO large-sized investment grant 175.010.2003.005); the Sophia Foundation for Medical Research (projects 301 and 393), the Dutch Ministry of Justice (WODC), and the participating universities. We are grateful to all adolescents, their parents and teachers who participated in this research and to everyone who worked on this project and made it possible. Further, we would like to thank Gijs Huitsing, Katya Ivanova, Anke Munniksma, Miranda Sentse, and Jelle Sijtsema, as well as the reviewers for their helpful comments on earlier drafts of this paper. An earlier version of this study was presented at the Educational Research Days (Onderwijs Research Dagen (ORD), Groningen, the Netherlands, June 2007.

Open Access

This article is distributed under the terms of the Creative Commons Attribution Noncommercial License which permits any noncommercial use, distribution, and reproduction in any medium, provided the original author(s) and source are credited.

References

  1. Adler, P. A., & Adler, P. (1995). Dynamics of inclusion and exclusion in preadolescent cliques. Social Psychology Quarterly, 58, 145–162. doi:10.2307/2787039.CrossRefGoogle Scholar
  2. Adler, P. A., & Adler, P. (1998). Peer power: preadolescent culture and identity. New Brunswick, NJ: Rutgers University Press.Google Scholar
  3. Aiken, L. S., & West, S. G. (1991). Multiple regression: testing and interpreting interactions. Thousand Oaks, CA, US: Sage.Google Scholar
  4. Baumeister, R. F., & Leary, M. R. (1995). The need to belong. Desire for interpersonal attachments as a fundamental human motivation. Psychological Bulletin, 117, 497–529. doi:10.1037/0033-2909.117.3.497.PubMedCrossRefGoogle Scholar
  5. Berndt, T. J. (1979). Developmental-changes in conformity to peers and parents. Developmental Psychology, 15, 608–616. doi:10.1037/0012-1649.15.6.608.CrossRefGoogle Scholar
  6. Berndt, T. J. (1982). The features and effects of friendship in early adolescence. Child Development, 53, 1447–1460. doi:10.2307/1130071.CrossRefGoogle Scholar
  7. Boivin, M., Dodge, K. A., & Coie, J. D. (1995). Individual–group behavioral similarity and peer status in experimental play groups of boys: the social misfit revisited. Journal of Personality and Social Psychology, 69, 269–279. doi:10.1037/0022-3514.69.2.269.PubMedCrossRefGoogle Scholar
  8. Bukowski, W. M., & Hoza, B. (1989). Popularity and friendship: issues in theory, measurement, and outcome. In T. J. Berndt, & G. W. Ladd (Eds.), Peer relations in child development, (pp. 15–45). New York: Wiley.Google Scholar
  9. Bukowski, W. M., Gauze, C., Hoza, B., & Newcomb, A. F. (1993). Differences and consistency between same-sex and other-sex peer relationships during early adolescence. Developmental Psychology, 29, 255–263. doi:10.1037/0012-1649.29.2.255.CrossRefGoogle Scholar
  10. Buhrmester, D. (1990). Intimacy of friendship, interpersonal competence, and adjustment during preadolescence and adolescence. Child Development, 61, 1101–1111. doi:10.2307/1130878.PubMedCrossRefGoogle Scholar
  11. Chang, L. (2004). The role of class norms in contextualizing the relations of children’s social behaviors to peer acceptance. Developmental Psychology, 40, 691–702. doi:10.1037/0012-1649.40.5.691.PubMedCrossRefGoogle Scholar
  12. Cialdini, R. B., & Richardson, K. D. (1980). Two indirect tactics of impression management: basking and blasting. Journal of Personality and Social Psychology, 39, 406–416. doi:10.1037/0022-3514.39.3.406.CrossRefGoogle Scholar
  13. Cillessen, A. H. N., & Borch, C. (2006). Developmental trajectories of adolescent popularity: a growth curve modelling analysis. Journal of Adolescence, 29, 935–959.PubMedGoogle Scholar
  14. Coleman, J. S. (1961). The adolescent society. The social life of the teenager and its impact on education. New York: Free Press.Google Scholar
  15. Corsaro, W. A., & Eder, D. (1990). Children’s peer cultures. Annual Review of Sociology, 16, 197–220. doi:10.1146/annurev.so.16.080190.001213.CrossRefGoogle Scholar
  16. De Bruyn, E. H., & Cillessen, A. H. N. (2006). Heterogeneity of girls’ consensual popularity: academic and interpersonal behavioral profiles. Journal of Youth and Adolescence, 35, 435–445. doi:10.1007/s10964-005-9023-4.CrossRefGoogle Scholar
  17. DeRosier, M. E., Cillessen, A. H. N., Coie, J. D., & Dodge, K. A. (1994). Group social context and children’s aggressive behavior. Child Development, 65, 1068–1079. doi:10.2307/1131305.PubMedCrossRefGoogle Scholar
  18. De Winter, A., Oldehinkel, A. J., Veenstra, R., Brunnekreef, J. A., Verhulst, F. C., & Ormel, J. (2005). Evaluation of non-response bias in mental health determinants and outcomes in a large sample of pre-adolescents. European Journal of Epidemiology, 20, 173–181. doi:10.1007/s10654-004-4948-6.PubMedCrossRefGoogle Scholar
  19. Dijkstra, J. K., Lindenberg, S., & Veenstra, R. (2007). Same-gender and cross-gender peer acceptance and peer rejection and their relation to bullying and helping among preadolescents: comparing predictions from gender-homophily and goal-framing approaches. Developmental Psychology, 43, 1377–1389. doi:10.1037/0012-1649.43.6.1377.PubMedCrossRefGoogle Scholar
  20. Dijkstra, J. K., Lindenberg, S., Verhulst, F. C., Ormel, J., & Veenstra, R. (2008) The relation between popularity and aggressive, destructive, and norm-breaking behaviors: moderating effects of athletic abilities, physical attractiveness, and prosociality. Journal of Research on Adolescence, in press.Google Scholar
  21. Erdogan, B. Z. (1999). Celebrity endorsement: a literature review. Journal of Marketing Management, 15, 291–314.CrossRefGoogle Scholar
  22. Eder, D. (1985). The cycle of popularity: interpersonal relations among female adolescents. Sociology of Education, 58, 154–165. doi:10.2307/2112416.CrossRefGoogle Scholar
  23. Ferguson, M. J., & Bargh, J. A. (2004). Liking is for doing: the effects of goal pursuit on automatic evaluation. Journal of Personality and Social Psychology, 87, 557–572. doi:10.1037/0022-3514.87.5.557.PubMedCrossRefGoogle Scholar
  24. Gifford-Smith, M. E., & Brownell, C. A. (2003). Childhood peer relationships: social acceptance, friendships, and peer networks. Journal of School Psychology, 41, 235–284. doi:10.1016/S0022-4405(03)00048-7.CrossRefGoogle Scholar
  25. Hartup, W. W. (1992). Friendships and their developmental significance. In H. McGurk, & N. J. Hillsdale (Eds.), Childhood social development: contemporary perspectives (pp. 175–205). London, UK: Erlbaum.Google Scholar
  26. Heidgerken, A. D., Hughes, J. N., Cavell, T. A., & Willson, V. L. (2004). Direct and indirect effects of parenting and children's goals on child aggression. Journal of Clinical Child and Adolescent Psychology, 33, 684–693. doi:10.1207/s15374424jccp3304_4.PubMedCrossRefGoogle Scholar
  27. Jackson, M. F., Barth, J. M., Powell, N., & Lochman, J. E. (2006). Classroom contextual effects of race on children’s peer nominations. Child Development, 77, 1325–1337. doi:10.1111/j.1467-8624.2006.00937.x.PubMedCrossRefGoogle Scholar
  28. Jarvinen, D. W., & Nicholls, J. G. (1996). Adolescents’ social goals, beliefs about the causes of social success, and satisfaction in peer relations. Developmental Psychology, 32, 435–441. doi:10.1037/0012-1649.32.3.435.CrossRefGoogle Scholar
  29. Kupersmidt, J. B., & Coie, J. D. (1990). Preadolescent peer status, aggression, and school adjustment as predictors of externalizing problems in adolescence. Child Development, 61, 1350–1362. doi:10.2307/1130747.PubMedCrossRefGoogle Scholar
  30. LaFontana, K. A., & Cillessen, A. H. N. (2002). Children's perceptions of popular and unpopular peers: a multimethod assessment. Developmental Psychology, 38, 635–647. doi:10.1037/0012-1649.38.5.635.PubMedCrossRefGoogle Scholar
  31. Lagerspetz, K. M., Bjorkqvist, K., & Peltonen, T. (1988). Is indirect aggression typical of females? Gender differences in aggressiveness in 11- to 12-year-old children.. Aggressive Behavior, 14, 403–414. doi:10.1002/1098-2337(1988)14:6<403::AID-AB2480140602>3.0.CO;2-D.CrossRefGoogle Scholar
  32. Lease, A. M., Kennedy, C. A., & Axelrod, J. L. (2002). Children's social constructions of popularity. Social Development, 11, 87–109. doi:10.1111/1467-9507.00188.CrossRefGoogle Scholar
  33. Lindenberg, S. (2001). Social rationality versus rational egoism. In J. Turner (Ed.), Handbook of sociological theory (pp. 635–668). New York: Kluwer/Plenum.Google Scholar
  34. Lindenberg, S. (2006). Prosocial behavior, solidarity, and framing processes. In D. Fetchenhauer, A. Flache, A. P. Buunk, & S. Lindenberg (Eds.), Solidarity and prosocial behavior: an integration of sociological and psychological perspectives. (pp. 23–44). New York: Springer.Google Scholar
  35. Luthar, S. S., & McMahon, T. J. (1996). Peer reputation among inner-city adolescents: Structure and correlates. Journal of Research on Adolescence, 6, 581–603.Google Scholar
  36. Merten, D. E. (1997). The meaning of meanness: popularity, competition, and conflict among junior high school girls. Sociology of Education, 70, 175–191. doi:10.2307/2673207.CrossRefGoogle Scholar
  37. Newcomb, A. F., Bukowski, W. M., & Pattee, L. (1993). Children’s peer relations—a meta-analytic review of popular, rejected, neglected, controversial, and average sociometric status. Psychological Bulletin, 113, 99–128. doi:10.1037/0033-2909.113.1.99.PubMedCrossRefGoogle Scholar
  38. Ojanen, T., Grönroos, M., & Salmivalli, C. (2005). An interpersonal circumplex model of children’s social goals: links with peer-reported behavior and sociometric status. Developmental Psychology, 41, 699–710. doi:10.1037/0012-1649.41.5.699.PubMedCrossRefGoogle Scholar
  39. Oldehinkel, A. J., Rosmalen, J. G. M., Veenstra, R., Dijkstra, J. K., & Ormel, J. (2007). Being admired or being liked. classroom social status and depressive problems in early adolescent girls and boys. Journal of Abnormal Child Psychology, 35, 417–427. doi:10.1007/s10802-007-9100-0.PubMedCrossRefGoogle Scholar
  40. Parker, J. G., & Asher, S. R. (1987). Peer relations and later personal adjustment: are low-accepted children at risk. Psychological Bulletin, 102, 357–389. doi:10.1037/0033-2909.102.3.357.PubMedCrossRefGoogle Scholar
  41. Parkhurst, J. T., & Hopmeyer, A. (1998). Sociometric popularity and peer-perceived popularity: two distinct dimensions of peer status. The Journal of Early Adolescence, 18, 125–144. doi:10.1177/0272431698018002001.CrossRefGoogle Scholar
  42. Rasbash, J., Browne, W., Goldstein, H., Yang, M., Plewis, I., Healy, M., et al. (2000). A user’s guide to MlwiN. London: Institute of Education.Google Scholar
  43. Rodkin, P. C., Farmer, T. W., Pearl, R., & Van Acker, R. (2000). Heterogeneity of popular boys: Antisocial and prosocial configurations. Developmental Psychology, 36, 14–24.PubMedCrossRefGoogle Scholar
  44. Rubin, K. H., Bukowski, W., & Parker, J. G. (2006). Peer interactions, relationships and groups. In: N. Eisenberg, Damon, W., Lerner, R. M., Hoboken, N. J. (Eds.), Handbook of child psychology. Social, emotional and personality development. (6 ed., pp. 571–645). New York: Wiley.Google Scholar
  45. Salmivalli, C., Lagerspetz, K., Björkqvist, K., Österman, K., & Kaukiainen, A. (1996). Bullying as a group process: participant roles and their relations to social status within the group. Aggressive Behavior, 22, 1–15. doi:10.1002/(SICI)1098-2337(1996)22:1<1::AID-AB1>3.0.CO;2-T.CrossRefGoogle Scholar
  46. Salmivalli, C., Lappalainen, M., & Lagerspetz, K. M. J. (1998). Stability and change of behavior in connection with bullying in schools: a two-year follow-up. Aggressive Behavior, 24, 205–218. doi:10.1002/(SICI)1098-2337(1998)24:3<205::AID-AB5>3.0.CO;2-J.CrossRefGoogle Scholar
  47. Salmivalli, C., & Voeten, M. (2004). Connections between attitudes, group norms, and behaviour in bullying situations. International Journal of Behavioral Development, 28, 246–258. doi:10.1080/01650250344000488.CrossRefGoogle Scholar
  48. Sentse, M., Scholte, R., Salmivalli, C., & Voeten, M. (2007). Person–group dissimilarity on involvement in bullying and its relation with social status. Journal of Abnormal Child Psychology, 35, 1009–1019. doi:10.1007/s10802-007-9150-3.PubMedCrossRefGoogle Scholar
  49. Skinner, C. H., Neddenriep, C. E., Robinson, S. L., Ervin, R., & Jones, K. (2002). Altering educational environments through positive peer reporting: prevention and remediation of social problems associated with behavior disorders. Psychology in the Schools, 39, 191–202. doi:10.1002/pits.10030.CrossRefGoogle Scholar
  50. Snijders, T. A. B., & Bosker, R. J. (1999). Multilevel analyses: an introduction to basic and advanced multilevel modeling. Sage, London.Google Scholar
  51. Stormshak, E. A., Bierman, K. L., Bruschi, C., Dodge, K. A., & Coie, J. D. (1999). The relation between behavior problems and peer preference in different classroom contexts. In Conduct Problems Prevention Research Group (Ed.), Child Development, 70, 169–182. doi:10.1111/1467-8624.00013.PubMedCrossRefGoogle Scholar
  52. Sullivan, H. S. (1953). The interpersonal theory of psychiatry. Oxford, UK: Norton.Google Scholar
  53. Vaillancourt, T., & Hymel, S. (2006). Aggression and social status: the moderating roles of sex and peer-valued characteristics. Aggressive Behavior, 32, 396–406. doi:10.1002/ab.20138.CrossRefGoogle Scholar
  54. Veenstra, R., Lindenberg, S., Zijlstra, B. J. H., De Winter, A., Verhulst, F. C., & Ormel, J. (2007). The dyadic nature of bullying and victimization: testing a dual perspective theory. Child Development, 78, 1843–1854. doi:10.1111/j.1467-8624.2007.01102.x.PubMedCrossRefGoogle Scholar
  55. Wright, J. C., Giammarino, M., & Parad, H. W. (1986). Social status in small groups: individual–group similarity and the social “misfit.”. Journal of Personality and Social Psychology, 50, 523–536. doi:10.1037/0022-3514.50.3.523.CrossRefGoogle Scholar

Copyright information

© The Author(s) 2008

Authors and Affiliations

  • Jan Kornelis Dijkstra
    • 1
  • Siegwart Lindenberg
    • 1
  • René Veenstra
    • 1
  1. 1.University of GroningenGroningenThe Netherlands

Personalised recommendations