How students perceive instruction determines the nature and quality of their learning processes. Instruction does not influence learning directly, but students’ perceptions of instruction influence learning and study behavior and eventually learning outcomes (Doyle 1977; Elen and Lowyck 1999; Entwistle and Tait 1990). Additionally, students tend to stick on learning preferences and habits and only use those elements of instruction that are in line with their habitual way of learning (Vermetten et al. 2002). It is therefore important for teachers to know how students perceive their instruction (i.e., to what degree different aspects of instruction, such as student autonomy, are observed to be present in a course by the students). However, teachers often are not sufficiently aware of the perceptions, desires, and possible criticisms of the students in their class (Holt et al. 2005; Watkins 2004). It seems valuable to make students’ perceptions and desires with regard to their instruction more explicit, since those perceptions and desires appear to have a direct effect on learning processes and are difficult to predict for teachers (Donaldson 1978; Kershner and Pointon 2000; Oldfather 1995a).

Nevertheless, there have been only few attempts to listen to students’ voices on education and a discourse between teachers and students about learning experiences is often lacking (Cook-Sather 2001). Students’ perceptions deserve a much more prominent place in instructional re/design than they currently have. There are signs that taking account of students’ perceptions and bridging the gap between teachers’ and students’ perceptions can contribute to the improvement of instructional design. In light of this notion, the current study investigated the effects of participation of students in the instructional redesign of a course, particularly how this participation affected students’ and teachers’ perceptions of the newly designed course. It was expected that participatory design would lead to more positive student perceptions of the re-designed lessons, increase the fit between students’ perceptions and desires with regard to the lessons, and diminish differences between the perceptions of teachers and students.

Discrepancies between students’ and teachers’ perceptions of instruction are likely to be detrimental to the teaching–learning process in several ways. First, the degree of congruence between students’ and teachers’ perceptions influences how well the behaviors of students and teachers are attuned to each other. Studies have shown that perceptions trigger corresponding behaviors (Bargh et al. 1996; Dijksterhuis and Van Knippenberg 1998), so discrepancies between perceptions of students and teachers are likely to cause divergent behavior patterns in the same teaching–learning environment. According to notions from human-factors engineering (e.g., Norman 1986, 1988), designers’ and users’ interpretations of any system have to be similar in order to achieve optimal functioning and effectiveness of a system (Bartholomew et al. 2001). This is also acknowledged in the field of education: Differences between the intentions of the instructional designers of an educational intervention and the interpretation of the students can cause a mismatch and, as a result, suboptimal use of the intervention (Elen and Lowyck 1999). Consequently, interventions may be ignored or used in a different way than intended. This is unfortunate, because instructional interventions are most effective if students use them as intended by the designer (Butler and Winne 1995; Winne and Marx 1982).

Second, large differences between students’ and teachers’ views of learning and teaching can have a negative effect on students’ learning processes. While small differences between students’ learning strategies and teaching strategies can challenge students to enhance their learning and thinking skills (Vermunt and Verloop 1999), this constructive friction can evolve into destructive friction if the differences are so large as to hamper students’ learning and thinking skills. Effects of this type are called mathemathantic effects (e.g., Lohman 1986). Clark (2001) explained these effects by referring to levels of self-efficacy of students. When task requirements are considered to be extremely high or even impossible to meet (for example, with novel and difficult tasks), students’ self-efficacy may reach such a low level that mental effort stops and attention is automatically directed at different or novel goals. To summarize, if discrepancies between teachers’ and students’ perceptions of instruction become too large, there is a risk that the intended learning process is cancelled.

Finally, when students’ and teachers’ views of instruction and the relative importance of different aspects of instruction diverge, the achievability of the instructional goals can be compromised. Teachers believe that the majority of students have much less sophisticated views of learning than they actually have (Watkins 2004). As conceptions of learning influence perceptions (Pratt 1992), teachers are similarly likely to underestimate students’ perceptions. If underestimation of students’ perceptions by teachers causes a concomitant underestimation of achievable learning outcomes, this will be reflected in teaching practices. For instance, students ranked independent learning and individual initiative as the third and fifth most important learning outcomes, respectively, while the teacher did not rank these outcomes among the first six out of ten possible learning outcomes (Doppelt 2004). Additionally, teachers and students have different perceptions of the impact of certain characteristics of instruction on learning outcomes. For instance, students ranked classroom discussions as the second most important aspect, whereas teachers only ranked these as the sixth most important characteristic (ibid).

The above considerations indicate that more insight into students’ perceptions of instruction is crucial and that it is important to find ways to diminish discrepancies between students’ and teachers’ perceptions. Educational research acknowledges the relevance of insight into students’ perceptions and written student evaluations are frequently used and efficiently provide information about sometimes large groups of students (De Jong and Westerhof 2001). However, insight into students’ views will have only a limited effect, if these views are only sought to evaluate the quality of teaching (Marsh and Dunkin 1992). When confronted with discrepancies between their own and students’ perceptions, teachers have been seen to attempt to reduce these discrepancies (i.e., cognitive dissonance reduction; Festinger 1957) by rejecting the student feedback as invalid or by changing their own perceptions rather than changing the lessons (Pambookian 1976). Thus, informing teachers about students’ perceptions does not necessarily guarantee that those perceptions will actually be taken into account.

The literature on student involvement, however, describes an alternative approach that might be more effective: Including students as partners in an ongoing dialogue about instructional design and teaching–learning processes (Cook-Sather 2001). “If school is about what students know, value, and care about, we need to know who students really are. We need to listen to them, pay attention to what they show us about themselves and their views… Students’ voices help us understand what they need and value as learners” (Dahl 1995, p. 124). Thus, listening to students could enable teachers to see lessons through the eyes of their students and give them a more accurate picture of how students interpret their lessons. Too often, students’ potential is neglected, because adults “underestimate the ability of children to be shrewd observers, to possess insight and wisdom about what they see and hear, and to possess internal resources we routinely underestimate” (Lincoln 1995, p. 89). Students are the primary stakeholders in education and experts on their own experiences (Oldfather 1995b). It might therefore be a fruitful initiative to bring qualities, insights, and observations of teachers and students together in a dialogue on how education can be improved. “Students should help shape rather than simply be shaped by educational policies and practices” (Cook-Sather 2003, p. 22).

In many areas outside the field of education, it is common practice to involve potential users of a product or system in the design phase. Participatory design aims to ensure active participation of the users of any system in the design process and in decisions that will affect them (Berns 2004; Kensing and Blomberg 1998; Mankin et al. 1997). Positive effects of this approach have been demonstrated in the fields of cognitive ergonomics and health promotion (Bartholomew et al. 2001; Meister and Enderwick 2002). Effective involvement of users in the design phase yields improved adjustment of the system to users’ needs, higher levels of acceptance of the final design by users, and better understanding of the design by the users, which in turn promotes usability (Damodaran 1996). A participatory design process consists of three phases: (1) analysis, (2) design, and (3) implementation (Cabana 1995). Participants analyze the current situation and assess its shortcomings taking their own needs and desires into account. Next, they come up with ideas for re/design and finally they devise a plan for the implementation of the new design.

So far, there have been hardly any initiatives for student participation in instructional design. This is in contrast to developments elsewhere. Children increasingly play the role of partner in the whole design process of new technologies for children (Druin 2002). This goes considerably further than involving children merely as testers or informants. Also, in research on human–computer interaction, ways are being sought to involve children in various stages of the design process (Markopoulos and Bekker 2003). However, these initiatives are limited to the development of electronic products. In school contexts, participatory design is fairly new. Although the importance of children’s views is acknowledged (Burke 2007), it is more important to enter into a dialogue with students rather than speak about them or for them (Fielding 2004). If students’ views are not actually heard, there is a risk that teachers’ own values will determine their judgments and teachers’ descriptions of students’ realities and interests are biased in favor of their own views. Moreover, if students contribute to the development of instruction, they are given a valuable opportunity to learn from a dialogue with their teacher. A study by MacBeath (2006) has shown that students who were asked to evaluate their teachers and peers became “more reflective and critical of received wisdom and more aware of their own ‘potential’ in the strongest sense of the abused notion” (MacBeath 2006, p. 205). Although participatory design seems promising, we were unable to find studies addressing the effects of this approach in the classroom other than studies that examined the involvement of students in the design of the physical school environment (e.g., Flutter 2006). It seems worthwhile to take up the challenge to adapt participatory design techniques to enable the involvement of secondary school students in the re/design of their courses and to investigate the impact on students’ perceptions of their education.

In the current study, students were involved in a redesign process that was based on ideas from participatory design which were adapted based on the results of a preliminary study (Könings et al. 2007b). This preliminary study, which sought students’ and teachers’ preferences with regard to a possible future implementation of participatory design, resulted in practical guidelines, such as: Participatory design meetings should be organized for one teacher and a small group of students (rather than the whole year group); students selected to participate in the design (co-designers) should represent heterogeneous views of the lessons, and participatory design meetings should not take up too much time. Participation of a small heterogeneous group of students in instructional redesign brings the risk that these students are not representative of the whole class, because are likely to differ in their perceptions and preferences. However, it has been shown that such a selection procedure yields redesign outcomes that are agreed on by the majority of the class (Könings et al., 2010a).

Students’ participation in a re/design process is expected to have a positive effect on their perceptions of a course. This is an important effect to pursue, considering that research has shown that perceptions determine students’ learning and study behaviors and eventually learning performances (Entwistle and Tait 1990). If students see their lessons as stimulating constructivist learning activities, their learning behavior is likely to change accordingly. Additionally, more favorable perceptions may enhance students’ motivation to learn: Their personal interest in learning increases and problems with motivation and concentration are likely to diminish (Könings et al. 2005b). In the current study we considered students’ and teachers’ perceptions as the degree of presence of different characteristics of instruction, such as possibilities for cooperative learning and student autonomy.

Apart from students’ perceptions, the fit between students’ perceptions and their desires on the instructional design of a course is likely to impact on their learning: Improving this fit can improve student motivation and engagement (Eccles et al. 1993), increase personally interested learning orientation, and decrease problems with motivation and concentration (Könings et al. 2005b). It has been shown that large discrepancies between students’ perceptions of their education and their desires of it are associated with multiple problems, such as poor grades, misconduct, and poor mental health (Roeser et al. 2000), and may end in low levels of student engagement and even disengagement and drop-out (Hijzen et al. 2007). Problems like these may be prevented if participatory design leads to a better fit of instruction and students’ wishes thereby diminishing the discrepancies between students’ perceptions and desires.

Finally, we expected positive effects of participatory design on discrepancies between students’ and teachers’ perceptions. Literature has shown that these discrepancies can be detrimental to the effectiveness of instruction. This poses a serious risk to educational practice, since research has shown that teachers and students frequently differ in their perceptions of instruction (for an overview, see Den Brok et al. 2003). In general, teachers tend to take a more positive view of their lessons compared to their students (Fraser 1982; Fraser and O’Brien 1985), students and teachers differ in their preferences with respect to instructional design (Doppelt 2004), and teachers are less likely than students to feel a need to change the set-up of their lessons (Könings et al. 2010b). Participatory design may promote mutual understanding of viewpoints between teachers and students and it might help to take student perceptions more into account.

Taken together, three aspects of students’ perspectives that are important for the effectiveness of instruction may be positively influenced by participatory design: (1) perceptions, conceived as the degree to which different aspects of instruction, such as student autonomy, are present in a course (Wierstra et al. 1999), (2) perceived-desired discrepancies, defined as discrepancies between what students perceive and what they desire with respect to instruction (ibid), and (3) teacher–student disagreement, defined as the discrepancy between students’ and teachers’ perceptions of a course.

The current study evaluated the effects of newly developed participatory design techniques that were applied in secondary education to improve instructional design. Students took part in a meeting with a teacher in which they exchanged positive and negative experiences with the lessons provided, discussed possibilities for improvement and overcoming negative points, and planned the implementation of the outcomes of the discussions. The participatory design process took place in six classes (with seven co-designing students and their teacher in each class) and for each class a different, tailor-made redesign was worked out. The effects were evaluated by measuring students’ perceptions of the lessons, perceived-desired discrepancies, and teacher–student disagreement both before and after the implementation of the instructional redesign. The study sought to answer the following research question: What are the effects of a participatory design meeting and subsequent redesign of a course on students’ (co-designers and non-co-designing students) and teachers’ perceptions of a course, students’ perceived-desired discrepancies, and student–teacher disagreement?

Method

Participants

In the experimental condition, the sample consisted of 10th grade pre-university students (about 16 years old) from two schools for secondary education in the Netherlands (N = 137). The teachers (five males, one female) of these students who voluntarily participated in the experiment taught mathematics, economics, and English as a foreign language. In each experimental class a small group of seven co-designing students (the co-designers) was selected as a representative sample of their class. The students did not object to their being selected to participate in the experiment. The students who were not directly involved in the participatory design process, but received lessons according to the redesign, are referred to as “the rest of the class”. The procedure for the selection of co-designers is explained in more detail in the “Procedure” section.

In the control condition, the sample consisted of 10th grade pre-university students from two schools for secondary education (N = 102). One school participated in both conditions; the other school participated only in the control condition. The control teachers (seven males) taught courses that matched the courses in the experimental condition.

Materials

Inventory of Perceived Study Environment Extended (IPSEE)

The IPSEE was designed to measure students’ perceptions of a particular learning environment and their desires with regard to the design of that environment. Discrepancies between perceptions and desires are a measure of the extent to which the learning environment fits students’ desires.

Of the 56 items of the IPSEE, 31 originated from the Inventory of Perceived Study Environment (IPSE; Wierstra et al. 1999), which was translated into Dutch by the Expertise Centre Active Learning of Maastricht University (Picarelli et al. 2006). We constructed another 36 items to measure characteristics of a learning environment, based on principles of cognitive psychology and constructivism and relating to the main goals of modern education: The acquisition of high-quality knowledge, problem-solving skills, self-directed learning skills, and transferability of knowledge and skills (see De Corte et al. 2003, and Könings et al. 2005a, for an overview). The items of the IPSEE are written in Dutch and consist of eight scales representing central characteristics of such learning environments. The internal consistencies of the perception items of the eight scales are presented in Table 1. The scales are sufficiently independent to warrant separate consideration (Könings et al. 2008b). Pairwise correlations and tolerance values—to check for possible colinearity between scales—revealed no statistical objections to considering the scales independently.

Table 1 Sample items for all scales of the IPSEE
Full size table

A sample item of each scale is presented in Table 1. All items contain a statement and two questions about one of the characteristics of the learning environment. For example:

All students do the same work at the same moment.

  1. (A)

    This happens.

  2. (B)

    I would like this to happen.

The questions are to be answered on a six-point scale, from totally disagree (1) to totally agree (6). Scores on question A give a measure of a student’s perception of the learning environment. Scores on question B give an indication of a student’s desired learning environment. The absolute difference between the scores on question A and B expresses the perceived-desired discrepancy: The higher the discrepancy the larger the difference between students’ perceptions and desires of the learning environment. Small discrepancies indicate a good fit between students’ perceptions and desires.

Inventory of Perceived Study Environment Extended-Teacher Version (IPSEE-T)

This 56-item questionnaire is a parallel version of the IPSEE in which some items are reformulated to reflect the teacher’s perspective. The questionnaire measures teachers’ perceptions and desires with respect to the learning environment. Together these measures give insight into the discrepancy between teachers’ perceptions and desires of the current environment. The internal consistencies of the scales of the IPSEE-T are all acceptable (Könings et al. 2007a): Cronbach’s alphas for the perception items are: 0.78 for the scale fascinating contents, 0.74 for productive learning, 0.77 for integration, 0.87 for student autonomy, 0.75 for interaction, 0.79 for differentiation, 0.77 for clarity of goals, and 0.68 for personalization.

Points of Improvement Evaluation Questionnaire (PIEQ)

This questionnaire was only administered at the posttest in the experimental condition and measured students’ perceptions with regard to the implementation of the formulated improvements. For example: To what extent did you notice that the teacher used more examples from daily life when explaining theory? The questions are rated on a scale from 1 (not at all noticeable) to 10 (highly noticeable). For each experimental class a different version of this questionnaire was composed to represent the specific improvements agreed on for each class. The number of questions corresponded to the number of formulated improvements (it turned out that the minimum was 3 and the maximum 6).

Change Question (CQ)

At the posttest one additional question was asked, the “change-question”, to examine if any changes had occurred in the control condition. Students in the experimental and the control condition were asked: “Did you notice any changes in the way this course was taught during the last two months?” Three response categories were provided: “no changes”, “yes, namely improvement”, “yes, namely worsening”. At the posttest this question was also put to the teachers with two response categories: “no changes” and “yes, the following changes: ….”.

Procedure

At the pretest, all students filled out the IPSEE. Before completing the questionnaire the students were instructed about the goal and contents of the questionnaire and about the scoring method. The teachers filled out the IPSEE-T and written instructions were provided on the first page of the questionnaire.

Based on the results of the IPSEE, seven students from each experimental class were selected to take part in the participatory design meeting. The aim was to select a representative and heterogeneous sample, consisting of students who had positive, neutral, or negative perceptions of the lessons. The selection was based on the scores of the students at the pretest. The teachers were not consulted about student selection. For the selection, an overall perception score was computed for each student: The mean of the scores on the eight IPSEE scales. Within each experimental class students were ranked according to these scores and divided into three groups of the same size: High perceivers, moderate perceivers, and low perceivers. For the purpose of student selection an extra question was asked (ten-point scale): “In your judgment how good or how poor is the teaching in this course?” From the group of high perceivers the student with the highest score on this question and the student with the lowest score on this item were selected. The same procedure was used to select two students from the group of low perceivers. Three students were selected form the group of moderate perceivers: One with the highest score, one with the lowest score, and a third one with a moderate score.

The participatory design meeting was scheduled during regular school hours and lasted 50 min. At the beginning of the meeting the chair (the first author) briefly explained the purpose of the meeting. The teacher, who participated as an equal group member, was asked to explicitly assure the students that critical comments on his/her teaching would have no consequences for them personally. The meeting consisted of three stages: (1) Brainstorming about positive and negative experiences during lessons. Students rolled a yellow and a black ball to each other to guide them in formulating positive and negative aspects of the lessons; (2) Describing and discussing the most important positive and negative aspects of the current educational practice. The students and the teacher individually wrote their comments on important aspects on green (positive), orange (doubtful or moderately negative) and red (negative) cards. The chair clustered the cards according to content and wrote the comments on the blackboard. This list of important comments on the current lessons was the input for the following discussion, and (3) Discussing ideas for improvement of the negative points, and formulating actions to be taken to improve the lessons. For each orange and red theme the students and the teacher discussed how the situation might be improved and formulated steps to be taken.

A few days after the participatory design meeting all the students who had not participated in the participatory design (the rest of the class) received a personal email with a short summary of the meeting, containing the main positive and negative aspects of the lessons as discussed during the meeting and the cooperatively formulated steps for improving the lessons. Further communication between teachers and students took place spontaneously and informally.

The teachers were expected to implement the improvements during the 2 months following the participatory design meeting, which comprised approximately six effective school weeks. At the end of the intervention period the posttest was conducted. Students filled out the IPSEE again and also answered the change question. Additionally, the students in the experimental condition filled out a version of the PIEQ that was adapted to the specific improvements formulated for their class. The teachers filled out the IPSEE-T and answered the change question.

Data-analysis

For each scale of the IPSEE and IPSEE-T mean scores were computed at T1 and T2. Perceived-desired discrepancy scores were computed as the absolute value of the difference between the desire and perception scores on each scale. Teacher–student disagreement scores, indicating the match of students’ and teachers’ perceptions, were computed as the discrepancy between a student’s and his/her teacher’s perception scores on a scale.

To control for unexpected changes in the learning environment, the answers to the Change Question (CQ) were analyzed and compared for the experimental and the control group. A χ2 test was used to investigate the differences between the frequencies of the categorical responses of the teachers and to test whether the frequencies of the students’ categorical responses to the CQ differed between three groups of students: The co-designers in the experimental condition, the rest of the class in the experimental condition, and the control group.

Analyses were conducted for each of the participating classes to examine the effects of the participatory design on the perceptions of characteristics of the lessons in each experimental class. Again three student groups were compared: Co-designers, the rest of the class, and the control group (matched with respect to course). ANOVAs were conducted to see whether the changes in perception scores at T1 and T2 differed between the three groups. Post-hoc tests (Tukey) were used to determine which groups differed significantly.

In the following section, we report the results that are significant at a level of p < 0.05. Because of the limited sample size, trends with p < 0.10 are also reported.

Results

The results for the CQ showed that the teachers in the experimental condition reported changes in educational practice between T1 and T2 far more frequently than the teachers in the control condition, χ2(1, N = 11) = 7.64, p < 0.01. All the teachers in the experimental condition reported changes, versus only one (of six) teacher in the control condition (who reported some increase in student activity during lessons, but no change in the instruction). The results of the analyses of the student responses to the CQ revealed significant differences between the responses of the co-designing students, the rest of the class, and the control group, χ2(8, N = 239) = 107.29, p < 0.01. Of the co-designers 86.8% noticed improvement in educational practice, 13.2% noticed no changes, and nobody noticed worsening. Of the rest of the class 64.1% noticed improvement, 32.6% noticed no change, and 3.3% noticed worsening. In the control group, 8.8% of the students noticed improvement, 90.2% noticed no changes, and 1.0% noticed worsening.

These results indicate that both students and teachers in the experimental condition experienced changes in the design of the learning environment, while no change was experienced by the students and teachers in the control condition. This supports our assumption that the classes in the control condition functioned as a reliable group in the comparison with the experimental classes.

In the next paragraphs, the formulated improvements and the extent of their implementation as perceived by the students are described for each experimental class separately. The extent of the implementation of improvements is indicated by the mean PIEQ score of all the students of a class and the PIEQ score of the teacher. Both scores are presented in brackets. The PIEQ scores range from 1 to 10. We consider a score below 5.5 as an indication of insufficient implementation. The results of the ANOVAs on changes in perception scores, perceived-desired discrepancy scores, and teacher–student disagreement scores are described only for those IPSEE scales that are relevant to the improvements formulated for that class. This is a sensible selection, since (over all classes) there were significantly more positive effects of the participatory redesign on the relevant scales than on the irrelevant scales, χ2(1) = 5.20, p < 0.05: Of the 25 significant effects on the relevant scales, 16 were positive (i.e., in the expected direction), while of the 20 effects on the irrelevant scales only six were positive.

Class 1

The participatory design meeting resulted in three points of improvement: (1) The teacher should clearly describe the contents of each lesson (M students  = 7.79, SD = 1.36; Score teacher  = 6.3) and explain and answer only questions relating to the planned contents (M students  = 6.52, SD = 1.35; Score teacher  = 10.0); (2) The teacher should be more strict with students who are talking loudly instead of whispering (also during task-related discussions), because this disturbs the concentration of the other students (M students  = 5.81, SD = 1.94; Score teacher  = 8.0), and (3) It should be explained more clearly where students can find additional exercises when needed (M students  = 6.05, SD = 2.28; Score teacher  = 7.0).

With respect to content, the first improvement relates to the IPSEE scale differentiation (diminishing differentiation in the lessons). Table 2 presents the descriptives and the results of the ANOVA on the differentiation scale. The perceived-desired discrepancy scores of the rest of the class decreased less than those of the control group. Actually, for the rest of the class perceived-desired discrepancy scores increased, while for the control group the scores decreased. The second and third points of improvement show no clear relationship with any of the IPSEE scales.

Table 2 Means (and SD) of the IPSEE scores of experimental class 1 and results of the ANOVAs
Full size table

Class 2

The participatory design meeting yielded three main points of improvement: (1) The teacher should briefly explain new subject matter during each lesson instead of the usual situation with students working individually on mathematics exercises and asking the teacher for individual instruction when they encounter a problem (M students  = 7.84, SD = 1.80; Score teacher  = 5.0); (2) The students wished that teacher support would be given more promptly when they had questions (i.e., shorter waiting time) (M students  = 5.57, SD = 2.25; Score teacher  = 7.0), and (3) Errors in the exercise book or learning materials should be explicitly reported in class to ensure that everyone is informed (M students  = 7.13, SD = 2.33; Score teacher  = 5.0). The PIEQ scores of the teacher indicate limited implementation of points 1 and 3, but the students were more positive about the implementation. The teacher, however, reported implementation in half of the lessons, implying that his score expressed frequency rather than value.

Table 3 presents the descriptives and the results of the ANOVAs on the relevant IPSEE scales. Points 1 and 2 relate to student autonomy and interaction. The third point of improvement has no clear relationship with any of the IPSEE scales. Teacher–student disagreement scores on student autonomy showed a larger increase among the co-designers and the rest of the class than in the control group. This undesirable result is caused by the combination of a decrease in the teacher’s perception scores (T1: 3.60, at T2: 2.80) and no change in the students’ perception scores between T1 and T2. The perceived-desired discrepancy scores on interaction of the rest of the class decreased less than those of the control group, which suggests a positive effect of the intervention.

Table 3 Means (and SD) of the IPSEE scores of experimental class 2 and results of the ANOVAs
Full size table

Class 3

Three points of improvement were formulated: (1) The teacher should give a short summary of the subject matter of the previous lesson to supplement individual and self-directed work, which is the prevailing approach in this class (M students  = 5.67, SD = 2.49; Score teacher  = 8.0); (2) Students said they wanted to be allowed to skip mathematics exercises if these were very similar (M students  = 5.10, SD = 2.51; Score teacher  = missing value), but the teacher said that the students themselves were responsible for skipping exercises, and (3) The teacher should be more precise in writing the numbers of worked-out exercises on the blackboard, so that it was easier for students to pick up on the explanation if they had not paid full attention from the beginning (M students  = 8.17, SD = 2.75; Score teacher  = 8.0). The second point was not implemented sufficiently according to the students. An indication of the implementation by the teacher was missing.

Table 4 presents the descriptives and results of the ANOVAs on the relevant IPSEE scales. The points of improvement are related to differentiation and student autonomy. On the one hand, increased differentiation and student autonomy was proposed by allowing students to skip exercises. On the other hand, the request for summaries can be interpreted as a reduction in student autonomy and differentiation. As for differentiation, the results show that the perception scores of the co-designers increased more strongly than those of the control group, whose scores decreased. Teacher–student disagreement scores in relation to differentiation decreased more strongly for the co-designers and the rest of the class than for the control group. A group effect was found for the teacher–student disagreement scores on student autonomy but post-hoc analyses did not show any significant effects between groups.

Table 4 Means (and SD) of the IPSEE scores of experimental class 3 and results of the ANOVAs
Full size table

Class 4

The meeting resulted in six points of improvement: (1) The teacher should summarize subject matter more frequently during the lessons (M students  = 6.70, SD = 1.58; Score teacher  = 8.0); (2) More examples from daily life and news items should be used to make the economics lessons and learning content more interesting (M students  = 5.80, SD = 1.99; Score teacher  = 5.0); (3) In order to overcome passive attitudes of students, which occurred occasionally, students should be stimulated to actively contribute to the lesson by asking questions, taking questions seriously, and trying to answer them (M students  = 6.04, SD = 2.58; Score teacher  = 7.0); (4) The teacher should provide exercises to help students to prepare better for the test (M students  = 6.05, SD = 2.65; Score teacher  = 9.0); (5) In order to optimize test results, the teacher should clearly state the maximum number of attainable points per question. This would make it easier for students to answer the most important questions in time (M students  = 7.99, SD = 2.79; Score teacher  = 10.0), and (6) More silence in the classroom (M students  = 6.03, SD = 1.54; Score teacher  = 5.0). The teacher indicated that points 2 and 6 had not been implemented. He argued that the content of the subject matter taught during the study period was not suitable for linkage with news items (point 2). The teacher was dissatisfied with the results with regard to more silence in the classroom (point 6).

The points of improvement relate to fascinating contents (using more examples from daily life), student autonomy and interaction (asking questions and taking each other’s questions seriously), and clarity of goals (providing exercises for tests and stating attainable points per question). Table 5 presents the descriptives and ANOVAs for these scales. With respect to fascinating contents, the perceived-desired discrepancy scores of the co-designers decreased while those of the control group increased. The teacher–student disagreement scores decreased more strongly for the co-designers and the rest of the class than for the control group. As for student autonomy, the perception scores of the co-designers increased more than those of the control group. No significant effects were found for interaction. With regard to clarity of goals, the perception scores of the co-designers and the rest of the class increased more than those of the control group. Perceived-desired discrepancy scores showed a greater decrease for the co-designers than for the control group.

Table 5 Means (and SD) of the IPSEE scores of experimental class 4 and results of the ANOVAs
Full size table

Class 5

Six points of improvement were formulated: (1) A clear statement of the minimum number of economics exercises to be completed during each lesson (M students  = 6.90, SD = 1.44; Score teacher  = 7.0); (2) Subject matter would be more interesting if the teacher used more examples from news items and daily life (M students  = 6.80, SD = 1.48; Score teacher  = 7.0); (3) The students should be stimulated to explain subject matter to each other in order to enhance understanding (M students  = 6.72, SD = 1.54; Score teacher  = 8.0); (4) The teacher should provide a procedure for solving difficult problems in order to improve students’ comprehension of the problem-solving process (M students  = 4.24, SD = 2.62; Score teacher  = 6.0); (5) Tests should better match the degree of difficulty of the problems dealt with during lessons and should not require an understanding that was deeper than that required for the exercises during the lessons (M students  = 5.06, SD = 2.38; Score teacher  = 8.0), and (6) Students occasionally got bored if the teacher took too much time to answer a question asked by one student. The teacher should answer such questions individually at the end of the lesson (M students  = 3.71, SD = 2.53; Score teacher  = 5.0). Students were dissatisfied with the implementation of points 4, 5, and 6, which they thought were not sufficiently noticeable during the lessons. The teacher rated the implementation of point 6 as low and remarked that he was able to explain and answer all questions during the lessons.

The proposed changes in the lessons are related to fascinating contents (using examples from daily life), differentiation (stating the minimum number of exercises to be completed—a request for less differentiation), interaction (students explaining subject matter to each other), clarity of goals (more congruence between degree of complexity of lessons and tests), and personalization (support for individual students at the end of the lesson instead of during the lesson). Table 6 presents the descriptives and the results of the ANOVAs on these scales. There were no effects on the scales fascinating contents and interaction. On differentiation, the teacher–student disagreement scores increased less strongly for the co-designers and the rest of the class than for the control group. The perception scores concerning clarity of goals decreased more strongly for the co-designers than for the control group, whose scores increased. Also, the perceived-desired discrepancy scores increased more for the co-designers and the rest of the class than for the control group. This may be due to the insufficient implementation of the improvements. On personalization, perception scores decreased more strongly for the rest of the class than for the control group, which may also be related to poor implementation.

Table 6 Means (and SD) of the IPSEE scores of experimental class 5 and results of the ANOVAs
Full size table

Class 6

Three points of improvement were formulated: (1) More time should be given to practicing English speaking skills and pronunciation during the lessons (M students  = 7.76, SD = 1.18; Score teacher  = 7.0); (2) Students wanted more practice in reading English texts (M students  = 6.71, SD = 2.14; Score teacher  = 7.0), and (3) The students wanted more explanations of grammar to the whole class instead of individual study of grammar from the course book (M students  = 7.59, SD = 1.19; Score teacher  = 6.0).

The content of the points showed no clear relationship with IPSEE scales, but could be linked to two scales. Practicing speaking and reading skills is likely to improve personal relevance and the challenging character of subject matter (fascinating contents), because students themselves pointed out these aspects as important. Also, clarity of goals could be relevant in this context, because the proposed additions to the lesson program could decrease the clarity of what is expected of students. Having to do something new can lead to increasing uncertainty about learning goals.

The descriptives and the results of the ANOVAs on the two scales are presented in Table 7. With respect to fascinating contents, the results revealed that the perceived-desired discrepancy scores increased more strongly for the rest of the class than for the control group. As expected, clarity of goals decreased during the intervention period. Specifically, the perception scores of the rest of the class decreased more than those of the control group. Additionally, the perceived-desired discrepancy scores increased more for the rest of the class than for the control group students, whose perceived-desired discrepancy scores even decreased.

Table 7 Means (and SD) of the IPSEE scores of experimental class 6 and results of the ANOVAs
Full size table

Conclusions and discussion

This study investigated the effects of participatory design in secondary education on students’ and teachers’ perceptions of the redesigned instruction. The hypothesis was that the intervention would lead to more positive student perceptions, decrease students’ perceived-desired discrepancy, and decrease disagreement between teachers’ and students’ perceptions of the instruction.

The effects of the intervention were investigated for each of the experimental classes separately and analyzed for those characteristics of instruction that most closely corresponded with the proposed educational changes. The findings provide some evidence of improvement in the perceptions of the co-designers: Their perceptions increased more than those of the controls (classes 3 and 4). There was also some evidence of positive effects for the rest of the class (class 4). Unexpectedly, in some cases, the perceptions of the co-designers decreased more than those of the control group (classes 5 and 6). This may be related to the extremely low evaluation scores on the improvement concerned (class 5) and the nature of the improvement (class 6).

With respect to perceived-desired discrepancies, the participatory design showed some positive effects for the co-designers, but often negative effects for the rest of the class. Perceived-desired discrepancies decreased more for the co-designers than for the control group (class 4), but for the rest of the class the perceived-desired discrepancy increased more (classes 1 and 6) or decreased less (class 2) than for the controls. In one instance, perceived-desired discrepancy increased more for the co-designers and the rest of the class than for the controls (class 5). This outcome, again, is probably due to the extremely low evaluation score on the point concerned. Students rated its implementation as insufficient, so positive effects were unlikely. Unsatisfactory implementation appeared to even have a negative effect on students’ perceptions.

As for teacher–student disagreement, the intervention had positive effects for the co-designers and, in most cases, for the rest of the class as well: The discrepancy between teachers’ and students’ perceptions diminished. Teacher–student disagreement decreased more for the co-designers and the rest of the class than for the control group (classes 3, 4, and 5). In one case, teacher–student disagreement increased for the co-designers and the rest of class, but this was due to a decrease in the teacher’s perception, which increased the discrepancy scores (class 2).

The findings for our research question on the effects of participatory design on students’ perceptions, perceived-desired discrepancies, and teacher–student disagreement show some positive effects for the co-designers, but limited or negative effects for the rest of the class.

The positive effects for the co-designers may have at least three underlying causes. First, the co-designers paid more attention to those aspects of the course that were discussed and re-designed in the participatory design meeting. Co-designers’ expectations were raised in the design meeting and this may have directed their attention to cues of successful implementation (i.e., selective attention, Olson et al. 1996). Second, increases in sense of control (Seifert and O’Keefe 2001), sense of agency, belonging, and competencies (Mitra 2004), as well as a better understanding of the teacher’s work and perspective (Cook-Sather 2002) can lead to increased engagement and a smaller distance to the teacher’s view on instruction. Third, interpersonal and affective aspects are very important when students evaluate their teacher or the course created by their teacher (Johannessen et al. 2002). For secondary school students, being a good teacher is almost equivalent to establishing good personal relationships with students (Beishuizen et al. 2001). Adolescent students feel a strong need to be supported, understood, and listened to (Blanco et al. 2002). The improved and intensified relationship between the co-designers and their teachers may thus have created a more positive learning climate.

In contrast to the co-designers, the rest of the class could only perceive some redesigned features in the course. Apparently, this did not cause a positive change in their perceptions. No effects were observed for perceptions and perceived-desired discrepancies even increased. The absence of effects may indicate that redesign per se is not effective and does not guarantee that all students profit from it. With regard to the increase in perceived-desired discrepancies, the rest of the class was confronted with co-designing peer students who had been able to influence the instructional design, whereas they had not been able to exert any control. The fact that part of the class was excluded from the design process may have fuelled negative feelings about the course in that group.

It cannot be excluded that the proposed changes were not sufficiently communicated to all the students. Research in change management has shown that limited communication can be an important factor when changes are introduced (Proctor and Doukakis 2003). Moreover, the quality of communication influences peoples’ openness to change and ultimately determines the effectiveness of change (Allen et al. 2007). Inadequate information may have frustrated the non-co-designing students in our study. Another possible explanation is that students became aware of possible shortcomings of the course. This can stimulate the external attribution of problems with learning and students may overrate the role of the instruction, and therefore become more critical. Finally, the increase in perceived-desired discrepancies may be attributable to the fact that the rest of the class were informed by the co-designers that certain aspects of the lessons needed to be changed, but missed cues that these changes were actually occurring. This may have made these students extra critical.

A practical implication of our study is that teachers should be supported in implementing collaboratively formulated redesign, because appropriate implementation cannot be taken for granted. According to both teachers and students, the majority of the improvements were implemented to a satisfactory degree. However, for some points the implementation did not succeed and, in those cases, positive effects of participatory redesign cannot be expected. It is a limitation of our study that the implementation of the proposed educational changes was not closely monitored. As the implementation of redesign is a crucial factor in examining the effects of participatory instructional (re)design, it is recommended that future studies should register how teachers implement the redesign. This provides additional information that can help to improve the effectiveness of participatory (re)design. Proper educational coaching of teachers may be a way to optimize the implementation of improvements. The effect of such coaching can also be evaluated by observing teachers’ behaviors in the redesigned course, in addition to students’ and teachers’ subjective perceptions of the instructional redesign.

Another practical implication and focus of future research relates to the finding that the co-designing students in all classes mentioned that student participation was more badly needed for other courses with other teachers. The teachers included in this study participated on a voluntary basis and were motivated to experiment with participatory design. We expect that not all teachers would be able or even motivated to implement the outcomes of a participatory design meeting in their educational practice. An educational advisor might be necessary to support teachers in implementing change.

A theoretical implication of our study is that effects of student participation in (re)design can be empirically tested. However, more research is needed to reach a more accurate model of the effects on students’ perceptions of instruction. The literature on this topic is incomplete, as is indicated by the unexpected finding of our study that participatory design can have negative effects for non-co-designing students (the rest of the class). It might be advisable to involve the rest of the class in the re/design process to improve the overall effectiveness of participatory design. This could be realized by regularly changing the composition of the group of co-designing students so that all students are given the opportunity to participate in re/design activities. Another way to enhance the effects of participatory (re)design could be to have the teacher and a small group of students list/discuss the main bottlenecks of a course, after which each co-designing student discusses the bottlenecks with a group of students of the rest of the class. These discussions could lead to suggestions for instructional redesign. The suggestions of the different groups can be reported and discussed by the original group of co-designers and the teacher, in the presence of the rest of the class. If participatory instructional (re)design is used more frequently in a school, students and teachers will get used to this new idea and the procedure will likely take less energy and time. So the efficiency of discussions is likely to increase with experience.

A limitation of the study is that the instruments used to measure the effects—IPSEE and IPSEE-T—were not fully attuned to the specific re-designed characteristics of each participating class. The IPSEE(-T) measures eight main characteristics of a learning environment, but these did not always correspond sufficiently with the proposed improvements. Thus, the questionnaires may have suboptimal sensitivity to measure the effects of specific changes in instructional design. Moreover, before the participatory design meeting (i.e., at T1) it was impossible to predict which aspects of a course would be redesigned. Consequently, when using a ‘pretest–posttest design’ it is impossible to use other than relatively rough measures for examining the effects of participatory design. This limitation is difficult to avoid because experimental testing of effects is crucial. In order to investigate the effects of specific points in the redesign we included retrospective measures in the PIEQ. However, such measures have the disadvantage that control group students cannot directly answer questions about a redesign that was not implemented their own class. Another limitation is that the experimental and control classes were not recruited from the same schools, due to organizational factors. So it cannot be excluded that school-related factors influenced the results, although we consider this highly unlikely. The answers to the Change Question did not give any indication of influences at the school level, neither in the experimental nor in the control condition. In addition, the experiment was so closely related to the instructional procedures at class level that factors at school level seem irrelevant for our experimental treatment.

This study provides some first insights into the use and effects of participatory design in education, but several questions are left for future research. First, it would be interesting to study long-term effects of participatory design meetings and subsequent redesign activities, in addition to the short-term effects investigated in this study. Second, direct effects on learning outcomes and the effectiveness of a course should be examined in more detail. This study focused on the effects of participatory re/design on perceptions, perceived-desired discrepancies, and disagreement between teacher and student scores. But student perceptions of instruction influence their learning and study behavior and eventually their learning outcomes (Elen and Lowyck 1999; Entwistle and Tait 1990). Influencing student perceptions through instructional redesign should therefore also be expected to affect and improve students’ learning behavior and, eventually, the effectiveness of education. Third, it would be interesting to focus in greater detail on the interplay of variables of the discrepancy measures. A decrease of teacher–student agreement can result from changes in both student perceptions and teacher perceptions triggered by the discourse that is going on. In the concept of students’ perceived-desired discrepancy, perceptions and desires are intertwined. Students’ perceptions change, but desires are also subject to changes over time. They may, for example, change as a consequence of disappointing study results (Könings et al. 2008a). More in-depth study of discrepancy measures may provide more precise information on the effects of participatory design. Fourth, it would be valuable to examine in more detail how participation affects students’ sense of belonging, agency, and general competences. Experiments measuring these variables may also help to explain the differential effects for co-designing students and the rest of the class. A related, fifth and final aim of future research is to explore ways to improve the effects of redesigned instruction for non-co-designing students. An obvious first approach would be to implement participatory design in such a way that all students are involved. Alternatively, the quality of communication about the re/design process with the rest of the class may have been suboptimal in our study, and better and more frequent communication might have been beneficial.

To conclude, this study reveals that participatory re/design can be used in education as a technique to make students’ perceptions part of the instructional re/design process. Co-designing students notice improvements in educational practice, but clear effects on their perceptions and perceived-desired discrepancies are limited. In contrast to our expectations, perceived-desired discrepancies on the course increased for the non-co-designing students. This effect deserves further, in-depth study. For both co-designing and non-co-designing students, teacher–student disagreement decreased. Despite these mixed findings, it seems worthwhile to study participatory instructional (re)design in more detail. If all students are involved, participatory re/design appears to be a promising tool to take account of students’ perceptions in efforts to improve instruction.