1 Introduction

As the objective of teaching is student learning, assessing the impact of teachers’ support on learning has been considered a major indicator of teaching quality in higher education institutions. This impact is made evident in students’ increased knowledge and skills as a result of their experiences. Though it seems to have been the case, whatever students learn could not always be attributed to teachers’ support. Instead, students’ assessment of teachers’ behavior is considered as evidence of the quality of the teaching.

Students’ descriptions of the characteristics of effective teachers are commonly used to develop measures of teaching quality. The most likely characterizations include ‘one who knows the subject well, communicates effectively, is approachable, etc’. It is uncommon to get descriptions such as ‘one who involves students in learning, designs tasks that help student learning, or guides student learning’. Part of the reason for giving more emphasis to teachers’ overt behavior as indicator of effectiveness could be the widely held belief that teaching is something the teacher does to make students learn. In fact, what the teacher does in the process of learning has a significant impact on student learning (e.g., Hattie 2003; McKeachie 2007). However, learning is less likely to occur as a result of what is presented by teachers. Rather, it is an activity that is accomplished by students, of course with the guidance and facilitation from teachers. Students have to be the major players in creating meaning out of their experiences. Some studies emphasize that student engagement in the learning process is more important for learning (e.g., Biggs 1999; Chickering and Gamson 1987; Shuell 1986). In the same vein, the students’ engagement in the process should be the focus of assessing the effectiveness of teaching. In light of this, it may be difficult to evaluate teacher performance in terms of its impact on student learning if the emphasis continues to be on the teacher’s behavior.

Teaching evaluation is conducted based on two related key assumptions. Primarily, teachers are seen as having an influence on student learning. The second one has to do with the students’ capacity to provide teachers with feedback to be used for teaching improvement. While there is little doubt about the second assumption, some contend that there is no empirical evidence that revealed improvements in teaching or student learning as a result of the feedback provided (e.g. Kember et al. 2002; Olivares, 2003). The nature of the items used to evaluate the effectiveness of teaching has been mentioned by Kember et al. (2002, p. 421) as one of the possible reasons for the lack of improvement. They stated that the emphasis of the rating questionnaire was on providing ‘judgmental’ instead of ‘developmental’ feedback, and this might have influenced the use of the feedback for improvement. In other words, the feedback may not have informed teachers what worked or didn’t work for the students and what needs to be improved, as the items refer to teacher performance. They suggested improving the instruments as well as changing the focus of the evaluation if feedback is to be used for teaching improvement.

In the higher education institutions in Ethiopia, students evaluate their teachers at the completion of a course using a 27-item generic Teaching Evaluation Questionnaire (TEQ). The teaching in these institutions has been described to be dominantly teacher centered (Daniel 2004; Zenawi et al. 2011. Recently, the Higher Education Proclamation (2009) clearly stated the need for higher education institutions to introduce learner centered instruction. However, the TEQ does not provide opportunities for students to evaluate the impact of teaching on their learning. The items in the questionnaire refer to what teachers should do in the process of teaching and learning. Thus, with the shift of emphasis to learner centered instruction, the use of an improved measure that provides opportunities for students to evaluate teaching in terms of their learning may encourage teachers to implement learner centered approaches.

2 Conceptual framework

The assumptions considered in designing the measures of teaching effectiveness influence the appropriateness of the feedback for improvement. The implied meaning of teaching and learning in the measures in turn has an impact on the teaching and learning approaches employed. A model proposed by Biggs (2003) that deals with the levels of thinking about teaching, expressed in terms of student learning, is used to explain the assumptions considered in designing student rating questionnaires. In this model student learning is described in three different ways: (1) learning is a function of individual differences between students; (2) learning is a function of teaching; and (3) learning is the result of students’ involvement in learning focused tasks.

Biggs argues that when learning is considered a function of individual differences between students, failure to learn is attributed to lack of ability on the part of students but not to problems in teaching. On the other hand, when learning is considered a function of the teaching, the focus is on teacher performance, not on what types of students the teacher has to deal with. Whether the teaching has the desired effect on student learning is not the concern since the role of the teacher is on transmitting knowledge. Finally, when learning is considered the outcome of student engagement, the emphasis is on what students have to do. In this case, the focus of teaching shifts from what the teacher does to what students have to do to understand the materials presented. Thus, the role of the teacher is facilitating student learning. When the ‘teaching as transmitting knowledge’ perspective is predominant, more emphasis is given to evaluating the effectiveness of teaching in delivering contents to students. A student rating questionnaire influenced by the ‘teaching as transmitting knowledge’ perspective is designed to evaluate the effectiveness of teacher performance, i.e. whether the teacher organizes, presents, and evaluates contents. In this case, teaching evaluation is limited to the issues of content coverage and presentation since it is assumed that students learn when the teacher presents contents. By contrast, when the ‘teaching as facilitating learning’ is the preferred perspective, student involvement in the process of learning is given more emphasis. This could be addressed in student rating questionnaires in terms of evaluating the teacher’s support and facilitation to engage students in learning.

Studies on teaching and learning indicate that both active student engagement and teacher guidance contribute to student learning. Learning is a function of teacher’s facilitation as well as students’ engagement in the process of learning. Students learn better when they are actively engaged in learning (Chickering and Gamson 1987; Biggs 1999). It is difficult to think of an instructional setting that is entirely teacher centered or student centered. Apart from benefitting students, such engagement proves to be helpful to teachers since they receive improved ratings from their students (McGowan and Graham 2009). Others argue that teaching skills of the teacher are equally important for teaching effectiveness (e.g. Hattie 2003; Pascarella and Terenzini 2005).

Some studies characterize the most widely used student rating questionnaires as teacher centered, which is expressed in terms of the emphasis given to teacher behavior as a basis for evaluating teaching quality (Abrami et al. 2007; Barr and Tagg 1995; D’Appollonia and Abrami 1997; Kolitch and Dean 1999: McKeachie 1997). The dimensions in these questionnaires inform students that to teach effectively means to present and evaluate contents. The fear is that students may give lower ratings to teaching that is more interactive and engaging (Kember and Wong 2000; Crumbly et al. 2001). As a result, teachers may be tempted to adhere to the requirements implicitly stated in the evaluation questionnaire.

One may ask, why bother about student learning in a measure designed to evaluate the effectiveness of teaching? As Abrami, et al. (2007) state, student ratings are not designed to measure student learning directly. It is rather indirectly inferred from the ratings since studies indicate that highly rated teachers contribute to student learning (e.g. Marsh 1987). However, such inference could be misleading for a number of reasons. For one thing, student ratings reveal only one aspect of the process, not the whole of it. In addition, students are better poised to rate their perceptions and experiences than the teacher’s behavior. Equally importantly, evaluating teacher performance through a questionnaire that entirely refers to teaching skills may not necessarily indicate students’ learning experiences. What the students have to do in order to learn is not considered as a basis for evaluating effective teaching. The following description by Abrami et al. (2007) clearly indicates the impact of using teacher centered evaluation questionnaires:

If non-global items from such rating forms are used for summative decisions, instructors may feel obliged to ponder to the built in bias these forms exhibit towards teacher-centered learning environments, and so the rating forms themselves would become a major obstacle to adoption of more student-centered active learning strategies that educational research has shown to promote conceptual change (Abrami et al. (2007), p. 451).

Some of the justifications for the need to consider students’ evaluation of their learning as indicators of teaching effectiveness come from studies that revealed that certain student related behaviours significantly contribute to their learning. For instance, students’ active engagement and collaboration was found to contribute to their learning (Chickering and Gamson 1987; Kuh et al. 1997; McGowan and Graham 2009). Students also reported improvements in their use of deep approaches to learning, their level of motivation, and even their academic work after attending a training that gave them the opportunity to take responsibility for their own learning (Dart & Clarke, 1991). The implication is that students will be in a better position to provide appropriate feedback for teaching improvement if the evaluation of teaching quality also addresses their learning experiences. The measure of effectiveness should enable students to evaluate the extent to which they were engaged in learning. Teachers will also be able to witness the impact of their teaching on student learning when teaching is evaluated in terms of the opportunities provided for students.

Some attempts have been made to consider students’ evaluation of their learning experiences as a basis for evaluating the effectiveness of teaching (Cabrera et al. 2001; Oliver et al. 2008). As opposed to rating forms designed to evaluate teacher performance, these questionnaires contain items that require students to express their level of engagement in the classroom, what helped or hindered their learning, their perceptions of the teachers’ strengths and weaknesses, and the most important skills, attitudes, and concepts they learned. Teacher’s role in this case is expressed in facilitating student learning through designing appropriate learning experiences. Certain characteristics related to student learning are not, however, addressed in the indicators. For instance, student learning is related to the effort they exert (e.g. Davis and Murrell 1994). The measure designed should also include items that address students’ role in learning since it is indicated that students learn better when they are encouraged to be responsible for their own learning (e.g. Weimer 2002). Some of the dimensions were identified based on the experiences in specific instructional contexts (e.g. Cabrera et al. 2001). What is more, not much is known about whether the measures actually enable students to evaluate teaching in terms of their own learning.

It can be inferred from the studies reviewed that students learn better when they are actively engaged in the process of learning (e.g. Biggs 1999; Chickering and Gamson 1987; McDowell et al., 2010) and given responsibility for their own learning (e.g. Weimer 2002). Improvements in student learning have also been reported when teachers make better course organization and presentation (e.g. Pascarella and Terenzini 2005) and provide feedback on progress (e.g. McDowell et al., 2010; Ramsden 2003). Thus, if students have to evaluate their teachers in terms of their own learning, the measure of teaching quality has to addresses the facilitation by the teacher as well as students’ evaluation of their engagement in the process of learning. In this study, it was hypothesized that such a measure has to address the following five dimensions: the way courses are organized and presented, the quality of assessment and evaluation, the feedback teachers give and receive, students’ level of engagement in the process of learning, and students’ judgments of their involvement in the process of learning.

3 Method

Student evaluation questionnaires are designed using one or a combination of the following approaches: (1) selecting certain characteristics of excellence and asking students to evaluate their teachers; (2) allowing students to write characteristics of excellence they believe describe effective teachers; and (3) interviewing teachers identified as excellent (Marsh 1987; Sherman et al. 1987). A slightly different approach was implemented in this study. First, five dimensions believed to address students’ evaluation of their progress and the support they receive from the teacher have been identified from the literature. Second, students evaluated their experiences while attending courses designed in accordance with learner centered approaches. Finally, a pool of items was drafted using students’ descriptions of their experiences and the dimensions in the hypothesized framework.

3.1 Participants

Two academic departments at Mekelle University, one from the social sciences and the other from the natural sciences, were selected to represent diverse instructional settings. The sample involved 189 second year students from the Department of Geography and Environmental Studies and 54 graduating class students from the Department of Mechanical Engineering.

3.2 Material

3.2.1 Student Learning Experience Questionnaire

A Learning Experience Questionnaire designed based on the tenets of learner centered instruction proposed by Weimer (2002) was used by students to evaluate their experiences. The questionnaire contains structured items presented in a Likert scale format (e.g. I am encouraged to make interpretations of the learning contents) and students were required to rate the frequency of the behavior. In the open ended items, on the other hand, students provided comments related to their experiences (e.g. Provide your suggestions on how to improve the process of instruction).

3.3 Procedures

The study was implemented in two phases. During the first phase, teachers were trained on the tenets of learner centered instruction (Weimer 2002) and on the instructional methods appropriate for learner centered instruction. Courses were redesigned in such a way that the students could do most of the learning by themselves. Specific objectives and tasks were presented for the different subtopics in the courses. During the implementation of learner centered instruction, students were providing comments for teaching improvement. During the second phase that commenced half way in the semester, students provided descriptions of their experiences using the Student Learning Experience Questionnaire. The items for the SELTQ were derived from the data obtained using the learning experience questionnaire and based on the hypothesized framework.

4 Results

4.1 Students’ evaluation of learner centered instruction

During the implementation of learner centered instruction, students evaluated the appropriateness of their experiences using the Student Learning Experience Questionnaire. One of the questions was related to their assessment of the approach employed in the course. 85% of the students replied that the teaching approach was different as compared to the one used in other courses. The students described that teachers in the learner centered classes designed courses to engage students in the process of learning, gave suggestion and facilitation, provided reading materials in advance, and conducted regular evaluation and feedback. They also stated that assignments and exams were designed to help them learn and they were given the opportunity to redo assignments based on teachers’ comments. Students provided their experiences in the courses designed in learner centered paradigm. 94% replied that they learned better in the learner centered classes, and they attributed this to their use of new learning skills such as group participation and presentation. They also added that they were informed why they learn a given topic and what they should do to grasp the concepts. 87% of the students replied that they performed the requirements in learner centered classes, such as working in group activities, performing desired tasks, and conducting peer as well as self evaluation.

4.2 Instrument development methodology

When designing a measure of teaching effectiveness, Levine and Wright (1987) suggest three major steps that should be accomplished: selection of measurement criteria, developing the rating scale, and checking the reliability and validity across instructional settings. Similarly, Berk (1979) mentions specifying the domains of indicators as the first step in designing evaluation instruments. During the development of the scale, a pool of 44 items was drafted based on suggestions from students as well as review of previous works on student learning (Biggs 1999; Chickering and Gamson 1987; McDowell et al., 2010; Pascarella and Terenzini 2005; Ramsden 2003; Weimer 2002). The list was presented under the five thematic areas in the hypothesized framework: course organization and presentation; level of student engagement; the nature of assessment and evaluation; the quality of the feedback from and to students; and students’ own judgment of their learning. The procedures implemented to estimate reliability and validity are presented below.

4.3 Validity estimates

4.3.1 Content validity

Content validity in this study refers to the judgment given by content experts on the appropriateness of the items. As a way of estimating content validity, the item pool was presented to five senior educators for comments. They provided their comments based on an improved version of a checklist designed by Berk (1979). The educators checked for clarity, redundancy, and unity. Based on their recommendations, the initial draft was reduced to 30 items. In another survey, 76 instructors randomly selected from six departments provided their comments based on the four criteria presented below:

  1. (a)

    Students can provide comments about the issue;

  2. (b)

    The item can be used across instructional contexts;

  3. (c)

    I can receive feedback for teaching improvement; and

  4. (d)

    The item is clearly stated.

The respondents were asked to provide their answers in a ‘yes’ or ‘no’ format after reading the four criteria for every item. Items with an average ‘yes’ response below 60% for each criterion were removed from the list since it was believed that nearly two-third of the respondents should agree if an item has to be retained in a given dimension. The mean values of the ‘yes’ responses for the remaining items range from 60-90%. This resulted in a total of 24 items presented under five categories.

4.3.2 Construct validity

Construct validity in the context of rating scale design refers to assessing whether or not the scale measures the hypothesized construct it claims to measure. To test the extent to which the newly designed rating scale has meaningful structures stated in the hypothesized framework, factor analysis was employed. Construct validity was assessed by examining whether the underlying constructs addressed in the five subscales were retained after running factor analysis. Factor loading was used as a criterion to select items for the scale, since items with higher loadings on a factor represent the underlying dimension.

The total number of valid responses (204 respondents for 24 items) is slightly lower than the 10 subjects to 1 variable ratio commonly used in most studies of factor analysis (Costello and Osborne 2005). However, the fact that Kaiser-Meyer-Olkin’s (KMO) index of sampling adequacy was high (.92), as compared to the recommended value of 0.60, implies that the sample size was adequate. Bartlett’s Test of Sphericty was also significant (p < .00), suggesting factorability of the correlation matrix.

The factors were selected after factor analysis has been conducted and rotated using varimax rotation to identify orthogonal (independent) factors. Although it was assumed that the new scale would represent five dimensions, factor analysis revealed four factors with Eigen values exceeding one. The two factors assumed to be independent in the hypothesized framework have been combined after the analysis. The four factors explained 56.68% of the total variance. The minimum value for retaining an item was .32, a value suggested as a good rule of thumb for the minimum loading of an item. Only two items, i.e. objectives and expectations were explained, and I was involved in group projects have communalities below 0.4, a minimum value suggested for social sciences (Costello and Osborne 2005). All the 24 items have loaded into the various categories and showed acceptable values of communalities. Absolute values less than .10 were suppressed and omitted from the cells.

4.4 Dimensions under the SELTQ

The newly designed Student Evaluation of Learning and Teaching Questionnaire (SELTQ) has four dimensions. The first dimension combined items that were presented under (1) evaluation and assessment and (2) feedback. Items under assessment and evaluation address whether assessment approaches evaluate comprehension, if students have been informed about the nature of assessment ahead of time, and if assessment was part of the instructional process (e.g. Assessment was designed to evaluate understanding). Items referring to the nature of feedback address issues related to whether there was prompt feedback following performance, if discussions were held following suggestions from students, and whether the instructor provided comments to draft works before submitting the final verson (e.g. The teacher revised assignments before exam).

The second dimension contains items referring to course organization and presentation.The items include whether teachers inform students about the objectives and tasks related to various topics, the perceived appropriateness of instructional methods to topic objectives, the encouragement given by the teacher to involve students in the process, and the opportunities for student-teacher discussion (e.g. The teacher created opportunities for dialogue).

The third dimension includes items referring to students’ self assessment of their learning. Items under this factor require students to make an evaluation of whether they knew what was expected of them in the process of learning, whether they have been asking the teacher any time they needed help, if they have been implementing suggestions provided by the teacher, and if they believe they are learning better as a result of their involvement (e.g. I implemented the feedback from the instructor).

The fourth dimension contains items refering to students’ level of engagement. Items that refer to students’ involvement in peer and self evaluation, in group projects and assignments, and in providing alternative explanations and elaborations to answers are included (e.g. I was encouraged to assess my own progress). It can be judged from the items that loaded into the four factors that the hypothesized factor labels were the same for the three paradigms, i.e. course organization and presentation, student self assessment, and student engagement. Only the first factor that combined the items had to be renamed into assessment and feedback. Table 1 reveals the paraphrased version of the items and factor coefficients.

Table 1 Summary of factor pattern loading (n = 204)
Full size table

4.5 Reliability estimates

Internal consistency reliability using Cronbach’s alpha was conducted on the 24-item scale. Results indicated that the standardized alpha coefficient for the scale was 0.93, revealing a high degree of reliability. Item-scale correlation confirmed this statistics, with all items exhibiting strong item-to-scale correlation. Only four items would make a slight difference to the overall reliability if deleted. However, as the increment is not significant, all the 24 items were retained in the scale. What is more, alpha coefficients for Course Organization and Presentation (.84), Assessment and Feedback, (.77) and Student Self Assessment (.85) were found to be high, apart for the Student Engagement subscale (.68).

5 Discussion and implications

In this study, the Student Evaluation of Learning and Teaching Questionnaire (SELTQ) was designed based on students’ experiences in learner centered instruction and a review of related works. Psychometric analyses revealed high internal consistency, reliability and good content and construct validity. Four underlying dimensions were identified, namely, (1) students’ self assessment, (2) students’ level of engagement, (3) the quality of feedback students give and receive, the appropriateness of the assessment employed, and (4) the way the course content is organized and presented. These dimensions explained 56% of the variance in ratings.

Certain peculiar features of the SELTQ are worth noting. The most commonly used student ratings are designed based on selected characteristics agreed by students and teachers to be indicators of teaching effectiveness (Marsh 1987). In the SELTQ, the items are designed based on students’ reflections of their experiences in learner centered instruction as well as a review of studies on student learning. In addition, the effectiveness of teaching is determined using dimensions that reflect teacher support and students’ evaluation of their learning. Students are expected to provide valuable information that reflects their level of engagement in the process of learning. Thus, the appropriateness of the measure can be seen in terms of combining indicators that address both the effectiveness of teacher performance and students’ evaluation of their own learning. Ory and Ryan (2001) have also stated that student rating questionnaires will be more valid indicators of teaching quality if they address all what effective teaching encompasses: students’ active engagement as well as the support and facilitation from the teacher.

Another difference between the SELTQ and the previously used Teacher Evaluation Questionnaire (TEQ) is that the new one has a factor that contains items referring to students’ self assessment of their learning. The items under this factor are designed in such a way that students can evaluate whether they have been accomplishing all the required tasks and, as a result, believe they are learning better. It can also inform students that they are responsible for their own learning. Equally importantly, the feedback students provide to teachers about their learning is also believed to help teachers improve students’ learning experiences.

The second factor that focuses on student engagement is also not commonly addressed in rating scales that give more emphasis to teacher performance. Students are the center of attention, and this is expressed in terms of involving them in the process of learning. Since what students accomplish during learning is equally important, a measure of teaching quality should provide opportunities for students to evaluate their level of engagement. The new questionnaire enables teachers to obtain feedback about students’ level of engagement in such a way that they can make improvements in teaching.

The dimension that deals with evaluation and feedback is not new as such since items referring to the validity of exams and the nature of the feedback provided have been included in the TEQ as well. However, the difference in the new rating scale is that these items are designed to help students evaluate the effectiveness of assessment and feedback in facilitating their learning. Students are required to judge whether the evaluation practices were contributing to their learning. The items focus on the extent to which students gave feedback for teaching improvement and if teachers made improvements following suggestions from students.

As the main objective of teaching is student learning, the effectiveness of teaching has to be determined using feedback that reveals the quality of the learning experiences described by students. The new measure has dimensions that enable students to evaluate the effectiveness of the teaching in terms of their own learning experiences. It also addresses students’ judgments of teacher performance and evaluation of their learning progress.

In conclusion, the findings of the study are believed to contribute to the discussion on the need for an improved measure of teaching quality. The current measure, the TEQ, focuses on evaluating teacher performance, not student learning. This, however, may not enable teachers to receive information about students’ learning progress as the feedback exclusively refers to the effectiveness of teachers’ presentation skills. Nor will they be able to improve their teaching unless the feedback informs them about what worked for students and what needs to be improved. Student engagement in learning should be given equal emphasis to teacher’s facilitation. In this regard, the dimensions that were identified in and make up the SELTQ reveal the emphasis given to student learning as a way of evaluating teaching quality. Unlike the TEQ, the new questionnaire combines both students’ evaluation of their learning as well as their teachers’ guidance and support. If the new questionnaire is effectively implemented, not only will it provide teachers with feedback about how they can promote student learning but it will also help students develop their ability to reflect on their learning. They will be able to evaluate their own learning progress as a way of evaluating the effectivenes of the teaching. The contents in the new questionnaire reflect that teaching is effective not only when teachers teach but also when students learn. The use of the questionnaire is also believed to change the widely held conception about teaching and learning, i.e. from teaching as transmitting information to teaching as facilitating learning. The dimensions in the improved questionnaire reflect that students are not passive spectators but active players in their own learning.