CRiSP: An Instrument for Assessing Student Perceptions of Classroom Response Systems

Abstract

This paper describes the development and validation of an instrument for evaluating classroom response systems (CRS). While a number of studies evaluating CRS have been published to date, no standardised instrument exists as a means of evaluating the impact of using the CRS. This means that comparing the different systems, or evaluating the benefits of using the CRS in different ways or settings, is very difficult despite the number of published reports, as indicated by Kay and LeSage (2009). An instrument was developed, called the classroom response system perceptions (CRiSP) questionnaire, which allows the evaluation of varied CRS on three scales: the usability; the impact on student engagement; and the impact on student learning. The development of CRiSP was undertaken in three universities, using different CRS, and evaluated through focus groups, one-on-one interviews and a factor analysis of the survey responses. We found no evidence of difference on the scales according to gender or age groups. The final CRiSP questionnaire consists of 26 base questions, with additional optional questions available. This paper proposes that the CRiSP Questionnaire could, in its current state or with minor changes, be used to evaluate the impact on learning of other classroom technologies also.

Appendix: The items in the revised questionnaire

The 26 ordinal scale items used in the initial questionnaire. All items are answered on a five-point ordinal scale (from Strongly Disagree to Strongly Agree)

Short description	Complete wording
Wasted time	Using [CRS] in lectures wasted too much time
Recommend use	I would recommend that the lecturer continue to use [CRS]
Overall value	The use of [CRS] helped increase the classes’ overall value
Motivation	[CRS] used in this unit motivated me to learn
Interaction	I found this method of interaction between students and lecturer effective
Instant feedback	[CRS] helped me get instant feedback on what I knew and didn’t know
Peer awareness	The use of [CRS] helped increase my awareness of my peers’ opinions and attitudes
Understand concepts	[CRS] allows me to better understand key concepts
Instructors used results	My instructor used the results from [CRS] questions to gauge class understanding and reinforce material that was not understood
Enhanced learning	Using [CRS] questions enhanced my learning of the subject
Control over learning	I believe that [CRS] provided me with more control over my learning than in units that do not use [CRS]
Think deeply	Using [CRS] helped me think more deeply about course materials
Correct but not understand	I often voted for the right answer without really understanding
More confident	Using [CRS] made me more confident to participate in class
Mostly used	I used [CRS] most times when it was used in class
Increased participation	[CRS] increased the frequency of my direct participation in the course
Active	The use of [CRS] helped me to be active in class
Pay attention	Using [CRS] helped me pay more attention in class
Concentrate	Using [CRS] has helped my concentration levels in lectures
Attendance	Using [CRS] has encouraged me to attend lectures
Easy to use	For me it was easy to use the [CRS] voting system
Too difficult	For me [CRS] was too difficult to use
Expectations hard	It was too hard to know what was expected of me using [CRS]
Tech problems	There were too many technological problems using [CRS]
Increased enjoyment	Using [CRS] has increased my enjoyment of lectures
Anonymity good	Other students could not see my answers, which encouraged me to be an active participant in the class

1. One item was omitted that appeared in the initial survey: Prefer More (“I would prefer more [CRS] questions during each class meeting”)

Other questions

Short description	Complete wording
Why Vote	When you did choose to vote in a [CRS] question, what was your reason for choosing to participate? Please select all that apply.
Why not vote	When you did not vote in a [CRS] question, what was your reason for choosing not to participate? Please select all that apply.
Age	Age
Gender	Gender

The following items are all answered on the five-point ordinal scale (from Strongly Disagree to Strongly Agree) unless noted otherwise.

Items for use with CRS that use phone-based technology:

• Do have access to a phone in class?

• Possible responses include: All classes; most classes; some classes; never.

• What company is your phone provided? (As used by Scornavacca et al. 2009):

• Students should select from a closed list of options, plus an “Other (specify)” option.

• I was distracted from the class after using my phone in class.

• I was distracted by other people using their phones in class after a [CRS] question was asked.

• I often failed to get a sufficiently strong signal to be able to use my phone.

• I didn’t like using my phone to vote.

For instructors who have used, or who are considering using, CRS technology to contribute towards grades:

• I would like to receive credit for the correct responses chosen while using [CRS]. That is, I believe that responses should be graded in some fashion.

• I would like to receive credit for using [CRS]. That is, I believe that participation in the voting using [CRS] should contribute towards grades in this course.

For CRS that require students to register before, they can use the technology:

• Registering to use the [CRS] was too much hassle for me.

• Registering to use the [CRS] was too difficult for me.

• Registering to use the [CRS] was too time-consuming for me.

• I did not want to register to use the [CRS] as I was required to supply some personal details.

For situations where the use of the CRS is anonymous for students to instructors:

• I was encouraged to vote because the lecturer did not know how I voted.