Background

Much time, money, and effort has been spent in developing innovative teaching pedagogies, documenting their effectiveness, and disseminating the results. Although these efforts have had some impact on teaching practices, there has been no systematic movement of college instruction at large research institutions toward consistency with research-based best practices (e.g., Wieman et al. [2010], Dancy and Henderson [2010], Handelsman et al. [2004], Henderson and Dancy [2009a], National Research Council [2003], Redish [2003]). Reformers typically develop a new pedagogy at their institution and then disseminate the new instructional model through talks, workshops, and papers. This development and dissemination model of reform assumes that telling faculty about good teaching ideas will lead faculty to integrate these ideas into their teaching practices. However, this intuitive ‘show them and they will adopt’ model has yet to produce desired large-scale transformations, indicating that a more robust research-based model of change is needed (Fairweather [2008]; Seymour [2001]; Henderson et al. [2011]).

In addition to lacking an overall model of how new teaching ideas spread, very few studies document what happens when secondary sites attempt implementations. Usually, an individual or small group develop a reformed pedagogy and disseminates it for use in many different educational settings. However, the unique combination of students, instructors, and structures at each location complicate transitions to other sites (Finkelstein and Pollock [2005]; Pollock and Finkelstein [2007]; Sabella and Bowen [2003]; Saul and Redish [1997]; Sharma et al. [2010]). Furthermore, secondary implementers often lack the grant funding, a project team, faculty release time, and education experts that contributed to success at the development site. As the most common type of site, secondary implementations deserve close attention especially from reform developers who want their innovative teaching strategies to spread successfully.

These sites also deserve more attention because secondary implementers can make significant modifications that can comprise the intended results of the reform. Instructors often adapt reform, and often, these changes are in the direction of traditional instruction (Dancy and Henderson [2010]; Henderson [2005], [2008]; Henderson and Dancy [2009a]; Henderson and Dancy [2009b]). Without investigating how the implementation has been adapted, ‘it may be difficult to interpret learning outcomes and to relate these to possible determinants’ (Fullan and Pomfret [1977], p. 338). This is especially important for less successful sites, since lower-than-expected gains can disappoint and frustrate adopters. For example, researchers studying learning gains at a secondary site found that improvements in student learning after using interactive lecture demonstrations were ‘nowhere near’ those claimed by developers (Sharma et al. [2010]). The researchers were unable to identify the reasons for these disappointing results. To understand why some sites are more successful than others, it is important to monitor the details of implementation, not just the reform they claim to be using.

This project examines how the Student-Centered Active Learning Environment with Upside-down Pedagogies (SCALE-UP) (Beichner et al. [2007]; Beichner [2008]) reform spread and how secondary sites implement the reform. Our intent is that this study will (1) contribute to a research-based understanding about how reforms spread, (2) identify typical modifications made in secondary implementations, and (3) develop recommendations to guide curriculum developers toward more successful dissemination. Although this study is focused on the SCALE-UP instructional strategy, we believe that the results are relevant more widely to instructional change in higher education.

What is SCALE-UP and why examine it for this study?

Dr. Robert Beichner developed SCALE-UP for large enrollment university physics courses at North Carolina State University in 1997. As we have found in this study, this reformed pedagogy and classroom environment has successfully crossed disciplines and continents and is currently used in at least 314 departments at 189 higher education institutions worldwide. This study closely examines the dissemination and implementation of this specific instructional strategy in order to develop general trends that may help other educational innovations become successful, like SCALE-UP.

SCALE-UP is a radical reform. In SCALE-UP, instructors modify their pedagogy to minimize lecture and make major physical changes to the classroom arrangement. When implemented well, radical reforms like SCALE-UP have been shown to lead to higher learning outcomes than more modest reforms (Redish [2003]). SCALE-UP changes the instructional methodology and course structure by integrating lab-lecture recitation and redesigning the classroom to a studio environment. Round or D-shaped tables, whiteboards on walls, and technology with projection capabilities facilitate collaboration and sharing of student work. No obvious ‘front’ of the room encourages instructors to minimize lecture, circulate and engage teams of students in Socratic dialogs, real-world problem solving, and technology-rich activities. Reforms that involve structural changes, like SCALE-UP, require departmental buy-in and thus may be harder to adopt. We hypothesize that this investment in structural changes may also decrease the chance of discontinuation once the reform is in effect.

SCALE-UP presents an interesting case study for multiple reasons. First, SCALE-UP is not discipline or curriculum specific. The large number of secondary sites allows us to identify discipline-based differences and investigate the ways people use the reform in an assortment of institutional settings. Additionally, information about SCALE-UP has been successfully disseminated through a variety of formal and informal means, both by the developer and by other sites. Finally, since so many sites have adopted SCALE-UP at different times over the past two decades, we can examine sites at different implementation stages and ones that have achieved different degrees of spread within the universities.

SCALE-UP has been shown to improve student problem-solving abilities, conceptual understanding, attitudes toward science, retention in introductory courses (Beichner et al. [2007]; Beichner et al. [2000]), and performance in later courses (Dori et al. [2003]). To spread these benefits successfully, it is extremely important to understand how secondary implementations of highly innovative curricula, like SCALE-UP, work. By examining why they are adopted, if/how modifications adjust the reform to their unique context, and the impact these modifications have on student learning, we can promote the diffusion of effective and sustained reforms.

Literature review

This section briefly summarizes several claims from the diffusion of implementations (DOI) literature and innovative teaching research that are relevant to this study. According to Rogers ([2003]), diffusion is the process in which an innovation is communicated through certain channels over time among the members of a social system. DOI theory describes this process, where socially constructed information about a new idea is communicated from person to person. Rogers ([2003]) developed these ideas after analyzing a wide variety of innovations - from farmers using a new hybrid seed to physicians utilizing a new drug to corporations incorporating a new technology - and the theory also applies to educational innovations (Nutley et al. [2002]; Berman and McLaughlin [1978]; Green and Johnson [1996]). We chose this framework since it explores the dynamic process between communication and innovation implementation in a social context. We hope it will help us understand how information about SCALE-UP is communicated and how this reform is adopted and utilized and changes behavior at secondary sites. This framework does not assume a simple, linear progression from product development to adoption and thus allows us to explore the stages of spread in detail. According to the DOI theory, the spread of information about an innovation and the implementation process is influenced by four factors: innovation attributes, communication channels, adopter characteristics, and adaptation.

The literature claims that the following factors influence the spread of information about an innovation and its implementation at a secondary site.

Innovation attributes

Rogers ([2003]) identified several innovation attributes that aid rapid diffusion. We consider these here, in relation to SCALE-UP. First, successful innovations are perceived to have a significant relative advantage over current alternatives. As previously mentioned, as a radical reform, SCALE-UP has been demonstrated to dramatically improve student learning (Beichner et al. [2007]; Beichner et al. [2000]; Beichner [2008]). Thus, we believe that departments seeking to substantially change their educational techniques will perceive SCALE-UP as a way to significantly change educational outcomes. Secondly, users are more likely to adopt innovations with high-perceived compatibility with past practices, current values, and existing needs. SCALE-UP is unlikely to be compatible with past practices in most traditional departments. Thus, it seems like that more innovative departments, who have been trialing more interactive educational pedagogies, may be among the first to adopt. Next, users are more likely to adopt low complexity innovations that can be readily understood and easily implemented. Although SCALE-UP requires many structural changes to implement, the underlying principles are relatively straightforward and intuitive to understand. Adopting a pedagogy and classroom space to support active and collaborative learning (Beichner [2008]) seems sensible to most users, which can aid the spread of SCALE-UP. Fourth, users are more likely to adopt innovations with high trialability, where new ideas can be tried out at a low cost before it is completely adopted. Since SCALE-UP requires modifying the classroom space and often the schedule (to combine the lecture, lab, and recitation sections of a course), a high-fidelity SCALE-UP implementation is not cheap or easy to trial on a temporary basis. However, we believe that once departments make the investment, observability, the fifth characteristic of successful reforms, significantly facilitates the success of SCALE-UP. Observability measures the degree with which others can view the use and benefits of the innovation, thus stimulating further uptake by others. Since a SCALE-UP classroom looks so different from a traditional lecture hall, we believe that observability will be a key factor that promotes the spread of SCALE-UP within and between institutions.

In addition to these classic innovation characteristics identified by Rogers, other researchers have identified potentially important characteristics, such as adaptability, centrality to the day-to-day work of the organization, and the minimal requirement for additional visible resources (Wolfe [1994]). For our study, another aspect of SCALE-UP is that a department, not an individual, is often the adopting unit. Since SCALE-UP requires such a dramatic restructuring and often a financial investment, the decision to adopt is often made at the department level. Thus, instead of analyzing the diffusion of ideas among individuals, we chose to look at the spread of ideas among departments and elect to use that grain size for the majority our study. This choice is supported by other research on innovations in organizations (Katz [1962]; Nutley et al. [2002]; Lewin [1947]; Wildemuth [1992]; Zaltman et al. [1973]) but can introduce complicating factors since the strength of evidence on whether adoption will lead to improved effectiveness does not seem to be the main factor influencing adoption decisions (Stocking [1985]; Westphal et al. [1997]) but instead depend more on current fads and fashions, especially during times of high uncertainty (DiMaggio and Powell [1983]).

Communication channels

According to DOI, the communication channels through which the message spreads also influence the rate of adoption. A distinction is generally made between interpersonal, where messages originate from local sources, and mass media channels, where messages comes from distant sources (Rogers [2003]).

Disseminating educational innovations via ‘mass market channels’ (Rogers [2003]) such as websites, conference presentations, and journal publications can raise awareness of an innovation. However, to move potential users from learning something new to changing their practices, interpersonal channels are often more influential when it comes to behavior (Borrego et al. [2010]; Rogers [2003]).

Furthermore, intermediaries between the innovators (curriculum developers in our case) and the broad mass of potential adopters are often an important communication channel between these dissimilar groups. These include opinion leaders who adopt the innovation themselves early on and tend to be more innovative and of a higher status than their near peers. This helps more conservative middle and late adopters overcome caution about the risks involved with adopter. Secondly, intermediary change agents can work proactively to expedite and widen innovation usage by reducing barriers, persuading adopters, and supporting adoption decisions. These leaders bridge the divide between technical experts and clients so they can mediate interactions and work effectively with both groups. The credibility of the change agent and level of contact with the change agent are both positively related to the decision to adopt (Rogers [2003]; Colemen et al. [1966]).

In this project, we investigate how people learned about SCALE-UP. In addition to mass market channels, we probe social connections - which theory indicates are essential to the reform process - through a SCALE-UP census. Our survey identifies users, asks how they learned about the reform, and examines whether the communication method impacts their use of the reform. We hypothesize that mass-market channels may be important early on but that interpersonal channels accelerate the spread of SCALE-UP.

Adopter characteristics

Typically, an innovation needs a critical mass of adopters before it can achieve a self-sustaining spread. For successful dissemination, the rate of adoption typically follows an S-shaped curve like that shown in Figure 1, where the rate of adoption ‘takes off’ once interpersonal networks become activated in spreading the idea from peer to peer within a system. The adoption rate begins to level off after half of the social system uses it, since non-knowers become scarce.

Figure 1
figure 1

S-shaped adoption curve, adapted from Rogers ( [2003] ) with adopter categories. The thick vertical line denotes the ‘chasm’ between adoption by the adventurous early users (16% of adopters) and more hesitant early majority.

Diffusion of innovation theory allows us to categorize adopters' characteristics based on their tendencies to adopt (Rogers [2003]; Moore [2002]). Although the characteristics and S-curve percentages are generalizations, developers should pay attention to this curve and ideally target each group with a different diffusion strategy (Green and Johnson [1996]). As shown in Figure 1, Rogers describes the first 2.5% of adopters as innovators. Innovators tend to be social, willing to take risks, and closely connected to scientific sources. Early adopters, the next group, are also educated, have a high social status, and typically have the highest opinion leadership of the adopter groups. According to Rogers, innovators and early adopters both respond to the newness of an idea and are not deterred by things that may not work perfectly. However, the early majority are much more conservative and cautious, which is when change agents and opinion leaders become important in spreading the idea amidst this population.

The implication is that change initiatives should change their strategies in accordance with changing adopter characteristics if they want their innovation to spread to the mainstream. Once an innovation has reached the innovators and early adopters (somewhere around 16% adoption), the marketing strategy needs to change to appeal to the more hesitant early majority (Moore [2002]; Rogers [2003]).

In this project, we will compare the diffusion of SCALE-UP to the S-curve and identify how close SCALE-UP is to the 16% tipping point. If SCALE-UP passes this tipping point, dissemination may need to involve change agents and opinion leaders to convince the mainstream.

Role of adaptation

Diffusion can range on a continuum from highly centralized (top down diffusion from expert to users) with a low degree of adaptation to decentralized systems with a high degree of local adaptation (where diffusion happens through horizontal networks with a wide sharing of power and control among members).

In general, research and development agencies tend to promote a high fidelity, top-down approach and may consider reinvention as a distortion of their original technologies. Thus, historically, dissemination of reforms has taken a linear, top-down approach in the development and dissemination model, presenting faculty with a packaged ready-to-go curriculum designed for unmodified use. This model does not acknowledge the expertise of adopting faculty, who rarely want to use reforms as designed and may adapt reforms back toward traditional instruction (Hutchinson and Huberman [1994]; Henderson and Dancy [2008]).

On the other hand, adopters tend to think that reinvention is a desirable quality because it promotes a closer fit with their needs and studies in education. Research suggests that instructors are more open to a collaborative approach, where researchers and users exchange specialized knowledge in a mutually constructed social context. As Cohen and Ball explain, ‘Teachers view themselves as independent, autonomous professionals…Even the most obedient and traditional teachers observed, enacted policies in their own ways and were proud of their contributions’ (1990, p. 253).

Researchers have shown that reinvention not only increases the likelihood of adoption but also reduces the likelihood of discontinuance (Berman and Pauley [1975]; Rogers [2003]). However, these same educational studies also suggest that when the level of reinvention is quite high, then the desired outcomes can diminish substantially (Henderson and Dancy [2008]; Penberthy and Millar [2002]; Sharp and McLaughlin [1997]; Silverthorn et al. [2006]; Hutchinson and Huberman [1994]; Berman and Pauley [1975]; Berman and McLaughlin [1978]).

This project identifies how instructors and departments adapt SCALE-UP, pedagogically and structurally, and examine how that relates to adoption, implementation, and sustainability. We hypothesize that a department's commitment to redesign the classroom may help reduce the tendency to revert back to traditional instruction, which can occur in educational reforms where instructors have freedom to make changes.

The literature lacks consensus regarding how to promote effective spread of complex innovations like SCALE-UP. SCALE-UP restructures the classroom environment and pedagogy in a way that often requires a formal decision by higher members of an organization. The larger adoption unit - a department in our case - for innovations like these introduce complicating factors which means that much of the literature on individual adoption of simple, product-based innovations (for example, in educational technology) cannot be generalized (Greenhalgh et al. [2004]). Since radical reforms like these have the best chance of improving student outcomes (Redish [2003]), the field still needs to examine how involved innovations spread effectively. This project aims to deepen our understanding of this by using the case of SCALE-UP to study how complex research-based reforms spread, within and across institutions, including how people learn about and implement them.

Research questions

  1. 1)

    To what extent has SCALE-UP spread? Are these rates of spread consistent with diffusion of innovation theory?

a How many physics departments has it spread to?

b How many other disciplines has it spread to?

c How much does SCALE-UP spread within a department?

d Are these rates consistent with diffusion of innovation theory?

  1. 2)

    Through what communication channels does knowledge about SCALE-UP spread?

  2. a.

    How do adopters learn about SCALE-UP?

  3. b.

    Does the communication channel affect the way adopters use the reform?

  1. 3)

    How do secondary sites adapt the pedagogy and structure of SCALE-UP? How does this impact the success of the implementation?

  2. a.

    How does the pedagogy and structure of enacted implementations vary by user status? Discipline?

  3. b.

    How does structural variation (classroom type) relate to the use of active learning pedagogies?

Methods

Data were collected via a web survey of faculty likely to be aware of SCALE-UP. This section will first describe the survey design and then discuss the survey implementation.

Survey design

The survey questions were designed to address the research questions and census SCALE-UP use internationally. Since we were exploring a respondent's interactions with and interpretations of SCALE-UP specifically, we were not able to use the preexisting survey instruments. The research team reviewed the literature and created a list of potential survey questions, based on past research and personal experience. For the questions designed to measure fidelity to the original reform we had, Beichner, the developer of SCALE-UP, describe his model and share what he has observed that is happening at other sites. This helped us create question options that captured subtle differences. The survey included multiple choice, multiple select, and free response questions that allowed respondents to describe their implementation. The survey is included in Additional file 1.

The web-based survey contained three main parts: (1) questions about the respondent's personal use of SCALE-UP at their institution (demographic information, instructional practices, self-described user status), (2) questions about SCALE-UP use in the respondent's department/institution (physical room set-up, contact person, duration of use and percentage of instructors using SCALE-UP in the department, other SCALE-UP departments at the institution), (3) questions about the spread of SCALE-UP (How did you learn about SCALE-UP? Who did you tell about SCALE-UP?). Additional survey details and specific questions will be discussed as needed in the ‘Results and discussion’ section. Branching was used so the survey would be appropriate for instructors, administrators, as well as past/potential/current users.

In order to improve the quality of data collected, the survey triangulated responses in two ways. First, the survey contained some questions that were designed to get at the same construct, but in different ways. Responses to these questions were compared during analysis. The second type of triangulation was that in many cases, we had multiple respondents in the same institution and department. Thus, we were able to determine the consistency of responses for questions relating to the department and institution.

We pilot tested the preliminary survey with three faculty members from three different universities and two different disciplines. After they completed the survey on their own, we interviewed these individuals and used their feedback to improve the clarity and appropriateness of the questions. After that, we distributed the survey to a small sample of 50 recipients to verify smooth administration of the instrument before inviting additional respondents.

Survey implementation

The goal of the survey was to develop a census of SCALE-UP use at higher education institutions. Current, past, or potential users of SCALE-UP style instruction were the target population of the survey. We did not wish to sample from this population, but rather to survey the entire population. This, of course, is an unreachable goal. We used several techniques in the survey implementation to identify respondents in our efforts to approach this goal of a SCALE-UP census.

First, surveys were sent to all current members of the SCALE-UP wiki database (http://scaleup.ncsu.edu/). The SCALE-UP wiki is a password-protected collection of resources about implementing SCALE-UP style instruction. At the time of the survey, there were 1,321 members of the wiki, virtually all of whom had contacted Beichner requesting information about SCALE-UP. In addition, we created an open survey link. This open link was distributed via relevant listservs (e.g., physics modeling, Arizona State University science faculty). Individuals were encouraged to share the link with other instructors in their discipline. Finally, snowball sampling was used to identify additional respondents. During the survey, the respondents were asked from whom they had learned about SCALE-UP and whom they told about SCALE-UP, including a contact person in their department. Survey invitations were sent to any people listed whom we had not previously contacted.

To ensure the thoroughness of the sample, we conducted internet searches to identify additional institutions using SCALE-UP instruction. Of the top 30 institutions returned by an Internet search, all but two of these sites were already in the database and had already been contacted for the survey. This provides some evidence that our list of survey respondents was reasonably complete. Note that we do expect that not all SCALE-UP implementations publicize their use of the innovation with a website.

Approximately 1,300 survey invitations were sent in rounds between December 2012 and August 2013. The e-mail invitation described the goal of the survey: conducting a census of people using SCALE-UP style instruction. Thus, people who filled out the survey associated themselves with SCALE-UP use, or at least acknowledged that the reform influenced their teaching. Three reminders were sent to non-respondents, and individuals could elect to be removed from the list if they thought the survey was irrelevant. In the end, 812 surveys were started with 84% of these respondents completing the entire survey. For this study, responses were only retained from respondents at American higher education institutions, leaving a sample of 659, a better than 50% response rate. SCALE-UP originated in the US so we chose to focus on universities with a similar cultural context. In other countries, different social and cultural norms could complicate the way SCALE-UP was implemented and we wanted to remove cultural factors from adding an additional variable to the study. Furthermore, since 84% of respondents came from the US, we decided to focus on getting a detailed understanding of domestic implementations.

We analyzed completed responses for each question answered, even for incomplete surveys. For some results, we used the department as the unit of analysis, combining responses from multiple respondents in a single department to form a picture of implementation at this level. When responses differed, we either went with the majority of respondents (if there was a majority that answered in the same way) or took an average response if there was not a majority.

Limitations of the study

Using a survey as a mode of data collection inevitably introduces some limitations to our study. Survey methods rely on self-report, which has been critiqued on various grounds including limited internal and ecological validity (Podsakoff and Organ [1986]; Stone et al. [1999]). In the field of higher education, researchers have expressed concerns about the ability of instructors to accurately report classroom happenings accurately in sufficient detail (e.g., Dancy and Henderson [2010]; Ebert-May et al. [2011]). Aware of these challenges, the survey did not ask respondents to provide a cohesive picture of all their teaching practices, just the key aspects that relate specifically to SCALE-UP. The questions were designed to be as general and direct as possible in an effort to avoid misrepresentations. For example, instructors were asked to select whether certain equipment was present in their classroom (whiteboards on walls, computers for small groups to access, projection capability) or estimate the percentage of time they spent lecturing, students spent problem solving, etc. (1% to 25%, 25% to 50%, 50% to 75%, 75% to 100%).

When there were multiple respondents from a department, we checked the responses on relevant questions against each other for agreement (for example, percentage of people in a department using SCALE-UP to teach introductory classes, years SCALE-UP has been used at a department, structural equipment in classroom). Having multiple respondents report similar descriptions of how SCALE-UP is used in their departments helps reduce some concerns about self-report. In some cases, there were up to five instructors from the same department whereas other departments only had one respondent. The questions analyzed at the departmental level, as all questions on the survey, were as general and direct as possible. Significant disagreements among faculty within a department were rare. Another limitation of our study is that we used a broad definition of SCALE-UP ‘users’ because almost all faculty members make modifications and have difficulties characterizing their use. Early in the survey, respondents were asked to describe their user status of SCALE-UP style instruction, which we defined in the survey as ‘SCALE-UP style instruction is characterized by the promotion of social interactions among students and instructors, use of engaging activities during class along with a substantial reduction in lecturing, and a focus on developing conceptual understanding and thinking skills. By having students attempt difficult tasks in a supportive environment, they experience the application of their new knowledge. Students have opportunities to practice teamsmanship, presentation of their own work, and evaluation of the work of others. Classroom furnishings are specifically chosen and/or arranged to facilitate this type of collaborative, interactive, guided inquiry’. The response choices were ‘user’ (25%), ‘modified user’ (33%), ‘influenced by’ (24%), ‘considerer’ (10%), ‘past user’ (4%), and ‘never heard’ (3%). For the purposes of this paper, we consider ‘users’ to be anyone who responded that they were ‘users’ as well as people claiming to be ‘influenced by’, ‘using,’ or ‘considering the use of’ the SCALE-UP. The reason for this broad definition of user is that in previous work (Dancy and Henderson [2010]), we have found that respondents interpret self-reported user categories in very different ways. As described above, instructors may only ‘use’ innovations in a very loose sense. For example, more knowledgeable instructors may deny being a SCALE-UP user if they consciously made a minor modification in their implementation while less knowledgeable instructors who are using only a few ideas from SCALE-UP may claim to be users. Additionally, since nearly all instructors make modifications, it becomes difficult to define ‘use’. As reported later in the analysis, different user categories had very few statistically significant differences in their use of pedagogical and structural aspects of SCALE-UP, thus justifying our decision.

In summary, relying on survey data inevitably adds concerns regarding self-report data. However, since empirical studies on the spread of complex educational innovations, such as SCALE-UP, is limited, these exploratory results lead to interesting and worthwhile hypotheses that we plan to further confirm and refine with interviews and site visits in subsequent phases of our study.

Results and discussion

RQ1: to what extent has SCALE-UP spread? Are these rates of spread consistent with Diffusion of Innovations theory?

1a. How many physics departments has it spread to?

Across all disciplines, the survey identified 314 departments at 189 institutions in 21 countries that claim to be influenced by or using some version of SCALE-UP style instruction. Over one third (114 departments) use SCALE-UP in physics (or related fields), making physics the most represented discipline. It is, of course, not surprising that SCALE-UP is most widely used in physics since SCALE-UP was originally developed for university physics courses.

1b. How many other disciplines has it spread to?

The other 63% of non-physics departments include over a dozen disciplines, as seen in Table 1. Most SCALE-UP implementations (81%) are in STEM departments. For the rest of the analysis, results presented by discipline will not display the ‘other STEM’ category because of its small percentage that includes diverse subject areas.

Table 1 Distribution of departmental affiliation for survey respondents who indicate that their department uses or has been influenced by SCALE-UP

Most implementations are in STEM departments, the majority of which are in institutions that award graduate degrees in their discipline. Thus, as expected, SCALE-UP is most widespread in schools and disciplines similar to the development site but, encouragingly, this census shows that education innovations can cross institution types and disciplines. Adopters have a wide range of characteristics, including the fields far from the originating field, such as social sciences and the humanities.

Other notable findings include:

16% of the departments using SCALE-UP are outside the United States

61% of the departments using SCALE-UP are from 4-year colleges/universities with a graduate degree in their discipline

23% of the departments using SCALE-UP are from 4-year colleges/universities with bachelors degrees in their discipline

8% of the departments using SCALE-UP are from 2-year colleges

8% of the departments using SCALE-UP are from other higher education institutions (those offering professional degrees, 3 year programs, 4 year programs without a degree in the discipline, etc.).

As discussed earlier, in the remainder of the paper, only results from United States institutions will be reported.

1c. How much does the reform spread within a department?

Even though SCALE-UP has spread widely across departments and institutions, within most departments, this mode of teaching is not the status quo. When compared by discipline, Figure 2 demonstrates that physics has the highest intradepartmental use of SCALE-UP in introductory courses. So even though a growing number of departments use SCALE-UP, across all disciplines, a minority of instructors use SCALE-UP style instruction in introductory classes. This indicates that this method of teaching is still relatively rare, and thus, the reform is still in the early stages of diffusion within most departments.

Figure 2
figure 2

Percentage of instructors within a department using SCALE-UP style techniques to teach introductory courses.

1d. Are these rates of spread consistent with diffusion of innovation theory?

According to diffusion of innovations theory, innovations that successfully spread follow an S-shaped adoption curve as shown in Figure 1. To see the spread of SCALE-UP over time, the survey asked respondents how long SCALE-UP style instruction had been used at their institution. As seen in Figure 3 below, physics has been widely using SCALE-UP the longest. The number of biology and non-STEM implementations jumped within the past 4 years. The time scale is not resolved enough to compare this figure to the typical diffusion S-curve (Figure 1) but the general pattern of the beginning stages of spread is similar. Locating SCALE-UP's position on this curve will allow change agents to orient themselves according to the users they may be reaching and may need to reach out to, to ensure the reform's continued spread along the curve.

Figure 3
figure 3

SCALE-UP departments in existence, by discipline. The 2013 survey asked respondents how long SCALE-UP had been used in their department with options: more than 6 years, 4 to 6 years, 1 to 3 years, and less than 1 year.

Unsurprisingly, the originating discipline of physics has the largest number of implementations using SCALE-UP the longest. The number of non-physics departments using this reform is increasing.

As mentioned in the literature review, DOI theory suggests that early adopters are different from later adopters. Thus, it is important for change agents to locate where their innovation is in the adoption curve. To identify where the use of SCALE-UP by US physics departments is on the adoption curve, we compare the 94 US SCALE-UP departments that offer physics degrees to the 751 undergraduate physics degree departments in the US (Nicholson and Mulvey [2012]). This suggests that SCALE-UP is currently (2013) used in about 12% of US physics departments that offer a physics degree. This percentage is approaching the 16% threshold that represents the shift between the early adopter and early majority stages predicted by DOI theory.

While reviewing the names of survey respondents using SCALE-UP in physics departments, we noticed that many actively participate in the physics education research community, a finding again consistent with DOI, which claims that early users are well educated and innovative in their fields. To continue the diffusion of SCALE-UP among the next group of ‘early majority adopters’, which Rogers ([2003]) characterizes as deliberate followers who seldom lead, SCALE-UP may need to change its message to appeal to a more mainstream faculty population. SCALE-UP spread appears to be in the early stages for other disciplines, so there is less urgent need to change dissemination strategies in subject areas where adventurous early users continue to adopt since developers can continue to appeal to their desire to be on the leading edge.

RQ2: through what communication channels does knowledge about SCALE-UP spread?

2a. How do adopters learn about SCALE-UP?

The survey asked respondents how they first learned about and how they learned the most about SCALE-UP. Responses to these two questions are combined in Figure 4. The most common way people learned about SCALE-UP was through in-person interactions (talks/workshops and colleague) rather than through the web or literature. As shown, respondents most frequently report discussions with colleagues as their source of information about SCALE-UP.

Figure 4
figure 4

Source of information about SCALE-UP by duration of implementation.

This finding is consistent with other studies (Dancy and Henderson [2010]; Borrego et al. [2010]; Rogers [2003]) and the research literature on change that social interactions are a critical component of the reform process. It is also a very important finding as the most utilized current reform model, development and dissemination, does not significantly leverage interpersonal networks, which points to a flaw in that approach.

Communication helps adopters spread awareness about SCALE-UP and share information about implementation. Many adopters learn about SCALE-UP through ‘mass-market’ channels like talks and workshops (28%), the Internet (14%), and literature (9%). With regard to education reform, mass-market channels like these can reach larger audiences, create knowledge, and may result in change, if people were almost ready. Audiences at talks and workshops already contain many of the characteristics of innovators and early adopters, who tend to adopt a reform first. This audience is probably well educated, social, more up-to-date with current research, and more open to improving their teaching. Mass-market channels often provide an overview of basic information about an innovation: 1) awareness that an innovation exists, 2) how-to knowledge about using an innovation properly, and 3) principles - knowledge about underlying functioning principles. However, information sources tailored to larger audiences do not provide a good platform to exchange personalized how-to knowledge with faculty who want to use the SCALE-UP approach in their unique settings.

Interpersonal exchanges are also significant mechanisms for communication and 11% of respondents claimed to learn about SCALE-UP from a departmental colleague, 11% from an institutional colleague outside the department, and 6% from a disciplinary colleague outside the institution. Interpersonal exchanges better allow individuals to clarify information, overcome some of the barriers associated with selective exposure, and tend to change strongly held beliefs more effectively (Rogers [2003]). The exchange of information between colleagues at an institution, but in different departments, may have facilitated the ‘natural’ spread of SCALE-UP into other subjects.

2b. How does the communication channel affect the way adopters use the reform?

To examine how the source of information about SCALE-UP might affect implementation, we created two indices from survey responses, ‘active indicator’ and ‘studio match’, to gauge the fidelity of the pedagogical and structural use of SCALE-UP, respectively. Since we wanted to see how the instructional practices and classrooms of secondary sites related to the original SCALE-UP model, we created these indices to very roughly compare different implementations on a spectrum of fidelity to the original model. These two indices summarize core features of SCALE-UP in a course, but meaningful way and allow us to compare categories of secondary sites in order to see trends in how they use the reform.

Active indicator

SCALE-UP seeks to minimize lecture in favor of problem solving and student presentations in a redesigned classroom that facilitates interaction. In the survey, respondents selected how frequently they typically lecture, have students solving problems, and have students present to the class (never, very rarely, rarely, often, very often). These qualitative responses were converted into nominal categories ranging from 0 to 4. The active indicator index was then calculated (problem solving + student presentations)/(problem solving + student presentations + lecture). So, an extensively interactive class with no lecturing would have an active indicator of 1 and a completely lecture-based class would have an active indicator of 0.

To put these numbers in perspective, we calculated a hypothetical active indicator for Peer Instruction (Crouch and Mazur [2001]), a popular physics reform. In Peer Instruction, ConcepTest clicker questions are interspersed throughout a lecture to expose common misconceptions associated with the material. Students are given a couple of minutes to formulate their response followed by a few minutes to reach consensus in small groups before re-voting. We estimate that this method would involve faculty lecturing often, students solving problems often, and presenting their solutions very rarely. This would correspond to an active indicator of approximately 0.6. For an ideal SCALE-UP implementation, students would be solving problems very often, present to the class very often, and the instructor would lecture very rarely, which would correspond to an active indicator of 0.9.

Studio match

The ‘studio match’ criteria uses the presence of tables designed to facilitate group work as a proxy for the presence of a studio classroom, a key feature of the SCALE-UP reform. Our survey asked respondents whether they had a specialized classroom for SCALE-UP and whether they had specific classroom features. We found that people's definitions of ‘specialized classroom’ vary and may include a renovated lab for some but not others. Furthermore, traditional classrooms can have classroom response systems, whiteboards on the wall, and/or projection capabilities. But the main purpose of the SCALE-UP room is to facilitate interaction. Of all the classroom features, round or special tables seemed to be a key to facilitating group work. We found that this captured 97% of the people who claimed to have a specialized classroom so the table criterion appears to be a good proxy for the presence of an interactive classroom consistent with SCALE-UP. So, the studio match can take on binary values of 1 (special or round tables exist) or 0 (special or round tables do not exist).

Most people learned about SCALE-UP through multiple sources (61%). Of the respondents, 10% learned about it exclusively by reading the literature or researching it on the Internet, 16% learned about it exclusively by talking to colleagues, and 14% have another/unknown source of information. Of the respondents, 53% learned about SCALE-UP from a talk or workshop but all of these people supplemented this with a second source of information,

As seen in Figure 5, the implementation of SCALE-UP does not seem to depend on the source of information. The one exception is that people who learned about SCALE-UP via completely passive modes (the literature/web) are less likely to have a studio-style SCALE-UP classroom.

Figure 5
figure 5

Active-studio indicators by source of information about the SCALE-UP reform. Strictly reading the literature or Internet, strictly learning about it from colleagues, multiple information sources, or other/unknown source.

RQ3: how do secondary sites adapt the pedagogy and structure of SCALE-UP? How does this impact the success of the implementation?

3a. How do pedagogical and structural adaptations of enacted implementations vary by user status? Discipline?

The survey asked the respondents whether they considered themselves ‘users’, ‘modified user’, ‘influenced by’, ‘considering’, or a ‘past user’ of SCALE-UP-like instruction. Only 22 respondents characterized themselves as past users and a follow-up question revealed that only 2 of these were true abandoners (0.3% of total respondents), who did not want to teach using SCALE-UP again. The other past users were not currently teaching with SCALE-UP because of changes in scheduling or teaching assignments but hoped to return to SCALE-UP style instruction in the future. This provides additional justification why it would be inaccurate to use individuals as the grain size to check the pervasiveness of SCALE-UP against the DOI curve. In most departments, not all faculty members are needed to teach using SCALE-UP-style instruction at any given time. Using the department as the grain size helps to account for this.

To see how respondents' self-categorization compared to researcher definitions, Figure 6 compares active studio indicators by user status. Not surprisingly, self-described ‘users’ have higher indicators, both pedagogically and structurally compared to people who claim to ‘use modified’ and be ‘influenced by’ the reform. The results of a one-way ANOVA suggests that the differences between the four user groups is statistically significant for studio match (F(3, 473) =36.58, p = .0000) and active indicator (F(3, 426) =23.32, p < .0001). A Tukey post hoc test revealed that the studio match indicators of users (.88 ± .32) were statistically significantly higher than for modified users (.69 ± .46) and influenced users (.37 ± .48). Similarly, Tukey post hoc test revealed that the active studio indicators of users (.77 ± .12) were statistically significantly higher than for modified users (.67 ± .15) and influenced users (.59 ± .17). For both indices, there were no statistically significant differences between users and past users (p = .993, .969 for studio match and active indicator, respectively). This makes sense because, as discussed above, in this study, ‘past users’ are not true abandoners.

Figure 6
figure 6

‘Active indicator’ and ‘studio match’ by self-described user status, with 0 being completely lecture-based and 1 being extensively active.

Since SCALE-UP ‘modified users’ and ‘influenced by’ do not have significantly differences in their use of interactive pedagogies and past users are not true abandoners, thus justifying the decision to include all three in our analysis. Users have the highest structural and pedagogical scores. ‘Modified users’ and ‘influenced by’ have similar (i.e., not statistically different) active indicators but ‘influenced by’ are less likely to have the structural infrastructure to support their use of interactive pedagogies. Thus, these people try to use SCALE-UP style, interactive pedagogies in a more traditional classroom setting.

As seen in Figure 7, when active studio indicators are calculated by discipline, physics has the highest studio match, demonstrating that structurally, physics has the highest fidelity implementation. Although the results of the one-way ANOVA suggest that differences between active indicators of the six different disciplines were statistically significant (F(6, 420) =2.662 p = .015), the Tukey post hoc test did not find any statistically significant differences between any of the pairs of disciplines. Results from the one-way ANOVA with regard to the studio match indicators suggest that the differences between the six disciplines were statistically significant (F(6, 575) =4.243 p < .0001). In contrast to the Tukey post hoc results for the active indicator, respondents in physics have statistically significantly higher studio match ratings than all other disciplines.

Figure 7
figure 7

Active-studio indicators by discipline.

3b. How does structural variation (classroom type) relate to use of active learning pedagogies?

The survey investigated enacted use of SCALE-UP at the individual and departmental levels. A full SCALE-UP implementation not only requires modification of existing pedagogy but also re-designs the classroom to promote interaction between students and their instructors and students with each other. Usually, classroom facilities were common to instructors within a department. Of the surveyed departments, 42% have rooms specifically designed to facilitate SCALE-UP style instruction.

Figure 8 displays the active indicator ratings for respondents with and without a studio-style classroom. Respondents without a studio-style classroom spend a higher portion of class time lecturing whereas the majority of respondents with a special classroom spend more time with students involved in activities. This suggests that a redesigned classroom is supportive of pedagogical changes related to SCALE-UP.

Figure 8
figure 8

Distribution of active indicator ratings for respondents with and without a redesigned classroom. The vertical axis displays the percentage of implementations within the no studio classroom and studio-style classroom, respectively.

In the original SCALE-UP model, students sit in three groups of three at round tables, with computers or laptops provided for each individual, handheld whiteboards for each group, whiteboards on the walls as public thinking space, projection capabilities to share student work and a classroom polling system, like clickers (Beichner [2008]). Respondents were asked which specialized classroom equipment they had access to. Computers for each individual/group and special tables are the most popular classroom fixtures, found in approximately half of the surveyed departments. However, we felt that these features were not unique to SCALE-UP classrooms so we did not include them in our studio match index.

Figure 9 displays the presence of specific specialized classroom equipment by discipline. Not surprisingly, based on the high studio match score, physics has the highest percentage of users with round/special tables, laptops, and clickers, thus following the developer's design. Out of the STEM disciplines, math/statistics have the least amount of specialized room equipment.

Figure 9
figure 9

Percentage of departments with specialized classroom equipment, scaled by discipline. Special tables include round or otherwise special tables, laptops/computer for individuals/groups, projection capability includes cameras and screens to display student work, and clickers include any classroom response system.

Some survey respondents elaborated on unique equipment they use. At some sites, students have access to iPads and scientific calculators and some bring their own technology. For polling, some instructors use smartphone-based polling, a discussion board called Yammer, or old-fashioned voting cards. Some instructors who do not have access to a classroom try to rearrange classroom furniture to sit in groups. Other instructors do the best they can in a typical lecture hall. The variation in special classroom tools demonstrates that structurally, SCALE-UP use varies widely between departments. There is evidence that secondary implementations design their classrooms to promote interaction but not all classrooms look the same. Several of these departments mention modifying the recommended room equipment to fit their situation, sometimes because of classroom budget/space restrictions or changes in technology (for example, incorporating the use of smart boards and using smart phones for polling).

To help explain differences in classroom structure, the survey asked the faculty how their department felt about building a specialized SCALE-UP classroom. As seen in Figure 10, approximately, a third of the instructors responded that they have access to a specialized classroom. Other options for responses included: faculty never discussed building a classroom, we had a classroom in the past but not anymore, no finances to build a classroom, or other. In the ‘other category’, over a third of these respondents explained that their classroom was currently in construction while others indicated that they use a classroom in another department, they are retrofitting a current classroom, or there was no space for this type of classroom.

Figure 10
figure 10

The departmental stance on building a SCALE-UP classroom, scaled by discipline.

The lack of financial resources and the absence of discussion inhibit the building of a classroom across all disciplines. In addition to affecting the status of the classroom, we saw financial limitations impacted how instructors chose to engage their students, for example, using voting cards for polling if a classroom response system was too expensive.

Conclusions

The main findings are summarized and discussed below.

Finding one: SCALE-UP has spread widely across disciplines and institutions

SCALE-UP-style instruction has influenced teaching practice in a minimum of 314 departments at 189 higher institutions in 21 countries. Furthermore, in the US, 63% of reported departments using SCALE-UP are outside the originating discipline of physics, and 20% are outside of STEM.

Implication one: encouragingly, this wide spread indicates that SCALE-UP is making a higher than average impact on teaching in higher education

It is beyond the scope of this paper to fully answer what makes the dissemination of SCALE-UP different from similar, but less successful, reform efforts. However, we can point to several attributes of this reform that may contribute to its success.

SCALE-UP is not discipline specific. As a flexible reform without an accompanying curriculum, SCALE-UP appeals to a broad audience. Having SCALE-UP in multiple disciplines at a given institution promotes cross-disciplinary pedagogical discussions and community building.

SCALE-UP supports instructor autonomy. Faculty members want ideas that they can adapt to their unique learning environment and personal preferences. They do not what to be told what to do (Cohen and Ball [1990]). SCALE-UP provides this guidance and supports a grassroots community via a website (http://scaleup.ncsu.edu/) to share ideas and curriculum.

SCALE-UP requires structural changes and, thus, has high observability. Renovating the classroom and investing in special equipment can be an initial barrier, but, once crossed, it may also support permanent change and increased dissemination within a department. Deciding to make these radical changes encourages department-wide discussions and buy-in and makes abandoning use less likely. Since structure impacts practice (Henderson and Dancy [2007]), it may encourage broader adoption of reformed techniques.

The developer has extensively leveraged interpersonal methods to disseminate the reform and has not relied exclusively on at-a-distance mechanisms, such as papers and websites. Beichner has given over 300 SCALE-UP talks, workshops, and departmental visits around the world.

Each of these potential key attributes will be further investigated in the interview-based, second phase of this study.

Finding two: the dissemination of SCALE-UP in physics may be at the tipping point between adventurous early users and the mainstream majority

The spread of SCALE-UP appears to fit diffusion of innovation theory in terms of the characteristics of early adopters and an accelerating rate of spread. For uptake in physics to continue to increase and follow the S-curve of a successful reform, the marketing message may need to change to appeal to more hesitant potential adopters.

Implication two: diffusion of innovation theory suggests the dissemination strategy of SCALE-UP should change

A rough estimate reveals that physics may be near a tipping point. To maintain S-shaped growth, it needs to cross the chasm from appealing to innovative early adopters to reaching the more practical, hesitant general population. Moore ([2002]) states that early adopters are more likely to embrace a radical discontinuity from traditional teaching, to champion the cause against local resistance, and to persevere through inevitable glitches. As many early users in this study were connected with the physics education research community, they are probably more educated about innovative pedagogies and willing to experiment with radical reforms in hopes of drastically improving learning gains. In contrast, the early majority wants to improve existing operations while minimizing disruption to their old ways. They tend to rely more on social networks to learn about and support their adoption, often demand evidence of effectiveness before deciding to adopt and are less tolerant of implementation difficulties (Moore [2002]).

To appeal to more hesitant faculty members, the dissemination of SCALE-UP may need to involve more change agents and opinion leaders to cross the chasm to the more hesitant majority. Specifically, implementers may need more guidance from near-peers and opinion leaders to reduce implementation challenges. Leveraging social networks could help facilitate a flow of information about SCALE-UP and provide an encouraging and supportive community. Enlisting the aid of change agents that use SCALE-UP in diverse educational settings may demonstrate to hesitant adopters that they can use elements of the radical reform without overthrowing the status quo. However, since much of the success of SCALE-UP comes from the radical nature of the reform, resources should be developed and shared to ensure that instructors can productively adapt SCALE-UP to their local setting.

Finding three: interpersonal interactions promote successful dissemination

While many people learn about SCALE-UP through typical dissemination via mass-market channels (talks/workshops and the Internet), just as many report learning through interpersonal interactions with colleagues. This is consistent with diffusion of innovation theory and the authors' prior research (Dancy and Henderson [2010]; Borrego et al. [2010]; Rogers [2003]).

Implication three: leveraging interpersonal networks can improve dissemination

The common development and dissemination model of reform does not account for how interpersonal interactions share and support reforms, potentially reducing the impact of existing reforms. A better dissemination model should recognize and utilize interactions as a way to encourage change, especially when trying to reach the mainstream adopters who may not seek out information about a reform. However, this alternate model should cautiously avoid an overreliance on informal dissemination. An overdependence on word-of-mouth may mean less familiarity with the underlying research outlined in primary sources. If potential adapters only see/hear about modified versions of SCALE-UP, the likelihood of (potentially detrimental) modifications will increase. Strategically, expanding the number of multi-disciplinary change agents familiar with the underlying research and could help control the quality of information spread through interpersonal networks.

Finding four: a studio classroom is an important feature of SCALE-UP

Our analysis indicates that the presence of a studio classroom is associated with higher levels of active learning. We also found sites without a classroom that attribute this to the lack of faculty discussion or financial resources, rarely because instructors were not interested.

Implication four: it is important to better understand the role of a specialized classroom in the spread of SCALE-UP

Our findings related to the studio classroom are associative, not causal. Learning more at the nature of these relationships may provide valuable guidance for those interested in promoting reform. The presence of a studio classroom is not enough to improve learning gains, if the pedagogy does not match the goal of the renovated room design (Lasry et al. [2014]). We found that the presence of a studio classroom is associated with a higher level of active pedagogies and plan to investigate this further.

Renovating a classroom typically requires a financial investment and administrative support so usually it gets departments talking about SCALE-UP. Our initial results indicating a low number of true abandoners strongly suggest that investing in a special classroom reduces the tendency to revert back to traditional instruction. As we continue our study with interviews, we plan to further investigate how building a classroom affects the number of adopters within a department. For example, how does the process of building a special classroom impact faculty who are on the fringes of the reform effort? Do research-based reforms that require departmental discussions (for example, to build a specialized classroom) increase uptake? Our data suggest that building a special space may reduce the chances of reverting back to traditional instruction. Interviews will continue to probe this initial finding, and future studies should investigate how to apply this to spreading other research-based reforms.

Finding five: reforms are modified

The wide variation of use of active learning pedagogy and studio classroom features demonstrates that instructors differ in their implementation of SCALE-UP. This supports previous research that instructors and departments modify SCALE-UP, both pedagogically and structurally (Henderson and Dancy [2008]; Penberthy and Millar [2002]; Sharp and McLaughlin [1997]; Silverthorn et al. [2006]; Hutchinson and Huberman [1994]). Respondents reported wide variation in their classroom equipment and active learning levels during their SCALE-UP implementation efforts.

Implication five: secondary sites will benefit from having research-based recommendations on how to adapt their implementation successfully

Faculty rarely implement an innovation ‘as is’, usually adapting ideas to their unique environment, goals, personality, and more. Developers should acknowledge this and focus on helping faculty to navigate the difference between productive and unproductive changes, by using change agents or coming up with written recommendations. They should provide advice on how to overcome structural barriers (i.e., budget limitations that prohibit the ideal classroom design) and other challenges, especially with the less independent, more impatient early majority adopters.

Summary

The results of this study indicate that the impact of SCALE-UP is growing and provides insights to guide those interested in educational transformation. Currently, we are conducting interviews to develop a more detailed understanding of how instructors and departments learn about, implement, and spread SCALE-UP. Since instructors tend to make modifications, finding out more about this process can help in the development of resources to support instructor adaptation of the reform to their local circumstances in productive and successful ways.

Additional file