National efforts are underway to transform undergraduate science, technology, engineering, and mathematics (STEM) education by encouraging STEM faculty members to use instructional strategies that improve the learning and success of all students (Freeman et al., 2014). One aspect of these efforts is discipline-based education research (DBER), which has emerged within the STEM disciplines to investigate and improve undergraduate learning and development and inform teaching reform efforts. As defined by the National Research Council (Singer, Nielsen, & Schweingruber, 2012), STEM DBER (referred to as “DBER” from here forward for succinctness) combines “expert knowledge of a science or engineering discipline, of the challenges of learning and teaching in that discipline, and of the science of learning and teaching generally” (p. 2) to address “discipline-specific problems and challenges” (p. 202). For example, the field of biology presents unique challenges for learning because of the extensive use of acronyms, use of multiple terms to describe a single phenomenon or physical entity (e.g., multiple names for the same protein), and evolution of the meaning of terms over time (e.g., changing definitions of “gene” or “species”) (Tibell & Rundgren, 2010). Understanding biology also requires reasoning across orders of magnitude (atomic to ecosystem) and ontological levels (e.g., DNA is information, a unit of inheritance, and a physical entity) (Tsui & Treagust, 2003). Although topics of DBER are disciplinary in nature and therefore familiar to STEM colleagues, the research questions and methods of DBER are often grounded in the social sciences. As such, DBER is truly an interdisciplinary field of study (Fig. 1).

Fig. 1
figure 1

DBER is an interdisciplinary field situated at the nexus of science, social science, and education (adapted from Lukes, LaDue, Cheek, Ryker, & St. John, 2015)

Full size image

Education evaluation, scholarly teaching, and the Scholarship of Teaching and Learning (SoTL) also aim to improve teaching and learning using educational data, but are distinct from DBER. Education evaluation aims to determine the merit, worth, value, or impact of a program or intervention (Scriven, 2003) with the goal of informing action rather than contributing to understanding teaching and learning, as is the case for DBER. Scholarly teaching involves teaching in ways that are consistent with research on learning, such as collecting assessment data from students to inform instructional decision making (Angelo & Cross, 1993), but with no intention of sharing the data beyond the classroom. SoTL extends scholarly teaching beyond the private environment of the classroom to the public domain through sharing and peer critique, bringing a level of systematicity and professionalism to improving instruction (Shulman, 2000). SoTL studies are typically descriptive and focus on innovations that addresses learning goals. Data collection and analyses are generally limited to one’s classroom or program with the aim of making local improvements; when published following peer review, SoTL can also serve as a tried-and-tested curriculum or instruction for other instructors to adapt for use with their own students. In contrast, DBER pursues research questions and hypotheses about teaching, learning, and ways of thinking in a discipline that extend beyond single classrooms and programs in order to yield original, generalizable, and mechanistic insights into educational processes and their effects.Footnote 1

DBER Positions: Development and Growth

There are multiple entry paths to DBER faculty positions. Some DBER faculty members have doctoral degrees in traditional areas of STEM and have either completed postdoctoral training in education research or developed education research programs through other mechanisms (e.g., collaboration with social scientists, self-teaching by reading and professional development). Other DBER faculty members have doctoral degrees in DBER or in educational research from a college of education and have gained STEM disciplinary understanding through their undergraduate STEM degrees or through collaborations, professional development, and reading. Important to any path is gaining first-hand insight into both STEM theory and practice and social science (e.g., education, cognitive science) theory and methodology.

Because DBER faculty members are recruited for and appointed in disciplinary departments, they are also uniquely positioned to help their STEM departmental colleagues apply DBER results in their teaching to improve student learning and success. They can bring a perspective to teaching and learning and to the study of education that reflects STEM priorities, worldviews, understanding, and practices. This “insider” status allows DBER faculty members to study many issues related to postsecondary STEM education and STEM faculty professional development, which are typically not a main focus of faculty members in Colleges of Education but are essential to improving teaching and learning in STEM fields.

The growth and productivity of DBER have prompted an increasing number of institutions to establish tenure-track DBER positions in STEM departments. In 2014–2015 alone, there were more than 25 active searches for tenure-track positions in biology education research. Training programs in DBER at the undergraduate and graduate level are also growing. The National Science Foundation (NSF) has funded DBER Research Experiences for Undergraduates sites and included DBER as an area of scholarship for their prestigious graduate fellowships. Although a handful of DBER doctoral programs have been in place for decades (e.g., the Chemistry Education Research program at Purdue University has existed for 31 years), many more are emerging. For example, seven of the 15 geoscience education research graduate programs in the U.S. were established since 2005 (Libarkin, 2015). The work of DBER faculty members has resulted in a huge growth of understanding in undergraduate education that can inform teaching and learning in STEM units and beyond (Singer et al., 2012). For example, Freeman et al. (2014) conducted a widely recognized and influential meta-analysis of 150 articles that demonstrated the effects of active learning for undergraduate STEM students; 90 of the 150 articles (60%) were published in DBER journals.

The Challenge of Evaluating DBER Scholarship

Faculty members hired into DBER positions are expected to contribute to understanding STEM education by establishing productive research programs, including publishing and garnering extramural funding. However, because these positions are in units where the new faculty member may be the only DBER scholar, the research programs must be evaluated by non-DBER STEM colleagues. Although senior STEM faculty members are accustomed to evaluating the teaching and service accomplishments of their junior colleagues, they may be less familiar with evaluating scholarly work that makes use of social science theory and methods to address questions about STEM teaching and learning. Traditional metrics for evaluating the impact of STEM research, such as numbers of citations, may miss important influences of DBER. This alone does not make DBER distinctive. In fact, DBER is similar to other interdisciplinary or applied research fields, the impacts of which extend beyond standard metrics such as citations. For instance, clinical biomedical research has been evaluated for its impact on patient care (e.g., Cox et al., 2009); and agricultural and extension research has been evaluated for its usefulness to the public (Weiser & Houglum, 1998). Scholars in these fields have academic homes with a tradition of evaluating “nontraditional” impact; that same tradition now needs to expand to also include the impact of DBER. This essay aims to address this issue by providing guidance on evaluating the scholarly accomplishments of DBER faculty members in a way that is useful to their departmental colleagues and administrators during the tenure and promotion evaluation process.

Recommendations for Evaluating DBER Scholarship

Nature of Specific DBER Positions

First and foremost, all evaluations must be conducted with the expectations of the DBER faculty member’s position in mind, the conditions of which should have been articulated and agreed upon at the time of hire. The faculty member, the administrators, and the members of the promotion and tenure committee all need to be working from the same set of expectations for what constitutes success in that position. For example, at the University of Georgia, DBER faculty members have been hired into positions that varied significantly in their research and teaching expectations. Some DBER faculty members have primarily teaching positions, with responsibility for 0.625 instruction and 0.125 research of a standard nine-month or 0.75 position. Others have predominantly research positions, with responsibilities similar to most STEM research faculty members, that is, 0.50 research and 0.25 instruction. The teaching loads and research expectations of these positions differ accordingly. For example, a peer-reviewed contribution to a widely-used and respected curriculum database such as the National Center for Case Study Teaching in ScienceFootnote 2 would be a valued scholarly contribution for a primarily teaching position, while peer-reviewed publications in respected journals would be expected for a primarily research position.

Regardless of the distribution of responsibilities, DBER scholarship is evaluated based on evidence of knowledge creation and impact. In the University of Georgia’s Department of Plant Biology, a candidate being considered for promotion from assistant professor to associate professor with tenure must have published a body of work sufficient to provide evidence of an emerging national reputation for excellence in teaching and creative scholarship in the pedagogy of the discipline.Footnote 3 For promotion from associate professor to full professor, the candidate must have published a body of work sufficient to establish a national reputation for excellence in teaching and creative scholarship in the pedagogy of the discipline. Publications generally are expected to appear in appropriate peer-reviewed journals that have earned high national and international status. United States Patents count as publications, and information deposited in national databases or distributed on the web may count as a publication if these are equivalent in impact to a standard peer-reviewed publication.

Examining where DBER Scholarship is Published

Although DBER is interdisciplinary, its primary impact is not. Rather, DBER aims to be useful, valuable, and influential to other DBER scholars and to the faculty members who are teaching in the discipline. Thus, one indicator of impact is the nature of the journals where DBER faculty members publish their work. This can be determined by examining the mission and readership of the journal and evaluating how specific papers contribute to the journal’s mission and the work of its readership. DBER faculty members may publish education pieces in STEM research journals; descriptions of instructional strategies in STEM teaching journals; and scholarly reviews and original research in journals of education, social science, or DBER. Each type of journal publishes articles that fit with their respective missions. For example, a journal such as Science magazine publishes occasional education articles of unusually broad relevance. Practice-oriented journals such as the Journal of College Science Teaching and The Physics Teacher publish reports and discussions of innovative teaching materials and methods for other instructors to adopt or adapt. DBER scholars may publish in education research journals, such as the Journal of Research in Science Teaching, American Educational Research Journal, Journal of Educational Psychology, and Instructional Science to reach readers who are education researchers, education psychologists, and cognitive scientists.

DBER journals, such as the ones we, the authors, represent, reach DBER faculty members and their STEM colleagues. These journals publish research and evaluation studies of discipline-specific teaching and learning, such as characterizations of how students think about the transformation of matter and the extent to which their thinking aligns with experts’ ideas. This work can be translated into classroom interventions and studied for effectiveness, such as whether an instructional strategy helps students develop more expert conceptions of how matter is transformed. Most DBER journals aim to be understandable and applicable to both education researchers and the thousands of scientists, engineers, and mathematicians who teach. Some DBER fields, such as chemistry education research, have published community rankings of top tier journals that can be used as an indicator of journal quality (e.g., Towns & Kraft, 2012). In other DBER fields, such as biology education research, it may be more important to publish in a society journal (e.g., the American Physiological Society’s Advances in Physiology Education) because it may be more widely read and thus influential among colleagues teaching in a specific sub-discipline.

Examining the use of DBER Scholarship

The ultimate goal of DBER is the improvement of student learning. The extent to which this occurred goes beyond counting citations (Feig, 2013; Singer et al., 2012). Therefore, using multiple methods of evaluating the impact of scholarship becomes important. Two additional metrics that can be useful are article views and downloads. For example, Hoskins and colleagues (Hoskins, Lopatto, & Stevens, 2011) published a study of a method for teaching undergraduates to read and evaluate primary scientific literature. The article has been cited only 61 times (Google Scholar), but the full-text html has been accessed ~2000 times; and the article PDF has been downloaded >2400 times at the time this piece was written. The most-cited article in CBE - Life Sciences Education (LSE), the DBER journal published by the American Society for Cell Biology, has been cited only 271 times (Google Scholar), but has had >23,500 html visits and over 15,000 PDF downloads (Crowe, Dirks, & Wenderoth, 2008). An article outlining new microbiology curriculum guidelines (Merkel, 2012) has been cited only 18 times (Google Scholar), but has been implemented by more than 150 faculty members (Horak, Merkel, & Chang, 2015), who in turn have the potential to influence hundreds of students per year. These guidelines have also influenced microbiology textbook reform and initiatives of the American Society for Microbiology (ASM) related to student assessment.

Citations of DBER scholarship can be limited not only by the practical nature of the research, but also by characteristics of DBER fields, which are small with a fairly limited number of journals (Harzing, 2012; Singer et al., 2012). For example, there is a single society-sponsored geoscience education research journal, the Journal of Geoscience Education, which limits opportunities for external citations within the field. The norms of citing prior work also differ. For example, Cultural Studies of Science Education strongly discourages the citing of more than one or two papers in support of a point (Tobin, 2008, 2009).

Evaluating Contributions to the Field

Regardless of the DBER field, there should be evidence of innovative and novel contributions. In this way, evaluating DBER is similar to evaluating scholarship in any sub-discipline; it must rely heavily upon input from experts in the subfield or at least upon individuals who have sufficient familiarity with and expertise in the discipline to determine what is novel. At institutions where there are multiple DBER scholars, it is relatively straightforward to find local experts. For example, Middle Tennessee State University (MTSU) evaluates the scholarship of DBER faculty members by including a representative from each of the focal areas in the department on the promotion and tenure committee. If a DBER candidate is the first to be considered for promotion and tenure in an MTSU department, a faculty member knowledgeable about DBER scholarship is asked to serve in this role even though they are not producing DBER scholarship themselves. The University of Arizona addresses the isolation of DBER faculty members in their College of Science by soliciting evaluations not only from departmental and college colleagues, but also from a separate Science Education Promotion and Tenure CommitteeFootnote 4 comprised of science education researchers across campus. Individuals on this committee bring expertise in education and social science research to the evaluation of DBER contributions that may not be available within particular science departments.

At institutions where neither of these approaches is feasible, external reviews from DBER scholars become even more critical. DBER experts can be identified in a variety of ways, such as through their service on DBER journal editorial boards, their authorship of high profile DBER publications, their role in DBER professional societies, and their roles as principal investigators of DBER grants or regional or national education programs. These outside experts can help disciplinary colleagues understand the nature of the journals where DBER candidates are publishing as well as the specific impact of the candidates’ scholarship on research design and methods, elucidating student understanding, educational practice, programming, and policy. Outside experts can also help disciplinary colleagues evaluate various forms of scholarship. For example, conferences where DBER faculty members present their work may employ a rigorous peer review process in selecting oral and poster presentations, such as the annual meeting of the National Association of Research in Science Teaching, which requires the submission of a five-page presentation that is reviewed through a double-blind process resulting in ~50% acceptance rate. Some conferences publish peer-reviewed proceedings (e.g., Physics Education Research Conference, Annual Conference on Research in Undergraduate Mathematics Education). At the ASM Conference for Undergraduate Educators, peer-reviewed poster presentations (60% average acceptance rate for 2014–2015) require extensive data on student learning, while peer-reviewed oral presentations have less stringent requirements (referred to as “microbrews;” 85% average acceptance rate in 2014–2015). This is distinctly different from many STEM meetings where most if not all poster presentations are accepted and comparatively fewer oral presentations are accepted. As context for their evaluations, both internal and external evaluators should identify their areas of expertise and make explicit in writing their expertise in commenting on specific aspects of DBER scholarship.

Examining Usefulness of DBER Scholarship

Whether effecting change at a local, regional, or national level, it is important to evaluate the usefulness of DBER for improving STEM education. When relevant, feedback should be solicited from instructors who have made use of a DBER faculty member’s research in their teaching, with the aim of describing how the research was translated into practice, its influence on faculty members, and its impact on student learning. When warranted, dossiers should include narrative descriptions of any influences of DBER publications on programs and policy accompanied by evidence. For example, DBER has led to the creation of effective professional development for those who mentor scientists-in-training (e.g., Handelsman, 2005; Pfund, Pribbenow, Branchaw, Lauffer, & Handelsman, 2006), tools for measuring student learning (e.g., Nehm, Beggrow, Opfer, & Ha, 2012), and curriculum and textbooks designed to fit how students learn (e.g., Reynolds, Johnson, Morin, & Carter, 2013). DBER has also been featured in reports from the National Academies of Sciences, Engineering, and Medicine; NSF program announcements; and policy recommendations and calls for action from other nationally influential groups (e.g., White House Office of Science and Technology Policy, education committees of scientific societies).

Conclusion

DBER faculty members are responsible for communicating and demonstrating the excellence, relevance, and impact of their research; but it is the responsibility of university administrators and the senior STEM faculty members to understand the landscape in which DBER is conducted and its potential for transformative change in disciplinary teaching and learning. As DBER positions grow in number and current DBER faculty members mature into leadership positions, navigating the road to tenure and promotion within DBER will become easier. We hope that this essay fills a current gap in this process.