1 Introduction

I last wrote about systematic review more than a decade ago when, having been immersed in conducting three systematic reviews for the Teacher Training Agency in England, I felt the need to reflect on the process. Writing a reflexive narrative (Nind 2006) was a mechanism for me to think through the value of getting involved in systematic review in education when there were huge questions being asked of the relevance of evidence-based practice (EBP) for education (e.g. Hammersley 2004; Pring 2004). Additionally, critics of systematic review from education were making important contributions to the debate about the method itself, with Hammersley (2001) questioning its positivist assumptions and MacLure (2005) focusing on what she proposed was the inherent reduction of complexity to simplicity involved, the degrading of reading and interpreting into something quite different to disinter “tiny dead bodies of knowledge” (p. 394). I concluded then that while the privileging of certain kinds of studies within systematic review could be problematic, systematic reviews themselves produced certain kinds of knowledge which had value. My view was that the things systematic reviews were accused of—over-simplicity, failing to look openly or deeply—were not inevitable. My defence of the method lay not just in my experience of using it, but in the way in which I was taught about it—and how to conduct it—and led to a longer term interest in the teaching of research methods.

At the time of writing this chapter I have just concluded a study of the Pedagogy of Methodological Learning for the National Centre for Research Methods in the UK (see http://pedagogy.ncrm.ac.uk/). This explored in some depth how research methods are taught and learned and teased out the pedagogical content knowledge (Shulman 1987) held by methods teachers and their often implicit craft knowledge (Brown and MacIntyre 1993). In the study we sought to engage teachers and learners as stakeholders in the process of building capability and capacity in the co-construction of understandings of what is important in teaching and learning advanced social science research methods (Nind and Lewthwaite 2018a). This included among others, teachers and learners from the discipline of education and teachers and learners of the method of systematic review.

This chapter about teaching systematic review combines and builds on insights from these two sets of research experiences. To clarify, any guidance included here is not the product of systematic review but of deep engagement with systematic review and with the teaching of social research methods including systematic review. To conduct a systematic review on this topic in order to transparently assemble, critically appraise and synthesise the available studies would necessitate there being a body of research in the area to systematically trawl through, which there is not. This is partly because, as colleagues and I have argued elsewhere (Kilburn et al. 2014; Lewthwaite and Nind 2016), the pedagogic culture around research methods is under-developed, and partly because EBP is not as dominant in education as it is in medicine and health professions. If we are teaching systematic review to education researchers we do not have the option of identifying best evidence to bring to bear on the specific challenge. However, the pedagogy of research methods is a nascent field; interest in it is gathering momentum, stimulated in part by reviews of the literature that I discuss next, identification of the need for pedagogic research to inform capacity building strategy (Nind et al. 2015) and new research purposefully designed to develop the pedagogic culture (Lewthwaite and Nind 2016; Nind and Lewthwaite 2018a, 2018b).

2 Contribution of Systematic Reviews and Other Literature Reviews

Wagner et al. (2011) took a broad look at the topics covered in the literature on teaching social science research methods, reviewing 195 journal articles from the decade 1997–2007. These were identified through:

a database search of the Social Sciences Citation Index, ScienceDirect, Academic Search Premier, EBSCOhost, PsycINFO, Swetswise and Google Scholar. The keywords research, teaching, training, methodology, methods, pedagogy, social sciences, higher education and curriculum were used in various combinations to search the databases … [plus] examining the reference lists of the accumulated material for additional sources, until a point of saturation had been reached (Wagner et al. 2011, p. 76).

No papers on teaching systematic review were identified. Their review “proceeded according to Thody’s (2006) five steps: recording, summarising, integrating, analysing and criticising sources” (Wagner et al. 2011, p. 78). From this they concluded that when it comes to teaching research methods there has been little debate in the literature, little cross-citation and limited empirical research.

Cooper et al. (2012) conducted a meta-study with a related focus, looking at thirty years of primary research on the experiences of students learning qualitative research methods. Their concerns were with learning from the past, not just about the students’ experience but about the research methods used to study them. Hence, their meta-study included:

a meta-method analysis of the methodologies and procedures used in the previous published primary research sources; a meta-theory analysis of the theoretical frameworks and conceptualization utilized in the previous published primary research sources; and a meta-synthesis of the results from the meta-data-analysis, meta-method analysis, and the meta-theory analysis to determine patterns between the results produced, the methodologies employed, and the theoretical orientations engaged (Cooper et al. 2012, p. 2).

While retaining a qualitative constructivist grounded theory approach in the analysis, the authors were influenced by the observation by Littell et al. (2008) of the increasing use of systematic review in education (and other social science) research. Their search focused on the Teaching and Learning Qualitative Research and Qualitative Research Design Resources database, ProQuest, ERIC, and Google Scholar with some hand-searching. This led them to identify 25 published articles providing the student perspective. Papers were appraised using a modification of the Primary Research Appraisal Tool (Paterson et al. 2001). They conclude “that the student experience of learning qualitative research is made up of three central dimensions—experiential, affective, and cognitive—which combine to form an experience of active learning necessary to understand and practice qualitative research” (pp. 6–7).

Next up, Earley (2014) undertook a synthesis of 89 studies (1987 to 2012) pertaining to social science research methods education (search terms and databases unspecified), asking

(1) What does the current literature tell us about teaching research methods?

(2) What does the current literature tell us about the learning of research methods?

(3) What gaps are there in the current literature related to teaching and learning research methods?

(4) What suggestions for further research can be identified through an exploration of the current literature on teaching and learning research methods? (Earley 2014, p. 243)

He followed Cooper’s (1998) five stages for conducting a research synthesis (problem formulation, literature search, assessment of the quality and applicability of the studies, analysis and interpretation, and presenting the results). Earley (2014) was able to show patterns in the research in how learners are characterised (largely unmotivated and nervous), teaching techniques covered (active learning, problem-based learning, cooperative learning, service learning, experiential learning and online learning), and teacher objectives (concerned with educating consumers or producers of research). More importantly perhaps, he identified problems that have been ongoing and that our Pedagogy of Methodological Learning study sought to address: unfulfilled need to establish what student learning of social research methods looks like and the literature being dominated by teacher reflections on their own classrooms rather than studies that cross contextual boundaries or look from the outside in.

As a bridge between previous reviews and new empirical work, my colleagues and I conducted a new literature review (Kilburn et al. 2014), purposefully constructed in terms of deep reading of the literature as opposed to systematic review. We engaged in thematic qualitative exploration of insights into how methods teachers approach their craft. We sought to identify all peer-reviewed outputs on the teaching and learning of social research methods, focusing on the endpoint for the Wagner et al. synthesis in 2007 through to 2013. We searched the ISI Web of Knowledge database and for the ‘high sensitivity’ search (Barnett-Page and Thomas 2009) used the search terms: “research methods” OR “methodology” OR “qualitative” OR “quantitative” OR “mixed methods” AND “teaching” OR “learning” OR “education” OR “training” OR “capacity building”. This led to sifting over 800 titles, moving to a potential pool of 66 papers and examination of 24 papers. As with Earley (2014), we found that most of the papers reported on teachers’ reflections on their practice and there was an emphasis on active and experiential learning. However, we also found greater “cause for optimism regarding the state of pedagogical practice and enquiry relating to social science research methods” in that “considerable attention is being paid to the ways in which teaching and learning is structured, delivered and facilitated” and “methods teachers are innovating and experimenting” in response to identified limitations in pedagogic practice and “developing conceptually or theoretically useful frames of reference” (p. 204).

The state of the research literature indicates a willingness among methods teachers to systematically reflect on their own practice, thereby making some connection with pedagogic theory, but that there is limited engagement with the practice of other methods teachers working in other disciplines or with other methods. It is noteworthy that none of the above searches turned up papers about teaching systematic review specifically. This situation may be indicative of the way in which education (and certainly not higher education (Bearman et al. 2012)) is not an evidence-based profession in the way that Hargreaves (1996) and Goldacre (2013) have argued it should be. If teachers of methods are relying on their own professional judgement (or trial-and-error as Earley (2014) argues), the knowledge of the team and feedback from their students, it may be that they do not feel the need to draw on a pool of wider evidence. They may be rejecting the “calls for more scientific research” and “reliable evidence regarding efficacy in education systems and practices” that Thomas (2012, p. 26) discusses when he argues that in education, “Our landscape of inquiry exists not at the level of these big ‘what works’ questions but at the level of personalized questions posed locally. It exists in the dynamic of teachers’ work, in everyday judgments” (p. 41). This disjuncture with systematic review principles poses real and distinctive challenges for teachers of systematic review method in education, as I shall go on to show.

Before moving on from the contribution of systematic reviews to our understanding of how to teach them we should note the systematic reviews conducted pertaining to educating medicine and health professionals about evidence-based practice. Coomarasamy and Khan (2004) synthesised 23 studies, including four randomised trials, looking at the outcome measures of knowledge, critical appraisal skills, attitudes, and behaviour in medicine students taught EBP. They concluded that standalone teaching improved knowledge but not skills, attitudes, or behaviour, whereas clinically integrated teaching improved knowledge, skills, attitudes and behaviour. This led them to recommend that the “teaching of evidence based medicine should be moved from classrooms to clinical practice to achieve improvements in substantial outcomes” (p. 1). Kyriakoulis et al. (2016) similarly used systematic review to find the best teaching strategies for teaching EBP to undergraduate health students. The studies included in their review evaluated pedagogical formats for their impact on EBP skills. They found “little robust evidence” (p. 8) to guide them, only that multiple interventions combining lectures, computer sessions, small group discussions, journal clubs, and assignments were more likely to improve knowledge, skills, and attitude than single interventions or no interventions. This and other meta-studies serve to highlight the need for new research and, I argue, more work at the open, exploratory stage to understand pedagogy in action.

3 The Pedagogy of Methodological Learning Study

The Pedagogy of Methodological Learning study was in large part my response to a policy demand for methods training to build capacity among social science researchers that was not yet recognising the contribution that pedagogic research could make, and to the limitations in the scope of the research to date. It was designed to find and share the pedagogical content knowledge of social science research methods teachers and to be conducted in a collaborative, non-judgemental spirit so that together we could better understand and develop our pedagogic practices. The study comprised a series of connected parts:

  • an international expert panel to explore—both individually and collectively—the practices and pedagogical content knowledge of methods teachers with extensive teaching experience, followed up with seven focus groups with methods teachers in the UK to further the insights and check the resonance of core themes from analysis of the experts’ data;

  • video stimulated recall, reflection and dialogue between teachers and learners of various social research methods in a series of focus groups to reflect on pedagogical decision-making and experience of research methods pedagogy in action;

  • a methods learning diary circle to access and explore together a range of learner perspectives on their methods learning journeys over an extended period;

  • in-depth case studies to add nuanced detail and test the emerging typology of pedagogic practice in situ.

The methods are discussed elsewhere, including their role in offering pedagogic leadership (Lewthwaite and Nind 2016) and in supporting pedagogic culture-building and dialogue (Nind and Lewthwaite 2018a). In this chapter I discuss the findings for the light they can shed on the teaching and learning of systematic review in the field of education. I draw in particular on video stimulated dialogue about the teaching of synthesis methods within systematic review.

The Pedagogy of Methodological Learning study has identified that the participating methods teachers have particular pedagogical content knowledge about how to teach with, through and about data, including the affordances of learner data and teacher data, and the value of authentic data, immersion in data and actively doing things with data. Teachers of qualitative methods understand that their work involves conceptually difficult material, which requires them to have deep knowledge of qualitative research and to foster reflexivity in their classrooms. They value and use authentic data and their own and learners’ standpoints in their teaching. Teachers of quantitative methods stress the teaching of technical skills, the necessary logic to make sound judgements and the role of actively practising on data. They understand that their work requires an understanding of the difficulty and sequencing of content and they use diverse strategies and tactics including chunking, bootstrapping, backfilling and scaffolding to convey knowledge, build competence and deepen learning (Nind and Lewthwaite 2018b). There is a recurrent narrative about underprepared, fearful, diverse and anxious quantitative methods students leading teachers to develop student-centred approaches that deploy visual or verbal non-technical strategies to support learning. Teachers of mixed methods understand the particularly challenging nature of supporting learners in going back and forth between deductive and inductive thinking and thinking critically as well as pragmatically.

Some participating methods experts and teachers struggled to articulate their pedagogic approach, some readily identified with a known, named pedagogic approach, and some articulated and named their own unique approach. They described using active learning, experiential learning, student-centred learning, peer/interactive/collaborative/dialogical learning, problem-based learning and independent learning approaches. The teaching of qualitative methods was associated with experiential learning approaches and the teaching of quantitative methods had a notable lack of collaborative approaches. Teachers in the study identified a range of conscious pedagogic strategies for structuring content, organizing the classroom and engaging students, often using data or drawing on their own experiences as pedagogic hooks. Within their classrooms they had tactics for supporting active learning, including generating effective exercises and creating space and scaffolds for reflection. They had tactics for being student-centred, including finding out about their students, attuning, empathizing, and connecting with students’ interests. They had tactics for connecting the techniques of research methods with real life research problems, including narrating stories and going behind the scenes of their own research work.

Through the various components of the study the participants and researchers probed together what makes teaching research methods challenging and distinctive, their responses to the challenges, and the pedagogical choices made. One of the first challenges is about getting a good fit between the methods course and the needs of the methods learner and a repeated refrain from learners and teachers was that mismatches were common. When writing this chapter I came upon this informative course description:

This course is designed for health care professionals and researchers seeking to consolidate their understanding and ability in contextualising, carrying out, and applying systematic reviews appropriately in health care settings. Core modules will introduce the students to the principles of evidence-based health care, as well as the core skills and methods needed for research design and conduct. Further modules will provide students with specific skills in conducting basic systematic reviews, meta-analysis, and more complex reviews, such as realist reviews, reviews of clinical study reports and diagnostic accuracy reviews.

We see here how embedded systematic review has become in health care and medicine as evidence-based professions. The equivalent would be unlikely in education where one could imagine something like:

This course is designed for education professionals and researchers seeking skills in contextualising, carrying out, and applying systematic reviews appropriately in education settings where there is considerable skepticism about such methods. Core modules will introduce the students to doing systematic review when the idea of evidence-based education is hugely controversial. …

I am being facetious here only in part, as this is an aspect of the challenge facing teachers of systematic review in education. Fortunately perhaps, advanced courses in systematic review are often multi-disciplinary and attitudes to systematic review are likely to be diverse. Diversity in the preparedness and background of research methods learners was a frequently discussed challenge among teachers in the study, but learners invariably welcomed diverse peers from whom they could learn.

In the study’s video stimulated dialogue about teaching and learning systematic review, in a focus group immediately following a short course on synthesis hosted in an education department, teachers immediately responded to an opening question about the challenges of teaching this material by focusing on the need to understand the diverse group. They expressed the need to find out about the background knowledge of course participants so as to avoid making errant assumptions and to follow brief introductions with ongoing questioning and monitoring of knowledge and of emotional states. As one participating teacher explained,

you really need to understand research in order to get what’s going on … we don’t want to have to assume too much, but on the other hand if you go right back to explaining basic research methods, then you don’t have time to get onto the synthesis bit, that which most people come for. So sometimes it’s a challenge knowing exactly where, how much sort of background to cover.

Participating students were equally aware of the challenge, one commenting on the usefulness of having an “overview of everything, because obviously everyone has come from slightly different arenas” and acknowledging

We didn’t do super-technical things, but I think that’s important because otherwise you get people that don’t understand and then you lose half the group, so it’s important that the tasks are feasible for everybody, but that they give you the technique so you can go home and do it yourself.

In this course, the disciplinary backgrounds of the students varied somewhat. The teachers managed this, in the way of many of the teachers in the study, by working out—and working with—the varied standpoints in the room. One of the teachers celebrated the pedagogical potential of having “people from different perspectives and different disciplines talking to one another”. This was the view of the students too, arguing that “the diversity, speaking to all the different people is, I think, is key in methods, and teaching in particular, because we’re all doing similar things, just in different topics”. The reasoning was clear too with the reflection that “if you’ve only got people who have exactly the same positionality, then how do you ever critique your own work and … reflect back and think why are we doing this”.

The focus group included a lively discussion about a point in the course when one student, as she put it, “disagreed very strongly with what was being said”, explaining that this “was because of disciplinary differences, because I don’t have a disciplinary allegiance to that sort of health promotion initiative”. The different disciplinary backgrounds supported debate about how synthesised data get reduced with students recognising that the “friction and tension … makes it so much more interesting to kind of discuss”.

This should help teachers of systematic review not to fear diversity among students; a standpoint, peer collaborative learning approach can be used to address different attitudes (Nind and Lewthwaite 2018b) and an active learning approach can address the differences in knowledge. The systematic review teachers in the Pedagogy of Methodological Learning study spoke of their tried and tested “slides and then practice, slides and practice”, using exercises developed and honed over time. Again the students liked the mix of input with opportunities to practice; “the quantitative stuff came really easily … And if it was applied, then I was really engaged … I could try those [calculations] myself and make sense for myself”. This student continued,

The qualitative exercises in particular I really liked, but I wouldn’t have naturally been drawn to them, but I found they were really interesting and found some strength I didn’t know I had in doing them, whereas I would have just crunched numbers instead, happily, you know without ever trying to break it into themes.

The focus group discussion turned from the welcome role of the exercises following the underpinning theoretical concepts to the welcome role of discussion between themselves in that “people came with quite a lot of resources in terms of their knowledge and experience and skills”. They concurred that they would have liked more time discussing, “to really work out what [quality criterion] was”. While the teacher spoke of concerns about the risks of leaving chunks of time in the hands of the students, the students reassured, “by that point we kind of knew each other well enough that it was really helpful doing this group work”. They noted that “it’s so much nicer talking in peer groups rather than just asking direct questions all the time, because … [for] little bits that you need clarification on, it’s easy to do with the person sitting next to you”. The complexity of the material and the need for active engagement was recognised by the students:

S3::

I think that was quite a hard session to teach.

S2::

Yeah.

S3::

Because what you wanted to do was to bring out different approaches to judging quality, they’re actually similar in many ways, so I think it could have, perhaps it could have been a bit more us sort of just experiencing these quality issues

…

T3::

my main objective wasn’t actually to do with the exercises. It was just to get you to realise how hard it was, and then realise that that’s right, it’s hard, that’s fine, now work together to make it manageable, and that can be done as well, you can begin to start to do that, and that’s really all I wanted out of the session

We are also able to learn from the video stimulated dialogue of this group about the way that pedagogic hooks work to connect students to the learning being targeted. We prompted discussion about a point in the day when everyone was laughing. Reviewing the video excerpt of that moment we were able to see how the methodological learning was being pinned to the substantive finding regarding a point about the sensitivity of the tool and the impact on the message that came from the synthesis. The group were enraptured by the finding that ‘two bites of the apple’ made a difference, which led them into appreciating how some findings made “a really good soundbite that you could disseminate … in a press release”. They appreciated “the point of doing good, methodologically sound studies is so we don’t have a soundbite like that based on crappy evidence … that’s why systematic reviews are so good”. This was important learning and the data provided the pedagogic hook.

Another successful strategy was to use the pedagogic hook of going behind the scenes of the teachers’ own research (echoed throughout our study). The teachers talked of liking to teach using their own systematic reviews as examples:

It’s a lot easier. I think because you know whatever it is backwards, … I mean that review, the two bites of an apple was done in 2003, so I don’t feel all that familiar with the studies anymore, but if you know something as well as that, it’s much easier to talk about it

Reflexivity played an important role in this practice too with another teacher in the team reflecting, “I found it easier to be critical about my own work, partly because I know it so well and partly because then I’m also freed up”. He spoke of becoming increasingly interested in the limitations of the work, “not in a self-defeating kind of way, but more just I find them genuinely interesting and challenging … what are we going to do? These limitations are there, how do we proceed from here?”. In teaching, he said, “I’m able to say more and be more genuinely reflexive, reflective about the work, just because I did it.” The students respected the value they gained from this, one likening it to “going to a really good GP” with knowledge of a broad range of problems. While the teachers valued their own experiences as a teaching resource because “we know the difficulties that we had doing them and we know the mistakes that we have made doing them”, the students valued the accompanying depth and credibility, “the answers that you could give to questions having done it, are much more complete and believed”. Systematic review as a method has been criticised for being overly formulaic, but this was not my experience in learning from reflective practitioners of the method and these teachers stressed this too, “it’s not a nice neat clean process, [whereby you] turn the wheel on a machine and out comes the review at the end, and it can look like that if you read some of the textbooks”. Even the students stressed, “you know the flowcharts and stuff, but actually there’s a lot more to consider”.

4 Conclusion

I first reflected on the politics of doing systematic review when, as Lather (2006) summarised, the “contemporary scene [was] of a resurgent positivism and governmental incursion into the space of research methods” (p. 35). This could equally be said of today and this makes it especially important that when we are teaching the method of systematic review we do some from a position in which teachers and students understand and discuss the standpoint from which it has developed and from which they choose to operate. Like Lather (2006), and many of the teachers in the Pedagogy of Methodological Learning study, I advocate teaching systematic review, like research methods more widely, “in such a way that students develop an ability to locate themselves in the tensions that characterize fields of knowledge” (Lather 2006, p. 47). Moreover, when teaching systematic review there are lessons that we can draw from pedagogic research and from other practitioners and students who provide windows into their insights. These enable us to follow the advice of Biesta (2007) and to reflect on research findings to consider “what has been possible” (p. 16) and to use them to make our “problem solving more intelligent” (pp. 20–21). I have found particular value in bringing people together in pedagogic dialogue, where they co-produce clarity about their previously somewhat tacit know-how (Ryle 1949), generating a synthesis of another kind to that generated in systematic review. However we elicit it, teachers have craft knowledge that others can, with careful professional judgement, draw upon and apply. Those of us teaching systematic review benefit from this kind of practical reflection and from appreciating the resources that students offer us and each other. Teaching systematic review, as with teaching many social research methods, requires deep knowledge of the method and a willingness to be reflexive and open about its messy realities; to tell of errors that researchers have made and judgements they have formed. It is when we scrutinize pedagogic and methodological decision-making, and teach systematic review so as to avoid a rigid, unquestioning mentality, that we can feel comfortable with the kind of educational researchers we are trying to foster.