Understanding the validity of data: a knowledge-based network underlying research expertise in scientific disciplines
- 2.1k Downloads
This article considers what might be taught to meet a widely held curriculum aim of students being able to understand research in a discipline. Expertise, which may appear as a ‘chain of practice’, is widely held to be underpinned by networks of understanding. Scientific research expertise is considered from this perspective. Within scientific disciplines, how research is conducted to solve different problems varies with concomitant effects on the validity of the data and the strengths of the claims made. Despite this variation, the overarching concepts of validity and reliability and the underpinning and interrelated network of more specific ‘concepts of evidence’ are applicable to understanding a wide range of research designs and the uncertainties in the resultant data and claims. These constituent ideas, which inter alia have been validated as relevant to professional and academic expertise, form an integrated knowledge base about evidence which can be visualised on a concept map. The network of ideas underpinning research expertise across scientific disciplines is outlined. Research from explicitly teaching the concepts of evidence is reported, and the implications for teaching and learning in science-based disciplines in HE are considered.
KeywordsConcept mapping Threshold concepts Expertise Concepts of evidence Research–teaching links Uncertainty
Curriculum development in HE, despite variations across institutions, is an area where academics still maintain their professional autonomy (Louvel 2013), deciding both what to teach and how, so as to meet their aims. This article considers what might be taught to meet a widely held curriculum aim of students being able to understand research in a discipline. While the focus is on an understanding of research in science, the ideas may be pertinent to other disciplines, such as in applied and social sciences, where research gives weight to data as evidence.
Students’ understanding of research is incorporated in notions of ‘graduateness’ (Steur et al. 2012) and generic graduate attributes (Barrie 2006) as well as being of importance in critical thinking (Moore 2011) and for postgraduate research (Cumming 2010). The varied research–teaching links (Willison 2013) widely established in HE in many countries and extensively reviewed by Malcolm (2014) also attest to the widespread engagement of undergraduates with research which requires that they have an understanding of evidence within their discipline. Understanding research is a key feature of the characteristic ways of thinking and practicing in a discipline (WTPs; McCune and Hounsell 2005) and of disciplinary expertise, both of which are now central notions in curriculum development in HE (Baillie et al. 2013; Entwistle 2009; Kinchin and Cabot 2010; Meyer and Land 2003, 2005) and can, in Hay’s (2011) view, be described as ‘researcher-led’ teaching.
Kinchin and Cabot (2010:153) argue that ‘The development of university pedagogy needs to consider the nature of expertise and how this can be modelled for students in such a way that teaching reflects the values of each academic discipline’. This article contributes to this aim with regard to research expertise in scientific disciplines.
If expertise is to inform curriculum development, then understanding expertise is important. Expertise has been extensively researched (Ericsson et al. 2006) while Amirault and Branson (2006) have identified particular aspects of importance to education. Only a few points can be addressed here.
Expertise appears intuitive (Kinchin and Cabot 2010), develops through extensive experience (Shopkow 2010) and is considered to be hard to articulate since it draws on tacit knowledge (Polanyi 1966; Sternberg 1999). These characteristics would seemingly challenge attempts to teach students explicitly to develop expertise. Does this mean that experts’ ‘route to expertise is blocked for the student’ (Shopkow 2010:320)? Several authors (for instance, Bradley et al. 2006; Kinchin et al. 2008) suggest that tacit knowledge can be elicited.
WTP research, with its focus on disciplinary expertise and ‘thinking like an expert’, is closely associated with research into threshold concepts (TCs; Meyer and Land 2003, 2005)—concepts associated with expertise, which are integrative and transformative and may be troublesome to learn. TC research has been influential (see, for instance, Flanagan n.d.), focusing disciplinary academics’ recent attention on the concepts required for expertise and how they might be made explicit to students.
Expertise can take many forms. Bradley et al. (2006) focus on the tacit knowledge specifically associated with experts’ decision-making which underpins action. Articulation of the knowledge base used in decision-making is an important means of demystifying the thinking underpinning expert practice, a key aspect of curricula with a focus on deep understanding of disciplinary WTPs (Entwistle 2009). A validated and clearly integrated knowledge base should go some way to helping the curriculum developer.
Kinchin, Cabot and Hay and other colleagues (see reference list) have used concept mapping (based on Novak and Gowin 1984) as a means of eliciting experts’ (both teachers and researchers) understandings underpinning their practice, a technique that explicitly shows the propositional links between the concepts and that makes clear the integrative nature of expert thinking (Davies and Mangan 2007). Kinchin and colleagues have mainly studied in clinical contexts but have argued that their fundamental ideas are applicable to non-clinical university disciplines as well (Kinchin and Cabot 2010).
That such networks of ideas, the articulation of understanding—on the right of Fig. 1—are made explicit is important both for curriculum developers and for learners (Entwistle 2009; Hay 2007; Kinchin and Cabot 2010; Novak and Cañas 2007; Shopkow 2010; Cañas et al. 2015). The ideas can arguably form ‘the “know that” part of the curriculum’ such that students understand the ‘know how’ (Muller and Young 2014:137). Kinchin et al. (2011) have used concept maps to inform curriculum development. Since expertise includes a conceptual basis, it should make it amenable to specification, teaching and learning.
In summary, research has suggested that despite the apparently tacit nature of much of what constitutes expert practice, it is possible to analyse and articulate the ideas important for expert decision-making; and that underpinning the observable chains of practice—which are more evident in some disciplines than others (Kinchin et al. 2008)—there is an integrated network of concepts.
So now we need to consider specifically how expertise in scientific research might be framed so that we can help students develop this understanding.
Framing research expertise in scientific disciplines
Researchers in scientific disciplines—those disciplines that ‘allow direct empirical investigation of an important question’ (National Research Council 2002: 6), a description that focuses on the means of research rather than specifically what they are researching—carry out a multitude of diverse practices as they enquire into phenomena. These practices characterise the WTPs of the disciplines.
However, research practices vary across disciplines and also within them (McCune and Hounsell 2005), and not only because of the specific substantive knowledge required. How research is designed and conducted to solve different problems varies according to the disciplinary circumstances and conventions (for instance, using tightly controlled laboratory-based research; or the observation of phenomena; in surveys which suggest links between factors; or in randomised control trials [RCTs]) with concomitant effects on the validity of the data and the strengths of the claims made. Research may also differ in, inter alia, the equipment employed, the manual skills required and the specific techniques selected. This presents a challenge to curriculum developers. How can such diversity of research practice be conceptualised to aid teaching and learning?
The model in Fig. 1 will be developed with respect to research expertise in scientific disciplines to aid such conceptualisation, using examples from research literature as well as drawing on 20 years of research and teaching that I have conducted with colleagues and to whom I am indebted.
Chains of practice?
It is easy to get the impression of scientific practice as chains. Research practice in science after all may well draw on ‘routines’, those standardised techniques and protocols, which act as shortcuts to a solution and minimise ‘thinking’, other than in their selection. They have become established to make practice efficient and to ensure quality, whether that is how an instrument is deployed, a protocol for a procedure or the ‘rules’ for a specific approach, such as an RCT. These ‘routine chains’ and their appropriate selection are clearly important components of expertise—they represent part of the ‘practice’ in a discipline’s WTPs.
However, expert research practice involves much more than just competence of routine chains! But experts’ familiarity with the context of their research and their quick thinking as they make necessary modifications to their practice can give the superficial appearance of chain-like procedures as they seek solutions to their research problems; the solution being to establish a valid pattern in their data that will be ‘good enough’ for the claims to be made.
That practice appears to consist of chains on the left-hand side of Fig. 1 is strongly reinforced when formal written accounts of research represent research practice as linear.
It is unsurprising, therefore, that a widespread traditional teaching approach in science disciplines in HE (Kinchin et al. 2010) has been to provide students with opportunities to familiarise themselves and enact chains of practice through the use of structured pre-specified practical activities, often designed to be illustrative of substantive ideas. Students are directed, by means of ‘methods’, written by experts and using disciplinary conventions, through the (more or less) detailed linear stages of the practical activity. However, with so many potential chains, the selection of practicals becomes an issue, as do the constraints of time.
Another approach to help students has been to describe commonalities amongst the chains. Willison and O’Regan (2007:399) argue that despite the observed variation in chains ‘the fundamental facets of inquiry are, however, identical, with common processes being acted out across all research endeavours’. Such an approach underpins descriptions of research practice—in effect, generic descriptions of the steps in Fig. 1’s chains—which can be couched in terms that Muller and Young (2014) describe as ‘know how’1 processes or skills. These are typified by terms like research skills or processes such as hypothesising, planning, designing, analysing, interpreting and evaluating, and they may be taught in research methods or statistics courses (Sotos et al. 2007). However, these descriptions of chains may provide a limited basis for teaching; the concepts required for the thinking to be able to do these processes are seldom specified, and in the absence of such guidance there is a danger that students revert to ‘copying’ practices they have seen before (Roberts et al. 2010). Descriptions of chains without the explicit specification of the conceptual basis to guide the teacher and learner are a poor basis for a curriculum specification that aims to develop expertise2 (Kinchin et al. 2008).
For expert researchers, many of the solutions to the decisions outlined in Fig. 2 could already be familiar to them and others may be more or less quickly resolved (Kinchin et al. 2008), possibly giving the appearance of practice as a chain (Fig. 1). It may well take very detailed observation (such as can be found in Roth 2009, 2013) to reveal that individual experts’ chains of practice may be a misleading model of expert practice in scientific research.
Expertise can develop through extensive practice and with time (Feltovitch et al. 2006; Shopkow 2010) but, for the most part, undergraduate students do not have this luxury. The traditional hope that through exposure to many different chains, over time students would develop expertise and pick up the tacit network of understanding is not supported by evidence. Literature shows (see, for instance, Taylor and Meyer 2010; Wilson et al. 2010) that aspects of scientific research expertise indicative of network thinking—such as the ability to design research from scratch; to understand how particular chains of practice have developed; to understand how and why to amend research plans; or the ability to evaluate others’ research and the quality of the claims made—are poorly developed in students following such traditional HE practice and are also unlikely to have been developed during school science (Roberts et al. 2010).
Nets of ideas
Figure 1 represents expertise as being underpinned by a network of ideas that can be constructed to form an understanding, and Fig. 2 suggests that decision-making is an important element in scientific practice; but what is it that experts in scientific research understand so that they can make decisions like this?
Identification of a conceptual underpinning for research expertise is the premise of recent TC-inspired research in different scientific disciplines. In any subject, the specific substantive knowledge is going to be of central importance to any research endeavour. In addition, concepts associated with conducting or evaluating research in scientific disciplines have been identified—referred to by some (see, for instance, Perkins 2006; Mead and Gray 2010; Ross et al. 2010), as the subjects’ epistemes.
For instance, research in biology has identified ‘hypothesis construction and testing’ as a TC (Taylor et al. 2011). Taylor and Meyer (2010:179) explain what is signified by this term: ‘Inherent in this role of hypothesis testing is the broader consideration of the conceptualisation of experimental design and the role that variation in this design may play in terms of verifying and extending previous findings. The concept of hypothesis testing thus provides a framework for experimentation and investigations’. Wilson et al. (2010:100) identified ‘measurement uncertainty’ as a potential TC for understanding in physics—which they explained as ‘an understanding of how to identify different sources of uncertainty, quantify their effects, take those effects into account in planning experiments, analysing data and making logical inferences from those data, and an appreciation of the consequences of uncertainty’. Despite these TCs having different names, the conceptual basis of what is being described has a lot of common ground.
That their TCs transcend disciplinary boundaries has been recognised by the teams researching them (see, for instance, Ross et al. 2010; Wilson et al. 2010). Hall (2010), studying understanding in the interdisciplinary area of climate change, found that uncertainty in the data and its weight as evidence was a key component. Other science researchers have identified aspects of research expertise to be critical and often problematic for students; for instance, Ryan (2014) working in Earth and Environmental science recognised the determination of patterns in data and issues resulting from researching complexity and uncertainty as TCs. Understanding in some cross-disciplinary aspects of scientific research such as understanding of statistics (Gordon and Nicholas 2009) and quantitative numeracy (Frith and Lloyd 2013) is also shown to be dependent on understanding how the data have been derived.
Whether these understandings related to research expertise are TCs per se or not is not the issue here; instead, the focus in this article is on the network of concepts alluded to but not fully articulated in all these works that enable research expertise to be better understood—the net of ideas shown in Fig. 1.
In the UK, the Royal Society’s motto ‘Nullius in verba which roughly translates as “take nobody’s word for it”’ (Royal Society, n.d.) emphasises the central importance in science of evidence over assertion, as scientists make claims following investigations into the real world. All scientific research is judged on the quality of its data regardless of how that scientific practice is enacted. Ultimately such understanding is important both for the conduct of an investigation as well as other aspects of academic expertise, such as in the review of others’ work. These are the WTPs of scientific disciplines, but with the emphasis now on the ‘thinking’ about evidence.
That the ‘doing’ of science includes, inter alia, an understanding of evidence was recognised by Millar et al. (1994). So what is this knowledge base? Gott et al. (n.d.) offered a tentative specification of the concepts of evidence—the ‘thinking behind the doing’—which constitutes a knowledge base for such an understanding; this was further exemplified in Gott and Duggan (2003). Research with experts in HE and science-based industries has indicated that an understanding of these ideas validly represents the thinking they use in their scientific practice (Gott et al. 1999; Roberts and Gott 1999); while Tytler et al. (2001a, b) and Duggan and Gott (2002) have also shown them to be important in detailed case studies of the public engagement with science, the importance of which is clearly made by Goldacre (2011).
Through illustration with a range of scientific practices from different disciplines, Roberts and Johnson (2015) have argued that all require interrelated and nuanced decisions using these ideas, regardless of the context of the work or the approach adopted, decisions that are essentially all directed towards establishing the validity of data and its subsequent weight as evidence for a claim. Readers are referred to the original paper for a full explanation in which contexts ranging from tightly controlled laboratory-based science, field trials, RCTs and ecological surveys are used to illustrate how the understanding of the validity of data, represented in Fig. 3, informs the decisions made in such approaches.
The emphasis of our work is on such an understanding. I will expand on this briefly next and consider its relationship with the observed chains of practice. The curriculum implications of the map will then be explored.
A concept map of evidence
Any specification necessarily requires the use of a language. Different approaches to research have, historically, developed specific terminologies, and the ‘territorial’ nature of research (Jones and Kinchin 2009) with specialised vocabularies is a challenge to the articulation of the concepts of evidence which, we believe, apply across disciplines. Green et al. (2014) refer to the notion of ‘signification’ (the relationship between a word and its meaning) and suggest that work on signification ‘may be especially useful when examining interdisciplinary fields’. In our original specification (Gott et al. n.d.) and in the explanation of the concept map (Roberts and Johnson 2015), we have illustrated the concepts—the signified—with examples from different fields of science to try to reduce any ambiguity associated with the terminology (the signifier). Our use of terms is intended to be ‘neutral’, with weak ‘semantic gravity’ (Maton 2009) and does not privilege any one method or approach.
The vocabulary associated with measurement on the right-hand side of the map is a case in point. Categoric (qualitative) or continuous (quantitative) values all require ‘measurement’ (although often referred to as ‘observation’; see, for instance Gray 2014). All measurements require an ‘instrument’, a term usually associated with the measurement of continuous variables. For categoric variables where the measurement entails the recognition of the defining features of the variable—identifying specimens of a species, for instance—the substantively informed discernment of the observer acts like an instrument. The reliability of any measurement depends on its degree of uncertainty; vocabulary familiar to users of conventional instruments but taken, in the map, to also include uncertainties in identification and classification of categoric variables.
Figure 3 indicates (by means of shadow on the box) those concepts that are directly informed by substantive knowledge. The map emphasises the intimate integration of substantive knowledge with scientific practice. Neither stands alone, each is only as good as the other. The production of data is conceived within, is guided by and uses instruments that depend on existing substantive knowledge. The soundness of substantive knowledge depends on the quality of the originating data as evidence. They are inextricably bound (which has curriculum implications to which we will return later), yet in this article the focus is on the understanding of the quality of data.
Our map focuses on the ideas involved in carrying out a ‘whole’ scientific investigation4 from initial observation to the resultant claim and its position in the broader substantive theory. The map centralises the question of the validity of data since the confidence in the validity in any research practice gives it weight as evidence for a claim—it is this that all investigators are striving for, regardless of what they are researching or what conventions they are adopting, and is at the forefront of expert researchers’ thinking whether they are researching in a laboratory or the field, doing ‘classical experiments’ or ‘observational study’ (Gray 2014). The many different approaches to research and the resultant validity of the data and strength for a claim (Hodson and Wong 2014) can be viewed as the consequence of differences in the nature of the two interrelated sides of the map: in the variables involved and their measurement (as discussed above) in any investigation. Across scientific disciplines and in different experimental contexts the degree to which variables can be isolated and their values manipulated, and the amount of variation in a defined variable (all of which need measuring) influence how relationships are sought and the strength of any resultant claims.
That expert practice involves an understanding of the quality of data—even if ‘doing’ research may appear to involve procedural chains—perhaps becomes more obvious when we consider the understandings experts draw upon when evaluating research. In both informal and formal peer review, others’ ‘routes to a solution’ are judged. Were the decisions made—during every aspect of the research—‘good enough’ for the claims the authors made? Such evaluation is not based on chains of practice of ‘doing’ but requires the ‘thinking behind the doing’ represented by the concept map.
Chains from the net
Figure 2’s model of scientific research as involving trialling and iterative decision-making suggests that the appearance of expert practice as a linear chain (the left of Fig. 1) may not always fit with detailed observations. However, both models point to expert practice being underpinned by an understanding that enables decisions to be made. Figure 3 attempts to articulate this network as an understanding of the quality of data.
The concept map (Fig. 3) represents the conceptual basis of this—the ‘thinking behind the doing’—not a series of procedures, processes or methods. It is not a flow diagram of the sort that is often associated with descriptions of scientific practices nor does it represent any particular pattern of reasoning (Cleland 2002). The focus is on the concepts and emergent understanding we suggest are required to make decisions. The arrow directions in conjunction with the linking terms are there to represent the propositional relationships which give meaning to the concepts and do not imply a procedural or reasoning sequence. The map represents a network of linked ideas, and it is the ‘joined up thinking’ and mental ‘juggling’ of these linked ideas that demonstrates higher-level thinking about research practice which is manifest in the iterative practice of Fig. 2 (even if, as discussed earlier, it may have the superficial appearance of the chain in Fig. 1). The map may go some way to articulating the understandings common to the different disciplinary TCs discussed earlier.
In established chain-like practices, whether ‘routines’ or the ‘methods’ provided in traditional laboratory manuals, all the decisions about the design to establish valid data have already been made. The ‘thinking behind the doing’ went into creating the chain so that practice can be standardised and little further thinking is required. For instance, in illustrative practicals to demonstrate a substantive concept, in contexts where variables can be manipulated, matters can be contrived to give a very small variation in comparison with the effect of changing the independent variable so that only a few (if any) repeated measurements of the dependent variable suffices (for example, as in Royal Society of Chemistry n.d.; Nuffield Foundation n.d.). In summary, Fig. 3 shows a map of the ‘thinking behind the doing’, whatever form the ‘doing’ might take. Its main purpose is in its being a concept map and so emphasising that scientific practice is about ideas to be understood and that can be specified—just like substantive knowledge—and that its teaching should follow accordingly. This, I propose, goes some way to distinguishing Kinchin et al.’s (2008) ‘underpinning expert knowledge’, and it is to the implications for a curriculum that aims to teach this that we now turn.
As an interviewee points out (Shopkow 2010:324):
The …various kinds of ‘know how’ supplement and depend upon the ‘know that’ or conceptual knowledge. They do not replace it … Crucially, the ‘know how’ abilities are dependent on the conceptual knowledge of the domain concerned in all but a small number of mechanical skills and techniques. The ‘skills talk’ that most worries us is that form of discourse which pays lip service to the importance of knowledge but then goes on to concentrate almost entirely on the ‘know how’ requirements of the curriculum. This has the effect of shoehorning the ‘know that’ part of the curriculum into a ‘know how’ box, which obscures the curriculum requirements of the conceptual knowledge—its requirements for sequence, pace, progression and level of difficulty.
Kinchin et al. (2010:85) advocate concept maps as a way of granting students ‘epistemological access’ to the discipline. I propose that a map like that of Fig. 3 would go some way to address this and would also help students better understand the connections of the net to the chains of practice which they are familiar with from teaching (Kinchin and Cabot 2010) and may be relevant to other disciplines in which research gives weight to data as evidence. The map can act as a pedagogic and curriculum planning tool for the lecturer as well as being of value in a metalearning activity for students (Meyer et al. 2015).
‘[Students] don’t have a map. Of how to understand the different parts of the puzzle or the complexity you want them to understand and they have this understanding that this should be a straight forward type of story and they don’t have a map to navigate themselves around through this sea of information’.
Since there has been little research from this cognitive perspective, we have little data on how students take to these ideas or how a curriculum might best be structured. In our experience, undergraduates prior to teaching have some knowledge of these ideas, at least in isolation (Glaesser et al. 2009a) but they fail to ‘join them up’ to develop an understanding that can be readily applied (Roberts et al. 2010). But if explicitly taught the understanding on the map, they are more critical (Roberts and Gott 2010) and are better able to investigate (Roberts et al. 2010). The ideas and understanding in the map are not ‘rocket science’, but they are concepts that are missing from traditional science curricula.
We have found that there is the need for activities aimed at explicitly developing students’ understanding of the quality of data; i.e. the ideas in the map contributing to the validity of data. This has involved students carrying out their own investigations, enacting Fig. 2. We start with simple contexts (similar to those described in Roberts and Johnson 2015 and exemplified in Gott et al. (n.d.) chosen so that they do not make high demands on specialised substantive understanding and where the outcome is not part of prescribed disciplinary content, better still if the outcome is genuinely unknown (an approach also advocated by, for instance, Sternberg 1999; Baillie et al. 2013). This allows the focus to be on getting good enough data to make a claim and develops an understanding of the interplay between the ideas on the map.
Students can feel very disconcerted by such work—it feels so different from the linear practice with ‘right answers’ that they have become accustomed to (Roberts et al. 2010; Roberts, in press). Demonstration of an understanding of the interaction amongst the ideas in the map is seen when students carry out trials and work iteratively in response to the data—making nuanced decisions based on the understanding in the map as they work—which are not features common to usually chain-like illustrative practicals.
Since the focus is on learning ideas and developing understanding, non-practical teaching activities are also appropriate; in our experience, explicit teaching of ideas from distinct sections of the map, with ample opportunities for students to discuss the effects of potential decisions in relation to real data and the quality of claims (their own or others’—see, for example, Bennett 2014) is valuable (Roberts, in press).
Once students have developed this basic understanding in different contexts, they will, arguably, be in a better position to understand the diverse practices and conventions employed across the sciences (Hodson and Wong 2014). Science research is intimately bound with disciplinary knowledge, and an understanding of this, as well as an understanding of the quality of data, is necessary conditions for success (Glaesser et al. 2009b). Students, of course, need to understand disciplinary conventions associated with research in their discipline, but an understanding of the ‘thinking behind the doing’ will, arguably, demystify some of the conventions they find problematic (Taylor and Meyer 2010).
Entwistle (2009:88) reminds us that it is important to determine the ‘inner logic of the subject and its pedagogy’ since how lecturers ‘understand their subject affects how they explain it to their students’. The broad concern of this article is with the network of concepts of evidence that our work shows underpins a deep understanding, vital to expertise, about the quality of data in scientific disciplines where research gives weight to data as evidence. Conceptualising evidence as having a knowledge base represents an ontological shift compared with practice described as processes and has implications for the curriculum; there are ideas underpinning research expertise that can be specified and taught.
The map shows scientific research expertise as drawing on both the specialist disciplinary knowledge base and an understanding of the quality of data and that the concepts of evidence in turn are framed by that substantive knowledge. As such, this map can be seen as extending the disciplinary knowledge base of scientific disciplines. Further research both within and across disciplines is needed to better understand how the specification and teaching of these ideas can best enhance students’ understanding of research expertise.
Muller and Young’s (2014) use of the term ‘know how’ implies ‘doing without thinking’ and the potential threat implicit within this to the place of expert knowledge. This is in contrast to, for instance, Manley and Garbett’s (2000:355) use of the term ‘know how’ which refers to the tacit knowledge involved in an expert’s chain of practice.
Many school science curricula around the world have been framed by such descriptions of performance, and there are widespread concerns as to school students’ understanding of scientific practices (Roberts and Johnson 2015).
It is worth noting that understanding scientific practices has not only been the focus of research in HE. Recently, other school science researchers (summarised in Roberts and Johnson 2015) have moved beyond describing what scientists do (wherein any understanding may be implicit) and now explicitly articulate some of the ideas required for scientific practice; the knowledge-base required to judge the quality of data and understand evidence.
The term investigation includes all types of inquiry. Different practices will place greater emphasis on some areas of the map than others, according to the context.
I should like to thank Philip M. Johnson for his helpful comments on a draft of this article.
- Amirault, R. J., & Branson, R. K. (2006). Educators and expertise: A brief history of theories and models. In K. A. Ericsson, N. Charness, P. J. Feltovich, & R. R. Hoffman (Eds.), The Cambridge handbook of expertise and performance (pp. 69–86). Cambridge: Cambridge University Press.CrossRefGoogle Scholar
- Bennett, K. A. (2014). Using a discussion about scientific controversy to teach central concepts in experimental design. Teaching Statistics. http://onlinelibrary.wiley.com/doi/10.1111/test.12071/epdf Accessed June 16, 2015.
- Cañas, A. J., Novak, J. D., & Reiska, P. (2015). How good is my concept map? Am I a good Cmapper? Knowledge Management & E-Learning, 7(1), 6–19.Google Scholar
- Entwistle, N. (2009). Teaching for understanding at university: Deep approaches and distinctive ways of thinking. Basingstoke: Palgrave Macmillan.Google Scholar
- Ericsson, K. A., Charness, N., Feltovitch, P. J., & Hoffman, R. R. (Eds.). (2006). The Cambridge Handbook of Expertise and Expert Performance. Cambridge: Cambridge University Press.Google Scholar
- Feltovitch, P. J., Prietula, M. J., & Ericsson, K. A. (2006). Studies of expertise from psychological perspectives. In K. A. Ericsson, N. Charness, P. J. Feltovich, & R. R. Hoffman (Eds.), The Cambridge handbook of expertise and performance (pp. 41–67). Cambridge: Cambridge University Press.CrossRefGoogle Scholar
- Flanagan, M. T. (n.d.). Threshold concepts: Undergraduate teaching, postgraduate training and professional development. http://www.ee.ucl.ac.uk/~mflanaga/thresholds.html. Accessed June 16, 2015.
- Frith, V., & Lloyd, P. (2013). Quantitative Literacy and Epistemological Access at University: Reflections on Using the Threshold Concept Framework for Research, Paper presented at Mathematics Education and Society 7th International Conference, 2-7 April 2013. South Africa: Cape Town.Google Scholar
- Glaesser, J., Gott, R., Roberts, R., & Cooper, B. (2009b). The roles of substantive and procedural understanding in open-ended science investigations: Using fuzzy set qualitative comparative analysis to compare two different tasks. Research in Science Education, 39(4), 595–624.CrossRefGoogle Scholar
- Goldacre, B. (2011). Battling bad science, TED Talk. https://www.ted.com/talks/ben_goldacre_battling_bad_science?language=en. Accessed November 11, 2015.
- Gordon, S. & Nicholas, J. (2009). Using examples to promote statistical literacy. In Proceedings of the Australian conference on science and mathematics education (formerly UniServe science conference): Motivating science undergraduates: Ideas and interventions (pp. 58–64). http://ojs-prod.library.usyd.edu.au/index.php/IISME/article/view/6204/6852. Accessed June 16, 2015.
- Gott, R., Duggan, S., Roberts, R., & Hussain, A. (n.d.). Research into understanding scientific evidence. http://community.dur.ac.uk/rosalyn.roberts/Evidence/cofev.html. Accessed 16 June 2015.
- Gott, R., Foulds, K., Roberts, R., Jones, M., & Johnson, P. (1999). Science Investigations 3. Collins Educational: London.Google Scholar
- Gott, R., & Murphy, P. (1987). Assessing investigations at ages 13 and 15. Science report for teachers: 9. London: DES.Google Scholar
- Green, D., Lewis, J., Loertscher, J., & Minderhout, V. (2014). Signification: unlocking threshold concepts in natural sciences with a key from the humanities? In Proceedings of Threshold concepts in practice: 5th biennial threshold concepts conference, Durham University, UK, July 9–11th, 2014. http://www.ee.ucl.ac.uk/~mflanaga/abstracts/TC14Abstract1.pdf. Accessed June 16, 2015.
- Hall, B. M. (2010). Teaching uncertainty: The case of climate change. PhD thesis, Gloucester: University of Gloucestershire.Google Scholar
- Hay, D. B. (2011). Academic practice studies and researcher-led teaching. Paper presented at the annual conference of the Society of Research into Higher Education (SRHE), 7–9 December, Newport, Wales. http://www.srhe.ac.uk/conference2011/abstracts/0153.pdf. Accessed June 16, 2015.
- Johnson, P., & Roberts, R. (in press). A concept map for ‘working scientifically’. School Science Review, 97(360), 15–22.Google Scholar
- Jones, S. C., & Kinchin, I. M. (2009). Integration or coordination? A consideration of two curriculum models in the context of an MPharm module. In Proceedings of the 3rd excellence in teaching conference, King’s College, London (pp. 5–12). http://www.kcl.ac.uk/study/learningteaching/kli/conferences/eitc/2007-2011/2009proceedings.pdf. Accessed June 16, 2015.
- Kinchin, I. M., Cabot, L. B., & Hay, D. B. (2010). Visualising expertise: revealing the nature of a threshold concept in the development of an authentic pedagogy for clinical education. In J. H. F. Meyer, R. Land, & C. Baillie (Eds.), Threshold Concepts and transformational learning (pp. 81–95). Rotterdam: Sense Publishers.Google Scholar
- Mead, J., & Gray, S. (2010). Contexts for Threshold Concepts (1): a conceptual structure for localising candidates. In J. H. F. Meyer, R. Land, & C. Baillie (Eds.), Threshold Concepts and transformational learning (pp. 97–113). Rotterdam: Sense Publishers.Google Scholar
- Meyer, J. H. F., & Land, R. (2003). Threshold concepts and troublesome knowledge: Linkages to ways of thinking and practicing within the disciplines. Enhancing teaching-learning environments in undergraduate courses: Occasional report 4 (pp. 1–12). http://www.ed.ac.uk/etl/docs/ETLreport4.pdf. Accessed June 16, 2015.
- National Research Council. (2002). Scientific research in education. Committee on scientific principles for education research. In R. J. Shavelson & L. Towne (Eds.), Centre for Education. Division of Behavioural Sciences and Education. Washington: National Academy Press.Google Scholar
- Novak, J. D., & Cañas, A. J. (2007). Theoretical origins of concept maps, how to construct them and use them in education. Reflecting Education, 3(1), 29–42.Google Scholar
- Nuffield Foundation (n.d.). Stretching copper wire (quantitative). http://www.nuffieldfoundation.org/practical-physics/stretching-copper-wire-quantitative. Accessed October 29, 2015.
- Patel, V. L., Arocha, J. F., & Kaufman, D. R. (1999). Expertise and tacit knowledge in medicine. In R. J. Sternberg & J. A. Horvath (Eds.), Tacit knowledge in professional practice: Researcher and practitioner perspectives (pp. 75–99). Mahwah: Lawrence Erlbaum Associates.Google Scholar
- Perkins, D. (2006). Constructivism and troublesome knowledge. In J. H. F. Meyer & R. Land (Eds.), Overcoming barriers to student understanding (pp. 33–47). Abingdon: Routledge Falmer.Google Scholar
- Polanyi, M. (1966). The tacit dimension. Gloucester: Peter Smith.Google Scholar
- Roberts, R. (in press). Understanding evidence in scientific disciplines: Identifying and mapping ‘the thinking behind the doing’ and its importance in curriculum development. Practice and Evidence of the Scholarship of Teaching and Learning in Higher Education, 10, [special issue Threshold Concepts and Conceptual Difficulty Eds. Land, R., Rattray, J. & Matthew, R. Due autumn 2015].Google Scholar
- Roberts, R., & Gott, R. (1999). Procedural understanding: Its place in the biology curriculum. School Science Review, 81(294), 19–25.Google Scholar
- Ross, P. M., Taylor, C. E., Hughes, C., Kofod, M., Whitaker, N., Lutze-Mann, L., & Tzioumis, V. (2010). Threshold Concepts: challenging the way we think, teach and learn in biology. In J. H. F. Meyer, R. Land, & C. Baillie (Eds.), Threshold Concepts and transformational learning (pp. 165–178). Rotterdam: Sense Publishers.Google Scholar
- Royal Society. (n.d.). The royal society: History. https://royalsociety.org/about-us/history/. Accessed June 16, 2015.
- Royal Society of Chemistry. (n.d.). The effect of concentration on reaction rate. http://www.rsc.org/learn-chemistry/resource/res00000743/the-effect-of-concentration-on-reaction-rate. Accessed October 29, 2015.
- Ryan, A. M. (2014). Seeing deeply in space and through time: Interdisciplinarity meets. Threshold concepts in earth and environmental science. In O’Mahony, C., Buchanan, A., O’Rourke, M., & Higgs, B. (Eds.), Threshold concepts: From personal practice to communities of practice, proceedings of the National Academy’s sixth annual conference and the fourth biennial threshold concepts conference (pp. 99–104). http://www.nairtl.ie/documents/EPub_2012Proceedings.pdf#page=109. Accessed June 16, 2015.
- Shopkow, L. (2010). What decoding the disciplines can offer Threshold Concepts. In J. H. F. Meyer, R. Land, & C. Baillie (Eds.), Threshold Concepts and transformational learning (pp. 317–332). Rotterdam: Sense Publishers.Google Scholar
- Sternberg, R. J. (1999). What do we know about tacit knowledge? Making the tacit become explicit. In R. J. Sternberg & J. A. Horvath (Eds.), Tacit knowledge in professional practice: Researcher and practitioner perspectives (pp. 231–236). Mahwah: Lawrence Erlbaum Associates.Google Scholar
- Taylor, C. E., & Meyer, J. H. F. (2010). The testable hypothesis as a threshold concept for biology students. In J. H. F. Meyer, R. Land, & C. Baillie (Eds.), Threshold Concepts and transformational learning (pp. 179–192). Rotterdam: Sense Publishers.Google Scholar
- Taylor, C., Ross, P., Hughes, C., Lutze-Mann, L., Whitaker, N. & Tzioumis, V. (2011). Threshold concepts in biology: Final report 2011. http://sydney.edu.au/science/biology/learning/threshold. Accessed June 16, 2015.
- Wilson, A., Åkerlind, G., Francis, P., Kirkup, L., McKenzie, J., Pearce, D., et al. (2010). Measurement uncertainty as a threshold concept in physics. In Proceedings of the Australian conference on science and mathematics education (formerly UniServe science conference) (16, pp. 98–103).Google Scholar
Open AccessThis article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.