Introduction

Assessing the quality of learning environments has a long tradition in pedagogical science and many surveys and questionnaires have been developed. Most of these instruments were developed to fit existing theories about factors which predict achievement and to investigate which factors in the learning environment predict educational outcomes. Parallel to the research on learning environments, cognitive psychologists have been investigating the transition from novice to expert, including suggestions about how to organise learning environments to foster the development of professional expertise. Although student development is considered in both strands of research, insights from research on the development of professional expertise have not yet been systematically implemented in instruments used for evaluating the quality of learning environments. This study was intended to bridge the gap between these domains by developing and validating a Supportive Learning Environment for Expertise Development Questionnaire (SLEED-Q).

The introduction of a National Curriculum in the Netherlands was intended to ensure that secondary educationFootnote 1 (15–18 years) focused more on skills, particularly domain-specific problem solving skills, instead of solely on the acquisition of conceptual knowledge (Bolhuis 1996). Problem solving has long been the focus of research into development of professional expertise. In the fields of professional expertise (Boshuizen et al. 2004) and academic domain learning (Alexander 2003), it is generally acknowledged that experts outperform novices in problem solving. This is due to experts’ (1) well-organised knowledge, (2) thorough problem analysis and problem representation (3) and strong self-monitoring abilities (Chi et al. 1988). The journey from novice to expert begins in the upper grades of secondary school and continues through higher education (formal education) into professional life (workplace). This developmental process can be characterised as a process of continually transforming a repertoire of knowledge, attitudes and skills to improve problem solving in a particular domain according to expertise standards (Boshuizen et al. 2004).

The knowledge taught in upper grades of secondary education and higher education can be characterised as largely decontextualised and formalised academic knowledge which is expressed in, for example, theories, domain principles, equations and graphs. In industrial and post-industrial societies, the formalised knowledge embodied in traditional academic domains such as history, science, mathematics and economics serves as the backbone of formal education (Alexander 2005). The introduction of the National Curriculum and the corresponding emphasis on problem-solving skills as well as formalised knowledge has introduced elements of expertise development into the upper grades of secondary education in The Netherlands.

This paper focuses on the very early phases in the development of professional expertise (i.e. on development of expertise during the upper grades of secondary education). Although secondary education serves several different purposes and is not always included in studies of expertise development, it is a recognised goal of secondary school programs to help students to take their first steps towards becoming experts (Alexander 2005; Bereiter and Scardamelia 1986; Goldman et al. 1999). The goal of secondary school learning environments is not to make experts of secondary school students but to help students to develop the types of knowledge representations, ways of thinking and social practices that define successful learning in specific domains (Goldman et al. 1999; Hatano and Oura 2003) and thus lay the foundations for development of expertise or, as Tynjälä et al. (1997, p. 479): argued “Education as an institution and educational practices have an important role in creating (or inhibiting) the preconditions for expertise.”

Recognising the importance of education in the development of expertise (Alexander 2005; Boshuizen et al. 2004; Goldman et al. 1999), Tynjälä (2008) developed a pedagogical model of development of expertise. This Integrative Pedagogy Model assumes that an ideal learning environment is one in which all the elements of being an expert—theoretical knowledge, practical skills and self-regulation (reflective and metacognitive skills)—are present and integrated. Tynjälä gives a clear account of the knowledge components and learning processes that together constitute a suitable learning environment for development of expertise, but little is known about the instructional principles which would enable researchers and educators to design and implement learning environments from this pedagogical perspective.

Several studies (e.g. Arts et al. 2002; Nievelstein et al. 2011) have provided evidence that development of professional expertise is a malleable process that is responsive to well-conceived, skilfully implemented interventions. These authors discussed educational and instructional implications which foster students’ development in professional expertise. Although an understanding of these implications would have significant pedagogical value, relevant data are scattered throughout the literature and have not yet been brought together in a systematic manner; as a consequence, they are still underexploited in educational practice and research. Following a review of the literature, Elvira et al. (in press) have presented ten instructional principles for the creation of a classroom environment which promotes expert-like behaviour in diagnostic domains (e.g. business, geography and biology), based on an integrative pedagogical perspective. The question of the extent to which instruction in classrooms is based on insights from research on development of professional expertise remains unanswered.

This paper reports the development and validation of an instrument that assesses the degree to which the classroom learning environment is consistent with known principles for promoting development of professional expertise in diagnostic domains. Many instruments for the evaluation of learning environments have their roots in psychosocial or learning sciences. By using Tynjälä’s Integrative Pedagogy Model as a starting point, we choose for a pedagogical perspective on learning environments grounded in professional expertise development research. The presentation of this classroom learning environment instrument is divided into six parts; following this introduction, the second section focuses on development of professional expertise and the third section looks at research on classroom learning environments. The design of our research instrument is explained in the fourth section, the results of a validation study are presented in the fifth section and the final section presents a conclusion based on discussion of the instrument.

Expertise development research

Within the domain of cognitive psychology, substantial effort has been devoted to defining the distinguishing characteristics of experts. “While top performance in any field, ranging from chess to composing, represented the main research interest in the 1970s and 1980s, expertise in professions has emerged as one of the most important areas in this decade the 1990s” (Tynjälä et al. 1997, p. 477). Based on cross-sectional studies of professional expertise which mostly focused on knowledge structures and the cognitive strategies used in domain-specific problem solving, Tynjälä et al. (1997, p. 477) summarised the characteristics of professional experts as follows: experts perceive large, meaningful patterns in their own domain; experts focus on relevant cues of the task; experts represent problems on a deeper level than novices; experts have better self-monitoring skills than novices; experts’ knowledge structures are hierarchically organised and have more depth in their conceptual levels than those of novices; experts categorise problems in their domain according to abstract, high-level principles; and experts’ knowledge structures are more coherent than those of novices (Chi et al. 1988; Ericsson and Lehman 1996, Ericsson and Smith 1991; Eteläpelto 1994; Saariluoma 1995 cited by Tynjälä et al. 1997). Experts are not created overnight. Development of expertise is a long and ongoing process, beginning in formal education and continuing throughout professional life, when the different elements of knowledge, skills and attitudes are continually transformed qualitatively and quantitatively, from the beginning of students’ formal education throughout their professional life (Boshuizen et al. 2004), to support better domain-specific problem solving.

Expertise, as we use the term here, is relative. This perspective on expertise assumes that there are several stages of proficiency between novicehood and expertise (Chi 2006). The literature on expertise development (Alexander 2003; Dreyfus and Dreyfus 1986; Schmidt et al. 1990) provides us with models that describe the path to expertise. This journey involves numerous transitions within and between stages of expertise (Alexander 2003; Schmidt et al. 1990). Stage theories imply a developmental continuum from novice to expert and identify characteristics and development activities at each stage (Grenier and Kehrhahn 2008). In this paper, we focus on the early stages of expertise development, namely, the novice phase, which starts in secondary education, when the foundations for expertise could be laid.

Tynjälä (2008) developed a pedagogical model in which learning towards expertise, in terms of getting better in problem-solving, plays a central role. This pedagogical model integrates the various elements of expert knowledge and learning processes underlying expertise development that unfold around problem solving (see Fig. 1). Problem solving plays a central role in Tynjälä’s (2008) Integrative Pedagogy Model. Various authors (Arts et al. 2006; Boshuizen et al. 2004; Herling 2000) have claimed that the key to expertise lies in an individual’s propensity for solving problems. Expert professionals are constantly solving problems and the ability to solve problems manifests the degree of expertise. The domain-specificity of expertise is reflected in problem-solving ability, such as diagnosing a X-ray photographs in radiology (Gunderman et al. 2001), analysing legal cases (Nievelstein et al. 2010), approving financial statements (Bouwman 1984) and choosing appropriate statistical techniques in applied research (Alacaci 2004).

Fig. 1
figure 1

Integrative Pedagogy Model (Tynjälä 2008)

Full size image

Expert knowledge, another key feature of professional expertise, consists of three kinds of knowledge that are tightly integrated with each other (Tynjälä 2008). Conceptual/theoretical knowledge is universal, formal and explicit in nature and depends on conscious, conceptual thought processes (Heikkinen et al. 2012) supported by texts, figures, discussions or lectures. The second constituent of expertise, practical knowledge, is manifested as skills or ‘knowing how’ and is seldom taught in university settings; it is usually gained through practical experience (Engel 2008). Knowledge based on practical experience is personal, tacit and similar to intuitions in that it is difficult to be expressed explicitly (Tynjälä 1999). The third type of knowledge is related to self-regulation; self-regulative knowledge, including metacognitive and reflective skills, is knowledge about self-regulated learning strategies, and how to plan, monitor and evaluate one’s own learning and work.

The Integrative Pedagogy Model offers an account of how these three knowledge components are both products of expertise and contributors to its development. Tynjälä (2008) argued that integration of the three types of knowledge occurs during problem solving through the following learning processes: transforming conceptual/theoretical knowledge into practical/experiential knowledge; explicating practical knowledge and reflecting on both practical; and conceptual knowledge by applying and developing self-regulative knowledge (see Fig. 1).

Transforming theoretical knowledge into practical knowledge requires that theories are considered in the light of practical experience (i.e. theoretical knowledge is applied in a practical context). Explicating practical knowledge into conceptual knowledge is the process of making practical knowledge accessible and explicit (in the form of texts, figures, discussions or lectures). The third learning process is reflecting on conceptual and practical/experiential knowledge using self-regulative knowledge; self-regulative knowledge is developed further in the process. This process enables students to make practical knowledge explicit and analyse both theoretical and practical or experiential knowledge (Tynjälä 2008); it is a means of increasing awareness of effective learning strategies and developing an understanding of how these strategies can be used in other learning situations (Ertmer and Newby 1996). The model’s premise is that “the processes that lead to expertise are intriguingly domain general in their view of developmental origins” (Wellman 2003, p. 247) but expertise is definitely not domain-general in terms of developmental outcomes and problem solving.

Tynjälä’s Integrative Pedagogy Model reflects the essential role that integration of the three elements of expert knowledge plays in development of expertise. For instance, during the problem solving process, expert knowledge is transformed and developed; simultaneously this knowledge is used as input for the problem solving process. The arrows in the model represent the continuous, holistic character of expertise development. Although Tynjälä outlined the kinds of learning processes which should be fostered in a learning environment, there has been little work from an integrative pedagogical perspective on instructional principles for such a learning environment.

Taking Tynjälä’s framework as a starting point, we have been searching for instructional principles for promoting development of expertise. We derived ten instructional principles for promoting the development of professional expertise in diagnostic domains such as biology, business and geography from a systematic literature review (Elvira et al. in press). The instructional principles derived from these studies were categorised according to their association with the three learning processes underlying expertise development. The instructional principles (Support students’ in their epistemological understanding, Enable students to understand how particular concepts are connected, Provide students with opportunities to differentiate between and among concepts and Target for relevance) were associated with transforming theoretical knowledge into practical knowledge. Two other instructional principles were associated with explicating practical knowledge into conceptual knowledge: Share inexpressible knowledge and Pay explicit attention to all students’ prior knowledge. The last four principles, Facilitate self-control and self-reflection, Support students in strengthening their Problem solving strategies, Evoke reflection and Practice with a variety of problems to enable students to experience complexity and ambiguity) were associated with the learning process reflecting on practical and conceptual knowledge by using self-regulative knowledge. These instructional principles are summarised in Table 1.

Table 1 Description of the ten scales of the SLEED-Q
Full size table

Research on classroom learning environments

Research over the past 40 years has shown that the quality of the classroom learning environment has a critical influence on achievement of educational objectives (Walker and Fraser 2005). Several approaches have been used to assess what aspects of a classroom learning environments facilitate learning achievement and learning outcomes. Various approaches are discussed in this section; they can be distinguished on the basis of the following characteristics:

  1. (a)

    Theoretical perspective

  2. (b)

    Perspective on the classroom learning environment

  3. (c)

    Type of data collection

  4. (d)

    Educational level.

One way of differentiating instruments for evaluation of learning environments is to consider the variety of (learning science) theories by which the research is influenced. For example, the Questionnaire Teacher Interaction (QTI; e.g. Wubbels and Brekelmans 1998) was influenced by the pragmatic perspective of the communicative systems approach (i.e. the effect of communication on someone else). The Learning Environment Inventory (LEI; Fraser et al. 1982), Classroom Environment Scale (CES; e.g. Moos and Trickett 1987) and My Class Inventory (MCI; Fraser and O’Brien 1985) were designed for use in teacher-centred classrooms, whereas the Individualised Classroom Environment Questionnaire (ICEQ; Fraser 1990) was developed to assess the factors which differentiate conventional classrooms from individualised ones involving open or inquiry-based approaches to learning. Other instruments are informed by important concepts in learning theories such as metacognition or social constructivism. These include the Metacognitive Orientation Learning Environment Scale-Science (MOLES-S; Thomas 2003) and the Constructivistic Learning Environment Survey (CLES; e.g. Taylor et al. 1997). The What Is Happening In this Class? (WIHIC; Fraser et al. 1996) was designed to bring parsimony to the field of learning environments research and might therefore be considered to be based on a pragmatic perspective (Dorman 2003).

Studies also differ in their perspective on the learning environment. An important distinction can be drawn between studies which focus on psychosocial factors and studies which also consider physical factors (e.g. Zandvliet and Fraser 2005). From a psychosocial perspective, the classroom learning environment should create favourable conditions for learning; in other words, attention should be paid to factors that, for example, affect student satisfaction, cohesiveness and autonomy. According to Moos (1974), each human environment—irrespective of the type of setting—can be described by three dimensions: Relationship Dimensions (the nature and intensity of personal relationships within the environment and the extent to which people are involved in the environment and support and help each other), Personal Development Dimensions (standard pathways for personal growth and self-enhancement) and System Maintenance and Change Dimensions (orderliness, clarity of expectations, degree to which control is exerted, responsiveness to change). The LEI (Fraser et al. 1982), CES (Moos and Trickett 1987), ICEQ (Fraser 1990), MCI (Fraser and O’Brien 1985), College and University Classroom Environment Inventory (CUCEI; Fraser and Treagust 1986), Science Laboratory Environment Inventory (SLEI; Fraser and McRobbie 1995), QTI (Wubbels and Brekelmans 1998), CLES (Taylor et al. 1997) and WIHIC (Fraser et al. 1996) are examples of questionnaires that focus on the psychosocial aspects of the classroom environment (Fraser 1998; Schönrock-Adema et al. 2012; Van der Sijde and Tomic 1992). Instruments which assess the physical learning environment focus on the physical, ergonomic, chemical and biological factors that can affect a student’s ability and capacity to learn (Zandvliet and Straker 2001). Certain elements of the physical environment (e.g. space, light, colour, noise, materials, thermal control, air quality) influence learning and development (Berris and Miller 2011). It is assumed that physically inappropriate learning environments can be barriers to learning (for example, a noisy classroom can impede concentration and make it difficult to hear the teacher). The Computerised Classroom Ergonomic Inventory (CCEI; Zandvliet and Straker 2001) and Childhood Physical Environment Rating Scale (CPERS; Moore and Sugiyama 2007) are examples of questionnaires that focus on the physical aspect of learning environments.

There is an important distinction between qualitative and quantitative studies in classroom learning environment research. Studies which take a qualitative approach use open-ended responses (Wong 1993), interviews (e.g. Ge and Hadré 2010; Lorsbach and Basolo 1998), participant observation (Kankkunen 2001), logbooks (Stevens et al. 2000), video recordings (Li 2004), field notes (Parsons 2002) and reflections (Harrington and Enochs 2009) to build a detailed picture of learning environments. Although valuable, these techniques are very time-consuming; a reliable and conceptually-sound instrument which provides quantitative snapshots of classrooms would be a valuable supplement to such activities (Walker and Fraser 2005). Quantitative studies involve numerical, structured and validated data-collection instruments, and statistics. A distinction can also be drawn between ‘alpha press’ and ‘beta press’. Murray (1938) used the term ‘alpha press’ to refer to an external observer’s perceptions of the learning environment and ‘beta press’ to refer to observations by the constituent members of the environment under observation. In many cases, classroom learning environment research uses student ratings. Students are in a good position to make judgements about classrooms because they have encountered many different learning environments and spend enough time in a given classroom environment to form an accurate impression of it (Fraser 1998).

Finally, studies can be distinguished according to the educational level on which they focus. Research and measurement of learning environments have been dominated by studies at the secondary school level and, to a lesser extent, elementary school and higher education levels (Fraser 1998).

To date, learning environment research focusing explicitly on the development of expertise has taken a qualitative, learning science perspective on higher education (Ge and Hadré 2010). Using a qualitative design, Ge and Hadré (2010) identified stages in the development of expertise and the processes by which novices gain competence. In this present article, we present the development and validation of a Supportive Learning Environment for Expertise Development-Questionnaire, SLEED-Q, drawing on pedagogical, learning theoretical and psychosocial insights (see Table 1). The SLEED-Q is a quantitative instrument for evaluating the extent to which a learning environment supports the development of professional expertise in diagnostic domains such as business, geography and biology.

Instrument design and development

In order to develop a useful instrument to assist researchers and teachers in assessing the degree to which a particular classroom’s learning environment is consistent with known principles of development of professional expertise, we derived an instrumental framework from the theoretical framework. The first step in developing the draft version of an instrument measuring the extent to which educators have created a Supportive Learning Environment for Expertise Development was conceptualisation of the dimensions. The ten instructional principles of a learning environment supportive for expertise development were reflected in ten dimensions (see Table 1). We wrote items that tapped each of the ten dimensions. The questionnaire was developed in Dutch and presented to Dutch students. The first author of this paper developed entirely new items for the scales epistemological understanding, prior knowledge, differentiation, connectedness and practice with complexity and ambiguity; for the other five scales, we combined original items with relevant items from the ‘metacognitive demands’ and ‘teacher encouragement and support’ scales of the MOLES-S (Thomas 2003), performance control and self-reflection scales from the Self-Regulated Learning Inventory for Teachers (SRLIT; Lombaerts et al. 2007) and the Planning and Monitoring scale, Relevance and Coherence scale and the Self-tackling assignments scale (Sol and Stokking 2008). Only the four items from MOLES-S used in the SLEED-Q were originally published in English. These four items were independently translated into Dutch by a native English speaker (with a Master degree) and a certified Dutch translator (with a Master Degree and knowledge of education in the age category 12–18 years) fluent in English.

The questionnaire items were reviewed by two researchers in educational sciences and one researcher in management sciences to establish content validity. These reviewers agreed that the set of statements was consistent with the underlying theoretical framework. In order to ensure that the items were considered relevant and comprehensible by teachers and students, we tested the instrument in a pilot study in which 36 secondary school students (with three different teachers) completed the digital version of the SLEED-Q. Students reported difficulties in understanding questions to their teachers. They were also asked about the user-friendliness of the digital instrument. After the students completed the questionnaire, two secondary school teachers reviewed the questionnaire to assess the clarity and absence of ambiguity of the items. We made some modifications based on recommendations and comments from the students, teachers and researchers. We improved the phrasing and clarity of some items and the usability characteristics of the instrument (e.g. we added a good introduction to the questionnaire and divided it into clearly arranged, tabbed pages). We also had to make certain changes to the electronic design of the questionnaire. The result of this process was the initial Supportive Learning Environment for Expertise Development Questionnaire (SLEED-Q), a 65-item instrument consisting of ten scales, each corresponding to one of the ten aforementioned instructional principles. The questionnaire makes use of a five-point Likert scale (1 = strongly disagree, 2 = disagree, 3 = neither agree nor disagree, 4 = agree, and 5 = strongly agree).

In order to determine the best factor structure to represent the SLEED-Q, we performed exploratory factor analysis (EFA) and confirmatory factor analysis (CFA). EFA was carried out on the initial instrument, CFA on the modified instrument. Between November 2011 and January 2012, the initial instrument was administered to 430 secondary school students (176 girls; 254 boys) following a business (Management and Organisation; M&O) track, in 18 different 10th to 12th grade pre-university and senior general higher education classes (14–18 years of age, average age 16.83 years; 10th grade: 40 %; 11th grade: 40 %; 12th grade: 20 %) at 9 Dutch schools.

In June and July 2012, the modified questionnaire was administered to 156 secondary school students (90 girls; 66 boys) following a business (Management & Organisation) track, in 8 different 10th to 12th grade pre-university and senior general higher education classes (14–18 years of age, average age 16.35 years; 10th grade: 59 %; 11th grade 41 %) at 5 Dutch schools. As well as completing the SLEED-Q, students provided demographic information (gender, age).

On the basis of evidence that student ratings tend to reflect personal preferences (Kunter and Baumert 2006), we added an external criterion. Students were asked to fill out a validated scale consisting of nine questions (Harackiewicz et al. 2008) about their individual interest in the school subject for which they completed the SLEED-Q to provide a measure of concurrent validity.

Analysis and results

Exploratory factor analysis

The data were subjected to maximum likelihood extraction with an oblimin rotation. We chose maximum likelihood because there was a hypothesis about the underlying structure. Next an oblimin rotation was performed because we assumed that the factors describing the structure were interrelated. The decision about the number of factors to retain was based on initial eigenvalues; we kept all factors with an eigenvalue of 1.0 or higher. A scree plot was also examined, looking for a change in the slope of the line connecting the eigenvalues of the factors. Only items with a loading of 0.40 or more were selected. Internal consistency and reliability of the various scales was assessed using Cronbach’s α, with a value of at least 0.60 as the criterion for acceptable internal consistency (Nunnally 1978; Robinson et al. 1991; DeVellis 1991). We also followed alpha Raubenheimer’s (2004) recommendation that there should be no fewer than three items per factor and that replacement items should be generated if the items of a scale in development do not exhibit sufficient reliability and validity. Based on these criteria, we removed items and added six new items: (V_18: “We learn how the subject M&O relates to other subjects”; V_75: “When working on an assignment we keep track of time ourselves”’; V_76: “The assignments from the book/hand-out deal with examples from the professional world”; V_77: “We get a variety of assignments taken from the professional world”; V_78: “We learn how to comment on fellow students’ work”; V_79: “The teacher teaches us how to deal with feedback”). Three of these new items were from the SRLIT (Lombaerts et al. 2007) and the other three were developed by the first author. Table 2 shows the factor loadings and where and why the modifications were made.

Table 2 Factor loadings for the SLEED-Q
Full size table

Four of the initially conceptualised dimensions were part of the SLEED-Q: Relevance, Self-control and self-reflection, Problem solving strategies and Epistemological understanding. Three new scales were derived from the data: Teaching for understanding, Support learning for understanding and Sharing and comparing knowledge. Following Kember et al. (2010), we titled one scale Teaching for understanding; this subscale tapped encouragement for students to employ a deep approach to learning. This scale (see Table 6 in Appendix) consisted of two Relevance items, two Differentiation items, two Connectedness items and one Self-control and self-reflection item, giving a total of seven items. The scale Support learning for understanding tapped understanding of (new) concepts and information needed to solve problems; it consisted of two items on Prior knowledge, and two items on Problem solving strategies. The scale Sharing and comparing knowledge consisted of two Reflection items and two Inexpressible knowledge items related to articulation of hidden, unspoken knowledge. One of the original SLEED-Q scales, Complexity and ambiguity, was eliminated as the three items comprising it did not load as one factor and did not load on any of the remaining seven factors. The modified version of the instrument consisted of seven scales consisting of 32 items and covering 9 of the 10 initially conceptualised scales.

Confirmatory factor analysis

CFA was performed to determine whether the seven-factor structure obtained using EFA would be confirmed in another dataset. A confirmatory factor model was tested using LISREL 8.80. The seven latent variables were the seven factors identified by the EFA. The 32 questionnaire items were the observed variables. The goodness-of-fit indices for this model, shown in the second row of Table 3, consistently indicated a good fit with the data. The Chi square/df ratio was less than 2.0, indicating that the model was a good fit with the data; the NNFI and the CFI were higher than the recommended value of 0.90 (Hu and Bentler 1999); and the RMSEA (0.070) also indicated acceptable fit (Browne and Cudeck 1993).

Table 3 Fit indices for confirmatory factor analysis models
Full size table

An examination of the modification indices of the seven-factor model revealed that there was still room for improvement. Item 61 “We have information that enables us to judge for ourselves whether our work is finished”, and Item 57 “We are used to checking our own work in the M&O classes” had an unsatisfactory loading tendency and an insignificant t-value for their respective factors and were therefore deleted; this improved the fit of the model (third and fourth row Table 3). We chose the 30-item solution with the sample of secondary school students. The 30 items and the path diagram of the items’ factor loadings are presented in Fig. 2. The factor loadings of the items varied between 0.28 and 0.85 (see Fig. 2). Table 4 shows the scale descriptions and a sample item from the modified SLEED-Q.

Fig. 2
figure 2

Factor-item correlation for the SLEED-Q after excluding Items 57 and 61

Full size image
Table 4 Scale descriptions and sample items for the modified SLEED
Full size table

Internal consistency reliability

Internal consistency reliability was assessed with Cronbach’s α coefficient using SPSS 19.0. As shown in Table 5, α-coefficients ranged from 0.60 for Epistemological understanding to 0.81 for Problem solving strategies.

Table 5 Cronbach’s α coefficients and bivariate correlations for the seven-factor SLEED-Q
Full size table

Discriminant validity

Table 5 gives discriminant validity data in the form of correlations between the factors under investigation. These data indicate overlap, but not to such an extent that it compromises the psychometric properties of the SLEED-Q. The conceptual distinctions between the scales are another reason for retaining them in the SLEED-Q. All the correlations were statistically significant at 0.05 level, except for the correlation between the factors of Problem solving strategies and self-control and self-reflection.

Concurrent validity

Concurrent validity measures the degree to which a test corresponds to an external criterion that is known concurrently (Rubin and Babbie 2007). The concurrent validity of the SLEED-Q was computed using bivariate correlations between the seven factors and total score on a measure of individual interest in the school subject for which SLEED-Q had been completed (see Table 5). The correlations between individual interest in the school subject scores and the seven factors were all significant at the p < 0.01 level. For concurrent validity, weak to moderate correlation coefficients were found for the seven scales of the SLEED-Q and total score on individual interest in the school subject (r = 0.17–0.52). In other words, students with high interest scores on the school subject had low or moderate scores on the seven scales of the SLEED-Q, meaning that the factors of the SLEED-Q and the total score for individual interest in the school subject were only distantly related.

Conclusion

This paper presented the development and validation of the SLEED-Q. Development was based on a systematic review of the literature on instructional practices which promote development of professional expertise in diagnostic domains. Sixty-five relevant instructional practices were derived from this review which theoretically fit in terms of Tynjälä’s Integrative Pedagogy Model and cluster into ten instructional principles. These ten instructional principles were reflected in ten scales. EFA and CFA of responses to the SLEED-Q showed that the questionnaire had a seven-factor structure: (1) Share and compare knowledge, (2) Relevance, (3) Self-control and self-reflection, (4) Epistemological understanding, (5) Teaching for understanding, (6) Support learning for understanding and (7) Problem solving strategies.

Four of the ten original scales (Epistemological understanding, Relevance, Self-control and self-reflection and Problem solving strategies) were confirmed. Three new factors emerged from the factor analysis: items referring to Evoking reflection and Inexpressible knowledge loaded together on the Sharing and comparing knowledge factor; items measuring differentiation, connectedness, self-control and self-reflection and Relevance loaded together on one factor (Teaching for understanding). Items which were originally part of the scales of Prior knowledge and Problem solving strategies loaded together on one factor, labelled Support learning for understanding. Sharing and comparing knowledge and Support learning for understanding reflect the learning processes Explicating practical/experiential knowledge into conceptual/theoretical knowledge and Reflecting on both practical and conceptual knowledge by using self-regulative knowledge. Although both new factors tap the same learning processes underlying development of expertise, they have different foci. The four items loading on Sharing and comparing knowledge relate to reflection and direct sharing of inexpressible knowledge. Interestingly, reflection and sharing inexpressible knowledge are sometimes considered crucial to fostering expertise. Kirsner (2000) argued that, without reflection, only ‘false expertise’ can develop. Several authors (e.g. Alacaci 2004; Nilsson and Pilhammar 2009) claim that sharing inexpressible knowledge of intermediates and experts in a domain contributes to an individual’s enculturation into the ‘expert’ group. Enculturation should be understood in two senses, as process of developing knowledge, skills, habits, attitudes which characterise a specific domain, and as the process of becoming an accepted and a legitimised member of a group (Boshuizen et al. 2004). The factor Support learning for understanding includes four items tapping the activation of elements of prior conceptual and practical knowledge. Chi et al. (1988) suggested that prior conceptual knowledge influences how problem solvers define, represent and solve problems. From an expertise development perspective, Support learning for understanding embodies the requirement for integrated conceptual knowledge and practical knowledge when solving domain-specific problems.

Teaching for understanding reflects the learning processes by which theoretical and conceptual knowledge is translated into experiential and practical knowledge and the process of reflecting on practical and conceptual knowledge by using self-regulative knowledge. The seven items measuring Teaching for understanding reflect instructional practices which promote a deeper understanding of the domain, such as encouraging students to compare and contrast concepts, helping students to see relationships between concepts, and encouraging students to apply domain-specific concepts to assignments. Self-directed learning can promote a more thorough understanding; this is captured by the item “We have deadlines for our assignments”. The factors Sharing and comparing knowledge, Teaching for understanding and Support learning for understanding appear more interrelated than the other factors (Relevance, Self-control and self-reflection and Problem solving strategies). The difference in interrelatedness (in terms of learning processes) of the factors indicates that, from the student perspective, instructional practices focus on each learning process individually as well as on integrating and synthesising the various learning processes.

The items assessing seven factors revealed in this study relate to nine of the ten original instructional principles. Our data provided empirical support for the validity of nine out of ten of the instructional principles identified in our literature review (Elvira et al. in press). It might be considered disappointing that the scale initially designed to measure the complexity and ambiguity of tasks did not survive the statistical analyses. One explanation for the absence of the complexity and ambiguity factor from students’ perspectives of their learning environments might be that teachers do not provide complex, ambiguous assignments at this early stage in development of professional expertise. Several authors have stated that caution is vital when confronting students with complexity; some authors (e.g. Botti and Reeve 2003; Brookes et al. 2011) propose that the complexity of problems, cases, representations and scenarios should be increased gradually: “Starting any sequence of representations with the most regular, simple forms available and minimising contextual features that could potentially confuse or distract the students, will enable students to ‘get their eye in’ when making sense of a representation” (Gilbert 2005 cited by Halverson et al. 2011, p. 816).

Some limitations of this study should be mentioned. First, although other scales had reliability coefficients close to or higher than 0.70, Epistemological understanding had a reliability of 0.6. Closer inspection of the data revealed that the individual items making up this scale did not show low corrected item-total correlations or low factor loadings. One explanation for the lower reliability is that this scale consists of only three items, which is the minimum number of items recommended for a scale (Raubenheimer 2004). Future development of the instrument should include consideration of adding items to this scale and re-examining the formulation of the current items.

Second, whilst an instrument with 30 items might be preferable to a longer instrument, the factor analyses revealed scales with different numbers of items and, although it is not necessary for each scale to have the same number of items, a more balanced distribution of items across factors might result in a more efficient instrument (Johnson and Stevens 2001).

Third, the SLEED-Q only measures student perceptions of the nature of the learning environment. Previous research (Kunter and Baumert 2006) has shown that there are differences between teachers’ and students’ perceptions of the same learning environment. Research into both teachers’ and students’ perceptions of learning environments is important, because divergence and convergence between students’ and teachers’ perceptions have proven to be informative in investigations of teaching and learning processes and could provide valuable information for teacher education programs (Brekelmans and Wubbels 1991, cited by den Brok et al. 2006). Future research should focus on developing a teacher version of the SLEED-Q.

Fourth, the face validity of items (i.e. whether the meaning was clear and the response scales easy to use for students) was evaluated indirectly by asking students to report possible problems with the questionnaire to their teacher. This gave us indirect information about possible misinterpretations; it might be preferable to interview a sample of respondents to obtain first-hand data on face validity.

Fifth, the SLEED-Q was administered in Dutch to Dutch children taught in a particular educational system; cross-validation of the SLEED-Q in other countries with different educational systems is desirable. Earlier learning environment research (MacLeod and Fraser 2010; Telli et al. 2007; Wong and Fraser 1995) has shown that various instruments (e.g. SLEI, WIHIC and QTI) had satisfactory internal consistency reliability and factorial validity, indicating cross-cultural validity. The SLEED-Q is included in the paper (Table 6 in Appendix) to stimulate further research on learning environment factors which promote development of expertise at secondary school level.

Table 6 SLEED questionnaire with corresponding principles
Full size table

Although several scholars (Tynjälä et al. 1997) have argued that educational practices have an important role in creating or inhibiting the preconditions for expertise, the pedagogical dimension of expertise is rarely incorporated into frameworks for development of instruments to evaluate learning environments. The process by which the SLEED-Q was developed is a first step in this direction. The SLEED-Q provides another means of evaluating classroom environments with respect to their potential to promote development of expertise and is complementary to often time-consuming, qualitative approaches. Studies using the SLEED-Q could stimulate discussions with educators about how to create an environment conducive to development of professional expertise, and secondary schools could use the SLEED-Q as a diagnostic tool as part of a quality-assurance process. Practical application of the SLEED-Q would require formulation and implementation of intervention strategies based on questionnaire results and re-administration of the questionnaire to establish the effectiveness of the interventions (Yarrow et al. 1997).