Theorising a contextual framework for moderation of internal assessment: development and opportunities

Suitable execution of moderation policy is challenging but crucial for the trustworthiness and credibility of internal high-stakes assessment systems. In formal education, policies are rarely implemented as intended. Instead, they are enacted in ways influenced by mediating factors including the internal and external contexts of organisations. Ball, Maguire and Braun’s (2012) contextual-dimensions heuristic provides a conceptualisation of organisation-specific contexts, which is useful when the organisation is the unit of analysis. However, comprehensive analysis of policy enactment—including that relating to moderation—warrants consideration of contexts narrower in scope than whole organisations and wider in scope than individual organisations. In this article, we modify Ball and colleagues’ heuristic, incorporating Biggs’ (1993) application of systems theory, to develop a new contextual framework for moderation that is applicable on multiple scales and enables such analysis. This framework is applied to a selection of contemporary moderation studies with scopes that vary from one course, to jurisdiction-wide, to illustrate its utility. Our framework captures the hierarchy of embedded, interacting systems within which moderation is enacted and makes contextual relationships visible, allowing consideration of perspectives between units of analysis. Our framework provides a nuanced conceptualisation of context that distinguishes between material and human factors, and intrinsic and extrinsic contexts. We present potential uses of the framework for education organisations, central agencies and researchers including as a tool for identifying contextual factors involved in executing moderation initiatives and identifying possible pressures, tensions and enablers.


Introduction
Policy change in education is complex.Policies themselves overlay and interact, and organisations enact-and react to-policy change in diverse ways.Thus, what is envisioned by policymakers is not always what occurs in practice.Instead, policies are interpreted and translated in ways that vary from organisation to organisation and, often, from the original policy intent (e.g.see Ball, Maguire, and Braun, 2012, for contextual factors that influence policy enactment in secondary schools).This article proffers a framework designed to provide a mechanism through which to understand variations between policy as intended and as enacted.
Policy that addresses the quasi-autonomous application of complex processes across multiple organisations is particularly susceptible to enactment variation (Ball et al., 2012;Keddie, 2019).The policy domain discussed in the present article-internal, high-stakes educational assessment of learner performance-is a salient example.In its broadest sense, the term internal assessment refers to any assessment judgement of learner performance made by an educator involved in the programme of learning being assessed, while external assessment refers to judgements made under the auspices of a superordinate authority.In some cases of internal assessment, it is not only assessment judgements that are made by the educator; responsibility for the assessment activities and grading systems might also be devolved.In many jurisdictions, internal assessment is used for credentialing, making it a high-stakes process subject to substantial political scrutiny.In these cases, quality assurance processes that maintain the integrity of the assessment process are essential to support the reliability, validity and credibility of qualifications that are based on those assessments.Most often, quality assurance of internal assessment takes the form of 'moderation'-processes designed to ensure an acceptable level of comparability between the assessments conducted by different assessors and organisations, and therefore between qualifications awarded on the basis of those assessments (Klenowski & Wyatt-Smith, 2014).
In this article, we develop a new contextual framework for moderation comprising two elements.The first is a modification of a heuristic developed by Ball et al. (2012), to represent the intra-and extra-organisational contexts of policy enactment.The second is a modification of Biggs' (1993) systems-theory representation of tertiary education.We combine these to describe a hierarchy of embedded and interacting systems within which moderation policy is enacted.This new framework contributes theoretically and empirically by providing an analytical device for shedding light on ways in which policies are enacted within and between the levels of a systems hierarchy, and illuminating contextual factors that exist for entities within and external to individual organisations.We illustrate the framework by discussing its key features in relation to a range of studies.Finally, we consider potential uses of the contextual framework.
Globally, internal assessment is widely used for summative purposes to credential students (i.e. to ascertain whether their demonstrated knowledge, skills and attitudes meet specified requirements of a standard or course; Harlen, 2007), including the award of formal qualifications (Crisp, 2013).To be credible, qualifications-and the assessments that contribute to them-must be robust and trustworthy (Broadfoot, 2007).Moderation assists with calibrating judgements across assessors and professional learning, and can be characterised as a quality assurance process.Construed as such, moderation focuses specifically on the quality and integrity of summative internal assessment (Harlen, 2007;Newton & Shaw, 2014).
Moderation is used as a quality-assurance process in many countries including New Zealand (Crooks, 2011), the UK (Crisp, 2017;Grant, 2012), Australia and Canada (Klenowski & Wyatt-Smith, 2014), the Netherlands (Johnson, 2013) and Switzerland (Mottier Lopez & Pasquini, 2017).The quality-assurance regimes to which moderation contributes are situated within the political and cultural environments of the jurisdictions, societies, sectors and organisations in which they occur.As such, the forms and foci of moderation processes are generally tailored to their contexts (Misko, 2015).In jurisdictions including Australia (Australian Skills Quality Authority, n.d.; Tertiary Education Quality & Standards Agency, 2021), England (Crisp, 2017) and New Zealand (New Zealand Qualifications Authority, 2021a, 2021c), it is mandatory for organisations to have in place systems of internal moderation or cross-organisation (i.e.moderation conducted within or between organisations, respectively) if they are to conduct summative assessment for credentialing purposes (Bloxham et al., 2016;Misko, 2015).In some jurisdictions (e.g.New Zealand), education organisations that conduct internal assessment against sector-based or nationally-specified standards for credentialing purposes are additionally required to engage with moderation conducted by external quality assurance bodies (Hipkins, Johnston, & Sheehan, 2016;New Zealand Qualifications Authority, 2021b).

Policy enactment and context
Individual organisations respond to and enact policy and regulatory requirementsincluding those pertaining to quality assurance regimes and moderation-in contextualised, creative and diverse ways.The enactment of organisation-specific policies (those that are established and enacted within an organisation in situated and idiosyncratic ways) is similarly diverse: such enactment is influenced by the contexts and cultures of organisations and the wider environments within which they are located.Hence, context shapes policy enactment (Ball et al., 2012;Keddie, 2019).Far from being solely the stage-set upon which organisations (or sub-organisation entities) function, '… context is an "active" force … [which] initiates and activates policy processes and choices … in relation to policy imperatives and expectations' (Ball et al., 2012, p. 24).Therefore, to better understand the complexity of policy enactment within and between organisations, ongoing attention to contextual factors is necessary (Keddie, 2019).

Four dimensions of the contexts of individual organisations: a heuristic
Ball and colleagues (2012) developed and tested four inter-linking contextual dimensions (situated contexts, material contexts, professional cultures and external contexts) as a heuristic to assist understanding of policy enactment at the organisation level in secondary schools in England.Situated contexts encompass aspects of organisations associated with geographic location and history.Ball et al. included the student intake of an organisation and the community the organisation serves as components of its situated context.Material contexts refers to the infrastructure of an organisation, including its facilities, buildings, information and communications technologies, budgets, staffing levels and other resources.The professional cultures dimension comprises the professional expertise, experiences, ethos, values, attitudes, philosophies and motivations of teachers and other staff.The external contexts dimension encompasses regulatory and legal requirements, and the expectations and pressures of local communities and wider society.

Levels of units of analysis subordinate and superordinate to organisations
In any comprehensive analysis of moderation policy enactment, it is useful to consider the impact of contextual factors on entities within, and external to, individual organisations, along with those where the entity is the organisation (as it was for Ball et al., 2012).For example, moderation practices and approaches may vary within an organisation, or even within a faculty (e.g.Adie et al., 2013).Sometimes a moderation initiative is only applied in a single department, programme or course.For example, the multiphase moderation initiative involving social and expert moderation approaches that Crimmins et al. (2016) reported on was conducted in a firstyear communications course at an Australian university.
The utility of the contextual dimensions of Ball and colleagues' (2012) heuristic device is most evident when the unit of analysis is the organisation.When the focal unit of analysis is an entity within an organisation-such as an individual faculty or course (e.g.Crimmins et al., 2016)-the professional cultures contextual dimension, such as of teachers' perspectives regarding professional development offered by the moderation initiative, remains internal to that entity, as described by Ball and colleagues.In contrast, the materials and situated contextual dimensions, which-according to Ball and colleagues' heuristic-are internal to an organisation, become contextual dimensions external to the unit of analysis.When applied to whole organisations as intended, Ball and colleagues' 1 3 Educational Assessment, Evaluation and Accountability (2023) 35:449-469 (2012) heuristic allows the illumination of different contextual dimensions of organisations.However, there is also nuanced insight to be gained by applying it to entities within organisations.
Moderation policy often concerns assessment and assessment practices at broader levels than individual organisations.It is important to know how these practices vary across communities of schools and other institutions, whether a particular assessment standard is assessed consistently across a jurisdiction, whether assessments and qualifications are sound, trustworthy to stakeholders, and whether the public can have confidence in assessment and qualifications systems.Thus, sometimes a comprehensive analysis of the enactment of moderation policy involves entities that encompass multiple organisations; in such cases the units of analysis are broader than an individual organisation alone.For example, sometimes moderation is conducted across or between organisations, to attempt to achieve assessment-or assessor judgement-comparability across those organisations, or across a sector.Such moderation may be coordinated by the organisations themselves, or non-regulatory external agents such as peak bodies or external research teams (Black et al., 2011;O'Connell et al., 2016).Moderation is also coordinated and utilised by central agencies to ensure the comparability of assessment or assessor judgements across a jurisdiction (Hipkins et al., 2016).In another variation, some moderation initiatives involve organisations from different sectors (e.g.Grant, 2012).
Moderation occurring at a broader level than that of individual organisations is subject to many of the drivers, pressures and opportunities that would be encapsulated by the external contexts dimension of Ball and colleagues' (2012) heuristic-if the unit of analysis in each case were an individual organisation.However, when the unit of analysis is broader than an organisation, some contextual factors remain external, while others become the focus of analysis themselves, and the individual organisations involved become constituents of Ball and colleagues' material and professional cultures contexts.
An example of the way in which what constitutes an external context dimension is affected by the level of analysis is provided by Adie's (2012) longitudinal study of the middle-school sector in Queensland, Australia.This study involved participants from multiple geographically diverse schools in online real-time social moderation meetings and focused on teachers' professional learning and identity.Nonetheless, because the focus was at the sector level, the organisations, assessors, assessment practices and assessor judgements involved formed elements of the material and professional cultures contexts.If, however, the unit of analysis had been individual organisations with a focus on inter-organisation comparability, other organisations, assessors and assessment practices, would have formed elements of the external contextual dimension.Thus, what comprises each of the contextual dimensions of the heuristic described by Ball et al. (2012) depends on the unit of analysis of interest.
It is important to distinguish between context and level of analysis: The context of an entity within the framework does not change with the focus of a study, whereas the level of analysis is a choice made by a researcher.However, when we change the level of the unit of analysis, what belongs to each contextual dimension shifts accordingly.Considering both micro-units of analysis (for which the entities of interest are encompassed within organisations), and macro-units (for which the entities of interest encompass multiple organisations), it appears useful to adapt Ball and colleagues' heuristic device to better illuminate the contextual mediators and influences of moderation policy enactment at different levels of analysis.
Systems theory provides a movable frame of reference through which to view a phenomenon of interest.We now turn to systems theory as a way of conceptualising contexts that recognises their embedded nature.

Systems theory
Systems theory is the multidisciplinary study of broadly applicable principles and theoretical constructs that pertain to entities-systems-comprising organised and interacting components, interactions and relationships between those components, and interactions between systems and their environments (Boulding, 2003;Midgley, 2003;von Bertalanffy, 2003).A component of a system 'may [itself] be [a system] of a lower order' (Hall & Fagen, 2003, p. 68) such that lower-order (micro-or sub-) systems can be embedded within higher-order (macro-or superordinate) systems.Interactions between system components describe ways in which changes in one component impact on other components and on the system as a whole.Likewise, interactions of a system with its environment-or a higher-order system of which the initial system is a constituent part-describe ways in which the changes in the system impacts on the environment and vice-versa.Nonetheless a feature of many systems as they thus respond to forces from within and without, is that they self-adjust towards equilibrium by absorbing pressure, or by adapting or evolving through feedback mechanisms towards either an established or new equilibrium.If feedback mechanisms are insufficiently effective for the system to adapt or evolve, that system may breakdown and fail (von Bertalanffy, 2003).Biggs (1993) applied systems thinking to tertiary education and conceptualised multiple levels of systems operating in relation to teaching and learning: a hierarchy of 'macro-and constituent micro-systems' (p.78).Working from the micro to the macro, a system might comprise a student, a classroom, a department or faculty, an institution or a community.Biggs represented this hierarchy as a set of concentric circles, with constituent micro-systems nested within a macro-system, which may also be a micro-system within a larger macro-system.
The inner-most component in Biggs' systems hierarchy is the individual student.The next level is the classroom, which comprises its students and teacher, the teaching context and content, teaching and learning activities, students' perceptions of those activities, the teacher's perceptions of the students, and students' learning outcomes.Classroom systems are themselves embedded in department or faculty systems, with components comprising staff, structures, policies, practices and classrooms.The next-most superordinate system is the education institution, which comprises the (preceding) constituent micro-systems, as well as organisation-wide policies and processes, staff, resources and infrastructure.The most 'macro' system in Biggs' hierarchy is the community, which includes the expectations and requirements imposed on tertiary education organisations (Biggs, 1993).Further applying 1 3 Educational Assessment, Evaluation and Accountability (2023) 35:449-469 systems thinking, Biggs proposed that the internal components and micro-systems of tertiary education systems interact to hold those systems in equilibrium, and that each system also works to maintain its superordinate systems in steady states.Thus, each micro-system in the macro-system of tertiary education 'has its degree of autonomy, but also its interdependence with adjacent systems' (Biggs, 1993, p. 83).He concluded, therefore, that whenever attention is focused on one aspect of the teaching or learning context, for example with an intent to effect change, it is necessary to consider both the micro-system of which that aspect is a constituent element and the adjoining systems in order to obtain a '[more] fruitful understanding of teaching and learning' (Biggs, 1993, p. 83).
As an example of a system adapting to maintain equilibrium in response to external forces, we can apply systems thinking to a period of profound change in the New Zealand education and qualifications system.The introduction of the National Certificates of Educational Achievement (NCEA) school-leaving qualifications in the early 2000s marked a major assessment reform in New Zealand, with a shift from a norm-referenced model of assessment to a standards-based (criterion-referenced) model.NCEA comprises a mix of external (predominantly examination-based) and internal assessment against discrete assessment standards, and the New Zealand Qualifications Authority (NZQA) is the central government agency with responsibility for managing the setting and marking of external assessment and the moderation of internal assessment.
In the initial execution of the reform, any plan that NZQA may have had in place for managing the reliability of large-scale standards-based assessment proved inadequate; for a full account, refer to Hipkins et al. (2016).However, in summary, over the period [2002][2003][2004], there was such a level of year-on-year variability in external examination results for individual standards that public and stakeholder confidence in the NCEA qualifications system was severely shaken.The variability contributed to a public and political outcry, which precipitated the New Zealand State Services Commission reviewing NZQA's performance in delivering the secondary school qualifications.The Commission's findings and recommendations led to the introduction of measures to better ensure reliability-in terms of consistency of results over time, and across subjects, standards and organisations-for NCEA external and internal assessments.
Hipkins and colleagues observed that, although the new measures were imperfect, they were 'much better than nothing; without their inception for the 2005 examination round, the variability problems would have persisted, and it is most unlikely that NCEA would have survived at all' (Hipkins et al., 2016, p. 92).Through the lens of systems thinking, we can recognise the assessment reform, the public and political disquiet and the State Services Commission report as external forces emanating from the superordinate system of central government and the NZ public.Through this lens we can also recognise the education and qualifications system as responding to those forces; NZQA as the component that initially exercised inadequate control mechanisms to maintain stability through the period of transition; the measures introduced in response to feedback in the form of the State Services Commission report; and adaptation in the utilisation of these measures for the 2005 examination round and beyond.This period in the early years of NCEA is therefore an example of a system correcting and adapting in order to survive.When viewed through the lens of systems thinking, the successive adjustments to NCEA in the ensuing years (also as described by Hipkins et al. (2016) suggest that these types of dynamics have been ongoing.

A contextual framework for moderation
The hierarchy of nested systems in tertiary education described by Biggs (1993) provides us with a structure within which to conceptualise embedded contexts that we can adapt and apply to moderation systems.His hierarchy offers a way of conceptualising the layers of sub-systems, systems and contexts within which internal assessment-and moderation-occurs, using a series of increasingly expansive lenses.
While the heuristic developed by Ball et al. (2012) takes both intra-organisational factors (the material and professional cultures dimensions) and extra-organisational factors (largely the situated contexts and external contexts dimensions) into account, it always takes an education organisation to be the focal unit of analysis.However, moderation practices may differ within an organisation (Adie et al., 2013), between organisations (Bloxham et al., 2016) and across entire education sectors (Crisp, 2017;O'Connell et al., 2016).Thus, a comprehensive analysis of moderation policy and practice requires focus both on entities that are subordinate to an organisation and on those that are superordinate.In our view, reconceptualisation of the dimensions of Ball and colleagues' device is necessary in order to support such a comprehensive analysis.
We propose a two-dimensional conceptualisation of context that distinguishes intrinsic and extrinsic contexts on one dimension and human and material contextual factors on the other.
Intrinsic contexts are those that are internal or subordinate to the unit of analysis, whereas extrinsic contexts are those that are external or superordinate.Note that, whether a context is intrinsic or extrinsic is relative to the unit of analysis.For example, if an analysis is at a level that focuses on government education agencies with responsibility to oversee the embedding of policy across the compulsory education sector, individual schools are each intrinsic contexts, because they are subordinate to the agencies.On the other hand, if an analysis is at the level of an individual school, (other) schools are extrinsic contexts because they are external to the school that is the focus of the analysis-as are government agencies because the agencies are superordinate to that school.
Human contextual factors comprise the people within a system, and their knowledge, constructs and experiences.Human contextual factors largely combine the professional cultures-and external-context categories in Ball and colleagues' heuristic-that is, the professional expertise, experience, ethos, values, attitudes, philosophies and motivations of people within a system, as well as expectations and regulatory and legal requirements.We also include students, communities and historical contexts in this category, which Ball and colleagues (2012) included in their situated dimension.Our material contextual factors comprise locale (the remaining constituent of the situated dimension of Ball et al.), as well as their material contexts.

3
Educational Assessment, Evaluation and Accountability (2023) 35:449-469 Our proffered framework for conceptualising moderation as a hierarchy of nested and contextualised systems is depicted in Fig. 1.The diagram comprises a set of concentric circles representing the layers of systems.Students are represented at the centre, in recognition of their centrality in education.Each system (layer) that surrounds this core contains both human and material contextual factors, as per our conceptualisation of context (above).
Whether each layer of the system represents an intrinsic or an extrinsic context depends on the layer at which analysis is focussed.The layer that surrounds students represents the learning context they experience (i.e. the teaching-learning-assessment cycle), and the courses or programmes of learning involved.We have called this the 'students' education' system-layer, similar to Biggs' (1993) immediate classroom micro-system.Human contextual factors in this system-layer include teachers and assessors involved with that learning context, as well as pedagogy and learning behaviour.Material contextual Fig. 1 A contextual framework for moderation factors comprise the physical and electronic resources, as well as the facilities and infrastructure, of that learning environment.
In our framework, the next-most superordinate system is the individual 'organisation'.The human contextual factors involved in this system include academic and other staff or personnel (e.g.teachers), academic processes and practices, timetabling, internal quality management systems, human resources and performance management processes.Material contextual factors involved include teaching and learning resources, buildings and facilities, budgets, information communication and technology (ICT) software and equipment, and other infrastructure.
Encompassing the organisation system in our framework is a layer representing the 'education-and qualifications systems' within which individual organisations operate.This layer comprises the human contextual factors of curricula, training schemes, assessment standards and qualifications, regulatory and quality-assurance bodies, standard-setting bodies and other education organisations.Material contextual factors include the budgets, and ICT and physical infrastructure of those bodies and organisations, and other infrastructure such as computer equipment and software supporting online databases.The macro-system in our framework is 'society', which comprises government, community, families, prospective students, graduates, industry and employers.The broad policy directions determined by government originate from this contextual layer and determine or influence what occurs in each of the subordinate layers of the framework.Thus, the macro-system within which moderation occurs has many layers of embedded sub-systems.Furthermore, each layer of this hierarchy has its own contextual influences.Internal assessment-the primary focus of moderation-features in the second, students' education layer.The systems that encompass this layer are increasingly peripheral to assessment itself.
There is flexibility in how the organisation and education-and qualifications systems layers of this framework can be conceptualised; each can be tailored to better reflect the nuances and characteristics of the individual organisation or jurisdiction of interest.The organisation layer is frequently stratified, containing subordinate layers-such as in an organisation where departments or faculties warrant being considered as distinct embedded systems within an overall organisational context.The configuration of the education-and qualifications layer is influenced by variables including the jurisdiction, its legislative structure, and the reach of regulatory and quality-assurance bodies relevant to internal assessment.Depending on these variables, the education-and qualifications systems layer may be conceptualised as singular or stratified (comprising distinct sub-systems where one leads to another, e.g.pre-compulsory education, compulsory education), and as uniform (i.e.applying throughout the jurisdiction) or as comprising overlaying parallel sub-systems (that do not necessarily articulate, e.g. that are state or district-specific).
To illustrate two configurations of education-and qualifications systems and how the framework can be made relevant to different international settings, we now apply it to the neighbouring countries of New Zealand and Australia with their contrasting parliamentary and legal systems.New Zealand is a unitary parliamentary democracy with a central government (Eagles et al., 2006) responsible for national policy, regulation and governance pertaining to the moderation of internal assessment in formal education.The Ministry of Education and NZQA have national regulatory, 1 3 Educational Assessment, Evaluation and Accountability (2023) 35:449-469 quality assurance and standard-setting roles.Quality-assured qualifications, including NCEA, are registered on the comprehensive New Zealand Qualifications Framework (Ministry of Education, 2015).From an internal assessment and moderation perspective, the New Zealand secondary education sector can thus be conceptualised as one uniform (nation-wide) sub-system forming an internal context of the broader stratified education-and qualifications systems.
In contrast, Australia is a commonwealth of federated states and self-governing territories with a federal parliamentary constitution (Eagles et al., 2006).The education-and qualifications systems of the framework are stratified and comprise both uniform aspects and overlaying parallel internal contexts (sub-systems).For example, uniform aspects include that national education and training policies for all levels of education are set at the federal level by the Department of Education, and there is a unitary national qualifications framework, the Australian Qualifications Framework (Department of Education, n.d.).The school-sector sub-system itself comprises further overlaying parallel state-or territory-level sub-systems.The delivery of the Australian Curriculum, and the development, assessment and quality assurance of state or territory senior secondary qualifications, are responsibilities of state or territory education authorities, for example, the Queensland Curriculum and Assessment Authority (Australian Curriculum, Assessment and Reporting Authority, 2016).

Viewing studies through the lens of the contextual framework for moderation
We now apply the contextual framework for moderation to a selection of contemporary studies addressing moderation of internal assessment to identify some of the contextual factors and system layers evident in each.The selection has been chosen from empirically-based studies of moderation that involved 'social' or 'expert' approaches, or a hybrid of the two (Klenowski & Wyatt-Smith, 2014), as identified through the lead author's doctoral literature review (Willliams, 2019).
The studies have been purposefully chosen to provide variation (Creswell, 2012) across each characteristic of: focal unit of analysis level (individual course to whole-of-jurisdiction); education sector (primary school to university); moderation approaches (social, expert, hybrid); internal, inter-organisation, or external quality assurance body moderation; and country.Where more than one study was identified as an example of particular characteristics (e.g.state-wide social moderation at the middle school level), the study that illustrated a specific contextual factor was chosen (e.g.information and communications technology infrastructure; per Adie, 2012).This discussion starts with studies focusing on elements near the centre of the framework, at the students' education system-layer, where internal assessment actually occurs, and moves outwards to consider studies addressing more peripheral systems, such as those that set or enact policy or engage in cross-organisation moderation.As might be expected in studies that deal with moderation of internal assessment, the assessment aspect of the teaching-learning-assessment cycle-itself comprised largely of human contextual factors and intrinsic to the students' education systemlayer of the framework-is evident in all studies about moderation that we considered.For example, internal assessment for a first-year (tertiary) communications course was the focus of the moderation initiative on which Crimmins et al. (2016) reported.Another example comes from an evaluation by O'Connell et al. ( 2016) of a multi-institutional intervention in the Australian university sector that involved nationally-defined accounting standards.The internal assessments at the centre of the intervention were components of the teaching-learning-and-assessment-cycle within the accounting courses at the various participating organisations.
Other aspects the cycle from the students' education layer of our framework are represented in Crimmins and colleagues' study.Crimmins et al. (2016) found that the scheduling of some moderation activities (e.g.moderation meetings to calibrate assessment judgement occurring early in a course) enabled staff to incorporate into their teaching what they had learned through those moderation activities, and thus, to better prepare students for assessment.Further aspects of the cycle are also evident in the King's Oxfordshire summative assessment project on which Black et al. (2011) reported.This was a 3-year, multi-stage intervention involving teachers from three schools in the lower-secondary sector in England that explicitly aimed, simultaneously, to enhance learning and summative assessment, and to improve comparability between schools.Over the life of the project, the assessment activities involved underwent multiple cycles of use and refinement to better achieve both aims.This process clearly situated assessment as being integrally linked to teaching and learning.
In noting that what suits one classroom or course may not suit another, Black and colleagues (2011) also alluded to the nuances and situated nature of each layer of the framework.During the King's Oxfordshire project, these researchers observed that a portfolio assessment approach exposed a tension between flexibility and comparability.In response to this tension, measures such as a clearer definition of the required portfolio composition, and use of intra-and inter-school moderation processes, were instigated.These measures were intended to enhance the comparability of assessor judgements while maintaining sufficient flexibility to enable the selection of appropriate evidence from the different learning contexts involved: different courses and programmes of learning; different classrooms with different teachers; and students in different schools.
The organisation layer of the framework is evident in many of the studies that we considered.Teachers, their professional learning, their development of shared understandings, their perceptions about moderation and the professional learning resulting from participation in moderation are all intrinsic, human contexts for organisations and featured strongly in the studies of Adie et al. (2013) and Grant (2012).In exploring how the staff in one Australian university faculty viewed moderation, Adie and colleagues set the focus of analysis at the level of organisational systems and, in particular, at the faculty sub-system level.The specific focus was on the human contextual factors within that sub-system-the academic teaching staff.
In another example, Grant (2012) reported on a 4-year initiative that involved staff from all primary and secondary schools under the jurisdiction of one Scottish educational authority.Like the study of Adie et al. and  we have already introduced (Adie, 2012;Black et al., 2011;Crimmins et al., 2016), Grant's study focussed on intrinsic, human factors associated with teachers' professional learning, including the development of shared understandings.The initiative aimed to provide professional development for teachers and to develop sustainable assessment approaches in schools and a sustainable inter-sectoral community of practice.Both Grant and Adie (2012) also made mention of other human contexts intrinsic to the organisation layer, reporting that individual teachers applied their learnings to their own practice in an idiosyncratic fashion, and that teachers took their professional learning from moderation activities and then disseminated the knowledge in their own communities of practice.
Academic systems and processes, which are also human contexts intrinsic to organisational systems, have also featured in several studies that we have referred to here.Internal moderation processes designed to enhance the comparability of assessor judgements within organisations featured in the interventions of Black et al. and Crimmins et al.Organisations' assessment and moderation practices-as reported by assessors within those organisations-were a focus for Halliday-Wynes and Misko (2013) in their Australian vocational education and training sector study.These included the widespread practice of organisations developing or modifying assessment instruments for their individual contexts, despite a lack of assessment competence and expertise amongst the assessors involved.
A material context intrinsic to the organisation layer is that of the facilities and infrastructure that support any organisational system.Information and communications technology infrastructure (both extrinsic and intrinsic to the organisation) played an important role in Adie's (2012) study.The social moderation meetings forming the basis of the intervention that she investigated were conducted in an online environment, using an online technology platform (which itself was an extrinsic material context).For teachers to participate, they needed access to an internet-connected computer and a telephone (intrinsic material factors).In some project meetings, multiple teachers from one school 'were clustered around one computer' (Adie, 2012, p. 46).
The education-and qualifications systems system-layer of the framework is also evident in a number of the studies that we examined.Various contexts for that system layer are apparent, including intrinsic human contexts such as education organisations or sector-and jurisdiction-wide standards, qualifications, and curricula.Extrinsic human contexts include regulatory and quality-assurance bodies.
The studies of Adie (2012), Black et al. (2011) andO'Connell et al. (2016) all involved participants from multiple organisations in social moderation approaches, facilitating the development of cross-organisation comparability of assessment and shared understandings between teachers of standards and assessment requirements.The initiative that Grant (2012) reported on was both cross-organisational and crosssector, participants from all primary and secondary schools in the jurisdiction of an education authority were involved.Furthermore, the local education authority itself, another intrinsic human context for the education-and qualifications systems system-layer, was integrally involved in the initiative; the authority commissioned Grant's research and had staff involved as participants.Yet another human contextual sub-system that is intrinsic to this layer and evident in Grant's (2012) study comprised the national guidelines used as standardised criteria for moderation activities.
Another example in which education-and qualifications systems are visible is the Australian Research Council Linkage project, reported on by Adie (2012).This project was instituted in response to a state-wide policy initiative involving the introduction of a standards-based curriculum and assessment approach, both intrinsic, human contexts of the system-layer.Sector-or jurisdiction-wide comparability of assessment and student achievement against national standards, also intrinsic, human contexts for this layer, were aims of the moderation systems reported on by O'Connell et al., (2016; sector-wide), Crisp (2017;jurisdiction-wide), and audited by the Controller and Auditor-General (2012; jurisdiction-wide).The moderation systems utilised by NZQA (Controller and Auditor-General), and those at the centre of Crisp's study (external moderation of the General Certificate of Secondary Education-GCSE-coursework in England) differed from those used in the other studies discussed here: both used expert moderation approaches involving external moderators from the relevant quality assurance bodies.In contrast, other studies used social moderation approaches involving teachers, as well-in the case of Crimmins and colleagues-as expert moderation with more senior academic staff in moderator roles.Thus, the moderation systems of NZQA and those that Crisp reported on reside only in the jurisdiction-wide education-and qualifications systems layer of the framework (although their impacts reverberate through the subordinate system-layers).In contrast, the moderation involved in the studies of Black et al. (2011) and Crimmins et al. (2016) were conducted within education organisations, which are intrinsic contexts to this superordinate layer.The studies of Adie (2012), Grant (2012) andO'Connell et al. (2016) reported on moderation process conducted between organisations (which Black and his colleagues also did)-each of which is an intrinsic context to this layer.
The macro-system of our framework, society, is evident, or intimated, in a number of studies that we examined, two of which we will discuss here.In outlining the purpose of her audit into the performance of NZQA, the Controller and Auditor-General (2012) made specific reference to the importance of NZQA's work to members of society.The stated aims of her audit were to provide assurance that NZQA was able to effectively assure the quality of internal assessment and, thereby, the credibility and consistency of NCEA qualifications.The stakeholders in this case included the New Zealand Parliament, the parents, caregivers and future employers of students, the students themselves and tertiary education providers.By stating that an aim of her audit was to 'provide assurance … about whether NZQA has procedures to ensure that there is no such manipulation' (Controller & Auditor-General, 2012, p. 11), the Controller and Auditor-General addressed a common criticism of internal assessment as being vulnerable to manipulation, and highlighted the function of her audit in maintaining public confidence in the ability of NZQA to quality-assure the internal assessment for NCEA.Thereby, the audit also functioned to maintain public confidence in moderation, which is itself-in part at least-a quality assurance process for internal assessment and qualifications.In terms of the framework for moderation, these functions inhabit the society system layer.Likewise, the Controller and Auditor-General and the stakeholders who were intended 1 3 Educational Assessment, Evaluation and Accountability (2023) 35:449-469 to be served by the audit are contextually intrinsic to this layer.Drawing from systems theory, the Controller and Auditor General's audit is an example of a feedback mechanism (von Bertalanffy, 2003) exerting adaptive pressure on NZQA as a result of the loss of public confidence in NCEA and in the wake of the 2005 State Services Commission report.In response to this pressure, NZQA increased the accountability focus of its moderation system for internal assessment (Hipkins et al., 2016).
In her study focussing on the consistency of internally assessed coursework for GCSE in England, Crisp (2017) also alluded to the societal context.She reported that, at the time of her study, there was continuing debate over the robustness and trustworthiness of internal assessment in England, which are similar to criticisms raised by the Controller and Auditor-General in New Zealand.Crisp observed that, subsequent to her data collection, coursework was replaced with 'controlled assessment', a more restricted version of internal assessment.Implicit in questions about the credibility of internal assessment for senior secondary qualifications is public confidence in the credibility and legitimacy of internal assessment for high-stakes purposes, and therefore, in the credibility of the qualifications themselves.Public confidence is a human context of society-the macro-system of our framework.One function of moderation and other quality assurance processes in education sectors is to maintain public trust and confidence in qualifications and to answer-if not prevent-questions about the integrity of those qualifications (Misko, 2015).Thus, the society system layer is always present in, albeit not always the focus of, studies and analyses of moderation of internal assessment-at least when that assessment is used for high-stakes summative purposes.
Of the studies that we examined, five evince, or at least connote, pressure from a superordinate system that stimulates a response in adjacent intrinsic contexts (i.e.subordinate systems).The initiative reported by Crimmins et al. (2016) was prompted, at least in part, by changes in federally-mandated policy in the Australian university sector, which required greater emphasis on the quality assurance of assessment and learning outcomes.O'Connell et al. (2016) reported on a project that was also established in response to those changes in federally-mandated policy.While we have already argued that the project reported on by O'Connell and colleagues focussed on the educationand qualifications systems layer, decisions and actions were required within the intrinsic contexts of subordinate organisations to allow, enable and facilitate participants' involvement in the project.Those in the control group needed to be able to undertake the pre-test and post-test tasks and required time and resources to do so.Those in the treatment group needed not only to undertake the pre-test and post-test tasks, but also to engage in the intervention: what appeared to be full-day in-person workshops, which, given the geographic spread of participants, would have likely involved travel and accommodation.Thus, components of the organisation layer are also evident in O'Connell and colleagues' (2016) study, albeit less explicitly than the adjacent and superordinate education-and qualifications systems layer.Similarly, while we have used the initiative reported on by Grant (2012) to illustrate the contexts of the education-and qualifications systems layer, participants' involvement in the study emanates from the organisation layer.The resources (and presumably logistics, including timetabling and teacher-release) that would have been required for teachers to participate over the lifespan of the project demonstrates substantial commitment on the part of each primary and secondary school, and each teacher, involved.
Other examples of human contexts within a superordinate system-which exert pressure on intrinsic contexts-are the roles that stakeholders and the public play; stakeholder and public confidence are strong drivers for moderation regimes.The Controller and Auditor-General (2012) emphasised the importance of the public having confidence in an education and qualifications system and in the education and qualifications gained by students.Moreover, stakeholder feedback can prompt debates and policy considerations, for instance, a lack of confidence expressed by stakeholders in the training and assessment delivered by parts of the Australian vocational education and training sector, which led to those parts being identified by the federal-level regulatory and quality assurance body for the vocational education and training sector, the Australian Skills Quality Authority (Department of Education, n.d.), as being at 'high risk' (Halliday-Wynes & Misko, 2013, p. 13).
Our new framework contributes to the theoretical literature.Through applying the framework to a selection of studies we have shown that it is applicable to different organisations, sectors and jurisdictions, and to studies with a range of foci.Across these studies we have elucidated all systems layers of the framework, and the multitude of human and material contexts involved.The involvement of multiple layers of focus in many of the studies that we examined, including those of Adie (2012), Black et al. (2011), Crimmins et al. (2016), Grant (2012) and O'Connell and colleagues (2016), highlights the capacity of our framework to shift between systems and sub-systems as frames of reference, with an associated shift in which (other) systems constitute intrinsic and extrinsic contexts for each frame.Further, the framework enables contextual relationships within and between system layers to be made visible, and to remain so irrespective of shifts in frame-of-reference, revealing the tensions, pressures, forces, actions and reactions present in, or engendered by, those relationships.In this way the framework makes explicit the structure of hierarchies, authority and influence inherent in all social systems.

Potential uses of the contextual framework
We now turn our attention to whom, and for what, this contextual framework for moderation could be of use.We suggest that the framework could be fruitfully used by education organisations, central agencies (e.g. regulatory or quality assurance bodies) and researchers as an empirical analytic tool for designing and evaluating policy initiatives.Taking advantage of its capacity to shift between frames of reference, likely pitfalls and unintended consequences of their enactment could be identified.The framework illuminates the human and material contextual factors within each system layer, allowing the roles that these factors play in policy initiatives and enactments to be made explicit.Furthermore, in drawing from systems theory, the framework could assist in highlighting that an attempt to effect change in one system layer is more likely to be effective if attention is given to interventions in the intrinsic and extrinsic contexts of that layer that might support the intended change.Those involved in policy work within education organisations and central agencies, and researchers, would find such insights to be of great value to their work.

Education organisations
Within education organisations, the framework could assist those involved with policy work at the organisation level, to interpret and understand policy directives or initiatives, and thus determine what to enact and how to do so.For example, it could be used as a structure to guide the identification of existing human and material contexts, both intrinsic and extrinsic, including existing policy enactments, competing pressures and other pertinent situational elements.Doing so may assist decisionmaking in the process of translating policy developed at superordinate systems levels into practise at the level of individual organisations.Similarly, the framework could be utilised to assist in this type of policy work for entities subordinate to an organisation, such as a department or faculty.
The framework could also be used as an empirical analytic tool in the design, evaluation or refinement of intra-or inter-organisational moderation initiatives.It could be used as a structure on which to map the intrinsic and extrinsic human and material contextual factors-including practices and processes, influences and forces, enablers and limiting factors-and to illuminate relationships between them.Such a mapping exercise could help to discern or prioritise where effort would best be applied.It could also help to identify the resource requirements of an initiative of interest across human and material contexts, (e.g. the person-hours involved, or infrastructure required), as well as to compare resource requirements of different initiatives.The framework could also assist the engagement of academic staff in the design or evaluation of a moderation initiative.It firmly positions assessment as an integral part of the teaching-learning-assessment cycle and, therefore, moderation as a powerful influence on this cycle via its action on assessment.Thus, the framework would have particular utility for initiatives that are aimed at enhancing teaching, learning and assessment.Involving staff in a mapping exercise similar to that described above, using a diagram of the framework to stimulate discussion and generate suggestions for how to better meet that aim, is an example of how the framework could be used in a practical activity.

Central agencies, for example, regulatory or quality-assurance bodies
Central agencies (e.g. regulatory bodies) could use the framework as an analytic tool for designing or evaluating policy initiatives.Its utility for this purpose might be to prompt the identification of contextual factors residing in each of the subordinate system-layers.Identifying and mapping the array of pressures, incentives, barriers, enablers, control mechanisms and other factors that exert forces within and acrossfrom macro inwards and vice versa-the layers of embedded contexts would make those forces visible.Making the forces visible would help to make sense of those forces, which, in turn, would assist policy makers to refine their policy work.Doing so might help policy makers to foresee and avert unintended, adverse or perverse consequences and behaviours, adjust a policy lever to bring systems back into equilibrium, or include control mechanisms to work at an adjacent system layer.
Quality-assurance bodies could use the framework to inform thinking about moderation and to design, evaluate or refine moderation initiatives and associated practices.It brings into focus broad, multi-layered contexts within which moderation of internal assessment occurs.As such it may assist policy makers to identify linkages between different policies or design policies that have such connections.The framework could also be used as an analytic tool through which to identify the human and material contexts involved in the relevant systems, and the barriers and enablers to a moderation initiative functioning optimally.Further, such analysis could help in the identification of resource requirements and support professional development.

Researchers
Researchers in the fields of education and quality assurance policy could make use of the framework as an analytic tool for the evaluation and critique of moderation policy enactments.Taking advantage of its capacity for shifting between frames of reference and for making human and material contextual factors visible, the framework would assist researchers to formulate recommendations for the refinement of policy initiatives and moderation regimes.Further, the framework would help researchers to interpret and make sense of findings regarding the effectiveness or not of moderation and other educational quality assurance processes.

Conclusion
In this article, we have proposed a new contextual framework for moderation, the development of which was driven by literature and theory.We have also trialled the framework in post hoc analyses of other studies.Notwithstanding these previous applications, refinements to the framework may be required for other jurisdictions or sectors, just as it is anticipated that the organisational system-layer of the framework would be tailored for individual organisations.
The framework draws together various participants, policies and artefacts that are directly and indirectly involved in moderation.It represents the complex layers of contextual systems within which any given frame of reference for a moderation process is situated.Thus, the framework could inform thinking about moderation, and the design and evaluation of moderation policies, systems and practices.It could be used as an empirical tool to help make sense of discussions about moderation, including those about research findings reflecting on the effectiveness or otherwise of particular moderation systems and processes.
Furthermore, drawing on systems theory, the framework offers a structure that illuminates ways in which contextual systems interact with a system selected as a frame of reference, and with each other.Thus, it brings into focus the multiple embedded systems within-and on-which moderation functions.If we consider a process of policy enactment from only one frame of reference without sufficient attention to the contextual roles of other systems that interact with that frame of reference, we heighten the risk of widespread variation in that enactment, which can 1 3 Educational Assessment, Evaluation and Accountability (2023) 35:449-469 result in egregious unintended consequences.Our new framework is intended as a mechanism to help to prevent and mitigate such consequences.
Funding Open Access funding enabled and organized by CAUL and its Member Institutions