Introduction

Globally, vocational education and training (VET) is considered important for ensuring the supply of skilled labour to the economy and economic competitiveness and also assisting both younger and older students secure an effective transition to working life, the development of domain-specific occupational capacities and effective integration into society. However, for historical and cultural reasons, VET systems around the world differ greatly in terms of learning goals, curricula, and institutional contexts. Hence, it is difficult to compare VET systems and to identify the factors that help or hinder the achievement of these goals in the various VET systems. This lack of international large-scale studies restricts the decision-making basis for VET policy and also represents a deficit for the scientific community in VET research compared to other areas of educational research. In response to the growing interest in VET among its member countries and inspired by its Programme for International Student Assessment (PISA), the Organisation for Economic Co-operation and Development (OECD) is developing a new international large-scale assessment focusing on the professional knowledge and skills of learners undertaking VET programmes. PISA-VET attempts to lever recent innovations in assessment technology to assess and compare learners in initial VET programmes (i.e., those programmes offering a first VET qualification, regardless of learners’ prior education) in different countries through the assessment of individuals’ professional skills/outcomes and their learning environments. After expert consultation and discussions with various stakeholders, the following occupational areas were selected: Automotive technicians, Business and administration, Electricians, Healthcare/Nursing assistants, and Hotel receptionists. The selection of the five occupational areas for assessment in the PISA-VET framework was based on their economic significance, their international relevance in terms of comparable professional skills, and their representation in VET programs across participating countries. These areas are also significant in reflecting a balance of gender participation and the impact of digital and green transitions.

The aim of the initiative is to enhance comparative policy insights for improving the relevance, quality, equity, and effectiveness of programmes of initial occupational preparation. This is aimed to be accomplished through delivering a comprehensive dataset on selected VET outcomes and related background factors. This dataset goes beyond participation and employment rates and aims to provide internationally comparable data on the outcomes of selected VET programmes in key occupational areas. Specifically, PISA-VET aims to:

  1. 1.

    Deliver Comprehensive Data: Generate a comprehensive dataset on selected VET learning outcomes and related background factors, surpassing traditional metrics, such as participation and employment rates and labour market outcomes.

  2. 2.

    Inform Decision Making: Provide internationally comparable data on the performance of selected VET programs to inform decision-making processes at the policy level.

  3. 3.

    Facilitate Peer Learning: Create opportunities for peer learning between countries based on insights derived from assessment, fostering collaboration and knowledge exchange in the field of VET.

  4. 4.

    Promote the Image of VET and raise the profile of VET students: Contribute to promoting the profile of VET globally through evidence-based insights and data.

Such a project is admittedly ambitious, given the different genesis, purposes, forms and enactments of VET provisions. Consequently, work on the ‘PISA-VET Assessment and Analytical Framework’ commenced in January 2023 involving input from more than 100 experts from 20 different countries. The draft framework was published in March 2024 (OECD, 2024) following extensive consultations with national experts, international organisations, social partners and employers. The framework is intended to guide the work of instrument developers in the next stage of the Development Phase of the initiative and to help policymakers understand the purpose and the appropriate use of the assessment results once these become available.

In the present paper, the lead experts involved in the work on the PISA-VET assessment framework discuss the current status of defining and measuring professional knowledge and skills in selected occupational areas as well as the set of transversal employability skills that have been included in the assessment. The key aims of this paper are to disseminate awareness of the OECD’s initiative to a wider audience and to stimulate a discussion amongst researchers and academics working on VET. This is particularly important due to the new international large-scale assessment of VET and the potential impact this might have on VET programmes and VET policies.

Overview of Targeted Skills and Assessment Approaches section of the paper provides an overview of the target population, designated test time, the targeted professional skills and employability skills, and the assessment approaches. System-Level Data and Contextual Assessment section contains information on the collection of system-level and contextual data. Assessment Approaches to Professional and Employability Skills section outlines the assessment approaches for each of the occupational areas as well as for the employability skills and ends with an overview of the assessment framework. Challenges and Promises: Towards a Research Agenda section provides a discussion of the challenges and promises and suggests a future research agenda.

Overview of Targeted Skills and Assessment Approaches

Target Population and Test Time

The PISA-VET scheme targets learners who are about to complete an initial vocational training programme, regardless of their age. However, adult learners returning to VET after longer periods of work and education can be included, but are analysed separately from the main target group, as their experience, educational backgrounds and experiences are likely to be of kinds that are quite distinct from younger attendees. VET programmes concerning the selected occupational areas can be school-based or combine school-based and work-based learning. They are classified as upper-secondary and post-secondary non-tertiary education levels and assigned to levels 3 and 4 of the International Standard Classification of Education (ISCED) and levels 3 and 4 of the European Qualification Framework (EQF). Furthermore, there is no universal age range, but most learners are aged 18 to 20 when they take the test.

A central challenge of PISA-VET arises from the restriction of test time. The total test time is expected to be 3 to 4 h. In addition to the assessments of professional and employability skills, this timeframe includes an introduction, questionnaires, and breaks in the testing process. It is assumed that the target population, in contrast to PISA, tolerates longer test times of domain-specific (i.e. occupational-specific) assessments as they are accustomed to full working days. However, the costs of a longer test time and test fatigue also play a role.

Overview of the Targeted Professional Skills and Employability Skills

The main goal of PISA-VET is the regular assessment of learning outcomes of selected initial VET programmes by measuring ‘… knowledge, skills, attitudes and values of learners and their capacity to apply these competencies creatively and effectively in work situations and real life’ (OECD, 2024, p. 11). This definition relates to common competence definitions in VET research (e.g., Mulder et al., 2006; Weinert, 2001) that have been widely accepted in empirical competence research in various vocational fields (e.g., Abele et al., 2014; Deutscher & Winther, 2018; Michaelis & Seeber, 2019; Rausch et al., 2016). As the design and implementation of initial VET programmes vary widely across the participating countries, the concept of occupational areas is used instead. An occupational area refers to one or a group of related occupations in the labour market. The following five occupational areas have been selected for PISA-VET: (1) Automotive Technician occupations, (2) Electrician occupations, (3) Business and Administration occupations, (4) Health Care occupations, as well as (5) Tourism and Hospitality occupations. Some of these occupational areas refer to broader occupational fields and have been narrowed down to bundles of work situations that are of high practical relevance and at the same time allow for an efficient and valid measurement as well as comparison across countries. Table 1 gives an overview of the occupational areas, an evaluation of their scope, relevant work processes and the corresponding professional knowledge and skills (for more details see Assessment Approaches to Professional and Employability Skills section). The relevance of these areas and rationales for their selection for PISA-VET are discussed in the respective subsections in Assessment Approaches to Professional and Employability Skills section.

Table 1 Overview of the occupational areas, relevant work processes and corresponding professional knowledge and skills
Full size table

In addition to assessing professional skills in these occupational areas, PISA-VET includes the assessment of employability skills referred to as foundational, transversal or transferable skills, cross-domain or generic skills, core skills, key competencies, soft skills, and 21st-century skills. They represent a broader ‘… set of transversal and foundational skills that are fundamental for success in the workplace and for navigating a changing labour market’ (OECD, 2024, p. 175). They are competencies which are relevant and transferable across occupational areas. OECD’s PISA (OECD, 2019a) and PIAAC (OECD, 2013) measured similar constructs, including problem solving, critical thinking, communication, collaboration, literacy, numeracy, information and communication technology (ICT) literacy. Given the long-standing debates about such general competences, the decision was made in favour of the following four employability skills: (1) Literacy, (2) problem solving, (3) task performance, and (4) collaboration, all of which had been studied in previous OECD assessments. Table 2 provides an overview of the four targeted employability skills.

Table 2 Overview of the four employability skills considered in PISA-VET
Full size table

Overview of the Assessment Approaches

Consistent with earlier OECD studies and other frameworks, competencies are defined as combinations of knowledge, skills, abilities, and other characteristics (KSAO) that may be more occupation-specific or more transversal (Mikhridinova et al., 2024; OECD, 2005). Practice-oriented professional competencies refer to KSAO needed to carry out selected professional activities that are typical for the work and business processes in the selected occupational areas. Regarding assessment approaches, these practice-oriented competencies are particularly difficult to measure with issues of validity and reliability compromised by the different circumstances in which they are manifested across nations, trades, and enterprises where they are utilised. Overall, the PISA-VET assessment framework lists five assessment approaches (OECD, 2024, p. 17):

  1. 1)

    Reading and answering questions about digital texts, which may include scrolling through texts, navigating through hypertexts or clicking on links (e.g., PIAAC literacy)

  2. 2)

    Solving scenario-based problems from everyday contexts (e.g., PIAAC problem solving)

  3. 3)

    Scenario-based questions of occupation-specific knowledge and skills

  4. 4)

    Digital simulations of occupation-specific professional skills in the workplace

  5. 5)

    Live or recorded demonstrations of occupation-specific professional skills

The assessment of transversal competencies is based on established instruments from previous OECD studies such as the assessments of literacy, problem solving or social and emotional skills. Although these approaches are not free from criticism (e.g., Dörner & Funke, 2017), they have proven to be very useful in large-scale assessments (OECD, 2013, 2015, 2021a, b).

The core challenge of PISA-VET lies in the development of authentic assessments of occupation-specific competencies. In the assessment literature, these types of assessments are referred to as performance assessments. Performance assessments are based on domain-specific, holistic, complex, and extended ‘hands-on’ problem-solving processes and usually require a constructed response (Kane et al., 1999; Lane & Stone, 2006; Messick, 1994). The PISA-VET framework, acknowledges the greatly varying approaches to performance assessments are summarised as a range from scenario-based questions to digital simulations and live or recorded demonstrations (see points 3 to 5 of the above list). The recommended approaches come with the typical validity-reliability-dilemma in the performance assessment (Trigwell & Prosser, 2014; Wilson, 2004). Validity requirements demand, among other things, that the test situations correspond as closely as possible to the situations of the respective work practice (for an overview, see AERA et al., 2014). This correspondence between test situations and real-life situations is often referred to as the authenticity of an assessment (Frey et al., 2012; Messick, 1994). Authentic assessments require test-takers to apply the same competencies that they need to apply in the given professional setting (Gulikers et al., 2004), which is not the case in multiple-choice tests (Frey et al., 2012). However, regarding reliability, authentic performance assessments of complex skills often come with poor psychometric qualities (Fischer et al., 2017; Kane, 2013). The reliability of a test is generally higher if many similar test items measure the same narrowly defined construct and if response spaces are less open which allows less room for interpretation. Hence, there is a trade-off between the depth and breadth of an assessment which is also related to test time, test motivation and fatigue (Kane, 2013; Seifried et al., 2020). These must be taken into account when evaluating the results.

Scenario-based questions of occupation-specific knowledge and skills emphasize psychometric qualities and the content-related breath of the assessment. An authentic scenario is presented by texts, images, and/or video vignettes as an introduction (i.e., item stem) followed by closed items with pre-defined response options (e.g., multiple choice) or short-answer items, as seen in the PISA studies. In the assessment literature, such assessments are also referred to as situational judgment tests (SJT). In SJTs, test-takers are presented with job-related problem scenarios and a list of plausible response options (Lievens et al., 2008; Whetzel et al., 2020). Due to the limited interaction possibilities, SJTs are considered low-fidelity simulations (Schäpers et al., 2020). Figure 1 shows a scenario from a situational judgment test on the social competencies of prospective medical assistants (Monnier et al., 2016).

Fig. 1
figure 1

A situational judgment test on the social competencies of prospective medical assistants (Monnier et al., 2016)

Full size image

Simulation-based assessments also provide a higher degree of authenticity, as compared to scenario-based questions, since they do not only present an authentic situation, but simulation environments also allow for the manipulation of this situation, such as in a flight simulator. At the same time, simulation-based assessments enable standardised test situations across test-takers (Cedefop, 2022; Duchatelet et al., 2022; Mislevy, 2011; Seifried et al., 2020; Williamson et al., 2006). Simulation-based assessments for various VET sectors have been developed within the ASCOT research and transfer initiatives funded by the German Federal Ministry of Education and Research (BMBF). Simulation-based assessments in the fields of automotive technicians, electronics technicians, and commercial employees showed promising psychometric qualities while enabling high authenticity (Abele, 2018; Klotz et al., 2015; Rausch et al., 2016; Walker et al., 2016). Figures 2 and 3 show screenshots from a car simulation and an office simulation.

Fig. 2
figure 2

Screenshot from a car simulation for assessing diagnostic skills in automotive technicians (Meier et al., 2022)

Full size image
Fig. 3
figure 3

Screenshot from LUCA Office Simulation for assessing domain-specific problem solving in commercial trainees (Ludwig et al., 2024)

Full size image

Direct or recorded observations of students performing in a real-world setting offers an even more authentic and engaging assessment environment (Lund & Kirk, 2010). Observation-based assessments of occupation-specific professional skills have been developed by WorldSkills International since the first competition, in 1950, in Madrid. WorldSkills has established a global movement with 86 member countries aiming to elevate the status and acknowledgment of vocational and professional skills, by organising major events like the biennial WorldSkills Competition (Hoey, 2009; WorldSkills, 2024). In 2022 over 1,000 competitors took part in more than 62 professional skill assessments. The professional skills in the WorldSkills portfolio range from bakery to cyber security, beauty therapy, renewable energy, and car painting. The assessments are designed to take between 15 and 22 h, within a period not exceeding 4 days (WorldSkills, 2019) and increasingly incorporate technology to represent real-world practice. Since the assessments resemble real-world work tasks and the participants are used to full working days, the long test time does not seem to impair test motivation. On the other hand, the participants in WorldSkills are rigorously pre-selected and are likely to be highly motivated due to the competitive and prestigious nature of the event. Besides the high face validity of the real-world work environments and an extensive quality assurance system, little is known about the psychometric quality of these assessments. The psychometric quality of the assessments is highly dependent on the judges who are trained and monitored within the WorldSkills programme. Finally, the relatively long test time, logistical challenges of providing real work environments and associated costs also play a role. However, WorldSkills International can serve as a model of live assessments for developing assessments for PISA-VET. Figure 4 shows a collage of WorldSkills assessment situations in various occupations.

Fig. 4
figure 4

WorldSkills assessment situations in various occupations

Full size image

Table 3 summarises the characteristics of the proposed assessment approaches.

Table 3 Overview of the characteristics of the proposed assessment approaches
Full size table

Some authors of the occupation-specific chapters of the framework have expressed preferences for one approach or another. However, all three assessment approaches are considered as equal options in all occupational areas in further development phases.

System-level Data and Contextual Assessment

Complementing the assessment of professional skills in an occupational area and employability skills, PISA-VET introduces system-level data collection and a contextual assessment. These two sources of data allow VET systems to be described and compared across countries. Additionally, they enable analysis of how student performances in VET programmes relate to institution-based learning and work-based learning, as well as to learners’ economic, social, and cultural contexts.

System-Level Data

The system-level data collection provides a categorization and a high-level international comparison of key features of VET systems. Internationally, different purposes of VET programmes produce great variability in programme structure and characteristics as well as target population and learners’ characteristics. The purpose of system-level data in the context of PISA-VET is to help the interpretation of results. This provides a basis to analyse the relationship between the test results of PISA-VET and various features of the design and delivery of VET in different countries. In contrast to PISA, the target population of PISA-VET is defined by a specific level of targeted learning outcomes, rather than age. The profile of learners will therefore vary across countries in terms of age and prior education, while the learning providers involved (e.g., school, college, company) will also be more diverse than in the context of PISA.

Much information on VET is available through existing data collections, in particular the UNESCO-OECD-Eurostat data collection, the Network on Labour Market, Economic and Social Outcomes of Learning (OECD, 2018a) and the Network for the Collection and Adjudication of System-Level Descriptive Information on Educational Structures, Policies and Practices (OECD, 2018b). These are publicly available online and/or are included as indicators published in the annual ‘OECD Education at a Glance’ report (OECD, 2023a).

Comparative data on VET are available on a range of key issues. For example, participation data shed light on the place of VET within national skills systems (e.g., one of the options in upper secondary schooling vs. mostly provided at postsecondary levels). Data are available on pathways to and from vocational programmes. The profile of VET learners is an important contextual element, with comparative data available on the age and gender of learners. How VET is delivered also varies across countries – key data here include the use of work-based learning. Data by field of study provide some insights on the skills targeted by VET. Data are also available on expenditure in vocational programmes, on teachers in VET, but not on in-company trainers. Finally, data on different social and economic outcomes provide an indicator of the effectiveness of VET.

A dedicated system-level questionnaire will complement these data, allowing respondents to specify if the response varies across programme types or occupational areas. Questions will focus on six issues:

  1. (1)

    Choice and selection in VET (e.g., factors that constrain student choice of the type of programme),

  2. (2)

    Profile of VET learners (focusing on socio-economic background in comparison to their peers in general programmes),

  3. (3)

    Delivery of VET (% of time spent in different settings; types of institutions providing VET),

  4. (4)

    Skills targeted by VET (e.g., level [national, sub-national] at which learning outcomes are defined; instruction time dedicated to general vs. vocational subjects),

  5. (5)

    Teachers and trainers (qualification requirements for VET teachers; requirements for companies that provide work-based learning and “training the trainers” initiatives), and

  6. (6)

    Governance (ministries involved, social partner engagement).

At the same time, these indicators are inherently imperfect as measures of the quality of vocational programmes in participating countries. The availability of relevant data, even at a national level in at least some countries, is a major constraint. For example, regarding the quality of work-based learning, questions will explore whether targeted training is mandatory for apprentice supervisors. However, further details such as the duration and content of training programs for supervisors, could not be included in the planned questionnaire due to time constraints among respondents and limited data availability. Similarly, the engagement of social partners is widely viewed as crucial in high-quality VET systems. Yet, the questions included in the system-level questionnaire will not be able to capture how social partners are engaged, particularly in terms of the quality of their engagement, in the design and delivery of VET. Improving the availability of comparable system-level data should be another development goal of the PISA-VET initiative in order to gradually improve the framework in this area as well.

Contextual Assessment

The contextual assessment of PISA-VET builds on the PISA 2022 Contextual Assessment, and particularly on the Educational Prosperity model (OECD, 2019c; Willms, 2018), on the PISA 2025 Science Framework in recognizing the greater agency of learners in the face of climate change (OECD, 2023b), and on the PISA 2025 Learning in the Digital World assessment framework (OECD, 2023c), to orient the focus of PISA-VET assessment towards digital transition and digital skills. The framework also borrows from and reflects on available indicators and instruments on the quality of VET learning contexts (e.g., Blom & Meyers, 2003; Deutscher & Braunstein, 2023) as well as on lessons learned from international studies on VET (e.g., Griffin, 2017), from national assessments, and from VET research studies conducted in the targeted occupational areas (e.g., Böhn & Deutscher, 2021; Nickolaus et al., 2008; Striković et al., 2022).

The theoretical underpinnings of the framework embed the experience of studying and working in initial VET programmes, where according to the 3P model (Biggs, 1999), learners experience different qualities of their learning environment on a presage, process, and product level (Tynjälä, 2013). This perspective is complemented with a broader description of learners’ life trajectories and their skill development for employment in the labour market. Following a life-course approach, the framework establishes that learners’ outcomes, participation, progression, and academic outcomes, at all levels of education, build on past educational tracks and depend on successful personal transitions (Ainsworth & Roscigno, 2005; Alba & Lavin, 1981; Bozick & DeLuca, 2005; Dougherty, 1987; Entwisle & Alexander, 1993). A life-course approach explains learning success as well as timely entry into post-secondary education or its premature termination. The interplay of personal transitions, educational and social resources, and institutional mechanisms leads to the construction and negotiation of educational careers that extend into midlife (Astone et al., 2000; Boudett et al., 2000; Kuh et al., 2006). Youth enter, progress, and achieve academic and personal goals in secondary and post-secondary education according to stratification mechanisms, by building on prior achievement, resulting from the sifting and sorting processes embedded in educational, work, and social institutions. Health and the socioeconomic advantage experienced early in the life course continue to impact youth development, with ripple effects on their outcomes at various ages, from childhood to adulthood.

The contextual framework proposes that measuring the success of learners enrolled in VET should go beyond occupation-specific and employability skills, by including learners’ health and well-being, progression and engagement in the programme, aspirations and attitudes towards VET as well as the development of occupational and organisational identity. These outcomes vary by sociodemographic characteristics and by educational and work background. Learners’ success depends on their previous experiences and relates to their current practices and interactions in vocational ‘communities of practice’ (Lave & Wenger, 1991), as well as interacts with the resources available in learners’ social, educational and occupational environments.

The contextual framework aims to measure quality, equality and equity in VET educational systems. Accordingly, while quality is measured through self-report scales and equality is measured by the differences among groups in the distribution of learners’ success outcomes. These are occupation-specific learning outcomes, employability skills, learners’ educational aspirations, engagement in work and learning, occupational and organisational identity, health and well-being, and work safety. Equity refers to the possibility for all groups of VET learners to experience success in the form of these outcomes, by having fair access to foundational resources, relationships, and processes for success. These are the quality of instruction and training in schools and workplaces, inclusive environments, effective usage of institution-based and work-based time, support by family, friends, and communities, material and human resources and a well-organised collaboration between VET institutions and workplaces. Unfair access to foundational factors increases inequalities in outcomes. Equity in VET can be effectively assessed and ensured by focusing on policy-specific “priority groups” and by monitoring when and where they experience barriers. Research consistently indicates that five types of learners should be prioritised, identified and monitored in their access, progression, and acquisition of skills (Bowman, 2004): (1) women of various age groups, (2) students from lower socioeconomic backgrounds, (3) students whose first language is not the language of the country of assessment, (4) learners with lower levels of proficiency in literacy and numeracy and with a history of school dropout, and (5) students with disabilities.

The measurement instruments of the contextual assessment are self-report questionnaires and include a learner questionnaire, a teacher questionnaire, and a VET institution questionnaire, administered to the VET institution’s leader. The VET institution questionnaire can include programme-specific items for each of the occupational areas. Its function is to collect data on the institution as a whole as well as on the specific VET programmes in the selected occupational areas administered in and by the particular institution. The questionnaires targeting the institutions and the teachers address questions on three different administrative and contextual levels: (1) Questions that are specific to organisational structure and resources of different occupational area programmes within an institution, (2) questions about the organisational structure and resources of the educational institution, and (3) questions about the context and the daily activities or practices of the single programmes. As an additional source of data, trainers in the work-based learning environment that are most knowledgeable about the participating learner’s training are requested to complete a one-page questionnaire. The trainer/employer questionnaire will only be administered in countries and programmes that have a substantial work-based component. The groups of respondents are selected to provide a mix of unique perspectives and combined views of outcomes, processes, and resources. Triangulation is implemented by combining data on the same general area of assessment from different respondents, using different instruments. All of this then sets out the intended approach to assessment across these occupational areas. However, similar limitations to those already discussed above for system-level data must also be emphasised regarding the contextual assessment. Measurement instruments must be continuously developed further.

Assessment Approaches to Professional and Employability Skills

In the following sections, the assessment approaches for each of the occupational areas is outlined and discussed.

Automotive Technician

There are four key reasons for selecting the occupation of automotive technician in the development phase of PISA-VET. Firstly, automotive technology is of great economic importance worldwide. In most OECD countries, motorised vehicles are the predominant form of transport. Automotive technicians repair, troubleshoot, and maintain these vehicles. Secondly, the knowledge and skills that automotive technicians should have are internationally comparable, as automotive technicians work with comparable technologies and therefore fulfil similar work tasks, even if the age of the vehicles and the technologies can vary considerably. Thirdly, it makes sense to include the occupation of automotive technician in PISA-VET, as this occupation is learned in VET programmes in many countries and by many people (Baethge & Arends, 2009). Fourthly, automotive technicians are an integral part of both digital and green transitions. Digital information and communication systems are becoming increasingly important in the context of the green transition and future technologies (e.g., electric vehicles). They require effective VET programmes that equip automotive technicians with the knowledge and skills related to these technologies. Around the world, automotive technicians generally complete a 3- to 4-year VET programme (e.g., United Arab Emirates: 3 years, Australia: 4 years). The vocational training programmes that qualify automotive technicians are aligned with levels 3 and 4 of the European Qualifications Framework (EQF) and the International Standard Classification of Education (ISCED).

Automotive technicians’ work settings fall into two categories: work in the production of vehicles or work in workshops. Following the feasibility study by Baethge and Arends (2009) and the 2022 WorldSkills Occupational Standards, automotive technicians working in workshops are selected as the focus for PISA-VET. In workshops, the task of automotive technicians is to investigate and rectify light vehicles. When investigating light vehicles, automotive technicians diagnose the causes of malfunctions or inspect the vehicles to determine whether the vehicles have undesirable conditions (e.g., brake pads that are outside the limits specified by the manufacturer). When rectifying light vehicles, they carry out corrective actions: They replace and (dis)assemble, restore and rebuild or adjust (parts of) vehicles. These corrective measures are generally referred to as “repairs”. The broader term “rectify” is used because it includes not only repairs but also replacements and adjustments. Contentwise, these work processes relate to the drive unit, the powertrain, the undercarriage, the assistance, and the chassis; however the latter is excluded because in some countries, it does not fall within the scope of automotive technicians.

The automotive technician domain is defined as the knowledge and skills required to fulfil these tasks and subtasks and as the constructs to be assessed in PISA-VET. Investigation skills refer to the ability of a vehicle technician to gather information about light vehicles in workshops, to draw conclusions from the information obtained and to justify the measures taken and the results achieved. Investigation skills include the ability to inspect and diagnose vehicles. Rectification skills are defined as the ability to establish or re-establish light vehicles to a desired condition, i.e., to remove damage, correct malfunctions, maintain functionality, meet manufacturer specifications or fulfil customer requirements. Rectification skills include the ability to (dis)assemble and/or replace, rebuild and/or restore, and adjust (parts of) light vehicles. The knowledge of automotive technicians includes conceptual knowledge of vehicles. The vehicle consists of the drive unit, power train, undercarriage, and assistance systems, although other subsystems can also be used for structuring. Based on this and on de Jong and Ferguson-Hessler (1996), PISA-VET should measure the factual knowledge (e.g., legal requirements), technical terms (e.g., resistances) and principles (e.g., functions of systems) that automotive technician learners have in four subsystem-related dimensions.

Altogether, automotive technicians need content knowledge (1) of technical terms and facts, such as legal requirements, requirements of vehicle manufacturers, types and signs of malfunctions, damage, and wear (e.g., visual appearance of corroded cables), most common designs, etc. for the respective system, (2) of scientific principles (e.g., principles of electronics, fluid mechanics, material science) relevant to the system in question, (3) of the structure and (4) of the functions of each system, the subsystems, components, and parts including their interdependences and connection types for the respective system (e.g., knowing the function of a fuel temperature sensor). Furthermore, automotive technicians need procedural knowledge to (1) obtain information from standardised sources (e.g., access mounting positions or wiring diagrams using repair manuals or digital expert systems), (2) operate measuring instruments (e.g., multimeter, brake tester, wheel aligning system), (3) perform measurements (e.g., measuring voltage on the pins of a connector, measuring the freezing point of engine coolant), (4) make adjustments (e.g., adjusting headlights, adjusting the amount of engine oil, tire balancing), and (5) restore or replace vehicle parts, components, or subsystems (e.g., carrying out a cable repair, replace the cabin filter).

For the assessment of automotive technicians’ skills and knowledge, the authors of the framework chapter emphasise the need to authentically and broadly represent the above constructs and contents. To assess investigation and rectification capabilities, technology-based authentic scenarios can be used. Technology-based assessments allow for an efficient assessment that is also feasible in international large-scale settings. There is evidence for the domain of automotive technicians that technology-based assessments can provide objective and reliable results in reasonable testing times (Abele et al., 2014), valid interpretations of these results (e.g., Gschwendtner et al., 2009), and objective process data which can be used to increase the informative value of assessments (Abele & von Davier, 2019). Altogether, this enables a comprehensive and accurate assessment of the set of knowledge and skills required by automotive technicians in today’s dynamic and technology-driven automotive industry.

Electrician

The rationale for the inclusion of the occupation of electrician in the Development Phase of the PISA-VET project is strong because of its continued economic importance and international comparability. Electric power is responsible for running every aspect of society including industry, business, entertainment, and domestic life. The role is in global demand and is dynamic – it is constantly affected by advances in technology and sustainability (United Nations, 2022). An equally important reason for selecting electrician for the International VET Assessment is that it is a highly regulated occupation with international standards that are universally recognised. In most countries, electrical installations comply with a set of regulations, issued by national authorities or by recognised private bodies. Globally, individuals undertaking initial training to be an electrician embark on a 3-to-4-year VET programme, which is broadly based and provides a strong foundation. The international programmes included in the Development Phase will be broadly aligned to Levels 3 and 4 of the European Qualifications Framework (EQF) and will have a wide range of titles.

The domain of electricians includes work processes of installing, commissioning, maintaining and adjusting electrical systems and components in accordance with regulations and professional standards which are critical for safe practice. The role is dynamic, continually responding to the advancement of technology and the need for sustainable practice. The environments in which the electrician works are far ranging. Following the feasibility study for a VET-LSA by Baethge and Arends (2009) and the WorldSkills International practice, electrician occupations can be divided into the work contexts of craft and industry, both of which are focused on in PISA-VET in a slightly different classification: The PISA-VET assessment evaluates learners’ performance in three contexts: the commercial and public sector, domestic and industrial. Electricians in the commercial and public sector often work in a busy environment with, for example, high voltage generators. This contrasts with domestic electricians who carry out tasks mostly inside residential flats/apartments, houses and bungalows. Industrial electricians work in large organisations which are characterised by complex equipment and heavy machinery. Electricians are disciplined to consistently follow a set of processes when approaching any task to ensure safe practice. In the above three contexts there is a focus on the following core processes: assessing/diagnosing, planning/designing, assembling/installing, repairing/maintaining, and inspecting, commissioning, testing and reporting. The diagnostic skills of electricians are critical for new tasks or for troubleshooting. Planning and design results in an electrical plan which includes specifications, instructions, drawings and diagrams. The assembly and installation process requires established industry safety procedures and manufacturers’ instructions to be followed. Repairing and maintaining electrical systems require the electrician to gather information to solve operating problems and undertake preventative procedures. Electrical inspection, commissioning, testing and reporting are critical final processes in ensuring that systems and equipment are safe, reliable, and functional.

The underlying capabilities for the electrician occupation that have been identified for PISA-VET combine occupation-specific professional skills and related employability skills:

  • Identifying and resolving electrical problems: the ability to identify, diagnose and resolve electrical problems in dynamic situations.

  • Responding to customers and colleagues: the ability to impart or exchange information with customers and colleagues, both verbally and “in writing”.

  • Collaborating on electrical tasks, problems, projects: the ability to work with other professionals on electrical projects and problems.

  • Using digital and communications technology for daily electrical tasks and problems: the ability to use digital technology, tools and networks within daily work routines.

  • Professional personal management skills for the electrician: the ability to regulate behaviours, thoughts, and emotions in the workplace in a productive way.

  • Critically assessing electrical situations: the ability to engage in reflection and independent reasoning, rigorously questioning ideas, and assumptions.

  • Safety consciousness in electrical situations: the ability to be constantly aware of hazards and to be alert to electrical dangers.

  • Reading electrical documentation and preparing reports: the ability to read manufacturer’s instructions, interpret electrical diagrams, write, speak and listen effectively.

  • Using mathematics and numeracy in daily electrical tasks and projects: the ability to deal with calculations and equations, e.g., Ohm’s Law, and trigonometry.

Regarding the assessment of the above capabilities, the experts propose a combination of online scenario-based questions of technical and practical knowledge, digital simulations of professional problem solving in the workplace, and live or recorded demonstrations of professional skills with expert judges. Electricians usually undertake practical sessions as part of their learning programmes and as part of their formative and summative assessments. Thus, there is a strong commitment to them as they reflect real work environments/situations. Generally, trainee electricians, within their national systems, undergo a live practical final test that is usually extensive and complex and, therefore, takes several hours to complete. One of the challenges in the development phase is to provide shorter assessments that still allow for a reliable and valid measurement.

Business and Administration

The rationale for including Business and Administration occupations during the Development Phase of the PISA-VET stems from the pivotal role of Business and Administration in a nation’s economic success, transcending sectors and regions (Dummert, 2021; Hoidn & Šťastný, 2021). Additionally, the global demand for trained commercial staff is evident, with commercial training being a preferred option in many countries. Notably, this occupational field stands at the forefront of significant global trends, such as automation (including artificial intelligence) and evolving work structures (including remote and asynchronous work). The feasibility of including Business and Administration occupations is underscored by the consistent performance of similar standard tasks by commercial staff internationally, adhering to universally recognised economic principles. Internationally, individuals pursuing initial training in Business and Administration typically enrol in a 2-to-5-year VET program. The international programs in the Development Phase will align with Level 4 of the European Qualifications Framework (EQF).

The Business and Administration domain encompasses a diverse range of activities centred around the management and organisation of business enterprises, resulting in a vast array of occupations within the domain. Specifically, occupations in Business and Administration span critical areas such as Accounting and Finance, Marketing and Sales, Human Resources, Operations and Strategic Management, and Information Technology. Following classic management theory, work activities can be classified along operational value chains (Porter, 1985; Rüegg-Stürm & Grand, 2020; Winther & Achtenhagen, 2009) as either management processes related to operational and managerial aspects of day-to-day operations, business processes concerning customer processes, production processes, and the distribution of goods, or support processes regarding fundamental business principles, effective business communication, and the preparation and presentation of relevant business information. More specifically and with regard to the operationalisation, the following relevant core tasks can be distinguished:

  1. 1.

    Working with (business) data: Encompassing tasks related to handling operational indicators, as well as identifying and presenting meaningful representations of data.

  2. 2.

    Working with and through communication: Focused on tasks involving effective correspondence and customer-facing skills.

  3. 3.

    Working in and for administrative projects: Encompassing occupational situations tied to operational project management and administrative work practices.

All of these tasks are characterised by the increasing trend towards digital working environments. This pertains to the utilisation of digital tools for tasks such as data preparation, communication, project administration, and project management. An increasingly digital work environment serves as a pivotal contextual factor for a multitude of professions within the Business and Administration domain.

In order to successfully perform such tasks, four steps in a work process are often differentiated (e.g., Rausch et al., 2021):

  1. 1.

    Identifying and understanding occupational information: This process addresses the search for workplace-related information and the recognition of its relevance. Information in the business domain is mainly presented in text form, such as emails from customers or supervisors, business letters, invoices or presentations from ERP systems.

  2. 2.

    Analysing occupational information and situations: This process describes the ability to analyse information and recognise relationships between different professional contexts in relation to a specific problem situation.

  3. 3.

    Applying occupational knowledge and skills: This process focuses on acting in professional situations by applying professional conceptual knowledge in conjunction with professional action knowledge.

  4. 4.

    Evaluating and reflecting on occupational issues: This process refers to the use of knowledge that goes beyond a specific professional situation, including reflection on one’s own professional behaviour during the work process.

To successfully navigate these occupational processes, VET learners need a set of domain-related skills (including general numeracy and literacy) and domain-specific knowledge (such as definitions, rules, and practice algorithms) (e.g., Deutscher & Winther, 2019; Gelman & Greeno, 1989; Winther, 2010). Regarding the areas of skills and knowledge, the following content categories are suggested:

  1. (1)

    Skills and knowledge to identify relevant operational indicators in business administration and to interpret and present data appropriately (see ‘working with business data’)

  2. (2)

    Skills and knowledge to apply methods, manage projects and implement effective work strategies (see ‘working in and for administrative projects’)

  3. (3)

    Skills and knowledge to appropriately correspond and communicate, also in cross-cultural contexts, and to deal with diverse customers and stakeholders (see ‘working with and through communication’)

For the assessment in the Business and Administration domain, the chapter authors highlight two approaches: (1) digital simulations of professional problem solving in the workplace and (2) scenario-based questions of technical and practical knowledge. Simulations in the business domain usually feature comprehensive introductions to company structures, including historical background, and graphical overviews of the company’s business situation. Furthermore, office simulations usually provide relevant information, business data, embedded office software, and digital communication tools. Embedded videos of authentic professional contexts can enhance the authenticity. Scenario-based questions serve as a supplementary tool, ensuring that implicit knowledge utilized during problem-solving in simulations can be explicitly addressed (e.g., Achtenhagen & Winther, 2014).

Health Care

The rationale for the inclusion of the healthcare occupation is strong considering its high economic and societal importance and its continued relevance in ageing societies as highlighted as the third UN Sustainable Development Goal: ‘Ensure healthy lives and promote well-being for all at all ages’ (United Nations, 2022). The core purpose of any roles in healthcare/nursing assistance around the world is to help meet the health and care needs of clients or patients, respecting their personal health needs, circumstances, and rights. Beyond this, there is considerable diversity in the scope and limits of these roles, and the preparation of young people and adults to fulfil them. Each nation’s and region’s healthcare system is a historical, cultural, social, and economic artefact, as diverse as the VET arrangements designed to serve it (Bjornavold & Chakroun, 2017). Hence, PISA-VET focuses on the competencies that are (a) universally recognised as “core” to the occupation, (b) amenable to interpretation, assessment, and evaluation at several levels, and (c) culturally neutral.

The healthcare occupational area comprises many different types of occupations at various levels of qualification requirements. Given that the focus of the PISA-VET Assessment is on VET programmes at ISCED levels 3 to 5, healthcare occupations that typically require bachelor’s, master’s or doctorate qualifications are excluded, and the focus is instead on medium-skill healthcare occupations. Based on international occupational classifications, there appears to be a broader role, referred to as healthcare assistant, and the narrower role of nursing assistant, which may be more specifically focused on nursing rather than nursing and caring. Since both roles sit within the pathway to registered nursing occupations, and in some instances nursing assistant is regarded as a subset of healthcare assistant, it is appropriate to embrace both within the definition of the occupation. As a result, the term ‘healthcare/nursing assistant’ is used to describe the occupational area within PISA-VET.

Healthcare/nursing assistants are defined as providing direct personal care and assistance with activities of daily living for patients and residents in a variety of health care settings and generally work under the direct supervision of medical, nursing or other health professionals or associate professionals. Home-based personal care workers provide routine personal care and assistance with activities of daily living for persons who need such care due to effects of ageing, illness, injury, or other physical or mental condition. For international comparative assessment, while care of the elderly is generally seen as a core part of the role, care of children lies outside this role’s compass. While there is a top-level distinction between a hospital and a home-like environment, in practice, there is a complex range of settings, the number and purpose of each depending on each nation’s and region’s healthcare system and history. For instance, in a country like Australia, what healthcare workers do varies widely depending on their geographical location. A nurse or nursing assistant in a large tertiary hospital in a metropolitan centre undertakes tasks that are quite distinct from those in regional base hospitals, and are distinct from those in small community hospitals often huge distances away from larger facilities. Legal, regulatory, and administrative requirements are a constant feature of healthcare assistance. Healthcare is a regulated profession at all levels, including intermediate roles. This feeds into its administration, which is the foreground of regulation. Regulation normally includes a framework for audit and quality: compliance, acceptable practice, and improvement.

Five core work processes have been identified that meet the criteria for international comparison, are balanced in their significance to the role and, when taken together, require all the capabilities sought after for initial vocational education and training to equip individuals for roles as healthcare/nursing assistants. The five occupational processes are: (1) work in professional teams, (2) identify needs and collaborate in healthcare planning, (3) support and enhance clients’ quality of life, (4) provide and support treatment and medical processes, as well as (5) review and evaluate care.

Fulfilling the role of a healthcare/nursing assistant and handling the above work processes requires a blend of social, emotional, physical, mental, and intellectual maturity. Some of these underlying capabilities resemble the OECD’s Survey for Social and Emotional Skills (SSES; see Table 1), such as collaboration with others and management of self, time, tasks, and workspace in the context of healthcare/nursing. Furthermore, access to and use of evidence and information to support healthcare solutions is required. At the heart of PISA-VET is the acknowledgement that an understanding of healthcare content, and the ability to apply that knowledge to interpret meaningful situations in occupational contexts and to solve contextualised problems is fundamental for success in the healthcare occupation. The five content categories of these skills and knowledge are (1) multi-professional collaboration, (2) communication and relationship building, (3) medical and related sciences, (4) client/patient care, service, and assistance, as well as (5) administrative and legal frameworks.

Regarding assessment, the experts emphasise that for national purposes, summative assessment for VET generally comprises live tests of practice plus tests of knowledge and theory. Thus, it covers actual performance while indicating the extent to which that performance is underpinned, generalised, and built upon through disciplinary, procedural, conditional, and epistemic knowledge and understanding. The case for live performance assessment within the mix of assessment types is strong, and particularly so for testing procedural knowledge. The scope of live performance testing for international VET assessment should be explored during the development phase of PISA-VET along with immersive scenarios and multiple choice questions.

Tourism and Hospitality

In the last six decades, the tourism sector has become one of the most dynamic and fastest-growing economic sectors in the world, being considered a driving force for job creation and for local economic development in many countries (OECD, 2018c). The Tourism and Hospitality occupational area contains many different occupations. To streamline the scope for the International VET Assessment, a decision was made to concentrate on a singular occupation. Following discussions with experts as well as a review of available evidence and resources, the choice was made to focus on the occupation of hotel receptionist. Various reasons underly this selection, including its international comparability, gender balance for the assessment (70% of hotel receptionists globally are female) and its importance in the overall tourism and hospitality area. The competencies needed to perform hotel receptionist tasks are typically acquired in 1-to-3-year VET programmes at ISCED Levels 3–4 with learning outcomes at EQF level 4, or equivalent.

The role of Hotel Receptionist is key to every hotel establishment. The hotel receptionist receives and serves different guests and other clients and individuals, with technical competence and knowledge, autonomy, creativity, and as an individual and as part of a team. The hotel receptionist receives and welcomes and communicates with different guests and clients through various communication channels; carries out reception procedures to promote guest and client satisfaction; and ensures quality and excellence in service. According to their relevance in the role of the hotel receptionist, four main work processes were identified: (1) check-in and check-out, (2) reservations, (3) management of complaints, and (4) touristic information.

The fundamental hotel receptionist capabilities to master the above work processes include:

  • Hospitality service orientation,

  • Interpersonal and communication skills,

  • Collaboration with other hotel staff

  • Problem Solving in the context of hotel reception

  • Task performance in the context of hotel reception

  • Critical thinking in hospitality service

  • Literacy in hospitality service

  • ICT Literacy for hotel reception

Furthermore, domain-specific content knowledge is vital for success in this role, as it enables problem-solving and contextual interpretation. The hotel receptionist domain is structured into specific content categories to ensure comprehensive coverage across various aspects of the occupation. This organisation emphasises crucial tasks and processes while detailing the specific knowledge associated with each content category selected for the assessment. The five categories of content knowledge are: hotel reservation procedures, checking-in procedures, administration and back-office procedures, checking-out procedures, and promotion of the local area.

According to the tourism and hospitality expert group, the knowledge and competences of hotel receptionist students should be assessed according to the above processes, skills, content and context. To ensure valid outcomes, the assessment must authentically represent the scope of hotel receptionist skills and knowledge. The assessment covers contextual aspects, content topics, and operational processes. It aligns with occupation-specific standards, emphasising not just performance but also the ability to justify actions and outcomes. Regarding knowledge, the assessment should encompass both strategic and procedural knowledge essential for executing these work processes. It also incorporates conceptual knowledge of the content topics necessary to comprehend, apply, and rationalise actions and results. Authentic scenarios using technology can effectively evaluate various hotel receptionist competencies. These scenarios simulate real tasks, environments, and guest interactions, replicating the hotel receptionist’s workplace realistically, like handling a guest complaint about service quality or accommodation. Live or video-recorded demonstrations could supplement these technology-based simulations, enhancing value if they fit within logistical and budgetary constraints. Additionally, traditional knowledge-based tests remain relevant for assessing certain elements of hotel receptionist competencies.

Employability Skills

Employability skills are defined as a set of cognitive and non-cognitive competencies that are relevant across occupational areas and are seen as a prerequisite for successfully participating in modern labour markets and societies (OECD, 2015). Similar concepts have been labelled as foundational skills, transversal skills, transferable skills, cross-domain skills, generic skills, core skills, key competencies, soft skills, and 21st-century skills. These skills have received increasing attention within several OECD frameworks (OECD, 2019a, b, 2021a) and have been addressed as problem solving, critical thinking, literacy, numeracy, information and communication technology (ICT) literacy, communication, collaboration, task performance, emotional regulation, open-mindedness, to name a few. In essence, these skills enable individuals to adapt to new technologies and rapidly evolving work environments (OECD, 2019b), prepare them for continuous learning and enable them to responsibly meet future, possibly unpredicted, challenges in the workplace.

Due to the breadth of employability skills and the limited time available for the assessment, difficult decisions had to be made as to which of the above-mentioned dimensions of employability skills to include. The final decision was made in favour of Literacy, Problem Solving, Task Performance, and Collaboration (see Table 2). 1) Literacy was included because it is regarded as a necessary skill in occupational areas where written communication plays an important role. Literate individuals can make use of a broad spectrum of written materials in the service of a wide range of activities and are knowledgeable of the standards of their communities of practice. Literacy skills also enable individuals to adapt to constant changes in the workplace and, hence, to develop additional skills within his or her occupational area. 2) Problem-solving was included because, in modern societies and labour markets exposed to several transformation processes, problem-solving is considered crucial for adapting to new circumstances and turning knowledge into action. Furthermore, problem-solving is one of the skills that are least exposed to automation (Lassébie & Quintini, 2022). Like literacy, problem-solving also serves as the basis for developing additional occupation-specific skills. 3) Task Performance was included because being self-controlled, being perceived as responsible by others, persistent and motivated is highly relevant for job performance and pro-social behaviour, particularly in work settings discussed under the label of ‘New Work’ which are characterised by autonomy, flat hierarchies and personal responsibility. 4) Collaboration was included because working collaboratively with colleagues and clients or customers is assumed to be increasingly important in future workplaces. Empathy, trust, and co-operation are linked to how individuals relate to others and how they solve problems collaboratively and hence, also correspond with job performance (OECD, 2017a, b; Sackett & Walmsley, 2014). To not include numeracy and ICT literacy as skills to be assessed in PISA-VET was the subject of controversial debate among experts and interest groups. It is argued that these skills are addressed in their domain-specific manifestation within the assessment of professional skills in the occupational areas. However, time constraints can be seen as the main reason for the limitation to four skills and the selection of these skills was a difficult process. Overall, the four selected skills were mostly rated as highly relevant by experts from the occupational areas (OECD, 2024, p. 196).

Another controversial discussion concerned the question of whether employability skills should be regarded as domain-general (i.e., independent of occupation-specific knowledge and competencies) or domain-embedded (i.e., including occupation-specific knowledge and competencies). The advantages of defining and assessing employability skills in line with a domain-general approach lie in the possibility of using already existing instruments and the comparability of results across occupational areas. However, it is argued that literacy, problem-solving, task performance and collaboration are, to some extent, modulated differently across occupational fields. For instance, the requirements for understanding and interpreting an error report as an automotive technician differ clearly from those for understanding and interpreting a patient’s medical record as technical language is domain-specific. These and further arguments (see OECD, 2024, p. 198) lead to the suggestion of considering both approaches by including a generic assessment and an occupationally-embedded assessment of employability skills.

Overview of the Assessment Framework

Altogether, the assessment framework suggests three pillars: 1) A domain-general assessment of employability skills, 2) a domain-linked assessment of employability skills for each occupational area and 3) a domain-specific assessment of professional skills in each occupational area (see Fig. 5). The assessment of professional skills in the occupational areas is presented generically. The content of the scenarios should be based on the identified work processes (see column 3 in Table 1) and the skills to be measured should be based on the defined capabilities (see column 4 in Table 1). For instance, in the occupational area of healthcare, there might be a scenario that includes supporting a client’s quality of life, which requires knowledge of client care, service, and assistance as well as skills for building a supportive relationship. Another scenario might focus on developing a long-term care plan, which also requires medical knowledge but in addition, skills and knowledge for multi-professional collaboration.

Fig. 5
figure 5

Three pillars of assessments in PISA-VET

Full size image

From a conceptual perspective, the three pillars might rather be considered a continuum. From an assessment perspective, the foundational core of the employability skills should be measured separately using existing OECD instruments. However, the assessments of occupationally modulated employability skills (pillar 2) and the assessments of professional skills (pillar 3) might overlap if some of the authentic scenarios are utilised to measure constructs from both pillars.

Challenges and Promises: Towards a Research Agenda

Effective VET systems are the key to developing the skills that are needed in today’s labour markets and societies (OECD, 2023d). The integration of PISA-VET into the current research landscape marks a pivotal stride towards enhancing our understanding of VET across the globe and in a way that allows for international benchmarking of VET systems. This first-of-a-kind initiative by the OECD opens new avenues for empirical research, offering insights into the effectiveness of VET programs and their impact on learner outcomes. It aims not only to benchmark skill levels and programme effectiveness but also to generate policy insights that can guide the understanding and development of VET systems worldwide. In the public debate, however, very negative positions are also to be expected, according to which standardised tests are not very suitable for capturing what is considered effective in heterogeneous practical contexts.

The overarching goal of PISA-VET is to provide an empirical basis for reforms and evidence-based decision making that align VET programs more closely with the demands of the labour market and societal needs in the 21st century. In March 2024, the OECD published the “Draft Assessment and Analytical Framework” that is intended to guide further instrument development, help policy makers understand the purpose and expected results of PISA-VET, as well as inform researchers and educators about methodological choices. In this paper, members of the various expert groups who were intimately involved in the framework development delve into the potential benefits, challenges, and the research agenda that emerge from this innovative approach. Of course, even the selection of members could be considered biased in the scientific community, as they can predominantly be assigned to the empirical paradigm. Yet, these members are aware that much of what is proposed here is debateable and will hopefully promote a fruitful discussion.

Emerging Avenues for Research and Policy Insights

PISA-VET enables analysis and comparison of skill levels among learners across participating countries within specific VET programs and program types, as well as the discernment of some differences between learners with distinct characteristics. Additionally, this approach can identify distinct or similar features of VET programs in selected occupational areas across and within countries, allowing for an examination of how these features are related to program outcomes. Employability skills measured across occupations also provide some level of empirical linkage between occupations both within and across countries. Methodologically, the data offers a unique opportunity to develop and evaluate assessment approaches to occupational skills, to compare them and related constructs across countries, and to explore relations to contextual characteristics and system-level designs of VET programmes. In practical terms, it can indicate effective characteristics and measures on various levels, though given the cross-sectional overall design, causal interpretations should be treated with caution. In particular, the cross-sectional nature of the data will make it difficult to derive firm conclusions of effective and efficient characteristics of VET-programs. Importantly, by providing a first robust empirical basis for comparing programme outcomes, it encourages a shift towards evidence-based policy-making in vocational education and thus builds on the legacy of earlier PISA initiatives. National add-on studies that look more closely at national specifics, regional variations and particular contexts or that follow-up on specific samples in a longitudinal manner can further contribute to a full exploitation of PISA-VET. In fact, national add-ons have been implemented for PISA in many countries and help to highlight certain, nationally-relevant aspects that might be beyond the international parts of the assessment. Furthermore, such add-on studies could also take a closer look at the limitations and possible further developments of the assessment approaches.

Challenges and Methodological Issues

An important point of discussion is, of course, the selection of the occupational areas, their scope and the identified work processes. Across the chapters of the PISA-VET framework, the five occupational areas are consistently characterised by global economic and societal importance, universal relevance across countries, alignment with technological and sustainability advances, international comparability of skills and standards, and adaptability and cultural neutrality. However, the discussion may also be biased by the particularities of participating countries that were more involved in this early phase. Furthermore, regarding the comparability of skills and standards, the occupational areas differ and the expert groups took different routes to deal with incompatibilities. In Business and Administration, for instance, the scope is quite broadly targeted at a set of occupations and so are the skills to be measured. In contrast, Tourism and Hospitality focuses on hotel receptionists and, accordingly, the skills are more specific. Both approaches are accompanied by advantages and disadvantages, but, as a general rule, international assessments that are designed to be relevant across numerous contexts always take a more generic approach that has to disregard some specific aspects relevant to national VET-programs. As mentioned above, national and regional add-on studies can redress this shortcoming and provide an evidence-based link between the international and the national components of the assessment.

A related discussion revolves around the assessment approaches employed in PISA-VET. The initiative’s reliance on scenario-based questions, simulation-based assessments, and live demonstrations represents a comprehensive effort to measure professional skills authentically and from various perspectives. However, these methods bring forth challenges in terms of reliability, validity, and the feasibility of large-scale administration. The trade-offs between the breadth of construct coverage and the authenticity of performance tasks as well as test time and cost restrictions underscore the need to carefully consider the various assessment approaches during the development phase. Booklet designs within a rotating block design for each occupational area are proposed to assure both an adequate construct coverage and a feasible test time. Table 4 summarises the characteristics of the proposed assessment approaches and presents rough evaluations made by the authors of this paper to initiate further discussions in later stages of the test development.

Table 4 Overview of the characteristics of the proposed assessment approaches
Full size table

It is evident that all approaches have their advantages and disadvantages and therefore, in principle, all three approaches are pursued in all occupational areas in the Development Phase. Furthermore, it should be emphasised that the preferences for or against certain approaches also differ within the expert groups.

Outlook and Need for Add-on Studies

This paper sets out some of the key elements of OECD’s ‘PISA-VET Assessment and Analytical Framework’ published in March 2024. It recaptures the milestones that have thus far been achieved in the Development Phase by identifying occupational areas, the skills to be assessed within those areas, and more generic employability skills. The Development Phase will continue until 2025 and include a methodological study comparing the reliability, validity, and costs of the instruments developed for assessing the five selected occupational areas and conclude with a second, more comprehensive document to recruit further countries to participate in the Pilot Phase. The Pilot Phase will run from 2025 to 2029 and include further development and validation of the assessment instruments, field trials and a main study resulting in an international report on the feasibility, manageability, affordability, and limitations. The Large-scale Implementation Phase is planned from 2029 to 2032. It is then planned to repeat assessments in occupational areas every few years and to recruit additional countries over time.

Like in PISA and PIAAC, further research questions that are not directly addressed in PISA-VET can be addressed in complementary studies on international and national levels. In PISA-VET, this strategy is even more important to pay tribute to the numerous national VET specialities, since VET systems differ greatly across countries and contain highly diverse target populations with different needs and backgrounds. Furthermore, additional studies can investigate the longitudinal outcomes of VET alumni, assessing long-term career progression and satisfaction. Finally, carefully designed intervention studies provide information on the effects of targeted measures and policy-making to enhance VET outcomes, since studies on instructional sensitivity have hardly been established in VET (see Wittmann et al., 2022, for the nursing domain).

Altogether, PISA-VET and respective add-on studies embody the potential for an “empirical revolution” in the understanding of VET systems with the possibility of evidence-based policy-making and international benchmarking within reach. By providing an empirical basis for comparing design features and program outcomes, it encourages a shift towards evidence-based policy-making in vocational education - a dearly needed novelty in most of the participating countries. At the same time, the PISA-VET initiative and its results will likely lead to controversial discussions and need for country-specific add-on projects and adjustments. The aim of this article is to bring these important discussions to the scientific community at an early stage of the initiative and to spark a scientific debate that has the unique chance of shaping VET across the globe.