Introduction

How, ‘in our messy, fuzzy, anarchic field of practice … can we produce neatly packaged bundles of evidence that might be useful to busy policymakers?’ (Field, 2015). In a society dominated by information technologies, and in which data plays an increasingly pervasive role in economic, social, cultural and political life, it seems natural that we should seek computer-aided support in making intelligent decisions by which to improve policy. To this end, the European Commission encouraged the creation of a prototype Intelligent Decision Support System (IDSS) to support policy-making in lifelong learning. As developed within the Enliven project, this had a particular focus on young people Not in Employment, Education or Training (NEETs). The aim was to provide a repository for existing programmes to enable anyone interested in offering programmes for young adults—particularly those from groups disadvantaged in terms of gender, ethnicity, culture and other factors—to examine what measures have worked, to identify what types of actions have previously been employed and to enable their assessment against suitable criteria. The work involved close collaboration between educational researchers and computer scientists and was enriched through the interaction of approaches from social and computer sciences. This allowed for examination of the key challenges and limitations of both disciplines in addressing one of the most intractable, or ‘wicked’, social problems of our times—one which repeatedly defies neat packaging into ‘bundles of evidence’.

Whilst we were clear on the ‘problem’ per se, understanding and characterising it, as well as piloting a computer-aided system which might create a mechanism for genuine policy adaptation, was deeply complex. From the beginning, creating a common discourse between two very different disciplines represented problems of ontology and epistemology. We needed to agree how we determined the important attributes or characteristics of ‘being NEET’, given the complex range of variables—age, location and geography, social and economic characteristics—across the Enliven project’s ten countries. From a computer science perspective, it became important to show the relationship between any defining characteristics and to determine a set of concepts and categories that represented the subject. From a social science perspective, it was vital to explore the factors which characterise this group of young people and the social constraints and barriers they face. We also sought to recognise how we would know whether the IDSS had worked, from the perspectives of practitioners, policy-makers and young people, how it would be evaluated and how we could demonstrate its impact in the longer term. This created an important phenomenological focus for social science partners. We attempted to gain access to young people’s own perceptions of what worked for them particularly through feedback from members of a Youth Panel.

The research aimed to uncover barriers to unemployment faced by young people furthest from the labour market and the programmes used to respond effectively to these issues. To add depth and nuance, and to gain access to more detailed evaluation material and data local to the research team, we evaluated ‘Young and Successful’, an employment support service for young people who had been unemployed for 12 months or more, based in Derbyshire and Nottinghamshire (two counties in England’s East Midlands region). The input of youth practitioners and youth representatives from this programme greatly enriched the research and gave access to practice-based knowledge.

This inter-disciplinary and inter-sectoral work proved groundbreaking and allowed for praxis: ‘reflection and action upon the world in order to transform it’ (Freire, 2000, p. 51). Our practitioner focus enabled us to examine the coming together of research paradigms. We discovered how civil society practitioners, social scientists and computational social scientists can work to share expertise, create greater knowledge democracy and address the ontological differences between data expectations among policy-makers and target communities. Such differences might include, for instance, policy-makers’ understanding of the reality of searching for work in areas where appropriate work is scarce, or a failure to adequately recognise the importance of prior work experience, or the importance of having mentors (and financial support) to overcome certain apparently small barriers (like access to buses and transport, or the right clothes) which can become insurmountable for vulnerable young people.

This chapter reflects on the exercise of collaborative cross-disciplinary and inter-sectoral working, as well as describing how the Enliven IDSS developed and giving insight into the lessons learned during the stages of its development. It offers reflections on both the limitations of the information available and the constraints of a computer-based response to a complex social problem. It also considers the instruments necessary to facilitate effective policy-making in lifelong learning. We have sought to explore the ontological differences between data expectations among policy-makers and among young people themselves, and how data can serve knowledge creation for those most excluded in society.

Policy Focus: NEETs

Across the world, NEET young people are considered to face particular barriers including: a lack of work experience, poor qualifications, heightened employer uncertainty, and—by some policymakers—considered to represent certain negative typologies (e.g. poor work, lazy, quitters). (Mawn et al., 2017, p. 2)

According to Eurostat, in 2018, around one in six (16.5%) of young people aged 20–34 in the European Union (EU) were defined as Not in Employment Education or Training (NEET), corresponding to approximately 15 million young people (Eurostat, 2018). The term NEET has risen to prominence in recent policy debate due to the disproportionate impact of the post-2008 recession on young people. The unemployment rate for those aged under 30 is nearly double the average (Eurofound, 2016); long-term scarring and disengagement are known to result. This population is very diverse, encompassing groups of more privileged young people who voluntarily become NEET whilst waiting for a particular opportunity or attempting to pursue alternative careers, as well as unqualified early school-leavers and those taking on family caring responsibilities (Mascherini, 2018). In 2018, the NEET rate in the EU for young people (aged 20–34) with a lower level of education was 37.2%, compared with 14.7% for those with an intermediate level of education and 9.5% for those with a high level of education. People with a lower level of education were thus almost four times less likely to be in employment, education or training than those with a higher level (Eurostat, 2018).

Young people defined as NEETs often face a complex interplay of social, economic, political and psychological factors, and these can present additional barriers in terms of their visibility (socially and economically), the opportunities presented to them, and the outcomes they achieve in the labour and employment markets. For many such young people, individual factors—physical (e.g. sickness, disability) and emotional/psychological (e.g. poor mental health, dependence)—intersect with problematic material circumstances (e.g. poverty, homelessness, inadequate or uncertain access to health care or education) and social context (lack of support from their family or peer group, absence of guidance in difficult situations, and immediate risks from the physical environment). A substantial body of research has also shown that the ‘post-industrial’ restructuring of the economy—flexibility, the ‘gig economy’ and so on—is generating a Europe-wide ‘precariat’ (Standing, 2011) for whom insecure jobs, work with little formal or informal training and no annual or sick leave or superannuation benefits are the norm. Taking the UK alone, recent statistics show that young people aged 16–24 may be disproportionately disadvantaged by working practices: a third of people on zero-hour contracts are in the 16–24 age range, compared to 12% for all people in employment (Office for National Statistics, 2018).

People working in such precarious conditions are likely to be more vulnerable to, and less resilient against, deteriorations in both economic and personal circumstances, such as changes in their health. If all of our energy is focused on meeting basic needs for safety and security, or ‘deficit needs’, the individual is subject to vulnerability factors which make personal development virtually impossible. The search for education or training, or even for satisfying work, can seem remote from lives caught up in dealing with the immediate exigencies of poverty, homelessness, caring responsibilities or seeking physical safety. Instead, in such situations, we are more likely to be focused on the need for adequate food, housing, clothing and stable work—which the Universal Declaration of Human Rights defines as fundamental human rights (Article 25). This is clearly a major inequality issue to which policy-makers at European level and within the EU’s member states have responded in various ways.

Computer-Aided Decision Support and Evidence-Based Policy

Decision Support Systems are information systems whose function is to improve the effectiveness and efficiency of decision processes (Pretorius, 2017). An Intelligent Decision Support System (IDSS) system ‘uses artificial intelligence, machine learning, taught algorithms and data analytics to help support decision-making in real-time, by setting out possible courses of action and evaluating the likely results of these proposed actions’ (Field, 2005). The intention is that policies will be based on evidence: ‘evidence-based policy’ has been defined as an approach that ‘helps people make well informed decisions about policies, programmes and projects by putting the best available evidence from research at the heart of policy development and implementation’ (Davies, 2004).

The aim of the IDSS has been to provide a repository, or case base, for existing programmes across Enliven’s nine European countries—along with a comparator (Australia)—to enable policy-makers and practitioners interested in offering programmes for young adults from disadvantaged backgrounds, and furthest from education, training and the labour market, to identify types of actions that have been previously employed, to assess their suitability against specific criteria and to examine how similar programmes and measures have worked. The IDSS enables the sharing of information on policies and practices. It is based on the understanding that learning from and about the experience of others through collected project data can enhance both the delivery of more efficient policies and the discovery of better solutions for specific policy problems. For this, we have drawn on findings from research conducted by European and international agencies and research projects, as well as from Enliven itself. We hoped the IDSS would enable policy-makers at EU, national and organisational levels to enhance the provision and take-up of learning opportunities for adults, leading to a more productive and innovative workforce and reducing social exclusion.

In terms of theoretical positioning, we brought together methodologies and theorisations from artificial intelligence (Case-Based Reasoning) and adult learning (bounded agency) to develop and evaluate an Intelligent Decision Support System (IDSS) to provide a new and more scientific underpinning for policy debate and decision-making about adult learning, especially for young adults. Case-Based Reasoning (CBR) is a knowledge-based methodology where previously solved cases can be retrieved from a case base by using a similarity measure. Compared with traditional knowledge-based methodologies, which return solutions to user enquiries only where an exact match is available, CBR seeks to retrieve cases similar to the enquiry case: if a direct project match is not available, similar projects will be suggested.

Enliven employed the theory of bounded agency to point up structural vulnerabilities and inequalities. Bounded agency recognises the complex interplay between personal or individual motivation and the broader structural and cultural conditions of a person’s life—specifically their institutional and labour market settings, and the social support available to them—and argues that such factors are vital in shaping adults’ decisions to engage in learning or education:

People living in specially disadvantaged circumstances are less likely to engage in lifelong learning, in part because they lack the financial resources to fund their studies and believe that there will be few economic benefits. In addition, their life experiences may have reinforced a sense of powerlessness and inability to control risk. (Róbert, 2012, p. 88)

Structural vulnerability results from unequal treatment by society in relation to a person’s gender, ethnicity, job type or social status. While young people are resourceful and often develop imaginative solutions to create fulfilling lives under trying circumstances, there is clear evidence (Hagell et al., 2018) that their lack of control over key aspects of their lives results in anxiety and stress and impacts on their psychological health: ‘Young people who are NEET are considered to be at greater risk of poor physical and mental health, being unemployed, and having low quality and low wage work in later life’ (Allen, 2014). There is also evidence that policies that respect and facilitate personal choice and autonomy—such as the UK-based Activity Agreement Pilots between 2006 and 2011 (Maguire et al., 2011), which trialled individualised approaches to re-engaging young people defined as NEET—avoid the temptation to penalise young people who are unsuccessfully negotiating difficult circumstances and help to increase their sense of control and well-being.

Part of the normative nature of capitalism is its capacity to confer on the individual the responsibility for lack of confidence, skills or qualifications, and for levels of work experience inadequate for the job market. This individualisation promotes a myth of autonomy at odds with structures and systems dominated by ‘competitive, self-interested individuals vying for their own material and ideological gain’ (Giroux, 2004, p. 106) and without a real choice of genuine ‘alternative activities’. It leads to ‘agency inequalities’ (Antonucci et al., 2014, p. 21): failing to manage risk is a matter of individuals’ responsibility; the real barriers that young people face are minimised.

Computer-aided policy-making methods have only recently started to be applied in lifelong learning. Little if any information has become available to assess its effectiveness. Wyatt (2017) provided a review of software to support computer-aided policy-making. An earlier review by Vennix (1990) noted that ‘It is intriguing to observe that on the one hand computer models for policy support abound in policy making settings, while on the other hand their actual impact on policy making is considered to be limited’. Vennix found that modellers made three kinds of recommendations:

  1. 1.

    Build small models rather than complex, detailed ones

  2. 2.

    Involve the client in the policy modelling efforts as much as possible

  3. 3.

    For wicked problems, models are best used as communication devices to gain insight rather than for prediction.

Wicked Social Problems

Whilst building the IDSS we realised that ‘being NEET’ is a ‘wicked’ social problem. Wicked problems are complex, intractable, open-ended and unpredictable (Alford & Head, 2017). From a computer science perspective, there are no overall solutions to such problems, although they can be mitigated. Examples of wicked problems are design planning (Rittel & Webber, 1973), social problems such as obesity (Finegood et al., 2010), and disadvantages faced by indigenous people (Australian Public Service Commission, 2007). The NEET phenomenon reflects key characteristics of wicked problems: it is difficult to define; it has multiple explanations and no single cause; it has no clear solution—even a combination of multiple solutions may be inadequate; there is no unambiguous indicator of when it has been resolved; addressing it very likely requires partnership arrangements or alliances between different agencies.

This identification of NEETS as a wicked social problem is critical in considering how intelligent information system tools can be utilised to aid policymakers focusing on this challenge. Such tools can assist policy-makers to locate information readily and enable differing opinions to be viewed and compared, so they can make informed choices. However, the design of any tool applied to the domain of NEETs must allow for the features of wicked problems—such as an absence of consensus on what the problem is—if it is to be of use. This section will describe the features of wicked problems which need to be considered when creating a decision support tool for developing programmes for NEETs and how these features impact on its design. Features of wicked problems relevant to the design of an intelligent systems tool are:

  1. 1.

    ‘There is no definitive formulation of a wicked problem’ (Rittel & Webber, 1973). This implies that a tool must provide for the views of multiple stakeholders.

  2. 2.

    ‘Wicked problems are often not stable’ (Australian Public Service Commission, 2007). The problem may evolve as policies are implemented.

  3. 3.

    ‘Wicked problems do not have an enumerable (or an exhaustively describable) set of potential solutions’ (Rittel & Webber, 1973). A tool must therefore be able to support multiple solutions and to present information in ways easily comprehensible by end users.

  4. 4.

    ‘Wicked problems are socially complex’, involving coordinated action by a range of stakeholders; they hardly ever sit conveniently within the responsibility of any one organisation (Australian Public Service Commission, 2007). This means an intelligent systems tool must be able to manage information across multiple organisations—which may vary considerably in their information technology capabilities and corporate data policies. A tool should also employ data structures which recognise social complexity and allow for fuzziness and uncertainty.

  5. 5.

    ‘Attempts to address wicked problems often lead to unforeseen consequences’ (Australian Public Service Commission, 2007). For example, finding jobs for one social group—young unemployed people, for instance—may result in other groups (perhaps migrants) encountering difficulties in finding employment. A tool must be able to present a balanced view of the effects of approaches to problems.

  6. 6.

    ‘Every wicked problem can be considered to be a symptom of another problem’ (Rittel & Webber, 1973). Wicked problems do not exist in isolation; they are related to other wicked problems.

The last feature requires a tool to be able to provide users with information on programmes considering related social issues. Causes of unemployment are diverse and complex:

Many of the barriers to employment are well known: illiteracy and innumeracy, and poor general educational attainment; weak employment history; contact with the criminal justice system; physical and mental ill-health and disabilities; alcohol and substance misuse, and more general indicators of a chaotic lifestyle; housing problems and homelessness; and long-term caring responsibilities. (House of Commons Work and Pensions Committee, 2016)

Practitioners designing a programme for NEETs need details of local and national programmes and funding relating to education, housing, health and other dimensions of social policy. This enables them to consider how to take advantage of, enhance, inform programme participants about, or identify gaps in, existing provision. Which features or methods could be applied to the NEET domain requires learning from other complex social problems and identifying intelligent support tools utilised within those problem areas.

NEETs and the IDSS

The definition of NEET agreed by the European Commission’s Employment Committee (EMCO) refers to young people aged 15–24 years who are unemployed or inactive and not attending any education or training courses (EurWork, 2013). The definitions of unemployed and inactive follow the International Labour Organization (2019): unemployed as being without a job, actively seeking work in the past four weeks, and available to start work in the next two weeks; an economically inactive person is someone outside the labour force. The definition of NEET was later broadened to include those aged 15–29 years (Eurofound, 2016).

However, programmes incorporated within the IDSS relate to young people aged 15–35 years. They include those whose needs or circumstances may prevent them from starting work or education immediately; they also include school children at risk of becoming NEET. Due to a lack of suitable programme data (discussed below), it was necessary to expand the definition of NEETs in order to locate sufficient information to populate the IDSS; the focus on vulnerability was maintained, however. The term NEET goes beyond unemployment in that it captures all unemployed or inactive young people who are not accumulating human capital through formal channels (Mascherini, 2018).

Key Challenges

Limitations of Information Available

In CBR systems, not all attributes can contribute to retrieving the cases most relevant or useful for informing decision making. In the Enliven IDSS, therefore, only key attributes which can effectively measure the relevance between cases in terms of policy-making were used in calculating the similarity between cases and thus to retrieve cases. The creation of the IDSS identified a list of 52 attributes required to capture the characteristics of programmes concentrating on NEETs. These fell into three groups: ‘Project information’ attributes, ‘Project participant’ attributes and ‘Project outcomes’ attributes (see Table 9.1).

Table 9.1 Categories and sub-categories for Intelligent Decision Support System (IDSS) key attributes
Full size table

To support informed decision-making, an IDSS also requires a source of current actions and evaluations of these actions. To allow practitioners and policy-makers to design programmes targeted at young adults from disadvantaged backgrounds (by learning from past programmes’ effectiveness), detailed descriptions of the past programmes, profiles of the participants they targeted and evaluations of project outcomes are required. Whilst searching for programmes designed to aid young people defined as NEET to populate the IDSS repository, we discovered not only a lack of detailed programme data but also non-standard data formats across programmes and inadequate programme evaluation data. Where programme evaluation data was available, its scope was limited in depth and there was no consistent approach to evaluation. This section describes this situation and provides examples.

Much publicly available programme data is not extensive enough to provide insights for practitioners. The well-known European Social Fund database ‘Creating Chances for Youth’,Footnote 1 for instance, provides short (200-word) project descriptions which vary in amount of detail provided on project aims and activities. Others consist of vignettes of individual students but no insights into the overall project, its objectives or its success in achieving its objectives.

Each programme was found to use its own format to record attributes describing its target groups, participants and outcomes. Two project descriptions of unemployment amongst participants illustrate this point: ‘56% of starts had been claiming unemployment benefit for over 3 months before starting’ (Haigh & Woods, 2016); and ‘Looking for work less than 1 month: 7%; 1–4 months: 39%; 5–8 months 22%; 9–12 months: 8%; Over 1 year: 16%. Less than 1 year: 76%; 1 year or more: 16%’ (Thornton et al., 2014). Clearly, the two sets of figures, referring to different categories and time spans, are not comparable.

Attributes also vary as to the stage in the programme when the data was gathered, or had unclear definitions. For example, the numbers entering employment might be recorded when each person left the programme, at the end of the programme, or at a specified time period after the end of the programme. The percentage of people achieving employment varied as to whether or not it included people who had failed to complete the programme. The definition of what ‘achieving employment’ meant (in terms of hours employed and duration of the employment) could often not be found. Although some guidance on providing programme data is given by the European Social Fund (ESF Support Centre, 2016), this is insufficient to ensure programmes utilise common data formats.

Youth members of the Enliven project’s UK Youth Panel identified another data issue pertaining to programme comparison. They thought finding employment could be the result of more than one programme, running either concurrently or consecutively. For example, an individual might be helped by both an employment programme and a mental health programme. There is no way of tracking the number of programmes with which an individual is involved—in order to discover successful programme combinations—as each programme uses its own participant identifiers.

This lack of programme evaluation data has been referred to in academic literature, but policymakers do not seem to be aware of the need for thorough evaluation: ‘a key finding’ of Mawn et al.’s (2017) review ‘was to highlight the need for future research to adopt high-quality evidence methodologies to determine what works best for this population’ (see also Impetus Private Equity Foundation, 2014; Britton et al., 2011).

It is worth noting that no single programme included in the Enliven IDSS repository (220 programmes in all) contained a full set of the attributes we regarded as necessary to describe programme characteristics. However, all the attributes had a value specified for at least one programme—showing that the programme provider required the attribute.

When undertaking programme evaluation, it is necessary to consider the reader. What outcomes are of interest will differ between, for instance, potential programme participants, practitioners and policy-makers. The process of finding well-evaluated programmes for the IDSS demonstrated that there is little commonality—in how, and for whom, interventions are evaluated—across countries or funding regimes.

To enable programme comparison, a core set of uniform indicators is needed. Enliven identified a set of attributes necessary to describe programmes focusing on young people defined as NEET. However, programme providers should also have the freedom to develop their own indicators, supplementing the core set: this would aid an ongoing process of learning about relevant indicators to take place. Indicators may, of course, be more (or less) appropriate depending on circumstances.

Within the limited number of evaluations of programmes which were available for inclusion in the IDSS repository, most were found to comprise process evaluations providing statistical analyses of programme outcomes. Practitioners and researchers need to work together to identify and develop appropriate methods for evaluating programmes (that aim to support young people who are NEET) which provide sufficient information to enable improvements in policy and practice to take place but whose implementation does not, at the same time, place too heavy a burden on practitioners.

The correct level for programme evaluation (national, regional, provider) should also be considered. Employment levels typically vary between regions within a country (Eurostat, 2021); so can programme implementation. For example, the UK’s Talent Match programme (Big Lottery Fund, 2018) has core ideals, based on the importance of building trust and ongoing communication, and offers a localised and individualised approach to address the heterogeneity of young people at risk of social exclusion. Each of its 21 regional partnerships is therefore autonomous, developing solutions in response to local needs.

Within the Enliven IDSS repository, there are many examples of programmes specifically focusing on regional areas, such as Moin Moin Hamburg—Welcome Tours for Refugees (Hawash & Gehrke, 2015), and Skilling Queenslanders for Work (Deloitte Access Economics, 2012). Regional contextual information needs to be precisely targeted as there can be considerable differences between adjoining regions due to factors such as demographics, transport linkages, and the balance of agriculture and industry. For example, the city of Nottingham contains both areas that are amongst the 20% least deprived and the 20% most deprived within the UK (Rae et al., 2016).

For the UK, detailed regional information is available. The Index of Multiple Deprivation (IMD) calculates information from seven domain indices for neighbourhoods in England with an average of 1500 residentsFootnote 2 (Department for Communities and Local Government, 2015). The UK Office for National Statistics’ NOMIS service provides UK labour market statistics for electoral wards with an average size in England of 7000.Footnote 3 However, for comparing regions across Europe, the best information available is provided by Eurostat. The social information on young people it provides is available only at the Nomenclature of Territorial Units for Statistics (NUTS) 2 regional level (NUTS 2016 Classification). NUTS2 regions have a population size between 800,000 and 3 million—too large to provide contextual data of sufficient detail for comparing programmes (European Parliament & Council of the European Union, 2016). There are thus limitations on the contextual information available for comparing NEET programmes across Europe

Information Modelling and ‘Soft’ Attributes

We discovered that programmes and their evaluations consist of a mix of textual data and nominal, ordinal and quantitative information. Textual data might be a description of programme improvements which could be made. Ordinal information consists of unordered categories—for example, the attribute ‘type of programme provider’ consists of the categories ‘Public sector’, ‘Not for Profit/ Third sector’, ‘Private sector’, and so on. Ordinal information comprises ordered categories, such as how successful (on a scale of 1–5) the programme was. Quantitative information is a numerical measurement, such as the number of participants in the programme. Some programme attributes are composed of both ordinal and numerical information—an ethnicity profile of participants, for instance, might consist of values such as White: 47%; Black: 23%; Asian 7%. An intelligent information system needs to incorporate suitable information models to store this complex mix and to utilise appropriate algorithms so users (policy-makers or others) can locate what they wish to find. Thus algorithms based on quantitative reasoning alone would not be appropriate.

Some programme attributes proved hard to pin down. For example, in the attribute ‘percentage of participants more optimistic about finding a job’, what does ‘more optimistic’ mean? Although ‘soft’ attributes are difficult to define, they can be of use in indicating whether a programme achieved its objectives when precise data is missing. When participants leave a programme, contact with them may be lost and the impact of the programme on their future employment and careers not recorded. Investigation is needed into how to define ‘soft’ attributes satisfactorily, so they can be understood by both programme participants and policy-makers. Commensuration—transforming qualities into quantities, difference into magnitude, reducing complex information into ‘numbers that can easily be compared’ is needed. Such transformations enable ‘people to quickly grasp, represent, and compare differences, offering standardized ways of constructing proxies for uncertain and elusive qualities’ (Espeland & Stevens, 1998). Commensuration is of course ubiquitous in modern society—examples include prices, temperature, grading of student essays, rankings (from football leagues to universities), cost-benefit ratios, censuses, financial instruments such as shares and futures, and Likert scales.

An initial start towards this was made by Rose et al. (2005). They created a motivation scoring scheme for clients attending a supported employment agency for people with disabilities. A detailed description was provided for each level of the scale (Rose et al., 2005). It is also necessary to be consistent about when information is gathered as participant opinions on ‘feelings’ may vary over time as skills are learnt and jobs are gained and lost. To capture, store and search complex information about social programmes efficiently—so policy-makers can use the information—we need to develop appropriate models.

The Individual Story

Enliven youth panel participants pointed out that each young person on a programme has a unique individual story, which is difficult to capture in a statistical format. ‘Blanket’ figures can miss individual barriers. For example, the attribute ‘25% of people secured employment’ does not indicate what characteristics those finding employment possessed. Just because certain people found work as a result of attending an employment programme, it does not follow automatically that others will be able to do so. Neither does this attribute indicate anything about the nature of the employment secured: for example, the salary, whether the employment is full-time, part-time or a zero-hours contract. Members of the UK Youth panel were concerned to gain not just employment but quality employment. This is difficult to measure: which employment conditions are deemed satisfactory depends on the individual’s perspective. The youth panel also suggested an attribute, ‘improved quality of life for participants’, could be used to assess programme effectiveness. Capturing this concept is problematic, but the young people proposed that it might be measured in terms of social engagement (such as the last time a person talked with a friend or relative) and financial independence. Such nuanced understandings require individual stories and narratives: input from youth panels and experienced practitioners proved vital in ‘humanising’ statistical data.

Long term outcomes—the eventual effect of a programme on employment—are also almost impossible to measure. People may remain unemployed for a period after completing a programme, or choose to attend a course of study before attaining employment. More research is needed into what proxy variables would be appropriate for detecting the success of a programme. For example, is attendance at job interviews an indication that employment will be found in the future? The ‘Proximity to Labour market’ measure (Sanderson & Wilson, 2015) is an initial attempt to provide a proxy for employment: it considers how likely a young person is to be in work given their characteristics, experiences and capabilities, and consists of 12 separate categories (including qualifications, experience, well-being and issues such as alcohol or drug dependency).

Within the IDSS knowledge base, a picture emerged suggesting that small, specialised programmes were more effective at finding employment for their participants than larger scale programmes. For example, the Luxembourg ‘Practical training for young construction workers’ programme had 19 participants and an outcome of 86% employment. The Romanian ‘Proiectul tinerilor cu initiative’—supporting Arad city’s candidacy for the European Capital of Culture programme 2021—provided participants with experience in public engagement; 80% of its 16 participants found employment (personal communication). By contrast, the UK traineeships programme, aimed at improving participants’ English and Maths to help them in gaining an apprenticeship or employment, had 19,400 participants, of whom 28% found employment and 22% an apprenticeship (Dorsett et al., 2019). A large number of small programmes is apparently more effective than a single national-level programme.

Whether this provisional finding is valid—and if so, why—is important and demands further investigation. Talent Match (2018)—a five year, UK-wide, initiative between 2014 and 2019—aimed to provide employability support services to young people furthest from the labour market on a regional and sub-regional basis, responding to local need. Similarly, the Activity Agreements, piloted in eight areas of England between 2006 and 2011, repeatedly showed the effectiveness of local, personalised programmes for the most vulnerable young people (Maguire et al., 2011). Both these intensive, personalised, initiatives included not only a financial incentive to secure young peoples’ engagement and participation and give them some degree of autonomy, but also impartial personal support and tailored learning over a specified period of time.

Funding Regimes

Competitive tendering for funding limits information sharing between programme providers (which may be rivals). To the detriment of the vulnerable young people whom the providers and funders seek to serve, the elements of a successful project may not be shared with competitors. When programmes funded for a limited time-span close, personnel leave and information and expertise are lost. Departing (and remaining) professionals may be unaware of the tacit knowledge they possess—which the organisation is losing—and it may be impossible to capture and store it for future use. Funders need to ensure that organisations have sufficient budget to record programme data and allow evaluation to take place. Data gathering is time-consuming and requires that sufficient, capable staff remain available.

Facilitating Effective Policy-Making: Learning from the IDSS

At a demonstration of the IDSS, an audience of adult educators (Boeren et al., 2019) pointed out the importance of users’ being aware of a programme’s context. Gathering evaluation data during and after a programme, and comprehending the data gathered in order to make use of in designing future programmes, are time-consuming, though they can be made easier and quicker by appropriate use of information technology.

When considering a knowledge management framework for evaluating a complex social problem (disaster management), Otim (2006) suggested that a component of the knowledge should be ‘Contextual information: data/information that pertains to a particular context’. As we have seen, participant narratives personalise and contextualise and can be useful in enabling programme providers to understand and interpret bald programme statistics.

Programmes do not operate in isolation. They are subject to external influences such as economic conditions, government policies and the effects of other interventions. These mean programme statistics are difficult to interpret. For example, economic conditions affect the number of jobs available, so a programme will appear less effective during a recession or in a depressed area; social security policies may determine whether certain types of jobs are financially viable; what transport is available allows (or prevents) access to workplaces. To understand programme evaluations, policy-makers need to be aware of such external influences.

If information technology is to help us learn from existing programmes, it must aid data collection from programme practitioners and participants and facilitate data understanding for practitioners and policy-makers. Programme data must also be preserved: otherwise, lessons learnt may be lost. This is key to knowledge management.

When gathering programme data to provide data sources for the IDSS, we discovered that a researcher with specific knowledge of youth unemployment was needed to understand the domain specific vocabulary used by the programme practitioners and evaluators. Enliven project youth representatives told us that wording mattered. Questions should be jargon-free and youth-friendly, and – to encourage answers – why the information was being collected should be explained. Designers of information technology tools should work with domain experts (both practitioners and programme participants) to ensure that those they create are fit for purpose.

To overcome problems with inconsistent evaluations (as described above), and to enable programmes to be compared and lessons learnt and applied in future, funding bodies should ask uniform questions of practitioners and participants. Providing a template based on an underlying information model would help ensure that answers follow a consistent format. The use of simple, easy-to-use forms and websites would help practitioners in gathering and inputting programme information.

When supporting programme design through learning from past policies, a knowledge-based repository of existing programmes (such as the Enliven IDSS)Footnote 4 is required. As many—or all—of the problems for which an IDSS is likely to be needed will be ‘wicked’ and constantly evolving, information techniques must ensure programme information is relevant and up-to-date. User feedback and comments must also be assimilated. A policy-maker, for instance, may be want national evaluation data, while a practitioner may prefer details of activities delivered by a single provider within a programme.

Information visualisation techniques can present programme information in ways which stakeholders can quickly assimilate. Appropriate representations of complex data can enable commissioners and policy-makers to understand better what the needs of a target community of disadvantaged young people are. Complex contextual information in particular can be enhanced by visualisation. There is research in this area: for example, the Organisation for Economic Co-operation and Development’s Well Being websiteFootnote 5 provides infographics enabling regions across the world to be compared in terms of eleven topics important for well-being, such as education, employment and access to services. Eurostat’s visualisation, ‘My country in a bubble’,Footnote 6 compares EU member states using 50 indicators grouped into themes, such as ‘Population and social conditions’, ‘Transport’, and ‘Industry, trade and services’.

While collecting programme data for the IDSS repository, we discovered that online information on previous programmes had often disappeared. Many project websites were not maintained after the project ended; URLs became defunct. Thus, the International Labour Organisation’s Youth Employment Inventory,Footnote 7 initiated by the World Bank, provided comparative information on youth employment interventions worldwide. Its global inventory (over 400 such programmes in over 90 countries) covered programme design, implementation and results achieved. Yet this informative and substantial database—recommended to us by a local practitioner—no longer has an internet presence. Programme information should be permanently archived so that it can be found and used by practitioners, policy-makers and scholars.

Ideally, programme information should be managed in a circular process whereby users learn from existing and former programmes, later recording information about their current programme’s experience, thus informing others. Progress in this can be assisted through information technology tools, such as document management software.

Conclusion

Reviewing the Young and Successful programme provided the authors with a rare opportunity to examine an intensive, tailor-made intervention. It also allowed us, using an interdisciplinary approach, to focus on how ‘distance travelled’ (measuring soft outcomes and steps towards entering education, employment and training) in employability programmes can be captured and demonstrated. The programme recognised that the journey to ‘employability’ for the most vulnerable young adults is not linear; it cannot be encompassed in rigid time limits; neither is it consistent between individuals. Attempts to standardise interventions have a high probability of failure and have unintended—often negative—results. ‘Hard’ outcomes, such as becoming employed, are too narrow. While smaller-scale, more person-centred approaches are labour-intensive and costly, we believe that embracing them would (as the Young and Successful programme suggests) ultimately help reduce levels of long-term economic and social exclusion for young people across Europe.

Computer-aided support for decisions would make policy-making more efficient, avoiding past mistakes. However, before information systems tools can be implemented, we need to ensure programme data are collected in a standardised model format. For this, research into appropriate methods of programme evaluation is required. The Enliven IDSS research has discovered the attributes necessary to capture the characteristics of NEET programmes. Intelligent information system tools for this domain must encompass the features of NEETs as a wicked social problem. If users are provided with contextual information and computer support throughout a programme’s life cycle, an IDSS can be a powerful tool. But policy-makers may have to reappraise their own ways of working.