The objective of the framework is to develop a best practice guide for use in promoting value generation and sharing of ideas within the big data and AI innovation ecosystem.
The goals are to:
-
Foster collaboration and promote sharing of best practices and know-how among CoEs and national initiatives
-
Provide expert guidance and (non-financial) support to member states looking to establish a new national CoE for big data and AI.
Within the framework as illustrated in Fig. 2, there is a process flow in the form of a value chain starting from the environment (which supplies input) through the core BDAI CoE capabilities (which process the input) to the output represented by the impact of the output received by the society under various categories: economic, scientific and societal. There is a backward flow (feedback) from the impact of a CoE back to the CoE and to the environment in which the CoE operates. For example, a CoE may hire personnel it trained as postgraduates or receive income from services rendered to a partner, which can return value to the CoE. Similarly, the impact created can influence the environment in which it operates, particularly regarding policymaking and funding decisions. The quality of output from a CoE is often the most significant determinant of funding decisions by funding agencies.
4.1 Environment
As described in the literature on organisational science, the “environment means forces difficult to control from inside that demanded a response” (Weisbord 1976). The external environment comprises forces that initiate organisational change (Burke and Litwin 1992). In the context of a BDAI CoE, the environment is defined as three forces: industry, policy and citizens.
4.1.1 Industry
The term industry refers to companies, start-ups and businesses that are carrying out economic activities related to big data and AI. While the big data and AI industry would directly affect the strategy and performance of a BDAI CoE, the relative strengths or weaknesses of other industrial sectors may be reflected in the core elements of the BDAI CoE framework. A recent Norwegian study indicated that the industry provides increasingly significant financial support (more than doubled since the 1980s) for academic research while the proportion of basic funding is decreasing (Gulbrandsen and Smeby 2005). In a study carried out among Norwegian university professors, a clear relationship exists between industry funding and research performance. Professors with industrial funding are often engaged in applied research and frequently produce entrepreneurial results, they collaborate more with other researchers both in academia and in industry, and they report more scientific publications (Perkmann and Walsh 2007; Gulbrandsen and Smeby 2005).
The industry in the context of the BDAI CoE framework is defined as follows:
“The ecosystem of companies surrounding a big data and AI Centre of Excellence that is associated with the creation of economic value, at both national and European levels.”
Establishing and maintaining strategic industry—research collaborations should be a priority for BDAI CoEs. Inter-organisational network relationships in the context of “open innovation”, the role of practices such as collaborative research, university-industry CoEs, contract research, and academic consulting are the basic needs of existing CoEs (Perkmann and Walsh 2007).
The industry demands for big data and AI tools and services drive research focus on the development of these innovative technologies through collaborative research, contract research and consultation services with industry participants. Industry-focused CoEs are highly user-centric in the design of their technologies, and, as such, they work very closely with their end-users to co-design functional solutions to the users’ respective challenges.
In the field of big data and AI, CoEs within the EU focus on different domains and trends, while the industries mainly drive decisions within a country. However, international development in science and technology also has an important impact on local trends and decision-making by the management of organisations. For example, within Ireland, the IT, medical and pharmaceutical industries are significant parts of the economy; therefore, data analytics research CoEs focus on providing cutting-edge technology tools and services for these sectors. Centres within economies dominated by petrochemicals focus on the development of data analytics for the digitalisation of oil and gas exploration and related developments in geology domains.
New or emerging CoEs should focus on the areas of interest of the country of operation to align the CoEs’ strategic interest with the national strategic interest. This enables the country to provide better funding and policy support for a CoE. As seen from the case studies, where interests diverge, a CoE could run into problems in balancing its priorities. Evidence from the survey indicates that internal capabilities, such as supportive governance, exemplary strategy implementation, the existing units for business development, a simplified IP arrangement process and advanced outreach programmes, are needed to promote university—industry collaborations.
4.1.2 Policy
Policies and regulations can be divided into two broad categories: research and innovation policy and data protection policy. The first policy defines the goals of funding available to CoEs and influences the alignment of the elements within a CoE with those goals. The second policy primarily focuses on clarifying rules about data usage, data ownership, data localisation and data portability (Ron 2016), which are critical to the operation of a CoE.
Policy in the context of the BDAI CoE framework is defined as follows:
“The policy is defined as the set of public laws, regulations and principles that govern research and innovation activities at the national and European level, as well as dictating the access, manipulation and distribution of data.”
A dedicated agency or agencies in each country support research activities and provide funding support when needed. The reason for the use of dedicated agencies to fund and support research institutions is that these agencies are specialised in designing arrangements and policies that help to align the research institutions’ strategic interests with the country’s overall educational system, particularly STEM subjects, research and development, and development of expertise. The agencies help to prioritise areas of research, not just for the country but also among existing CoEs in the country to avoid unnecessary duplication of research effort and funding. The agencies also monitor the performances of CoEs to ensure impacts are up to expectations considering the investment funding provided to them. For example, the Department of Business, Enterprise and Innovation (DBEI) in Ireland has the responsibility of enacting research-related policies and helps in setting national strategic directions regarding stem disciplines, including Science and Technology and Innovation (STI). In addition to the DBEI, the Science Foundation of Ireland (SFI), Enterprise Ireland (EI) and the Industrial Development Authority (IDA) are Irish Government agencies that not only fund Research and Innovation (R&I) development initiatives but also play crucial roles in planning and deciding the direction of the country’s technological development, including the development of expertise. Generally, policy formulation fosters academic-industry collaboration as a way to facilitate technology transfer from the academic/research institutions to the industry where research results are applied in practice. Successful CoEs have developed strong working relationships with these agencies to implement policy, but also to shape it.
It is essential for new and existing CoEs to ensure close alignment with funding agencies and national research and innovation agendas. For example, one CoE was aligned with a national digital transformation agenda. As part of the transformation process, the CoE was charged with the research and development initiatives for the CoE of a specific sector of national importance. There could arise a considerable number of funding issues, where a CoE interest fails to align well with the national research agenda that is pursued by the funding agencies.
4.1.3 Societal
Citizens or civil society communities play an important role within the external environment of a BDAI CoE. Social, political and cultural values influence the progress of scientific research and technological innovation in society (Bijker et al. 1987). The state of a societal environment around a BDAI CoE can be assessed using frameworks produced by organisations such as the Organisation for Economic Co-operation and Development (OECD) or the United Nations (UN). In this regard, we use the following three indices: the Human Development Index (HDI), the Global Competitiveness Index (GCI) and the Global Innovation Index (GII).
Societal in the context of the BDAI CoE framework is defined as follows:
“The societal environment of a BDAI CoE comprises the state of human development as measured by composite statistics and indexes, and the national priorities for human development regarding the UN Sustainable Development Goals and H2020 Societal Challenges.”
There is a feedback loop between the societal influence on a CoE and the impact of the CoE’s output on the society. Society influences a CoE through various policies and research agenda directives. The societal influences on a CoE include the existing science and technology goodwill of a country, the ability to attract high-level research expertise and industry, and the ability to harness the available expertise and research output. The presence of more expertise and companies enables research institutions to produce higher-quality outputs that are driven by the demand for the output and the availability of quality skills. The identified interdependence between society and research institutions works systematically to sustain the research environment as well as the industrial environment. In this sense, the industrial or corporate entities serve as the user entities for research output, as well as research collaborators providing the problems and challenges for which solutions need to be designed.
Thriving research organisations prioritise the publication of research output, attend international science and technology conferences, and get involved in collaborative research contracts or projects. These are avenues that publicise the inventions of a CoE and add to its popularity, helping it to stand out from the crowd. The CoEs within our study had an excellent national and international record of performances in science and technology development initiatives. The CoEs support the countries’ rise in the Global Indicators, which creates a positive feedback loop by attracting an inflow of personnel and companies which further drive quality output.
4.2 Strategic Capabilities
The strategic capabilities of the framework include strategy, governance, structure, funding, people and culture.
4.2.1 Strategy
Strategy in the context of the BDAI CoE framework is defined as follows:
“The means by which a CoE intends to achieve its overall mission and goals.”
A dynamic and innovative research environment has a clear and visible strategy which has been formulated by a senior research and management group (Schmidt and Krogh Graversen 2017). Successful CoEs have well-defined, distinct, narrow-ranged research areas which are unique in their region (or country) (Schmidt and Krogh Graversen 2017). The strategy of a CoE is not limited to corporate body management activity. Unlike companies which define their future goals and can independently plan how they achieve them, CoEs often have research agendas handed down to them by funding agencies in a top-down approach. This commonly results in a situation where CoEs force severe performance challenges, which can create occasional conflicts of interest between a CoE and its funding partners and host university or affiliated educational institutions. The management act of strategising is needed to define goals to be pursued by the CoE and to plan ways to achieve them. Prioritising strategic goals is critical to make the best use of available resources and create a focus on the mission of the CoE.
The BDAI CoE study discovered that the strategy design processes in the studied CoE cases were similar. For example, in all cases studied, the management
-
clearly defines and lists its strategic goals and objectives
-
tries to align the CoE’s strategic goals and mission with national (and European) research goals and objectives
-
is market-focused and directed towards industry challenges and demands when designing its future goals and objectives
-
is working hard to attain knowledge development and technology transfer to the industry through collaboration with industry partners
On the other hand, there are specific approaches that are different in the case studies.
For example, some CoEs carry out widespread consultations to gather information to formulate strategies. Such consultations included dialogue with stakeholders in the research ecosystem and with their staff, and research and funding organisations and affiliated educational institutions.
Some CoEs break down strategic goals into manageable objectives or activities and use Key Performance Indicators (KPIs) to measure performances towards objectives, goals, mission and vision. These KPIs cover impact areas including economic, commercialisation and academic, and they are operationalised.
Applied CoEs focus on developing a robust interface with industry partners. This approach helps the CoEs to:
-
Identify technical, social and cognitive barriers to use of technology
-
Define, reinforce and maintain mutual understanding and a shared vision with industry partners
-
Track evolving technologies and challenges
-
Establish new industrial collaborations
Through this approach, CoEs can identify constraints in existing tools, identify opportunities for changes to transform end-user work practices, and transfer knowledge and expertise via a feedback loop in the innovation cycle. This end-user knowledge allows them to engage in industrial projects and to justify continued basic funding from funding agencies.
Finally, decision-making through consensus of all members at the CoEs on major matters requires holding several meetings and using procedures to prepare and anchor decision-making and to run processes that enable achievement of a consensus.
The BDAI CoE study reveals that in the strategy design process, CoEs consider the following factors in the definition and design of future goals, objectives and priorities:
-
Design strategic goals to align with the national research agenda
-
Design strategic goals to align with market demands and trends, bearing in mind future needs and developments
-
Break down overloaded research agenda from funding institutions into strategic goals and objectives
-
Break down goals into manageable activities and measure each with KPIs
-
Operationalise the KPIs into daily activities
Strategy Formulation
A broad dialogue is necessary to design robust strategies for a CoE. The formulation of the strategy needs to go beyond the senior management group and be inclusive of all stakeholders, including researchers and students. The process of soliciting contributions to strategy design needs to be all-inclusive. For example, one CoE holds an annual general strategy meeting to gather ideas from everyone on how to advance the CoE. It is also crucial that the strategy formulation opens a dialogue with industry stakeholders, host university(s) and researchers from the broader ecosystem. This dialogue with stakeholders is regarded as very important for a CoE’s future success as it offers the stakeholders an opportunity to articulate their priorities. For instance, some stakeholders may prefer the development of an international profile, while others suggest the development of national and local priorities.
Alignment of KPIs with Strategy
As part of the strategic initiatives of a CoE, the management should strive to design KPIs to measure the performances of their organisation towards the set goals.
In this sense, the CoE’s management should operationalise some clearly defined strategies by formulating them into objectives that are measurable using properly designed KPIs. The measurement of those KPIs should be on a regular periodic basis, for example quarterly, bi-annually or annually.
4.2.2 Governance
Governance in the context of the BDAI CoE framework is defined as follows:
“The means by which a CoE achieves decision-making and operations.”
Joynson and Leyser (2015) propose a set of good research practices for high-quality science regarding research governance and integrity, which include training in good research practice, openness about the consequences of misconduct, and adoption of appropriate ethical review processes.
Core to the effective governance of a CoE is a strong governance body and management team. The governance body of a CoE can go by a range of names, which include Governing Council (GC), Centre Steering Committee (CSC) or General Assembly (GA). The composition of the governing body usually consists of both internal and external members. Internal members typically include the CoE’s Director or Chief Executive Officer (CEO) and a few top-level officials which could be both academic and non-academic staff. External members can be drawn from industry partners. Despite the similarity in the composition of the governing body, differences exist to some extent. For example, some CoEs include an independent observer, an official from the Technology Transfer Office (TTO), or members of governmental departments.
The governing body of a CoE holds regular meetings, about twice a year in some CoEs and up to three or four times a year in other CoEs. Some CoEs use a Strategy Board to complement the activities of the governing body. The Strategy Board is charged with the responsibility of drafting the strategic goals as well as overseeing the day-to-day operations of the CoE. These boards are composed of the top leadership personnel of the CoE. Often CoEs maintain an Executive Team and together with the CEO of the CoE report the CoE’s operations to the GC. In reverse, the GC disseminates its information through the Executive Team to the general members of the CoE. This approach is bottom-up and top-down information dissemination.
The management team of a CoE needs to plan and coordinate research activities, define and prioritise research target areas, and emphasise research productivity and quality (Schmidt and Krogh Graversen 2017). The management team should lead by example by supporting high ethical standards and paying attention to the responsible conduct of research. They should ensure policies that promote being the “best” within the scientific enterprise, and within a context that encourages responsible conduct (Schmidt and Krogh Graversen 2017).
In general, the governing body has the role of making the top-level decisions and approving the strategic goals, objectives and priorities of the CoE. Whatever the composition is, there is a significant value that each member brings to the governing board. For example, an independent observer assumes the role of suppressing biases in judgements or dealing with areas of conflict of interest during decision-making processes. Similarly, the role of the Principal Scientific Investigator in the governing body is to introduce ideas from an in-depth research point of view, which is necessary for delivering research targets.
The bottom-up and top-down information dissemination approach is useful in ensuring accountability, contribution to the decision-making process and an allowance for general inclusivity. It also enables the governing body to monitor the CoE’s performances through KPIs.
4.2.3 Structure
Structure in the context of the BDAI CoE framework is defined as follows:
“How a CoE is designed in terms of levels, roles, units, decisions, and accountability.”
An appropriate CoE structure depends on the type of institutions and the level of decision-making, as defined by Bleiklie and Kogan (2007)
-
CoE as a “republic of scholars”: Leadership and decision-making is at the collegial level by independent scholars.
-
CoE as a “stakeholder organisation”: (1) Institutional autonomy is considered a basis for strategic decision-making by leaders who are assumed to see it as their primary task to satisfy the interests of major stakeholders. (2) The interests of other stakeholders therefore circumscribe academic freedom.
Schmidt and Krogh Graversen (2017) identified that dynamic research environments have flexible organisational structure which may consist of a core researcher group and some attached members or affiliates. Successful CoEs have an organisational structure with high adaptability to internal and external changes.
One of the most critical findings in the case studies is that the structure of a CoE is designed to ensure representation of stakeholders, including host institutions (or affiliate educational institution), industry partners, funding agencies and key staff of the CoE. The structure is designed to facilitate operations and support decision-making and governance that enables coordination and integration of the activities of the CoE for consistency and synergy.
In the design of the structure of a CoE or in guiding the evolving features of the structure, it is important to consider the size of the CoE and the scope of activities. It is also essential to consider the interdependency of the various roles that must work together to optimise resource utilisation to maximise outcomes. Structures enable the efficient running of an entity – the roles, the reporting lines and the accountability for the respective responsibilities. The structure facilitates information dissemination and enforcement of rules and regulations, and thus can also play a key role in the development of suitable cultural practices.
4.2.4 Funding
Funding in the context of the BDAI CoE framework is defined as follows:
“The availability, diversity and sustainability of the monetary support for carrying out research and educational activities in the CoE.”
Funding practices for a CoE need to ensure that it is provided with sufficient funding and that it has diverse external funding sources to supplement basic research funding. Funding practices in CoEs with a focus on applied research look to secure funding in the form of collaborative or contract research, with industry partners facilitating technology transfer. Joynson and Leyser (2015) highlight two good research practices for high-quality science through the adoption of diverse funding approaches and the clear communication of funding opportunities and assessment criteria funding that are critical to the recruitment of new researchers, which is a key success factor of a CoE.
From the BDAI CoE case study, the result shows that funding models are provided in a cycle with a fixed period to address specific long-term objectives (e.g. 4, 6 or 8 years). Funding schemes come in mixed models comprising diverse funding sources. A mixed funding model pushes a CoE to explore multiple funding sources such as national, industry (local and international) and European funding sources (e.g. H2020). The industry funding sources could further be broken down into contract research with large multinational companies or with small and medium enterprises (SMEs), as well as with start-up companies. However, there are challenges involved in dealing with SMEs and start-up organisations because of their income level and undefined strategies and goals. Extra funding sources beyond the basic sources usually supplied by funding agencies can also be in the form of services delivered as consulting services by CoEs to other corporate entities or organisations in the not-for-profit sector or even educational sector. The extra funding could also come from national funders that facilitate organisations to sponsor projects financially for a CoE to execute them. In the European Commission (EC), most international funding sources come from EC H2020 and FP7 projects. Participation in projects sponsored by these funding sources in addition to collaborative research with industry partners helps CoEs to obtain extra income to augment their basic funding requirements.
A CoE’s sources of funding can be listed as follows:
-
Engagement or collaboration with industry partners in collaborative research and consultation services to industry members.
-
Participation in EU projects under mainly Horizon 2020 and also FP7 Research and Innovation projects.
-
A business development role can be used to develop better engagement capability with the industry partners, which helps a CoE to negotiate more contracts for more income.
-
Some CoEs pursue commercialisation and spinouts as well as scientific inventions and publications. However, some are more specialised in the scope of research and innovation development.
Additional funding is often needed to enable a CoE to finance specific interests that the funding agencies may not want to fund. However, funding policy requirements may pose some challenges for a CoE in that it may be required to provide a given amount of its funding needs to become eligible for funding supply from its financiers. For example, one CoE studied needs to provide up to 25% of its funding needs to be eligible for continued funding from funders. This places the management under pressure to collaborate with industrial partners even when it is not a priority to enter into such a contract.
4.2.5 People
People in the context of the BDAI CoE framework are defined as follows:
“The people required to carry out specific tasks towards the goals of the organisation.”
CoEs are affiliated to educational institutions, which appear, in most cases, to be the primary sources of personnel supply, particularly CoEs that run academic courses such as master’s, PhD and postdoctoral training. In the case of all CoEs, the host universities provide the human resources policies that guide the personnel practices in the CoE.
To gain a broader scope of expertise to bring into their CoEs, the management of research institutions advertise vacancies in both local and international fora, and this enables them to build a range of options into the selection process. CoEs also use some cultural practices:
-
To keep their people in a good state of mental health and social well-being (e.g. community volunteering programmes, excursions, walking and cycling activities). If the CoE is not hosted at a single physical site, these can be online activities (e.g. online mindfulness, virtual coffee sessions and online game tournaments, especially during the Covid-19 pandemic).
-
To help build their skill and careers through various activities—for example, lunch seminars with invited speakers and on-the-job training of partners’ workers on internship programmes. A programme at the CoE combines researchers and partners representatives for cross-fertilisation of skills, weekly meetings featuring occasional invited speakers and thesis programmes involving public speaking.
-
To help in the integration of new in-takes through mentoring programmes and to get them up to speed with others.
-
To eliminate preferential treatment (e.g. unconscious bias programme) and ensure gender equality and gender mix, programmes like staff diversity, gender equality and women’s networking, are organised. In addition, local language training programmes for non-native speakers take place.
To make people feel at home, CoEs use programmes to bring about a feeling of togetherness in a common purpose. For example, one CoE organises an International Cultural Day, which is an event where the different cultures of the various represented nationalities are displayed and celebrated, including the provision of food from various nationalities. A feeling of togetherness can also be achieved through the creation of a friendly environment, where individuals voice their concerns. This helps achieve collaboration and teamwork necessary for productivity in the CoE. Joynson and Leyser (2015) propose a set of good theoretical research practices for high-quality science. These practices include providing adequate training programmes for researchers, being open and clear about consequences of misconduct, and the adoption of appropriate ethical review processes.
4.2.6 Culture
Culture in the context of the BDAI CoE framework is defined as follows:
“The underlying values, beliefs and norms that drive the teams and the CoE as a whole.”
Culture is a critical part of the CoE. Schmidt and Krogh Graversen (2017) identified that a successful CoE has a working climate based on internalised norms grounded in a research tradition. The working environment should be open to new ideas, methods and approaches. Staff within the CoE have research autonomy during the research process. The working climate is based on teamwork with close cooperation among research staff. Finally, they identify that culture encourages internal professional and social dialogues.
The case results point to the common fact that most CoEs have a mix of local and international culture. A key question is how CoEs use cultural practices to achieve a spirit of togetherness and inclusivity that reduces conflicts, eliminates preferential treatment and maximises productivity.
The effective use of cultural practices in CoEs helps the management to mitigate problems and helps staff to attain high levels of productivity:
-
Integration of new in-take
-
Collaboration and teamwork
-
Welfare programmes
-
Researchers/staff personal skills development
-
Inclusivity and voice
-
Support for outreach
-
Elimination of preferential treatment and achieving gender equality
Culture plays a vital role in the level of interrelationship and interaction existing between people in an organisation. Culture is connected to the degree of collaboration that is possible in an organisation and has a direct impact on the success of the organisation. Culture is developed or guided to evolve into practices that support healthy sharing, caring and support of one another, a situation that enables people in an organisation to feel a sense of togetherness, giving them an opportunity to voice their concerns and contribute to decision-making processes and general shared goals. Like corporate organisations, research institutions also recognise the strong need for good cultural practices in a workplace and how to use their impact to direct success.
The BDAI CoE study reveals that CoEs use various programmes to enhance cultural practices and to make things happen in the way they are desired. For example, in the case of integration of new in-takes, some CoEs use mentoring and orientation programmes to familiarise recruits with their operation and culture. Welfare programmes cater for students and staff to make them feel valued and to get the best out of them for their success and that of the CoE. As they cooperate and collaborate to deliver for the success of their institutions, researchers in research institutions, particularly student researchers, often have personal career development needs. To compensate for their individual needs, leading research institutions provide career and personal development programmes for their workers.
4.3 Operational Capabilities
Operational capabilities in the context of the BDAI CoE framework are defined as follows:
“The operational capability is the ability of a CoE to perform a coordinated set of tasks, utilising organisational resources for the achievement of its mission and goals.”
The BDAI CoE framework identifies a set of operational capabilities needed to operate a CoE. These capabilities are detailed in Table 1.
Table 1 Core operational capabilities of the BDAI CoE framework Capabilities maintained by a CoE are partly dependent on their areas of focus and partly conditioned by their need to meet stakeholder demands. There is a wide range of capabilities within the studied CoEs. Some of the highlights from the case studies are as follows:
One CoE exercises an elaborate plan of outreach in the form of Education and Public Outreach (EPE) programmes for which a Subject Matter Expert is employed. The elaborate EPE process is informed by the importance attached to it by the government’s interest in making the public aware and also taking advantage of science and innovation outcomes.
A CoE with an applied focus to bring the best of services and products to their industry partners adopts a practical process of demonstrating their prototypes contained in a catalogue of demonstrators, IPs and the state-of-the-art analytics and visualisation technology reviews to their partners. This capability brings research outcomes to its network of industry members to which it also delivers services such as seminars, conferences and consultation to create awareness and disseminate information to the end-users of its technologies. The process of garnering collaboration with partners uses two calls for demonstrator proposal. Later, a team filters the proposals received and rates the accepted ones. Finally, the rated proposals are decided upon by the senior management of the CoE, which makes final proposal choices.
Another CoE developed an iterative three-stage process of innovation cycle methodology called Scalable Innovation Cycle (SIC), in which the CoE carries out a user-led generation of ideas and validation of results. The CoE’s processes are highly user-centred, and hence it aligns them closely with the end-user-centric methodologies. The goal of this methodology is to combine research with real-world deployment to meet real business problems. Being iterative, SIC requires the use of a series of feedback among pilots, prototypes and experiments to identify new challenges and gaps to perfect results.
The results of these case studies show that there are various capabilities, and these capabilities tend to differ from CoE to CoE depending mostly on their strategic research domain and end-user needs. With this in mind, it is hard to pinpoint one capability as the best approach to research as there are reasons that support the use of individual capabilities in each CoE.
However, whatever capability is in use in a CoE, there is a need for it to be regularly well operationalised and measured for the desired outcome. KPIs should be designed by breaking down a capability into stages of work, and metrics should be put in place to measure performances at each stage over a given time interval or periodically.
4.4 Impact
Impact in the context of the BDAI CoE framework is defined as follows (Harland and O’Connor 2015):
“The direct and indirect ‘influence’ of research or its ‘effect on’ an individual, a community or society as a whole, including benefits to the economic, social, human and natural capital.”
The definition of the impact metrics and their measurement methods are a significant part of the impact assessment methodology. The following subsections provide guidelines from the literature on how to measure the economic, scientific and societal impact of research output. The impact on the environment and society would be seen in reports of innovation activities derived from field research about impact areas such as economic, scientific and societal. The parameters to understand impacts could be measured through the KPIs being monitored by the BDAI CoE and those monitored by the country government agencies in which the BDAI CoE is located. For example, the economic impact could be how a CoE and industry partnership or collaboration in research and technology is bringing about a measurable increase in commercial activities, companies created through commercialisation, spinouts and jobs creation, and skills development. There are reports which provide a narration of these measures for the government and government agencies to use in support of policymaking for performance review and educational purposes.
4.4.1 Economic Impact
Economic impact in the context of the BDAI CoE framework is defined as follows:
“The economic impact is the effect on commerce, employment, or incomes generated from Big Data and AI research in general and by the CoE in particular.”
As described in Adams (2016), the examples of best practices for the assessment of economic impact are:
-
Funders need to be sure that job creation is reported consistently across multiple organisations so researchers need an agreed standard such as “full-time equivalent jobs created” to avoid counting part-time roles.
-
Claims of impact remain assertions unless there is an independent validation of impact evidence.
-
Evaluators require an audit trail to use impact data for evaluation purposes.
-
Impact evidence must be collected over time, attributing each impact to original research or expertise and tracing the developing sequences of activities.
-
Evidence types can vary widely depending on the discipline, the stakeholders and the changes that have occurred.
-
Impact evidence can include quantitative reports of increased sales for a commercial stakeholder or quality of life improvements.
-
Qualitative testimonials can directly attribute changes to the research, or the contributions made by researchers because of their expertise.
-
Impact information needs a standard structure and categorisation.
A digital research report by Digital Science & Research Ltd. that was released in March 2016 suggests the following best practices for a Research Excellence Framework to improve both the quality and value of future CoEs (Adams 2016):
-
To ensure that the full range of meaningful impacts can be recognised, consider extending eligible periods both for impacts and for the research on which they were based.
-
Require listing of funders and grant references in the case study template.
-
To aid assessment and further use, consider developing guidance on certain types of evidence where appropriate, e.g. sales, staff numbers, company investment.
-
Where possible, re-use information from other systems, e.g. ORCID.
4.4.2 Scientific Impact
Scientific impact in the context of the BDAI CoE framework is defined as follows:
“The scientific impact of a CoE is the returns on research investment assessed qualitatively or quantitatively within the academic sphere.”
The assessment of the scientific impact of a CoE helps funding agencies to evaluate returns on research investment from a research impact perspective. The scientific result can be assessed qualitatively or quantitatively. An analysis carried out by Sutherland et al. (2011) identifies the following practices for quantifying the impact and relevance of scientific research:
-
Qualitative approaches: This approach involves expert panels evaluating impact, for example as high, medium or low, based on written descriptions of impact.
-
Quantitative approaches: This approach involves numerical indicators derived from scoring systems or questionnaires focused on the various possible impacts of a research programme or project.
4.4.3 Societal Impact
Societal impact in the context of the BDAI CoE framework is defined as follows:
“The societal impact of a CoE is its impact on human lives and health, organisational capacities, societal behaviours and the environment.”
A variety of frameworks and models are proposed to quantify and measure societal impact (Penfield et al. 2014; Bornmann 2013; Sutherland et al. 2011). Such a variety of frameworks might also be reflected by the impact assessment methods adopted by national funding agencies across Europe. Regardless of the specifics of assessment tools or methods, the underlying objective of assessing societal impact is to understand the social externalities of research and innovation activities undertaken in a BDAI CoE.
Impact on the environment and society can be captured by reporting activities which are conducted by several agencies such as the United Nations Human Development Index (UNHDI), GCI, GII, Knowledge Impact (KI) and Knowledge Fusion (KF) rankings agencies or organisations. These rankings are measurements that also categorise measures into impact areas such as economic, scientific and societal. The parameters to understand impacts could also be measured through some KPIs being monitored by the individual BDAI CoE, on the one hand, and those monitored by the research-funding agencies and other government agencies of the country in which the BDAI CoE is located, on the other hand.
Societal impact can be reached through various practices that CoEs can adopt to influence the relationship between research and society (non-academic community). Societal impacts, as defined by Molas et al. (2002), are part of a conceptual framework for analysing third-stream activities and categorised as follows:
-
Research CoEs have capabilities in two main areas: (a) knowledge capabilities and (b) physical facilities. These capabilities are developed as CoEs that carry out their core functions of teaching and research.
-
Using the means at their disposal, CoEs carry out three main sets of activities; they (c) do research, (d) teach, and (e) communicate the results of their work.
The type of economic impact a CoE has on the economy in which it exists is dependent on the research areas it specialises in and how that drives economic output. For example, a large-scale CoE may have broad research areas which cut across data analytics applicable in many domains such as media analytics, optimisation and decision analytics. It also participates in other domains such as personal sensing, sustainable IT, e-government, machine learning and Semantic Web. On the other hand, a CoE may have a narrower domain focus with industry-centric capability for producing various data analytics and visualisation tools. Centres may also focus on a single industrial domain. The visible outcome of a CoE does not depend entirely on its output because it also depends on the amount of publicity the CoE has provided on its scientific outcome. Publicity on a CoE’s research result is essential in that it helps to create public awareness (locally and internationally) and attract partners for collaboration, creating an avenue for technology transfer.
Conversely, collaboration opportunities previously involved have the potential to bring more opportunities to the CoE because previous engagements serve as an opening for further engagements. This is the cyclical aspect which calls for adequate investment in various ways by which research output can be publicised, and it should include the national agenda of the country in which the CoE is located, as well as the funding agencies’ contribution towards publicity and exposure to opportunities. Many countries have put in place policies to drive outreach activities from CoEs to the public, while individual CoEs also make an effort to get involved in presentations at conferences as well as sending entries to scientific publications. Another important consideration for impact is the quality of research output. Good-quality and innovative research output sells itself while bad results fail. This would therefore be a good reason to invest in world-class researchers and infrastructure, in addition to a continuous study of the trends in the markets both in the local and international environment.
Scientific impact is constituted by additions to the state of the art in science and technology which are made known to the public through publications in scientific journals and conferences, as mentioned above. A culture of documentation of research processes and findings on a regular basis can help provide information necessary for preparing articles on the outcome of research endeavours. Documentation should be given priority in research exercises not only for project purposes but also for article writing and presentation at scientific conferences. Societal impact is linked to economic impact with the use of research outcomes in the industry, thereby creating new companies, jobs and economic values which benefit the entire society. Also, societal impact refers to the direct benefit derived by people when they use technology items and when technology helps to create better conditions around them, e.g. reduction in poverty levels and crime and disease control and prevention, as well as helping humanity sustain a greener environment in any way possible.
4.4.4 Impact Measured Through KPIs
Whichever category an impact belongs to, it can be measured through specific indicators that can capture perceivable improvements due to the outcomes of a CoE. KPIs (as described in Table 2) are basic indicators that can be measured with defined metrics designed to provide measures of benefits produced regarding economic, scientific and societal advantages. For example, in Ireland, the principal research financing agencies, such as SFI, EI and IDA, have together developed a set of KPIs to measure research performances and their impacts on the nation’s goals based on their research outcomes. SFI demands that a research centre’s targets be ambitious and achievable and reflect the strategic and commercial positioning of the centre. The centre’s targets will therefore be part of the basis for evaluation of the centre’s proposal. Also, funded centres’ metrics will be reported against defined KPIs and evaluated against the targets on an annual basis (Roche et al. 2013). SFI selected 13 KPIs and used these to score each centre under relevant performance indicators and targets broken down into four categories: academic outputs, human capital outputs, funding diversification and commercialisation. All of these must be aligned with the objectives of the research centres’ programmes as well as the overall SFI objectives per Agenda 2020.
Table 2 Sample impact KPIs SFI evaluates a research centre’s performance periodically using evaluation instruments such as the Metrics Governance report and balanced score card, the annual report of the centre, the annual census report (including financial reporting) and site visitations with the external panel (Roche et al. 2013).