The Radical Technology Inquirer (RTI) tool for technology anticipation and evaluation: introduction and quality criteria analysis

Abstract

This paper contributes to the recent discussion about the standards and quality of futures research. The quality criteria proposed by Kuusi (Futura 1/15, 2015) concern the internal and external validity of a futures research process and its outcomes. This paper introduces a new tool for technology anticipation and evaluation called the Radical Technology Inquirer (RTI) and analyses its internal and external validity in light of the recently proposed quality criteria for futures research. The aim is to provide a practical use case for the quality criteria where the validity of a new methodology is analysed.

Introduction

The topic of quality in futures research has gained attention in the recent years. Establishing standards and common criteria to evaluate the quality of future-oriented studies would improve the credibility and understanding of the studies in the field. Authors have proposed variations of criteria [1, 2] and discussed the epistemological underpinnings of foresight [3]. One of the recent contributions to the scientific discussion concerning the quality of futures research is the set of quality criteria proposed by Kuusi, Cuhls and Steinmüller [4]. These criteria address both the internal validity of the research process and the external validity of its results. The aim of this paper is to contribute to the discussion by providing a practical, examplary use case of these criteria. This paper applies these criteria to evaluate the quality of a recently introduced methodology for technology anticipation and evaluation called the Radical Technology Inquirer (RTI) [5]. Therefore, this paper is both an introduction to the RTI tool and an example of the application of the quality criteria for futures research. The paper first introduces the quality criteria proposed by Kuusi, Cuhls and Steinmüller [4]. It then describes the elements, principles and process of the Radical Technology Inquirer (RTI) tool [5]. Lastly, it applies the quality criteria to the RTI tool for an analysis of the tool’s quality.

Introduction of the foresight quality criteria

Kuusi, Cuhls and Steinmüller [4] make a distinction between internal and external validity for evaluating the quality of futures research, and argue that an internally valid futures research process in turn promotes valid results, i.e., external validity of the research process. The following sections outline these sets of internal and external quality criteria for futures research.

Internal validity

According to the Kuusi et al. [4], internal validity relates to the use of sound research methods in a well-organized manner. Internally valid futures research processes are considered to ‘shape the future’ into pragmatic and organizational approaches.

Kuusi et al. propose that a good way to analyse the internal validity of a futures research process is to answer the following questions originally defined by the EFFLA Group [6]:

• a) What is the objective of the whole foresight activity? Are there hidden agendas?

• b) What type of activity has to be considered for what type of issues/time spans/knowledge?

• c) What is the scope of foresight? What is the scope of relevant intelligence and sense-making? Is there specific strategic intelligence or are there sense-making projects to be launched? How focused or wide should their scope be?

• d) What is an appropriate set of/combination of methods to make use of the strategic intelligence of the specific actors? And how can this be organized?

• e) What are the intended outcomes of the different stages in the process? In general, reports are written but often, the activity as such is an outcome. How are the results presented?

According to the EFFLA Group, foresight, or ‘applied futures research’, projects should always follow a process that integrates strategic intelligence, sense-making activities and a link to the policy cycle. More specifically, the EFFLA group has suggested the following phases of foresight: Strategic Intelligence, Sense Making, Selecting Priorities and Implementation [6]. The above questions of internal validity also reflect this process.

External validity

Kuusi et al. suggest six pragmatic criteria with which the external validity of a futures research process can be appraised. More specifically, these six criteria help validate the futures map, which is the result of a future research process. The futures map includes all possible futures that have been identified during the research process, and therefore, depicts a “whole picture” rather than singled-out elements of the research. The authors argue that this makes the futures map a suitable frame to analyse the quality of the research results. Using the following six criteria, one can evaluate which futures map (FM) is more valid, when the customers of both are the same or their intersts are the same [4].

The six pragmatic criteria of external validity are:

• 1. FM1 suggests more possible futures than FM2 that might be relevant from the point of view of the vision or acceptable futures. (Wide scope of possibly relevant paths)

• 2. FM1 is able to identify most relevant futures better than FM2. (Important relevant futures)

• 3. FM1’s scenarios are causally in line with more futures’ relevant facts than FM2’s scenarios. (More interpreted causally relevant facts)

• 4. FM1’s number of facts that get causal interpretation in scenarios divided by the number of scenarios is higher than in FM2. (Effectively with scenarios interpreted facts)

• 5. FM1 is understood by more customers than FM2. (Many understand)

• 6. FM1 is better understood by those customers who understand FM2. (Better understand)

Introduction and description of the Radical Technology Inquirer (RTI) framework

The Radical Technology Inquirer (RTI) tool was published in 2013 by the Finnish Parliament’s Committee for the Future, which has anticipated and evaluated social impacts of new technologies since the 1990’s. The RTI tool, which was developed for national foresight purposes, is a new approach to technology anticipation and evaluation, and its design draws from the Committee’s previous cooperation with the EPTA (European Parliamentary Technology Assessment network) [5].

The aim of the RTI is to provide a framework for the evaluation of emerging radical technologies and for ranking them in order of importance from the user’s perspective. The tool also allows the user to arrange emerging technologies from the perspective of anticipated market demand, thereby expanding the scope of the tool from mere anticipation of technologies to understanding their impact on societal level. The tool therefore supports both the ‘demand pull’ and the ‘technology push’ viewpoints.

The following sections describe the elements of the tool, the calculation principle for evaluating and ranking the technologies and the process of the tool.

Elements of the Radical Technology Inquirer (RTI) and their effect relationships

The elements of the RTI and their effect relationships are illustrated in Fig. 1. The RTI tool consists of two main elements: 20 Global Value-Producing Networks (henceforth also referred to as GVPNs) and 100 Radical Technological Solutions (henceforth also referred to as RTSs), and five additional elements, which are estimated to have an impact on the breakthrough potential of the RTSs (see Fig. 1). These five elements are the anticipated maturity of the technological solution in 2020–2030, its scientific promise, the global market R&D activity focused on the solution, (Finnish) national competence in the solution, and (Finnish) national access to relevant application areas of the solution [5].

Fig. 1

Elements and their effect relationships in the RTI framework

The Global Value-Producing Networks (GVPNs)

The Global Value-Producing Networks are combinations of emerging technological possibilities and customers’ needs or potential demands. The underlying assumption of the GVPNs is that people and organisations will satisfy most of their needs through these demand clusters by 2030. 20 such clusters of global customer demand have been identified in the RTI [5]. The key features of the GVPNs are defined as follows:

• 1. A global value-producing network describes an area of technological and societal change based on global demands. The global value-producing networks have been chosen from a Western perspective, which is familiar to Finns. In the pilot study, the economic or social impacts of the networks have been described using the impact figures on the Finnish economy. The figures anticipate possible savings or welfare impacts in the Finnish home market as well as the potential in the Finnish export markets. Besides Finland, the figures also indicate the relative importance of the value chains in similar developed countries.

• 2. Because of the big global market, the exact size of the global market is not relevant from the point of view of an economy the size of Finland. The export potential of a network for Finland might be great even if the global market related to the network is not especially large on global scale. The total size of each selected global value-producing network is so large that a niche in the value-producing network can be highly significant for a small country like Finland.

• 3. Though the twenty global value-producing networks are selected especially from the Finnish perspective the authors of the pilot study anticipated that by 2030, people and organisations worldwide will satisfy most of their demands or welfare producing activities through them [5, p. 15].

It is assumed that organisations and individuals participate in more than one GVPN simultaneously, and therefore, traditional industry categorisation has not been used in the RTI, although some GVPNs do resemble conventional industries [5].

In the RTI, each GVPN is given a 1–3 page introduction that includes the current state of the GVPN from the perspective of a country like Finland and its people. The anticipated, possible new operating model that technological development opens is also described. The possible savings and added value created by the new operating model are discussed as well as challenges of the transfer period. The introduction of each GVPN also discusses the threats of the emerging GVPN and how to prepare for them [5].

The 20 GVPNs identified for the first version of the RTI are:

• 1. Automation of passenger vehicle traffic

• 2. Automation of commodity transport

• 3. Manufacturing close to customers

• 4. Virtualisation of retail trade and services

• 5. Local or functional food

• 6. Distance presence and remote control of tools

• 7. Individualisation of learning and guidance

• 8. Self-care based and personalized healthcare

• 9. New capabilities for those who have lost their functional health

• 10. Equipment that increases awareness of the environment

• 11. Functional materials and new material technologies

• 12. Functional added value of intelligent goods

• 13. Sustainable energy technologies

• 14. Raw materials from untapped areas of the Earth and space

• 15. Participatory forms of entertainment, culture and influence

• 16. National defence and antiterrorism

• 17. Functionalization of spaces and structures

• 18. Operation models for self-organising communities

• 19. Virtualisation of identities and social structures

• 20. Democracy, freedom and social cohesion

The Radical Technological Solutions (RTSs)

Each of the 100 technological solutions have been chosen for the RTI on the condition that they are either important for many GVPNs or they are crucial for some GVPNs. This is assumed to imply that the technological solution has the potential to change current practices of the GVPNs either by saving costs, easing people’s everyday lives or by increasing comfort or strenghtening or weakening power structures. The technological solution should also be available in the market by 2020 at the latest, and its impacts should be vast by 2030. To have been chosen on the RTI’s list of 100, the technological solution must have already been proven in a scientific publication.

The Radical Technological Solutions refer to clusters of technologies that aim to meet some shared challenge, as opposed to single technologies. In other words, there can be many technical execution options of the same function, and these execution options are included in the same RTS if their development efforts have a shared target function.

Each of the 100 RTSs are described in the RTI through a summary of the recent state-of-the-art of the solution and its future prospects by 2020 and 2030 if the solution delivers on its promise as anticipated. Today’s various leading achievements related to solution are also described, and hyperlinks to the most relevant background documents are presented for the reader. These hyperlinks function as the empirical basis for the anticipations and evaluations of the RTSs. Evaluations of the most probable application areas and most promising execution options of the functional target are based on the information gathered from the hyperlinks. After the hyperlinks are presented, the maturity or development stage of the technology is presented, in addition to the scientific interest towards the solution, the global market R&D activity focused on the solution, and the evaluated linkages to the GVPNs [5].

Maturity

Each of the Radical Technology Solutions are categorized into four levels of maturity or development stage as follows:

• 1. Scientific principles that make the technological breakthrough possible are proved and the functionality is demonstrated in a peer-reviewed scientific paper.

• 2. Prototype that is scientifically or commercially demonstrated. The functionality of the prototype fulfills requirements of the commercialization.

• 3. Enough actors that have financial resources develop the technological breakthrough that is close to commercialization.

• 4. Increasing amounts of products are delivered to customers, new application areas emerge and prices of products decrease [5, p. 49].

Scientific promise

In the RTI, a key assumption is that the likelihood of the success of a technological solution in the long term is strongly affected by the scientific interest toward the field. Global scientifc interest supports development opportunities and other vast and active development work related to the solution. In the RTI, the most important background sources for the evaluations of scientific promise have been the Science Map studies of the Japanese NISTEP, the National Institute of Science and Technological Policy [7].

Global market R&D activity

In the RTI, it is assumed that the future potential of a technological solution is higher if there is vast development work related to the technological solution being done on global commercial markets, public sector or among hobbyists and user communities [5].

National competence

It is assumed in the RTI that the technological solution is of higher significance and potential to a country, if there is notable, existing R&D knowledge or basic research knowledge in the country in areas that are essentially related to the solution [5].

National access

Based on Michael Porter’s National Diamond model [8], the RTI also assumes that the technological solution is of higher significance to a country if there is a clear, existing connection to a global client base that could benefit from the new technological solution or could utilize it as an add-on to their existing products. The technological solution is of even higher significance to a country if these conditions are fulfilled and the country’s existing position in the potential market segment is very strong [5].

Calculation principle

The breakthrough potential of each of the RTSs is evaluated using a special calculation principle. The relative significance of an RTS is calculated by, firstly, summing the RTS’s impact values on the GVPNs and, secondly, summing the values of the five other factors that impact the potential of the RTS. The two sums are then multiplied by each other. Among the five factors that have an effect on the RTS’s potential, the maturity factor’s importance is highlighted by it receiving double weight compared to the other four factors [5].

The potential values for each factor are:

• 1) Anticipated impacts of the RTS on the GVPNs: values 20, 10, 5, 3 or 1

• 2) The anticipated maturity of the RTS in 2020–2030: values 1–4

• 3) The scientific promise of the RTS: values 0–2

• 4) Global market R&D activity related to the RTS: values 0–1

• 5) National competence related to the RTS: values 0–1

• 6) National access to relevant application areas of the RTS: values 0–3

Based on the above calculations, a list of the most promising technological solutions can be created. The list is ranked into order from the number one highest ranking technological solution to the 100th lowest ranking solution. The highest scoring 25 solutions are given a four-star rating (****), the second highest 25 were a three-star rating (***) and the lowest quarters two stars (**) and one star (*) respectively [5].

Process

The RTI is based on a systematic study of open data sources of the Internet, crowdsourcing and evaluations of experts. The 100 Radical Technological Solutions presented in the first version of the RTI were found through an open online discussion on Facebook, which was hosted by one of the RTI tool’s authors. Approximately 600 persons had registered to the forum and out of those, about 100 were active in suggesting promising technological breakthroughs and related Internet sources. Several of these Internet sources were also published alongside the descriptions of the 100 most promising technologies, in an attempt to improve the transparency and usability of the tool. The authors selected the 100 most promising technological solutions from all the suggestions made by the active members of the forum, and evaluated and ranked them using the indicators and calculation principle described in the previous section. The descriptions of the Global Value-Producing Networks (GVPNs), Radical Technological Solutions (RTSs) as well as the calculation principle were published online on the Finnish Parliament’s website and in a limited number of hardcopies. The Facebook group was left open and had ongoing active discussions related to the recent developments of the 100 described technologies as well as new ones at the time of writing this article [5].

Application of the quality criteria to the Radical Technology Inquirer (RTI) framework

The Radical Technology Inquirer (RTI) is a tool originally developed for national technology assessment purposes in Finland. The first pilot version has recently been published. This section will present an initial analysis of the new methodology’s strenghts and weaknesses in terms of its validity by considering the foresight quality criteria proposed by Kuusi et al [4].

Internal validity of the Radical Technology Inquirer (RTI)

• a) What is the objective of the whole foresight activity? Are there hidden agendas?

The objective of the RTI is to provide a frame for the evaluation of emerging radical technologies. Various lists of emerging technologies are made around the world, but most of them lack a set of consistent criteria according to which the technologies are chosen on the list and how they are ranked into an order of importance. Therefore, their value in guiding any serious technology investments is low. Often the purpose of the listing is to support the agenda of the person creating the list or they are created solely for entertainment purposes. The RTI, on the other hand, has been developed by leading experts in technology assessment and is based on a set of criteria that allows the technologies to be scrutinized. The objective of the RTI is to produce reliable information to guide and support decision-making regarding national R&D investments.

In addition, compared to other lists of technologies, the RTI also presents the customer demand-pull aspect rather than the technology-push aspect alone. The RTI has an inbuilt mechanism to evaluate the Radical Technological Solutions in terms of their relevance to the markets and customers.

• b) What type of activity has to be considered for what type of issues/time spans/knowledge?

The RTI relies heavily on the ‘wisdom of the crowds’ and utilizes the Internet and crowdsourcing to identify a large number of potential breakthrough technologies. A facilitated online discussion was held on Facebook for the pilot version of the RTI. There might be other similar or better ways to organize the information gathering, or Strategic Intelligence, phase of the process. For example, the online discussion was held on Facebook, which might leave out important experts and active contributors, who don’t have a Facebook profile. In addition to possibly leaving out important voices, this method might bring in uninformed opinions. According to Ilmola-Sheppard and Kuusi [9], in virtual environmental scanning processes anonymity of the participants tends to produce more diversity in the data, whereas a scanning process based on social interaction produces less diversity. Against this backdrop, a platform that allows anonymity might produce a different outcome than Facebook where the participants mainly use their real names and identity. Moreover, the Facebook discussion was held in the Finnish language, which again might leave out important contributors due to the language barrier.

The platform on which the crowdsourcing is organized should be well thought out. In the case of the first RTI pilot, the Facebook approach may have supported the idea of citizen participation in the national decision-making process and it may have been the most efficient method for gathering the information. However, using other online platforms either alone or as complementary sources, might increase the variety and scope of the inquiry, and be more suitable for other contexts where the RTI is used. The use of designated technology scouts is common in technology foresight methods used especially in large multinational companies [10–12]. This would ensure that the voices of relevant experts are heard.

• c) What is the scope of foresight? What is the scope of relevant intelligence and sense-making? Is there specific strategic intelligence or are there sense-making projects to be launched? How focused or wide should their scope be?

In the crowdsourcing phase, or the Strategic Intelligence phase, the intention is to retrieve as many suggestions for breakthrough technologies as possible. The idea is to keep the scope as wide as possible, and therefore, there were no limits to, for example, suitable areas of technologies for the RTI. It was in the Sense Making phase that the authors narrowed down the scope of the technologies and the chosen 100 technologies were qualified and evaluated according to the principles described above.

Keeping the scope completely open in the Strategic Intelligence phase tends to produce more diversity in the results [9]. However, leaving the search totally undirected might also leave out important areas of technologies, as the results will depend solely on the crowd’s personal interests. One might argue that to ensure that at least those areas that the organisers think are important are covered, the discussion should be directed and facilitated to a certain extent.

• d) What is an appropriate set of/combination of methods to make use of the strategic intelligence of the specific actors? And how can this be organized?

The clearly defined calculation principle of the RTI enables good use of the crowdsourced information in the Sense Making phase. In the pilot version of the RTI, the authors were responsible of conducting the Sense Making phase. As resources allow, the process might benefit from consulting other experts in the Sense Making phase and complementing the process with other futures research methods such as a Delphi panel with specified experts. Involving the decision-makers in the Sense Making phase might also aid the implementation of the results of the process.

• e) What are the intended outcomes of the different stages in the process? In general, reports are written but often, the activity as such is an outcome. How are the results presented?

The outcomes of the RTI are an online and offline report with direct hyperlinks to the Internet sources of each of the 100 Radical Technological Solutions. In addition, the results are shown on an excel-spreadsheet, which can be used to easily analyse potential impacts of various factors in the frame of the calculation principle. However, this Excel spreadsheet is available from the authors by request only. To improve the openness and availability of the results, the Excel could be made more readily available through an online representation on an open website. Other visual representation models should also be considered in the future to improve the outcome of the tool.

External validity of the Radical Technological Inquirer (RTI)

The first two criteria of external validity are related to the Strategic Intelligence phase of the foresight process. The relevance in criteria 1, 2, 5 and 6 means the relevance of the futures maps to the customers or users of the maps.

• 1. FM1 suggests more possible futures than FM2 that might be relevant from the point of view of the vision or acceptable futures. (Wide scope of possibly relevant paths)

• 2. FM1 is able to identify most relevant futures better than FM2. (Important relevant futures)

Due to the crowdsourcing element, the RTI is able to produce a wide variety of different technological visions. As each of the 100 Radical Technological Solutions (RTSs) consists of many single technologies that aim to solve the same problem, the RTI actually covers several hundred individual technologies. Moreover, as each of the RTSs contribute to enabling one or more of the consumer demand visions defined as the 20 Global Value-Producing Networks (GVPNs), the result is a genuinely wide scope of possible relevant paths to the many different futures scenarios.

The relevance of the RTI is improved by the calculation principle, which includes the factors concerning national competence and national access to markets related to the Radical Technological Solutions. This allows the user of the RTI to tailor the results to his/her own country’s perspective.

The external validity criteria 3 and 4 concern the causal reasoning of the futures maps:

• 3. FM1’s scenarios are causally in line with more futures’ relevant facts than FM2’s scenarios. (More interpreted causally relevant facts)

• 4. FM1’s number of facts that get causal interpretation in scenarios divided by the number of scenarios is higher than in FM2. (Effectively with scenarios interpreted facts)

Especially the Maturity, Scientific promise and Global R&D activity factors related to the Radical Technological Solutions of the RTI concern the past and recent facts related to the technologies. Each of these factors gives indication to how the technology develops in terms of pace and applications. When the RTI process is repeated, the descriptions of the technologies are updated, giving improved indication of the technology’s potential future development and validation to the assumptions of the previous round.

The technological solutions are enablers of the future scenarios described in the 20 Global Value-Producing Networks. The maturity of the technological solutions that enable each GVPN are described, as well as the anticipated challenges of the transition period from the present to the future scenario. Therefore, it can be argued that the scenarios, or the GVPNs, are causally in line with relevant facts, or technological solutions, and that the facts are effectively interpreted with scenarios.

The external validity criteria 5 and 6 are again related to the futures maps’ relevance to their customers or users. These criteria are used to appraise how well the futures maps are communicated and how well they create shared understanding among relevant stakeholders.

• 5. FM1 is understood by more customers than FM2. (Many understand)

• 6. FM1 is better understood by those customers who understand FM2. (Better understand)

Criterion 5 concerns the universal comprehensibility of the futures map. The RTI’s pilot version’s outcomes are a report and an excel-based calculation and ranking of the technologies. A visual representation of the results would greatly benefit the communication of the RTI’s results to a wide audience. Other technology foresight methods have developed visual illustrations of the outcome, which improve the comprehensibility of the results [10, 11]. Criterion 6, on the other hand, concerns how well those (and sometimes preferably only those) who have an interest in the visions of the futures maps understand the results, and for those the RTI provides a very thorough view of the potential of the different technological solutions and allows them to analyse them in depth in terms of their varying competences and access to relevant markets where the solution’s impact is most likely the highest.

Conclusions and suggestions for further research

Due to the increase of future-oriented studies in the recent years, the futures research community is searching for common ground to move the field toward scientific status. The aim of this article has been to offer a pragmatic, user-oriented perpective to the recently proposed quality criteria for futures research by Kuusi et al. [4] by providing a practical use case where the validity of a new futures research methodology is analyzed by applying these criteria.

The criteria provided a framework for a reflective analysis of the Radical Technology Inquirer (RTI) tool. The questions related to the internal validity criteria provoked thoughts, reflection and appraisal of the process of the methodology, but they did not offer rules, guidelines, a checklist or other best practice type of suggestions. As was argued by Kuusi et al. [4], the questions would serve best the planning of a foresight project by leading thoughts to each step of the process beforehand. The external validity criteria offered a list of standards to aim for with the results of each phase of a foresight process, namely the Strategic Intelligence, Sense Making and Selecting Priorities phases/Implementation phases.

Based on the internal validity analysis, one might conclude that the Strategic Intelligence phase of the RTI process could be improved by using an anonymous online platform for increased diversity of data and perhaps by using designated technology scouts as complementary sources of information. The Sense Making phase might benefit from involving decision-makers in the process and the Selecting Priorities/Implementation phase from a more visual presentation of the results. The external validity analysis showed that the RTI might be considered quite strong with regard to criteria 1, 2, 3 and 4, and less strong with regard to criteria 5 and 6.

Originally, the RTI was developed for national level technology assessment purposes. However, the tool might prove beneficial for other sectors as well. Case studies will be conducted by the author to test the usability of the tool in private manufacturing companies. The Quality Criteria could be applied to other technology foresight tools previously implemented in the corporate context [e.g., 10, 11] to understand the strenghts, weaknesses and possible complementary elements of each. The RTI is a very comprehensive and throrough tool and as such, it might prove to be too resource-intensive for smaller organisations to implement. Outsourcing parts of the process, such as the facilitation and management of the online crowdsourcing phase, might make the tool more affordable for smaller organisations. The Strategic Intelligence phase might also be scaled down from the completely open version of the RTI’s pilot, and designated technology scouts might ensure the variety and depth of the information for a smaller organisation. Industry coalitions or clusters might also be able to serve their member companies by undertaking an RTI process and managing the Strategic Intelligence phase or the whole process on their clients’ behalf. If the RTI becomes a repetitive process by the Finnish Parliament’s Committee for the Future, or other countries’ governments for that matter, it could also highly benefit the private sector.

The RTI’s impact could also be strenghtened through a more visual presentation of the results. How the data could be presented in the most understandable and clear way should be studied in further research.