Keywords

Challenge

A plethora of different metrics focused on the evaluation of the maturity of products, systems, frameworks and procedures exist nowadays. Even though significant efforts have been made to integrate widely used frameworks, methodologies and indicators for measuring the maturity of products and systems, there has not been any framework or methodology that integrated multiple readiness level domains under a holistic framework and, most importantly, tailored to security solutions. As a result, the development of a robust scaling framework is of utmost importance to achieve the improved cross-disciplinary assessment of the maturity of novel solutions (developed for security practitioners).

Moreover, there is a need for more efficient use of maturity assessment frameworks to convey technology readiness, synchronize parallel projects, forecast deployment and support decision-making in security investment planning.

The MultiRATE Approach: Focusing on Technological, Manufacturing and Commercialization Aspects

MultiRATE conducts research in multiple fields to create a solid ground for the development of the holistic, homogeneous and harmonized RL evaluation methodology. The fields of research regarding the RLs will be the following: technology (TRL), societal (SocRL), security (SecRL), legal, privacy and ethics (LPERL), integration (IRL), commercialization (CRL) and manufacturing (MRL). In each field, appropriate indicators will be identified per level (e.g. RL1–9) to accurately evaluate the solutions developed for the security domain. Where appropriate, the consortium will identify the indicators for both security practitioners and non-practitioners. Moreover, the project aims to evaluate solutions in all the aforementioned fields by developing a unique methodology and specific indicators for each field based on the different aspects and objectives. MultiRATE will integrate the above RLs under a holistic RL evaluation methodology and tool, which will be used by the EU R&D community to harmonize their approach during security solutions development.

MultiRATE solution is designed in an agile and robust manner, and will be tested in multiple domains (e.g. cybersecurity, border management, fight against crime and terrorism) and through multiple rounds of testing, maximizing its accuracy. In the following sections, three very important readiness level assessors—TRL, CRL and MRL—will be discussed. These can be used as a set for assessing the maturity of a technological solution along the innovation pathway.

Technology Readiness Level

NASA introduced the TRL in the 1970s and had seven levels, which were formally defined in the late 1980s focused on space missions and the relevant technological requirements and components. In the 1990s NASA altered this to a nine-level scale (the lowest level is TRL 1 while the highest is TRL 9), and in 2013 the International Organization for Standardization produced the ISO 16290:2013 standard [Space Systems—Definition of the Technology Readiness Levels (TRLs) and their criteria of assessment] [1]. The focus of the TRL was to evaluate the maturity of developments in space technology. The TRL was introduced in EU-funded projects in 2014 [2] after the term of each level was altered to fit the Horizon 2020 programme, but the overall meaning of each level remained the same. Nevertheless, no assessment guideline or criteria per level were developed, and thus, each R&D consortium was assessing the TRL of the technologies they were developing. MultiRATE developed a concrete methodology, initially focused on assessing the maturity of software solutions and components designed and developed under the horizon Europe Cluster 3: ‘Civil security for society’, based on a set of indicators per level, which can be categorized into four groups as presented below (Table 38.1).

Table 38.1 MultiRATE TRL indicators’ categories
Full size table

Each level is compiled by indicators falling under one of the four categories. Furthermore, to be more simplistic and easier to use, it follows the progression of each indicator from one level to the next level (where relevant) using specific keywords either individually or in combination as depicted in Table 38.2.

Table 38.2 Keywords affecting the status of MultiRATE TRL indicators
Full size table

Manufacturing Readiness Level

In 2003, the Government Accountability Office (GAO) recommended in GAO Report 03-476 [3] to establish cost, schedule and quality targets for product manufacturing early on in technology development in order to obtain process maturity. The report suggests that design and manufacturing knowledge should be obtained early in product development for a product to be successful. In response, the Joint Defense Manufacturing Technology Panel developed MRL definitions as well as manufacturing readiness assessments (MRAs). This MRL scale helps programme managers assess manufacturing risks, which will facilitate the identification of areas that require additional management attention or investment. Manufacturing readiness is as important to the successful development of a system as the readiness and capabilities of the system. Though MRLs were created from the manufacturing perspective to evaluate―the manufacturing readiness of a product and supplement existing TRLs, they, too, have limitations. A limitation of the MRLs is that the lower MRL levels can be difficult to correlate to corresponding TRL numbers due to the technology’s immaturity. MultiRATE adjusts these existing methodologies and develops indicators, specifically for EU projects’ security-related technologies, seeking to harmonize the levels of the MRL with the TRL and the rest of the RLs.

The proposed assessment approach comprises five phases, strategically designed to analyse current conditions and pinpoint manufacturing risks. The primary objective is to empower end users in formulating plans to mitigate or eliminate these risks, facilitating evidence-based decision-making. The development of the MultiRATE MRL framework aligns with the five manufacturing maturity phases, namely Material Solutions Analysis, Technology Maturation and Risk Reduction, Engineering and Manufacturing Development, Production and Deployment, and Operations and Support. These phases are intricately evaluated through the 10-level MultiRATE MRL scale, serving as a tool to gauge the manufacturing maturity of projects within the EU security domain and aid the identification of associated risks.

To enhance the efficacy of the MRL Criteria Matrix, a set of indicators has been established, thoroughly aligned with the criteria. These indicators (accompanied by a total of 68 questions) provide measurable parameters, enabling a more precise evaluation of manufacturing maturity of state-of-the-art technologies in the security field. A high-level overview of the selected eight indicator categories is presented in Table 38.3, aiding project consortia, in comprehending the impact of each one.

Table 38.3 MultiRATE MRL indicators’ categories
Full size table

The utilization of MRLs proves to be an effective methodology for confirming manufacturing feasibility, establishing a predictable schedule, managing and potentially reducing costs, identifying and mitigating risks, enhancing product quality and optimizing manufacturing processes and supply chains. In summary, adopting MRLs empowers end users to track progress, identify areas for improvement and make informed, data-driven decisions to enhance their manufacturing capabilities.

Commercialization Readiness Level

The CRL framework assesses various indicators, which influence the commercial and market conditions beyond just the technology maturity. This enables key barriers to be addressed to support the commercialization of a technology. CRL refers to how ready a product or service is, to go to market as a commercial offering for a group of customers. CRL is a more recent concept, and definitions and operationalization of commercialization readiness are therefore far less widely accepted and solid than TRL. However, the focus on market readiness has become increasingly prevalent as, for example, expressed by Horizon2020s’ sharpened focus on the market aspect of product development [4]. Thus, MultiRATE addresses this by developing an assessment methodology tailored to evaluate the commercial readiness of security-related solutions in European R&D through the identification of specific indicators and harmonization of the CRL with other project’s RLs.

Such a comprehensive methodology assesses market access, financial capital, manufacturing possibilities and users’ profits, emphasizing the commodification of technology. MultiRATE commercialization process, compliant with that of literature and online tools [5,6,7], follows the phases of Ideation and Research, Development and Assessment, and Marketing and Sales. Adapted to the security research domain, it is further analysed in nine steps, including criteria assessment and guiding the user from research idea to a viable market, reducing risk, facilitating evidence-based decisions and ensuring a holistic evaluation of the element’s commercial potential.

The dedicated MultiRATE CRL scale offers a robust framework for evaluating security innovations for successful market deployment, recognizing the pivotal criteria influencing commercialization readiness. These criteria address technological innovation, market potential, IP protection, regulatory compliance, business model, scalability, consortium expertise and financial viability, collectively shaping the commercialization readiness level (Table 38.4). Stakeholders can leverage these criteria for valuable insights, aiding informed decisions about commercialization and funding prospects.

Table 38.4 MultiRATE CRL indicators’ categories
Full size table

Independent of the steps followed during commercialization process, the aspects that have been identified to affect both the level but also the evolution of the process include the market potential and awareness, the regulations and certifications that should be complied with or acquired, the business models and commercialization strategies developed, and the relationships established both with suppliers and customers.

A Technology, Manufacturing and Commercialization Assessment Case

Several stakeholders and use cases that could benefit from the MultiRATE approach have been identified. For example, when writing a proposal, consortium members should identify the RLs of the proposed solutions and the efforts needed to achieve the proposed RL. Another case would be that of project reviews, during which the EU agencies and reviewers (with the support of consortium members) could identify the RL of the developed solutions and compare them with the envisioned or previously assessed RLs.

In order to conduct the assessment of an element’s readiness level, two main aspects should be considered, that of the nature of the element (software, hardware, methodology, etc.) and that of the environment in which it is supposed to operate (e.g. security domain, operational conditions). Based on the above, the most appropriate assessment criteria (as further described in the section above) are identified and a tailored assessment process is defined (i.e. planning, data collection, data validation, attribute rating and reporting).

As depicted in Fig. 38.1, potential users such as consortium members may use the TRL, MRL and CRL by answering specific questions associated with each category of indicators, identifying the corresponding level of maturity for their product. As a very simplified example, if the following questions are true, then the respective indicator is considered complete, and the corresponding maturity level is achieved:

  • Demo solution available for demonstration in an intended operational environment → Yes → TRL6

  • Are manufacturing process and tooling mature? → Yes → MRL6

  • Full and complete understanding of the competitive landscape, target applications, competitive products/services and market has been achieved → Yes → CRL5

Fig. 38.1
The T-M-C R L assessment methodology diagram depicts the assessment within a defined security domain and operational environment. It encompasses elements, consortium partners, indicators for measurement, the T M C assessment tool, resulting R Ls outcomes, and a comprehensive report to stakeholders.

T-M-C RL assessment methodological process

Full size image

A set of questions are provided per indicator category and level of the assessment tool. Considering the specific threshold of indicator needed to be completed, the user will be able to get a report on the levels achieved so far and plan for further activities and efforts that will leverage the maturity of the element assessed, from the technological, manufacturing and commercialization point of view. It must be underlined that achieving a maturity level in the TRL, for example, is not directly linked with a level in the MRL or CRL. Thus, a solution can be more mature in one of the three assessment aspects.

Conclusions

Research and innovation, including that for security and resilience of citizens and infrastructure, proved to be among the most powerful of European policies to boost the union’s economies and competitiveness at the global scale. To support the exploitation and market uptake of security products and services, assessment of the readiness of such R&I projects’ results is very important both by the projects themselves and the European Commission’s services.

MultiRATE identified that EU research projects even in the same domain of security have distinct characteristics and differences in terms of market, stakeholders, financing and innovation development and uptake. These differences reflect the unique challenges and opportunities within each domain (e.g. disaster resilient societies, fight crime and terrorism) and highlight the need for tailored approaches to innovation development and uptake.

The main factors influencing the uptake of EU-funded security research outcomes include the following (Table 38.5).

Table 38.5 Hindering and enabling factors influencing the uptake of EU-funded security research outcomes
Full size table

As presented in this study, an evaluation framework of more than TRL, which when used alone has proven to provide limited or unclear outcomes (e.g. TRL-9 does not mean ready-for-the-market), is of utmost importance in order to have a holistic assessment of the maturity of the project’s solution. Thus, when combined with the MRL, assessing the development process, and the CRL to track and support the pathway to go-to-market, shall provide to users and consortia a useful tool to enhance their solutions maturity, but also to the EC services a more detailed evaluation assessment.