Machine Learning Security and Trustworthiness

Abstract

Machine Learning (ML) is the innovation powerhouse for Intelligent Multimodal Security Systems (IMSS). Along with obvious benefits, ML brings unique risks that require thorough assessment. ML system security builds on "traditional" cybersecurity controls and spreads to cover the expanded attack surfaces for the new ML assets. Moreover, the problem of ML trustworthiness stands in the way of taking full advantage of ML advancements. This chapter introduces the challenges and risks of ML, highlights key trends in the global ML policies, and best practices, summarizes key standardization activities, and provides a detailed description of assets and threats. Putting the above into perspective, we present a practical framework for addressing ML-based IMSS security and trustworthiness and discuss relevant implementation options.

“A robot may not harm humanity, or, by inaction, allow humanity to come to harm.”

—Zeroth Law of Robotics, Isaac Azimov

Machine Learning (ML) is the innovation powerhouse for Intelligent Multimodal Security Systems (IMSS). Along with obvious benefits, ML brings unique risks that require thorough assessment. ML system security builds on “traditional” cybersecurity controls and spreads to cover the expanded attack surfaces for the new ML assets. Moreover, the problem of ML trustworthiness stands in the way of taking full advantage of ML advancements. This chapter introduces the challenges and risks of ML, highlights key trends in the global ML policies, and best practices, summarizes key standardization activities, and provides a detailed description of assets and threats. Putting the above into perspective, we present a practical framework for addressing ML-based IMSS security and trustworthiness and discuss relevant implementation options.

From myths and legends to modern day fiction, the idea of artificial intelligence (AI) has always fascinated people. The 1950s mark the symbolic birth of AI with the workshop held in Dartmouth.¹ It took another 50 years for the industry to reach the point predicted by Moore’s law to provide enough computing power to enable complex neural net computation algorithms capable of approaching human accuracy for limited tasks. At the same time, the exponential growth of data triggered the demand for technologies that can better utilize data for making business decisions. This became the catalyst for Machine Learning (ML), the applied field of AI focused on building applications that can learn from data, make predictions based on the data, and improve the accuracy of decision over time without explicitly changing the applications.

ML-based innovation rapidly grows and has a potential to generate immense economic value giving rise to an entirely new set of services. Several analysts, including Garner’s Hype Cycle,² and Advanced AI and Analytics IDC, forecast that connected IOT devices, are expected to generate about 80ZB of Data in 2025.³ Accumulation of enormous data paves the path for faster adoption of ML technologies across a broader spectrum of market segments. It has been widely publicized that more and more businesses across the world are increasing investments into ML research and solutions. Today we witness active engagement and adoption of ML to gain a competitive edge across a variety of market segments, e.g. retail, healthcare, industrial and many others. Organizations adjust and transform business processes to enable ML-based operations. The rapid adoption acts as a stimulus for further development of ML technologies, such as Tiny ML,⁴ Quantum ML,⁵ and Auto ML.⁶ The latter comes with the promise to democratize ML for broad adoption, making it accessible for the organizations who don’t have technical expertise to create ML-based solutions.

As any rapid technological innovation ML not only offers business and customer benefits but also introduces security risks due to the lack of security considerations during the early phases of ML solutions development. This includes a new class of risks that are unique to ML and require mitigations beyond what traditional cybersecurity practices. While the industry embarks on AI-powered digital transformations, those security risks may not only become a hurdle for adoption and scale of ML solutions but also can lead to significant negative consequences for enterprises (brand, compliance, financial loss), individuals (physical and safety risks) and nation’s security (functioning of critical infrastructure).

The intense demand for real-time analysis and handling of a large volume of complex, sensitive data drives the need for ML processing at the edge; closer to the data origin at smart endpoints. Computing at the edge presents unique challenges and requirements for the ML solution. Proliferation of the ML at the edge triggered security oncerns in society that resulted in several policy initiative emerging across governments, the private sector, trade organizations, and alliances.

Usage of Machine Learning in IMSS

A fast adoption of cameras across the industry resulted in a drastic increase in the amount of video data available for operation and analysis. Over the last decade, the rapid proliferation of technology-enabled high-quality video streaming solutions at affordable cost, are widely used in consumer, commercial, and industrial settings. According to the IDC report,⁷ we expected solid growth for the video surveillance camera market as smart camera systems and analytical software enable new use cases. In 2020, despite the COVID-19, 82% of 22.6ZB of data created by IoT devices is from surveillance systems and this trend will continue to increase. More video data presents greater opportunities to analyze it, monitor risks, predict events, and recommend corrective actions. However, video data is mostly unstructured, and therefore traditional algorithms are not able to perform the predictive tasks on this type of data. AI/ML unleashes the power of unstructured data processing, making it usable for numerous new and exciting use cases beyond traditional video surveillance.

Using machine vision and object recognition makes checkout-free frictionless shopping possible. Amazon Go utilizes video observation and smart shelving in more than 20 new stores in the United States and recently introduced a new store in the United Kingdom to enable customers to autonomously walk through the store and select items for purchase. Upon finishing, purchases in the shopping bag will be processed and an electronic receipt will be sent to the customer when they leave the store. Amazon Go⁸ uses the same types of technologies that are found in self-driving cars, such as computer vision, sensor fusion, and deep learning. The combination of the AI, computer vision, and sensor data ensure that customers are only charged for the products that they finally picked, detecting when products are taken or returned to the shelves. It is recognized that the COVID-19 pandemic drastically changed social behaviors, including shopping experiences. Approaches based on AI and computer vision for frictionless retail shopping have obvious benefits in an age of social distancing, limiting human interaction, but they also speed retail transactions and help brick-and-mortar businesses be more competitive with online shopping.

Another exciting use case involves video data analytics to detect and correct problems in supply and distribution channels. Machine vision solutions coordinated with back-end systems track the inventory as it moves through the supply-chain, monitoring the quality and timeliness.

Manufacturing is also benefiting from machine vision and, specifically, computer vision. It has been used for inventory control, spectral image analysis of chemicals, maintenance and repair operations, and intelligent robotics. To address the performance issues with manufacturing equipment, local or cloud analytics perform equipment telemetry and predict when failures might arise. Cameras are now being embedded not only into the assembly line but also into labor management (time reporting, etc.) as well as warehousing and distribution.

Video surveillance technologies can be a great benefit for the healthcare domain. Thermal imaging combined with the appropriate analytics assesses the health of people entering and leaving controlled areas. Patient tracking in hospitals lets healthcare providers reach patients and improve efficiency in healthcare facilities. Video analysis is also being used as a diagnostic tool, a use case that has become far more popular in recent years. By combining video with analytics tools, healthcare providers can rely on artificial intelligence to aid their diagnostic capabilities. Advanced medical imaging can analyze and transform images and model possible situations. One of the well-known solutions in this space was brought to market by SkinVision,⁹ which enables one to find skin cancer early by taking photos of skin with a phone and getting to a doctor at the right time. AI-powered medical imaging is also widely used in diagnosing COVID-19 cases and identifying patients who require ventilator support. Intel® AI Builders member Huiying Medical¹⁰ has developed a medical imaging diagnostic solution that uses CT chest scans to assist with the early detection of coronavirus infections that complement standard lab testing with 96% accuracy.

These use cases have evolved even further to utilize multiple sensors such as vision, audio, and smell as Multisensory Surveillance solutions. Figure 5-1 Multisensory Surveillance Solution Evolution¹¹ demonstrates the 2017–2019 timeline.

Figure 5-1

Multisensory surveillance solution evolution

The COVID-19 pandemic introduced further demand for thermal cameras and solutions that aggregate video and thermal sensors. Thermal body temperature solutions are supposed to detect changes in human skin temperatures, identify individuals that may have a virus, detect face masks, and provide organizations with an additional layer of protection to their facility from increased exposure to the coronavirus.

Challenges and Risks

Traditional programming methods for multisensor fusion struggle with ambiguous and conflicting inputs. AI and ML solutions are providing more robust solutions for these difficult problems. There is no shortage of headlines about the unintended consequences of new AI/ML adoption when AI/ ML systems are tricked, misled, circumvented, and therefore fail. However, many vendors are still largely unaware of the risks they are taking when they are adopting cutting-edge AI/ML technologies.

First, let’s view the AI/ML and Security intersection. There are three major perspectives to explore when considering how AI is impacting security:

• 1. AI Systems Protection. This requires securing the AI System’s assets, including training data, training pipelines, and models.

• 2. AI for Cybersecurity. This implies the use of AI/ML tools and techniques to autonomously identify and/or respond to cyber threats based on behavior analysis and automation of cybersecurity practices while augmenting the actions of human security analysts.

• 3. Anticipated nefarious use of AI by attackers. The benefits of AI can be used for malicious purposes. Identifying these attacks and defending against them will be an important addition to traditional cybersecurity practices.

In this book, we focus on the AI systems’ protection aspects. Protecting AI-Powered Systems presents new attack surfaces and thus increases security risks. Leaders who plan to use advanced AI and ML analytics, must understand the regulatory guidelines, comprehend the risks, and take appropriate measures to address the risks, some of which were documented when Mckinsey conducted extensive research on artificial intelligence risks.¹² According to the study, AI/ML systems involve a very broad range of stakeholders and include not only the organizations, consumers, or individuals but also human society at large and the environment. The following Figure 5-2 is the summary of the multifaceted risks for individuals, organizations and society highlighted in Mckinsey research.

Figure 5-2

Unintended consequences of AI (Mckinsey)

These risks span across the entire life cycle of AI/ML solutions and arise from the data used to train the AI/ML system, as well as from the design and operation of the systems. Organizations that plan to adopt the AI/ML technology should implement specific AI risk management practices. These practices leverage existing industry-recommended principles such as ISO 31000:2018 Risk Management Guidelines¹³ and emerging ones that intend to address the complexity of the supply chain of AI/ML systems and challenges of Data collection for large volumes of AI training and inferencing data, including texts, images, videos, audio, geo-local data, and other data modalities

The enormous growth of data introduces the challenges for proper data management. According to IDC,¹⁴ IoT devices will generate up to 79.4 ZB of data by 2025. The consumption of this massive data is a complex problem, as the data needs to be triaged for sensitive information and conform to the privacy regulations, such as the pivotal European Union’s General Data Protection Regulation (GDPR) and several US state-level initiatives, including the California Consumer Privacy Act (CCPA). Failure to follow privacy guidelines introduces huge reputation risks as well as financial penalties. When Hackers attacked cloud-based enterprise security startup Verkada’s video surveillance cameras, they gained access to over 50,000 of the IoT devices and customers’ associated video archives. The victims of this attack include such Verkada customers as a Tesla suppliers, Cloudflare, Equinox, various hospital networks, police departments, correctional facilities, and schools. This attack raised an intense dispute in society about how businesses, health care, and public safety workers monitor and treat people, as well as how they gather, store, and use data. In addition, many Verkada customers were concerned about Verkada’s employees’ access, raising privacy concerns.

Gartner Research published a recommendation¹⁵ highlighting the risk of data manipulation, “Machine learning presents a new attack surface and increases security risks through the possibility of data manipulation.” Solution developers and adopters must anticipate and prepare to mitigate potential risks of data corruption, model theft, and adversarial samples. Another scenario to consider is when the decision produced by ML will be translated and actuated in the physical world. A popular approach for data processing and decision-making systems is reinforcement learning. Reinforcement learning is a form of machine learning where the acting AI interacts with its environment and tries to maximize the reward that is determined by a function that, if determined correctly, should incorporate all goals (Sutton and Barto, 2017). This approach expands the perimeter of the system; along with inferencing, it includes online training susceptible to data manipulation that causes model drift. Thus, it substantially increases the surface of failure.

Data is a crucial component to developing every AI, therefore limiting access to sensitive data is the basis for building trustworthy AI/ML. As for any compute solution, AI/ML will be prone to security risks. If security measures are insufficient, it will be possible to plant an attack that forges the results and impacts recommendations, or decisions made by solution.

AI/ML models represent the core of the solution, and their implementation can introduce problems when they deliver biased results, become unstable, or produce results that are explainable or traceable. In 2019, Apple Card was investigated after gender discrimination complaints. When the husband and wife applied for Apple Card and compared their spending limits, they found out that the husband got 20 times more credit limit than his wife.¹⁶ Safety represents one of the most critical risks connected with AI. The death of Elaine Herzberg was the first recorded case of a pedestrian fatality involving Uber’s self-driving car, when the machine learning system failed to account for jaywalking¹⁷ and the vehicle operator was distracted, failing to monitor the AD system. As a result, not only did Uber cease its testing program in Arizona but also caused its rival, Google’s Waymo, to increase safety requirements.

Risks with AI models misbehaving become more prominent with large scale AI/ML deployments, where service providers aggregate a big number of pre-trained models from a diverse set of community developers. From this new service perspective, model theft attacks (in this case of intended misuse) are the most vital problem to be solved from a business standpoint. There were several cases when researchers demonstrated methods of “recreating” model IP. Two researchers, Aaron Gokaslan and Vanya Cohen,¹⁸ managed to replicate the OpenAI model that intends to automatically generate text. Initially, OpenAI did not completely disclose the underlying model due to concerns about malicious applications of the technology. Another interesting example is the study¹⁹ that describes the successful model theft attack on an intelligent IoT medical platform. Attacks like these demonstrate that intellectual property such as the trained AI models using private and sensitive information can be stolen and, as a result, the business could sustain both brand damage and investment losses due to weak AI/ML system protections.

For AI/ML supply chain vendors who build and deploy solutions, it is critical to comprehend the full range of risks of intelligent solutions introduced by specific AI/ML technologies that can lead not only to theft of the models and of the input training data sets but also incorrect results.

Policy and Standards

In this section, we will highlight the major trends, rapidly evolving worldwide policies and standards for IMSS. As mentioned earlier, IMSS is an essential segment of evolving IOT edge usages. Various forms of sensor-driven data processing, including computer vision and other forms of video data use cases contribute to the growth of IoT devices, edge infrastructure, and solutions. The widespread adoption of IoT/Edge devices, including the IoT deployment in high-risk environments, the expansion of the attack surface made IoT and Edge attractive targets for malicious actors that resulted in a surge of high-visibility attacks such as DDoS Mirai,²⁰ Strontium APT,²¹ Mozi,²² and many more. It started from poorly secured/configured IoT devices, Botnets, and evolved to high-profile distributed Denial of Services attacks, the most prominent one was Mirai in 2016 which targeted a wider range of IoT Devices. Botnets built from the Mirai codebase (e.g., Mozi) continue to disrupt in the IoT arena,²³ with cyberattacks taking advantage of loopholes in IoT device security to plant widespread attacks. Experts forecast²⁴ that even more nefarious threats will result from the surge of the attack volume and sophistication. Ransomware, physical attacks, sensitive data leakage, and attacks on privacy are some of the new attacks that emerged on the IoT horizon (Figure 5-3 IoT Threat Landscape).

Figure 5-3

IoT Threat landscape

These trends incentivize policymakers to propose a specific regulatory framework to address raising security and privacy concerns resulting in global industry-wide collaboration on the definition of IoT/Edge security frameworks.

Another major factor that contributes to global policy and standards activities is AI/ML technologies. For many decades, AI was limited in adoption; however, with the rise of ML that widened and enhanced the applicability of AI/ML technology, we see AI/ML solutions in almost every market – physical security, healthcare, industrial, retail, financial services, smart city, energy, utility transportation, agriculture, and many more.

The breadth of AI/ML technologies is expanding to leverage complex heterogeneous multi-party compute and at the same time, as it is for many innovative technologies, it is revealing personal, societal, and economic risks that can be exploited by malicious players. Over the past several years, several companies and institutions produced guidelines that proposed principles for artificial intelligence. Harvard’s Berkman Klein Center conducted the analysis²⁵ of 35 documents on Principles worldwide and in January 2020 published the paper “Principled Artificial Intelligence: Mapping Consensus in Ethical and Rights-based Approaches to Principles for AI.” This study systemized and visualized the key convergence trends covering segments such as government (US National Science & Technology Council, China, UK, Germany, G20, EU Commission, OECD), Private Sector and trade organizations (prominent players such as Microsoft, Google, Tencent, IBM, ITI, Telefonica), and think tanks such as academics, alliances, and civil society (e.g., IEEE, AI Industry Alliances, Academic institutions). Figure 5-4 Harvard AI Guidelines Analysis, illustrates in the following the study represented in a wheel shape where documents are represented by spokes with the highlighted scale of themes coverage. This project outlines very important trends. First, there is a strong convergence across the wide variety of industry efforts. Second, researchers identified eight common key themes for AI Principles:

• 1. Privacy

• 2. Accountability

• 3. Safety and Security

• 4. Transparency and Explainability

• 5. Fairness and Non-discrimination

• 6. Human Control of Technology

• 7. Professional Responsibility

• 8. Promotion of Human Values

The major conclusion is that these highlighted principles lay the foundation for measurable AI practices for both policymakers and technical professionals. The significance of this study is in the demonstration of the industry’s demand and readiness to embrace the complex challenging problems of AI. Understanding the industry momentum around AI is critical for the IMSS developers.

Currently industry is embracing the journey from the general declaration of AI principles to implementable practices. IDC,²⁶ Gartner²⁷ analysts predict that companies will redesign their AI/ML systems to address explainability, fairness and operationalize the AI development pipeline to meet regulation requirements. Legislators worldwide move to adopt regulation by design. Regulation is a major way in which the government influences the market; however, traditionally, the market doesn’t like these interventions.

It is critical to make sure that regulation is efficient and doesn’t impede business. AI/ML adopters face a dilemma. Evolving regulatory frameworks might significantly impact the business ability to adopt and scale the technology. At the same time, regulation is at the conceptualization stage when technology is still maturing, the industry is still accumulating the knowledge of impacts and finally, new laws and best practices are still evolving. When would be the right time for IMSS technology developers and adopters to mediate? The specific action at this stage is to comprehend the policy and regulatory factors and focus on concrete actions that they can undertake to ensure the IMSS doesn’t violate any existing and emerging laws and regulations.

Figure 5-4

Harvard AI guidelines analysisFootnote

https://cyber.harvard.edu/publication/2020/principled-ai

To address effective AI/ML system governance goals, regulators and lawmakers rely on standards organizations to support policymakers’ goals, besides regulation presents limited opportunities for experts to engage and provide feedback. Two major international standards bodies that are currently developing AI standards, International Standards Organization (ISO)/International Electrotechnical Commission (IEC) and the Institute of Electrical and Electronics Engineers (IEEE). JTC1 (joint ISO/IEC committee, established in 1987) SC42²⁹ subcommittee serves as the focal point for AI standardization development and has already published ten standards and more than 20 are in development. A diverse stakeholder ecosystem calls for close industry collaboration across domains (e.g., IoT), considering the usage of AI/ML technologies in the context of market and computation approaches of AI Systems. This is a very complex problem: the standardization approach cannot be conducted in a single area, and it requires interoperability approaches that go beyond current solutions. SC42 established a liaison collaboration with SC38³⁰ (Cloud Computing and Distributed Platforms is a standardization subcommittee) and SC27³¹ (Information security, cybersecurity, and privacy protection is a standardization subcommittee). SC27 developed several widely adopted standards for security controls, services, cryptography, security evaluation/testing, and privacy technologies. Given the wide range of issues brought up by AI/ML, it is important to highlight the importance of trustworthiness, security, and privacy within the context of AI usage. Along with international efforts, it is important to consider the efforts of national bodies, such as the National Institute of Standards and Technology (NIST) in the United States, and the European Telecommunications Standards Institute (ETSI) in the European Union.

The developers of IMSS solutions should consider composite policy implications. Figure 5-5 IMSS Policy Landscape summarizes the policy domains: IoT specific regulation, global privacy protection trends, AI/ML trustworthiness, and risk management.

Figure 5-5

IMSS policy landscape

An independent expert group that advises the European Commission recommends banning the usage of AI in some “unacceptable” scenarios, for example, facial recognition and job application as they present great potential risk to society and individuals.

Regulatory Landscape

As it was mentioned, governments around the world launched national AI legislation activities (for example, European Union (EU), China, the United Kingdom, Canada).

The EU is the most prominent, advanced, risk-based approach to AI systems. In 2021 the European Commission (EC) introduced the first ever complex regulatory framework on AI, “Proposal for a Regulation laying down harmonized rules on artificial intelligence.”³² In 2021, Cyberspace Administration of China (CAC) passed the “Internet Information Service Algorithm Recommendation Management Regulations” that regulates the implementation and use of recommendation algorithms. There is direct implication for the AI/ML empowered IMSS; it tackles the problems of the transparency of algorithms function, discriminatory data practices, opaque recommendation models, and labor violations.

Both the EU and Chinese legislations will have a deep impact both on the global economy and lawmaking. If the EU passes the AI law, this will become mandatory for anyone who wants to operate in the EU market as well as based on learnings from the General Data Protection Act (GDPR), it will be leveraged by other nations for their legislative strategy. In the United States, there are several AI regulatory activities across agencies; to name a few – Department of Commerce (DoC), Federal Trade Commission (FTC), and the Federal Drug Administration (FDA). State lawmakers also are considering the benefits and challenges of AI.

A harmonized approach will be critical for AI/ML systems adoption and scale. Regulatory frameworks need to evolve to provide clear guidelines for AI solution adopters to avoid misinterpretation and reduce the compliance cost. At the same time, there is strong concern about the regulatory intervention to the technology space that can hamper innovation. It is critical for technologists to comprehend the legislative approaches and contribute through standards, best practices to avoid overregulation and achieve a balanced risk-based framework.

IoT Security Baseline

The IoT regulatory and policy landscape are rapidly evolving. There is a growing number of state and federal legislation, best practices, and standards that span numerous market segments and countries include the EU, the United Kingdom, Japan, Brazil, Australia, and many others. For the United States, the IoT Cybersecurity Improvement Act of 2020 was pivotal because it defines that all IoT devices used by government agencies have to comply with US NIST-(National Institute of Standards and Technology)defined standards. This is not only a critical step for the security of solutions used by the US government, but it is expected that this security guideline will define the requirements for the broad segments of consumer and commercial IoT vendors and devices.

In 2016, NIST established the cybersecurity IoT program with the mission to “cultivate trust in the IoT and foster an environment that enables innovation on a global scale through standards, guidance, and related tools.”³³ This program drove several key initiatives (Figure 5-6 NIST CYBERSECURITY FOR IOT PROGRAM (Source: NIST). NIST 8259 Series guides for manufacturers and supporting parties creating IoT devices and products. SP 800-213 is intended for Federated Agencies looking to deploy IoT devices in their systems. Consumer IoT Products are addressing Executive Order 14028.

Figure 5-6

NIST cybersecurity for IOT program (Source: NIST)

NIST recommendations are results of collaborative industry-wide effort that was conducted by partnering with industry experts and in the spirit of facilitating the harmonized approach.

Collection of NISTIR 8259 reports (see Figure 5-7 NIST 8259 Series Roadmap and Federal Profile (as of January 2021) provides guidance for IOT device manufacturers on how to design, develop, and maintain IoT devices with foundational security.

Figure 5-7

NIST 8259 series roadmap and federal profile (as of January 2021)

Three final documents have already been released:

• NISTIR 8259: Recommendations for IoT Device Manufacturers: Foundational Activities³⁴

• NISTIR 8259A: Core Device Cybersecurity Capability Baseline³⁵

• NISTIR 8259B: IoT Non-Technical Supporting Capability Core Baseline³⁶

These recommendations and baseline help enhance cybersecurity for all new IoT devices. One of the major complexities is the range of the IoT device compute power with a large portion of the IoT market consisting of low or medium-complexity devices so called constrained devices. This complexity makes NIST recommendation even more important as it defines the baseline that is applicable across all IoT devices and that is recognized globally. A consensus baseline grounded in international standards with broad support across the industry will help enable interoperable IoT security policies worldwide.

NISTIR 8259 guidance addresses our IoT Security problem directly by providing steps that manufacturers should follow, which are grouped into two phases; pre-market, before the device is sold, and post-market, after the device is sold (see Figure 5-8 NIST recommended activities for IoT device developers, Source: NIST).

Figure 5-8

NIST recommended activities for IoT device developers. Source: NISTFootnote

https://nvlpubs.nist.gov/nistpubs/ir/2020/NIST.IR.8259.pdf

NISTIR 8259A publication outlines six technical capabilities, cross-referenced with applicable industry and federal standards, as a default for minimally securable IoT devices.

• Device identification: The IoT device can be uniquely identified logically and physically.

• Device configuration: The configuration of the IoT device’s software can be changed, and such changes can be performed by authorized entities only.

• Data protection: The IoT device can protect the data it stores and transmits from unauthorized access and modification.

• Logical access to interfaces: The IoT device can restrict logical access to its local and network interfaces, and the protocols and services used by those interfaces, to authorized entities only.

• Software update: The IoT device’s software can be updated by authorized entities only using a secure and configurable mechanism.

• Cybersecurity state awareness: The IoT device can report on its cybersecurity state and make that information accessible to authorized entities only.

NISTIR 8259B complements 8259A with guidance on nontechnical processes that manufacturers should implement that support IoT device cybersecurity, such as documenting updates, information collection and dissemination practices, and training for customers on how to implement them.

Together NISTIRs 8259 A and 8259B are a complementary pair, providing balance of technical and non-technical requirements, and giving comprehensive guidance for manufacturers executing the six activities outlined in NISTIR 8259 (see Figure 5-9 NISTIR 8259 IoT Security Technical and Non-Technical Baseline (Source NIST)

Figure 5-9

NISTIR 8259 IoT security technical and non-technical baseline (Source NISTFootnote

www.nist.gov/itl/applied-cybersecurity/nist-cybersecurity-iot-program/nistir-8259-series

)

The draft NISTIR 8259C “Creating a Profile Using the IoT Core Baseline and Non-Technical Baseline”³⁹ describes a process that can be leveraged by organizations to build IoT cybersecurity requirements set for particular market ecosystems, customers, applications, and/or environments. It starts with the core baselines provided in NISTIR 8259A and B and explains how to integrate those baselines with vertical or application-specific requirements to develop a profile suitable for specific IoT device usages This can be used for customer’s vertical segment requirements, industry standards, or regulatory guidance.

SP 800-213, IoT Device Cybersecurity Guidance for the Federal Government: Establishing IoT Device Cybersecurity Requirements,⁴⁰ accumulated 8259 learnings and provides guidance for federal organizations in defining their IoT cybersecurity requirements. It demonstrates the results of applying the NISTIR 8259C process in a federal government customer space, where the requirements of the FISMA (Federal Information Security Management Act of 2002) and the SP 800-53⁴¹ security and privacy controls catalog are the essential guidance. SP 800-213 includes SP 800-213A, the IoT Device Cybersecurity Requirements Catalog, a set of technical and non-technical cybersecurity controls defining IoT device capabilities and supporting non-technical actions that a federal agency can apply in documenting their IoT cybersecurity requirements. The catalog includes mappings to SP 800-53 and NIST Risk Management Framework⁴² as well as to IoT cybersecurity baseline controls set in SP 800-53B that’s derived from 8259 publications. Federal Profile provides comprehensive IoT market segment Cybersecurity guidance for device developers.

In November 2020, Consumer Technology Association published ANSI/CTA 2088⁴³ Baseline Cybersecurity Standard for Devices and Device Systems. This standard specifies baseline security requirements and recommendations for consumers facing IoT devices and systems of different complexity and focuses on addressing the adversarial impacts of the botnets and other security threats. These standards were developed in alignment with NIST guidelines.

On the European arena, it is important to learn about European Union Agency for Cybersecurity (ENISA) Baseline Security Recommendations for IoT,⁴⁴ and European Standards Organization ETSI 303-645 standard for the Cyber Security for Consumer Internet of Things: Baseline Requirements.⁴⁵ ENISA’s recommendation was developed in 2017 with the goals to specify the security requirements of IoT, map critical assets and relevant threats, assess attacks, and identify practices and security approaches to protect IoT systems in the context of Critical Information Infrastructure. This document has good foundational information on cybersecurity for the IoT systems consumers and provides an elaborated list of security measures and good practices, to mitigate the threats, vulnerabilities, and risks identified in the study that affect IoT devices and environments. This recommendation could be applied to the different IoT segments and environments and deployments in a horizontal manner as a baseline, in contrast to the vertical-specific recommendation.

ETSI 303-645 is a standard specifically designed for consumer IoT devices, including smart cameras and other smart household devices. It contains a set of security and privacy requirements and recommendations that manufacturers should follow to build secure products. These requirements are split into the following 13 categories:

• 1. No universal default passwords.

• 2. Implement a means to manage reports of vulnerabilities.

• 3. Keep software updated.

• 4. Securely store sensitive security parameters.

• 5. Communicate securely.

• 6. Minimize exposed attack surfaces.

• 7. Ensure software integrity.

• 8. Ensure that personal data is secure.

• 9. Make systems resilient to outages.

• 10. Examine system telemetry data.

• 11. Make it easy for users to delete personal data.

• 12. Make installation and maintenance of devices easy.

• 13. Validate input data.

In addition, ETSI EN 303 645 standard provides recommendations for data protection requirements to help manufacturers protect users’ personal data.

With a plethora of emerging IOT Cybersecurity recommendations and standards, it is important to have a harmonized approach that is intended for the broad IoT industry, providers, manufacturers, consumers, and regulators. This is a consistent message from NIST and ENISA. Over 20 other industry groups and technology organizations came together to develop a global, industry-driven consensus on IoT security baselines convened by the Council to Secure the Digital Economy (CSDE)⁴⁶ and contributed to the NISTIR 8259 series. The International Standards Organization (ISO) expert group is working on defining the IoT security and privacy device baseline requirements standard (ISO/IEC 27402⁴⁷) that aims to address this important problem of the connected IoT world.

The US Food and Drug Administration (FDA) regulates the use of AI in covered healthcare products and plays an important role in ensuring the safety and effectiveness of those products. In January 2021, FDA introduced the Artificial Intelligence/Machine Learning (AI/ML)-based Software as a Medical Device (SaMD) Action Plan.⁴⁸ SaMD covers a wide range of AI-enabled products such as applications that run on smartphones, but also software that can detect and analyze stroke based on CT images of the brain or diagnose fractures based on X-ray imaging. FDA plan considers the entire lifecycle of a device, promotes transparency, real-world performance monitoring, and methodologies to assess algorithmic bias. Racial, ethnic, and gender bias is one of the key problems for the healthcare industry, that is even more critical for AI efficiency and safety.

It is important to consider that these standards apply to a wide range of IoT device types. Minimum compliance may be adequate for many devices, but IoT devices that protect valuable Machine Learning models theft and from tampering that will damage trust will require the highest levels of security.

Privacy Compliance

Privacy considerations are critical for AI/ML solutions. IMSS that support ML-based solutions must comply with generic privacy regulations and practices. While there are many local and national privacy regulations, the most dominant regulation in privacy is the European Union’s General Data Protection Regulation (GDPR). It defines the most widely used and enforced regulatory definition for privacy.

In April 2020, the United States Federal Trade Commission (FTC) published guidance⁴⁹ on using Artificial Intelligence and Algorithms. Later, in January 2021, the FTC took extraordinary law enforcement measures⁵⁰ after finding that artificial intelligence company Everalbum, the maker of the “Ever” photo, had deceived customers about its data collection and use practices. FTC ordered Everalbum to delete or destroy any ML models or algorithms developed in whole or in part using biometric information it unlawfully collected from users, along with the biometric data itself.

In addition to these, there are market segment-specific regulations, such as Health Insurance Portability and Accountability (HIPAA) for health care. Consumer Technology Association (CTA) has also developed voluntary privacy principles⁵¹ for health data not covered by HIPAA (Health Insurance Portability and Accountability Act of 1996).

US privacy law proposals increasingly focused on AI accountability and discrimination, and continue to evolve. Keeping personal health information private and protected is a core component of trust. This is critical for AI/ML usage for health care where personal health information is the raw data for most AI systems. Consumers expect that personal data will be protected and want an assurance that organizations will keep their information confidential.

One of the privacy compliance implications is that the model/workload needs to have the ability to be revoked, for example, re-evaluate the right to use at any given time, based on noticing criteria have changed or (at least) operator direction. Depending on the platform application, this may require restarting the platform or the main application on the platform (e.g., the application may require continuous service, such as electric power).

A model licensing implementation which causes the model to be verified once and then operates continuously (potentially forever) would not be able to comply with an order to remove the model based on this kind of criteria.

Personal data in the healthcare systems can be divided into two categories. The first category is data collected by health plans and health providers, which is directly protected by HIPAA. The second category comprises such health information which has been collected by individuals or wellness data they have generated using health applications. While this second category is not protected by HIPAA, this protection is critical to ensure consumers trust the service providers and use the applications and services.

By providing information on how personal health information is used, shared, stored, and managed, organizations can promote the consumer trust necessary to encourage AI use.

HIPAA details some requirements for the secure storage of data and notification in the event of a security breach, but some organizations who provide AI-based services may not be covered by HIPAA. For example, an article in USA Today⁵² demonstrated that data profiles from health tracking devices are shared with employee companies, as well as with healthcare providers who manage corporate wellness programs. As a result, Personal Identifiable Information (PII) data could later be used to identify who they are and link their identities to detailed medical profiles that can be bought by companies, researchers, or anyone else. Consumers may not be aware of the scope of HIPAA coverage and, as a result, don’t realize that the information, they share is not protected by healthcare regulations (note, it doesn’t exclude coverage by other regulations). It is expected that manufacturers and sellers of AI solutions will likely be subject to future regulations if they are not already covered by HIPAA.

Additionally, there are often regional privacy regulations that apply to IMSS. OEMs, consultants, system integrators, and system operators should monitor and seek legal advice on applicable local regulations. It is recommended that developers should consider data security whenever information is collected and stored. Organizations should assess their risk of breach and take steps to minimize the risk of exposure.

GDPR and Emerging EU AI Regulation

European Union GDPR⁵³ (General Data Protection Regulation) is a prominent data privacy and security law with significant implications for businesses both in the EU and globally. AI is not explicitly outlined in the GDPR guidelines, but many provisions in GDPR directly apply to AI. According to the study conducted by the European Union Panel for the Future of Science and Technology,⁵⁴ AI system developers should consider the values and principles of the GDPR and “adopt a responsible and risk-oriented approach.” However, given the complexity of the AI and ambiguities present in the GDPR, it is not straightforward for the AI system ecosystem to translate it into practices. This paper indicates that further development is required to generate appropriate responses, based on shared values for stakeholders (industry and society) and effective technologies. According to the study, consistent application of data protection principles, when combined with the ability to efficiently use AI technology, can contribute to the scalability of AI applications, by generating trust and preventing risks.

In essence, AI breaks the traditional data processing assumptions reflected in GDPR. This conflict between AI and GDPR is specific to the field of data processing, but not new given the known dichotomy between innovative technologies and regulation and requires both sides to absorb and adopt. Several publications that discuss the intersection of GDPR and AI, outlining that GDPR provides limited guidance on how to achieve data protection. In 2019, Andre Tang published “Making AI GDPR compliant”⁵⁵ addressing the challenges with GDPR compliance for AI solutions. Among highlighted conflicts (see Figure 5-10 AI and GDPR conflicts and possible remediations. Source ISACA) is the accuracy of automated decision-making, the right for erasure, data minimization, and the transparency principle. Some of the recommendations make sense to implement and we see the industry already moving in that direction (for example, federated Learning and Transfer Learning), others such as “rights to erasure” would be challenging to implement if we think about broad-scale deployment of models developed based on the personal data. Data minimization should not exclude the use of personal data for machine learning purposes. GDPR should not prevent the creation of training sets and the building of AI models whenever data protection rights compliance is addressed.

Figure 5-10

AI and GDPR conflicts and possible remediations. Source ISACA

AI/ML Trustworthiness

Trustworthiness Journey

Trustworthiness has emerged as a fundamental security concept addressing AI systems being worthy of physical, cyber, and social trust. In 1999, the Trust in Cyberspace National Academies report⁵⁶ introduced the foundations of trustworthy computing. In 2002, Bill Gates wrote the famous “Trustworthy Computing” memo⁵⁷ where he raised the importance of trustworthy software and hardware products and identified four pillars to trustworthiness: security, privacy, reliability, and business integrity.

Since then, several programs researched the concept of trust, for example, Secure and Trustworthy Cyberspace Program (2011).⁵⁸ The practice of trustworthy computing adopted in standardization practices by Institute of Electrical and Electronics Engineers (IEEE) and The International Electrotechnical Commission (IEC)/The International Organization for Standardization (ISO) and defined as following:

• “Trustworthiness of a computer system such that reliance can be justifiably placed on the service it delivers.”⁵⁹

• “Trustworthiness is a quality of being dependable and reliable”⁶⁰

• “Trustworthiness is the ability to meet stakeholders’ expectations in a verifiable way.”^{61, 62}

Today trustworthy computing remains one of the key topics in the development and operation of complex computing systems. The growing complexity of the edge to cloud compute ecosystem, the scale and widespread use of edge computing devices, makes the problem of trustworthy computing implementation even more difficult. This is especially true for the innovative AI/ML systems.

Figure 5-11 represents the timeline of the key European Union (EU) policy activities for the AI, starting with 2018 call to the EU Strategy for AI, following by Ethical Guidelines for Trustworthy AI in 2019, Assessment list of Trustworthy AI (ALTAI) in 2020 and finally the key initiative in AI space, the EU Regulation on Fostering a European Approach to AI ( also known as EU AI ACT). The term “Trustworthy AI” was introduced by the High-Level Expert Group on Artificial Intelligence (AI HLEG).⁶³ This body was set up by the European Commission to develop a comprehensive EU AI strategy that “centers on excellence and trust, aiming to boost research and industrial capacity and ensure fundamental rights of the new technology.”⁶⁴ In 2019, HLEG published Ethics the Guidelines for Trustworthy AI that captured a set of key requirements that AI systems should meet in order to be trustworthy⁶⁵:

• 1. Human Agency and Oversight: fundamental rights, human agency, and human oversight

• 2. Technical Robustness and Safety: resilience to attack and security, fall back plan and general safety, accuracy, reliability, and reproducibility

• 3. Privacy and Data Governance: respect for privacy, quality and integrity of data, access to data

• 4. Transparency: traceability, explainability, communication

• 5. Diversity, Non-discrimination, and Fairness: avoidance of unfair bias, accessibility, and universal design

• 6. Societal and Environmental Well-being: sustainability and environmental friendliness, social impact, society, and democracy

• 7. Accountability: auditability, minimization and reporting of negative impact, trade-offs, and redress

In 2020, HLEG further translated the Ethical guideline into a practical tool and presented an Assessment List of Trustworthy AI⁶⁶ (ALTAI), as well as a prototype web-based tool version.⁶⁷ ALTAI’s goal is to enable the evaluation process for Trustworthy AI self-evaluation. AI system developers and adopters can leverage relevant from ALTAI or augment to reflect their needs, based on the market sector they operate in. This helps to operationalize the Trustworthy AI and risks that AI systems might generate. Per HLEG and ALTAI, trustworthiness is key to enabling “responsible competitiveness,” by providing the “foundation upon which all those using or affected by AI systems can trust that their design, development and use are lawful, ethical and robust.” ALTAI’s goal is to enable the evaluation process for Trustworthy AI self-evaluation. Figure 5-12 Interactive ALTAI tool for self-assessment illustrates some questions for the Privacy and Data Governance section, and Figure 5-13 ALTAI self-assessment scoring results demonstrate the sample outcome of the tool that also provides the recommendations for each of the assessment categories.

Figure 5-11

Timeline of EU AI Guidelines

Figure 5-12

Interactive ALTAI tool for self-assessment

Figure 5-13

ALTAI self-assessment scoring results

In April 2021, the EU Commission proposed the new legislation called the Artificial Intelligence Act,⁶⁸ the first legal framework to address AI concerns and GDPR limitations. This is the first step toward AI regulation, gathering responses from developers and adopters of the AI/ML framework to assess the proposal and prepare for broad-scale enforcement. The proposed regulation defines four AI risk-based systems categories (Figure 5-14 AI system classification per proposed EU regulation).

Figure 5-14

AI system classification per the proposed EU regulation

Limited- and minimal-risk AI systems include many of the AI applications currently used throughout the business world, such as AI chatbots and AI-powered inventory management.

Unacceptable-risk AI systems include subliminal, manipulative, or exploitative systems that cause harm, real-time, remote biometric identification systems used in public spaces for law enforcement, and all forms of social scoring, such as AI or technology that assesses person’s trustworthiness based on social behavior or behavioral predictions. This is the category that will be outright prohibited.

High-risk AI systems include those that evaluate consumer creditworthiness, assist with recruiting or managing employees, or use biometric identification. The systems in this category will be subject to obligatory compliance. It proposes labeling based on existing Conformité Européenne (CE⁶⁹) to indicate that systems meet EU safety, health, and environmental protection requirements. The European Union will be regularly revisiting this category to update the list of systems in this category. Without logo systems will not be accepted on the EU market. To be accepted, systems designed according to five categories of requirements derived from AI ethics principles – Data and Data Governance, Transparency for Users, Human Oversight, Accuracy, Robustness and Cybersecurity, Traceability and Auditability.

While the proposed EU regulation is in the transition period, corresponding standards would be developed, and the governance structures would get ready to operationalize the legislation. It is expected that regulation would come into effect in 2024 (earliest) and the EU market would require evidence of conformity assessments.

The United States has several AI policies and best practices activities in development, Figure 5-15 NIST AI Publications depicts key triggers and published papers.

Figure 5-15

NIST AI publications

The anchor point for this development is Executive Order (EO) 13859,⁷⁰ “Maintaining American Leadership in Artificial Intelligence,” issued in 2019. Executive orders are not law, but they influence policy. EO 13859 calls federal agencies to engage in AI standardization to promote US global leadership in AI. Actions to be implemented by agencies include foundational AI research and development to regulate and provide guidance for AI technology deployment and usage.

The National Institute of Science and Technology (NIST) is the agency in charge of developing a comprehensive approach to AI standards that could be the basis for a common understanding of how to achieve and measure trustworthy AI. In 2019, NIST published “A Plan for Federal Engagement in Developing Technical Standards and Related Tools”⁷¹ to address the direction of EO’s direction and “ensure that technical standards minimize vulnerability to attacks from malicious actors and reflect Federal priorities for innovation, public trust, and public confidence in systems that use AI technologies; and develop international standards to promote and protect those priorities.” AI Trustworthiness is identified by NIST as an emerging area for standardization and proposed following AI trustworthiness attributes (Figure 5-16 NIST AI Trustworthiness Attributes): accuracy, reliability, resiliency, objectivity, security, explainability, safety, and accountability.

Figure 5-16

NIST AI trustworthiness attributes

It is also important to mention the US Executive Order 13960 from 2020, “Promoting the Use of Trustworthy Artificial Intelligence in the Federal Government”⁷² with the objective “to ensure they design, develop, acquire, and use AI in a manner that fosters public trust and confidence while protecting privacy, civil rights, civil liberties and American values.” This EO outlines the following principles for use of AI in Government:

• (a) Lawful and respectful of our Nation’s values

• (b) Purposeful and performance-driven

• (c) Accurate, reliable, and effective

• (d) Safe, secure, and resilient

• (e) Understandable

• (f) Responsible and traceable

• (g) Regularly monitored

• (h) Transparent

• (i) Accountable

Clearly, there is a strong correlation of principles outlined in the Executive Order with EU guidance and reflection of the NIST trustworthiness principles.

NIST advanced the plan, introducing several initiatives presented on Figure 5-15 NIST AI Publications timeline.

In 2019, NIST published NISRIR 8312⁷³ that outlines four principles of Explainable AI:

• 1. AI systems should deliver accompanying evidence or reasons or their outputs

• 2. AI systems should provide meaningful and understandable explanations to individual users

• 3. Explanations should correctly reflect the AI system’s process for generating the output

• 4. The AI system “only operates under conditions for which it was designed or when the system reaches sufficient confidence in its output.”

“A Taxonomy and Terminology of Adversarial Machine Learning” NISTIR 8269⁷⁴ report was introduced by NIST in 2019. The data-driven approach of ML introduces additional security challenges in training and inference of AIS operations. Adversarial ML focused on the design of ML algorithms that can resist security challenges, the study of the capabilities of attackers, and the understanding of attack consequences. The taxonomy is arranged in a conceptual hierarchy that includes key types of attacks, defenses, and consequences (Figure 5-17 The Taxonomy and Terminology of Adversarial Machine Learning – NIST 8269).

Figure 5-17

The Taxonomy and Terminology of Adversarial Machine Learning – NIST 8269

In 2021, several new reports were presented by NIST. Special Publication 1270 “A Proposal for Identifying and Managing Bias in Artificial 6 Intelligence,”⁷⁵ was published in June 2021. Ethical implementation and consequently the use of AI technologies are one of the top societal concerns. To implement AI ethically, it is necessary to ensure that AI is acting within the defined scope, and that its behavior meets the expectation of fairness and potential harm. If an AIS can’t fulfill these expectations, then it cannot be trusted and therefore cannot be adopted in public infrastructure. This NIST paper recommends an approach for identifying and managing AI bias that is tied to three stages of the AI lifecycle (Figure 5-18 Example of biases across AI Lifecycle per NIST SP 1270): pre-design, design, and development, and deployment (and post-deployment). Instead of chasing specific biases for individual use cases, for better mitigation and effectiveness, it is suggested to address the context-specific nature of AI implicitly in the life cycle of system development.

Figure 5-18

Example of biases across AI Lifecycle per NIST SP 1270

Finally, NISTIR 8332, Trust and Artificial Intelligence, examines how humans experience trust as they use or are affected by AI systems. NIST has the following proposal. If the AI system has a high level of technical trustworthiness, and the values of the trustworthiness characteristics are perceived to be good enough for the context of use, then the likelihood of AI user trust increases. According to NIST co-author Brian Stanton, the issue is whether human trust in AI systems is measurable – and if so, how to measure it accurately and appropriately.

NIST is also researching the problem of measurable trustworthiness. NISTIR 8332⁷⁶ “Artificial Intelligence and User Trust” is a proposal on how to evaluate user trust in AIS based on nine weighted attributes of AI trustworthiness (Figure 5-16 NIST AI Trustworthiness Attributes), including privacy. The weight of these nine factors may change, depending on the use case, reflecting how the individual is using the AI system and the level of risk involved in that particular use. For example, user trust in a low-risk application, such as a music selection algorithm, may be tied to certain factors that differ from those factors influencing user trust in a high-risk application, such as a medical diagnostic system. Figure 5-19 NIST User Trust Decision⁷⁷ illustrates how a person may be willing to trust a music selection algorithm, but not the AI “medical assistant” used to diagnose cancer. It is interesting to note that NISTIR 8332 is the result of cross-disciplinary cooperation, one of the co-authors, Brian Stanton, is a psychologist, and another, Ted Jensen, is a computer scientist. The authors suggest that “If the AI system has a high level of technical trustworthiness, and the values of the trustworthiness characteristics are perceived to be good enough for the context of use, and especially the risk inherent in that context, then the likelihood of AI user trust increases. It is this trust, based on user perceptions, that will be necessary of any human-AI collaboration.” Thus, building solutions with measurable trustworthiness could be a competitive advantage.

Figure 5-19

NIST user trust decision

As was mentioned before, it is essential to have harmonized international standards, the ISO/IEC SC42⁷⁸ subcommittee is the anchor point for international efforts across the definition of AI trustworthiness, bias, safety, performance, risk management, and governance. ISO defines trustworthiness as the “ability to meet stakeholders’ expectations in a verifiable way, requiring a combination of organizational process with KPI and non-functional requirements.”⁷⁹ Consumers of AI Systems (AIS) technologies expect trust to be established and maintained at each layer of AIS to sufficiently protect infrastructures for data collection, storage, processing infrastructure, and AIS assets.

ISO/IEC published a technical report “Overview of trustworthiness in artificial intelligence”⁸⁰ that surveys methods to establish the trust in AIS, threats, vulnerabilities and mitigation, methods to evaluate and achieve accuracy, security, and privacy. It is proposed to use the following mitigation measures:

• 1. Transparency – enablement of the AIS components inspectability (data, models, training methods, development practices)

• 2. Explainability – attempt to communicate understanding; ex-ante (before use, features) and ex-post (in use, decision-making) explanations of AIS

• 3. Controllability – mechanisms for operator to take control of AIS; human-in-the-loop control points

• 4. Bias reduction – analysis of the provenance and completeness data sources; model training processes; testing and evaluation techniques could be used to detect bias

• 5. Privacy – syntactic methods (such as k-anonymity) or semantic methods (such as differential privacy)

• 6. Reliability, resilience, and robustness

  • a. perform its required functions under stated conditions for a specific period of time

  • b. recover operational condition quickly following an incident

  • c. maintain its level of performance under any circumstances

• 7. System HW faults – mask or work around failures using hardware (e.g., n-plication at a course or fine grain), information (e.g., check bits), or time (e.g., re-computation at different, usually random times) redundancy; software faults are protected against by software redundancy (e.g., software diversity, or other forms of moving target mitigation)

• 8. Functional safety – monitor the decisions taken by the AI in order to ensure that they are in

  • a. a tolerable range or bring the system into a defined state in case they detect problematic behavior

Among other notable activities in the domain of Trustworthy AI, it is important to mention AI Principles developed by the Organization for Economic Co-operation and Development (OECD).⁸¹ OECD is an intergovernmental organization founded with the objective to stimulate economic growth and world trade. Its AI Policy Observatory focused on multi-disciplinary analysis of AI and produced OECD AI Principles,⁸² which became the basis for the G20 AI Principles endorsed by Leaders in 2019⁸³ and later adopted by many governments around the world. (Figure 5-20 Governments that have committed to the OECD AI Principles [source OECD]).

Figure 5-20

Governments that have committed to the OECD AI Principles (source OECD)

Additionally, Consumer Technologies Association (CTA) leads two AI standardization efforts for consumer use cases:

• Guidelines for Developing Trustworthy Artificial Intelligence Systems (ANSI/CTA-2096)⁸⁴

• The Use of Artificial Intelligence in Health Care: Trustworthiness (ANSI/CTA-2090)⁸⁵

In 2021, NIST published a draft “Taxonomy of AI Risk”⁸⁶ to solicit public input toward building an AI Risk Management Framework (AI RMF). The document categorizes risks outlined in OECD, EU Ethics Guidelines for Trustworthy AI, and US Executive Order 13960 Principles of Trustworthy AI. A hierarchical approach is proposed for categorization to simplify the risk management for the stakeholders and three broad categories of AIS risk sources outlined (Figure 5-21 Taxonomy of AI Risk):

• 1) Technical design attributes: factors that are under the direct control of system designers and developers, and which may be measured using standard evaluation criteria that have traditionally been applied to machine learning systems, or that may be applied in an automated way in the future.

• 2) How AI systems are perceived: mental representations of models, including whether the output provided is sufficient to evaluate compliance (transparency), whether model operations can be easily understood (explainability), and whether they provide output that can be used to make a meaningful decision (interpretability).

• 3) Guiding policies and principles: broader societal determinations of value, such as privacy, accountability, fairness, justice, equity, etc., which cannot be measured consistently across domains because of their dependence on context.

Figure 5-21

Taxonomy of AI Risk

NIST not only proposed the taxonomy but also provided the mapping to the relevant policy documents (Figure 5-22 Mapping of proposed Taxonomy to OECD, EU, US EO [source NIST]). At the current stage of emerging cross-Geo policy development, it is important to have developed an approach that can satisfy the global needs while industry is working on harmonizing the policy approach.

Figure 5-22

Mapping of proposed taxonomy to OECD, EU, US EO (source NIST)

AI Model and Data Provenance

If simplified, the ML model can be visualized as a function or a program that takes an input as data, applies logic, and produces an output as data. Traditional software programs are developed by humans, and it takes human knowledge as input to produce results. In the case of the AI/ML, the model’s logic is driven by its training data and the quality of the data has a paramount impact. Models are only as trustworthy as the data they are built on and contain implicit information about how training data used to create the models was collected, organized, labeled, and processed.

Basic provenance enables a system operator or consumer to know where the model came from and that it has not been tampered with. This provenance data is essentially a cryptographic hash of the model which has been cryptographically signed by the creator or publisher that can be verified with the publisher’s public key provided by a certificate authority. In addition, because the model is an embodiment of the training data, being able to verify the source and integrity of the training data gives the system operator or consumer deeper trust.

The highest level of transparency for model provenance includes more information about the training data and the training environment. How the training data was gathered, labeled, and how it was pre-processed for AI model training can be known by also providing provenance metadata for the training data. The annotation tools, statistical analyses, division into training, test, and evaluation subsets, mini-batches used for smaller training epochs, hyperparameter selection, and the security configuration of the platform also can contribute provenance for even more trust. This level of transparency can be used for system operators, customers, regulators, and compliance certifications that require the highest level of transparency and trust. The provenance of the source, labeling, and pre-processing of the training data can provide transparency for bias and discrimination reduction and verification to foster fairness and equity.

When the data that is input to the AI inferencing algorithm has provenance metadata, the system operator has the assurance that the data came from the source that signed it and that it has not been tampered with. This can help mitigate manipulation of the input data to produce false positives or false negatives and can help protect against oracle attacks that are attempting to reverse engineer the model or the training data. AI algorithms that are doing the training while deployed for inferencing (e.g., reinforcement learning) are particularly susceptible to manipulation, so the value of provenance on the input data is multiplied for these algorithms.

Provenance applied to the AI output data allows consumers of that data to verify the integrity and source of that content. If that output data is signed by the model and by the system operator, with the provenance of the input data also available, the consumer has all of the information available to trace the provenance of the entire system to gain the highest level of trust in that output data.

Data provenance and data lineage are evolving and there are two important vectors of development:

• Automation of data provenance practices. Given the amount of data required for ML, manual operations and practices are no longer a viable and ultimately present a bottleneck for ML solution scaling. Therefore, automated tools for data provenance and, most important, integration of those tools into the ML lifecycle.

• Metadata schema definitions are the starting point for the data provenance. This includes machine-readable structures to capture the provenance of all aspects that apply to the AI output. Standardization and harmonization of the metadata schema will greatly influence the ecosystem of ML solutions.

• Metadata efficient processing is another critical factor as it might become a barrier of the adoption of the provenance tooling. The cryptographic workload for provenance metadata generation and processing is not negligible and is overhead for the main “useful” data processing functions. Hierarchies and (secure) linkages can help to minimize the overhead when the trust in the source and risk factors do not require a deep dive into provenance.

• Security best practices must be applied to the data provenance tools and metadata, integrity of the processing and integrity of the metadata are critical factors.

One of the organizations that work on defining the best practices for provenance is Coalition for Content Provenance and Authenticity (C2PA),⁸⁷ a joint development foundation project, formed through an alliance between Adobe, Arm, Intel, Microsoft, and Truepic. This project was motivated by the problems in digital media, where the ability to trace the provenance of media has become critical. C2PA focuses on developing technical specifications for establishing content provenance and authenticity.

The C2PA defines a Provenance Manifest as a series of statements that cover asset creation, authorship, edit actions, capture device details, bindings to content, and many other subjects. These statements, called Assertions, make up the provenance of a given asset and represent a series of trust signals that can be used by a human to improve their view of trustworthiness concerning the asset. Assertions are wrapped up with additional information into a digitally signed entity called a Claim. These assertions, claims, credentials, and signatures are all bound together into a verifiable unit called a C2PA Manifest (see Figure 5-23 A C2PA Manifest)

Figure 5-23

A C2PA ManifestFootnote

https://c2pa.org/specifications/specifications/1.3/specs/C2PA_Specification.html

Figure 5-24 C2PA elements demonstrate the C2PA architecture for image transformation, origin as a camera type, location and time, transformation in form of the filter and compression, and final digital content with asset metadata.

Figure 5-24

C2PA elementsFootnote

https://c2pa.org/specifications/specifications/1.3/specs/C2PA_Specification.html

AI Risk Management

As mentioned before, the goal for policymakers, standardization bodies, and broad industry stakeholders is to address the risk of AI/ML systems and solutions and remove barriers for broad deployment and scale. ISO 31000⁹⁰ defines the general-purpose practical guide risk management method, provides principles, a framework, and a process. According to ISO 31000, risk management is defined as “coordinated activities to direct and control an organization with regard to risk,” where risk is “effect of uncertainty on objectives” with the following notes⁹¹:

• An effect is a deviation from the expected. It can be positive, negative, or both, and can address, create or result in opportunities and threats.

• Objectives can have different aspects and categories and can be applied at different levels.

• Risk is usually expressed in terms of risk sources, potential events, their consequences and their likelihood.

The IMSS and AIS supply ecosystem are very complex, it spreads beyond a single organization and includes developers, distributors, solution providers, customers, their partners as well as human society. The risk management for this complex ecosystem should evaluate and address the concerns of all stakeholders. ISO/IEC 242028 outlinesthe risk management process of the AI system, starting with the identification of risk sources based on the trustworthiness principles (see AI/ML Trustworthiness), translating those to control objectives and corresponding mitigation, and further mapping of those to set for guidelines or measures according to the policy (see Figure 5-25 Risk Management Process). Well-established risk management is a continuous process that includes validation, assessments, and measurements of the approaches, including performance metrics and field trials.

Figure 5-25

Risk Management Process

As a part of the broad plan for Federal Engagement in AI standards, (Figure 5-15 NIST AI Publications) NIST works on the development of AI Risk Management practices. The goal is to stimulate the development of methods to address the trustworthiness of the AIS (see Figure 5-16 NIST AI Trustworthiness Attributes). Again, this framework should encompass principles during the entire life cycle of AI systems, design, deployment, use, and evaluation of AI systems, hence establishing trustworthiness. NIST published the concept paper for public input.⁹² This paper describes the synergy between AI RMF and NIST Cybersecurity Framework⁹³ and Privacy Framework.⁹⁴ It is important for IMSS/AIS developers to realize the dependency between Security, Privacy, and AI Trustworthiness. An AI RMF is required to be maintained continuously throughout the life cycle entireness, see Figure 5-26 Risk Management Throughout the AI System Life Cycle with four sample categories to be implemented during Pre-design, Design and Development, Test and Validation and Deployment:

• 1. Map: Context is recognized, and risks related to the context are enumerated.

• 2. Measure: Enumerated risks are analyzed, quantified, or tracked where possible.

• 3. Manage: Enumerated risks are prioritized, mitigated, shared, transferred, or accepted based on measured severity.

• 4. Govern: Appropriate organizational measures, set of policies, processes, and operating procedures, and specification of roles and responsibilities are in place.

Figure 5-26

Risk Management throughout the AI System life cycle (NIST RMF)

The AI Risk Management process helps to alleviate challenges that arise from the broad and evolving AI use cases. As we mentioned earlier, the AI ecosystem is complex with a multitude of diverse stakeholders, including developers, users, deployers, and evaluators. This makes risk management even more complicated when entities must identify, assess, prioritize, respond to, and communicate risks across business and social boundaries at scale. Adoption of a consensus-driven framework such as the one proposed by NIST, can address regulation, establish trust in AI systems, and unleash business opportunities.

IMSS with ML Protection

Regulatory, Policy, and Standards development demonstrates the need for a holistic approach when assessing the security, privacy, and trustworthiness of AI/ML-based IMSS.

AI systems classification framework, developed by OESD⁹⁵ could help stakeholders to define high-level elements of solutions and map implications for key policy areas.

Figure 5-27 AIS Classification (source OECD) presents the OECD framework which has four key dimensions:

• Context: Socioeconomic environment where the system is being deployed and used. For example, what’s the industry, business function, and scale of the deployment.

• Data and Input: The data that the system uses and the input it receives, including data collection to build a representation of the environment.

• AI Model: Computational representation of real-world processes, objects, ideas, people, and/or interactions that include assumptions about reality, including underlying AI technology, that makes up the AI system. For example, the type of the model (symbolic, statistical, or hybrid), and how the model was built (e.g., supervised or unsupervised learning).

• Task and Output: Tasks the system performs and the outputs that make up the results of its work (for example, recognition, personalization, etc.) and resulting action.

Figure 5-27

AIS classification (source OECD)

Why is this important? Different types of AI/ML systems raise unique policy considerations in their use context. This framework allows cataloging and organizing the AI system. In the context of measurable trustworthiness having such a classification will be a critical factor for building scalable solutions.

Here is the OECD definition of the AIS⁹⁶ that can be directly applied to ML IMSS:

An AI system is a machine-based system that is capable of influencing the environment by producing an output (predictions, recommendations, or decisions) for a given set of objectives. It uses machine and/or human-based data and inputs to (i) perceive real and/or virtual environments; (ii) abstract these perceptions into models through analysis in an automated manner (e.g., with machine learning), or manually; and (iii) use model inference to formulate options for outcomes. AI systems are designed to operate with varying levels of autonomy.

Generic ML IMSS architecture includes the collection of data from the Sensors (data ingestion), and an ML models that operates on the ingested data and produces results that will feed into decision-making (Actuators), see Figure 5-28 AI/ML Flow. The distinctiveness of ML is the fusion of data and ML Model. Data has dual importance as training sets are used to produce the model at the development stage and, upon deployment, the ML Model will be used for inferencing to generate insights.

Figure 5-28

AI/ML flow

To address the problem of the protection and ML security risks, first we must comprehend the IMSS solution lifecycle, including development, distribution, and execution. Here is the definition of AI lifecycle per OESC (Figure 5-29 AIS Lifecycle (source OECD):

AI system lifecycle phases involve: i) ‘design, data and models’; which is a context-dependent sequence encompassing planning and design, data collection and processing, as well as model building; ii) ‘verification and validation’; iii) ‘deployment’; and iv) ‘operation and monitoring’. These phases often take place in an iterative manner and are not necessarily sequential. The decision to retire an AI system from operation may occur at any point during the operation and monitoring phase.

Figure 5-29

AIS Lifecycle (source OECD)

AI/ML applications can be narrowed down to 2 major functions: Training and Inferencing. IMSS solutions are greatly leveraging compute resources from edge to cloud. The Figure 5-30 IMSS with Machine Learning summarizes ML solution scenarios with corresponding data/IP flows:

• 1. Training in the Cloud using data supplied from the Edge

• 2. Inferencing in the Cloud based on the data received from the Edge

• 3. Inferencing at the Edge based on Model developed/distributed from the Cloud and the data received from multiple End Nodes

  • 3.1. Inferencing on End-Node at the Edge

  • 3.2. Inferencing on-Prem at the Edge

• 4. Training based on Federated Learning approach where models get trained locally on-premises without revealing the data to the central cloud entity

  • 4.1. Federated Learning on End-Node at the Edge

  • 4.2. Federated Learning on-Prem at the Edge

Figure 5-30

IMSS with machine learning processing nodes and flows

The preceding flow identifies three major types of processing nodes: 1) End-Node at the Edge (smart camera, robots, drones); 2) On-premises Edge (IoT gateways or servers) that resides within premise within close proximity to the End-Nodes; and 3) ML services that are usually deployed in the cloud environment.

As it is stated above Organizations should embrace risk management practices throughout the entire lifecycle and across diverse stakeholders to establish the trustworthiness of their solutions and scale through the markets. In the case of the IMSS risks should be extended to include the consequences of data or model bias, new security threats such as adversarial inputs, threats to privacy due to leakage of personally identifiable information, and the lack of explainability and accountability.

Stakeholders and Assets

Risk management practices require scope and classify the system’s stakeholders and assets. Stakeholders cover all parties (organizations and individuals) who are involved in, or affected by, systems, directly or indirectly. Some stakeholders can play an active role in the life cycle, being directly responsible in the deployment and operation of the IMSS. ISO/IEC TR 24028:2020⁹⁷ defines following stakeholders for the AI value chain that can be directly applicable to the IMSS, see Table 5-1 ML ecosystem stakeholder per ISO/IEC TR 24028:2020.

Table 5-1 ML Ecosystem Stakeholder Per ISO/IEC TR 24028:2020

The Technical report also outlines the AI assets that need to be protected. The Table 5-2 ML Assets per ISO/IEC TR 24028:2020 summarizes tangible and less tangible assets that might be impacted by the IMSS ML product.

Table 5-2 ML Assets per ISO/IEC TR 24028:2020

Threats

Threat analysis is a key element in the design and development of a secure solution, and getting the threat taxonomy defined is a key factor for building trustworthy solutions. IMSS ML inherits the traditional cybersecurity threats for Edge and Cloud deployments and at the same time introduces new threats that are not addressed by traditional security measures.

Edge deployments require strong physical security, which cannot depend on the assumption of physical perimeter security such as monitoring the environment for access control and surveillance. Edge devices operate in potentially hostile environments and require physical defense mechanisms built into edge system designs. The levels of a physical defense vary based on the market segment use cases. Additionally, cloud services customers may be concerned about physical threats even though there is an expectation that the service providers practice perimeter security. Security technologies that mitigate physical threats are also seen as a benefit to the cloud services providers in that it not only provides better security assurance, but also reduces their liability for losses and breaches. Physical attacks are also reflected in the following diagram that provides Microsoft Azure vision for the Edge threats.⁹⁸

Figure 5-31

Security threats at the edge (Source: Microsoft)

In addition to edge threats and traditional vulnerabilities, an IMSS ML system must address new attack methods. The Adversarial ML Threat Matrix project,⁹⁹ led by Microsoft, with participation from other industry leaders, MITRE, Bosch, IBM, NVIDIA, and others, developed the knowledge base of ML systems attacks. The motivation here is to leverage the ATT&CK¹⁰⁰ framework for ML attack enumeration. Definitional starting point is the categorization of the machine learning attacks across inferencing and training use cases (see Table 5-3 ML System Attack Categories (source: GITHUB).

Table 5-3 ML System Attack Categories (source: GITHUB)

These categories were fed into the Adversarial ML Threat matrix to produce a meaningful tool for the security analyst to navigate the complex landscape of ML threats. This framework is populated with a curated set of vulnerabilities and adversarial behaviors that Microsoft and MITRE have vetted to be effective against production ML systems (Figure 5-32 Adversarial ML Treat Matrix (source: Github)). There is an axis with seven tactics, though in this case, they are focused in the area of ML: Reconnaissance, Initial Access, Execution, Persistence, Model Evasion, Exfiltration, and Impact. Each column has the techniques, categorized into two types: orange techniques are unique to ML systems, and white/grey techniques that apply to both ML and non-ML systems and come directly from Enterprise ATT&CK.

Figure 5-32

Adversarial ML Treat Matrix (source: Github)

Berryville Institute of Machine Learning (BIML) Threats took a different approach in their “An Architectural Risk Analysis of Machine Learning Systems: Toward More Secure Machine Learning.”¹⁰¹ The BIML conducted the risk analysis of generic ML system decomposed to individual components that are associated with various phases of ML lifecycle, such as 1) raw data in the world, 2) data set assembly, 3) data sets, 4) learning algorithm, 5) evaluation, 6) inputs, 7) model, 8) inference algorithm, and 9) outputs. After identifying risks in each component, the aggregated risks for ML system as a whole were identified what they believe are the top 10 ML security risks presented in the Table 5-4.

Table 5-4 BIML Top Ten ML Security Risks

Threats for the Training Process

There are IMSS use cases that perform training in edge devices, which brings the direct training threats into the inferencing environment.

The training data itself may be subject to data confidentiality and privacy constraints that can be exposed during inferencing by classic cybersecurity, property inference, membership inference, or model inversion attacks. Following the cybersecurity basics in this text will help with the classic cybersecurity attacks; however, the latter three attacks occur because of having control of the input and access to the output data during normal operation. The risk is primarily for the application developer, but system operators and their integrators and contractors may be required to provide cybersecurity controls to minimize their responsibility.

Also, the trustworthiness of the inference algorithm depends in large part on the trustworthiness of the training data. As you learned in the policy and standards section, the training data and resultant algorithms must be legal and ethical to use, and the system operator must ensure that.

Machine Learning algorithms are often trained with generic data that may not be representative of the deployed IMSS. This is a very common problem leading to poor accuracy in a deployed IMSS.

See the section on protection solutions for methods to improve performance, accuracy, and reduce risk.

Threats for the Inferencing Process

Real world data sampling is turning actual events into a digital form that can be processed by a machine learning algorithm. The performance will only be accurate and trustworthy anchored on the expectation that those sensors and subsequent data processing are secure.

In addition, real world deployed systems often exhibit noise, distortion, and artifacts that are not present in the training data. And in some cases, called transfer learning, algorithms are used outside of the scope they were trained for. Sometimes this works well, but some of the algorithms will randomly produce wildly inaccurate results when presented with data that they were not trained for.

The Machine Learning applications must also be trustworthy. The applications may carry malware trojans or backdoors that will compromise IMSS, leading to ransomware, functional failures, or data manipulation that produces false positives or negatives. Or they may be tampered with within the inferencing system.

The hyperparameters that control Machine Learning applications at runtime can cause the models to be overconfident (producing false positives) or underconfident (softening the probability distribution leading to classification errors).

It is rare for an IMSS to run in a complete air gapped network with only highly secured devices on the network. For example, most systems allow a remote user to access data via a phone, tablet, or laptop which can allow access to the secure network. If the inferencing is being done in a public cloud, the models and the input and output data may be exposed to untraceable attackers posing as cloud tenants due to poor security, misconfiguration, and misplaced inherent trust. Zero Trust protocols provide layered protection for these use cases. The main concept behind zero trust is elimination of implicit trust and continuously validating every stage of a digital interaction. The core principle of Zero Trust is “never trust, always verify.” The Zero Trust objective is to protect compute environments using strong authentication methods, use least privileged access, and minimize the exposure. NIST recently published “Zero Trust Architecture” SP 800-207,¹⁰² which provides an abstract definition of zero trust architecture (ZTA) and gives general deployment models.

In the same manner that the integrity and provenance of the input sensor information is critical for trustworthiness, the outputs must also be secure and trustworthy.

Methods to mitigate these threats are described in the ML-based IMSS Protection section of this chapter.

Training at the Edge

Edge training as a use case is driven by data compartmentalization requirements of various industries. There are regulations in Health and Life Sciences, Public Sector, and Finance technology (FINTECH) markets that drive training at the edge (especially Federated Learning). In some parts of the world, national data sovereignty laws do not allow data to be exported beyond the borders of that nation.

And there are a growing number of jurisdictions with privacy laws, such as the EU General Data Protection Regulation, similar laws in many other nations, recommendations for NIST in the United States, and local state laws in the United States. These are coming from a general desire for privacy which results in consumer demand for privacy-preserving analytics that benefit the consumers while preserving their privacy.

Federated Learning meets the goals of highly accurate analytics based on large diverse data sets, with the individual’s privacy being protected. Federated learning is a distributed machine learning approach that enables organizations to collaborate on machine learning projects without sharing sensitive data, such as, patient records, financial data, or classified secrets (McMahan, 2016¹⁰³; Sheller, Reina, Edwards, Martin, & Bakas, 2019¹⁰⁴; Yang, Liu, Chen, & Tong, 2019¹⁰⁵). The basic premise behind federated learning is that the model moves to meet the data rather than the data moving to meet the model. Therefore, the minimum data movement needed across the federation is solely the model parameters and their updates.

Figure 5-33

Federated learning (source Overview — OpenFL 2022.2 documentation)

Figure 5-33 present Federated learning systems with new components introduced to the AIS data pipeline.¹⁰⁶

• Collaborator: A collaborator is a client in the federation that has access to the local training, validation, and test datasets. By design, the collaborator is only component of the federation with access to the local data. The local dataset should never leave the collaborator.

• Aggregator: A parameter server sends a global model to the collaborators. Parameter servers are often combined with aggregators on the same compute node. An aggregator receives locally tuned models from collaborators and combines the locally tuned models into a new global model. Typically, federated averaging (a weighted average) is the algorithm used to combine the locally tuned models.

• Round: A federation round is defined as the interval (typically defined in terms of training steps) where an aggregation is performed. Collaborators may perform local training on the model for multiple epochs (or even partial epochs) within a single training round.

Reinforcement Learning and Transfer learning are other ML methods that conduct training on the edge system or gather data from the edge, to provide accuracy improvements while performing inferencing.

Both learning methods are fundamentally different from traditional supervised learning methods, conducted in an isolated environment, traditionally on cloud systems.

Reinforcement learning is the training method when learning happens from the interaction within the target environment, either real or simulated, that allows he production of versatile models for specific usage. In this scenario, action will be evaluated for a positive or negative outcomes, based on an established reward. Upon receiving feedback, the model “learns” whether that decision was good or bad and gets self-adjusted.

Transfer learning is a method of reusing a pre-trained model for a new problem when the model was trained on one dataset and then fine-tuned to work with another dataset and perform modified tasks. This is especially important for the broad scale deployments, when models developed based on the traditional supervised methods stored and distributed through “Model Zoo”s, still require tuning to operate in specific use cases.

When the IMSS models are developed using sensor data based on reinforcement or transfer learning, poisoning the training data provides a new pathway for the threats. It can amplify spurious correlations, cause the model to drift, amplify bias, and intensify general inaccuracies and, as a result, compromise the IMSS produced outcomes.

The security required for these use cases is unique. It includes confidentiality and privacy requirements for data involved in a training or retraining path. Depending on the Context (see Figure 5-27 AIS Classification [source OECD]), it might be required to use different keying and protocols to protect and isolate the inferencing data from the data originally used in training, or different stages of training. More importantly, it brings new considerations for data poisoning and provenance protection that are not present in a traditional enterprise feed-forward training environment.

ML-based IMSS Protection and Trustworthiness Framework

Considering cross dependency in addressing security, privacy, and trustworthiness risks of ML-based IMSS, defense in depth strategy should be employed with layered approach to protect IMSS assets and establish trust for stakeholder. Figure 5-34 Layered ML Protection shows composed defense layers. The proposed model uses an OSI (Open Systems Interconnection Model)-type, bottom-up layers.

Figure 5-34

Layered ML Protection and trust

Foundational Device Security 

Any security solution requires foundational or baseline security capabilities to be built-in on the device. The concept of the baseline is aligned with the industry momentum on defining the IoT cybersecurity capabilities (NIST, ENISA, ISO/IEC). With increased attention to the high level of software, attackers are drilling down into the platform stack, exploiting vulnerabilities in the platform firmware and hardware.

Hardware-enabled security techniques can help mitigate these threats by establishing and maintaining platform trust – assurance in the integrity of the underlying platform configuration, including hardware, firmware, and software. The Root of Trust (RoT) is the anchor for establishing trust, it provides protection for identities and workloads, hardened data crypto services, and authentication for a variety of applications. A Hardware-based RoT is an immutable foundation for establishing platform integrity. Platform attestation or verifiable evidence is the basis of creating a trusted platform, where the measurement of platform firmware and configuration chained with the rest of the Software Bill of Material (BOM) verify the boot loaders and then OS, hypervisor, or container runtime layers. The transitive trust described here is consistent with the concept of the chain of trust (CoT), a method where each software module in a system boot process is required to measure the next module before transitioning control. Hardware-rooted platform integrity and trust strengthen and complements the extension of the CoT and can be extended even further to include data and workload protection, critical protection problems for the IMSS ML.

Hardware-based protections lay the foundation for protecting multi-tenant data and workloads in edge compute devices. And in systems that do not have multiple tenants, the compute contexts themselves rely on foundational hardware security and the CoT to provide robust isolation so the different contexts are secure from each other.

Workload Protection

Machine Learning Analytics models and applications represent a wide range of development costs. These costs range from free-to-use open-source pre-trained models to custom models trained from proprietary data sets using custom processing layers and custom topologies costing millions of dollars to develop. The data sets themselves may be expensive to gather or may contain proprietary information or trade secrets. On top of the data, most ML training methods require annotation of the data, which involves a human providing the annotation. For example, in an application such as MRI images, a highly paid specialist spends on average seven minutes per image. And the data may be subject to privacy laws and regulations requiring the subjects to approve the use of their personal data. These add more elements of cost and value to the curated input data.

Training an algorithm can take lots of time on a server, and if the model topology and layers are also being developed, it may take many trial iterations to arrive at an optimal solution. There is no guarantee as to how many iterations it takes for the training to converge – or even a guarantee that they will converge.

Finally, the current market expectations, and hype are heightening the perceived market value of analytics solutions. Because they can be relatively faster and more reliable than humans, they may represent significant cost savings or quality improvements. And in some cases, the reduced response latency, accuracy, and reliability may save lives compared to traditional systems.

These factors mean that developers are highly motivated to protect their investments. They want to prevent integrators or end users from copying or cloning the applications, and from reverse engineering them.

IP Protection

ML Intellectual Property (IP) is the core of ML systems and introduces the specific challenges related to IP opportunities and pitfalls. Traditional IP Protection tools include copyrights, patents, contracts, and trade secrets. ML innovation targets are ML algorithms/models, model evaluations and optimization, data pre-processing and post-processing. Extensive high-quality, representative training datasets are extremely important for reliable performance of an AI model when processing new data. Business value can be found in the following:

• Protecting AI/ML models and/or algorithms

• Software and systems with embedded models/algorithms

• Training, evaluation, and/or optimization strategies

• Training data

• Results data

Institutions around the world are addressing a variety of issues associated with AI/ML IP protection to obtain legal exclusivity to secure assets. Patents may be attained for application of ML in solving a technical problem. In October 2020, the US Patent and Trademark Office released the report “Public Views on Artificial Intelligence and Intellectual Property Policy.”¹⁰⁷ The objective of this report was to seek feedback on whether the current patent-related laws are adequate for AI IP protection. It is interesting to note that the report highlighted many of the concerns raised around AI and IP rights as the technology of AI evolves.

Gartner’s Hype Cycle for Legal and Compliance Technologies¹⁰⁸ (Figure 5-35 Hype Cycle for legal and Compliance Technologies, 2020 (Source: Gartner)) provides vision for legal and compliance technologies to anticipate, evaluate the impact of emerging legal technology and business innovations. It is interesting to see the several AI/ML-related aspects highlighted in the analysis, specifically AI governance is on the rise and Explainable AI reaching the peak of expectations.

Figure 5-35

Hype Cycle for legal and compliance technologies, 2020 (Source: Gartner)

To rely on trade secret protection mechanisms, an owner must implement related internal policies and adequate protective measures. However, the democratization of the Edge and cross-industry collaborative execution model makes this unattainable. Besides the drawbacks of complexity, traditional IP protection tools might jeopardize existing and novel business models. These drawbacks are critical impediments for high-scale AI/ML usages across Edge to Cloud for multi-tenant IMSS deployment.

There is an emerging analytics-as-a-service market where cloud service providers enable tenants to bring their data to be processed in the cloud. In this environment, not only do the tenants distrust other tenants, but also, distrust the service providers, their personnel, or the service provider’s infrastructure. Consequently, to gain business benefits, service providers want to reassure that they can enforce IP Protection mechanisms.

OpenVINO™ Model Protection Security

OpenVINO™¹⁰⁹ is an Intel initiative and toolset to address the Machine Learning industry fragmentation, optimize and deploy deep learning solutions across multiple Intel platforms, and maximize the performance of applications for any type of Intel-provided inference engine. The Intel Distribution of the OpenVINO™ toolkit packages a host of tools and optimized functions enabling computer vision and deep learning inference deployment.

Figure 5-36

Inference Workflow with OpenVINO™

OpenVINO™ supports more than 100 public machine learning topologies and custom models, and it accepts models from standard frameworks such as Caffe, TensorFlow, MxNet, and others.

With reference to Figure 5-36 Inference Workflow with OpenVINO™, a model is built and trained with a common framework (left side). Next, models are imported into the model optimizer, a Python-based tool that converts and optimizes them for performance, accuracy, and power efficiency. This can occur on the development machine or the target hardware for deployment. Intermediate representation files, or IR files, are then sent to the inference engine. This is a common API interface with dynamic plugins to easily deploy across multiple hardware types. The model optimizer allows for performance comparison or tuning across different accelerators without having to recode. The inference engine also allows heterogeneity – for example, by providing fallback from custom layers on an FPGA to a CPU. The toolkit also contains optimized OpenCV,¹¹⁰ media encode and decode functions, OpenCL¹¹¹ drivers, pre-trained models, and a host of samples for full workload pipeline integration.

ML Model developers Often make a significant investment in the training data set and in the actual training of the models. So, they are concerned that the models might be copied, cloned, or reverse-engineered during runtime.

The OpenVINO Security Add-on (OVSA) (see Figure 5-37) is an end-to-end IP security solution that applies to analytics models developed using Intel’s OpenVINO™ toolchain that enables retargeting pre-trained models for optimal execution on various Intel HW accelerators (or back-ends). OVSA cryptographically protects IP at the Model Developer (or ISV) location, through delivery, and while in use by customers or the Model User (i.e., during run time).

Figure 5-37

OpenVINO Security AddOn

OVSA applies cryptographic confidentiality and integrity protection to the model’s Intermediate Representation (IR) and binds it to a customer license that describes the terms of use of the model. Licenses are specified on a per-customer basis. A model is usable by a customer, as long as the terms of use specified by the license are valid. Once these terms of use are exceeded, the model is no longer usable.

The OpenVINO™ Security Add-on along with the OpenVINO™ Model Server (OVMS) provide solutions for Model Developers to use secure packaging and model execution to enable access control to the OpenVINO™ models, and for model Users to run inference within assigned limits.

The OpenVINO™ Security Add-on consists of three components¹¹²:

• OpenVINO™ Security Add-on Tool: As a Model Developer or Independent Software Vendor, you use the OpenVINO™ Security Add-on Tool to generate an access-controlled model and master license.

• OpenVINO™ Security Add-on License Service: Use the OpenVINO™ Security Add-on License Service to verify user parameters.

• OpenVINO™ Security Add-on Runtime: Users install and use the OpenVINO™ Security Add-on Runtime on a virtual machine.

Figure 5-38 OpenVINO™ Security Add-On for Deployment demonstrates the IP protection during the model deployment. Protected models are launched in a model container of OpenVINO™ Model Server. It can load multiple models into memory and customer applications can submit analytics jobs to Model Server for processing by a specific model. The OVSA license checks are performed when a protected model is loaded into OVMS. Protecting cryptographic keys is essential in security protocols. OVSA protects the encryption key by wrapping it with the customer’s public key and binding it to the platform Trusted Platform Module or to Intel® Platform Trust Technology.¹¹³ The customer’s private key decrypts the model inside a trusted execution environment (TEE). OVMS and OVSA execute within a TEE to ensure the model is protected at run time. The TEE provides isolation from other applications, including the calling application that invokes the model. OVSA leverages the Linux Kernel Virtual Machine (KVM)¹¹⁴ hypervisor and virtual machine isolation mechanisms or a hardened type 1 virtual machine environment.¹¹⁵ For modern Xeon platforms, OVSA also supports Intel® Software Guard Extensions (SGX)¹¹⁶ and in the future, Intel® Trusted Domain Extensions (TDX)¹¹⁷ HW enforced Virtual Machines. TDX is a future technology, the details are beyond the scope of this document.

Figure 5-38

OpenVINO™ Security Add-on for deployment

When OpenVINO targets an accelerator with a back-end plug-in, the trusted execution environment is extended to the accelerator by encrypting the interface over the PCIe bus. So, the models never appear in plaintext, they are always protected by encryption or by trusted execution environments.

The focus of OVSA is the cryptographic IP protection and IP licensing mechanism. As mentioned earlier, OVSA uses existing run-time isolation mechanisms. The same cryptographic and licensing mechanisms will be applied to protected data streams in a subsequent version of OVSA. This will enable control over data ownership, access, use, and provenance. Once complete, the OVSA framework will provide both IP and data protection and use controls.

Data Protection, Privacy, and Provenance

Data makes up the most critical aspect of ML security. It is important to understand the data categories in ML systems. As it was mentioned earlier in the asset definition, data assets in an IMSS are 1) the inputs: generally, event and sensor data; 2) the outputs: processed events, and processed sensor data; 3) the results of analytics inferences derived from the input data; and 4) data used for the training.

The inputs can be simple events like a door opening indicator to more complex data from an audio or still image or video systems. Output derived from these data ranges from a basic understanding of characteristics of the data source to a complex understanding with narrowly defined differentiating criteria, to high cognitive understanding such as situational awareness. Many different input data types and sources are used to derive the output.

The value of the data assets depends on the cost of gathering, curating, and producing the data and the benefit derived from its use. Some data has obvious value, such as proprietary company data or trade secrets. Other types of data may be expensive to gather, such as Magnetic Resonance Imaging data.

The value of the output data is higher than that of input data because of the additional knowledge provided by the analytics. The benefits of high accuracy, high throughput, and the amount of information that can be processed can save lives, provide improved outcomes, enable faster response, and more complete understanding based on more data than was possible until now.

Additionally, for some systems, the data must be kept private or confidential, it must also be protected from tampering. US federal jurisprudence rules of evidence allow for “self-authenticating” evidence that includes proof that it has not been altered from the time of capture. The emerging tools for manipulating images and data are getting good enough that it is becoming difficult to prove intrinsically whether data is true, so it is beneficial to authenticate the data itself combined with encrypting data streams to maintain their confidentiality.

Sensor outputs and their analytical product are generally the property of the system owner. In some cases, laws confer ownership to the subject that the data is gathered from, and the system owner is a custodian of the data with legal responsibilities and limits for how it can be used or made public. This is particularly true for privacy-related data under the growing body of laws and regulations such as the EU General Data Protection Regulation (GDPR), US state-level legislation, such as the California Consumer Privacy Act (CCPA). These laws reflect how much people value their privacy and want to control personal data.

Classic cybersecurity methods grounded in a hardware Root of Trust will assure that the data came from a source that can be trusted and that the data has not been tampered with as well as maintain confidentiality and privacy of the data. Provenance methods employ cryptographic integrity and authentication methods that prove the data comes from a particular entity (that digitally signs the data) and that the data has not been altered since it was signed.

The threats section introduced specific threats to training and inferencing processes that are beyond the reach of traditional cybersecurity methods. Here are specific data and IP protection methods for those threats that will enhance the trustworthiness of the machine learning results.

Inaccurate data can be signed, authenticated, and subsequently verified. When using data to train machine learning, there must be an independent way to establish the veracity of the training data, essentially a ground truth reference. The ground truth reference must be reserved from the training process to use in the evaluation phase to establish the accuracy of the model. For in-loop training algorithms like Federated Learning and Reinforcement Learning, that same reference must be used periodically in a field evaluation to prevent model drift and thereby maintain trust.

The training dataset has to be large enough and the sampling has to be representative of the real-world diversity that will be experienced during inferencing. There is a necessary phase of processing the training data to eliminate redundancies, incomplete data, etc. During this process, developers may be tempted to manipulate the data samples to give better accuracy figures by, for example, eliminating anomalies. Proper use of statistical sampling methods to measure sampling error vs. the population, and randomness of the selection of training, test, and evaluation data sets will mitigate manipulation and provide data-based estimates of the true accuracy of the fielded algorithm. Using a FIPS-140-3 approved random Number Generator provides truly random numbers to mitigate the introduction of errors due to poor dataset sampling randomization.

Expert legal and ethics reviews are recommended to ensure the training set is legal and that the data has the necessary diversity, and the data, especially the annotation, is free from bias.

There are tuneable hyperparameters that are adjusted by developers during the training process. These are used to reduce false positives or false negative, and to control the confidence of the results. Overconfident or overfitted algorithms that have essentially memorized the training data will be more brittle when fielded and the results will be easier to manipulate with small perturbations of the input data. Adjusting the other way makes the algorithms more resilient but also less discriminating and more likely to produce alternate results with high confidence.

While system operators may not be directly responsible for this work, this must be included in the algorithm selection process because operators are responsible for selecting a trustworthy machine learning application.

Federated Learning

Federated learning applies to usages where the data must be compartmentalized, but the overall accuracy of the inferencing application requires a larger data set than the compartmentalized data contains. The compartmentalization can be driven by privacy considerations such as data sovereignty regulations at a national level, privacy regulations such as HIPAA or GDPR or the CCPA (and many others), compartmentalization for national security such as defense intelligence data, or as a general privacy preservation technology providing privacy-sensitive customers the greater assurance of privacy. In some cases, data transmission and storage constraints may also lead to a federated learning solution.

Because of the data compartmentalization, the data cannot be brought to a cloud backend, so the training must go to the local data domain. The edge devices used for federated learning range from high-performance server class computers to the IoT edge devices themselves, essentially where the sensors are.

Figure 5-39

Centralized vs. Federated Learning

The centralized or federated training process receives machine learning parameters from the edge collaborator nodes and combines them to gain the benefit of a large training data set represented by the multiple edge node collaborators.

Intel® Open Federated Learning is a github¹¹⁸ project that demonstrates Python 3 library with two components: the collaborator, which uses the local dataset to train a global model, and the aggregator, which receives model updates from collaborators and combines them to form the global model. The aggregator is framework-agnostic, while the collaborator can use any deep learning framework, such as Tensorflow or PyTorch. Additional documentation is available at openfl.readthedocs.io/en/latest/index.html.

While Federated learning enables dynamic collaboration while maintaining collaborator node data privacy, establishing trust and providing end-to-end security and privacy safeguards are critical to deployment and adoption of Federated Learning. Data and the model both should remain confidential and private, protected from multiple threats, including theft and manipulation that could lead to data leaks or model tampering that would undermine the accuracy and integrity of the solutions, damaging the trustworthiness of the AI/ML system. Intel published a paper on “Federated Learning through Revolutionary Technology,”¹¹⁹ where the role of hardware-rooted security technologies and Trusted Execution Environments (TEE) are employed to protect Federated Learning system nodes and connectivity to ensure trustworthiness.

TEE provides a mechanism for hardware-based isolation with memory encryption for code and data during execution. In the context of federated learning, foundational security functions provided by a TEE are confidentiality of the execution to mitigate attacks such as model IP stealing out of memory as the training process executes, the integrity of the execution to mitigate attacks that alter the behavior of the code, and remote attestation of the execution, wherein a TEE can provide some measurements as a proof for participating entity trustworthiness.

Intel® Software Guard Extensions (SGX) is a hardware-based TEE that is available on the market and helps to protect against data and code snooping and manipulation by malware. As Figure 5-40 Intel® SGX shows, technology is rooted in the hardware with a minimized trusted computing base that limits the attack surface and is more robust than pure SW methods. This follows a basic principle of cybersecurity and complexity, that is, the lower the complexity (the amount of code in the TCB) the better the security of the system is.

Figure 5-40

Intel® SGX

Intel® SGX can be integrated at both collaborator nodes and aggregation engine. This will ensure that both aggregated model, data used to construct the model, and communication will be confidential and protected from tampering. Figure 5-41 Federated Learning system with SGX enclave protection shows how SGX enclaves are used in the aggregator, collaborator, and governor nodes to provide confidentiality, and execution integrity, with robust attestation and isolation from the OS and Virtual Machine in each processing node.

Figure 5-41

Federated Learning system with SGX enclave protection in Gramine containers

During federation, data of a collaborator remains with the collaborator node for local training and never leaves it. Instead, model updates from each collaborator node are sent to an aggregator node so that they can be combined into a global “aggregated” model. The global model is returned to the collaborator nodes for a further round of local training. Thus, secure communication is a critical element for establishing secure federated learning systems. Two-way TLS¹²⁰ is a well-known method of establishing secure communication. SGX TEE requires significant modifications to enable the execution of the ML training code inside a TEE that increases development efforts in a user’s application. To address this extra overhead, lightweight, open-source library OS Gramine¹²¹ was developed for running unmodified user applications inside Intel SGX. This method allows users to run OpenFL code seamlessly without any modifications within TEE.¹²² For more information on how to integrate gramine to OpenFL see OpenFL GitHub.

Homomorphic Encryption

Another method to address data privacy protection is Homomorphic Encryption (HE) which allows ML computation to be performed over the encrypted data without the need to be decrypted. However, the usage of HE-encrypted data is limited to only simple types of machine learning models, which are not proven efficient and accurate with more practical and advanced datasets. HE acceleration is a promising and prospective solution to enable HE for traditional training methods. The Intel® Homomorphic Encryption Toolkit (Intel® HE Toolkit)¹²³ is designed to provide a well-tuned software and hardware solution that boosts the performance of HE-based cloud solutions running on the latest Intel® platforms. The vision is to lead the homomorphic encryption transformation by providing advanced HE technology on Intel® architecture, allowing customers to gain valuable insights while protecting highly private and sensitive data.

Intel is enabling the emerging HE ecosystem by accelerating HE to meet commercial performance requirements on real-world future use cases. The toolkit has been designed as an extensible solution to take advantage of the latest Intel® Advanced Vector Extensions 512¹²⁴ (Intel® AVX-512) acceleration instructions. The toolkit can also be combined with future purpose-built accelerator technology.

In the Trusted Zone, data is encrypted using a homomorphic encryption scheme and sent to the Central Non-Trusted Zone for computation (Figure 5-42 HE solution). In the case of IMSS, edge components will generate data and implement Trust Zone, and send it to the cloud for processing (training or inferencing).

Figure 5-42

HE solution

Data Provenance

As mentioned earlier, Data Provenance solutions are developing and fast growing. There are solutions both from market leaders such as Microsoft and IBM as well as from novice visionary companies.

IBM DataStage¹²⁵ is a data integration tool that combines analytics, on prem or cloud environments, governance, and data warehouse in a single platform built for AI development.

Microsoft Purview¹²⁶ unified data governance solution to help manage and govern data on-premises, multi-cloud, and software as a service (SaaS) data, mapping data with automated data discovery, sensitive data classification, and end-to-end data lineage.

Trustworthiness

As it was stated earlier, Trustworthiness is the most complex category as it covers several aspects that need to be assessed, and measured for an ML solution implemented with values-based principles. As standards are getting developed and policy is getting more and more mature, what does the technical community have to do to stay ahead of the curve and remove barriers to broad adoption of IMSS technologies? The first question for organizations who are developing and deploying systems would be to determine the right time to intervene.

Here we face a well-known problem of uncertainties for emerging technologies. In 1980, David Collingridge, a renowned researcher, published The Social Control of Technology,¹²⁷ where he introduced “The Collingridge Dilemma”¹²⁸ to explain the difficulty of controlling risks with given uncertainties, the dichotomy between insufficient information and power problems.

“When change is easy, the need for it cannot be foreseen; when the need for change is apparent, change has become expensive, difficult, and time-consuming.”

According to Collingridge, at the conceptual development stage, there is more ability to influence innovation, whereas, at the same point, there is limited information on the knowledge on the impact. When technology reaches maturity, we would have sufficient information on impacts but will have a lesser ability to control. This dilemma directly applies to AI. There is increased consensus that AI technology is at a juncture, and this is the right time to embrace the opportunity to start exploring the tools for trustworthiness. Deployment of these tools, integrated with IMSS will increase customer trust and adoption of solutions, remove the regulatory market entry barriers, and differentiate the solution on the market.

The following are examples of some industry trustworthiness tools that address fairness, explainability, interpretability, and robustness:

• AI Fairness 360 (AIF360)¹²⁹ – IBM has released this open-source library containing techniques developed by the research community to help detect and mitigate bias in machine learning models throughout the AI application lifecycle.

• Microsoft Fairlearn¹³⁰ – Open-source toolkit that empowers data scientists and developers to assess and improve the fairness of their AI systems.

• Explainable Artificial Intelligence (XAI)¹³¹ – Program by DARPA has the goal of developing a toolkit library consisting of machine learning and human-computer interface software modules that could be used to develop future explainable AI systems.

• InterpretML¹³² – An open-sourced code by Microsoft toolkit to improve explainability. It can also be used to explain predictions and enable auditing to meet compliance with regulatory requirements.

• Adversarial Robustness Toolbox (ART)¹³³ – An LF AI Foundation project that provides tools that enable developers and researchers to defend and evaluate Machine Learning models and applications against the adversarial threats of Evasion, Poisoning, Extraction, and Inference.