Keywords

1 Introduction

Modern day agile manufacturing [6] requires developing a framework of AI solutions that capture and process data from various sources including from human-AI collaboration [1]. Enhancing a manufacturing process by (semi-)automatized AI solutions that can support different stages in a production process that involves inter-company data infrastructure is one of the challenges in data-intensive AI for manufacturing. This challenge is exacerbated by the lack of contextual information and nontransparent AI models. In this chapter, we describe the concept of domain knowledge fusion in human-AI collaboration for manufacturing. Human interaction with AI is enabled in such a way that the domain expert not only inspects the output of the AI model but also injects engineered knowledge in order to retrain for iterative improvement of the AI model. It discusses domain knowledge fusion, the process to augment learned knowledge of AI models with knowledge from multiple domains or sources to produce a more complete solution. More specifically, a domain expert can interact with AI systems to observe and decide the accuracy of learned knowledge and correct it if needed.

The purpose of a domain ontology is to serve as a repository for domain-specific knowledge. Ontology enrichment system, as a part of human-AI collaboration, enables domain experts to contribute their expertise with the goal of enhancing the knowledge learned by the AI models from the patterns in the data. This enables the integration of domain-specific knowledge to enrich the data for further improvement of the models through retraining.

Domain knowledge fusion is a technique that involves combining knowledge from multiple domains or sources to produce a more complete solution by augmenting learned knowledge of AI models. It is used to improve the accuracy of predictive models, e.g., to guide feature selection in a machine learning model, resulting in better predictive performance [16]. Domain knowledge fusion also helps improve effectiveness of predictive models by supporting efficient dimension reduction techniques that are able to capture semantic relationships between concepts [17].

After reviewing prior research, we describe our concept domain knowledge fusion in agile manufacturing use case scenarios for human-AI interaction. We identify two kinds of knowledge: (i) learned knowledge, i.e., the knowledge generated by the AI model and (ii) engineered knowledge, i.e., the knowledge provided by the domain expert. We identify three aspects of domain expert interaction with our AI systems to observe and (a) reject if the learned knowledge is incorrect, (b) accept if the learned knowledge is correct, (c) adapt if the learned knowledge is correct but needs modification. We demonstrate these concepts for researchers and practitioners to apply human-AI interaction in agile manufacturing.

The rest of this chapter is organized as follows: in the Related Works section, we examine related works in order to identify research gaps of human-AI interaction in agile manufacturing. In the Human Feedback into AI Model section, we discuss the methodology (sub-components and interfaces) developed in the human-AI collaboration to enhance agile manufacturing. The Interaction for Model Selection and Parameter Optimization section covers the implementation of the proposed system and presents the preliminary results. Finally, in the Conclusion and Future Works section, we summarize the results and outline the future research works of the chapter.

2 Related Works

Fact-checking, a task to evaluating the accuracy of AI models, is a crucial, pressing, and difficult task. Despite the emergence of numerous automated fact-checking solutions, the human aspect of this collaboration has received minimal attention [13], though some advancement is being observed in conversational AI [8]. Specifically, it remains unclear how end users can comprehend, engage with, and build confidence in AI-powered agile manufacturing systems. In other words, enabling interaction of domain experts to AI model outputs, in order that they inspect the output and provide their feedback, helps fix errors that could lead to undesirable outcomes in production process.

Existing studies on human-AI collaboration predominantly focus on user interface (UI) and user experience (UX) aspects, i.e., whether (and how) the AI systems provide an intuitive user interface. A number of them assessed human-AI collaboration with respect to human-AI interaction guidelines as opposed to features that enable human actor to provide feedback to the AI model [4, 9]. Regarding its effect on decision-making of users has been studied using different eXplainable AI (XAI) interface designs [12].

Apart from data fact-checking and UI/UX, human-AI interaction can be done for data labeling. For example, data such as time series measurements are not intuitive for users to understand, and AI is used to generate descriptive labels [10] for a given data. Expert knowledge can augment the result of AI models by inspecting the output and complementing it. Ontology enrichment is being studied in areas of knowledge management [7], natural language processing [14], medical [3], and energy [5]. Although manufacturing with human in the loop is recently getting traction, studies on ontology enrichment have minimal attention for manufacturing.

There are several existing human-AI collaboration solutions that aim to leverage the strengths of both humans and AI systems through human-AI collaboration. Advanced applications like virtual assistants like Google Assistant [11] and Apple Siri [2] are common examples of human-AI collaboration systems. These AI-powered voice-activated assistants interact with humans to perform tasks, answer questions, and control connected devices. Most of these advancements concentrate in natural language processing, healthcare, and energy. However, the role of AI in manufacturing is mainly focused on automation and control.

This chapter focuses on data analytics and insights generation through AI models that had minimal attention despite the fact that manufacturing domain generates massive amount of sensor data. Processing and analyzing the large volumes of data can help identify patterns, trends, and anomalies, providing valuable insights to support decision-making. The chapter develops a tool that enables humans to collaborate with AI systems through intuitive interfaces that help domain experts in interpreting insights, validating the findings, and applying domain knowledge to gain a deeper understanding of the data.

3 Human Feedback into AI Model

The purpose of human feedback into AI model is enabling domain experts to inject their knowledge via predefined interfaces allowing for collaboration with the system in order to connotate previous knowledge with semantics, as for instance with a description of a specific process or data. It helps better understanding of the data, as it also provides the possibility for a better evaluation of the whole AI pipeline. In other words, human-AI collaboration is a component that offers interfaces between domain expert and AI system. The functionalities offered by the human-AI collaboration are to enable human feedback for domain experts, i.e., machine operators and managers without the need to understand the intricacies of AI models.

As shown in Fig. 1, the human-AI collaboration is composed of multiple sub-components and interfaces that enable communication with external systems such as data sources, model repositories, machine configurations, and decision support systems. The main sub-components described below are interface abstraction, model and data selection, parameter optimization, configuration adaptation, domain knowledge enrichment, and domain knowledge repository.

Fig. 1
A block diagram illustrates the Human-A I collaboration interface. The indicated elements include machine configurations, A I model, domain knowledge, configuration, A I model, knowledge repository, interface abstraction, U I, D S S, and domain expert.

Human-AI collaboration components and interfaces

Full size image

3.1 Interface Abstraction

Interface abstraction component is a container for configuration adaptation, model/data selection, parameter optimization and adaptation, and domain ontology enrichment components. It provides an interface to the domain expert through the decision support system through intuitive and user-friendly interfaces that enable effective communication and cooperation between humans and AI. Interface abstraction is beneficial for human-AI collaboration by playing a crucial role in enabling seamless cooperation and enhancing the productivity and usability of AI technologies. The goal of interface abstraction is to bridge the gap between the capabilities of AI systems and the understanding and expertise of human users. It allows users to interact with complex AI technologies without requiring them to have in-depth knowledge of the underlying algorithms of AI models. In effect, it empowers users to leverage the capabilities of AI systems while focusing on their own areas of expertise. By abstracting the complexities of AI algorithms and technologies, interface abstraction facilitates effective communication and collaboration between humans and AI.

3.2 Model and Data Selection

The human-AI collaboration system offers features for data and model selection. Operators select models and data from the available list of options in order to execute them for a specific scenario. Model and data selection are critical factors in human-AI collaboration. Because the choice of model and data significantly influences the performance, accuracy, and overall effectiveness of the AI system. When considering human-AI collaboration, several key considerations come into play. One aspect is determining the specific requirements of the task at hand, understanding the problem domain, the type of input data, and the desired output. This knowledge helps guide the selection of an appropriate model for the dataset. Another aspect is understanding capabilities of the AI model because different AI models and algorithms are suitable for the task. Considering factors such as the model’s architecture, complexity, interpretability, and scalability affect the choice of a model that aligns with the task requirements.

3.3 Parameter Optimization

Parameter optimization is an important step in human-AI collaboration to ensure optimal performance and effective interaction between humans and AI systems. Operators and managers perform the optimization of parameters that offer the best outcome for the given scenario. The system provides them with an interface where the operators can select the parameters, try various values, and observe the results.

It involves continuous evaluation of the performance of the system and collecting feedback from the human collaborator. This feedback can be used to identify areas for improvement and guide the parameter optimization process to iteratively refine and fine-tune the parameters based on the evaluation results and feedback. The parameter optimization is necessary for the domain expert to deal with trade-offs between different performance metrics or constraints that need to be satisfied. For example, optimizing for accuracy may lead to longer response times, which may impact the user experience.

The first step in parameter optimization is to identify the metrics or criteria that will be used to measure success. This will help guide the parameter optimization process. Once the parameters are identified, the next step is to determine the metrics that evaluate the performance of AI model, such as efficiency and accuracy. In this chapter, an example implementation of parameter optimization is shown in Fig. 4.

3.4 Configuration Adaptation

Configuration adaptation is the process of adjusting or fine-tuning the configuration settings of AI systems to better align with the needs, preferences, and context of human users. It involves customizing the equipment, parameters, or policies of AI models to optimize their performance. Feedback of domain expert plays a vital role in configuration adaptation as it provides valuable insights into the effectiveness and suitability of the AI model’s behavior in that the AI system can learn and adjust its configuration settings to improve its performance and align more closely with the user’s requirements in response to incorporating domain expert feedback. Moreover, when a model offers a need for specific configurations of machines that need to be modified, operators/managers can adapt the configurations of machines so that it suits to the model under consideration, for example, if new machines need to be added to the human-AI collaboration system, their configuration should be extracted and stored in such a way that they are accessible and usable to the modules.

3.5 Domain Knowledge Enrichment

Enriching learned knowledge with engineered knowledge describes the scenario where the AI model analyzes the given data for a task (e.g., outlier detection) and produces its result (e.g., that a given data point is an outlier), the domain expert realizes that the output of the model is not right (e.g., that the data point is not an outlier), and the information provided by the domain expert (i.e., the data point is not an outlier) is stored in the repository of ground truth and sent back to the AI model for retraining. It is used by operators and managers to enrich the knowledge repository with new entries obtained from the execution of the system using diverse setting of models, parameters, and configurations.

The key challenge of this approach is that it relies on the availability of domain experts. Scarcity of domain experts (that most of them spend their time on machine monitoring, operation, and management), limited availability of domain expertise, rapidly evolving AI landscape, and its demand for interdisciplinary skills make this challenge difficult to handle. Developing AI models often requires deep domain expertise in specific fields, e.g., manufacturing in this case, and experts who possess both domain expertise and a solid understanding of AI techniques are difficult to find. Moreover, effective collaboration between domain experts and AI practitioners often necessitates interdisciplinary skills. Domain experts need to understand AI concepts and methodologies, while AI practitioners need to comprehend the nuances and complexities of the specific domain. The scarcity of individuals with expertise in both areas makes the task of domain knowledge enrichment challenging.

In this chapter, the limitation of AI model development knowledge of domain expert is taken into account. Having listed these challenges, this research assumes that the involvement of humans in enriching knowledge will potentially reduce over time. As such, an initial set of AI models are trained and made available for the domain expert to experiment with them before trying to perform parameter optimization and feedback provision. At the beginning of a collaboration between humans and AI, there will be a significant effort to optimize parameters and transfer human knowledge and expertise to the AI models by providing more data, defining rules, and setting up the initial knowledge base. However, through retraining, the AI model learns and accumulates more data that it can gradually require less feedback from the domain expert.

Organizations can benefit from this system despite these challenges by defining feasible objectives of the human-AI collaboration in the manufacturing setting whereby they identify the specific areas where AI can enhance manufacturing processes, such as quality control, predictive maintenance, or supply chain optimization and establish key performance indicators (KPIs) to measure success. For example, companies can utilize this approach to perform what-if analysis in order to explore the potential implications of different scenarios and make more informed decisions by combining the analytical capabilities of AI models with human judgment, expertise, and contextual understanding. Domain experts can modify the input parameters, adjust variables, or introduce new constraints to observe the potential changes in the outcomes. The AI system then performs the simulations and presents the results to the human collaborator.

3.6 Domain Knowledge Repository

Domain Knowledge is the repository of knowledge (both learned knowledge generated by the AI model and engineered knowledge curated by the domain expert. Machines and production processes are undergoing a rapid digital transformation, opening up a wide range of possibilities. This digitalization enables various opportunities, including early detection of faults and pricing models based on actual usage. By leveraging sensor data analytics, it becomes possible to monitor machine operations in real time, providing valuable insights and applications. This is better achieved if the domain experts assist in improving the quality of AI model output by providing domain knowledge, for which this component is responsible to store.

4 Interaction for Model Selection and Parameter Optimization

Improving the effectiveness of AI model requires a comprehensive understanding of the model’s design and implementation and it can be achieved in a number of ways: (i) reviewing the input data, including the quality, completeness, and relevance, to determine if it can be modified to improve the output, (ii) analyzing the output data of the interaction model can help identify patterns and trends that can be used to modify the model’s output and identify areas for improvement or optimization, and (iii) modifying the algorithms used in the interaction model can help improve the output. In this chapter, the second method is used, i.e., the domain expert provides feedback on the output of the AI model.

An example scenario that shows the procedure for the human-AI interaction is shown below:

  • Fetch predicted labels form the output of automatic label detection of models.

  • Present the data with predicted label to the domain expert.

  • Present the domain expert with choices to (i) accept the predicted label and (a) confirm the predicted label or (b) offer an alternative label or (ii) reject the predicted label and offer the correct label.

    • If the domain expert accepts and confirms the label, the process ends.

    • If the domain expert accepts the predicted label and offers an alternative label or a refinement of the label or rejects the predicted label altogether and offers the correct label, the domain expert’s input will be sent as input to retrain the model.

  • Visualization of behavior of the model with/without the domain expert’s input will be shown for comparison of the effect of the domain fusion.

  • Human-AI collaboration system will expose an API of the visualization to the DSS component through which the user will inspect the outputs of the model.

Figure 3 shows the process of data/model selection and parameter optimization including data flow and UI mockup for model selection and parameter optimization user interface through which the domain expert selects the model and parameter and optimizes the parameter values. The UI presents visualization of processing results for the selected model, parameter, and values. Once the domain expert determines the model, parameter, and values, the UI then enables the domain expert to export the result which will then be consumed by the Decision Support System (DSS).

The domain expert selects a section of the visualization and provides engineered knowledge, i.e., manual labeling of data points. This helps the user to visually inspect the dataset and enrich it with domain knowledge to boost the quality of the data to be used as training dataset for better performance of the ML model. For example, for an AI model built for anomaly detection, this is achieved by enabling the user to select the data point on the visualization plot in order to display and review (and when applicable, modify) the data that are marked by the system as anomalies. This is implemented by providing point, box, or lasso [15] selection where the user can select a single (or multiple data points on the graphs) and get the corresponding data points back, to provide the domain knowledge.

As depicted in Figs. 2 and 3, the domain expert will load data and models from the model repository, run the models on the data, observe the visualization, and adjust parameters in order to achieve the desired behavior of the AI model. Once the domain expert obtains satisfactory output from the model, she/he can then provide feedback. The algorithm shown in Fig. 4 shows the detailed operations during the domain knowledge enrichment.

Fig. 2
A flow diagram denotes the flow from the catalog algorithms going through the A I model, data to accompany the A I model, followed by different user interfaces before reaching the model training. It loops back to the catalog algorithm after the model training.

Human-AI collaboration flow diagram

Full size image
Fig. 3
A screenshot of a user interface denotes the options to select internal temperature, pressure, and ambient temperature features on the left panel. The primary section exhibits the options to upload the c s v file, select model, and select preprocessor, followed by 2 line graphs.

Human-AI collaboration main interface

Full size image
Fig. 4
A set of lines of pseudocode represents the algorithm for domain knowledge enrichment. The input is the u r l of the model repository. The output is the data enriched by domain expert feedback.

Pseudocode showing high-level process of human-AI collaboration

Full size image

5 Conclusion and Future Works

This chapter discusses the concept of a framework of human-AI collaboration in manufacturing for injecting domain knowledge provided by human experts into AI models, as provided by machine learning processes in order to iteratively improve AI models. It explores the importance of human feedback in enhancing the effectiveness of AI models and improving usefulness of their outputs through incorporating human feedback. It describes a use case scenario to showcase the implementation of human-AI interaction where human feedback is utilized to enrich learned knowledge.

There are a number of future works in this chapter. An implementation of a full-fledged human-AI software prototype should be implemented and deployed so as to measure its effectiveness and also to experiment on the varied use cases to measure its actual usability in the real world. For this, the questions such as how the introduction of human-AI interaction affects the performance and effectiveness of the AI model, and for the given test, how much of the output of the AI model is rejected, accepted, and modified need to be answered.

Another aspect of the future research is to analyze whether AI model produces erroneous results even after retraining using expert feedback, i.e., whether the accuracy of the retrained AI model shows any improvement.

Yet another aspect is a study on scaling human-AI collaboration that takes into account the scarcity of domain experts who have understanding of intricacies of AI modeling or AI developers who have sufficient knowledge of the domain of manufacturing. Long-term studies of how to scale human-AI collaboration are of paramount importance because scaling up the human-AI collaboration approach to large-scale manufacturing settings presents challenges in maintaining consistent collaboration and effectively incorporating human feedback across various use cases, domains, and data volumes. Therefore, it is important to further research scaling up human-AI collaboration in large-scale manufacturing settings through a well-planned approach that involves aligning technology, data, technical/domain expertise, and processes to create a seamless integration of human and AI capabilities, ultimately enhancing productivity, quality, and efficiency in manufacturing operations.