Keywords

These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

1 Introduction

Advanced manufacturing promises an evolution of industrial production processes from management to the shop floor. It combines novel approaches, methods and technologies to increase the efficiency, effectivity as well as quality of industrial manufacturing. Visual computing plays a decisive role in making abstract data and connections between data visible and understandable for the user in order to enable well-founded work decisions. However, due to the growing complexity and heterogeneity of manufacturing data, we not only face the problem of visualizing Big Data but we also need to address the growing cognitive demands of the user accordingly.

For monitoring and control of manufacturing processes the user is confronted with a multitude of heterogeneous data sources and systems. Enterprise resource planning systems (ERP), manufacturing execution systems (MES) or even task-specific but isolated applications may help him to analyze large and complex data volumes from the manufacturing shop floor. Yet, their variety in providing different visualization and interaction approaches when working with data also challenges the user’s mental model and reduces his efficiency [2].

For this reason, data visualizations need to incorporate and blend data from different systems as well as resemble familiar aspects and metaphors from the real work domain in order to reduce the cognitive workload of perceiving, processing and applying the presented information embedded in manufacturing processes. Our work aims to address these issues for the data visualization on three levels:

  • on the contextual level, we first transfer context knowledge about the working domain into a domain ontology to structure the typically unrelated data from ERP, MES, and other manufacturing data systems,

  • on the conceptual level, we then use annotations as a central means to not only encode the contextual relationships between both abstract data and information concepts from the work domain ontology, but also to incorporate additional information not yet covered by existing systems, and

  • on the cognitive level, we provide a meta-visualization based on the user’s mental model of the work domain as an overview of the data and the interrelations in which contextual information of a work task are embedded.

The remainder of this paper details the related work in Sect. 2. In Sect. 3, we introduce our contextualized annotation approach. In Sect. 4, we illustrate two domain specific implementations of our concept for the concrete case of manufacturing. Section 5 concludes this paper and states ideas for future work.

2 Related Work

A fundamental problem of the multitude of heterogeneous data sources and applications is the information overload challenging the user’s cognitive capabilities. This problem exacerbates the more data sources and thus the more applications handling them have to be used. Landesberger et al. [8] identified the lack of consistency as one of the major reasons for this problem making it a crucial design issue for integrated data visualizations. On the one hand, it can be addressed by using the same basic visualization and interaction means, e.g., from a standard library, if the heterogeneity of data allows for it. On the other hand, consistency is also a question of the user’s perception. Here, research in cognitive engineering revealed that the user’s mental model as well as his macrocognitive context plays a growing role in designing interactive applications [7]. Each user has an individual small-scale mental model of certain aspects of reality, which influences his understanding of and interaction with information. Conversely each information application is embedded into the user’s cognitive processes of perception, understanding, decision making, learning, and behavioral control forming the macrocognitive context. Thus, each application has to resemble familiar metaphors in order to simplify its cognitive processing.

A common way for providing a more consistent and uniform access to the data is based on the introduction of an additional structure such as networks to link and connect the heterogeneous data sources. On the one hand, these structures provide overviews as given in [12] to select suitable data sources relevant for a specific task. On the other hand, they can be used as a guideline to access the different sources in a specific order during the analysis [11]. Especially in the manufacturing context where many hierarchical dependencies exist, more specific data structures such as ontologies have been used [6]. An ontology in this context describes an explicit specification of a shared conceptualization [5]. In cooperation with domain users, knowledge about a specific domain is gathered and usually formalized in a tree-like fashion that consists of concepts and different kinds of relationships between them. The main structure is often similar to a taxonomy representing hierarchical relationships between general concepts and their more specific counterparts. Additionally, the ontology establishes non-taxonomic relationships between concepts. This formal knowledge representation enables machine-interpretation and sharing of knowledge.

The practice of annotating documents is a familiar activity performed by diverse groups of people with the intention of classification, communication or documentation of information [3]. Students are writing annotations in their textbooks or people are putting sticky notes at objects to annotate them; the activity of annotating is easily done and useful to support information sharing and processing by human beings. In the context of machine-interpretability, annotations are defined as metadata objects that (semantically) enrich documents with additional information. The semantic enrichment of documents through annotations usually intends to support computers in processing the information context [9]. In this way, they support activities like searching for information, structuring and shaping a document, as well as enabling service interoperability.

In this paper, we intend to combine both: ontologies and annotations to support the cognitive and computational processing of heterogeneous manufacturing related data. In this sense, we extend the utilization of annotations from a mere mean for the automatic computational processing to a more flexible tool for providing additional information for a given task.

3 Contextualized Annotation and Visualization

Following our discussion in the previous sections, the visualization of abstract data requires suitable and familiar visual metaphors close to the work domain’s original context in order to reduce the cognitive interpretation efforts. For this reason, we combine the contextualized visualization of heterogeneous manufacturing data with a conceptual annotation framework based on an interlinked domain ontology. Basically, this approach consists of five main components (an overview of the first four more data-related components is given in Fig. 1) that are grouped according to the aforementioned three levels:

  • Contextual level:

  • (1) all relevant contextual elements of the work domain are modeled with semantically interrelated concepts in a domain ontology,

  • Conceptual level:

  • (2) the data from systems like ERP and MES is attached to the ontology by annotations connecting data objects to corresponding domain concepts,

  • (3) additional information supporting the work process can be attached on the fly to both: concepts and data objects by manual annotations,

  • Cognitive level:

  • (4) the context of a given task is then used as a context filter for selecting the required information from the annotated domain ontology to be visualized,

  • (5) a contextualized meta-visualization provides a contextual overview of the data based on the user’s mental model in which the task-specific information is embedded.

We detail the basic parts of our conceptual approach, after a brief introduction to the manufacturing domain context.

Fig. 1.
figure 1

Conceptual approach of linking context knowledge with manufacturing data as well as additional information. Context knowledge is covered by the domain ontology consisting of concepts (grey circles) that are connected by hierarchical (grey lines) and other relations (dashed grey lines). The data from manufacturing systems (orange circles) is attached to concepts (grey circles) of the ontology. Additional information (blue circles) may be attached to existing data and concepts. A context filter (green area) selects appropriate concepts and additional information based on the structure of the ontology. Here, a filter has been defined for a specific data object selecting its ontology concept and attached information (small green area). Following the ontology structure the filter has been extended (larger areas) to include related information (color figure online).

Full size image

Our work addresses the general management of manufacturing work processes, focusing on the management of resources, their planning and monitoring. For us it is the work task (see Fig. 2) connecting the main context elements people, places and things, if we follow the classical definition of context from [1]. Furthermore, the users are used to dual information encodings. Working with information from the construction sub-domain, e.g. 2D drawings or 3D models, there is a close proximity between physical objects and virtual representations (see Fig. 3). However, working with production planning and monitoring applications leads to a break with the cognitive consistency. Real objects are represented by their abstract identifiers and numerical or discrete values. Analysis tasks which include the processing of both information types need to bridge this gap between the real or imaginable world and its abstract data representation, in order to keep both cognitive consistency and connectivity for the user.

Here we find the rationale for our proposed visualization approach which includes the visual adoption of metaphors and schemes from the contextual background of the work domain.

Fig. 2.
figure 2

Contextual dependency between people, places and things centered around the work task

Full size image
Fig. 3.
figure 3

3D virtual representation of a production hall containing different types of machines

Full size image

3.1 Domain Ontology

To provide a consistent and uniform view on the data, all relevant contextual elements of the work domain are modeled by semantically interrelated concepts in an ontology. An ontology describes a shared conceptualization which formally represents a set of concepts (grey circles in Fig. 1) and the relationships (grey lines in Fig. 1). The domain ontology contains domain-specific knowledge as well as adaptions to the individual business. Especially general concepts and relations such as manager assigns task to worker or product consists of parts are transferable to businesses of the same domain. However, specific concepts such as a particular product is usually bound to an individual business. Our ontology is structured according to the manufacturing context as shown in Fig. 2:

  • The people subgraph of the ontology formalizes the personas with their corresponding responsibilities and skills (e.g. planning engineers assign workers).

  • The places subgraph of the ontology summarizes the local arrangements of the work places (e.g. the production hall contains different working groups).

  • The things subgraph of the ontology summarizes all production-related materials such as tools, parts or products and also encodes their composition as specific interrelations.

  • The work tasks subgraph formalizes the different work tasks, from general work tasks such as monitoring, planning and assembly to more specialized work tasks such as sticking and soldering. Moreover, the concepts of the work tasks subgraph connect all subgraphs.

Within these subgraphs, concepts are mainly ordered by hierarchical relationships, whereas non-hierarchical relationships encode more complex relations between concepts within a single subgraph and also connect concepts of different subgraphs. A small example for such a domain ontology showing these different relations is given in Fig. 4 in which the concepts of these four subgraphs are colored in black. The assembly work task in particular relates to all other subgraphs as it is performed by a worker (people), at a work station (places) and consumes parts to produce a product (things). It is this powerful semantic knowledge base which allows for all of the following steps.

Fig. 4.
figure 4

Excerpt from an annotated example domain ontology. General context knowledge is covered by domain ontology concepts (black font) connected by hierarchical and other relations and grouped by the four context subjects: people, places, things and work task. The data from manufacturing systems (orange font) are attached to the corresponding ontology concepts. Additional information (blue font) may be attached to both, existing data as well as ontology concepts. While there are also relations connecting people, places and things (e.g. persons are usually working at a specific work station), they have been omitted from the figure for the sake of clarity (color figure online).

Full size image

3.2 Connecting Data to the Ontology

Up to this point, the aforementioned domain ontology already provides a uniform way to describe the heterogeneous data that may stem from different manufacturing systems such as ERP and MES. Yet, to also provide access to this data in a consistent way it has to be integrated into our framework. For their integration, the data of the different systems is directly linked to the corresponding concepts of the domain ontology. This linking is performed on a per object basis connecting each object to the most specific related concepts (usually leaves) of the ontology (see the orange circles in Fig. 1). Here, a single object may be attached to multiple concepts (a worker for instance may fulfill multiple professions).

While all manufacturing systems together may present a heterogeneous data basis, each individual system is generally based on a more homogeneous and well-defined data storage for handling their information. Hence, all objects of a single data source can be linked in the same way and thus their linking needs only be defined once per manufacturing system. The basis of this definition is a simple object specification listing all properties with their description and type as well as a mapping function for linking objects to concepts according to their properties. In Fig. 4 the connected data is visualized in orange.

3.3 Additional Information by Annotations

Aside from the data provided by external systems such as ERP and MES there is much more knowledge in a business that also has to be formalized and contextualized for universal access. This knowledge may contain information related to business internal processes (e.g. maintenance schedules), documents supporting the execution of work tasks (e.g. handbooks, images, videos or even interactive visualizations), or feedback of workers (e.g. documentation with text and photos). This additional information can be related to abstract concepts or concrete data objects as symbolized by the blue circles in Fig. 1.

Because of its variety, the additional information ranges from well-defined and structured documents to not yet formalized individual knowledge of the work task and related objects. Additionally, personas may have different motivations, technical capabilities and resources for providing the annotations. For instance, a stressed worker on the one hand usually has only a handheld device and not enough time to provide a complete and structured documentation of his work that may be effectively reused. On the other hand, a planning engineer who is used to organize and distribute general resources often has more time to prepare the information in a more structured way. Therefore, it is reasonable to also provide personas with different ways of specifying annotations for different use cases ranging from rather simple interfaces to more rigid and stringent forms. In the following, some examples are given to underline the differences in the way additional information can be annotated.

Flexible Annotations. The knowledge an individual worker acquires by experience is rarely sufficiently documented in a way to make it beneficial for the efficient completion of future processes. It is especially valuable, because it is unique and not easily formalized and available, e.g. from external data sources. Manually created flexible annotations can be used to make this individual knowledge available to some extent to the whole business. The particular objective of this annotation activity is the documentation of the work tasks, arisen challenges and especially the solutions.

For the annotation activity it is important that the worker receives the flexibility to first and foremost capture all available information but not forcing him to annotate formally. A user-unfriendly static input mask might irritate and discourage him from giving additional interesting information. Therefore, he can take photos, write short texts and record audio or video documentations in a quick way without disrupting the workflow significantly. These flexible annotations are easily understandable for other humans and supports the cognitive understanding and collaborative work. However, the lack of standardization or formalization affects the computational processing of the data and thus the automatic reusability. For this reason, these annotations are in turn enriched by links to the current object, work task and related concepts of the domain ontology.

Structured Annotations. Information, already to some extent available, but not yet integrated into the framework, includes for instance manuals, guides, maintenance instructions, visual models, statistical visualizations and machine configurations. This information is extensive and general, so it is needed repeatedly for similar tasks. Yet, they are also more complex and more costly to integrate.

Therefore, the managing staff first links the information to the corresponding data objects and ontology concepts of the knowledge base. For instance, the machine configuration and its maintenance schedule are directly linked to the machine itself, whereas its manual is instead linked to the concept representing its machine type making it available for all other machines of that type. They then provide the type of the information, classifying whether it is for example a manual, a visualization, or a schedule resulting in a different visual reflection of the information. Depending on the chosen type different descriptions are required and filled in, such as the dates for a maintenance schedule or the work task to be performed according to this schedule. With these additional descriptions about the content of the documents, there are more possibilities for their automatic integration and visualization.

While the planning and managing staff are used to organize and distribute these resources, providing the necessary descriptions is nevertheless a burden that needs to be supported. For example, to aid a production manager in his monitoring task a statistical visualization can be attached as an annotation to his task providing him with an overview of the parts and products (as depicted by the annotation in the upper right of Fig. 4). However, for such an annotation one does not only have to specify which input parameters to show in a visualization, but also to select an appropriate visualization from a possibly large list, to actually support the given work task. While the selection of an appropriate visualization is still an open research question, there are first works that support this selection by ranking the visualizations according to their suitability for a given task [10].

3.4 Contextual Filtering

By linking the data objects to semantically meaningful concepts as well as by attaching additional annotations, we have enriched the pure manufacturing data with additional context information which was not included in their native management systems. However, we do not need to process all objects and annotations for specific work tasks, such as monitoring the current production state of a manufacturing site. In this case, we focus on the work task’s subcontext only, meaning a subset of concepts directly related to the work task which requires a visualization or data analysis. In order to reduce the necessary data for the work task, we can use its context with its particular information demands as the main information filter.

At the same time, we can provide additional information on demand by following the relationships of the ontology. Here, the semantic relationships identify further concepts with connections to the main concepts in focus similar to the contextualized network graphs method as proposed in [4] which uses the semantic proximity to measure the relevance of connected network nodes (concepts). Following these relations, we can identify all relevant concepts with respect to the three other information hierarchies in our domain ontology (people, places, things) starting with the actual work task concept. Semantically close concepts are candidates to extend this filter for a more detailed data visualization and analysis as reflected by the green areas in Fig. 1. The domain ontology helps us here to interactively walk through the main information hierarchies in which we find the most detailed level at the leaves. Each time the user changes the work context, e.g. by switching between different analysis activities, the corresponding work task concept also changes and so do the other related information concepts to be visualized.

3.5 Contextualized Meta-Visualization

While the ontology and the attached information provides a uniform and consistent description of and access to the manufacturing data, for most persons it is too abstract to be used as an overview of the data. Hence, we utilize the user’s mental model as a base to create such a meta-visualization for presenting a general overview of the work task’s context. In general, most humans perform better in recognizing familiar faces or places instead of abstract data or concepts. Hence, a good basis for our meta-visualization are the people or places subjects of the manufacturing context (see Fig. 2). As we are especially interested in the work tasks and where they are performed, we focus primarily on the places. We therefore ground our meta-visualization on the spatial context of the manufacturing situations such as a 2D plan or a 3D representation. An example for a meta-visualization based on a 3D view of a production hall containing different types of machines is shown in Fig. 3. In this meta-visualization, we can then embed the information required for a specific work task. As the user has a high spatial understanding and knows where a specific machine is located, he can easily relate the presented information to the real world. How such an embedding of information can be realized is demonstrated for two concrete cases in the next section.

4 Plant@Hand and Plant@Hand3D

Our approach presented in the previous section was implemented in two different but conceptually similar manufacturing monitoring applications: Plant@Hand and Plant@Hand3D. Whereas Plant@Hand specifically addresses the monitoring and control of assembly works in shipbuilding industries, Plant@Hand3D focuses on the monitoring and control of production processes at a manufacturing site. Both systems integrate different data analysis and visualization functionalities, as they are usually provided by ERP or MES individually, within a consistent multi-touch user interface. Furthermore, the Plant@Hand solutions allow the connection of external displays, like big monitors, tablets, smartphones and smartwatches. While the multi-touch table generally gives an overview about the context of the work tasks, the distributed displays show the information that is most relevant for the current task and worker. In this way, they support multiple users working collaboratively with different contextualized data visualizations.

Plant@Hand aims at simplifying the central planning and supervision of installation works in ships to support the assembly across all stages of preparation, implementation and documentation. For this purpose, the relevant data and information from the leading manufacturing systems, such as ERP or MES, as well as planning documents and installation progress reports are merged and integrated with constructive models and drawings. A CAD construction plan serves as the meta-visualization representing the spatial context of the data and work tasks. The overall information is reduced to the essential details of the current task by selecting only information concerning the related area of the construction plan. Supplementary information can be accessed by clicking on objects directly in the construction plan or following links in the different windows where available. Further details can be manually added by the worker. For documentation purposes, he can annotate by text, photo, video or analogous on paper via smart pen.

Fig. 5.
figure 5

Screenshot of the Plant@Hand3D System. A 3D model of the production plant serves as the meta-visualization representing the spatial context of the data. Embedded windows provide detailed information about the work task: Diagrams visualizing trends of temperature and electricity consumption of the selected machine (upper window) and a history of key performance indicators of the same machine (bottom window).

Full size image

Plant@Hand 3D is a centralized monitoring and controlling solution for production plants. The system visualizes data from manufacturing systems, databases and file systems in an integrative and comprehensible way by using a 3D model of the production plant as the base meta-visualization. The central environment for planning and controlling is a big multi-touch table which allows an intuitive and interactive access to the information (see Fig. 5). For each workplace detailed information can be retrieved, such as key performance indicators (KPI) or information about the machine status, work tasks, orders, stock or personnel. The mobile devices allow for a workplace independent visualization and interaction. In this system, the subjects of the context model from Fig. 2 are visualized by aggregating the information about work tasks (e.g. time schedule), things (e.g. parts in stock) and people (responsible personnel) and associating it to the corresponding workplace. Accordingly, the displayed information is filtered primarily based on the selected place providing details of the related machine such as its status, KPI, or work tasks.

5 Conclusion and Future Work

In this paper, we presented an approach for a situation-based information delivery by integrating and visualizing heterogeneous information from different sources in one for the working environment familiar visual metaphor. We classified the data objects by their semantic context into people, things, places and work tasks and interlink them. For this purpose we used an ontology as the semantic knowledge base. Additional information is added by annotations. The contextualized information is then filtered according to the context of the work task and visualized intuitively using a familiar metaphor of the work environment. Here, a meta-visualization is used as an overview of the data in its spatial context in which we embed the information required for a specific work task. Our Plant@Hand solutions show how the application of big multi-touch tables and smaller displays support the information visualization from overview to detail.

The linked semantic knowledge base can enable proactive assistance and adaptive information visualization. By providing means to automatically identify a specific situation the user can be supplied with relevant information he did not know about and thus would not have searched for. While official manuals or other formalized information is ordinarily given for a work task, knowledge of colleagues is in most cases only inquired after encountering a problem (if at all). Early knowledge of challenges and their solutions obviates these problems. In this way, proactive assistance can increase the work processes. But as soon as there are too many information objects to similar topics, the filtering by the ontology concepts will most likely not be enough to prevent information overflow. Therefore, an additional ranking of the information is reasonable (e.g. by the expertise of the information composer and priority of information).

As manual formalization is inconvenient for users and easily neglected, due to the lack of motivation and time, a higher level of automation for context detection, annotating and linking of data is desirable. For instance, reliable activity recognition could enable an automatic documentation by annotations (e.g. tracking the steps of a work task). Further research should therefore take place in the area of automatic ontology evolution.