1 Introduction

Many accidents in servicing/mechanical repair result from improper control and supervision of service works, insufficient knowledge about the current state of a service environment, or lack of specialist knowledge about the repair of, often highly technically advanced, machinery. Proper safety protocols can save lives, reduce accidents, reduce downtime, and increase productivity.

To manage service and repair works, especially in an environment dangerous to life and health, the developed proprietary INRED system (described in Section 2) integrates copyrighted and innovative solutions based, inter alia, on augmented reality (AR) that provides employees with computer generated information (e.g., 3D models of the serviced machines), which enhances their perception of reality and supports them during complex repair and renovation procedures.

The INRED system mainly aims to improve the quality of highly complicated repair and service processes by ensuring their proper execution, continuous surveillance, and reporting their accomplishments. The functionality of the system system allows to define each service procedure as a workflow process, automates some activities (e.g., measurements control and decision-making), and delivers a detailed description of commissioned tasks and adequate technical documentation (pictures, technical drawings, 3D models with animations, movies, and technical parameter values). Each serviceman can be equipped with various tools (a mobile terminal, MS HoloLens, cameras, Bluetooth communication devices, etc.) for automatic/procedural control of device settings, transfer of measurements and their instant validation based on existing standards. The Mobile Devices Communication Subsystem is used to manage the integration of the mentioned tools with the INRED system. Mobile and stationary cameras are used to document the tasks carried out, to provide the constant surveillance, and to automatically correct services based on advanced image recognition algorithms which detect several types of events. All components of the INRED system (surveillance system, workflow engine, repositories, databases, etc.) are integrated through the intelligent Wi-Fi/Bluetooth-based communication system. The INRED system combines many elements of the concept of Industry 4.0, Internet of Things (IoT), and Industrial Internet of Things (IIoT). The innovativeness of the INRED systems relies on creating a workflow engine that supports the renovation services, implementation of an editor to handle renovation procedures, and creating from scratch the system to organize wireless communication between the system elements, to register tool, and possibly to correct settings.

The rest of the paper is organized as follows. The INRED system is described in more detail in Section 2. Section 3 gives an overview of related works. The INRED-Workflow system is discussed in Section 4. The Service Procedures and the mobile device communication system are described, respectively, in Section 5 and Section 6. The operation of the INRED-Workflow is shown in Section 7 on the example of the Check valve state process. The paper ends with concluding remarks.

2 INRED System

This section provides more details on the INRED system. As already mentioned in the introduction, the main objective of the INRED system is to control all phases of the repair and service processes performed in a dangerous industrial environment. The functionality of the system is organized into services, with each service delivered from a specialized module. The main modules (see Fig. 1) of the system include:

  • Corporate Competence Center – a set of user interfaces that enable controlling currently realized processes and access to archived data.

  • Platform Process Management Knowledge Base (PPMKB) – manage the exchange of information between the Corporate Competence Center and service units and thus enables automation of services processes.

  • Communication Management System – self-organizing communication network.

  • Enterprise Process Management – manages service processes (workflows); this module is a key element of the INRED-Workflow system.

  • Mobile Devices Management – manages wireless devices (tools) used in the INRED system.

  • 3D device models – stores and manages 3D animations and models of the serviced machines.

  • System Knowledge Repository – advanced ontology-based repository system; it is responsible for storing and processing data coming from different sources, e.g., from processes execution; this data is used by other INRED modules such as Enterprise Process Management, Platform Process Management Knowledge Base, and Objects and Threats Recognition

  • Objects and Threats Recognition – intelligent system for environmental monitoring; it is used, inter alia, to predict and detect suspicious, dangerous, or unusual situations, i.e., irregularities or deviation of the object behaviour from the normal behaviour (in an unusual location, an unusual motion pattern such as movement in the wrong direction can be observed, a sudden movement of a person, a group of persons or an object can occur, a person or an object can enter in or access a restricted area, etc.). All these extremely complex tasks are managed by the system by using advanced algorithms based on machine learning (Bayes network, decision tree, neural network, deep learning, soft computing, rule-based system, and approximate optimization).

  • Individual Diagnostic System – an advanced system of surveillance and collecting biometric data of employees.

Fig. 1
figure 1

General scheme of INRED system

Full size image

To fully support the INRED system, additional modules will be implemented in the future, such as:

  • Data and Video Streams Management Platform.

  • Mobile Devices Communication Subsystem.

  • Semantic Search Engine.

Smart Procedures are an important element of the INRED system; they merge services supplied by modules such as Objects and Threats Recognition (e.g., using machine learning algorithms and image recognition), Mobile Devices Communication Subsystem and Mobile Devices Management. Smart Procedures have a standard programming interface, so they can be easily integrated (already at the stage of their creation) with workflow procedures.

3 Related works

Workflow Management (WfM) systems aim at modeling and and controlling the execution of application processes involving both human and machine-based activities, in complex technical and organizational environments. In August 1993, the Workflow Management Coalition (WfMC) [27] was established to standardize workflow systems. In 1995, WfMC published the workflow reference model which standardises a set of interfaces and data interchange formats between workflow products to achieve interoperability at a variety of levels.

WfM systems are rapidly evolving, and are increasingly often used in a variety of industries. However, despite there are several producers of the workflow systems (e.g., Creatio, Nintex Workflow, Appian, BP Logix, and Oracle Workflow), there is no any system allowing for integration of service processes with wireless tools, augmented reality technology, and image recognition algorithms (these and other limitations of existing workflow systems are discussed in many papers [8, 26]). The INRED system aims to, at least partially, fill this gap. Below we present an overview of the literature on image recognition, and data acquisition through wireless devices and IoT.

Image analysis based-methods consist of a number of sequentially implemented algorithms, such as camera calibration, preprocessing, or image segmentation (extraction of objects of interest), and measurement algorithms. Xu & Brownjohn [28] presented a detailed description of these algorithms and discussed their advantages and limitations. The theoretical considerations were illustrated by an example of the vertical displacement of the Humber Bridge.

The accuracy in an automatic measurement of flat, thin, rectangular objects by using computer vision techniques is discussed, e.g., in [15]. The proposed therein system consists in a CCD camera (calibrated by using a standard gauge) positioned in the upper viewer of the object. The authors highlight the difficulties with automated inspection of images. A geometric part measurement system for shaft parts in presented in [16]. This machine vision-based system uses a stationary camera mounted on the calibrated stand to obtain the image of a shift part. The obtained image is first preprocessed using wavelet denosing, then the edge contour of the part image is detected using single pixel edge detection method which is based on the Canny detection operator, and finally the geometrical quantity algorithm is applied for the measurement purposes. The repetitive measurement error of this method was less than 0.01 mm.

Data acquisition based on Internet of Things (IoT) is a subject on many publications. Most of them refer to the implementation of systems allowing data flow from IoT devices to digital repositories. Data stored in a repository can, when needed, be preprocessed and used to improve the processes carried out in various systems [25]. Luis et al. [17] presented a framework to exploit IoT devices for real-time decision-making on the construction site. First, sensor information at their point of origin is synthesized into resource state, and then this state is used as feedback into a process model of an operation. Then, decisions programmed into the process model are made automatically based on real-time status of the operation and relayed back to the entities on the construction site through the IoT infrastructure [17]. Belli et al. [1] discussed the security threats and vulnerabilities of the IoT heterogeneous environment. They proposed possible solutions to improve the IoT security architecture. Bermudez-Edo et al. [2] presented a methodology to analyse data streams obtained from various data sources, e.g., IoT devices. They analysed data streams using different spatio-temporal correlation-based algorithms and discuss co-occurrence and causation phenomenon on the Smart City Infrastructure example.

Castano et al. [3] presented a Bluetooth/spl trade/-based wireless solution to monitor wireless industrial sensors in small to mid-size industrial environments. Gore et al. [9] described a solution based on Bluetooth Low Energy (BLE) technology to connect sensor nodes to Internet-based services and applications using a gateway in an industrial plant. The operation of this solution was shown in a case study of a typical industrial plant. Additionally, the performance of BLE technology is assessed using various performance parameters. Magliulo et al. [18] presented a Bluetooth-Activity Tracker (BT-ACT) for workflow management and patients’ performance monitoring in a Radiation Oncology Department. They implemented the integration of ACT with a novel electronic medical record archiving and retrieving system and demonstrated that patient workflow and bio-metrics can be successfully monitored with BT-ACT during radiotherapy.

Kammerer et. al. [12] presented context-aware guidance of cyberphysical processes (CPS – a computer system which controls and monitors a mechanism by computer-based algorithms, e.g., workflow). Their approach enables the context-driven injection of predefined process fragments into a basic template of the cyber-physical process during run time. Users are supported in interaction with the process through an augmented reality application, which integrates position and marker sensors into the context graph.

Haslgrbler et al. [11] presented how IoT devices are used to perform workflow processes and attention recognition. They also showed how IoT devices are used to provide critical notifications that can be perceived by users in an industrial environment to improve the quality of process performer decisions and prevent harmful situations.

Methods to understand the behavior of human activities through visual cues in industrial environments are proposed by Makantasis et al. [19]. The authors define the problem of recognition of human activity as the classification of each frame to one of the available tasks in the workflow process. The classification task was performed using the convolutional neural network (CNN).

The problem of formalization of industrial processes and data modeling using the entity-relationship (E-R) diagram is addressed in [22], whereas [29] focuses on some important aspects of constructing business training platform based on the workflow management systems.

Dasani et al. [6] presented methods to ensure process safety and workflow procedural compliance. The system continuously monitors and compares operator actions with standard workflow operating procedures, and identifies in real-time procedural violations that could compromise process safety.

Kyjanek et al. in [14] proposed an interactive production process where participants interact with a robot system through an augmented reality (AR) interface. Worker can use Microsoft Hololens (augmented reality head-mounted display) as an interface to plan robotic trajectories, influence production sequencing, and view superimposed diagnostic feedback in Timber Prefabrication.

The concept of image-based measurements we propose in the present study differs from other image-based measurements mainly in that the position of the camera is set dynamically. More precisely, the measurements are made based on the image obtained from the camera mounted on the helmet of an employee or on the tripod, which causes that the distance between the camera and the object is not fixed. Clearly, the precision of the measurement depends on the size of the object in the image, but generally distance is not a problem.

Moreover, the proposed in this paper approach to workflow management differs from the existing approaches in that the process control procedures (e.g., Smart Procedures) are independent of the process and can be freely introduced on an ongoing basis in its various parts using various measuring devices. It should be noted that all gathered data has an owner who is responsible for data quality, and that control procedures can affect the execution of the process by changing the decision parameters. In addition, to increase the productivity and reliability of process performers, 3D devices models can be displayed using an augmented reality interface.

4 Process management

The INRED-Workflow system, which is one of the key components of the INRED system, provides the infrastructure for process (workflow) automation. Each workflow consists of a sequence of tasks and transitions between these tasks. Tasks in a workflow may be performed in sequentially or in parallel by a machine, a human, or both. The INRED-Workflow system stands out from other workflow systems in that it has the following unique features:

  • The ability to define and operate the so-called Smart Procedures.

  • The ability to annotate images.

  • Built-in media types to support images, animations, and movies.

The system is ready to provide hints on how to perform a specific task (for example, when trained, but often inexperienced, personnel perform a service procedure) at every stage of the workflow process. A hint can take the form of a technical drawing with selected elements that require action at a given moment or an animation that allows the user to view elements and activities from different perspectives and select the most interesting. The system can also view videos of previously performed service works. Additionally, each multimedia material can be annotated. The INRED-Workflow system includes the following components:

  • INRED-ProcessDesigner – graphical editor of workflow processes which allows to define a process according to the BPMN (Business Process Model & Notation) or XPDL (XML Process Definition Language) standard.

  • INRED-WorkflowClient – designed to support process participants; it provides work items to all process participants, supervises the execution of tasks, and supports process participants in the execution of certain tasks.

  • INRED-WorkflowEngine – performs workflow processes; the engine interprets the process definition, creates tasks, and supervises their execution.

  • INRED-AuditAndReportingTool – tracks and captures all tasks and actions related to a particular workflow; it stores all data (e.g., user decisions, measurement parameters) from the execution of processes for further analysis and reporting.

5 Repair and service procedures

The operation and maintenance of industrial systems is, in many cases, time-consuming, complex, and resource-intensive. The proposed process management system enables automation of various very complex industrial processes by using the developed procedures. An example of such a procedure can be the procedure of the refurbishment of the steam rotor of the power generator in the power plant or the valve repair procedure. In many cases maintenance and service procedures have to preserve a strictly predefined order and should provide data confirming the quality of the tasks performed.

Therefore, the activity performer (process participant) is controlled at every stage of the procedure by the process management system. Moreover, if required, the process supervision can be carried out by participants with special permission, such as quality managers and technologists.

5.1 System Knowledge Repository

All data gathered from the process is stored in the System Knowledge Repository (SKR) for future use. The gathered data includes, among others (Fig. 2):

  • current state of ongoing processes,

  • full audit of completed processes: performers, time of actions, positions, etc.,

  • data collected from the tools involved in the process execution (parameter values obtained during the measurement of the parts of repaired objects),

  • multimedia data (photos and movies gathered during the service processes).

The SKR includes the following input data for the service process:

  • Workers and Organisation Units – organizational chart.

  • Parameter Target Values.

  • Technical Device Specifications.

  • Repair Processes Definition – workflow procedures.

  • 3D device models.

  • Movies/3D Animations.

  • Repair and Maintenance Manuals.

  • Metadata.

  • Smart procedures.

Fig. 2
figure 2

System Knowledge Repository with input and output data for/from process management

Full size image

The use of external procedures (cf. Fig. 3) allows newly defined algorithms to be easily implemented in the workflow, e.g., by using machine learning methods.

Fig. 3
figure 3

Implementation of external procedures in workflow process

Full size image
Fig. 4
figure 4

Example of 3D models - valve

Full size image
Fig. 5
figure 5

Example of 3D models - power generator

Full size image

Processes can be implemented and modified using a special tool, the Workflow Editor. Apart from the process flow, process control parameters, and Smart Procedures, the Process Editor allows attaching to the process 3D device models (see Figs. 4 and 5) and animations with annotations in 2D and 3D space, definition of the reports, permission to the execution of the activity by the exactly specified employee, groups of employees or organisational unit.

5.2 Valve repair and service procedure

To illustrate the repair and service procedure, let us consider the main flow of the valve repair process depicted in Fig. 6. The presented process involves several stages to repair different types of valves. Each stage (also called activity) of the main flow is implemented as a sub-process which returns data to the main process. We would like, however, to point out two activities:

  • Valve classification stage during which the mobile communication subsystem automatically delivers the repaired valve parameters. A process participant sends the obtained parameters directly to the workflow process using dedicated tools, such as a micrometer screw or caliper, equipped with the Bluetooth communication interface, which is used to transfer data. If the transferred parameters are correct, the process is continued; otherwise, the process forces the repetition of incorrectly performed tasks. The mobile device communication subsystem is described in Section 6.

  • Check valve state stage during which the quality of the performed activity is assessed by using intelligent image processing methods (one of such methods is described in Section 7. These methods allow, for example, to detect the incorrect valve state, i.e., the position different from the three acceptable positions/states (specified by the process creator during the process description): opened, partially opened, or closed. If the detected position does not match any of the three target positions, the valve position should be corrected by repeating the adequate activity. This repetition can be implemented, as shown in Fig. 6, by the feedback between the activity Check valve state and the activity Set target valve state.

Fig. 6
figure 6

Main flow of Valve Service and Repair workflow process

Full size image
Fig. 7
figure 7

Implementation of Check valve state workflow sub-process in Valve Service and Repair workflow process (Smart Procedure)

Full size image

The activity (sub-process) Check valve state (cf. Fig. 7) is implemented by using (smart) procedures which wrap the algorithm that evaluates the state of the valve by analysing the video collected from a stationary or helmet-mounted camera. At the beginning, the sub-process obtains the screw parameters which allow the image analysis algorithm to assess whether the valve is open or close. If the acquired image quality is sufficient, then Screw visual assessment activity is performed and the appropriate value of the ValveState parameter is set.

6 Mobile device communication subsystem

Environmental data in INRED system can be measured by various mobile devices and other instruments spread around the facility. Therefore, a special INRED subsystem, called INRED Mobile Devices Subsystem (Fig. 8), was developed to enable communication with measurement tools and devices over multiple technologies and protocols, and what follows to automate as much as possible measurement procedures. This subsystem was designed to provide system communication with Bluetooth-enabled, Wi-Fi or other mobile devices. Quite complex subsystem architecture was established here. It includes a server cluster and two types of special hardware nodes: Bridge processors and mobile media converters (bridges). With such an architecture, subsystem deployment will be possible in almost any kind of physical environment. Low range piconet media (Bluetooth, ZigBee, etc.) will be extended (by remotely and automatically managed media converters, bridges) to a standard Wi-Fi network. These mobile bridges could be considered either as personal devices (hosted by people in the environment) or deployed in the environment infrastructure nearby low range wireless devices. Measurements will be passed through bridges to other nodes called bridge processors that can process the acquired data including protocol conversion, device monitoring, statistical analysis, etc.

Since there is a vast number of Bluetooth-enabled device types on the market, the INRED subsystem software must additionally provide extremely flexible device integration solutions. The problem here is to identify and integrate new device types, sometimes even not formally presented by manufacturers in manuals. Because each mobile device provides an unknown set of features over an unknown protocol the integration mechanism must foresee the unpredicted behaviour of new devices accepting and identifying in such cases new device types in real-time. These devices could not even exist during system development and still integration should be possible in the future. To achieve that, a layer of micro-services (Fig. 9) was prepared in the system. Micro-services can be provided by outside vendors and function as device drivers. Each new (yet unknown) Bluetooth device can now be linked to a micro-service and can be recognized by the system once the driver is installed. To gain scalability and system security (micro-service is a foreign code) they will not be hosted by a central server, but spread around on bridge processors. Bridge processors will convert and pass data between the system core and bridges.

Fig. 8
figure 8

Architecture of mobile devices subsystem and data-flow between modules

Full size image
Fig. 9
figure 9

Implemented microservice architecture

Full size image

To ensure the extensibility of the system, all messages passed between system components will be formatted in JSON notation. The system will also achieve high scalability thanks to the integration of high numbers of bridge processors, bridges, and mobile devices without performance problems. Bridge processors will act as signal-conversion units translating device messages to TCP or UDP traffic. From that point devices will be available for users. Measurements can be taken automatically from the device. In more complex scenarios, the user can be instructed step by step on how to perform the measurement. To provide subsystem functionality to endpoint users client applications are also designed. Finally that software will render measurement information on a desktop application 3D scene or VR hologram images in VR goggles.

7 Smart procedure for visual assessment of final stage of valve renovation

At the final stage of the valve renovation process, the condition (image) of the device after successful completion of the repair activity is verified. The target stage can be treated partially - as the end of an intermediate operation, or globally - as a final measurement to determine that the repair has been completed (see Fig. 10). The method used to recognize the condition of the object after the renovation is presented on the example of the operation of fully opening the valve [13].

Fig. 10
figure 10

Example of valve; QR code is used to identify valve, hand-wheel is used to set length of screw, and caliper is used to measure length of caliper

Full size image

It consists in measuring the length of the screw extending beyond the handwheel used to open the valve. The length of the protruding part of the screw should match the length stored in the database. The preferred method is to measure the length of the screw with a special caliper, transmitting the result via the Bluetooth. However, to protect against the lack or possible damage of the caliper, an alternative (computer vision based) method was proposed in [5].

The correct result of the algorithm requires a lot of involvement of the employee who must properly acquire the digital image. It is essential:

  • to obtain a large zoom of the part of the spindle protruding beyond the handwheel,

  • to set the camera so that the image of the screw is located parallel to the horizontal edges of the image,

  • to provide an appropriate background, which should be reasonably uniform and bright.

The employee obtains this information together with a sample image from the workflow program, which also shows the subsequent stages of image analysis. If the algorithm is not working properly, the employee can repeat the analysis by taking another photo. The general scheme of the analysis has not changed from that presented in the previous work [5]. The following description highlights the steps that need to be automated:

  • the image is converted to grayscale and smoothed using median filtering,

  • the image histogram is calculated,

  • the background pixel range is determined,

  • the image is binarized – pixel values outside the background range are set to 1,

  • the image is subjected to morphological operations and then projected onto the axes of the coordinate system,

  • the distances (in pixels) corresponding to the bolt diameter and length are calculated,

  • the length of the screw (\(l_{mm}\)) is calculated according to formula (1):

$$\begin{aligned} l_{mm} = (l_{pix} \cdot d_{mm}) / d_{pix}. \end{aligned}$$
(1)

where \(d_{pix}\) is the bolt diameter in pixels, \(l_{pix}\) - screw length in pixels, \(d_{mm}\) - screw diameter in millimeters - read from the database. \(d_{mm} = 16 [mm]\) for the valve located on the test stand at AGH University of Science and Technology in Krakow. For constructing and testing the algorithm, 4 images (960x540 pixels \(\times\) 24 bpp) were used (see Fig. 11). As one can see in the figure, the pixels belonging to a screw are darker than the background, which is consistent with the assumptions. On the other hand, reflections are visible on the screw, so to perform a correct segmentation it is necessary to detect the background boundaries and then apply two-threshold binarization.

Fig. 11
figure 11

Images used to construct and test algorithm for determining screw length: a) Img162small, b) Img163small (to construct), c) Img164small, d) Img166small, e) Img161small

Full size image

7.1 Background boundaries

The background area should dominate in the image, so the analysis begins with finding the maximum of the histogram. To locate the background boundaries, several methods were tested, two of which were selected for presentation: Gaussian approximation (single and bimodal) and the method of calculating the envelope. Gaussian approximation does not require the definition of additional parameters, it is only necessary to declare whether the approximation is to be single or bimodal.

The results are shown in Fig. 12. The shape of the background histogram is asymmetric, so neither of the two Gaussian functions approximate it correctly. This is especially visible on the right side of the background histogram. The envelope of the function (in this case, the histogram) can be calculated by various methods: using the Fourier transform, FIR filters or approximation by splines around local maxima. For testing, an approximation was chosen with a moving window w in which the Root Mean Square function rms(x) is calculated. Figure 12 shows the results of the approximation. All three envelopes generated for window lengths \(w=5,7,11\) have minimum points that can be used to detect background boundaries. This is shown in the enlargement of one of the areas in Fig. 12, where the black dashed line passes through the minimum of the envelope generated for the window \(w-7\). Therefore, to define the background boundaries one has to:

  • calculate the image histogram (hh),

  • determine the histogram envelope by using Matlab function envelope(hh.Values, w, ’rms’), where \(w = 7\) is the window width,

  • find the maximum value of the histogram and divide it by 2,

  • determine the values of the envelope corresponding to half of the histogram (see the black horizontal line in Fig. 12),

  • determine the local minima for the left and right sides of the envelope,

  • the corresponding values on the x axis should be taken as the background boundaries - on the left and right sides of the histogram, respectively (see vertical black dashed lines in Fig. 12).

Fig. 12
figure 12

Histogram approximation: solid black and magenta lines - Gauss1 and Gauss2, red lines: solid - window = 11, dashed - window = 7, dotted - window = 5 - envelope. The magnification shows the minimum point for window = 7, which was taken as the background border of the histogram left side

Full size image

The next step is to binarize the image (ob) using the Matlab code:

ob_bin = (ob < bkg_aut(1)) | (ob > bkg_aut(2));

where bkg_aut(1) and bkg_aut(2) are the background boundaries determined in the above-described algorithm and the ’|’ character is the ’OR’ operator.

7.2 Screw parameters

To determine the length of the screw (in [mm]), one needs to know its real diameter (in [mm]) and two distances measured in the image (in pixels): diameter and length. According to the concept of the algorithm presented in [5] the binary image is projected onto the axes of the coordinate system. In this way, two functions arise (see Fig. 13).

Fig. 13
figure 13

Img163small image analysis: a) binarized image, b) y-axis projection (red, rotated by 90 degrees), c) x-axis projection (red), d) histogram of grey-level image; green arrows show order of image transformation: from histogram to binary image and finally to projections. The blue arrows indicate the directions of projections’ analysis

Full size image

Large changes in value must be detected on both functions. For this purpose, the differences in the function values are calculated:

$$\begin{aligned} \Delta y(i) = \mid y(i+2) - y(i)\mid , \end{aligned}$$

which are then compared with the thresholds determined based on the statistical dependencies of the function \(\Delta y(i)\). Figure 13b) and c) show the functions \(\Delta y(i)\) (black), the order of comparison with the threshold of both functions (blue arrows), the detection threshold (blue dashed line) and the values found (vertical blue lines).

For the position of the screw as shown in Fig. 13a), the search for high values of the differential function starts at the beginning of the coordinate system. After finding the first value exceeding the threshold, the next one is sought (see Fig. 13c). The difference of their positions determines the length of the screw (in [pix]).

The binary image projection on the y-axis is analysed in different ways. The analysis begins with the search for the point where the differential function \(\Delta y(i)\) exceeds the threshold (see Fig. 13b), but the second value, limiting the bolt diameter on the right, is searched in the opposite direction - from the end of the axis to the beginning. This is due to the frequent occurrence of high values of the function \(\Delta y(i)\) inside the bolt area.

7.3 Results

Table 1 compares the results of the measurements obtained in [5], where the points of interest were determined manually, with the measurements made automatically - using the presented here algorithm. As one can see, there are pixel differences, both in determining the diameters and in calculating the length of the screw. Despite these differences, the results summarized in the fourth column of Table 1 do not differ much from the measurements made with a caliper gauge (last column).

Table 1 Summary of measurements in pixels and calculated screw lengths
Full size table

A larger difference occurs for the image Img161small (the last row of Table 1). This stems from the deviation of the screw image from the horizontal plane (see Appendix).

8 Conclusions

The innovativeness of the solutions used in the INRED system, described in this article, consists in:

  • creating a workflow engine that supports the renovation worker step by step. For this purpose, multimedia techniques are used: instructions and voice messages as well as presentations of examples of subsequent stages of work in the form of photos and short films,

  • implementation of an editor that matches the general workflow algorithm to a specific renovation procedure by defining multimedia and a set of tools necessary to perform the renovation. A report template is also generated, describing - partially automatically - the work performed,

  • the use of several types of tools communicating with the workflow algorithm via Bluetooth (e.g., calipers, torque wrenches),

  • creating from scratch the Mobile Devices Communication Subsystem, whose task is to organize Bluetooth/WiFi communication, register tools in the system, and possibly correct erroneous settings.

  • 3D models of the serviced machines can be presented in the 3D space which, in turn, can improve the quality and reliability of work.

In addition, it was shown how the key stages of renovation can be controlled using digital image analysis methods. An example is the algorithm for determining the length of the valve screw after the renovation, which is described in the previous section. The obtained value, after comparison with the database, informs about the correct (complete) opening of the valve.