Testing driver warning systems for off-road industrial vehicles using a cyber-physical simulator

Abstract

ADAS (Advanced Driver Assistance Systems) are becoming increasingly popular in on-road vehicles due to their safety, productivity, and cost savings. In the same way, off-road vehicles can benefit from ADAS systems to improve the security of drivers and workers in industrial settings. In this work, we study, design, and develop a novel security system to be integrated into industrial vehicles. This system is built to provide one-way Human Computer Interaction, from the computer to the human, so providing, through the interaction with the ADAS system, feedback to drivers about their surroundings, such as nearby workers, and thus helping to avoid collisions and prevent incidents. The study evaluates the quality of different feedback mechanisms, with the goal of designing the ADAS that produces the best User eXperience (UX). These feedback mechanisms are generated by LEDs in different display formats and colors, as well as with haptic feedback. We created a hybrid testbed using a realistic ADAS and a forklift simulator, integrating the system into a physical structure that resembles an industrial vehicle (a forklift) and used a computer-based simulation of a warehouse to gather the information from users. We performed a study with 36 people for the evaluation of the different feedback mechanisms tested, evaluating the results both from an objective point of view based on the results of the simulation, and a subjective point of view through a questionnaire and the stress of the users in each test.

1 Introduction

Driving in industrial environments and operating large off-road vehicles are challenging tasks. Most of the vehicles are large and heavy, having moveable and articulated parts that might seriously damage other vehicles or the building structure. Even more serious, in the case that a worker is directly hit, there is a good chance that the harm caused will be severe and may even be fatal.

Additionally, the environments in which these machines are used may be low visibility environments, such as dimly lit indoor environments, having to operate at night, with adverse weather conditions, or with the presence of suspended dust. The low visibility can even be caused by the actual day-to-day tasks, such as having to handle the load. Furthermore, heavy machines that move swiftly are utilized in different situations, such as logistical activities within an industrial facility, where they move with the presence of employees on foot throughout the industrial facility. It can often be challenging to detect moving impediments because of ambient noise, poor illumination, or the design of the workplace.

All these situations call for a high level of skill and mental effort from the professionals involved. Accident reduction in industrial contexts is now essential in the context of the twenty-first-century factory [1]. For example, in the year 2022, 108 accidents involving a forklift were recorded in the United States of America [2]. Additionally, there are studies that indicate that a forklift is involved in one of every six serious work accidents in the United States and that pedestrians are also involved in 80% of accidents [3]. As a result, accidents caused by industrial vehicles while moving are one of the most significant dangers in all types of workplaces such as logistical centers. For these reasons, in the upcoming years, reducing or, if possible, eliminating accidents will be a first-level technology challenge in the connected industry. To address these challenges, we look at the advances in security in other related sectors, such as on-road vehicles.

The use of ADAS (Advanced Driver Assis tance Systems) is increasing and, in some cases, is even mandatory. For this reason, we explore the use of ADAS in the context of off-road vehicles to advance towards reducing accidents in industrial environments. An ADAS is a set of electronic technologies designed to assist drivers in operating their vehicles more safely and efficiently. These systems use sensors, cameras, and other technologies to monitor the vehicle’s surroundings and provide real-time information to the driver, helping reduce the risk of accidents and improve the overall driving experience. ADAS in industrial environments are not very popular even though they provide important benefits such as increased safety, productivity, and cost savings. ADAS can help reduce the risk of accidents and injuries by providing drivers with real-time information about their surroundings, such as obstacles or other hazards. Additionally, ADAS can help reduce the costs associated with accidents, equipment downtime, and other inefficiencies.

When ADAS are integrated into off-road vehicles—like forklifts—which are not used in the same ways as other industrial vehicles, like trucks, their use has to be reevaluated. They must be especially made for the job that is done in industrial settings, where there may be dust, noise, workers moving around, etc. Each off-road vehicle can have a specific set of movement patterns. For instance, a forklift is expected to move freely throughout a plant or warehouse, whilst an excavator will mostly be in one place and move in a specific work area. This requires the customization of the ADAS system as well as the feedback mechanisms designed for it.

More concretely, forklift driving is a specialized skill that necessitates certain traits for safe and efficient operation. Forklift drivers must be focused, alert, and aware of their surroundings, including people and obstacles.

Good hand-eye coordination is essential for maneuvering the forklift accurately, particularly when dealing with heavy loads. The job can be physically demanding, requiring stamina for long hours and repetitive tasks. Safety is of utmost importance, with drivers adhering to guidelines, wearing appropriate protective equipment, and minimizing risks. Patience and attention to detail are necessary, especially when navigating tight spaces or handling delicate loads.

One of the most important parts of a system such as the one proposed here, is the interaction with the user (i.e., driver), because, ultimately, it is the driver who is going to carry out the action conveyed by the feedback system, be it braking or changing direction. Hence, when designing an assistance system for this specific use case, we need to rely on feedback mechanisms that are not invasive and do not represent a heavy mental burden for the driver. Furthermore, we sought to provoke in the user, a very quick and almost instinctive response to an alert or warning. We need to verify that even for non-specialized users, the feedback mechanisms used offer consistent results.

To effectively interact with drivers of off-road vehicles, we need to consider the aforementioned factors to design the base system for the interaction with the users (i.e., drivers). The base system-user interaction model is unidirectional, in the sense that the user will not be able to give input but will receive the output information from the system to react accordingly. The system-user interaction targets the evaluation of visual and haptic feedback, seeking the best reaction time.

We now present the basic user-interaction system, which will be detailed in Sect. 3. The testbed provides two basic user-interaction systems that are controlled by the ADAS, which are a set of LEDs, configured as two vertical strips at the sides of the cabin, and one LED matrix on the top of the cabin. Additionally, the driver will have haptic feedback on the belt. The ADAS will send the feedback to the user, based on the input from the computer-based simulation of the forklift moving in a virtual warehouse, warning, or alerting when an obstacle is nearby.

This work goes beyond the state of the art, evaluating different sensory feedback mechanisms to be used to interact with the driver of industrial vehicles, requiring little mental processing from the driver, whilst facilitating a speedy response in case of warning or alert. This work builds upon related work, which demonstrates that red LED strips in a vertical disposition are a satisfactory option to offer feedback to the driver [4] and the fact that the peripheral view is subject to a speedier response as these feedback mechanisms are processed by the reptilian brain, rather than in the pre-frontal cortex [5].

We use this base system, to design, build, and test a new set of feedback mechanisms. This paper corroborates the findings of [4], and at the same time explores the feasibility of additional feedback mechanisms. On the one hand, we add another color with a specific meaning to the system, amber. The colors red and amber are deeply embedded into our subconscious, as we are accustomed to seeing them in traffic lights and other contexts with the same meaning: danger and warning, respectively. Furthermore, we use in addition to the LED strip in vertical disposition, a LED matrix to convey additional meaning through symbols such as directional arrows to indicate the relative position of an obstacle to the driver. Finally, we add a different type of feedback using a haptic safety belt, to convey danger and prompt the user to stop the vehicle to reinforce the red LED light message.

The base interaction system is integrated into a physical structure, to evaluate the behavior of drivers in a virtual environment that simulates a warehouse, where they must respond as quickly as possible to the different feedback mechanisms established in the cyber-physical structure. To evaluate the new base interaction system, we have built a simulator that locates the user in a virtual warehouse or plant, where the user can drive following a simple set of instructions, that will seek to gather the reaction times of each user when any of the alerts from the feedback mechanisms are emitted.

The evaluation environment comprises a cyber-physical structure where the feedback mechanisms are located, and the warehouse simulation, which has visible and non-visible obstacles. Approaching non-visual obstacles will generate an alert that will be received by the ADAS and send the proper signals to activate the feedback mechanisms. We gather objective information from the simulation that provides the real reaction time from the user, and we also provide subjective information about the mental load using the NASA TLX form. We complete the evaluation, with the estimation of the stress of the user, by using a heart rate sensor of the Polar band to measure how the user reacts to each of the tests. A heterogeneous population was selected for the evaluation, with more than 30 [6] individuals of different ages, gender, and backgrounds. We aim to confirm the results of [4], related to the effectiveness of the feedback mechanisms on peripheral vision and understand if they can be improved with additional mechanisms.

The remainder of this article is as follows: Section 2 reviews the related work that provides the context and background for this work. Section 3 presents the evaluation system for the proosal of the different feedback mechanisms. The results of the evaluation are presented in Sect. 4. Section 5 provides the conclusions and future work.

2 Related work

Many accidents in the industrial environment can be caused by driver errors, a lack of awareness about workplace safety, an inadequately designed work environment, or a combination of these factors. Driver errors can occur owing to a lack of adequate training, stress, fatigue, or due to the monotonous task to be performed [7]. Manufacturers of industrial off-road vehicles offer driving assistance systems such as automatic parking brakes, cruise control speed, and an automated braking system when the vehicle takes a curve or passes through an area where there is a possibility of pedestrians or other vehicles. In this section, we review various approaches and technologies employed in industrial environments to enhance safety, particularly focusing on Advanced Driver Assistance Systems (ADAS) and their interaction with operators. These systems aim to mitigate accidents caused by driver errors, lack of awareness, or inadequately designed work environments. Additionally, we explore methods of communication and interface design aimed at optimizing safety measures.

2.1 Positioning technologies

Positioning technologies play a crucial role in industrial safety, facilitating the tracking and identification of vehicles, cargo, and personnel. Traditional systems like radar, ultrasonic, and optical solutions have been extensively studied. For instance, Cao et al. [7] discuss various methods and assistance systems for forklift drivers, including radar and ultrasonic solutions for collision avoidance. Kelemen et al. [8] highlight advancements in ultrasound technologies, addressing historical performance issues. Ultrasound-based indoor positioning systems have evolved significantly, with solutions such as asynchronous Ultrasound Trilateration [9], high-accuracy ultrasound systems [10], and time-of-flight based systems [11]. These advancements aim to provide reliable location information without relying heavily on infrastructure, reducing costs and maintenance efforts.

Similarly, technologies like Ultra-Wide Band (UWB) [12] and Bluetooth [13] offer promising alternatives for indoor positioning, even in ad-hoc systems not requiring fixed infrastructure for location [14]. While these technologies may require infrastructure, efforts are being made to develop infrastructure-less solutions that provide relative positioning information.

Having an overview of the different technologies used in location helps us understand the possibilities in terms of communicating information to the driver to increase awareness about the surrounding areas. That is the base of this work, to efficiently communicate to the driver what is happening and do it in such a way that the response is quick and requires little mental effort.

2.2 Advanced driver assistance systems (ADAS)-user interaction

A key area of research and innovation, which is the focus of this work, when reducing accidents is that related to communication and interfacing with drivers. In this sense, we need to consider how an ADAS notifies the driver about a dangerous situation. Even though there are solutions to improve security, their use is still limited due to their high cost. In addition, there are no studies in the literature that examine which methods are the best for interacting with the driver from the standpoint of cognition and the effectiveness of the warnings. What kind of warning causes the shortest reaction time? Which method is the least intrusive? and so forth. Manufacturers integrate various Advanced Driver Assistance Systems (ADAS) into industrial off-road vehicles to enhance safety. These systems include automatic parking brakes, cruise control speed, and automated braking systems [7]. Additionally, positioning systems interface with portable devices to reduce security risks and improve workplace efficiency.

Camera-based systems are also prevalent, aiding in pedestrian detection and collision avoidance [15]. Some systems utilize light pulses to illuminate potential hazard zones around vehicles [16]. These advancements underscore the importance of integrating sensor technologies to enhance situational awareness and mitigate risks.

Efficient communication and interaction between ADAS and drivers are essential for effective accident prevention. However, the effectiveness of warning signals and interface design remains a topic of research. Several studies evaluate different feedback mechanisms, such as LED displays [17], image overlays [18], and peripheral vision cues [4].

Finally, prevention and proactive measures are critical in preventing accidents. To accomplish this, assistance systems must consider the context, workload, environment, and physical and mental state of the industrial vehicle’s driver. The research by Stein, et al. [4], assesses which types of lights would be most useful in alerting the driver of a potential collision. The study includes reduced size lights in the center and lateral zones of the windscreen, as well as different-sized sidebands. The results show that lateral bands are more effective in capturing the driver’s attention. Therefore, we go further than this related work to develop a more comprehensive study.

This work goes beyond the state of the art by focusing on the study of the user experience, and the suitability of a set of ADAS-based feedback mechanisms, such as different color LEDs in two configurations (matrix and strips), two colors (red and amber) and haptic feedback in the seatbelt. Furthermore, we measure the effectiveness of the mechanisms, reaction time, and stress, both objective and subjective, using the data measured by the simulation and the impression of the user, based on questionnaires.

3 Materials and methods

In this section, we present the proposal for this article to evaluate the industrial ADAS, based on different feedback mechanisms. First, we present the testbed that we use to evaluate the ADAS. Then we introduce the experimental design to evaluate the ADAS-based feedback mechanisms. Finally, we present the results of the evaluation and discuss the implications of the different feedback mechanisms and their expected impact on the drivers.

3.1 Description of the testbed

For the evaluation of the proposed ADAS, we used a hybrid system composed of a physical structure equipped with different feedback mechanisms and a computer-based simulator of a warehouse.

3.1.1 ADAS-enhanced physical structure and feedback mechanisms

The test environment has a cabin-shaped physical structure, with dimensions like those of a forklift. To simulate handling, there are industrial vehicle controls composed of a steering wheel with a rotary knob and two pedals to perform forward and backward movements. In the front part of the physical structure, there is a 65-inch television for the projection of the simulator. The physical structure also allows the placement of the different elements of interaction with the driver who will use the ADAS to be evaluated, such as screens, LED strips, LED matrices, speakers, belts with haptic feedback etc. Finally, the system has a power supply to provide the necessary power to all items.

In this work, we devise the following feedback mechanisms: 1) LED strips, 2) LED matrix, and 3) Haptic feedback. The disposition of the feedback mechanisms is shown in Fig. 1.

The LED strips are set up following the disposition presented in [4]. In this sense, they have been collated vertically, on both sides of the driver’s cab. The use of the LED strips in this configuration follows two objectives, to corroborate the findings of the related work and to act as a baseline in this work. The addition in this work, going further than the related work, is to implement two-color feedback with distinct meanings. Red indicates imminent danger, prompting the driver to stop as soon as possible, while amber indicates the driver should reduce speed and increase awareness. This feedback is the simplest and does not imply any more information, not requiring the driver to make any complex interpretation of the meaning beyond the color indication.

The LED matrix goes a step further. It is located on top of the driver’s cab and may be subject to low visibility depending on the driver’s height. The information conveyed in the LED matrix aims to be richer in meaning as an arrow will be shown, indicating where the obstacle to be avoided can be found using four directions: front, rear, left, and right. In addition, the arrow will be displayed using the same color code as that used in the LED strips: red for immediate danger, and amber to request increased caution.

Fig. 1

Forklift structure with feedback mechanisms: a Physical infrastructure, b Logical representation

For the haptic feedback, we use a seatbelt with the necessary mechanisms to vibrate, following the danger indication. That is, the haptic feedback will turn on when the red light from the LEDs is emitted. The objective of the haptic feedback is to reinforce the red color when the visibility might not enable a quick response.

3.1.2 Computer-based warehouse simulation

To simulate driving in a factory, we developed a simulation using the Unity game engine. The game consists of driving an industrial vehicle, in this case a forklift, through a warehouse or factory.The dynamics are simple, since it is not a matter of evaluating the driver’s handling capabilities, but of determining if the ADAS to be evaluated is effective or not. For this reason, the objective of the game is to focus the user’s attention on driving, as a routine activity that does not require high cognitive abilities. The virtual factory was designed with a rectangular plan in which different construction modules are placed, thus constructing a complex structure of corridors. The modules are opaque, so the driver cannot see around a corner when turning. Figure 2 shows a top view of the factory with the construction modules.

Additionally, many obstacles (boxes, barrels, etc.) have been distributed in the corridors of the factory. The objective of having this visible obstacle is to preclude the driver from moving without having to pay attention. The construction modules and obstacles also force the driver to make changes in direction. A key element of the simulator is the hidden obstacles. These are not visible in any way to the driver. Their purpose is to trigger events in the ADAS being evaluated. These obstacles do not have any physical representation. The reason for this design is that if they were visible or could be interacted with in the simulation (e.g., collided with), it would not be possible to determine what triggered the driver’s reaction, seeing the object or the information from the ADAS. In Fig. 2 we can see the hidden obstacles deployed throughout the factory.

When the vehicle enters the configurable warning zone of an obstacle, the simulator sends a message to the ADAS. This information can be customized to provide the data each ADAS needs to represent the information to the driver. It can include details generated by the simulator such as the relative location of an obstacle in relation to the vehicle (in front, rear, left, or right). The forklift is operated using the vehicle controls that have already been described. The simulated vehicle has been designed to provide a driving experience as close to a real forklift as possible. In the design, easy handling has been sought, and special skills are not required. This allows evaluations with non-specialized drivers.

Fig. 2

Warehouse structure and obstacle position

To provide objective information about the performance of each user, the warehouse simulator generates a log line in a log file every time a hidden obstacle event occurs. These log lines allow us to study the response times of the driver and the movement of the vehicle. In addition, they allow us to analyze the response depending on the type of signal emitted and the obstacles, located in different positions in the factory and with a possible influence on response times.

The simulator also provides configurable background noise, allowing us to create an immersive experience. To increase realism, this noise varies in different areas of the factory. In addition, the vehicle’s engine noise changes according to its speed.

The event log lines include the following fields: the position of the vehicle within the factory at the time of the event, a timestamp at the instant the event was sent, the vehicle position at the instant the driver stops accelerating, and a timestamp of the instant the driver stops accelerating, an indicator that the driver has not braked before 5 s, the hidden obstacle that activated the ADAS alert and type of warning generated by the simulator: obstacle zone (in front, behind, left, right) and perimeter (amber, red).

A driving simulation scene is shown in Fig. 3.

Fig. 3

Snapshot showing the simulation with obstacles

3.2 Experimental design

Before the user starts driving, a general questionnaire is filled in to ascertain basic information allowing us to know his or her profession, age, level of studies, handedness, etc. Details of the basic information form are shown in Appendix A. Then, we ask them to use the heart band monitor Polar [6], to measure the stress levels experienced by the user during each task and afterward we explain how the tests work. For the main part of the evaluation, we have devised a set of three tests with the following generic characteristics.

In each test, the user must perform a task consisting of driving in the simulated factory for 5 min. To carry out every task, the user must reach the exit as many times as possible in the 5 min each test lasts. When the user is aware of any feedback mechanism from the ADAS, the forklift should reach a complete stop. For each test, before starting the next one, we asked the user to fill out the NASA-TLX form, to evaluate the subjective experience of the user with the performed task. The general procedure of the experiment is shown in Fig. 4. Next, we highlight the differences of each test which are labelled ADAS modes, identifying the set of feedback mechanisms used in each test.

Fig. 4

General procedure of the experiment

3.2.1 ADAS mode 1: LED strips only

The first test involves only the vertical LED strips representing two events, warning (amber) and danger (red). The response expected from both events is the same, to stop the forklift as soon as possible.

3.2.2 ADAS mode 2: LED strips and matrix

The second test adds to the first one, the LED matrix with the added meaning of positional arrows, indicating the location of an obstacle, relative to the user’s driving direction. The colors are maintained with the same meaning as the LED strips. The location of the LED matrix is above the driver’s cabin, and we understand that visualizing the symbols would depend on the height and peripheral vision of the user.

3.2.3 ADAS mode 3: complete LED and haptic feedback

The last test adds haptic feedback through the seatbelt. In this sense, we reinforce the danger signal conveyed by the red LEDs with the vibration of the seatbelt. In this sense, we expect this element to act as a fallback mechanism in case the light is not perceived for any reason (external light causing momentary blindness to the driver, or similar).

3.2.4 Objective information: event log and heart rate monitor

To convey if the feedback mechanisms are effective or not, we base our evaluation on objective and subjective information. As described previously, the objective information is all the data obtained from the simulation that tells us how long it took the user to react to an event, causing the vehicle to stop. Also, we measure the heart rate and evaluate the stress of the user in each test.

HRV (heart rate variability) is a non-invasive technique for measuring the variation of heartbeat intervals. It is a biomarker that records the small changes between each pulse, known as “RR intervals”, and has a close relationship with the neurological system, which regulates many biological activities [6]. Because low parasympathetic activity is frequently related to changes in HRV variables, HRV can be used as a biomarker of stress, and those with high HRV may be more resilient to stress. It should be mentioned that there is an inverse relationship between HRV and HR (heart rate), which means that a high HRV (RR interval) is associated with a low HR and vice versa. To measure the HRV accurately, a compatible heart rate monitor that can sense R-R intervals is required, as most wristbands and watches do not have the required accuracy. For that reason, we use the Polar heartbeat band [6].

3.2.5 Subjective information: NASA-TLS form

To get a sense of whether each feedback mechanism is effective or not, we asked the users to fill out a NASA-TLX form after each test, to understand how they felt in terms of workload. The NASA Task Load Index (NASA-TLX) is a widely used assessment tool that rates the perceived workload to evaluate tasks, systems, or teams’ effectiveness [19]. It consists of six subjective subscales to rate mental demand, physical demand, temporal demand, performance, effort, and frustration level. The tool also includes pairwise comparisons to create an individual weighting of the subscales to obtain an overall task load index. It has been used in various domains, including aviation, healthcare, and other sociotechnical domains. It has been cited in over 4000 studies, highlighting its influence on human factor research.

3.3 Description of the population

The experiment was performed with 36 users, aged 18 to 59. Table 1 shows a pyramid representation of the distribution of the population by gender and age groups, based on the Sturges rule [20].

Table 1 Ages of the Participants based on the Sturges Rule

The population of the experiments is composed of 27.78% women and 72.22% men. Even though recruiting participants has inherent difficulties, as can be appreciated, all age groups have at least one participant.

4 Results

To extract the conclusions of the experiment, we performed several statistical analyses. First, we carried out an individual study of both the objective and the subjective information of the different users. With this, we undertake comparative studies of the different warning systems to draw conclusions. The goal is to determine if there are statistically significant differences between two or more groups of variables involved in the warning system. We used the most common tests for this type of analysis, according to Crawley [21].

These tests depended on the assumptions of normality and homoscedasticity of the data and the number of subjects in each comparison group. First, we checked normality with KolmogorovSmirnov or Shapiro-Wilk tests, depending on the size of the groups and homoscedasticity with the Levene test [22]. When data met both normality and homoscedasticity, we used one-way ANOVA tests to compare the data. If differences existed, we used Tukey or Duncan tests with a confidence coefficient of 95% to perform pairwise comparisons. If normality or homoscedasticity failed, we used the Jonckheere–Terpstra test [23, 24] or the Kruskal-Wallis test. To compare groups of small and different sizes, we also used the Mann-Whitney U test [25].

4.1 Analysis of the objective information

An individual study of each of the participants has been carried out and their response in each one of the tests performed. Figure 5 indicates the format of the traces obtained by the monitoring system. In each scenario (ADAS Mode), five minutes of driving are carried out and several stages (the user must reach the exit) must be overcome, which correspond to work to be carried out in the factory.

Fig. 5

Traces obtained from driving simulation in the warehouse

During the tests, between 20–24 hidden obstacles are monitored in each scenario, depending on the user’s driving ability. In this case, the trace corresponding to the ADAS Mode 3 scenario, stage 1, is shown. The warehouse simulator generates a log line every time a hidden obstacle event occurs. Every time the industrial vehicle approaches a hidden obstacle, the warning signals are activated (led, matrix and haptic in the ADAS Mode 3 scenario) and the driver must stop the vehicle.

With all the available information, the individual response of each of the users has been analyzed. Figure 6 shows the reaction time to obstacles of Driver#3 in the three tests (scenarios). The reaction time represents the time from when the driver receives the ADAS warning about the presence of a hidden obstacle until he stops the vehicle. The aim of the boxplot in Fig. 6a is to determine if any of the warning methods (ADAS Mode) provide a shorter reaction time, and with Fig. 6b the objective is to ascertain if the direction in which the obstacle appears has an impact on the driver’s reaction.

Fig. 6

Reaction time depending on a scenario and b direction of the obstacle

The distance travelled until the driver brakes (Fig. 7a), the speed of the vehicle (Fig. 7b), the duration of the stages in each scenario (Fig. 7c) and the position of hidden obstacles in the warehouse (Fig. 7d) have also been monitored. Speed, reaction time and the distance until braking are obviously related, such that if the user drives at a high speed, the distance covered will be greater, increasing the probability of colliding with the obstacle before the vehicle stops. Figure 7c allows us to know how many stages the driver completed and their duration during the 5-minute test in the analyzed scenario. The more stages completed, the greater the driving skill. Figure 7d shows which part of the warehouse the driver traveled through, and the position of the hidden obstacles encountered.

Fig. 7

Monitoring driver #3 on ADAS Mode 3 scenario

Among the many variables that were monitored during the tests are those related to the driver’s heart activity. Thus, the heart rate has been monitored in the different scenarios, as shown in Fig. 8a. The boxplot in Fig. 8b shows the median and percentiles of the recorded heart rate of Driver#3 for each of the scenarios. ADAS Mode = 0 represents the time when there is no driving, either before star ting the simulation or during the time intervals between scenarios.

Fig. 8

Heart Rate of driver #3: a HR time evolution; b Boxplot in the scenarios

Likewise, in addition to HR, we have analyzed other variables related to cardiac activity to detect mental stress in the driver [26, 27]. In this study, the following variables have been considered to measure stress [28]:

• Heart rate variability (HRV) or R-R intervals, measured by the variation in the beat-to-beat interval.

• SDRR, the standard deviation of RR intervals.

• RMSSD, the square root of the mean of the squares of the successive differences between adjacent NNs.

• pNN50, the proportion of NN50 divided by the total number of NNs, where NN50 is the number of pairs of successive NNs that differ by more than 50 ms.

• SD1/SD2, the ratio of SD1/SD2, measures the unpredictability of the RR time series.

As an example, Fig. 9 illustrates Driver#3 in the ADAS Mode 3 scenario, the comparisons of biometric variables (R-R interval) with reaction time and with the direction of the hidden obstacles.

Fig. 9

Driver #3 a Reaction time and RR-interval b Direction of the obstacle and RR-interval

4.2 Analysis of the subjective information: NASA-TLX form

Once each experiment is finished, users filled out a NASA-TLX form to evaluate the workload of the task. It consists of six subjective subscales to rate mental demand, physical demand, temporal demand, performance, effort, and frustration level. Table 2 shows, on a scale of 0 to 100, the average and standard deviation of the workload scores for each of the scenarios, both for the overall workload index and for each of the dimensions classified by NASA-TLX. To obtain the global index, a weighted score has been applied following the indications of the NASA-TLX method.

Table 2 Average scores and standard deviation in each of the dimensions of the NASA-TLX and global index for each task

The results in Table 2 show that the workload of each task is low. On a scale from 0 to 100, the global index does not exceed 53 points in the evaluated scenarios. These results demonstrate that the warning systems proposed and evaluated in each task require little physical and mental processing by the driver. Differentiating by dimensions, although the workload is not excessively high in any of them, it should be noted that the highest scores were obtained in mental demand and performance, while the lowest scores were in physical demand and frustration. Only the mental demand exceeds 50 points in each task performed. Although it is not excessively worrying, this result should be taken into account to improve future warning systems. Mental demand represents the mental and perceptual activity necessary to respond to the alert of an obstacle, which implies that thinking and deciding how to act in this situation is more important for drivers than other dimensions, such as physical demand. We have also performed a comparison between the subjective measures (NASA-TLX test) of the three ADAS Modes. We have used verification of normality assumptions through the Kolmogorov-Smirnov test for each mode (ADAS Mode 1: p-value = 0.186>α = 0.05; ADAS Mode 2: p-value = 0.200> α = 0.05; ADAS Mode 3: p-value = 0.200> α = 0.05) and homogeneity of variances through the Levene test ( p-value = 0.633 >α = 0.05). These results are applied to ANOVA to compare the subjective measures obtained after performing the tasks. This test has a p value = 0.843>α = 0.05, which indicates that there is no statistically significant difference between the subjective measures in the 3 tasks.

4.3 Comparative analysis of the reaction time and stress

Once the data of all the users has been analyzed at an individual level, a comparative study will be carried out to answer various questions, and extract conclusions. The monitored information of each driver has been processed, and we have obtained, for each scenario, the average values of the NASA-TLX tests, the percentage of obstacles in which the vehicle has stopped, the average reaction time, the average heart rate and R-R interval, as well as the stress indicators pointed out in the previous section (SDRR, RMSSD, pNN50, SD1/SD2). The driver traces are available in the format indicated in Table 3, which include subjective information from the questionnaire and the objective information from the monitoring system in the three scenarios.

Table 3 Traces with subjective and objective information of all drivers

The first objective of this work is to verify the validity of our proposal, that is, if our basic ADAS system (ADAS Mode 1: LED strips) allows detecting hidden obstacles. The drivers under test confronted a total of 2,500 obstacles, stopping in 97.94% of the cases, which demonstrates the validity of the implemented system. It must be considered that these are hidden obstacles, and that the driver can only detect their presence by the indications of the warning systems (LED strips in this case). The next objective is to determine if the system improves by incorporating new warning methods. The three implemented systems have been compared with the results indicated in Table 4.

Table 4 Percentage of hidden obstacles detected by the warning systems and reaction time

It can be seen how the application of new warning methods improves the percentage of detected obstacles. The largest number of obstacles detected is produced with the haptic feedback system, which verifies the off-the-record impression of the majority of the drivers who did the test. Once the validity of the warning systems has been verified, we proceed with the analysis of the reaction times. Table 4 shows how the response time is lower in the haptic system. However, we have carried out a more detailed statistical analysis in order to determine if there are significant differences in the response times of the three implemented warning methods. As there are more than 30 records per task, normal results are assumed. Therefore, ANOVA can be used to verify whether there are statistically significant differences between the performed tasks. To apply the ANOVA, we will only test the homogeneity of variances using Levene’s test. For all analyses, a significance level of α = 0.05 is assumed. Based on Levene’s test, sig > 0.05, therefore, the hypothesis of homogeneity of variances is not rejected. Through the ANOVA table, it is verified that for each task sig>0.05, which indicates that there is no statistically significant difference between the results obtained during the performance of the tasks. This indicates that, although a simplistic analysis as in Table 4 seems to indicate the opposite, average reaction time to obstacles does not change from task to task. Likewise, we have carried out the same tests to compare the cardiac activity and the increase in the level of stress between the different tasks (ADAS Mode). The results of the ANOVA test indicate that the variables Heart Rate, RR-interval, SDRR, RMSSD, pNN50, SD1/SD2 do not change from task to task. In all cases sig>0.05. It can therefore be concluded that stress does not increase as more tasks are performed.

4.3.1 Effect of age on the reaction time

When comparing the results by age group, we now have small samples, which means that parametric methods for comparing these samples are not feasible. As we have K independent samples to compare, the Jonckheere-Terpstra test will be used. The Jonckheere-Terpstra test is a rank-based, nonparametric test that can be used to determine whether there is a statistically significant trend between an ordinal independent variable and a continuous or ordinal dependent variable. It tests the null hypothesis that the distribution of the response variable does not differ between classes. It is also known as the Jonckheere-Terpstra test for ordered alternatives. This test is similar to the Kruskal–Wallis H test. However, the KruskalWallis H test does not predict how the differences in the orders of the dependent variable will depend on the ordinal nature of the groups of the independent variable, but the Jonckheere-Terpstra test allows us to do so based on the chosen test. Based on these results (Table 5), we can conclude that there are no statistically significant differences when comparing the reaction times by age group in each performed task.

Table 5 Summary of the hypothesis test for age group

4.3.2 Effect of gender on the reaction time

When the groups are divided by gender, there are less than 30 individuals. Then, the most appropriate test is the Mann-Whiney U test, which is the non-parametric test corresponding to the t test for comparing independent samples. This test makes it possible to assess the effect of the participant’s gender on the result obtained for the measures obtained. The U test, for a significance level of 5%, showed that the gender of the participant does have an influence.

• Reaction time (sec) – Scenario ADAS Mode 1 (U = 39; pvalue = 0.02 < 0.05)

• Reaction time (sec) – Scenario ADAS Mode 2 (U = 18; pvalue = 0 < 0.05)

• Reaction time (sec) – Scenario ADAS Mode 3 (U = 37.5; p-value = 0.002 < 0.05)

Table 6 shows the results obtained by applying the Mann-Whitney U test to the ADAS Mode 1 scenario (LED strips). Similar results are obtained with the rest of the scenarios ( p-value<0.05), which indicates that the driver’s gender influences the reaction time to hidden obstacles.

Table 6 Mann–Whitney U Test Summary Independent Samples (ADAS Mode 1)

Figure 10 illustrates the reaction time according to the gender of the participants in the ADAS Mode 1 scenario.

Fig. 10

Participant gender ADAS Mode 1. Mann–Whitney U test independent samples

As it can be seen, the reaction time of men is shorter than that of women. Similar results are obtained in the remaining scenarios.

4.3.3 Effect of hand orientation or level of studies on the reaction time

When comparing the results by level of studies, we have small samples and therefore, parametric methods for comparing these samples are not feasible. As we have K independent samples to compare, the Jonckheere-Terpstra test will be used. Based on this test, we can conclude that there are no statistically significant differences when comparing the reaction times by level of studies in each performed task. The same conclusions are obtained when analyzing the stress, there being no significant differences in the stress variables by level of studies. Figure 11 graphically shows the test results for reaction time and R-R interval in ADAS Mode 1. Applying the Mann–Whitney U test to verify the effect of the dominant hand of the driver, it is also obtained that there are no significant differences in reaction times or in stress variables.

Fig. 11

ADAS Mode 1. a Reaction time vs level of studies. b R-R interval vs level of studies

4.3.4 Relationship between subjective estimation NASA-TLX and objective measures

It is also interesting to verify the correlation between the subjective perception of the physical and mental load of each task (NASA-TLX test) and the objective responses during the experiment. Thus, we have calculated the correlation coefficients in each task, with the results indicated in Table 7. These results indicate a non-significant correlation at a significance level of 0.01, as can also be visually appreciated in Fig. 12.

Table 7 Correlation NASA-TLX vs. reaction time, HR and RR interval

Fig. 12

Relationship between NASA-TLX test and a reaction time, b heart rate

5 Conclusion and future work

Our paper advances on system-user interaction in the field of ADAS in off-road industrial vehicles with an important set of contributions. We have developed and evaluated warning systems for the drivers, that are integrated into off-road vehicles with the goal of improving security and then reducing accidents in industrial environments.

These contributions are especially important if we consider that there is little research on driving assistance systems for industrial vehicles. To carry out the evaluation, we have built a hybrid test environment, real and simulated, consisting of a forklift structure equipped with warning systems to the driver and acceleration and brake pedals to respond to the generated alerts. Likewise, the structure integrates a simulator that allows the user to carry out a task in a virtual warehouse with visible and hidden obstacles that generate alarms and activate warning systems that drivers must react to.

We have proposed three warning methods that require little physical and mental processing from the driver and, at the same time, provide a quick response in case of danger. The proposed warning mechanisms are amber and red LED strips, a LED matrix and a haptic safety belt that allows us to perform an incremental testing.

Firstly, our results show that the performed tasks in the three scenarios do not involve a high physical and mental load, as shown by the results of the NASA-TLX test, carried out on users at the end of each task. Also, a comparative analysis indicates that there is no statistically significant difference between the subjective measures in the 3 tasks. Based on this, we can conclude that the proposed alert mechanisms do not imply an excessive workload for drivers, which is one of the conditions that these assistance systems must meet.

Secondly, our studies have shown that with the basic mechanism of two-color LED strips, 97.95% of obstacles are detected, and the vehicle is stopped on time. The results improve with the incorporation of the LED matrix and, if the haptic system is included, 99.76% of hidden obstacles can be avoided, which represents a substantial improvement over the basic LED strip system. Regarding reaction times, although the results seem to indicate that with the haptic system the driver reacts more quickly to warnings, a more detailed statistical analysis demonstrates that average reaction time to obstacles does not change from task to task. The same statistical tests have been carried out to determine the level of stress between the different tasks. The results allow us to conclude that stress does not increase as more tasks are performed.

Another interesting conclusion is that subjec-tive impressions (NASA-TLX test) do not always coincide with empirical results (reaction time, cardiac activity), which indicates that in addition to considering, obviously, the real performance of the system, the subjective information must be considered to improve the assistance system.

Although performance is not affected by certain user characteristics such as age, hand orientation or level of studies, the driver’s gender has an influence on the reaction time.

The results of the study allow us to conclude that the suggested alert systems meet the requirements initially proposed, allowing the detection of hidden obstacles with short reaction times, in addition to not assuming an excessive physical and mental burden for drivers. With the haptic system, combined with the LED strips and matrices, the best results are obtained in terms of the number of obstacles detected without increasing the workload or stress. Therefore, this alert mechanism is positioned as the most appropriate, as indicated by the informal comments of the drivers at the end of the tests.

Future work will be undertaken to perform multi-day tests with the basic system to analyze the learning curve and study if failures and response times improve over time. We also intend to test with new alert methods, such as progressive lights, some flashes for extreme situations, etc. Another improvement that we are working on is to carry out evaluations with driving in reverse, driving with a load, to analyze if the utility is maintained. Expanding the functionality of the ADAS to, for example, warn if a pallet can be loaded, height warnings when passing through a door, etc., are other aspects that can be worked on in the future.

Appendix A

General Information Form

Here we show the generic information we asked the users in order to identify and classify the population.

TEST PROTOTYPE FORKLIFT FEEDBACK

PARTICIPANT CHARACTERISTICS

AGE (YEARS): _____

GENDER

- MALE

- FEMALE

- PREFER NOT TO SAY

- OTHER: _____

INDICATE YOUR LEVEL OF STUDIES

- No Studies

- School

- Compulsory Secondary Education

- High School Graduate

- Bachelor’s degree

- Master’s degree

- Doctorate

Do you have a driving license: YES/NO

Are you a professional driver: YES/NO

Have you ever driven a forklift? YES/NO

Indicate your handedness: _____

Are you color-blind? YES/NO

Do you have experience with:

Video games? YES/NO

Driving video games? YES/NO

Days a week do you use the car? _____

Do you get distracted easily while driving? YES/NO

Do you often use a driving assistance system? YES/NO

Do driving assistance systems seem useful to you? YES/NO

How many hours have you worked today? _____

How many hours do you sleep as a mean? _____

Indicate your tiredness level before the tests: 1 (not tired) ... 5 (very tired)