1 Introduction

Today, very many people use air travel as a means of transport. The number is increasing every year, as well as the number of long haul flights. At the same time, average flight duration has increased because more fuel-efficient aircraft have made intermediate landings unnecessary. The air travel market is highly competitive. In order to attract the passengers, airlines try every possible way to cut down the cost, or to provide a better service, or both. Short haul operators such as Ryanair and easyJet tend to lower the cost by providing minimum service (O’Connell and Williams 2005), while airlines such as Singapore Airline and KLM try to operate their long haul airlines with better service within an endurable cost (Sultan and Simpson 2000) .

One of the ways to cut down the cost is to maximize the number of seats (Quigley et al. 2001). This very often results in a limited amount of individual seating space in economy class (Hinninghofen and Enck 2006). In this context, the EU project “SEAT” (Smart tEchnologies for stress free Air Travel) was set up (SEAT Project Consortium 2006). This project, sponsored by the European commission, focuses on improvements and solutions to increase comfort in long haul air travel. The partners in this project are Imperial College London, Acusttel, Aitex, Antecuir, Czech Technical University, Wearable Computing Lab ETH Zürich, Inescop, Queen Mary University of London, Starlab, Eindhoven University of Technology, Thales, and DHS. The work described here is part of the SEAT project, and was executed in Eindhoven University of Technology (TU/e). TU/e’s focus in SEAT is on the in-flight entertainment, especially on the adaptive recommendation systems (Liu et al. 2008, 2009a, 2009b, 2009c, 2010; van de Westelaken et al. 2008).

The combination of long flight duration, limited space, and an unusual environment in the cabin (in terms of air pressure, humidity, and continuous noise) causes physical and psychological discomfort for a large group of passengers as well as the crew (Hickman and Mehrer 2001). Most of the airlines offer in-flight entertainment to their passengers. This provides mental distraction, and can possibly lead to the reduction of psychological stress. To reduce physical stress, most of the airlines recommend in-flight exercises to passengers (Hinninghofen and Enck 2006; O’Donovan et al. 2006). The way these exercises are presented varies from printed paper to complete instructional videos, 3D animations, and crew demonstrations. The main goal of these exercises is to stimulate the blood flow and to prevent health-related issues such as deep vein thrombosis (DVT), stiffness, and fatigue. One of the problems with these recommendations is that it is up to the passenger whether the exercises are carried out. Unless the passenger pays enough attention to the danger of DVT and is motivated to do these exercises, these recommendations tend to be ignored. In addition to recommended exercises, some of the passengers are advised to wear compression stockings to prevent the formation of DVT (Ball 2003). Usually this precaution is taken only by those with an increased risk of DVT. Physical stress can cause a number of health-related problems, as shown by different institutes such as the World Health Organisation (2010) and the British Medical Association (Dowdall and Evans 2004). If the recommended exercises are carried out, physical stress can be reduced. However, the motivation to do these exercises in a confined space during a long-haul flight seems to be a problem. Therefore, this work focuses on the motivation problem. The solution discussed here is an airplane seat equipped with sensors to detect the body movements and gestures of the passenger. The detected gestures are then used as input for interactive applications in in-flight entertainment systems.

The idea of controlling interactive content by means of body movement and gesture recognition is not new. On the contrary it is very popular at the moment, especially in movement-controlled games supported by gaming platforms such as the Nintendo Wii and Sony Playstation2. Some of these games are even designed for fitness purposes. One example is the Wii Fit. These games prevent the player from sitting in a fixed position while moving only their fingers to push the controller buttons. The fun factor of the games motivates the players to move their bodies in a particular way. However, most of these games are designed for home use, which assumes that the player has enough space to move in. This assumption does not hold for the seating space in an economy class aircraft cabin. Here the amount of space is very limited, and active movements in such a space can be annoying for other passengers, which means that new solutions have to be explored.

2 Problem and context

Prolonged immobility from sitting in a seat during long-haul flights can lead to the pooling of blood in the legs. It is known that immobility causes the formation of blood clots in the body and especially in deep veins (Deep Vein Thrombosis, or DVT in short). “Larger clots may cause symptoms such as swelling of the leg, tenderness, soreness and pain. Occasionally a piece of the clot may break off and travel with the bloodstream to become lodged in the lungs. This is known as pulmonary embolism and may cause chest pain, shortness of breath, and, in severe cases, sudden death. This can occur many hours or even days after the formation of the clot.” (World Health Organisation 2010).

To reduce the risk of DVT, the wearing of compression stockings is advised (Liu et al. 2009a). Compression on the leg surfaces forces blood to flow from the small surface vessels into the larger, deep venous system. It also prevents back-flow of blood and the formation of clots. However, these stockings do not deal with physical discomfort in general, such as muscle stiffness and fatigue.

To reduce this physical discomfort, contraction of muscles is very important. Muscle activity helps to keep the blood flowing through the veins, particularly in the deep veins. One way of generating muscle activity is by moving around in the cabin. However, people do not want to inconvenience neighboring passengers by passing them, or feel too tired to start walking around. Also, potential health benefits should be balanced against the risk of having many passengers on their feet in the airplane in a case of unexpected turbulence. For these reasons it is recommended that muscle activity while seated should be stimulated. Many airlines already provide their passengers with instructions for a number of exercises that can be performed during the flight. The goal of these recommended exercises is to provide the passengers with ways of reducing discomfort, fatigue, stiffness, and DVT. O’Donovan et al. evaluated lower leg exercises recommended by airlines to investigate which exercises induce optimum calf muscle pump activity(O’Donovan et al. 2006) (Fig. 1). The exercises that involve active plantar flexion of the foot induce significant levels of calf muscle activity (exercise 1, 4, 6, 7 and 8).

Fig. 1
figure 1

Recommended exercises, and EMG produced in calf muscle (O’Donovan et al. 2006)

Full size image

In the economy class cabin of an airplane, the dimensions of the seats and the space between the seats are important factors for enabling these exercises. There are no global regulations concerning seat spacing in airplanes. The “Civil Aviation Authority” (CAA) based in the United Kingdom has formulated regulations entitled “Airworthiness Notice No. 64 (AN64)” (Ball 2003) concerning seat spacing for planes registered in the UK. These Regulations date from 1989 and have not been revised since. Currently these regulations on seat spacing are still used as a basis in Joint Aviation Authorities (JAA) countries that include a large number of European countries. The seat requirements are split into three primary dimensions, shown in Fig. 2.

Fig. 2
figure 2

Seat dimensions according to AN64 (Civil Aviation Authority 2006)

Full size image

These dimensions introduce a real challenge, because the AN64 regulations satisfied 95 percent of the passengers when they were formulated in 1989. Nowadays, however, the same regulations satisfy only 77 percent of European passengers as a result of the increased average height of Europeans.

The AN64 regulations do not prescribe a minimum width of the seats. Seat widths in economy class among different airlines range from 420 to 480 mm. Looking at the current seat dimensions in an economy class cabin one may conclude that the limitations in seating space could be a significant constraint for a solution that is based on in-seat exercises. This project focuses on reducing physical stress in an economy class cabin by stimulating passengers to move in such a limited space. This should result in decreased risk of DVT and a decreased feeling of stiff muscles and fatigue (Liu et al. 2008). In order to motivate passengers to perform these movements, the in-flight entertainment system can be used as a persuasive system. The assumption is that a design that links the entertainment system to physical movements in the seating space would stimulate people to move, which would reduce physical stress.

3 Concepts

With the problem and the context as starting points, an idea generation session was held with four industrial designers. The most promising ideas generated during this session are described briefly below.

  1. 1.

    Quiz game questions are answered by body movement.

  2. 2.

    Driving game with physical steering wheel and pedals.

  3. 3.

    Agility game balancing digital objects by physical movement.

  4. 4.

    Copying game try to copy movements of others as well as possible.

  5. 5.

    Movement of passengers will be triggered by changes somewhere in the cabin.

A common detail is that all ideas except the last one are based on games. It is not surprising because games are considered to be typical interactive content that motivates people to take active roles (Breuer 2006; van de Mortel and Hu 2007).

A benchmark based on a Pugh matrix (also called a decision matrix) was used to identify the most promising idea. A Pugh matrix is a way of assessing ideas on the basis of pre-selected criteria and matching weight factors. The result of this matrix should show the most promising idea(s) according to the criteria. The assessment criteria used, in descending order of importance, are:

  1. 1.

    Applicability of the basic principle whether the basic principle behind the idea is applicable for a wide variety of games and exercises or is applicable only for a small range of games and exercises.

  2. 2.

    Feasibility within the limited seating space to what extent is the limitation of space a problem in performing the suggested exercises.

  3. 3.

    Movement intensity to what extent do passengers have to move in terms of the intensity and number of repetitions.

  4. 4.

    Disturbance of other passengers to what extent would the idea lead to disturbance or annoyance of other passengers.

The benchmark showed an almost equally high score for several ideas. The common factor between these ideas is that they contain digital content that is controlled by physical movement. On the other hand, the major difference is the specific digital content used for the different ideas. These ideas all suggest a generic interaction platform, which could be a solid basis for a wide range of applications.

The next step towards a solution is to find a consistent and reliable way to translate physical movement and body gestures into digital information. Solutions for measuring movements can be found in two directions: separate sensors and built-in sensors, as detailed below Fig. 3.

Fig. 3
figure 3

Control of in-flight entertainment by body movement

Full size image

3.1 Separate sensors

An initial set of solutions is based on separate sensors (some of which need to be attached to the user). Figure 4a shows the idea of controlling a racing game using separate pedals. Figure 4b shows the idea of having a separated device that is attached to a passenger’s body (a device attached to shoes to measure leg movements and a headphone to measure head and neck movements). This introduces some feasibility problems that can be summed up in three keywords: safety, effort, and dependency.

Fig. 4
figure 4

Suggested attached sensors

Full size image

  • Safety separate objects could cause serious safety issues in certain circumstances such as turbulence or use by passengers for unexpected purposes.

  • Effort the user has to put in extra effort in order to wear these objects or to keep these objects in correct positions. For some users this threshold is too high, with the result that they will not use the system.

  • Dependency regular objects that are already available in the airplane can be used to obtain data from the user. For example the standard headphones could be equipped with sensors to measure a passenger’s head and neck movements. The problem is that not all passengers like the same headphone, and many would bring their own.

The use of separate objects is undesirable, and therefore ideally the solution should be integrated into the airplane.

3.2 Built-in sensors

A number of built in sensors were considered. These solutions are described below.

  • Video-based gesture recognition This form of gesture recognition is not ideal because the amount of free space is very limited and the light conditions can vary. Although the light conditions can be compensated by using special cameras (for example night vision cameras), the restricted space means that it is inevitable that the movements of the passenger will partly block the view of the camera, making it impossible to detect the movements and gestures accurately. Table 1.

    Table 1 Significance of difference between before and after test for the three groups
    Full size table

  • Sensors built into the floor with sensors integrated in the cabin floor it is possible to measure the angle and distance of passengers’ feet in relation to the floor; and the lower body position of a passenger can be calculated. This could be quite accurate because human anatomy and the space available for movement are known. However, such a system must be built into the cabin floor, and this makes adaptations or repairs very difficult and costly.

  • Sensors built into the seat it is possible to measure a person’s weight distribution using a grid of pressure sensors built into an airplane seat. When there are changes in this weight distribution it is possible to derive a person’s position or gesture. Table 2.

    Table 2 Significance of difference between the control group and test group
    Full size table

The last suggestion seems to be the most promising. However, the question is, how many sensors are needed to obtain a certain accuracy level, and how many gestures can be recognized. Since the main goal is to stimulate people to move, motivating people to move is more important than knowing exactly which movements are made. The system might work even if it can recognize only a few different gestures. Therefore, a system with relatively low accuracy might be sufficient. However, this assumption needs further validation.

4 Solution and prototyping

The suggested solution is an airplane seat equipped with a grid of pressure sensors to detect a passenger’s body weight distribution. The seating surface is the most important part because there is always contact between this surface and the passenger. At the same time, the weight changes in this surface are relatively large compared to weight changes in the seat back. This does not mean that sensors in the back are useless, as these could complement the data obtained from the seating surface. The sensor values obtained are processed, and the corresponding gestures are derived by means of pattern recognition. These gestures can then be interpreted as the intended inputs to the in-flight entertainment system.

A prototype (Fig. 5a) was built to validate the concept and to prove its effectiveness. The prototype seat is made of MDF with dimensions similar to those of an average economy class airplane seat. The seating surface of this seat is equipped with a grid of 28 force sensing resistor (FSR) sensors. A 4 mm thick rubber material is placed directly over the FSRs to distribute the weight evenly over the FSRs. For each FSR, a circular MDF plate with a diameter of 40 mm is placed on top of the rubber material to increase the surface that applies force to the FSR. The grid of sensors is connected to an ATMEL ATMEGA168 microcontroller. The data is sent to a computer through a serial data connection at a rate of approximately 20 hz. The sensor values are fed into a neural network, and the output of this neural network suggests the corresponding gesture. The prototype was later implemented in a more realistic flight cabin simulator (Fig. 5b) at Department of Industrial Design, Eindhoven University of Technology.

Fig. 5
figure 5

Prototype

Full size image

4.1 Sensor layout

Several sensor layouts were tried in order to find the most accurate one for detecting the gestures. The final layout takes human anatomy into account as well as the fact that people can sit more to the left or to the right of the seating area. Figure 6 shows the final sensor layout. The red circles are the FSRs and the grey circles are the MDF discs that extend the surface that presses on each FSR.

Fig. 6
figure 6

Sensor layout used in prototype

Full size image

4.2 Hardware

The FSRs are connected by a 4–16 line decoder (SN74HC154NT) to four analogue input channels of an ATMEL ATMEGA168 microcontroller. The four address inputs from the decoder are controlled by four digital outputs of the same ATMEL microcontroller. The FSRs are placed in a voltage dividing circuit (Fig. 7) to measure the different pressures applied to the FSRs.

Fig. 7
figure 7

FSR in electronic circuit

Full size image

4.3 Neural network

The neural network is implemented using the Java Object Oriented Neural Engine (JOONE) (Jeff 2005; Marrone 2007). The neural network used for the prototype is a three-layer sigmoid network that consists of 28 input neurons. Between the input and output layer there is one hidden layer consisting of 30 neurons. The output layer contains nine neurons representing nine different gestures.

The raw sensor values are integer values ranging from 0 to 1,024. The value is normalized to a decimal number in the range 0.0–1.0 before it is fed into the neural network. For any input, all but one of the output neurons will be low (0). The neuron with a high value (1) suggests the calculated gesture. The network used for the prototype was trained in advance with sample data using back propagation.

The gestures that can be recognized by the neural network are: sitting straight with both legs resting on the seat, sitting straight with right or left leg lifted in two gradations, sitting straight with both legs lifted, sitting slumped in the seat with both legs down, sitting slumped in the seat with right or left leg lifted.

4.4 Game

To test if the system can motivate people to move, it was necessary to develop a game as part of the system. It provides a digital response to the physical movements made by a user. Instead of using an existing game, a new game was developed, firstly because the user must be able to recognize how his/her physical gestures are reflected in on-screen changes in the game. Secondly, the user could have expectations or experience of an existing game, which could affect the results. The developed game is challenging both for users who play it for the first time and for more skilled users who have played it before.

The game developed for the prototype is a skill game. The objective is to catch falling balls (displayed on the screen) in a container that slides along a bar (Fig. 8). The slope of the bar is controlled by lifting the right or left leg. The longer a user can stay in the game the more difficult it becomes, so the user is continuously challenged. The game was created with Macromedia Flash. This tool is ideal for creating relatively simple interactive content such as a game needed to test the prototype.

Fig. 8
figure 8

Screenshot of game

Full size image

4.5 Complete system

The separate parts described above are all integrated into one system (Fig. 9). The heart of the system is an application written with Microsoft Visual Basic 6.0. This application deals with the sensor input, the neural network, and the game as output. To summarize, the 28 FSRs are connected by a 4–16 line decoder to four analogue inputs of an ATMEL ATMEGA168 microcontroller. This microcontroller is connected to a computer via a RS232 connection. The microcontroller uses four digital outputs connected to the address port of the decoder to control it. After setting the decoder to the desired channel, the values of four sensors are read simultaneously and sent directly to the computer. This is a continuous process that starts with the first row of four sensor values and repeats after getting the last row of four sensor values.

Fig. 9
figure 9

Overview of all the components in the prototype

Full size image

The application on the computer side reads the sensor values sent from the microcontroller. The sensor values are stored in a buffer until all 28 values of the corresponding FSRs have been received. The data is then sent to the neural network as input. At the same time, the buffer is refreshed and is then ready for new sensor input. The communication between the Java application and the visual basic application works via a server-client construction over a local TCP socket.

After the input of a complete set of FSR values, the most probable gesture is suggested by the neural network and returned to the visual basic application. The Flash game is executed from the visual basic application. Each new user gesture triggers an event listener in the Flash game. Finally the Flash game reacts according to the detected gesture.

5 User evaluations

5.1 Experimental setup

A test was conducted with the described prototype in order to answer questions regarding the effectiveness of the concept in terms of a passenger’s physical stress reduction. The formulated research questions are:

  • Q1 Does the concept stimulate people to move? In other words is there a significant difference in terms of passenger movement compared to current situations.

  • Q2 Do passengers move enough to reduce the feeling of physical discomfort compared to current situations?

5.2 Method

A test environment was built in a small room. The room was equipped with the prototype, with a similar seat in front of it to simulate the limited seating space available in an economy class cabin. During the test, participants could use in-flight entertainment presented on a 12″ touch screen positioned in the back of the seat in front. The content consisted of a number of movies and music channels.

Every test took 1 h, and the participants were instructed to stay in the seat during the test. The 1 h duration was necessary to give rise to stiff muscles and to create a feeling of physical discomfort.

5.3 Participants

Three different groups were defined; two control groups and one test group. The groups are necessary to compare current situations with the concept.

  • Group 1 (control group) The participants in this group can use only the in-flight entertainment system.

  • Group 2 (control group) The participants in this group start the experiment by watching an introduction movie that shows recommended exercises (comparable with introduction movies of real airlines). They also have a sheet available with recommended exercises that can be used at any time during the test. They can also use the in-flight entertainment system.

  • Group 3 (test group) The participants in this group can use the in-flight entertainment system. When the user has been inactive for 25 min the in-flight system will suggest that the user play a game that implements the gesture recognition system as the game control. The user can easily ignore the suggestion if he/she does not want to play the game.

5.4 Instruments and test variables

During the test a number of variables were recorded, one of which is the amount of movement made by the user during the test. Research from O’Donovan et al. has proved that calf muscle activity is important for reducing the risk of health-related issues such as DVT (SEAT Project Consortium 2006). O’Donovan et al. have also developed a reliable way of measuring calf muscle activity by means of two accelerometers attached to the participant’s leg (Liu et al. 2010). This method is used to register the number of movements with calf muscle activity.

Another variable that is important concerns the user’s feeling of physical stress. To measure this variable a method developed by Prof. Dr. Wilhelm Janke and Dr. Günter called “Eigenschaftswörterliste” (characteristics word list) is used. This method can be described as a multidimensional way of describing aspects of how somebody feels. This method uses a list of 123 different words divided into 15 categories, which describe different aspects of various human psychological states. To get an impression of a participant’s psychological state it is necessary that he/she indicates whether his/her feeling at a certain moment agrees or disagrees with the words. The users have to fill out a list before and after the test to get the difference in state.

Only two of the categories are used, because they are relevant for answering the second research question. The categories are:

  • Activeness does somebody feel decisive, active, and energetic to a great extent or not at all (include words such as decisive, indefatigable, diligent, working, active, competent).

  • Fatigued does somebody feel passive and fatigued to a great extent or not at all (include words such as negligent, indifferent, fatigued, tired, slow, lethargic).

Finally all the tested variables are compared among the different participant groups.

5.5 Results

Twelve participants were divided equally over the three test groups. Observation proved that the control group with recommended exercises did not do any exercises at all. In this case there is no difference between the two control groups, because the recommended exercises provided to one of the two test groups had no influence on the amount of movement made by the participants. For this reason the two control groups are combined for analyzing the data. This means that there were eight participants in the control group and four in the test group.

5.5.1 Amount of calf muscle activity

The box plot of the data shows that there are relatively big differences between the groups (Fig. 10), in particular between the control groups and the test group. An ANOVA analysis showed a significant difference (significance of 0.004) between the test group and the control groups. It is evident that the test group made more movements involving calf muscle activity compared with the control groups. In other words, by comparison with current situations the concept does stimulate people to move.

Fig. 10
figure 10

Times of active calf muscle pump during test

Full size image

5.5.2 User’s feeling

To make a judgment on how the use of the prototype influences the user’s feeling of activeness and fatigue the participants had to indicate their psychological state before and after the test. The values retrieved from the questionnaires range from 0 to 1 in both categories (Fig. 11, 12). An ANOVA analysis is applied between the differences per group per category. This shows that the participants in the control group felt significantly less active after the test. There is also an indication that participants in the test group felt less fatigued after the test.

Fig. 11
figure 11

Results of activeness category

Full size image
Fig. 12
figure 12

Results of fatigued category

Full size image

The ANOVA method was also used to test whether there is a significant difference between the control and test groups. It proved that participants in the test group felt significantly less fatigued compared to the participants in the control group. It also indicates that participants in the test group felt more active compared to the participants in the control group, only this is not proved with a high level of significance.

6 Discussion and conclusions

The SEAT project focuses on a stress free hence better travel experience for passengers instead of the lower cost for airlines. But with limited investment the airlines can provide a better service hence have a better competitive position. This is rather speculative. However, the cost-benefit analysis of the proposed solution is beyond this paper and the SEAT project. The proposed solution suggests embedding sensors into the airlines, and these sensors are, like any other additional electronics adding to an aircraft, yet to be checked and verified against aviation regulations and security measures.

Nevertheless, to reduce both psychological and physical stress in air travel, the work presented here suggests as a solution the integration of sensors in airplane seats to detect gestures as input for in-flight entertainment systems. The content provided by the entertainment systems would help to reduce psychological stress, and the gesture recognition is used as input for the interaction and thus stimulates people to move, reducing physical stress as well.

The proof-of-concept prototype was built to demonstrate the feasibility, and for user research, focusing on the user experience, i.e. whether the solution simulates passenger to move and whether it reduces the feeling of discomfort. The prototype was not suitable for evaluating the effectiveness and the efficiency of the technology itself. If the concept is accepted by commercial airlines, more rigid engineering is necessary to turn the prototype into an effective and efficient product.

Initial user research has proved that users of the concept have significantly more active calf muscle pump activity in comparison with the current situation. In other words, the concept is able to stimulate people to move. Future research is needed to validate the next research question on whether the risk of health-related issues such as thrombosis is really reduced by the use of this seat. In terms of how users feel after using the concept as compared with the current situation, the research showed that the users feel more active and less fatigued after a long period of sitting in a limited seating space.

The concept has been adopted in the SEAT project by other partners. Further development is being undertaken using sensors integrated into the fabric covers, leading towards commercial products.