Keywords

1 Introduction

During our vacation time, especially in the summer, we would like to interrupt our daily routine and to regain the energies we lost during the whole year. At the beginning of the summer, many people do not like the image reflected in the mirror, and it is common to start the vacation with the intention of spending some time in physical activities. Mirrors do not lie... But what if mirrors were able to suggest solutions rather than complain?

This is the objective of the Virtuoso project: to create a seamless interactive support for fitness and wellness activities in touristic resorts. The overall idea is to evaluate the current physical state of a resort guest through a mirror metaphor. Then, the system will try to help her in enhancing it during the stay, suggesting different activities available in the resort. Both the analysis and the suggestions should be provided in a playful way. In this paper, we introduce the project requirements and we provide a survey on the existing enabling technologies for creating the mirror metaphor and to acquire data from the user. It can be helpful for researchers and practitioners interested in creating a similar environment.

From a technological point of view, creating such environment poses a set of challenges, both technical and related to the user experience. The first challenge is how to design and realize the smart mirror and its related apparels. Different companies provide off-the-shelf smart mirrors whose functions are related to controlling and supporting smart home environments, multimedia controllers and augmented reality (AR) devices. In this paper, we will provide a detailed review of the existing devices in all three categories, discussing their advantages and disadvantages in general, and in particular for the Virtuoso project.

The second challenge for our project is how to support the evaluation of the user’s physical state. Such evaluation does not require the same precision if compared to a real diagnosis, which is out of our scope. In addition, since a resort is a relaxed environment where users expect to enjoy themselves, it is not feasible for Virtuoso to use intrusive equipments or methodologies for acquiring data. Therefore, in this paper we analyse different measurements that can be quickly performed on a user. For each of them, we detail the procedure for acquiring the data, assessing its cost both money-wise (i.e., the equipment to be used) and user-wise (i.e., how bothering it can be).

At the end of the vacation the mirror should profile our user again, if the activities had their effect, the resort guest can bring home a valuable souvenir: an improved physical shape. Or at least some good suggestions!

2 Virtuoso

The main aim of the Virtuoso project is to provide a set of tools, easy to use, that does not necessarily require to be used by specialized health personnel (medical or paramedical), to assist tourist operators. The typical user of the system will be the fitness and/or wellness coach. Such a kind of professional is a figure that is more and more diffused also in a contexts of vacation resort. The activity of the wellness coach is to identify the tourists’ need and guide them in choosing and practicing fitness activities offered by the resort.

The main tool we want to develop will be a computer-based application for the general assessment of the psychological and physical conditions of the customer. We aim, in other words, to measure his or her well-being with some simple and compact indicators. The application used by the center will have a companion mobile app, for end users, that allows to browse the information produced in the initial evaluation session, to log the activities during the vacation, to socially share it, and to bring back home the lesson learnt and continue to practice the good customs.

To reach this goal we will study and develop methods and techniques for the analysis of the general health of an individual, based on heterogeneous data acquired from non-invasive and non-intrusive sensors like, for example, cameras and platforms with sensors, contactless or minimally contact, such as multifunction armbands.

The basic idea behind the whole project is the ability to assess the physical and psychological condition of an individual and give him or her good advices on how to improve, through an analysis of diagnostic signs collected using different techniques and technologies.

3 Smart Mirrors

In this section we review the existing technologies that may be used for building interactive mirrors. Recent introduction of new materials and the overall ICT proceedings make different solutions for building interactive mirrors possible, in order to transform it from a simple reflecting surface to a more interactive device. We can group them in three sets: (1) Multimedia players (e.g. music or videos), (2) Interactive home controllers and (3) Augmented reality devices. We discuss their advantages and disadvantages in general, and in particular for the Virtuoso project.

3.1 Multimedia Players

The multimedia players category contains devices equipped with a touch screen, or TV enhanced with external full body tracking devices (e.g. Microsoft Kinect). However, most of them support only entertainment tasks, like playing music or videos.

  • Mirror 2.0 [1] combines the advantages of a smartphone and a mirror. It contains an LCD display, positioned behind the glass. It provides news and weather information and it allows the playback of both videos and music. The user controls its functionalities through gestures and vocal commands.

  • Smart Washbasin [2] displays different information on a washbasin mirror such as mails, weather forecast, the water temperature and pressure, the calendar and the user’s weight measured through a built-in scale in the base portion. The device consists of an Android tablet that displays the widgets on the basin mirror, made with a semi-reflecting glass put on top of an LCD display. It is possible to control it without touching the screen surface, since it is equipped with proximity sensors able to track the hands position and motion.

  • NEOD Framed Mirror TV [3] is a standard LCD screen (up to 50 in.), covered by a mirror, specifically designed for the screen. The screen provides only TV functionalities, and it does not provide more interactive features.

3.2 Interactive Home Controllers

Interactive home controllers allow to control different home appliances through the interface displayed on the mirror. The following is the list of the devices belonging to this category.

  • The Smart Mirror for home environment [4] allows to control all the smart devices at home. It relays on face recognition for authenticating the user and displays personalised information (news, mail, messages etc.). The system exploits a touch screen monitor and two webcams, one for the face recognition and one for the home surveillance.

  • The Multi Display in Black Mirror [5] by Toshiba is a prototype that combines the functionalities of a tablet together with the reflecting surface of a mirror. It provides two configurations taking into account two different home environments: the bathroom and the kitchen. Considering the bathroom, the prototype provides useful information for the beginning of the day such as the weather forecast and fitness information coming from personal devices. In the kitchen, the setting includes a camera allowing the user to interact through gestures while preparing recipes and controlling the appliances.

3.3 Augmented Reality Devices

Many mirror devices provide Augmented Reality (AR) features, which are useful especially for advertisement purposes in shop showcases. In addition, there are different attempts to use smart mirrors for supporting routine activities. In this category we can include, for instance, an application that monitors and guides a user while brushing teeth through the information displayed on the smart mirror. Finally, the so called “medical mirrors” are particularly interesting for the Virtuoso project, since they can measure different physical parameters such as heart and breath rate.

  • The Reveal project [6], created in the New York Times research and development department, consists of an LCD Display covered by a mirror glass. The device exploits a Microsoft Kinect for tracking user’s movements in real-time. It visualizes different information on its surface (calendar, mail, news, online shopping websites, instant messenger etc.). In addition, it responds to vocal commands. A peculiar feature is the medicine box scanner, which allows the user to buy medicines recognizing their packages.

  • The Cybertecture Mirror [7] is a complete PC contained into a 37 in. mirror, equipped with a 32 in. LCD screen. Through a smartphone application, the user accesses different information overlaid on the reflected image. The interface allows to visualize instant messages, the calendar, the mailbox, and the weather forecast. In addition, it provides information on the user’s physical state. Indeed, the device provides a set of external wireless sensors that allow to measure the user’s weight, fat, muscle and bone mass.

  • The Interactive Mirror [8] by Panasonic seems to be an ordinary mirror: neither camera nor other sensors suggest the features of a smart object. Once the user sits down in front of it, the mirror displays an enlarged frame for her face, together with menus for accessing different functionalities. The system analyses the face hydration, wrinkles and other details in order to recommend products and treatments to take care of her skin (e.g. to make it softer etc.), to slow ageing and so on. The mirror supports the user in buying such products. In addition, it provides make-up style previews, simulating lighting and ambient conditions (e.g. at home, outdoor, shopping center etc.).

  • The Connected Store Demo [9] by eBay and Rebecca Minkoff provides interactive experiences in both the store showcase and in the fitting room, equipped with a mirror surface overlaid by store user interface. In the showcase setting, the user explores the different items in the store. Once she finds something interesting, she requests to try it in the fitting room. Once finished, the shopper prepares the fitting room with all the items. Inside the fitting room, the user exploits the mirror for looking for other items and/or providing feedback. In addition, she may select some of them for buying.

  • The Brushing Teeth Mirror [10] displays the information collected by a smart brush about inflammations or infections of the teeth and gums.

  • The Medical Mirror [11] combines computer vision and signal processing techniques for measuring the heart rate from the optical signal reflected of the face. The prototype consists of an LCD display with built-in camera and a two way mirror fitted onto the frame. The smart mirror recognizes the presence of a user when she stands in front of it and, after about 15 seconds, it displays the heart rate below the user’s reflected image.

4 Physical State Acquisition

In this section we report different physiological parameters that can be evaluated through different sensors. We summarize their definition and the measurement procedure. After that, we compare them in Table 1, according to three dimensions:

  1. 1.

    How much the procedure is annoying or requires uncommon actions for a user (Impact). We consider the project’s audience: people monitoring their physical status for wellness purposes and not for a real diagnosis (they are spending their holidays in a resort). We consider three possible degrees: High if the procedure takes a long time or it requires difficult actions, Low if the procedure is short or very easy, and Medium for values in between.

  2. 2.

    Whether the sensors used for measuring the parameter require some Contact or not.

  3. 3.

    The Cost of the equipment for measuring the parameter. We considered three cost levels (Low, Medium and High).

For instance, acquiring user’s weight through a pressure sensitive board (e.g., the Wii Balance Board) and her height through a laser tool, makes it possible to calculate the Body Mass Index (BMI). Combining such data with a volumetric representation of the user’s body allows to analyse the body weight distribution without any direct contact with the user. So the BMI has a low cost in terms of intrusiveness and a reasonable cost in terms of hardware. Such measurement together with other information provide the evaluation data and could drive the activity during the stay. The following is the list of parameters we considered:

  1. 1.

    Metabolic Balance of Fat and Glucose. Breath acetone is a parameter for detecting the correct exploitation of all energetic substrates [12]. Indeed, some metabolic diseases cause a predominant use of the fat substrate instead of glucose. This increases the production of acetone which is expelled through the lungs. The presence of acetone in the breath may indicate infectious diseases or diabetes. It can be measured through colorimetric reactions on a disposable transparent support with a reactive substance. We can detect the colour change with a camera.

  2. 2.

    Glucose metabolism. There is a correlation between the saliva and the blood glucose level. A variation of the value from the reference levels reveals an altered glucose metabolism, caused by organ or hormonal diseases. It can be measured again through colorimetric reactions.

  3. 3.

    Circulation of blood. An irregular body heat distribution indicates blood circulation problems such as e.g. the venous stasis. Therefore, we can obtain such distribution through an infrared camera and analysing the thermal image [13].

  4. 4.

    The Body Mass Index (BMI) is the ratio of a person’s weight and height squared. This biometric parameter indicates whether the weight correlates well with a height. In general, a person having a BMI greater than 25 is overweight, greater than 30 is obese, while below 18.5 is underweight. For calculating this index it is sufficient to measure the weight through a pressure board (e.g. the Wii Balance Board [14]) and the height with a laser sensor

  5. 5.

    Fat distribution. It is possible to use the ratio between waist and hips circumference for identifying an excessive visceral adipose tissue mass accumulation, which is strongly correlated with cardio vascular diseases. The circumferences can be measured through laser and infrared sensors [15].

  6. 6.

    Heart and breath functionalities can be evaluated through the oxygen-haemoglobin saturation levels (indicating a good lung ventilation and blood) and the heart rate. We can measure these parameters using a pulse oximeter, which requires a light pressure on a finger.

  7. 7.

    Stress level. Persons react to specific visual stimuli with different face movements, which can be evaluated for establishing the stress level. Therefore, it is possible to create a test where the user looks to a sequence of pictures and a software analyses the face movements [16].

  8. 8.

    Muscular tone evaluation. The muscular strength allows to evaluate the physiological condition of the active lean mass. We consider the strength of the quadriceps and of the dominant hand and forehand muscles, since they represent better the general condition of the entire muscle mass. We can measure the quadriceps strength with a dynamometer set in a fixed position while we measure the hand strength measuring the grip force [17].

  9. 9.

    Body mass distribution. The intracellular and extracellular water volume allows to evaluate the lean mass percentage (through a simple ratio). We can measure such volumes through the bioimpedance analysis: the user stands on impedance board and grabs two handles to allow the electricity flow from hands to feet [18].

  10. 10.

    Hearing can be evaluated with an audiometric curve: a device emits a sound scale, ranging from 15 to 20000 Hz. The volume is variable between 0 and 100 dB. The user listens to the sounds and signals at which pitch s/he starts hearing [19].

  11. 11.

    Night vision. We evaluate the functionality of the rods (the black/white and shape photoreceptors) testing the adaptability of the eye to night vision, asking the user to recognize the objects in a set of images in low light conditions. We evaluate the recognition and detail degree of their image descriptions.

  12. 12.

    Blood pressure. It provides information about the circulation of blood and also on the stress level. We can measure it with a sphygmomanometer applied at the user’s wrist. The method is unintrusive and requires some contact.

Table 1. Physical state parameters summary.
Full size table

5 Setup Discussion

The ideal setup of the system developed in the Virtuoso project is as following.

Imagine a tourist arriving at his or her chosen resort, in summer, in Sardinia, to enjoy a relaxing week or fortnight of vacation. Most probably, he or she does not like to undergo a set of medical exams to figure out what is his or her state of health. But, what if, he or she will enter wearing a swimsuit, a funny room whit a board on ground, and a Microsoft Kinect mounted on a rotating arm capable of turning around the board, and a mirror interacting with him or her asking to answer simple questions just waiving a hand?

This will be a kind of fun, subtracting just, maybe, five or ten minutes to the beach, and will be the basic set of information to compute a profile containing age, height, weight, BMI, body volume, fat vs muscle ratio, and some more parameters that, put together in a model, will tell which kind of activity is best suit and, why not, the best restaurant you can find in the resort for your needs. This is the long term goal of the project, scheduled for the summer of 2016.

In order to put such objectives into practice, we plan to create our mirror device prototype, starting from existing consumer hardware (Kinect 2.0 and a wide LCD screen), similarly to those described in Sect. 3.3. Considering the physical state parameters, we plan to include those having a low or medium cost, without any contact and with a low user impact in the measurement.

With respect to the evaluation of the prototype, we plan to validate the Virtuoso results in a holiday resort scenario. Being this setting one of the most important for wellness in both research and industrial effort, we expect to collect important insights on the effectiveness and the acceptance of the approach. We plan to perform both technical test and evaluations with end users.

On the technical level, the evaluation should address two points. The first one is assessing the reliability of the physical insights collected through our prototype with ground-truth data. In order to do this, we will evaluate a group of people whose physical state has been already assessed with proper diagnosis techniques and we will compare the results. The second point is related to the deployment of the entire solution in a real resort setting, considering hardware, software and the staff training.

Last but not least, we plan to evaluate the user overall usability of the approach through a long-term study. We will deploy the system in a well-known resort in Sardinia, and we will collect both qualitative and quantitative data. In particular, we plan to apply the SUM model [20] for combining different usability metrics into a single score. In order to collect the data, considering that it would be annoying for resort guests to complete questionnaires, we plan to instrument the software for tracking task completion, time, and errors. With respect to the post-task satisfaction ratings, we will provide a playful interface for collecting the answer, e.g. punching the rating number as physical exercise. Such evaluation method is convenient since it would be possible to automatize the data collection and to perform analysis at both a global and a task level.

6 Conclusion and Future Work

In this paper, we reported on the state of the art technologies for building smart mirrors and for sensing the user’s physical. We described smart mirrors devices and prototypes working as multimedia players, interactive home controllers and augmented reality devices. Moreover, we described and compared different parameters for evaluating the user’s physical state, considering whether they require annoying or uncommon actions, the contact between the sensors and the user and the procedure cost.

Finally, we introduced the scope and the objectives of the Virtuoso project, which aims to create a seamless interactive support for fitness and wellness activities in touristic resorts. We explained how, in future work, we are planning to create the project setup and to evaluate its results.