Keywords

1 Introduction

The taste information obtained from the taste cells present in the tongue accounts for the majority of the human gustatory sense [1]. However, it is known that the gustatory sense is affected not only by taste information but also by other sensory information such as visual information [2, 3]. Recently, studies have been increasingly being made to change dining experience by externally changing sensory information [4]. In particular, the influence of olfactory and visual information on taste sensation is large, therefore a lot of studies have been done by using this [5,6,7,8]. These effects are recognized also in psychological experiments [9]. For example, Meta Cookie gives a change to the feeling of human gustatory sense by changing both visual information and olfactory information [8]. It’s also said that auditory information when chewing food, tactile information such as the texture of food and chewy feeling influences the feeling of taste [10,11,12]. Previous research suggests that changes in sensory information from the outside can also change taste information. However, quantitative evaluation of how taste information is affected by changes given to sensory information has not yet been made. If a quantitative evaluation is possible, more applicative control of the gustatory sense becomes possible. Among the sensory information affecting the taste like this, the visual information can change most easily the feeling of the taste significant. Unlike other sensory information, the visual information can be separated three primary colors of light, therefore it is easy to present any visual information. Therefore, by changing the color of the food through the visual sensor and visually recognizing the food through the display, it is possible to change the visual information of the food. In addition, as the penetration rate of smartphones is increasing in recent years, there are many scenes of possessing visual sensors such as cameras. There are also many Head-Mounted Display (HMD) that can provide VR environment easily by wearing smartphones, therefore it is possible to impart real world to a chromatic transition of food.

Also, in recent years, content production related to dining experience has been actively performed [13, 14]. Dining experiences that are familiar to humans should be always new and enjoyable by adding such entertainment characteristics. It is expected that the control of the gustatory sense will greatly contribute to such content creation in the future.

In this research, we aim to verify the influence of a chromatic transition of food through a HMD on the impression of taste for food. For this purpose, we constructed a system that gives a chromatic transition to the color of food through HMD. In this research, we also examine the difference between visually recognizing food with the naked eye and the recognition it through HMD. Therefore, to control the impression of human taste by optimal color change, it is necessary to consider the difference between the case of visually recognizing with the naked eye and the case of seeing through the display.

2 VR System of a Chromatic Transition

In this research, smartphone and HMD were used as a color change presentation device. iPhone 7 Plus which iOS 10 introduced by Apple Co., Ltd. was introduced, and VOX + 3DVR goggle provided by VOX was used for HMD. We can really find the image which we acquired from the camera of the smartphone like the image actually in front of the eyes by attaching HMD which set the smartphone. By giving a color change to the image acquired from this camera and outputting it to the display, it is also possible to give a chromatic transition to the user. These devices are shown in Fig. 1.

Fig. 1.
figure 1

Experiment environment and a virtual chromatic transition system (Color figure online)

Full size image

And, we installed the application created in Xcode on iPhone 7 Plus and used it to detect target food and change color to that food. The source code of the application is written in Objective-C and introduced the OpenCV framework for image processing such as a color change to target food. A flow chart showing the flow of the created application is shown in Fig. 1, and detailed specifications are shown below.

  1. 1.

    Acquire an image from a camera mounted on a smartphone

  2. 2.

    Convert all the pixels in the image obtained from the camera from the original RGB color space to the HSV color space

  3. 3.

    Of the parameters of the HSV color space, make specified color change for pixels with V (Value, Brightness) above a certain value

  4. 4.

    Output the pixel on which the color change was made on the original image to the display

The food whose color has been changed by the above operation is outputted to the display and made to be visible to the human through the HMD.

3 Experiment of Verification

3.1 The Experimental Environment and Target Food Item

This experiment was conducted for 11 males and females in their twenties who are healthy with both visual and taste. The experimental environment is shown in Fig. 1. In this experiment, the application was carried out in a room that turned off the light, unifying the background to black so that the application can easily detect white food. A target food was placed on the inside of which processing was made by attaching a black felt fabric around an acrylic box which is a cube having one side of 300 mm and the front side does not exist, and this subject was visually recognized by the subject.

In this experiment, the self-made fizzing candy was used as the target food item. All of the ingredients were made equal in 0.1 g unit so that each of the fizzing candy had the same taste. Those colored by using coloring agents were prepared for each of them. The colors used this time are five types of red, blue, yellow, green and purple shown in Fig. 1. Experiments were carried out using a total of six kinds of color fizzing candies to which white of the reference color was added.

3.2 Method of Experimentation

In this experiment, we asked each subject to answer the questionnaire about the impression of the taste of the target food. In addition, the experiment was conducted under six situations in which three factors, “before or after eating,” “viewing through HMD or naked eyes,” “with Virtual chromatic transition or without” were changed. A table summarizing these is shown in Table 1, a diagram showing these conditions is shown in Fig. 2. To this “six colors of fizzing candy” will be added.

The questionnaire prepared two kinds of contents different “before or after eating”. The contents of the questionnaire before eating are two items, “What kind of taste do you think it is?,” and “What kind of food do you imagine?”. Next, the contents of the questionnaire after eating are three items, “What kind of taste did you feel?”, “What kind of food did you imagine?”, “Were there any differences of impressions before eating?”. The first questionnaire item evaluated what kind of taste it felt before and after eating about five colors excluding white, in 5 grades each of the five flavors. At this time, white was defined as the evaluation value 3 in all the five flavors before and after eating, and it was used as a standard.

Table 1. Situation of experiment
Full size table
Fig. 2.
figure 2

Situation of experiment

Full size image

3.3 Result

From the experimental results, evaluate how far away the impression was from the white fizzing candy received as a reference against the colored. At this time, we define the taste space as 5-dimensional space and evaluate the evaluation value of each basic taste answered by the subject to Euclidean distance when white is taken as the origin and evaluate.

First, in order to compare “before or after eating” and “viewing through HMD or naked eyes”, we perform Analysis of variance using the result of experiment 1, 2, 3 and 4, as factors. Next, in order to compare “before or after eating” and “giving or not virtual chromatic transition”, we perform an analysis of variance using the result of experiment 3, 4, 5 and 6, as factors. Graph showing these results is shown in Fig. 3.

Fig. 3.
figure 3

Result of analysis of variance

Full size image

The abscissa of Fig. 3 shows the six situations of experiment shown in Table 1, and the ordinate shows the average value of the Euclidean distance of the result calculated for each situation of the experiment.

Firstly, as a result of analyzing the experiment #1, #2, #3 and #4, the interaction of “before or after eating” and “viewing through the HMD or naked eyes” was superior at 5% level or less. It was also revealed that the impression before eating was different depending on the presence or absence of the HMD by the simple main effect test. It seems to be due to the difference in the amount of visual information obtained when comparing the state in which the food is viewed through the HMD and the state in which the food is visually recognized with the naked eye. The information acquired from the camera has a difference in information amount as compared with information obtained from the naked eyes due to various factors such as image quality and viewing angle. It seems that the difference in this amount of information affected the impression of the food before eating.

Next, as a result of analyzing the experiment #3, #4, #5 and #6, the interaction of “before or after eating” and “with virtual chromatic transition or without” was superior at 1% level or less. Moreover, it was revealed that the impression before eating is caused by the presence or absence of color change by examination of simple main effect. This seems to be related to the amount of visual information as well. The amount of visual information decreases if it passes through a display like a camera. The application used in this experiment was to add a chromatic transition from the top of the image acquired from the camera. In other words, when a chromatic transition is added by the application used in this experiment, it can be said that it is easier to obtain visual information because clear colors can be seen compared with the case where no chromatic transition. It is seemed that the difference in the amount of visual information has influenced the impression of the food before eating.

4 Conclusion

In this study, we examined the difference in the feeling of taste especially when visually recognizing the food with the naked eye and through the HMD. For this purpose, we installed an application on the HMD and made an application to change the color of the image acquired from the camera mounted on the smartphone. The experiment was conducted under six situations in which three situations, “before or after eating”, “viewing through HMD or naked eyes”, “with virtual chromatic transition or without” were changed. In each case, differences in the taste impression due to these factors were verified by subjective evaluation about the taste impression of the target food. The result of analysis revealed that “viewing through HMD or naked eyes,” and “with virtual chromatic transition or without” respectively influence the impression before eating food. Therefore, the experiments suggested that the impression of food tastes were changed due to the difference in the amount of visual information between the naked eye and the screen. In other words, when giving a chromatic transition to a food through a screen such as HMD, it is possible that considering the difference in impression of the case of visual recognition with the naked eye leads to more advanced control of taste.

Moreover, verification experiments have revealed that there is almost no influence on the feeling of the taste of eating food merely by changing the visual information. Changes in olfaction, tactile sensation, and auditory information are thought to be useful for effecting this. From now on, we will investigate how they affect taste sensation by changing these, and also realize a taste control system by constructing a multimodal interface including these.