Abstract
Shopping on the Internet has many advantages, compared to shopping on-site. However, shopping on the Internet has limited user experiences, especially for large and heavy furniture products. In addition, users cannot change the materials and dimensions of the online furniture products. As a result, online shoppers often cannot make right purchasing decisions. This paper presents an augmented reality furniture customization system, which provides users the abilities to view and change the materials and dimensions of three-dimensional virtual furniture, within the contexts of real environments. A Kinect is used to track the human body motions. Occlusions for real and virtual objects in different depths are considered to increase the realism of the system. A calibration algorithm is developed to find the depth, IR image, and RGB image information of the Kinect, to improve the image quality in the augmented reality environment. Furniture customization functions are provided to improve user experiences. User test results show that users consider the augmented reality-based furniture customization system realistic and natural to use.
Keywords
1 Introdcution
Augmented Reality (AR) is a technology which combines virtual objects with real scene captured from cameras. Applications of AR are various. For example, Ling and Shuyu used AR technology in textbook design to improve students’ learning [1]. Carozza et al. used AR technology in city design [2]. In order to improve the realism of AR scenes, virtual objects and real objects must have correct spatial relationships. However, traditional AR always places virtual objects in front of the real objects. It may confuse users and cause decision failure. Therefore, solving occlusion problems becomes an important issue in AR. Hayashi et al. solved the occlusion problems by combining stereo information to find the depth of a moving object [3]. Fortin and Hebert fixed the locations of cameras and physical scenes to find depth information of real objects to solve the occlusion problems [4].
Interaction between users and virtual objects is another important issue in AR. Seo and Lee tracked the positions and motions of hands [5]. When users clicked an image on a plane board, they could interact with the virtual objects. Radkowski and Stritzke used a Kinect to track human bodies to control a virtual cursor for assembly and disassembly training [6].
Correct occlusion and smooth interaction can improve the realism of an AR application. Most previous AR furniture display systems do not allow users to view virtual furniture with accurate relative positions, orientations, and sizes, in the context of the physical environment. In addition, they do not allow users to change the dimensions and materials of virtual furniture in the AR environment.
In order to allow users to directly, intuitively, and naturally interact with virtual objects, in this study, a Kinect was used to obtain the depth information of the real world. A two-step method was developed to align the depth images and RGB images. Z-buffer was overlapped with the calibrated depth image to solve the occlusion problem. This study built a furniture display system to verify the developed method. A user test was conducted to evaluate the effectiveness of the system.
2 Method
2.1 Calibration of Kinect
Although the Kinect is capable of providing depth information, the depth images and RGB images are mismatched, as shown in Fig. 1. Because of different angles and positions of the infrared camera and color camera, the Kinect needs to be calibrated to align the depth images and RGB color images. An object in the world coordinate first needs to be projected to the camera coordinate by the external parameters of the camera, and then projected to the image coordinate by the internal parameters of the camera.
Stereo Calibration.
Kinect camera calibration (stereo calibration) needs to find the external parameters between the color camera and infrared camera to obtain the correct relative rotation, translation, and scaling relationships between the two cameras. This study used the internal parameters of the color camera and infrared camera and the image pairs, which were taken simultaneously by the two cameras, to perform stereo calibration. The calculation method was carried out using the Matlab Camera Calibration Toolbox.
Translation Correction Between Depth Image and Infrared Image.
The translation correction was performed with a star-shaped calibration board and an ellipse fitting method. The depth image and infrared image were taken simultaneously. The system detects ellipse edges in the depth image and find the corresponding edges in the infrared image to calculate the least square deviation for translation, as shown in Fig. 2. Figure 3 shows the images before and after translation. The result shows that the depth images and infrared image can fit together.
2.2 Occlusions in AR
AR is a computer-generated environment combined with real scene and virtual objects. The key to construct a realistic AR environment is to have a correct transformation relationship between the camera coordinate system and the world coordinate system. This study uses NyARtoolkit (http://nyatla.jp/nyartoolkit/wp/) as the development toolkit. NyARtoolkit is an open source, based on marker tracking. This study verifies the correctness of the AR coordinate system by placing a virtual block of 160 mm (width) × 80 mm (height) on a marker of 160 mm (width) × 160 mm (height), as shown in Fig. 4. The result shows that the dimension of the virtual object can correctly fit with the real object in the AR environment.
Occlusions.
The occlusion problem was solved by overlapping the calibrated depth image with the Z-buffer of the DirectX. Thus, the Z-buffer represents the real depth of the physical world. Before rendering the virtual objects, the z coordinates of the virtual objects are compared with the depth information in the Z-buffer. If the virtual objects are nearer to the observers, they are rendered. Otherwise, the virtual objects are occluded. In this study, a 32bit Z-buffer and DirectX 9.0c were used. Figure 5 shows an example of the occlusion results.
2.3 Interaction in AR
This study creates a cubic event trigger which is used to fire the interaction event. To improve the stability in controlling virtual objects, a bigger cube which is apart from the event trigger 200 mm was created as a separation detector. When users’ hands touch or enter into the event trigger, an interaction event happens and the event trigger will show up as a transparent red area. When users’ hands move out the separation detector, the interaction event ends and the transparent red area disappear. This study created four kinds of basic interactions, which were resize, translation, rotation and animation. The four interactions should be triggered by both hands. With Kinect’s body tracking capability, the positions and parameters of both hands can be obtained to manipulate the virtual objects. Figure 6 shows that when the user’s hands enter into the event trigger, a transparent red area appears to show the interaction mode.
3 Experiment
Two experiments were carried out to test the developed AR furniture display system. Furniture models were taken from the TF3DM open source (http://tf3dm.com/3d-models/furniture). There were 12 men and 11 women in the test. In the first experiment, subjects needed to interact with the virtual furniture and place the virtual furniture at the correct locations, Fig. 7 shows an example. A questionnaire with a Likert scale of 1 (extremely disagree) to 7 (extremely agree) was used to evaluate the AR system, after the subjects finished the task. Table 1 shows the questionnaire results. Most subjects considered the AR system helpful, interesting, and smooth to use. Most subjects also would like to use this system when purchasing furniture online.
The second experiment tested the effects of occlusion. Subjects first arranged the furniture as the first experiment. Then, subjects followed the instruction to go to certain locations, with and without occlusions. Another questionnaire was used to evaluate the occlusion effects. Table 2 shows that the AR system with occlusion is significantly more realistic and intuitive than that without occlusion.
4 Conclusions
With the advance of computer hardware and software, AR applications are becoming more popular. Correct occlusion and direct interaction can improve the realism of an AR environment and help users be immersed in the AR environment. This study developed an interactive AR system using natural user interface with occlusion. The discrepancy between the depth image and infrared image was resolved. Users can directly interact with the virtual objects with their bare hands. The results of the experiments also show that most subjects had positive responses toward the AR system. They considered the system helpful, interesting, realistic, intuitive, and smooth to use.
References
Ling, Z., Shuyu, Z.: Design research and practice of augmented reality textbook. In: International Conference of Educational Innovation Through Technology EITT 2014, pp. 16–20, Brisbane (2014)
Carozza, L., Tingdahl, D., Bosché, F., Gool, L.: Markerless vision-based augmented reality for urban planning. Comput.-Aid. Civil Infrastruct. Eng. 29(1), 2–17 (2014)
Hayashi, K., Kato, H., Nishida, S.: Occlusion detection of real objects using contour based stereo matching. In: Proceedings of the 2005 International Conference on Augmented Tele-Existence, New York, USA, pp. 180–186 (2005)
Fortin, P.A., Hebert, P.: Handling occlusions in real-time augmented reality: dealing with movable real and virtual objects. In: The 3rd Canadian Conference on Computer and Robot Vision, pp. 54–54 (2006)
Seo, D.W., Lee, J.Y.: Direct hand touchable interactions in augmented reality environments for natural and intuitive user experiences. Expert Syst. Appl. 40(9), 3784–3793 (2013)
Radkowski, R., Stritzke, C.: Interactive hand gesture-based assembly for augmented reality applications. In: The 5th International Conference on Advances in Computer-Human Interactions, Valencia, Spain, pp. 303–308 (2012)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2016 Springer International Publishing Switzerland
About this paper
Cite this paper
Young, TC., Smith, S. (2016). An Interactive Augmented Reality Furniture Customization System. In: Lackey, S., Shumaker, R. (eds) Virtual, Augmented and Mixed Reality. VAMR 2016. Lecture Notes in Computer Science(), vol 9740. Springer, Cham. https://doi.org/10.1007/978-3-319-39907-2_63
Download citation
DOI: https://doi.org/10.1007/978-3-319-39907-2_63
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-39906-5
Online ISBN: 978-3-319-39907-2
eBook Packages: Computer ScienceComputer Science (R0)