Interactive Augmented Reality Pop-Up Book with Natural Gesture Interaction for Handheld
- 76 Downloads
Handheld augmented reality (AR) has been widely used with smart and portable device in the applications such as education, games, visual experience, and information visualization. However, most of the handheld applications do not fully support natural interaction, and the existing 3D pop-up book has used touch-based to interact with 3D content. Therefore, this entry describes a fundamental to design an interactive AR pop-up book with natural gesture interaction using real hand. Subsequently, the real hand gesture tracking in handheld AR is explored to examine how it can track user’s hands in real time. Thus, this entry describes about gesture interaction to allow the user to directly interact with the virtual objects. The user feels more realistic to interact with 3D objects using their bare hands on 3D pop-up book.
Augmented reality (AR)33 is a technology that allows computer-generated information or digital information including text, video, 2D virtual images, and 3D virtual objects to be overlaid onto the real-world environment in real time (Ismail and Sunar 2013). The main reason people intend to develop AR application is to merge the real world into the virtual world to provide the users with information-enhanced environment (Billinghurst et al. 2008). The connection between these two worlds seems impossible back then, but now it becomes an attraction, and its potential was very overwhelming. Usually, the virtual elements are generated by the computer and made to be overlaid onto the real world, to enhance the user’s sensory perception of the augmented world they are seeing or interacting with.
Nowadays, the concept of AR technology is used widely in entertainment, military training, engineering design, robotics, manufacturing, and other industries. AR technologies bring a lot of advantages to perform a task especially once it involves with design and planning. AR has the ability to perform 3D object manipulation and can provide natural user interaction techniques (Ismail and Sunar 2013). All developers take an advantage on AR technologies and believe it could help them to perform real task in virtual way easily besides reducing cost for real task and able to solve many issues which cannot be remedied in the real world.
The level of immersion for both elements of virtual and real objects in AR application refers to the merging of real and virtual worlds to produce AR environments and visualizations where real and digital objects coexist and interact in real time (Azuma et al. 2001). According to Ismail and Sunar 2013, a tracking process is very important in developing AR application and in running it in real time. The main requirements for trackers are high accuracy and little latency at a reasonable cost. The tracking of objects in the scene amounts to calculating the pose between the camera and the objects. Virtual objects can then be projected into the scene using the pose.
Augmented Reality Handheld Interface
There are three main fundamentals that can be found: tracking, display technology, and interaction (Billinghurst et al. 2008). Tracking is one of the fundamental parts in enabling technologies in AR, and it still have many problems that are unsolved (Ismail and Sunar 2013). Interaction technique issues in mobile AR and multimodal AR are becoming more popular. In vision-based interaction, hand and fingertip tracking and hand gesture recognition method are widely used to provide an easy way to interact with virtual object in AR (Chun and Lee 2012). A real-time vision-based approach was introduced to manipulate the overlaid virtual objects dynamically in a marker-less AR system using bare hand with a single camera (Cohen et al. 1989). It is natural that the collision between the human hand and the augmented object can occur during manipulation of the virtual 3D object. In AR, however, the collision happened between a virtual object and a real object; thus, the collision detection approach may be different compared with the ways in the real world. Most AR-handheld applications are not applying a natural interaction, and the user interactions mostly are using touch-based (Kim and Lee 2016). Therefore, this entry describes the interaction in an interactive pop-up book with natural gesture interaction using real hand in handheld interface.
The existing AR book which is generally known as the magic book contains 3D virtual and animated content registered on real book pages, mimicking a traditional “pop-up book” (Markouzis and Fessakis 2015). AR pop-up book is a book that involves the process of overlaying a virtual content onto the pages of a physical pop-up book. The current existing AR book that uses similar metaphor is MagicBook (Billinghurst et al. 2001). MagicBook offers the user the ability to experience the full reality-virtuality continuum. This is because the MagicBook itself is capable of changing the mode between AR and VR. Through AR display, the user is able to see a scene alike, and they could change the view mode to an immersive virtual environment. Another application that adopted the AR book metaphor is AR coloring book (Clark et al. 2011). The AR coloring book aims at augmenting an educational coloring book with user-generated AR virtual content.
Handheld interfaces have four interaction techniques that have been recently explored: touch-based interaction (Kim and Lee 2016), midair gesture-based interaction (Vuibert et al. 2015), device-based interaction (Samini and Palmerius 2016), and direct interaction (Hilliges et al. 2018). The traditional touch-based interaction methods for handheld AR cannot provide intuitive 3D interaction due to a lack of natural gesture input with real-time depth information (as agreed by Bai et al. 2013). Therefore, this entry aims to illustrate the design of natural interaction techniques in 3D spaces by handheld AR devices. Positions and movements of the user’s fingertips are corresponding to the manipulations of the virtual objects in the AR scene (as recommended in Bai et al. 2013).
Augmented Reality Pop-Up Book
There are three phases carried out to develop AR pop-up book that are described in the following subsections.
Phase 1: Defining Interactivity and Storytelling for AR Pop-Up Book
The interactivity for an interactive book happens when it contains story and activities which required the user to perform and interact. The real pop-up book does offer a lot of advantages, but in the transformation to more digital and interactive experience, the book offers a lot more than just a pile of heavy paper. Digital books recently have been widely restructured and recycled, yet it enhances the reading experience and more interactive than the conventional printed books. The main advantages of a digital book are that it can be customized to meet the reader’s prospect (Markouzis and Fessakis 2015). This phase is conducted to design and construct the 3D contents for AR pop-up book. The 3D object built with animation is developed during this phase since the physical pop-up book does not in a digital mode. It was a fully printed copy.
An interactive storytelling enables user to take part and affects the plot of the story, creating a new genre of narrations that is much more engaging and adaptive. Several levels of interactive storytelling start from a simple branching plot to fully dynamic narration models. Interactive storytelling constitutes a new genre of literature which promises considerable learning effectiveness. This stage also defined that the appropriate 3D animation could be applied on the virtual object so the visual is more appealing and interesting. However, the storytelling has been chosen based on the current available conventional pop-up book which is entitled Beauty and the Beast. The physical fairytale pop-up book is being used to provide pop-up book with the storytelling. Therefore, we were transforming the existing format for real pop-up book into AR transitional and tangible in order to measure the AR experience.
Phase 2: Setting Up AR-Handheld User Interface
As illustrated in the diagram, it can be seen the hardware configuration. In order to overlay the virtual element on the top of real environment, the data of 3D object are loaded binding with 2D textures. In order to display the AR interface, handheld device is chosen as AR display technology. The standard vision-based tracking system works to recognize the input, marker, and user hand. It recognizes the registered marker before it was loaded with the appropriate 3D object onto the scene. The user’s hand required to be captured by the leap motion device (Guna et al. 2014). User interacts with the AR environment by using their bare hand as an interaction tool. The application is able to recognize user’s one hand to interact with the virtual object, and the other hand holds the handheld device. Users can interact with the virtual animation by performing a define gesture that is recognized by the system.
Phase 3: Pop-Up Book Feature Tracking
This phase is conducted to design and construct the 3D contents for AR pop-up book. The 3D object built with animation is developed during this phase since the physical pop-up book does not in a digital mode. It was a fully printed copy. The phase is carried out that includes determining the display technique, tracking technique, and interaction method.
The main challenge in AR pop-up book application is to ensure the registrations and hand tracking problem are effectively solved. AR pop-up book utilizes the benefit of hand gesture recognition technique as an interaction tool in the AR environment. The tracking library is used to track the page of the pop-up book that utilizes a feature-based tracking technique.
Phase 4: Developing Hand Gesture Interaction
This phase focuses on exploring the gesture interaction for the user to interact with AR pop-up book. The study of the pop-up book concept and its interactivity processes has been carried out in Phase 3. In order to enhance the realism in AR environment for conventional pop-up book, we merge the AR pop-up book with the live character, and the story elements of the pop-up book come alive. The character will follow user’s hand movement, and the story elements will activate the animation effects once user’s hands touch them. To actualize this realism effects, user interaction is crucial to precisely hit the characters. To look more natural, the user can use their bare hands to directly contact with the virtual elements.
Natural Gesture Interaction
This section explains on natural gesture interaction which was divided into the following process.
Phase 1: Acquiring Gesture Inputs
There are three gesture inputs that have been defined such as TouchGesture, SwipeGesture, and CircleGesture. TouchGesture represents a virtual object that will call an appropriate animation as a feedback once it is being touched. SwipeGesture represents a virtual object that is being swiped, while CircleGesture is being retrieved and updated whenever user performed a circling gesture at designated position in the AR environment and call appropriate animation.
The SwipeGesture is a gesture input where the user swipes their index finger to interact with the virtual object of the AR environment. The gesture is defined in this particular project by calculating the velocity and speed of the tip of the index finger and the collision detection between the finger and the virtual object that can be interacted with.
The TouchGesture is a gesture input where the user touches the virtual object by using their index finger to enable certain features in the AR environment and interact with the virtual object. The gesture is defined in this particular project by making collision detection whenever the tip of the index finger collider of the user collides with the interactable virtual object in the AR environment.
Phase 2: Integrating Gesture with Handheld AR
Through the PUN network protocol that is being implemented in this stage, the user hand tracking data (position and rotation) from the real world is being sent by desktop (sender) to the handheld device (client or receiver). Photon network always uses a master server and one or more game servers. The master server manages the currently available games and does matchmaking. Once a room is found or created, the actual gameplay is done on a game server. All servers run on dedicated machines.
Phase 3: Executing Gesture Interaction in AR Pop-Up Book
Problems and Restriction
The AR pop-up book is demonstrated in this entry as an interactive AR environment that enables users to play with the storytelling. The gesture interaction provides the user to directly interact with the virtual objects. The user feels more realistic to interact with 3D objects using their bare hands, and the realism of the 3D objects appears on the top of the pop-up book in the real world. Hence, there are several problems that arise regarding the real-time 3D gesture sensing in AR pop-up book. The first problem is the accuracy of the hand’s detection because when the hands move into a certain position, it loses the accuracy. Accuracy in tracking is vital to ensure intuitive user interaction with the virtual elements (Lv et al. 2015). The second problem is that the user feels detached from the AR environment because of the indirect interaction method. However, these problems still persist especially when involving the precision of the hand’s detection which can cause problem in the performance. It is natural that collision between the human hand and the augmented object can occur when manipulating the virtual 3D object. In AR, however, the collision happened between a virtual object and a real object; thus, the collision detection approach may be different compared with the ways in the real world. In the user’s observation, with a handheld, the screens are often restricted and sometimes can be rotated between portrait and landscape. Handheld is small enough to hold and operate in the hand; nevertheless the user cannot use their both hands since the other hand needs to hold the device.
Based on the development stages described in the previous section, the standard guidelines emphasize on developing the handheld AR interface for AR pop-up book application that applies natural gesture interaction instead of touchscreen. The AR pop-up book development is explained in this entry but does not study the education pedagogy and the development stresses on AR technology to bring the physical book into more appealing and interesting handheld AR application. On the physical book, the virtual environment was overlaid in real time. The study on education purposes can be further explored the potentials and future work. Also, more future work in user’s interaction for usability aspect can be carried out such as invoking the multimodal interaction that may bring AR pop-up book to be more interactive when speech input complements the gesture. Multimodal interaction is seen to advance interaction technique in AR which can improve user’s experience in AR (Ismail and Sunar 2014; Piumsomboon et al. 2014). Handheld AR has been widely used with smart and portable device in the applications such as education, games, visual experience, and information visualization. However, most of the handheld applications have used touch-based to interact. Subsequently, the real hand gesture tracking in handheld AR is explored to examine how it tracks user’s hands in real time. This entry describes the gesture interaction that allows the user to directly interact with the virtual objects. Thus, the user feels more realistic to interact with 3D objects using their bare hands.
- Azuma, R., Behringer, R., Feiner, S., Julier, S., Macintyre, B.: Recent advances. In EEE Computer Graphics and Applications, 2011(December), 1–27 (2001)Google Scholar
- Bai, H., Gao, L., El-Sana, J., Billinghurst, M.: Markerless 3D gesture-based interaction for handheld augmented reality interfaces. In Mixed and Augmented Reality (ISMAR), 2013 IEEE International Symposium on, pp. 1–6. IEEE (2013)Google Scholar
- Billinghurst, M., Kato, H., Poupyrev, I. Tangible Augmented Reality. ACM SIGGRAPH ASIA 2008 Courses, 7, pp. 1–10 (2008)Google Scholar
- Chun, J., Lee, S.: A vision-based 3D hand interaction for marker-based AR. Int J Multimed Ubiquit Eng. 7(3), 51–58 (2012)Google Scholar
- Clark, A., Dünser, A., Grasset, R.: An interactive augmented reality coloring book. In: Mixed and Augmented Reality (ISMAR), 2011 10th IEEE International Symposium on, pp. 259–260. IEEE (2011)Google Scholar
- Cohen, P.R., Dalrymple, M., Moran, D.B., Pereira, F.C., Sullivan, J.W., Cohen, P.R., Sullivan, J.W.: Synergistic use of direct manipulation and natural language. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems Wings for the Mind – CHI ’89, vol. 20, pp. 227–233. ACM Press, New York (1989)CrossRefGoogle Scholar
- Hilliges, O., Kim D., Izadi S., Molyneaux D., Hodges S.E., Butler D.A.: Augmented reality with direct user interaction. U.S. Patent 9,891,704, issued February 13 (2018)Google Scholar
- Ismail, A.W., Sunar, M.S.: Intuitiveness 3D objects interaction in augmented reality using S-PI algorithm. Indones J Electr Eng Comput Sci. 11(7), 3561–3567 (2013)Google Scholar
- Ismail, A.W., Sunar, M.S.: Multimodal fusion: gesture and speech input in augmented reality environment. In: Computational Intelligence in Information Systems: Proceedings of the Fourth INNS Symposia Series on Computational Intelligence in Information Systems (INNS-CIIS 2014), vol. 331, p. 245. Springer, Cham (2014)CrossRefGoogle Scholar
- Markouzis, D., & Fessakis, G.: Interactive storytelling and mobile augmented reality applications for learning and entertainment – a rapid prototyping perspective. In: Interactive Mobile Communication Technologies and Learning (IMCL), 2015 International Conference on, pp. 4–8. IEEE (2015)Google Scholar
- Network, P.U.: How to Create an Online Multiplayer Game with Photon Unity Networking (2015)Google Scholar
- Piumsomboon, T., Altimira, D., Kim, H., Clark, A., Lee, G., Billinghurst, M.: Grasp-Shell vs gesture-speech: a comparison of direct and indirect natural interaction techniques in augmented reality. In ISMAR 2014 – IEEE International Symposium on Mixed and Augmented Reality – Science and Technology 2014, Proceedings, pp. 73–82 (2014)Google Scholar
- Samini, A., Palmerius, K.L.: A study on improving close and distant device movement pose manipulation for hand-held augmented reality. In The 22nd ACM Symposium on Virtual Reality Software and Technology (VRST), Munich, Germany, November 02-04, 2016 (pp. 121–128). ACM Press (2016)Google Scholar
- Vuibert, V., Stuerzlinger, W., Cooperstock, J.R.: Evaluation of docking task performance using mid-air interaction techniques. In: Proceedings of the 3rd ACM Symposium on Spatial User Interaction (pp. 44–52). ACM (2015)Google Scholar