Abstract
New technologies, especially mixed reality devices, such as Microsoft HoloLens, are blurring the difference between the real and virtual worlds. These developments provide an exciting opportunity to re-define virtual library while maintaining utility and effectiveness. Developing a mixed library system requires identifying the set of services to be supported by the system and identifying well-defined service protocols. We describe a mixed reality based system, a prototype mixed library, that provides a variety of affordances to support embodied interactions and improve the user experience. By leveraging embodiment and context awareness, a mixed reality based user interface can provide many ways to aid a user. We use Microsoft HoloLens device to augment the user’s experience in the real library and to provide a rich set of affordances for embodied and social interactions. In addition, the user can also access services of the virtual library when not in the real library.
Keywords
1 Introduction
Virtual Library is traditionally defined as a searchable collection of online books, journals and articles available on the Internet. In essence, the traditional notion of a library as a structured collection of printed materials is more or less directly mapped to online or digital medium. However, modern libraries are more than that. They are places of social interactions, active learning and new services (e.g., 3D printing). At the same time, new technologies, especially augmented reality (AR) or mixed reality (MR) devices such as Microsoft HoloLens [4] are blurring the difference between the real and virtual worlds. These developments provide an exciting opportunity to re-define virtual library while maintaining utility and effectiveness after the initial excitement and ‘coolness’ expires. Instead of distinguishing between the real and virtual library let us consider creating a ‘mixed library’.
Rather than only looking from outside in, i.e., how to replicate a traditional library in a virtual space, we should also look from inside out, how to expand the traditional library and the corresponding physical space (built environment) with virtual artifacts. One of the possibilities is to create a virtual environment that replicates, perhaps partially, the real environment of a library. The first step in that direction is establishing a connection between real and virtual worlds. That includes marking/tagging physical artifacts, tracking people, etc. However, the challenge is how to make this more than an architectural walkthrough in a built environment to support information and service exploration as well as social interaction. This challenge can be addressed by leveraging MR technology to bridge and blur the differences between real and virtual library and functions in both worlds while maintaining the user’s embodiment.
Embodied interactions demonstrate the importance of the body’s interactions with the physical world. Interaction of our body and the surrounding physical world affect our cognitive processes and embodied cognition [20]. Embodiment cognition leverages the notion of affordances, potential interactions with the environment, to support cognitive processes [8].
MR is a mixture of both real and virtual realities where the user can see both the real world and virtual artifacts. The use of MR is well suited for built environments because the user interface can enhance the built environment with virtual constructs. For example, if a user is visiting the library for the first time, the user may need directions to the reserve desk. A user interface based on MR could provide these directions by drawing virtual arrows directing the user to their desired location. In this case, the built environment is using its context awareness to determine if the user is new to the environment and is leveraging the user’s embodiment to direct them to their desired location.
By leveraging embodiment and context awareness, a user interface for a built environment can provide many affordances to aid the user. By allowing the user to define a context for a specific service or task, the system could use that context to help the user when performing similar tasks.
Therefore, in a real library, MR devices expand the real world with virtual artifacts and information to help better use the services. In a virtual library, physical surrounding serves as a framework to attach information. A student accessing the library from a dorm room can create a model of 3D object and send it directly to 3D printer in the library. Those, perhaps oversimplified, examples of a mixed library provide an insight into exciting opportunities that could be provided by a mixed library. A support for multimodal interactions can provide additional affordances. For example, a voice-recognition-based user interface can be used to search for a book and then guide the user where to pick the book (voice or visual display).
MR systems inherently depend on the surrounding space to support user interactions. Creating an actual mixed library is a tremendous task but a pilot study can provide guidelines and directions how to design and implement a mixed library. We describe an MR system, a prototype mixed library, that provides a variety of affordances to support embodied interactions and improve the user experience. We use Microsoft HoloLens device to augment the user experience in the real library and to provide a rich set of affordances for embodied and social interactions.
Figure 1 shows illustrative snapshots of the MR user interface taken at the Virginia Tech’s library (Fig. 1 center). Figure 1 top left and top right illustrate how to provide a representation of a user accessing the mixed library and its services. Figure 1 bottom left and bottom right illustrate opportunities for situating virtual collaboration in the physical library.
The collaboration with the library staff and researchers is essential to develop the prototype MR application and supporting infrastructure. This collaboration is a translational innovation that can be further refined for specific educational and research areas requirements through the involvement of the faculty. The outcome is likely to improve user experience, reduce cognitive load and enhance interactions among the users.
2 Related Work
Virtual reality (VR) and AR/MR technologies [3] technologies/applications are becoming more affordable and thus more accessible to general users. While there is over forty years of research in this area, we still need more findings to better understand challenges when it comes to developing MR applications.
There are several AR tools that enable developers to design AR applications and experience. Two examples include ARToolKit [6] and Vuforia [18]. ARToolKit is an open-source SDK mainly focused on implementing augmented reality applications. Vuforia SDK provides recognition and tracking capabilities can be used on a variety of images and objects, including support for a fully customizable marker that can be placed on any surface.
Although AR tools can provide general tools for reproducing virtual contents in the real world but these tools can not provide all the needs for educational purposes. Also, research on AR mostly about technology aspects and few papers focus on using AR for education [21].
Cubillo et al. [5] describe a learning environment based on augmented reality, ARLE (An Augmented Reality Learning Environment). ARLE can be used by both the teacher to develop AR educational resources and students to acquire knowledge in the area. The main goal is to provide a user-friendly, AR environment to create educational content without requiring programming skills. ARLE incorporates multimedia resources, such as video and audio, and supports personalized content by including additional information. Users have access to the library of virtual contents, which all resources are available to all users in a simple and clear way. They found a significant learning improvement for users who used ARLE compared to the users who did not use ARLE. In other words, AR application for educational purposes can improve student performance and also help teachers to provide better and more engaging content to study.
Underwood and Kimmel [19] conducted a study of the use of MR simulation to introduce social justice and understand culturally relevant pedagogy to pre-service school librarians. In their study, each candidate had a book talk assignment. They had to chose a book from one of the multicultural selected categories. Instead of candidates selecting their audience for their book talking assignment, they asked to give their talk in a teaching simulation to a group of five avatars in MR. The avatars were controlled remotely in real time by actors and they were modeled in a way to introduce cultural diversity. The study tried to answer two questions.
The first question is what are candidates perceptions of the multicultural book talking before and after using MR-based simulation. The study shows that participants focused more on the literature and did not pay attention to the book talking process. This lead to the conclusion that they characterized avatars as the students that are disrespectful and disruptive. The major challenge for the participants was they could not go through their list of books in the presence of an active avatar. They blamed the technology for the difficulty that they faced.
The second question is what is the efficacy of MR-based simulation for teaching culturally relevant pedagogy. Nearly all of the participants reported awareness for the need of multicultural resources for school library collections. Although participants critiqued the animation of the avatars as misbehaving and distracting, but it provided unique experience about culturally relevant pedagogy.
Hantono et al. [11] describe the use of an AR agent in education. The agent functions continuously and autonomously on behalf of a user and works and learns from its experience over time.
Providing an agent in AR applications can attract user’s perception of the state in both virtual and real world [2]. Oh and Byun [17] presented an interactive agent in the AR learning system. The agent generates puzzles based on the user’s action. To evaluate the system, the authors compared “help-to-bloom” task in the AR application with and without the agent. User satisfaction was measured using a questionnaire after the experiment. The result shows that having an animated agent in AR learning environment will increase user’s satisfaction significantly.
The use of collaborative AR system is another area that has a potential for the learning environment. Matcha et al. [14] explored the potential of AR spaces to provide communication cues and to support collaboration in the learning environment. They implemented an AR prototype system called AReX (Augmented Reality Experiment). The goal was to help students learn the concept of light dispersion and combination. The study shows that AR supports “various communication cues” and “the AR system shows a significant support for collaborative learning environment” [14].
Some of the challenges in creating AR systems were explored by Dunleavy et al. [9]. They studied the limitation and affordances of an AR system. The results showed that although using AR system could significantly increase student engagement, there are still hardware and software issues. For example, the GPS failure rate was between 15–30%. Because this study took place in an outdoor environment, the displayed information was difficult to read on bright sunny days due to the glare. Another problem was the participant’s cognitive overload. They were overwhelmed with the amount of material and complexity of the tasks they were asked to process.
AR/MR technologies are used to reconstruct historical buildings and monument and preserve historical data [16]. Such virtual heritage can be used in education as a platform for learning, motivating and understanding of historical events and locations.
When the physical size of the physical environment is large, navigating the corresponding virtual environment could be challenging. User interaction techniques for navigation task should also support users’ spatial learning performance more effectively and conveniently [7, 12, 13].
3 Mixed Library
Leveraging MR technologies to support common mixed-library user tasks can enhance the user experience. One such task is exploring a digital index of the available books before moving to some aisle, following the signs and instructions, to fetch the book of choice. MR technologies can provide a more convenient way to guide the user to where the book is located. A virtual librarian in the form of a life-size human avatar (Fig. 1 top right) can walk the user to the book location before pointing at it, just the same way an actual librarian would do. This relieves the user from looking for signs or finding a librarian to ask and provides a more natural way of guiding someone to somewhere. Avatars can be created as many as needed and they can take over the simple tasks to help librarians assisting the users.
Given the initial location of a user, an MR device can track the user’s location and orientation. This information can be used to move the avatar in the physical space with respect to user’s location, where the avatar can speed up or slow down according to the user’s movement. It may also call or walk back to the user if the user got lost or went to the wrong direction. For added convenience, the avatar may give the user a short note about the book during their way to book location.
Microsoft HoloLens device support for shared holograms allows for sharing the same avatar between two or more users. This allows for a shared experience, which might be needed for collaborative tasks. For instance, an avatar can give a library tour for a group of users.
A library can provide a variety of services (e.g. borrowing books, printing, scanning, etc.). Each of those services should have a well-defined protocol that consists of a set of steps to be taken by the user in order to receive that service. Initially, the user should be informed about the available services before selecting the service of interest. Afterwards, an avatar should act as a dedicated librarian to the user and provide guidance according to the service protocol. The protocol may contain different types of steps ranging from obtaining permission to use the service (e.g. login, reservation, payment) to technical instructions on how to use that service (e.g. operating a 3D printer). Following the service protocol, an avatar should guide the user by showing when, where and how to perform a given step.
Developing a mixed library system requires identifying the set of services to be supported by the system and identifying well-defined service protocols. Services should be categorized as either shareable or mutual exclusion services. If the library has multiple instances of a given service (e.g. multiple printers), the system should take that into account and perform load balancing by having the avatar recommending the instance with the shortest waiting line.
3.1 System Architecture
Figure 2 shows the system incorporates four basic modules, a service portal, a user tracker, an avatar controller, and a resource scheduler. The service portal module provides the user with an interface through which the user can request or cancel a previously requested service. The user tracker is responsible for determining the user’s position and orientation. The tracking information is used by the avatar controller to adjust the avatar’s movement and actions. The avatar controller is responsible for generating and controlling avatars. An avatar can be dedicated to an individual user or shared among a group of users.
Each service should have a protocol to be followed in order to consume that service. As shown in Fig. 3 left, a service protocol incorporates a sequence of actions that involve the use of different resources. Some of these resources are shareable (e.g., an elevator) and some are not (e.g., a printer). Each resource type can have one or more instances, as shown in Fig. 3 right, that can fulfil the same service (e.g., several copies of the same book). The scheduler module is responsible for turning the service protocol, which is an abstract description of the subsequent use of resources, into a procedure that specifies the resource instances to be used. The selection of the resource instances depend on common factors such as load balancing. Moreover, the scheduler can benefit from context information to better select resource instances. Location information of both the user and the different instances of a given resource can help the scheduler selecting the instance that is nearest to the user. Selecting a resource instance is an optimization problem that should take several factors into account including load balancing, service time, and user’s location.
4 A Mixed-Library Prototype
We are developing a mixed library prototype to evaluate the proposed mixed-library system architecture. Figure 4 shows an example of a service portal through which the user can request a service. After requesting a service, the avatar controller will generate an avatar, a virtual librarian, at the user’s current location. Following the service protocol, the virtual librarian controller communicates with the scheduler module upon the execution of each protocol step to retrieve information about which resource instance to target.
The virtual librarian can guide the user to the location of the instance selected by the scheduler. Afterwards, the avatar may provide the user with instructions on how to use the resource. For simple tasks, those instructions can be verbal or textual (e.g., asking the user to wait in a line). For more advanced tasks (e.g., 3D printing), the avatar may perform a simulation of the task or play a video illustrating how to perform that task as shown in Fig. 5.
Common library services that used to be available through a web portal can be supported in MR as well such as reading a book or watching a video. Moreover, books may be augmented with holograms, sound recordings, and videos that can take the reading experience to the next level (e.g., turning a book figure into a 3D model, displaying animal holograms while reading a kids story).
Supporting collaboration can be achieved by allowing two or more users to share the same hologram (e.g. avatar, digital book). The system not only can support collaboration but also it can suggest collaboration. Using user location information as well as learned interests, the system may recommend joining a reading group with similar interests. In order to preserve user privacy, the user can control whether to enable this feature so that other users can know about possible collaboration chances.
5 Evaluation
Some of the advantages of the proposed system include virtuality (adding virtual objects to the real world), augmentation (augmenting real objects by virtual annotations), cooperation (multiple users can collaborate), independence (a user has a separate viewpoint), and individuality (provided service, resource, and information are user dependent). However, there are challenges, such as occlusion and depth perception, visual differences between real and virtual objects tracking, and using head mounted devices. We need to apply both quantitative and qualitative measures to assess the developed system while taking into account these advantages and challenges.
The resource scheduler is at the core of the system as it manages the allocation of library resources trying to optimize the resource utilization [15] and quality of service [10]. The quality of service can be measured in terms of service time and user frustration. We plan to conduct a comparative user study to evaluate the proposed system compared to asking a librarian for help or following instructions and signs. Besides questionnaires about the usability of the system, biometric measures can be used as indicators of the level of user frustration. Librarian frustration should be taken into account as well since the system aims at both helping the users and relieving librarians from handling frequent simple tasks.
MR applications offer the opportunity to teach and learn in an augmented environment. We need to explore the ability to unify input from different sources to enhance motivation and engagement (increase interest level), effective learning (increase receptiveness) and cognitive learning (increase analytical ability).
As a part of a related project, we have developed an infrastructure to record biometric data about the user (Fig. 6). The infrastructure incorporates three devices connected to a data collection PC as well as a video recording PC. The video recording PC can capture what the user sees during the study including both physical and virtual objects. The data collection PC collects time-stamped data from the EEG, Empatica, and HoloLens devices.
The EEG device senses the electrical activity of the brain at different spots as well as the 3-axis acceleration of the user’s head. The Empatica device is a wristband that senses different types of data including 3-axis acceleration of the user’s hand, blood volume pulse, galvanic skin response, skin temperature, and heart rate. The HoloLens device records timestamped events for synchronization purposes with the data collected from the other two devices. This variety of collected biometric data can be used as indicators for assessing user frustration when evaluating the mixed library system.
6 Conclusion
As stated in Introduction, creating an actual mixed library is a tremendous task. Our findings and a prototype implementation using the Microsoft HoloLens device demonstrated the feasibility of the proposed concept. By leveraging embodiment and context awareness, the developed system can provide many ways to aid a user. Using a service based approach will facilitate gradual expansion of the mixed library system and its functionality.
We plan to conduct a summative evaluation to compare relevant computer-based, VR-based, and MR-based library applications. The analysis of the summative study data will be used to validate functional and non-functional requirements identified in the focus group interviews and the pilot study. The task performance will be compared with the baseline established in the pilot study. The collaboration with the library staff and researchers will inform future refinements and development.
References
16x9onglobal: 3D printing: Make anything you want (2017). https://www.youtube.com/watch?v=G0EJmBoLq-g. Last accessed 10 Feb 2017
Barakonyi, I., Schmalstieg, D.: Augmented reality agents for user interface adaptation. Comput. Anim. Virtual Worlds 19(1), 23–35 (2008)
Bimber, O., Raskar, R.: Spatial Augmented Reality: Merging Real and Virtual Worlds. A K Peters, Wellesley (2005)
Corporation, M.: Microsoft HoloLens. https://www.microsoft.com/microsoft-hololens/ (2017). Last accessed 10 Feb 2017
Cubillo, J., Martín, S., Castro, M., Diaz, G., Colmenar, A., Botički, I.: A learning environment for augmented reality mobile learning. In: 2014 IEEE Frontiers in Education Conference (FIE) Proceedings, pp. 1–8, October 2014
DAQRI: ARToolkit. http://www.artoolkit.org (2016). Last accessed 10 Feb 2016
Davis, M.M., Gabbard, J.L., Bowman, D.A., Gracanin, D.: Depth-based 3D gesture multi-level radial menu for virtual object manipulation. In: Proceedings of the 2016 IEEE Virtual Reality Conference, pp. 169–170 (2016)
Dourish, P.: Where the Action Is: The Foundations of Embodied Interaction. The MIT Press, Cambridge (2001)
Dunleavy, M., Dede, C., Mitchell, R.: Affordances and limitations of immersive participatory augmented reality simulations for teaching and learning. J. Sci. Educ. Technol. 18(1), 7–22 (2009)
Gračanin, D., Zhou, Y., DaSilva, L.: Quality of service for networked virtual environments. IEEE Commun. Mag. 42(4), 42–48 (2004)
Hantono, B.S., Nugroho, L.E., Santosa, P.I.: Review of augmented reality agent in education. In: Proceedings of the 6th International Annual Engineering Seminar (InAES), pp. 150–153, August 2016
Kim, J.S., Gračanin, D., Quek, F.: Sensor-fusion walking-in-place interaction technique using mobile devices. In: Proceedings of the IEEE Virtual Reality Conference (VR), pp. 39–42, 5–8 March 2012
Kim, J.S., Gračanin, D., Yang, T., Quek, F.: Action-transferred design approach for navigation techniques in 3D virtual environments. ACM Trans. Comput.-Human Interact. 22(6), 30:1–30:42 (2015)
Matcha, W., Awang Rambli, D.R.: Preliminary investigation on the use of augmented reality in collaborative learning. In: Abd Manaf, A., Sahibuddin, S., Ahmad, R., Mohd Daud, S., El-Qawasmeh, E. (eds.) ICIEIS 2011. CCIS, vol. 254, pp. 189–198. Springer, Heidelberg (2011). doi:10.1007/978-3-642-25483-3_15
Mulligan, G., Gračanin, D.: A comparison of SOAP and REST implementations of a service based interaction independence middleware framework. In: Rossettii, M., Hill, R., Johansson, B., Dunkin, A., Ingalls, R. (eds.) Proceedings of the Winter Simulation Conference, pp. 1423–1432, 13–16 December 2009
Noh, Z., Sunar, M.S., Pan, Z.: A review on augmented reality for virtual heritage system. In: Chang, M., Kuo, R., Kinshuk, Chen, G.-D., Hirose, M. (eds.) Edutainment 2009. LNCS, vol. 5670, pp. 50–61. Springer, Heidelberg (2009). doi:10.1007/978-3-642-03364-3_7
Oh, S., Byun, Y.C.: The design and implementation of augmented reality learning systems. In: Proceedings of the 2012 IEEE/ACIS 11th International Conference on Computer and Information Science, pp. 651–654. IEEE Computer Society, Washington, DC (2012)
PTC Inc.: Vuforia. http://www.vuforia.com (2017). Last accessed 10 Feb 2017
Underwood, J., Kimmel, S., Forest, D., Dickinson, G.: Culturally relevant booktalking: using a mixed reality simulation with preservice school librarians. Sch. Libr. Worldwide 21(1), 91–107 (2015)
Wilson, M.: Six views of embodied cognition. Psychon. Bull. Rev. 9(4), 625–636 (2002)
Zhou, F., Duh, H.B.L., Billinghurst, M.: Trends in augmented reality tracking, interaction and display: a review of ten years of ISMAR. In: Proceedings of the 7th IEEE/ACM International Symposium on Mixed and Augmented Reality, pp. 193–202, September 2008
Acknowledgments
This work was supported in part by a grant from Virginia Tech Institute for Creativity, Arts, and Technology (ICAT).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2017 Springer International Publishing AG
About this paper
Cite this paper
Gračanin, D., Ciambrone, A., Tasooji, R., Handosa, M. (2017). Mixed Library — Bridging Real and Virtual Libraries. In: Lackey, S., Chen, J. (eds) Virtual, Augmented and Mixed Reality. VAMR 2017. Lecture Notes in Computer Science(), vol 10280. Springer, Cham. https://doi.org/10.1007/978-3-319-57987-0_18
Download citation
DOI: https://doi.org/10.1007/978-3-319-57987-0_18
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-57986-3
Online ISBN: 978-3-319-57987-0
eBook Packages: Computer ScienceComputer Science (R0)