Skip to main content
SpringerLink
Log in

Towards Advanced Evaluation of Collaborative XR Spaces

  • Conference paper
  • First Online:
Sense, Feel, Design (INTERACT 2021)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13198))

Included in the following conference series:

  • 937 Accesses

Abstract

Extended Reality (XR) technologies such as head-mounted displays are deemed beneficial for the collaboration of co-located as well as distributed people. As such, XR technologies appear particularly promising for supporting distant and hybrid teaching which became highly relevant during the Covid-19 pandemic. Despite the potential awarded to such technologies, practical applications are still very rare. In order to investigate the impediments to the practical adoption of XR technologies, the respective systems should be evaluated in real-world settings. Existing evaluation tools are, however, not suited for this purpose. In this paper, we explain why today’s evaluation tools such as questionnaires, observation, and performance measurements are not sufficient for evaluating long-time, exploratory, and collaborative tasks that are typical in educational settings. To address this gap, we follow a top-down approach: Based on an existing model of user acceptance, we specify the variables that are to be optimized by HCI research and outline the potential of wearable-based measuring instruments to quantitatively assess these parameters. Eventually, we point out related research gaps that should be addressed by future research.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Beatty, J.: Task-evoked pupillary responses, processing load, and the structure of processing resources. Psychol. Bull. 91(2), 276–292 (1982). https://doi.org/10.1037/0033-2909.91.2.276

    Article  Google Scholar 

  2. Bian, Y., et al.: A framework for physiological indicators of flow in VR games: construction and preliminary evaluation. Pers. Ubiquit. Comput. 20(5), 821–832 (2016). https://doi.org/10.1007/s00779-016-0953-5

    Article  Google Scholar 

  3. Büschel, W., Lehmann, A., Dachselt, R.: MIRIA: a mixed reality toolkit for the in-situ visualization and analysis of spatio-temporal interaction data. In: Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, pp. 1–15. ACM (2021). https://doi.org/10.1145/3411764.3445651

  4. Cranford, K.N., Tiettmeyer, J.M., Chuprinko, B.C., Jordan, S., Grove, N.P.: Measuring load on working memory: the use of heart rate as a means of measuring chemistry students’ cognitive load. J. Chem. Educ. 91(5), 641–647 (2014). https://doi.org/10.1021/ed400576n

    Article  Google Scholar 

  5. Cruz-Neira, C., Sandin, D.J., DeFanti, T.A.: Surround-screen projection-based virtual reality: the design and implementation of the CAVE. In: Proceedings of the 20th Annual Conference on Computer Graphics and Interactive Techniques, pp. 135–142. ACM (1993). doi: https://doi.org/10.1145/166117.166134

  6. Csikszentmihalyi, M.: Flow and the Foundations of Positive Psychology. The Collected Works of Mihaly Csikszentmihalyi. Springer, Dordrecht (2014). https://doi.org/10.1007/978-94-017-9088-8

  7. Davis, F.D.: A technology acceptance model for empirically testing new end-user information systems: theory and results. Ph.D. dissertation, MIT, Cambridge, MA, USA (1986)

    Google Scholar 

  8. Davis, F.D.: On the relationship between HCI and technology acceptance research. In: Zhang, P., Galletta, D. (eds.) Human-Computer Interaction and Management Information Systems: Foundations. Advances in Management Information Systems, vol. 5, pp. 395–401. M.E. Sharpe, Armonk, NY, USA (2006)

    Google Scholar 

  9. Dishaw, M.T., Strong, D.M.: Extending the technology acceptance model with task–technology fit constructs. Inform. Manage. 36(1), 9–21 (1999). https://doi.org/10.1016/S0378-7206(98)00101-3

    Article  Google Scholar 

  10. Duchowski, A.T.: Eye Tracking Methodology. Theory and Practice. 3rd edn. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-57883-5

  11. Duchowski, A.T., et al.: The index of pupillary activity: measuring cognitive Load vis-à-vis task difficulty with pupil oscillation. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, pp. 1–13. ACM (2018). https://doi.org/10.1145/3173574.3173856

  12. Goodhue, D.L., Thompson, R.L.: Task-technology fit and individual performance. MIS Q. 19(2), 213–236 (1995). https://doi.org/10.2307/249689

    Article  Google Scholar 

  13. Hess, E.H., Polt, J.M.: Pupil size in relation to mental activity during simple problem-solving. Science 143(3611), 1190–1192 (1964). https://doi.org/10.1126/science.143.3611.1190

    Article  Google Scholar 

  14. Hillmert, M., Bergmüller, A., Minow, A., Raggatz, J., Böckelmann, I.: Psychophysiologische Beanspruchungskorrelate während kognitiver Belastung. Zentralblatt für Arbeitsmedizin, Arbeitsschutz und Ergonomie 70(4), 149–163 (2020). https://doi.org/10.1007/s40664-020-00384-9

    Article  Google Scholar 

  15. Ishimaru, S., Bukhari, S.S., Heisel, C., Kuhn, J., Dengel, A.: Towards an intelligent textbook: eye gaze based attention extraction on materials for learning and instruction in physics. In: Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing: Adjunct, pp. 1041–1045. ACM (2016). https://doi.org/10.1145/2968219.2968566

  16. Kahneman, D., Beatty, J.: Pupil diameter and load on memory. Science 154(3756), 1583–1585 (1966). https://doi.org/10.1126/science.154.3756.1583

    Article  Google Scholar 

  17. Kloiber, S., et al.: Immersive analysis of user motion in VR applications. Vis. Comput. 36(10–12), 1937–1949 (2020). https://doi.org/10.1007/s00371-020-01942-1

    Article  Google Scholar 

  18. Lindlbauer, D., Feit, A.M., Hilliges, O.: Context-aware online adaptation of mixed reality interfaces. In: Proceedings of the 32nd Annual ACM Symposium on User Interface Software and Technology, pp. 147–160. ACM (2019). https://doi.org/10.1145/3332165.3347945

  19. Marks, S., White, D.: Multi-device collaboration in virtual environments. In: Proceedings of the 2020 4th International Conference on Virtual and Augmented Reality Simulations, pp. 35–38. ACM (2020). https://doi.org/10.1145/3385378.3385381

  20. Marshall, S.P.: The index of cognitive activity: measuring cognitive workload. In: Proceedings of the IEEE 7th Conference on Human Factors and Power Plants, pp. 7–5–7–9. IEEE (2002). https://doi.org/10.1109/HFPP.2002.1042860

  21. Milgram, P., Kishino, F.: A taxonomy of mixed reality visual displays. IEICE Trans. Inf. Syst. 77(12), 1321–1329 (1994)

    Google Scholar 

  22. Milgram, P., Takemura, H., Utsumi, A., Kishino, F.: Augmented reality: a class of displays on the reality-virtuality continuum. In: Das, H. (ed.) Telemanipulator and Telepresence Technologies, vol. 2351, pp. 282–292. SPIE (1995). https://doi.org/10.1117/12.197321

  23. Murdock, B.B.: The serial position effect of free recall. J. Exp. Psychol. 64(5), 482–488 (1962). https://doi.org/10.1037/h0045106

    Article  Google Scholar 

  24. Papakostas, M., Kanal, V., Abujelala, M., Tsiakas, K., Makedon, F.: Physical fatigue detection through emg wearables and subjective user reports – a machine learning approach towards adaptive rehabilitation. In: Proceedings of the 12th ACM International Conference on PErvasive Technologies Related to Assistive Environments, pp. 475–481. ACM (2019). https://doi.org/10.1145/3316782.3322772

  25. Saggio, G., Tombolini, F., Ruggiero, A.: Technology-based complex motor tasks assessment: a 6-dof inertial-based system versus a gold-standard optoelectronic-based one. IEEE Sens. J. 21(2), 1616–1624 (2021). https://doi.org/10.1109/JSEN.2020.3016642

    Article  Google Scholar 

  26. Wells, T., Houben, S.: CollabAR – investigating the mediating role of mobile AR interfaces on co-located group collaboration. In: Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, pp. 1–13. ACM (2020). https://doi.org/10.1145/3313831.3376541

Download references

Acknowledgements

Funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) – 252408385 – IRTG 2057.

Author information

Authors and Affiliations

  1. Human Computer Interaction Lab, Technische Universität Kaiserslautern, 67663, Kaiserslautern, Germany

    Vera Marie Memmesheimer & Achim Ebert

Authors
  1. Vera Marie Memmesheimer
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Achim Ebert
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Vera Marie Memmesheimer .

Editor information

Editors and Affiliations

  1. Polytechnic University of Bari, Bari, Italy

    Carmelo Ardito

  2. University of Bari Aldo Moro, Bari, Italy

    Rosa Lanzilotti

  3. Computer Science Department, University of Pisa, Pisa, Italy

    Alessio Malizia

  4. Reykjavik University, Reykjavik, Iceland

    Marta Larusdottir

  5. University of Cagliari, Caglieri, Italy

    Lucio Davide Spano

  6. Department of Information, University of Minho, Braga, Portugal

    José Campos

  7. Department of Communication, University of Copenhagen, Copenhagen, Denmark

    Morten Hertzum

  8. Fachbereich Informatik, Trier University of Applied Sciences, Trier, Germany

    Tilo Mentler

  9. ITI/Larsys, University of West London, London, UK

    José Abdelnour Nocera

  10. Knowledge Media Institute, Open University, Milton Keynes, UK

    Lara Piccolo

  11. University of Paderborn, Paderborn, Germany

    Stefan Sauer

  12. Faculty of Sciences, Department of Computer Science, Vrije Universiteit, Amsterdam, The Netherlands

    Gerrit van der Veer

Rights and permissions

Reprints and permissions

Copyright information

© 2022 IFIP International Federation for Information Processing

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Memmesheimer, V.M., Ebert, A. (2022). Towards Advanced Evaluation of Collaborative XR Spaces. In: Ardito, C., et al. Sense, Feel, Design. INTERACT 2021. Lecture Notes in Computer Science, vol 13198. Springer, Cham. https://doi.org/10.1007/978-3-030-98388-8_40

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-98388-8_40

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-98387-1

  • Online ISBN: 978-3-030-98388-8

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Societies and partnerships

Search

Navigation