Developing a Symbiotic System for Scientific Information Seeking: The MindSee Project

  • Luciano Gamberini
  • Anna Spagnolli
  • Benjamin Blankertz
  • Samuel Kaski
  • Jonathan Freeman
  • Laura Acqualagna
  • Oswald Barral
  • Maura Bellio
  • Luca Chech
  • Manuel Eugster
  • Eva Ferrari
  • Paolo Negri
  • Valeria Orso
  • Patrik Pluchino
  • Filippo Minelle
  • Bariş Serim
  • Markus Wenzel
  • Giulio Jacucci
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9359)

Abstract

This paper describes an approach for improving the current systems supporting the exploration and research of scientific literature, which generally adopt a query-based information-seeking paradigm. Our approach is to use a symbiotic system paradigm, exploiting central and peripheral physiological data along with eye-tracking data to adapt to users’ ongoing subjective relevance and satisfaction with search results. The system described, along with the interdisciplinary theoretical work underpinning it, could serve as a stepping stone for the development and diffusion of next-generation symbiotic systems, enabling a productive interdependence between humans and machines. After introducing the concept and evidence informing the development of symbiotic systems over a wide range of application domains, we describe the rationale of the MindSee project, emphasizing its BCI component and pinpointing the criteria around which users’ evaluations can gravitate. We conclude by summarizing the main contribution that MindSee is expected to make.

Keywords

Symbiotic system Implicit measures BCI Information seeking User experience 

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Licklider, J.C.R.: Man-computer symbiosis. IRE Trans. on Hum. Factors Electron. 1, 4–11 (1960)CrossRefGoogle Scholar
  2. 2.
    Jacucci, G., Spagnolli, A., Freeman, J., Gamberini, L.: Symbiotic interaction: a critical definition and comparison to other human-computer paradigms. In: Jacucci, G., Gamberini, L., Freeman, J., Spagnolli, A. (eds.) Symbiotic 2014. LNCS, vol. 8820, pp. 3–20. Springer, Heidelberg (2014) Google Scholar
  3. 3.
    Shute, V.J., Zapata-Rivera, D.: Adaptive technologies. In: Handbook of Research on Educational Communications and Technology, pp. 277–294 (2008)Google Scholar
  4. 4.
    Åman, P., Liikkanen, L.A., Jacucci, G., Hinkka, A.: OUTMedia – symbiotic service for music discovery in urban augmented reality. In: Jacucci, G., Gamberini, L., Freeman, J., Spagnolli, A. (eds.) Symbiotic 2014. LNCS, vol. 8820, pp. 61–71. Springer, Heidelberg (2014) Google Scholar
  5. 5.
    Moreno, J.C., et al.: Symbiotic wearable robotic exoskeletons: the concept of the biomot project. In: Jacucci, G., Gamberini, L., Freeman, J., Spagnolli, A. (eds.) Symbiotic 2014. LNCS, vol. 8820, pp. 72–83. Springer, Heidelberg (2014) Google Scholar
  6. 6.
    Pluchino, P., Gamberini, L., Barral, O., Minelle, F.: How semantic processing of words evokes changes in pupil. In: Jacucci, G., Gamberini, L., Freeman, J., Spagnolli, A. (eds.) Symbiotic 2014. LNCS, vol. 8820, pp. 99–112. Springer, Heidelberg (2014) Google Scholar
  7. 7.
    Serim, B.: Querying and display of information: symbiosis in exploratory search interaction scenarios. In: Jacucci, G., Gamberini, L., Freeman, J., Spagnolli, A. (eds.) Symbiotic 2014. LNCS, vol. 8820, pp. 115–120. Springer, Heidelberg (2014) Google Scholar
  8. 8.
    Hore, S., Tyrvainen, L., Pyykko, J., Glowacka, D.: A reinforcement learning approach to query-less image retrieval. In: Jacucci, G., Gamberini, L., Freeman, J., Spagnolli, A. (eds.) Symbiotic 2014. LNCS, vol. 8820, pp. 121–126. Springer, Heidelberg (2014) Google Scholar
  9. 9.
    Barral, O., Jacucci, G.: Applying physiological computing methods to study psychological, affective and motivational relevance. In: Jacucci, G., Gamberini, L., Freeman, J., Spagnolli, A. (eds.) Symbiotic 2014. LNCS, vol. 8820, pp. 35–46. Springer, Heidelberg (2014) Google Scholar
  10. 10.
    Aranyi, G., Kouider, S., Lindsay, A., Prins, H., Ahmed, I., Jacucci, G., Negri, P., Gamberini, L., Pizzi, D., Cavazza, M.: Subliminal cueing of selection behavior in a virtual environment. Presence: Teleoperators Virtual Environ. 23(1), 33–50 (2014)CrossRefGoogle Scholar
  11. 11.
    Barral, O., et al.: Covert persuasive technologies: bringing subliminal cues to human-computer interaction. In: Spagnolli, A., Chittaro, L., Gamberini, L. (eds.) PERSUASIVE 2014. LNCS, vol. 8462, pp. 1–12. Springer, Heidelberg (2014) CrossRefGoogle Scholar
  12. 12.
    Cetnarski, R., Betella, A., Prins, H., Kouider, S., Verschure, P.F.: Subliminal Response Priming in mixed reality : The ecological validity of a classic paradigm of perception. Presence: Teleoperators Virtual Environ. 23(1), 1–17 (2014)CrossRefGoogle Scholar
  13. 13.
    Negri, P., Gamberini, L., Cutini, S.: A review of the research on subliminal techniques for implicit interaction in symbiotic systems. In: Jacucci, G., Gamberini, L., Freeman, J., Spagnolli, A. (eds.) Symbiotic 2014. LNCS, vol. 8820, pp. 47–58. Springer, Heidelberg (2014) Google Scholar
  14. 14.
    Vidal, J.J.: Toward direct brain-computer communication. Annu. Rev. Biophys. Bioeng. 2(1), 157–180 (1973)MathSciNetCrossRefGoogle Scholar
  15. 15.
    Wolpaw, J.R., Wolpaw, E.W.: Brain-computer interfaces: something new under the sun. In: Wolpaw, J.R., Wolpaw, E.W. (eds.) Brain-Computer Interfaces: Principles and Practice, pp. 3–12. Oxford University Press, Oxford (2012)CrossRefGoogle Scholar
  16. 16.
    Birbaumer, N., Ghanayim, N., Hinterberger, T., Iversen, I., Kotchoubey, B., Kübler, A., Perelmouter, J., Taub, E., Flor, H.: A spelling device for the paralysed. Nature 398(6725), 297–298 (1999)CrossRefGoogle Scholar
  17. 17.
    Sellers, E.W., Vaughan, T.M., Wolpaw, J.R.: A brain-computer interface for long-term independent home use. Amyotroph. Lateral Scler. 11, 449–455 (2010)CrossRefGoogle Scholar
  18. 18.
    Wolpaw, J.R., Birbaumer, N., McFarland, D.J., Pfurtscheller, G., Vaughan, T.M.: Brain-computer interfaces for communication and control. Clin. Neurophysiol. 113(6), 767–791 (2002)CrossRefGoogle Scholar
  19. 19.
    Pfurtscheller, G., Allison, B.Z., Brunner, C., Bauernfeind, G., Solis-Escalante, T., Scherer, R., Zander, T.O., Mueller-Putz, G., Neuper, C., Birbaumer, N.: The hybrid BCI. Front. Neurosci. 4 (2010)Google Scholar
  20. 20.
    Blankertz, B., Tangermann, M., Vidaurre, C., Fazli, S., Sannelli, C., Haufe, S., Maeder, C., Ramsey, L., Sturm, I., Curio, G., Müller, K.R.: The Berlin brain-computer interface: non-medical uses of BCI technology. Front. Neurosci. 4 (2010)Google Scholar
  21. 21.
    Brunner, C., Birbaumer, N., Blankertz, B., Guger, C., Kübler, A., Mattia, D., Millán, J.D.R., Miralles, F., Nijholt, A., Opisso, E., Müller-Putz, G.R.: BNCI Horizon 2020: towards a roadmap for the BCI community. Brain-Computer Interfaces (ahead-of-print), 1–10 (2015)Google Scholar
  22. 22.
    Acqualagna, L., Bosse, S., Porbadnigk, A.K., Curio, G., Müller, K.R., Wiegand, T., Blankertz, B.: EEG-based classification of video quality perception using steady state visual evoked potentials (SSVEPs). J. Neural Eng. 12(2), 026012 (2015)CrossRefGoogle Scholar
  23. 23.
    Porbadnigk, A.K., Treder, M.S., Blankertz, B., Antons, J.N., Schleicher, R., Möller, S., Curio, G., Müller, K.R.: Single-trial analysis of the neural correlates of speech quality perception. J. Neural Eng. 10(5), 056003 (2013)CrossRefGoogle Scholar
  24. 24.
    Venthur, B., Blankertz, B., Gugler, M.F., Curio, G.: Novel applications of BCI technology: psychophysiological optimization of working conditions in industry. In: Systems, Man and Cybernetics (SMC 2010), pp. 417–421. IEEE Press, New York (2010)Google Scholar
  25. 25.
    Schultze-Kraft, M., Birman, D., Rusconi, M., Allefeld, C., Görgen, K., Dähne, S., Blankertz, B., Haynes, J.D.: Predicting and interrupting movement intentions with a closed loop. In: Presented at the 21st Annual Meeting of the Organization for Human Brain Mapping, Honolulu, USA (2015)Google Scholar
  26. 26.
    Di Flumeri, G., Borghini, G., Aric, P., Colosimo, A., Pozzi, S., Bonelli, S., Golfetti, A., Kong, W., Babiloni, F.: On the use of cognitive neurometric indexes in aeronautic and air traffic management environments. In: Blankertz, B., Jacucci, G., Gamberini, L., Spagnolli, A., Freeman, J. (eds.) Symbiotic 2015. LNCS, vol. 9359. Springer International Publishing, Cham (2015)CrossRefGoogle Scholar
  27. 27.
    Bos, D.P.O., Reuderink, B., van de Laar, B., Gürkök, H., Mühl, C., Poel, M., Nijholt, A., Heylen, D.: Brain-computer interfacing and games. In: Tan, D., Nijholt, A. (eds.) Brain-Computer Interfaces: Applying our Minds to Human-Computer Interaction, pp. 149–178. Springer, London (2010)Google Scholar
  28. 28.
    Haufe, S., Treder, M.S., Gugler, M.F., Sagebaum, M., Curio, G., Blankertz, B.: EEG potentials predict upcoming emergency brakings during simulated driving. J. Neural Eng. 8(5), 056001 (2011)CrossRefGoogle Scholar
  29. 29.
    Haufe, S., Kim, J.W., Kim, I.H., Sonnleitner, A., Schrauf, M., Curio, G., Blankertz, B.: Electrophysiology-based detection of emergency braking intention in real-world driving. J. Neural Eng. 11(5), 056011 (2014)CrossRefGoogle Scholar
  30. 30.
    Brunner, C., et al.: BNCI horizon 2020 – towards a roadmap for brain/neural computer interaction. In: Stephanidis, C., Antona, M. (eds.) UAHCI 2014, Part I. LNCS, vol. 8513, pp. 475–486. Springer, Heidelberg (2014) Google Scholar
  31. 31.
    White, R.W., Roth, R.A.: Exploratory search: Beyond the Query-response Paradigm. Synthesis Lectures on Information Concepts, Retrieval and Services. Morgan and Claypool, San Francisco (2009)Google Scholar
  32. 32.
    Hinrichs, U., Schmidt, H., Carpendale, S.: EMDialog: Bringing information visualization into the museum. IEEE Trans. Vis. Comput. Graph. 14(6), 1181–1188 (2008)CrossRefGoogle Scholar
  33. 33.
    Jacucci, G., Morrison, A., Richard, G.T., Kleimola, J., Peltonen, P., Parisi, L., Laitinen, T.: Worlds of information: designing for engagement at a public multi-touch display. In: Proceedings of the 28th International Conference on Human Factors in Computing Systems, pp. 2267–2276. ACM (2010)Google Scholar
  34. 34.
    Coutrix, C., Kuikkaniemi, K., Kurvinen, E., Jacucci, G., Avdouevski, I., Mäkelä, R.: Fizzyvis: designing for playful information browsing on a multitouch public display. In: Proceedings of the 2011 Conference on Designing Pleasurable Products and Interfaces, vol. 27. ACM (2011)Google Scholar
  35. 35.
    Hearst, M.: Search user interfaces. Cambridge University Press, Cambridge (2009)CrossRefGoogle Scholar
  36. 36.
    Bates, M.J.: Where should the person stop and the information search interface start? Inf. Process. Manag. 26(5), 575–591 (1990)CrossRefGoogle Scholar
  37. 37.
    Marchionini, G.: Exploratory search: from finding to understanding. Communications of the ACM 49(4), 41–46 (2006)CrossRefGoogle Scholar
  38. 38.
    Teevan, J., Alvarado, C., Ackerman, M.S., Karger, D.R.: The perfect search engine is not enough: a study of orienteering behavior in directed search. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 415–422. ACM (2004)Google Scholar
  39. 39.
    Tombros, A., Sanderson, M.: Advantages of query biased summaries in information retrieval. In: Proceedings of the 21st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 2–10. ACM (1998)Google Scholar
  40. 40.
    Manning, C.D., Raghavan, P., Schütze, H.: Introduction to Information Retrieval, pp. 1–6. Cambridge University Press, Cambridge (2008) CrossRefMATHGoogle Scholar
  41. 41.
    Limbu, D.K., Connor, A., Pears, R., MacDonell, S.: Contextual relevance feedback in web information retrieval. In: Proceedings of the 1st International Conference on Information Interaction in Context, pp. 138–143. ACM (2006)Google Scholar
  42. 42.
    Puolamäki, K., Salojärvi, J., Savia, E., Simola, J., Kaski, S.: Combining eye movements and collaborative filtering for proactive information retrieval. In: Proceedings of the 28th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 146–153. ACM (2005)Google Scholar
  43. 43.
    Kelly, D., Teevan, J.: Implicit feedback for inferring user preference: a bibliography. ACM SIGIR Forum 37(2), 18–28 (2003). ACMCrossRefGoogle Scholar
  44. 44.
    Arapakis, I., Konstas, I., Jose, J.M.: Using facial expressions and peripheral physiological signals as implicit indicators of topical relevance. In: Proceedings of the 17th ACM International Conference on Multimedia, pp. 461–470. ACM (2009)Google Scholar
  45. 45.
    Damasio, A.: Descartes’ error: Emotion, reason and the human brain. Random House, New York (2008)Google Scholar
  46. 46.
    Pfister, H.R., Bhm, G.: The multiplicity of emotions: A framework of emotional functions in decision making. Judgm. Decis. Mak. 3(1), 5–17 (2008)Google Scholar
  47. 47.
    Ajanki, A., Hardoon, D.R., Kaski, S., Puolamki, K., Shawe-Taylor, J.: Can eyes reveal interest? Implicit queries from gaze patterns. User Model User-Adapt Interact. 19(4), 307–339 (2009)CrossRefGoogle Scholar
  48. 48.
    Gwizdka, J.: Distribution of cognitive load in web search. J. Am. Soc. Inform. Sci. Technol. 61(11), 2167–2187 (2010)CrossRefGoogle Scholar
  49. 49.
    Gwizdka, J., Cole, M.J.: Inferring Cognitive States from Multimodal Measures in Information Science (2011)Google Scholar
  50. 50.
    Eugster, M.J., Ruotsalo, T., Spapé, M.M., Kosunen, I., Barral, O., Ravaja, N., Jacucci, G., Kaski, S.: Predicting term-relevance from brain signals. In: Proceedings of the 37th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 425–434. ACM (2014)Google Scholar
  51. 51.
    Barral, O., Eugster, M.J., Ruotsalo, T., Spapé, M.M., Kosunen, I., Ravaja, N., Kaski, S., Jacucci, G.: Exploring peripheral physiology as a predictor of perceived relevance in information retrieval. In: Proceedings of the 20th International Conference on Intelligent User Interfaces, pp. 389–399. ACM (2015)Google Scholar
  52. 52.
    Treder, M.S., Schmidt, N.M., Blankertz, B.: Gaze independent brain computer interfaces based on covert attention and feature attention. J. Neural Eng. 8(6) (2011)Google Scholar
  53. 53.
    Schmidt, N.M., Blankertz, B., Treder, M.S.: Online detection of error-related potentials boosts the performance of mental typewriters. BMC Neurosci. 13(1), 19 (2012)CrossRefGoogle Scholar
  54. 54.
    Pomplun, M., Sunkara, S.: Pupil dilation as an indicator of cognitive workload in human-computer interaction. In: Proceedings of HCI International 2003, vol. 3, pp. 542–546. Erlbaum, Mahwah (2003)Google Scholar
  55. 55.
    Iqbal, S.T., Zheng, X.S., Bailey, B.P.: Task-evoked pupillary response to mental workload in human-computer interaction. In: CHI 2004 Extended Abstracts on Human Factors in Computing Systems, pp. 1477–1480. ACM, April 2004Google Scholar
  56. 56.
    Chen, S., Epps, J.: Using task-induced pupil diameter and blink rate to infer cognitive load. Hum-Comput. Int. 29(4), 390–413 (2014)CrossRefGoogle Scholar
  57. 57.
    Oliveira, F.T., Aula, A., Russell, D.M.: Discriminating the relevance of web search results with measures of pupil size. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 2209–2212. ACM (2009)Google Scholar
  58. 58.
    Laeng, B., Sirois, S., Gredebck, G.: Pupillometry a window to the preconscious? Perspect. Psychol. Sci. 7(1), 18–27 (2012)CrossRefGoogle Scholar
  59. 59.
    Lessiter, J., Freeman, J., Miotto, A., Ferrari, E.: Ghosts in the machines: towards a taxonomy of human computer interaction. In: Jacucci, G., Gamberini, L., Freeman, J., Spagnolli, A. (eds.) Symbiotic 2014. LNCS, vol. 8820, pp. 21–31. Springer, Heidelberg (2014) Google Scholar
  60. 60.
    Spagnolli, A., Guardigl, E., Orso, V., Varotto, A., Gamberini, L.: Measuring user acceptance of wearable symbiotic devices: validation study across application scenarios. In: Jacucci, G., Gamberini, L., Freeman, J., Spagnolli, A. (eds.) Symbiotic 2014. LNCS, vol. 8820, pp. 87–98. Springer, Heidelberg (2014) Google Scholar
  61. 61.
    Buenaflor, C.B., Kim, H.C.K.: Six Human Factors to Acceptability of Wearable Computers. IJMUE 8, 103–114 (2013)Google Scholar
  62. 62.
    Barfield, W., Mann, S., Baird, K., Gemperle, F., Kasabach, C., Stivoric, J., Bauer, M., Martin, R., Cho, G.: Computational clothing and accessories. In: Fundamentals of Wearable Computers and Augmented Reality, pp. 471–509. Erlbaum, Mahwah (2001)Google Scholar
  63. 63.
    Duval, S., Hoareau, C., Hashizume, H.: 7 humanistic needs as seeds. In: Wong, W.K., Guo, Z.X. (eds.) Smart Clothing, pp. 153–187. Taylor and Francis Group, Boca Raton (2014)Google Scholar
  64. 64.
    Baber, C.: Wearable computers: a human factors review. Int. J. Hum-Comput. Int. 13(2), 123–145 (2001)CrossRefGoogle Scholar
  65. 65.
    Rubin, J., Chisnell, D.: Handbook of usability testing: how to plan, design and conduct effective tests. John Wiley and Sons, New York (2008)Google Scholar
  66. 66.
    Gamberini, L., Spagnolli, A., Prontu, L., Furlan, S., Martino, F., Solaz, B.R., Alcañiz, M., Lozano, J.A.: How natural is a natural interface? An evaluation procedure based on action breakdowns. Pers. Ubiquit. Comput. 17(1), 69–79 (2013)CrossRefGoogle Scholar

Copyright information

© Springer International Publishing Switzerland 2015

Authors and Affiliations

  • Luciano Gamberini
    • 1
    • 5
  • Anna Spagnolli
    • 1
    • 5
  • Benjamin Blankertz
    • 2
  • Samuel Kaski
    • 3
  • Jonathan Freeman
    • 4
  • Laura Acqualagna
    • 2
  • Oswald Barral
    • 6
  • Maura Bellio
    • 5
  • Luca Chech
    • 1
  • Manuel Eugster
    • 3
  • Eva Ferrari
    • 4
  • Paolo Negri
    • 1
  • Valeria Orso
    • 1
  • Patrik Pluchino
    • 5
  • Filippo Minelle
    • 5
  • Bariş Serim
    • 6
  • Markus Wenzel
    • 2
  • Giulio Jacucci
    • 3
    • 6
  1. 1.Department of General PsychologyUniversity of PaduaPaduaItaly
  2. 2.Neurotechnology GroupTechnische Universität BerlinBerlinGermany
  3. 3.Helsinki Institute for Information Technology (HIIT)Aalto UniversityEspooFinland
  4. 4.Department of Psychology, I2 Media ResearchGoldsmiths University of LondonLondonUK
  5. 5.Human Inspired Technology Research Centre (HIT)University of PaduaPaduaItaly
  6. 6.Department of Computer Science, Helsinki Institute for Information Technology (HIIT)University of HelsinkiHelsinkiFinland

Personalised recommendations