Advertisement

Phylter: A System for Modulating Notifications in Wearables Using Physiological Sensing

  • Daniel Afergan
  • Samuel W. Hincks
  • Tomoki Shibata
  • Robert J. K. Jacob
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9183)

Abstract

As wearable computing becomes more mainstream, it holds the promise of delivering timely, relevant notifications to the user. However, these devices can potentially inundate the user, distracting them at the wrong times and providing the wrong amount of information. As physiological sensing also becomes consumer-grade, it holds the promise of helping to control these notifications. To solve this, we build a system Phylter that uses physiological sensing to modulate notifications to the user. Phylter receives streaming data about a user’s cognitive state, and uses this to modulate whether the user should receive the information. We discuss the components of the system and how they interact.

Keywords

fNIRS Adaptive interfaces Brain-computer interfaces Google glass 

Notes

Acknowledgments

We thank Shiwan Zuo, Beste Yuksel, Alvitta Ottley, Eli Brown, Fumeng Yang, Lane Harrison, Sergio Fantini, and Angelo Sassaroli from Tufts University, Erin Solovey from Drexel University, and Michael Rennaker, Timothy Jordan, and Alex Olwal from Google. We also thank Google Inc. and the NSF for support of this research (NSF Grants Nos. IIS-1065154 and IIS-1218170). Any opinions, findings, conclusions, or recommendations expressed in this paper are those of the authors and do not necessarily reflect the views of Google Inc. or the National Science Foundation.

References

  1. 1.
    Afergan, D., Peck, E.M., Solovey, E.T., Jenkins, A., Hincks, S.W., Brown, E.T., Chang, R., Jacob, R.J.K.: Dynamic difficulty using brain metrics of workload. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 3797–3806. ACM (2014)Google Scholar
  2. 2.
    Afergan, D., Shibata, T., Hincks, S.W., Peck, E.M., Yuksel, B.F., Chang, R., Jacob, R.J.K.: Brain-based target expansion. In: Proceedings of the ACM Symposium on User Interface Software and Technology, pp. 583–593. ACM (2014)Google Scholar
  3. 3.
    Baddeley, A.: Working memory. Science 255(5044), 556–559 (1992)CrossRefGoogle Scholar
  4. 4.
    Bailey, B.P., Iqbal, S.T.: Understanding changes in mental workload during execution of goal-directed tasks and its application for interruption management. ACM Trans. Comput.-Hum. Interact. (TOCHI) 14(4), 21 (2008)CrossRefGoogle Scholar
  5. 5.
    Bailey, B.P., Konstan, J.A.: On the need for attention-aware systems: measuring effects of interruption on task performance, error rate, and affective state. Comput. Hum. Behav. 22(4), 685–708 (2006)CrossRefGoogle Scholar
  6. 6.
    Chen, D., Vertegaal, R.: Using mental load for managing interruptions in physiologically attentive user interfaces. In: CHI 2004 extended abstracts on Human Factors in Computing Systems, pp. 1513–1516. ACM (2004)Google Scholar
  7. 7.
    Cutrell, E.B., Czerwinski, M., Horvitz, E.: Effects of instant messaging interruptions on computing tasks. In: CHI 2000 extended abstracts on Human Factors in Computing Systems, pp. 99–100. ACM (2000)Google Scholar
  8. 8.
    Cutrell, E., Tan, D.: BCI for passive input in HCI. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM (2008)Google Scholar
  9. 9.
    Girouard, A., Solovey, E.T., Hirshfield, L.M., Chauncey, K., Sassaroli, A., Fantini, S., Jacob, R.J.K.: Distinguishing difficulty levels with non-invasive brain activity measurements. In: Gross, T., Gulliksen, J., Kotzé, P., Oestreicher, L., Palanque, P., Prates, R.O., Winckler, M. (eds.) INTERACT 2009. LNCS, vol. 5726, pp. 440–452. Springer, Heidelberg (2009) CrossRefGoogle Scholar
  10. 10.
    Girouard, A., Solovey, E.T., Hirshfield, L.M., Peck, E.M., Chauncey, K., Sassaroli, A., Fantini, S., Jacob, R.J.K.: From brain signals to adaptive interfaces: using fNIRS in HCI. In: Tan, D.S., Nijholt, A. (eds.) Brain-Computer Interfaces, pp. 221–237. Springer, London (2010)CrossRefGoogle Scholar
  11. 11.
    Girouard, A., Solovey, E.T., Jacob, R.J.K.: Designing a passive brain computer interface using real time classification of functional near-infrared spectroscopy. Int. J. Auton. Adapt. Commun. Syst. 6(1), 26–44 (2013)CrossRefGoogle Scholar
  12. 12.
    Hirshfield, L.M., Solovey, E.T., Girouard, A., Kebinger, J., Jacob, R.J.K., Sassaroli, A., Fantini, S.: Brain measurement for usability testing and adaptive interfaces: an example of uncovering syntactic workload with functional near infrared spectroscopy. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 2185–2194. ACM (2009)Google Scholar
  13. 13.
    Horvitz, E., Apacible, J.: Learning and reasoning about interruption. In: Proceedings of the International Conference on Multimodal Interfaces, pp. 20–27. ACM (2003)Google Scholar
  14. 14.
    Hussain, M.S., Calvo, R.A., Aghaei Pour, P.: Hybrid fusion approach for detecting affects from multichannel physiology. In: D’Mello, S., Graesser, A., Schuller, B., Martin, J.-C. (eds.) ACII 2011, Part I. LNCS, vol. 6974, pp. 568–577. Springer, Heidelberg (2011) CrossRefGoogle Scholar
  15. 15.
    Iqbal, S.T., Bailey, B.P.: Effects of intelligent notification management on users and their tasks. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 93–102. ACM (2008)Google Scholar
  16. 16.
    Iqbal, S.T., Zheng, X.S., Bailey, B.P.: Task-evoked pupillary response to mental workload in human-computer interaction. In: CHI 2004 extended abstracts on Human Factors in Computing Systems, pp. 1477–1480. ACM (2004)Google Scholar
  17. 17.
    Jackson, T., Dawson, R., Wilson, D.: The cost of email interruption. J. Syst. Inf. Technol. 5(1), 81–92 (2001)CrossRefGoogle Scholar
  18. 18.
    Jones, B., Hesford, C.M., Cooper, C.E.: The use of portable NIRS to measure muscle oxygenation and haemodynamics during a repeated sprint running test. In: Huffel, S.V., Naulaers, G., Caicedo, A., Bruley, D.F., Harrison, D.K. (eds.) Oxygen Transport to Tissue XXXV, pp. 185–191. Springer, New York (2013)CrossRefGoogle Scholar
  19. 19.
    Maior, H.A., Pike, M., Sharples, S., Wilson, M.L.: Examining the reliability of using fNIRS in realistic HCI settings for spatial and verbal tasks. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (In Press). ACM (2015)Google Scholar
  20. 20.
    Mandic, D.P., Obradovic, D., Kuh, A., Adali, T., Trutschell, U., Golz, M., De Wilde, P., Barria, J.A., Constantinides, A.G., Chambers, J.A.: Data fusion for modern engineering applications: an overview. In: Duch, W., Kacprzyk, J., Oja, E., Zadrożny, S. (eds.) ICANN 2005. LNCS, vol. 3697, pp. 715–721. Springer, Heidelberg (2005) Google Scholar
  21. 21.
    Miyata, Y., Norman, D.A.: Psychological issues in support of multiple activities. In: Norman, D.A., Draper, S.W. (eds.) User Centered System Design: New Perspectives on Human-Computer Interaction, pp. 265–284. Lawrence Elbaum Associates, Hillsdale (1986)Google Scholar
  22. 22.
    Owen, A.M., McMillan, K.M., Laird, A.R., Bullmore, E.: Nback working memory paradigm: a meta analysis of normative functional neuroimaging studies. Hum. Brain Mapp. 25(1), 46–59 (2005)CrossRefGoogle Scholar
  23. 23.
    Parasuraman, R., Caggiano, D.: Neural and genetic assays of human mental workload. In: McBride, D.K., Schmorrow, D. (eds.) Quantifying Human Information Processing, pp. 123–149. Rowman & Littlefield Publishers Inc., Lanham (2005)Google Scholar
  24. 24.
    Peck, E.M., Afergan, D., Jacob, R.J.K.: Investigation of fNIRS brain sensing as input to information filtering systems. In: Proceedings of Augmented Human International Conference, pp. 142–149. ACM (2013)Google Scholar
  25. 25.
    Peck, E.M., Afergan, D., Yuksel, B.F., Lalooses, F., Jacob, R.J.K.: Using fNIRS to measure mental workload in the real world. In: Fairclough, S.H., Gilleade, K. (eds.) Advances in Physiological Computing, pp. 117–139. Springer, London (2014)CrossRefGoogle Scholar
  26. 26.
    Pierce, J.S., Nichols, J.: An infrastructure for extending applications’ user experiences across multiple personal devices. In: ACM Symposium on User Interface Software and Technology, pp. 101–110. ACM (2008)Google Scholar
  27. 27.
    Piper, S.K., Krueger, A., Koch, S.P., Mehnert, J., Habermehl, C., Steinbrink, J., Obrig, H., Schmitz, C.H.: A wearable multi-channel fNIRS system for brain imaging in freely moving subjects. Neuroimage 85, 64–71 (2014)CrossRefGoogle Scholar
  28. 28.
    Prinzel, L.J., Freeman, F.G., Scerbo, M.W., Mikulka, P.J., Pope, A.T.: Effects of a psychophysiological system for adaptive automation on performance, workload, and the event-related potential P300 component. Hum. Factors J. Hum. Factors Ergon. Soc. 45(4), 601–614 (2003)CrossRefGoogle Scholar
  29. 29.
    Repovš, G., Baddeley, A.: The multi-component model of working memory: explorations in experimental cognitive psychology. Neuroscience 139(1), 5–21 (2006)CrossRefGoogle Scholar
  30. 30.
    Shibata, T., Peck, E.M., Afergan, D., Hincks, S.W., Yuksel, B.F., Jacob, R.J.K.: Building implicit interfaces for wearable computers with physiological inputs: zero shutter camera and phylter. In: Proceedings of the adjunct publication of the ACM Symposium on User Interface Software and Technology, pp. 89–90. ACM (2014)Google Scholar
  31. 31.
    Solovey, E.T., Afergan, D., Peck, E.M., Hincks, S.W., Jacob, R.J.K.: Designing implicit interfaces for physiological computing: guidelines and lessons learned using fNIRS. ACM Trans. Comput.-Hum. Interact. (TOCHI) 21(6), 35 (2015)Google Scholar
  32. 32.
    Solovey, E.T., Girouard, A., Chauncey, K., Hirshfield, L.M., Sassaroli, A., Zheng, F., Fantini, S., Jacob, R.J.K.: Using fNIRS brain sensing in realistic HCI settings: experiments and guidelines. In: Proceedings of the ACM Symposium on User Interface Software and Technology, pp. 157–166. ACM (2009)Google Scholar
  33. 33.
    Solovey, E., Schermerhorn, P., Scheutz, M., Sassaroli, A., Fantini, S., Jacob, R.J.K.: Brainput: enhancing interactive systems with streaming fNIRS brain input. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 2193–2202. ACM (2012)Google Scholar
  34. 34.
    Sthalekar, C.C., Koomson, V.J.: A CMOS sensor for measurement of cerebral optical coefficients using non-invasive frequency domain near infrared spectroscopy. IEEE Sens. J. 13(9), 3166–3174 (2013)CrossRefGoogle Scholar
  35. 35.
    Strangman, G., Culver, J.P., Thompson, J.H., Boas, D.A.: A quantitative comparison of simultaneous BOLD fMRI and NIRS recordings during functional brain activation. Neuroimage 17(2), 719–731 (2002)CrossRefGoogle Scholar
  36. 36.
    Stripling, R., Coyne, J.T., Cole, A., Afergan, D., Barnes, R.L., Rossi, K.A., Reeves, L.M., Schmorrow, D.D.: Automated SAF adaptation tool (ASAT). In: Schmorrow, D.D., Reeves, L.M. (eds.) HCII 2007 and FAC 2007. LNCS (LNAI), vol. 4565, pp. 346–353. Springer, Heidelberg (2007) CrossRefGoogle Scholar
  37. 37.
    Tremoulet, P., Barton, J., Craven, P., Gifford, A., Morizio, N., Belov, N., Stibler, K., Regli, S.H., Thomas, M.: Augmented cognition for tactical Tomahawk weapons control system operators. In: Schmorrow, D., Stanney, K., Reeves, L. (eds.) Foundations of Augmented Cognition, pp. 313–318. Strategic Analysis Inc., Arlington (2006)Google Scholar
  38. 38.
    Villringer, A., Chance, B.: Non-invasive optical spectroscopy and imaging of human brain function. Trends Neurosci. 20(10), 435–442 (1997)CrossRefGoogle Scholar
  39. 39.
    Wickens, C.D.: Multiple resources and mental workload. Hum. Factors J. Hum. Factors Ergon. Soc. 50(3), 449–455 (2008)CrossRefGoogle Scholar
  40. 40.
    Young, M.S., Brookhuis, K.A., Wickens, C.D., Hancock, P.A.: State of science: mental workload in ergonomics. Ergonomics 58(1), 1–17 (2015)CrossRefGoogle Scholar
  41. 41.
    Zander, T.O., Kothe, C., Welke, S., Roetting, M.: Utilizing secondary input from passive brain-computer interfaces for enhancing human-machine interaction. In: Schmorrow, D.D., Estabrooke, I.V., Grootjen, M. (eds.) FAC 2009. LNCS, vol. 5638, pp. 759–771. Springer, Heidelberg (2009) Google Scholar

Copyright information

© Springer International Publishing Switzerland 2015

Authors and Affiliations

  • Daniel Afergan
    • 1
  • Samuel W. Hincks
    • 1
  • Tomoki Shibata
    • 1
  • Robert J. K. Jacob
    • 1
  1. 1.Tufts UniversityMedfordUSA

Personalised recommendations