Skip to main content
SpringerLink
Log in

Evaluating the Future of HCI: Challenges for the Evaluation of Emerging Applications

  • Conference paper
Artifical Intelligence for Human Computing

Part of the book series: Lecture Notes in Computer Science ((LNAI,volume 4451))

Abstract

Current evaluation methods are inappropriate for emerging HCI applications. In this paper, we give three examples of these applications and show that traditional evaluation methods fail. We identify trends in HCI development and discuss the issues that arise with evaluation. We aim at achieving increased awareness that evaluation too has to evolve in order to support the emerging trends in HCI systems.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Alben, L.: Quality of experience: defining criteria for effective interaction design. Interactions 3(3), 11–15 (1996)

    Article  Google Scholar 

  2. Gaver, B., Martin, H.: Alternatives: exploring information appliances through conceptual design proposals. In: Proceedings of the conference on Human factors in computing systems (CHI’00), The Hague, The Netherlands, pp. 209–216. ACM Press, New York (2000)

    Chapter  Google Scholar 

  3. Jaimes, A., Sebe, N., Gatica-Perez, D.: Human-centered computing: A multimedia perspective. In: Proceedings of the ACM international conference on Multimedia, Santa Barbara, CA, October 2006, pp. 855–864. ACM Press, New York (2006)

    Google Scholar 

  4. Pantic, M., et al.: Artificial Intelligence for Human Computing. In: Huang, T.S., et al. (eds.) ICMI/IJCAI Workshops 2007. LNCS (LNAI), vol. 4451, pp. 47–71. Springer, Heidelberg (2007)

    Chapter  Google Scholar 

  5. Coen, M.H.: Design principles for intelligent environments. In: Proceedings of the National Conference on Artificial Intelligence (AAAI’98), Madison, WI, pp. 547–554 (1998)

    Google Scholar 

  6. Intille, S.S., et al.: Tools for studying behavior and technology in natural settings. In: Renals, S., Bengio, S. (eds.) MLMI 2005. LNCS, vol. 3869, pp. 157–174. Springer, Heidelberg (2006)

    Google Scholar 

  7. Streitz, N., Nixon, P.: Introduction: The disappearing computer. Communications of the ACM 48(3), 32–35 (2005)

    Article  Google Scholar 

  8. Weiser, M.: The computer of the 21st century. Scientific American 265(3), 66–75 (1991)

    Article  Google Scholar 

  9. Nielsen, J.: Noncommand user interfaces. Communications of the ACM 36(4), 83–89 (1993)

    Article  Google Scholar 

  10. Ju, W., Leifer, L.: The design of implicit interactions. Design Issues, Special Issue on Design Research in Interaction Design (to appear)

    Google Scholar 

  11. Benford, S., et al.: Expected, sensed, and desired: A framework for designing sensing-based interaction. ACM Transactions on Computer-Human Interaction 12(1), 3–30 (2005)

    Article  Google Scholar 

  12. Forlizzi, J., Battarbee, K.: Understanding experience in interactive systems. In: Proceedings of the conference on Designing Interactive Systems (DIS’04), Cambridge, MA, pp. 261–268. ACM Press, New York (2004)

    Google Scholar 

  13. Hassenzahl, M., Tractinsky, N.: User experience: a research agenda. Behaviour & Information Technology 25(2), 91–97 (2006)

    Article  Google Scholar 

  14. Hassenzahl, M., Sandweg, N.: From mental effort to perceived usability: transforming experiences into summary assessments. In: Extended abstracts on Human factors in computing systems (CHI’04), Vienna, Austria, pp. 1283–1286. ACM Press, New York (2004)

    Google Scholar 

  15. Reidsma, D., et al.: Towards bi-directional dancing interaction. In: Harper, R., Rauterberg, M., Combetto, M. (eds.) ICEC 2006. LNCS, vol. 4161, pp. 1–12. Springer, Heidelberg (2006)

    Chapter  Google Scholar 

  16. Thomas, P., Macredie, R.D.: Introduction to the new usability. ACM Transactions on Computer-Human Interaction 9(2), 69–73 (2002)

    Article  Google Scholar 

  17. de Vreede, G.J., et al.: Fifteen years of GSS in the field: A comparison across time and national boundaries. In: Proceedings of the Hawaii International Conference on System Sciences (HICSS’03), Big Island, HA, p. 9. IEEE Computer Society Press, Los Alamitos (2003)

    Chapter  Google Scholar 

  18. Nunamaker Jr., J.F., Briggs, R.O., Mittleman, D.D.: Electronic meeting systems: Ten years of lessons learned. In: Coleman, D., Khanna, R. (eds.) Groupware: Technology and Applications, Prentice Hall, Englewood Cliffs (1995)

    Google Scholar 

  19. Grudin, J.: Groupware and social dynamics: Eight challenges for developers. Communications of the ACM 37(1), 93–105 (1994)

    Article  Google Scholar 

  20. Briggs, R.O., de Vreede, G.J., Nunamaker Jr., J.F.: Collaboration engineering with thinklets to pursue sustained success with group support systems. Journal of Management Information Systems 19(4), 31–64 (2003)

    Google Scholar 

  21. Schmidt, A., Kranz, M., Holleis, P.: Interacting with the ubiquitous computer: towards embedding interaction. In: Proceedings of the joint conference on Smart objects and ambient intelligence (sOc-EUSAI’05), Grenoble, France, pp. 147–152. ACM Press, New York (2005)

    Chapter  Google Scholar 

  22. Davies, N., Gellersens, H.-W.: Beyond prototypes: Challenges in deploying ubiquitous systems. IEEE Pervasive Computing 2(1), 26–35 (2002)

    Article  Google Scholar 

  23. Wiberg, C.: Usability and fun: An overview of relevant research in the hci community. In: Proceedings of the CHI Workshop on Innovative Approaches to Evaluating Affective Interfaces, Portland, OR (April 2005)

    Google Scholar 

  24. Preece, J., et al.: Human-Computer Interaction. Addison-Wesley Longman Ltd., Reading (1994)

    Google Scholar 

  25. Sharp, H., Rogers, Y., Preece, J.: Interaction Design: Beyond Human Computer Interaction, 2nd edn. John Wiley and Sons, Chichester (2007)

    Google Scholar 

  26. Dix, A., et al.: Human Computer Interaction, 3rd edn. Prentice Hall, New York (2004)

    Google Scholar 

  27. Shneiderman, B., Plaisant, C.: Designing the User Interface: Strategies for Effective Human-Computer Interaction, 4th edn. Addison-Wesley, Reading (2005)

    Google Scholar 

  28. Norman, D.A.: The Design of Everyday Things. MIT Press, Cambridge (1998)

    Google Scholar 

  29. Nielsen, J.: Usability Engineering. Academic Press, Boston (1993)

    MATH  Google Scholar 

  30. Bevan, N., Kirakowski, J., Maissel, J.: What is usability? In: Proceedings of the international Conference on HCI, Stuttgart, Germany, September 1991, pp. 651–655 (1991)

    Google Scholar 

  31. Rauterberg, M.: Quantitative measures for evaluating human-computer interfaces. In: Proceedings of the International Conference on Human-Computer Interaction, Orlando, Florida, August 1993, pp. 612–617. Elsevier, Amsterdam (1993)

    Google Scholar 

  32. Shneiderman, B.: Leonardo’s Laptop: Human Needs and the New Computing Technologies. MIT Press, Cambridge (2002)

    Google Scholar 

  33. Mao, J.-Y., et al.: The state of user-centered design practice. Communications of the ACM 48(3), 105–109 (2005)

    Article  Google Scholar 

  34. Reeves, L.M., et al.: Guidelines for multimodal user interface design. Communications of the ACM 47(1), 57–59 (2004)

    Article  Google Scholar 

  35. Card, S.K., Newell, A., Moran, T.P.: The Psychology of Human-Computer Interaction. Lawrence Erlbaum Associates, Mahwah (1983)

    Google Scholar 

  36. Rengger, R.E.: Indicators of Usability based on performance. In: Human Aspects in Computing: Design and Use of Interactive Systems with Terminals, pp. 656–660. Elsevier, Amsterdam (1991)

    Google Scholar 

  37. Abowd, G.D., Mynatt, E.D.: Charting past, present, and future research in ubiquitous computing. ACM Transactions on Computer-Human Interaction 7(1), 29–58 (2000)

    Article  Google Scholar 

  38. Bellotti, V., et al.: Making sense of sensing systems: Five questions for designers and researchers. In: Proceedings of the Conference on Human Factors in Computing Systems (CHI’02), Minneapolis, MN, April 2002, pp. 415–422 (2002)

    Google Scholar 

  39. Fiscus, J.G., et al.: The rich transcription 2005 spring meeting recognition evaluation. In: Renals, S., Bengio, S. (eds.) MLMI 2005. LNCS, vol. 3869, pp. 369–389. Springer, Heidelberg (2006)

    Chapter  Google Scholar 

  40. Phillips, J., et al.: Preliminary face recognition grand challenge results. In: Proceedings of the Conference on Automatic Face and Gesture Recognition 2006 (FGR’06), Southampton, United Kingdom, April 2006, pp. 15–24. IEEE Computer Society Press, Los Alamitos (2006)

    Chapter  Google Scholar 

  41. Nijholt, A., Rist, T., Tuijnenbreijer, K.: Lost in ambient intelligence? In: Extended abstracts on Human factors in computing systems (CHI’04), Vienna, Austria, pp. 1725–1726. ACM Press, New York (2004)

    Google Scholar 

  42. Höysniemi, J., et al.: Children’s intuitive gestures in vision-based action games. Communications of the ACM 48(1), 44–50 (2005)

    Article  Google Scholar 

  43. Oviatt, S.L.: Multimodal interfaces. In: The Human-Computer Interaction Handbook: Fundamentals, Evolving Technologies and Emerging Applications, pp. 286–304. Lawrence Erlbaum, Mahwah (2003)

    Google Scholar 

  44. Van Bunningen, A.H., Feng, L., Apers, P.M.: Context for Ubiquitous Data Management. In: International Workshop on Ubiquitous Data Management (UDM’05), Tokyo, Japan, pp. 17–24 (2005)

    Google Scholar 

  45. Whittaker, S., Terveen, L., Nardi, B.A.: Let’s stop pushing the envelope and start addressing it: a reference task agenda for HCI. Human Computer Interaction 15(2-3), 75–106 (2000)

    Article  Google Scholar 

  46. Blythe, M.A., et al.: Funology: From Usability to Enjoyment. Human-Computer Interaction Series, vol. 3. Kluwer Academic Publishers, Dordrecht (2003)

    Google Scholar 

  47. Nardi, B.A., et al.: Video-as-data: Technical and social aspects of a collaborative multimedia application. Computer Supported Cooperative Work 4(1), 73–100 (1995)

    Article  Google Scholar 

  48. Grudin, J.: Why CSCW applications fail: problems in the design and the evaluation of organizational interfaces. In: Proceedings of the Conference on Computer Supported Cooperative Work (CSCW’88), New York, USA, pp. 85–93. ACM Press, New York (1988)

    Google Scholar 

  49. Newman, W.M.: Better or just different? On the benefits of designing interactive systems in terms of critical parameters. In: Proceedings of the conference on Designing interactive systems: processes, practices, methods, and techniques, Amsterdam, The Netherlands, August 1997, pp. 239–245 (1997)

    Google Scholar 

  50. Petersen, M.G., Madsen, K.H., Kjær, A.: The usability of everyday, technology-emerging and fading opportunities. ACM Transactions on Computer-Human Interaction 9(2), 74–105 (2002)

    Article  Google Scholar 

  51. Intille, S.S.: The goal: smart people, not smart homes. In: Proceedings of the International Conference on Smart Homes and Health Telematics, Belfast, United Kingdom, June 2006, pp. 3–6. IOS Press, Amsterdam (2006)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. University of Twente, Human Media Interaction Group, Enschede, The Netherlands

    Ronald Poppe, Rutger Rienks & Betsy van Dijk

Authors
  1. Ronald Poppe
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Rutger Rienks
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Betsy van Dijk
    View author publications

    You can also search for this author in PubMed Google Scholar

Editor information

Thomas S. Huang Anton Nijholt Maja Pantic Alex Pentland

Rights and permissions

Reprints and permissions

Copyright information

© 2007 Springer Berlin Heidelberg

About this paper

Cite this paper

Poppe, R., Rienks, R., van Dijk, B. (2007). Evaluating the Future of HCI: Challenges for the Evaluation of Emerging Applications. In: Huang, T.S., Nijholt, A., Pantic, M., Pentland, A. (eds) Artifical Intelligence for Human Computing. Lecture Notes in Computer Science(), vol 4451. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-540-72348-6_12

Download citation

  • DOI: https://doi.org/10.1007/978-3-540-72348-6_12

  • Publisher Name: Springer, Berlin, Heidelberg

  • Print ISBN: 978-3-540-72346-2

  • Online ISBN: 978-3-540-72348-6

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation