Skip to main content
SpringerLink
Log in

Quality characteristics and measures for human–computer interaction evaluation in ubiquitous systems

  • Published:
Software Quality Journal Aims and scope Submit manuscript

Abstract

The advent of ubiquitous systems places even more focus on users, since these systems must support their daily activities in such a transparent way that does not disturb them. Thus, much more attention should be provided to human–computer interaction (HCI) and, as a consequence, to its quality. Dealing with quality issues implies first the identification of the quality characteristics that should be achieved and, then, which software measures should be used to evaluate them in a target system. Therefore, this work aims to identify what quality characteristics and measures have been used for the HCI evaluation of ubiquitous systems. In order to achieve our goal, we performed a large literature review, using a systematic mapping study, and we present our results in this paper. We identified 41 pertinent papers that were deeply analyzed to extract quality characteristics and software measures. We found 186 quality characteristics, but since there were divergences on their definitions and duplicated characteristics, an analysis of synonyms by peer review based on the equivalence of definitions was also done. This analysis allowed us to define a final suitable set composed of 27 quality characteristics, where 21 are generic to any system but are particularized for ubiquitous applications and 6 are specific for this domain. We also found 218 citations of measures associated with the characteristics, although the majority of them are simple definitions with no detail about their measurement functions. Our results provide not only an overview of this area to guide researchers in directing their efforts but also it can help practitioners in evaluating ubiquitous systems using these measures.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14

Similar content being viewed by others

Notes

  1. Some papers did not have the abstract registered in the database. For those papers we downloaded the complete paper to start the review (next step screening the papers).

  2. http://lapes.dc.ufscar.br/tools/start_tool.

  3. The sum of the numbers in the graphic exceeds the total number of measures because there are duplicate measures among the papers.

  4. ISO/IEC 25000 2014 presents the following definitions:

    Quality measure element (QME): measure defined in terms of a property and the measurement method for quantifying it.

    Property to quantify: property of a target entity that is related to a quality measure element and which can be quantified by a measurement method

    Quality measure (QM): derived measure that is defined as a measurement function of two or more values of quality measure elements.

    Measurement function: algorithm or calculation performed to combine two or more quality measure elements.

  5. We could use the standard ISO/IEC 9241, specific for HCI, for our analysis. However, we chose to work with the SQuaRE since it aggregates the other quality standards (ISO/IEC 9126 2001) and (ISO/IEC 14598 1999), and the characteristics from ISO/IEC 9241 are also all defined in it.

References

  • Abi-Char, P. E., Mhamed, A., El-Hassan, B., & Mokhtari, M. (2010). A flexible privacy and trust based context-aware secure framework. In Proceedings of the aging friendly technology for health and independence, and 8th international conference on smart homes and health telematics, ICOST’10. Springer. http://dl.acm.org/citation.cfm?id=1894439.1894443.

  • Ammar, L., Trabelsi, A., & Mahfoudhi, A. (2015). A model-driven approach for usability engineering of interactive systems. Software Quality Journal, 1–35.

  • Bezerra, C. I. M., Oliveira, K. M., Andrade, R. M. C., et al. (2014). Challenges for usability testing in ubiquitous system. In l’Interaction Homme-Machine.

  • Cappiello, I., Puglia, S., & Vitaletti, A. (2009). Design and initial evaluation of a ubiquitous touch-based remote grocery shopping process. In First international workshop on near field communication, 9–14.

  • Carvalho, R. M., Andrade, R. M. C., & Oliveira, K. M. (2015). Using the GQM method to evaluate calmness in ubiquitous applications*. In HCI international.

  • Chalmers, D., & Sloman, M. (1999). A survey of quality of service in mobile computing environments. IEEE Communications Surveys & Tutorials, 2(2), 2–10.

    Article  Google Scholar 

  • Chang, Y.-H., & Lin, B.-S. (2011). An inquiry-based ubiquitous tour system. In International conference on complex, intelligent and software intensive systems.

  • Da Silva, C. M. R., Da Silva, J. L. C., Rodrigues, R. B., Do Nascimento, L. M., & Garcia, V. C. (2013). Systematic mapping study on security threats in cloud computing. arXiv preprint arXiv:1303.6782,

  • Damián-Reyes, P., Favela, J., & Contreras-Castillo, J. (2011). Uncertainty management in context-aware applications: Increasing usability and user trust. Wireless Personal Communications, 56(1), 37–53.

    Article  Google Scholar 

  • De Moor, K., Ketyko, I., Joseph, W., et al. (2010). Proposed framework for evaluating quality of experience in a mobile, testbed-oriented living lab setting. Mobile Networks and Applications, 15(3), 378–391.

    Article  Google Scholar 

  • Dey, A. K. (2001). Understanding and using context. Personal and Ubiquitous Computing, 5(1), 4–7.

    Article  MathSciNet  Google Scholar 

  • Evers, V., Cramer, H., Van Someren, M., & Wielinga, B. (2010). Interacting with adaptive systems. In Interactive collaborative information systems (pp. 299–325). Springer.

  • Evers, C., Kniewel, R., Geihs, K., & Schmidt, L. (2014). The user in the loop: Enabling user participation for self-adaptive applications. Future Generation Computer Systems, 34, 110–123.

    Article  Google Scholar 

  • Fenton, N., & Pfleeger, S. (1997). Software metrics: A rigorous and practical approach. PWS Pub.

  • Haapalainen, E., Kim, S., Forlizzi, J. F., & Dey, A. K. (2010). Psycho-physiological measures for assessing cognitive load. In Proceedings of the 12th ACM international conference on ubiquitous computing, Ubicomp’10. ACM. http://doi.acm.org/10.1145/1864349.1864395.

  • Hernandes, E., Zamboni, A., Fabbri, S., & Thommazo, A. Di. (2012). Using GQM and TAM to evaluate StArt—A tool that supports systematic review. CLEI Electronic Journal, 15, 3.

    Google Scholar 

  • Iqbal, R., Sturm, J., Kulyk, O., Wang, J., & Terken, J. (2005). User-centred design and evaluation of ubiquitous services. In Proceedings of the 23rd international conference on design of communication—documenting and designing for pervasive information. http://www.scopus.com/inward/record.url?eid=2-s2.0-32044471829&partnerID=40&md5=9a711a41b59e2b258fd13669a3826bc7.

  • ISO 9241-11. (1998). Ergonomic requirements for office work with visual display terminals. In The International Organization for Standardization

  • ISO/IEC 14598. (1999). Information Technology—Software Product Evaluation—Part 1.

  • ISO/IEC 9126. (2001). Software engineering—Product Quality—Part 1.

  • ISO/IEC 25000. (2014). Software Engineering—Software Product Quality Requirements and Evaluation (SQuaRE)—Guide to SQuaRE.

  • ISO/IEC 25010. (2011). ISO/IEC 25010. Systems and software engineering—Systems and software Quality Requirements and Evaluation (SQuaRE)—System and software quality models, v. 2011.

  • Jafari, S., Mtenzi, F., O’Driscoll, C., Fitzpatrick, R., & O’Shea, B. (2010). Privacy metrics in ubiquitous computing applications. In International conference for internet technology and secured transactions.

  • Jafari, S., Mtenzi, F., O’Driscoll, C., Fitzpatrick, R., & O’Shea, B. (2011). Measuring privacy in ubiquitous computing applications. International Journal of Digital Society, 2(3), 547–550.

    Article  Google Scholar 

  • Jia, L., Collins, M., & Nixon, P. (2009). Evaluating trust-based access control for social interaction. In 3rd international conference on mobile ubiquitous computing, systems, services, and technologies. http://www.scopus.com/inward/record.url?eid=2-s2.0-77951440915&partnerID=40&md5=241ed20a5c57e71b94e32fcaec543a9e.

  • Karaiskos, D., Kourouthanassis, P., & Giaglis, G. M. (2009). Towards a validated construct for information systems pervasiveness: An exploratory assessment. In BLED 2009 proceedings, p. Paper 12.

  • Karvonen, H., & Kujala, T. (2014). Designing and evaluating ubicomp characteristics of intelligent in-car systems. In 5th International conference on applied human factors and ergonomics.

  • Kemp, E. A., Thompson, A.-J., & Johnson, R. S. (2008). Interface evaluation for invisibility and ubiquity: An example from E-learning. In Proceedings of the 9th ACM SIGCHI New Zealand Chapter’s international conference on human–computer interaction: Design Centered HCI, CHINZ’08. ACM. http://doi.acm.org/10.1145/1496976.1496981.

  • Kim, H. J., Choi, J. K., & Ji, Y. (2008). Usability evaluation framework for ubiquitous computing device. In Proceedings—3rd international conference on convergence and hybrid information technology, ICCIT 2008. http://www.scopus.com/inward/record.url?eid=2-s2.0-57849159822&partnerID=40&md5=2965dfe3adcf63a35f173ca1fd4cb7e9.

  • Kim, Y., & Lee, K. (2006). A quality measurement method of context information in ubiquitous environments. In 2006 International conference on hybrid information technology. http://ieeexplore.ieee.org/xpl/freeabs_all.jsp?arnumber=4021269.

  • Kitchenham, B. A., Budgen, D., & Brereton, P. (2010). The value of mapping studies—A participant-observer case study. In Proceedings of evaluation and assessment of software engineering, EASE.

  • Kitchenham, B. A., & Charters, S. (2007). Guidelines for performing systematic literature reviews in software engineering. In Technical report. EBSE-2007-01, Keele University.

  • Ko, I.-Y., Koo, H.-M., & Jimenez-Molina, A. (2010). User-centric web services for ubiquitous computing. In Advanced techniques in web intelligence-I.

  • Kourouthanassis, P. E., Giaglis, G. M., & Karaiskos, D. C. (2008). Delineating the degree of “pervasiveness” in pervasive information systems: An assessment framework and design implications. In Pan-Hellenic conference on informatics, PCI.

  • Kryvinska, N., Strausss, C., & Zinterhof, P. (2011). “Variated availability” approach to the services manageable delivering. In Fifth international conference on innovative mobile and internet services in ubiquitous computing (IMIS).

  • Lee, J., & Yun, M. H. (2012). Usability assessment for ubiquitous services: Quantification of the interactivity in inter-personal services. In IEEE international conference on management of innovation & technology.

  • Lee, J., Song, J., Kim, H., Choi, J., & Yun, M. H. (2008). A user-centered approach for ubiquitous service evaluation: An evaluation metrics focused on human–system interaction capability. In Asia-Pacific conference, APCHI.

  • Liampotis, N., Roussaki, I., Papadopoulou, E., et al. (2009). A privacy framework for personal self-improving smart spaces. In International conference on computational science and engineering.

  • Montagud, S., Abrahão, S., & Insfran, E. (2012). A systematic review of quality attributes and measures for software product lines. In Software Quality Journa, v. 20.

  • Nielsen, J. (1994). Usability engineering. Access Online via Elsevier.

  • Novais, R. L., Torres, A., Mendes, T. S., Mendonça, M., & Zazworka, N. (2013). Software evolution visualization: A systematic mapping study. Information and Software Technology, 55(11), 1860–1883.

    Article  Google Scholar 

  • Oriol, M., Marco, J., & Franch, X. (2014). Quality models for web services: A systematic mapping. Information and Software Technology, 56(10), 1167–1182.

    Article  Google Scholar 

  • Petersen, K., Feldt, R., Mujtaba, S., & Mattsson, M. (2008). Systematic mapping studies in software engineering. In Proceedings of the 12th international conference on evaluation and assessment in software engineering, EASE’08. British Computer Society. http://dl.acm.org/citation.cfm?id=2227115.2227123.

  • Petersen, K., & Gencel, C. (2013). Worldviews, research methods, and their relationship to validity in empirical software engineering research. In Proceedings—Joint conference of the 23rd international workshop on software measurement and the 8th international conference on software process and product measurement, IWSM-MENSURA 2013 (pp. 81–89).

  • Petersen, K., Vakkalanka, S., & Kuzniarz, L. (2015). Guidelines for conducting systematic mapping studies in software engineering: An update. Information and Software Technology, 64, 1–18.

    Article  Google Scholar 

  • Poppe, R., Rienks, R., & Dijk, B. Van (2007). Evaluating the future of HCI: Challenges for the evaluation of emerging applications. AI for Human Computing, 234–250.

  • Ranganathan, A., Al-Muhtadi, J., Biehl, J., et al. (2005). Towards a pervasive computing benchmark. In International conference on pervasive computing and communications workshops.

  • Reis, R. A. C. (2015). Usability evaluation approaches for (ubiquitous) mobile applications: A systematic mapping study usability evaluation approaches for (ubiquitous) mobile applications: A systematic mapping study. n. September.

  • Rocha, L. S., Ferreira, J. B., Lima, F. F. P., et al. (2011). Ubiquitous software engineering: Achievements, challenges and beyond. In Brazilian symposium on software engineering (in Portuguese)

  • Ross, T., & Burnett, G. (2001). Evaluating the human–machine interface to vehicle navigation systems as an example of ubiquitous computing. In International Journal of Human–Computer Studies,

  • Rubio, J. M. L., & Bozo, J. P. (2007). Approach to a quality process for the ubiquitous software development. In Electronics, robotics and automotive mechanics conference.

  • Ryu, H., Hong, G. Y., & James, H. (2006). Quality assessment technique for ubiquitous software and middleware. Research Letters in the Information and Mathematical Sciences, 9, 13–87.

    Google Scholar 

  • Sanchez-pi, N., & Carb, J. (2012). An evaluation method for context—Aware systems in U-Health. In 3rd international symposium on ambient intelligence (ISAmI 2012).

  • Santos, R. M., Oliveira, K. M., Andrade, R. M. C., Santos, I. S., & Lima, E. R. R. (2013). A quality model for human–computer interaction evaluation in ubiquitous systems. In Latin American conference on human computer interaction.

  • Schalkwyk, J., Beeferman, D., Beaufays, F., et al. (2010). Your word is my command: Google Search by Voice: A case study. Advances in Speech Recognition.

  • Scholtz, J., & Consolvo, S. (2004). Toward a framework for evaluating ubiquitous computing applications. IEEE Pervasive Computing,

  • Sears, A., & Jacko, J. A. (2009). Human–computer interaction: Development process. Boca Raton: CRC Press.

    Book  Google Scholar 

  • Silveira, P. A. M., Machado, I. C., McGregor, J. D., Santana, E., & Meira, S. R. L. (2011). A systematic mapping study of software product lines testing. In Information and Software Technology

  • Song, J., Park, K. R., Kwon, S., Lee, J. H. J. H., & Yun, M. H. (2009). The development of human-system interactivity metrics for ubiquitous service applying user-centered design methodology. In World Congress on Services.

  • Sousa, B., Pentikousis, K., & Curado, M. (2011). UEF: Ubiquity evaluation framework. Wired/Wireless Internet Communications

  • Spínola, R. O., & Travassos, G. H. (2012). Towards a framework to characterize ubiquitous software projects. Information and Software Technology

  • Sun, T., & Denko, M. K. (2008). Performance evaluation of trust management in pervasive computing. In International conference on advanced information networking and applications, AINA.

  • Tahir, T., & Jafar, A. (2011). A systematic review on software measurement programs. In Frontiers of Information Technology (FIT), 2011 (Vol. 73, pp. 39–44).

  • Thompson, S. G., & Azvine, B. (2004). No pervasive computing without intelligent systems. In BT technology journal

  • Toch, E. (2011). Super-Ego: A framework for privacy-sensitive bounded context-awareness. In ACM international workshop on context-awareness for self-managing systems.

  • Viana, J. R. M., Viana, N. P., Trinta, F. A. M., & Carvalho, W. V. De (2014). A systematic review on software engineering in pervasive games development. In Brazilian symposium on computer games and digital entertainment. http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=7000032.

  • Wagner, S., Toftegaard, T., & Bertelsen, O. (2012). Requirements for an evaluation infrastructure for reliable pervasive healthcare research. In International conference on pervasive computing technologies for healthcare, IEEE.

  • Waibel, A., Stiefelhagen, R., Carlson, R., et al. (2010). Computers in the human interaction loop. Handbook of Ambient Intelligence and Smart Environments.

  • Weihong-Guo, A., Blythe, P., Olivier, P., Singh, P., & Nam Ha, H. (2008). Using immersive video to evaluate future traveller information systems. In IET intelligent transport systems,

  • Weiser, M. (1991). The computer for the 21st century. In Scientific American.

  • Wieringa, R., Maiden, N., Mead, N., & Rolland, C. (2005). Requirements engineering paper classification and evaluation criteria: A proposal and a discussion. In Requirements Engineering.

  • Wohlin, C. (2014). Guidelines for snowballing in systematic literature studies and a replication in software engineering. In 18th international conference on evaluation and assessment in software engineering (EASE 2014) (pp. 1–10).

  • Wohlin, C., Runeson, P., Da Mota Silveira Neto, P. A., et al. (2013). On the reliability of mapping studies in software engineering. Journal of Systems and Software, 86(10), 2594–2610.

    Article  Google Scholar 

  • Wu, C. L., & Fu, L. C. (2012). Design and realization of a framework for human–system interaction in smart homes. IEEE Transactions on Systems, Man, and Cybernetics—Part A: Systems and Humans, 42(1), 15–31.

    Article  Google Scholar 

  • Zhang, Y., Zhang, S., Tong, H., & Yong Zhang, S. Z. (2006). Adaptive service delivery for mobile users in ubiquitous computing environments. In International conference on Ubiquitous Intelligence and Computing.

Download references

Acknowledgments

We thank FUNCAP (Ceará State Foundation for Support of Scientific and Technological Development, Brazil) and CNRS (Centre National de la Recherche Scientifique, France) for the financial support of this work, which is a result of the Maximum Project—A Measurement-based Approach for the Quality Evaluation of Human–Computer Interaction in Ubiquitous Systems, under grant number INC-0064-00012.01.00/12. We also thank CAPES for sponsoring Rainara Maia Carvalho and Ismayle de Sousa Santos with PhD scholarships, and CNPq for sponsoring Rossana Maria de Castro Andrade with a Researcher Scholarship - DT Level 2.

Author information

Authors and Affiliations

  1. Group of Computer Networks, Software Engineering, and Systems (GREat), Computer Science Department (DC), Federal University of Ceará (UFC), Fortaleza, Brazil

    Rainara Maia Carvalho, Rossana Maria de Castro Andrade, Ismayle de Sousa Santos & Carla Ilane Moreira Bezerra

  2. Laboratory of Automatic Control, Mechanics and Computer Science for Industrial and Human-Machine Systems (LAMIH), CNRS UMR 8201, University of Valenciennes and Hainaut-Cambrésis (UVHC), Valenciennes, France

    Káthia Marçal de Oliveira

Authors
  1. Rainara Maia Carvalho
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Rossana Maria de Castro Andrade
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Káthia Marçal de Oliveira
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Ismayle de Sousa Santos
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Carla Ilane Moreira Bezerra
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Rainara Maia Carvalho.

Appendix: Software Measures

Appendix: Software Measures

This appendix presents the 219 extracted software measures from the systematic mapping presented in this paper. These measures are classified and organized in tables according to the quality characteristics that they are aimed at evaluating (Tables 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27).

Table 6 Software measures for acceptability
Full size table
Table 7 Software measures for attention
Full size table
Table 8 Software measures for availability
Full size table
Table 9 Software measures for calmness
Full size table
Table 10 Software measures for context-awareness
Full size table
Table 11 Software measures for device capability
Full size table
Table 12 Software measures for efficiency
Full size table
Table 13 Software measures for mobility
Full size table
Table 14 Software measure for network capabilities
Full size table
Table 15 Software measures for privacy
Full size table
Table 16 Software measures for reliability
Full size table
Table 17 Software measures for reversibility
Full size table
Table 18 Software measures for robustness
Full size table
Table 19 Software measure for scalability
Full size table
Table 20 Software measures for security
Full size table
Table 21 Software measures for simplicity
Full size table
Table 22 Software measures for transparency
Full size table
Table 23 Software measures for trust
Full size table
Table 24 Software measures for usability
Full size table
Table 25 Software measure for user satisfaction
Full size table
Table 26 Software measures for utility
Full size table
Table 27 Software measures without characteristic
Full size table

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Carvalho, R.M., de Castro Andrade, R.M., de Oliveira, K.M. et al. Quality characteristics and measures for human–computer interaction evaluation in ubiquitous systems. Software Qual J 25, 743–795 (2017). https://doi.org/10.1007/s11219-016-9320-z

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11219-016-9320-z

Keywords

Search

Navigation