Skip to main content
SpringerLink
Log in

An assessment of a semiotic framework for evaluating user-intuitive Web interface signs

  • Long paper
  • Published:
Universal Access in the Information Society Aims and scope Submit manuscript

Abstract

A semiotic framework [Semiotic Interface sign Design and Evaluation (SIDE)] was developed to help designers to deal with user-intuitive interface signs. Examples of signs are small images, navigational links, buttons and thumbnails, which users make use of when interacting with Web UIs. This paper assesses the SIDE framework on the quality of the evaluation of interface signs, and the contributions of the framework as perceived by evaluators. Two empirical user studies were carried out, involving 23 participants. Data were collected via interviews, problem-solving assignments and feedback questionnaires, and analyzed quantitatively and qualitatively. The study shows that the evaluation of using the SIDE framework leads to acceptable score on quality metrics; and the subjects evaluate the framework’s ease-of-use, contribution, and usage in practice positively. The SIDE framework is applicable to design and evaluate interface signs and contributes to the improvement in understanding the intuitive nature of interface signs.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

References

  1. Andersen, P.B.: A theory of computer semiotics: Semiotic approaches to construction and assessment of computer systems, vol. 3. Cambridge University Press, Cambridge (1997)

    Google Scholar 

  2. Barr, P., Biddle, R., Noble, J.: A semiotic model of user-interface metaphor. In: Liu, K. (ed.) Virtual, Distributed and Flexible Organisations: Studies in Organisational Semiotics, pp. 189–215. Kluwer Academic Publisher, Dordrecht (2004)

    Google Scholar 

  3. Bekker, M.M., Baauw, E., Barendregt, W.: A comparison of two analytical evaluation methods for educational computer games for young children. Cognit. Technol. Work 10(2), 129–140 (2008)

    Article  Google Scholar 

  4. Bento LFH, Prates RO, Chaimowicz L (2009) Using semiotic inspection method to evaluate a human- robot interface. In: Proceeding of Latin American Web Congress, pp. 77–84

  5. Bilotta, E., Pantano, P.: A semiotic approach for analysing icons in graphical user interface. Bollettino del CIC, No. 3. http://galileo.cincom.unical.it/pubblicazioni/editoria/period/Numeri/quad3htm/Quad35.htm (1995). Accessed Jan 2013

  6. Bolchini, D., Garzotto, F.: Quality of web usability evaluation methods: an empirical study on MiLE+. In: Proceedings of the Web Information Systems Engineering–WISE 2007 Workshops, pp. 481–492. (2007)

  7. Bolchini, D., Chatterji, R., Speroni M.: Developing heuristics for the semiotics inspection of websites. In: Proceedings of the 27th ACM International Conference on Design of communication, ACM, pp. 67–72. (2009)

  8. Cockton, G., Lavery, D., Woolrych, A.: Inspection-based evaluations. In: Jacko, J., Sears, A. (eds.) The Human–Computer Interaction Handbook: Fundamentals, Evolving Technologies and Emerging Applications, pp. 1118–1138. Lawrence and Erlbaum Associates, Mahwah (2003)

    Google Scholar 

  9. De Angeli, A., Matera, M., Costabile, M.F., Garzotto, F., Paolini, P.: On the advantages of a systematic inspection for evaluating hypermedia usability. Int. J. Hum. Comput. Interact. 15(3), 315–335 (2003)

    Article  Google Scholar 

  10. De Souza, C.S.: The Semiotic Engineering of Human-Computer Interaction. The MIT press, Cambridge (2005)

    Google Scholar 

  11. De Souza, C.S.: Semiotic engineering: bringing designers and users together at interaction time. Interact. Comput. 17(3), 317–341 (2005)

    Article  MathSciNet  Google Scholar 

  12. De Souza, C.S., Barbosa, S.D.J., Prates, R.O.: A semiotic engineering approach to HCI. In: Proceedings of the CHI’01 Extended Abstracts on Human Factors in Computing Systems, ACM, pp. 55–56 (2001)

  13. De Souza, C.S., Barbosa, S.D.J., Prates, R.O.: A semiotic engineering approach to user interface design. Knowl. Based Syst. 14(8), 461–465 (2001)

    Article  Google Scholar 

  14. De Souza, C.S., Leitão, C.F., Prates, R.O., da Silva, E.J.: The semiotic inspection method. In: Proceedings of VII Brazilian Symposium on Human factors in computing systems, ACM, pp. 148–157 (2006)

  15. De Souza, C.S.: The semiotic engineering of user interface languages. Int. J. Man Mach. Stud. 39, 753–773 (1993)

    Article  Google Scholar 

  16. De Souza, C.S.: Semiotics. In: Soegaard, M., Dam, R.F. (eds.). The Encyclopedia of Human-Computer Interaction, 2nd Ed. http://www.interaction-design.org/encyclopedia/semiotics_and_human-computer_interaction.html (2013). Accessed July 2014

  17. Derboven, J., De Roeck, D., Verstraete, M.: Semiotic analysis of multi-touch interface design: the MuTable case study. Int. J. Hum. Comput. Stud. 70(10), 714–728 (2012)

    Article  Google Scholar 

  18. Derboven, J., Geerts, D., Grooff, D.D.: Researching user interpretation beyond designer intentions. In: Proceedings of the CHI ‘13 Extended Abstracts on Human Factors in Computing Systems, ACM, pp. 367–372. (2013)

  19. Eco, U.: A theory of Semiotics. Indiana UP, Bloomington (1976)

    Google Scholar 

  20. Fenton, N.E.: Software Metrics: A Rigorous Approach. Chapman & Hall Ltd., London, UK (1991)

  21. French, T., Polovina, S., Vile, A.: Semiotic for e-commerce: shared meanings and generative futures. In: Proceedings of BIT, Vol. 99, (1999)

  22. Hartson, H.R., Andre, T.S., Williges, R.C.: Criteria for evaluating usability evaluation methods. Int. J. Hum. Comput. Interact. 13(4), 373–410 (2001)

    Article  Google Scholar 

  23. Islam, M.N.: A systematic literature review of semiotics perception in user interfaces. J. Syst. Inf. Technol. 15(1), 45–77 (2013). doi:10.1108/13287261311322585

    Article  Google Scholar 

  24. Islam, M.N.: Towards user-intuitive web interface sign design and evaluation: a semiotic framework. Manuscript submitted for publications (2014)

  25. Islam, M.N., Tetard, F.: Exploring the impact of interface signs’ interpretation accuracy, design, and evaluation on web usability: a semiotics perspective. J. Syst. Inf. Technol. 16(4), 250–276 (2014)

    Article  Google Scholar 

  26. Islam, M.N., Ali, M., Al-Mamun, A., Islam, M.: Semiotics explorations on designing the information intensive web interfaces. int. Arab. J. Inf. Technol. 7(1), 45–54 (2010)

    Google Scholar 

  27. Islam, M.N., Tetard, F.: Integrating semiotics perception in usability testing to improve usability evaluation. In: Garcia-Ruiz, M.A. (ed.) Cases on Usability Engineering: Design and Development of Digital Products, pp. 145–169. IGI Global, Hershey (2013)

    Chapter  Google Scholar 

  28. Landauer, T.K.: The Trouble with Computers: Usefulness, Usability, and Productivity. MIT Press, Cambridge (1995)

    Google Scholar 

  29. Liu, K., Crum, G., Dines, K.: Design issues in a semiotic description of user responses to three interfaces. Behav. Inf. Technol. 17(3), 175–184 (1998)

    Article  Google Scholar 

  30. Morris, C.W.: Foundations of the Theory of Signs. University Press, Chicago (1971)

    Google Scholar 

  31. Nadin, M.: Interface design: a semiotic paradigm. Semiotica 69(3–4), 269–302 (1988)

    Google Scholar 

  32. Nadin, M.: One cannot not interact. Knowl. Based Syst. 14(8), 437–440 (2001)

    Article  Google Scholar 

  33. Newman, W.M.: On simulation, measurement, and piecewise usability evaluation. Hum. Comput. Interact. 13(3), 316–323 (1998)

    Google Scholar 

  34. Nielsen, J., Mack, R.L.: Usability Inspection Methods. Wiley, New Work (1994)

    Book  Google Scholar 

  35. Nielsen, J.:10 Usability heuristics for user interface design. http://www.nngroup.com/articles/ten-usability-heuristics/ (1995). Accesses July 2014

  36. O’Neill, S., Benyon, D.:A semiotic approach to investigating presence. In: Proceedings of the 3rd Conference on Computational Semiotics for Games and New Media, UK, (2003a)

  37. O’Neill, S., Benyon, D.: An exploration of a semiotic model of interaction through interactive media. In: Proceedings of the Workshop in HCI, the Arts and the Humanities, York, (2003b)

  38. O’Neill, S., Benyon, D., Turner, S.R.: Semiotics and Interaction Analysis. Proceeding of the 11th European Conference on Cognitive Ergonomics, pp. 44–50. Catania, Sicily (2002)

    Google Scholar 

  39. O’Neill, S.: Exploring a semiotics of new media. PhD thesis, Edinburgh Napier University, UK (2005)

  40. Peirce, C.S.: Collected Writings (8 Vols.). In: Hartshorne, C., Weiss, P., Burks, A. (eds.) Harvard University Press, (1931–1958)

  41. Reis, D.S., Prates, R.O.: Assessing the semiotic inspection method: the evaluators’ perspective. In: Proceedings of the 11th Brazilian Symposium on Human Factors in Computing Systems, pp. 287–296, (2012)

  42. Reis, D.S., Prates, R.O.: Assessing the semiotic inspection method: the evaluators’ perspective. Revista de Informática Teórica e Aplicada (J. Theor. Appl. Inf.) 20(3), 155–182 (2013)

    Google Scholar 

  43. Roberto, V., Toppano, E.: Semiotic design of a hypermedia for cultural heritage. In: Foggia, P. (ed.) Image Analysis and Processing—ICIAP, pp. 425–433. Springer, Berlin (2009)

    Google Scholar 

  44. Salgado, L.C.D.C., De Souza, C.S., Leitão, C.F.: A semiotic inspection of ICDL. In: Carlos, J.P.D.L. (ed) Monografias em Ciência da Computação, No. 31/09, Brazil (2009)

  45. Scolari, C.: The sense of the interface: applying semiotics to HCI research. Semiotica 2009(177), 1–27 (2009)

    Article  Google Scholar 

  46. Sears, A.: Heuristic walkthroughs: finding the problems without the noise. Int. J. Hum. Comput. Interact. 9(3), 213–234 (1997)

    Article  Google Scholar 

  47. Shneiderman, B.: Designing the user interfaces: Strategies for effective human computer interaction. Addison-weseley, Reading (1987)

    Google Scholar 

  48. Speroni, M.: Mastering the semiotics of information-intensive web interfaces (Doctoral dissertation). University of Lugano, Switzerland. http://doc.rero.ch/record/5891/ (2006)

  49. Speroni, M., Bolchini, D., Paolini P, Interfaces: “Do Users Understand Them?”. In: Proceedings of the Museums and the Web 2006, Albuquerque, New Mexico, March 22–25, 2006. http://www.archimuse.com/mw2006/papers/speroni/speroni.html (2006). Accessed July 2014

  50. Triacca, L., Inversini, A., Bolchini, D. Evaluating Web usability with MiLE+. In: Proceedings of the Seventh IEEE International Symposium on Web Site Evaluation (WSE 2005), pp. 22–29, (2005)

Download references

Acknowledgments

The authors would like to thank all the participants of this study. Finnish Economic Education Foundation (LSR), Nokia Foundation, TeliaSonera Research and development Foundation in Finland provided the grant that has made this research possible. For this, LSR, Nokia Foundation and TeliaSonera are gratefully acknowledged. To Dr. Franck Tétard (Associate Professor, Uppsala University, Sweden), thank you for your feedback and suggestions for strengthening this paper.

Author information

Authors and Affiliations

  1. Department of Information Technologies, Åbo Akademi University, Turku, Finland

    Muhammad Nazrul Islam

  2. Section of Information and Communication Technology, Faculty of Technology, Policy and Management, Delft University of Technology, Delft, The Netherlands

    Harry Bouwman

Authors
  1. Muhammad Nazrul Islam
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Harry Bouwman
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Muhammad Nazrul Islam.

Appendices

Appendix 1

The equations used to measure the quality metrics:

Thoroughness measures the proportion of real problems identified by an evaluation method to the real problems existing in the systems:

$${\text{Thoroughness}} = \frac{{{\text{Number}}\;{\text{of}}\;{\text{real}}\;{\text{problems}}\;{\text{identified}}}}{{{\text{Number}}\;{\text{of}}\;{\text{real}}\;{\text{problems}}\;{\text{that}}\;{\text{exist}}}}$$

Validity measures the proportion of problems identified by an evaluation method that are real problems:

$${\text{Validity}} = \frac{{{\text{Number}}\;{\text{of}}\;{\text{real}}\;{\text{problems}}\;{\text{identified}}}}{{{\text{Number}}\;{\text{of}}\;{\text{problems}}\;{\text{identified}}}}$$

Effectiveness measures the simultaneous effect of thoroughness and validity in a ‘figure of merit’:

$${\text{Effectiveness}} = {\text{Thoroughness}} \times {\text{Validity}}$$

Reliability measures the consistency of evaluation results across different (groups of) evaluators:

$${\text{Reliability}} = {\text{Maximum}}\;\;\left\{ {0,\left( {1 - \frac{{{\text{stdev}}\;({\text{number}}\;{\text{of}}\;{\text{real}}\;{\text{problems}}\;{\text{identified}}}}{{{\text{Average}}\;({\text{number}}\;{\text{of}}\;{\text{real}}\;{\text{problems}}\;{\text{identified}}}}} \right)} \right\}$$

Efficiency measures how fast a method helps to identify a problem:

$${\text{Effeciency}} = \frac{{{\text{Number}}\;{\text{of}}\;{\text{real}}\;{\text{problems}}\;{\text{identified}}}}{{{\text{Total}}\;{\text{evaluation}}\;\;{\text{time}}}}$$

Learnability measures the ease of learning of a method [7], based on the following factors: (a) the time needed by an evaluator to become familiar with a method and to be able to carry out an inspection with a reasonable level of performance; (b) participants’ perceived difficulty in terms of learning and/or understanding the method, and being able to carry out an evaluation, and (c) participants’ perceived difficulty with regard to using the method in a real case.

Accuracy measures how well the framework helps identify the intuitiveness of interface signs:

$${\text{Accuracy}} = \frac{{{\text{Number}}\;{\text{of}}\;{\text{sign's}}\;{\text{int}}{\text{uitiveness}}\;{\text{level}}\;{\text{identified}}\;{\text{correctly}}}}{{{\text{Total}}\;{\text{number}}\;{\text{of}}\;{\text{evaluated}}\;{\text{interface}}\;{\text{signs}}}}$$

Appropriateness measures the percentage of correctly applied heuristics:

$${\text{Appropriateness}} = \frac{{{\text{Number}}\;{\text{of}}\;{\text{heuristics}}\;{\text{or}}\;{\text{guidelines}}\;{\text{applied}}\;{\text{correctly}}}}{{{\text{Total}}\;{\text{number}}\;{\text{of}}\;{\text{heuristics}}\;{\text{or}}\;{\text{guidelines}}\;{\text{applied}}}}$$

Appendix 2

See Table 10.

Table 10 Semiotic heuristics of the SIDE framework
Full size table

Appendix 3

The data were collected using the following open-ended questions:

  • What do you like most about the proposed framework? (16 responses)

  • What do you like least about the proposed framework? (15 responses)

  • Do you have any suggestion to improve the proposed framework? (12 responses)

  • Please comment on the procedural guidelines to evaluate the interface signs. (11 responses)

  • Any other comments on the proposed framework. (7 responses)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Islam, M.N., Bouwman, H. An assessment of a semiotic framework for evaluating user-intuitive Web interface signs. Univ Access Inf Soc 14, 563–582 (2015). https://doi.org/10.1007/s10209-015-0403-6

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10209-015-0403-6

Keywords

Search

Navigation