HCI Vision for Automated Analysis and Mining of Web User Interfaces

  • Maxim BakaevEmail author
  • Sebastian Heil
  • Vladimir Khvorostov
  • Martin Gaedke
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10845)


Most techniques for webpage structure and design mining are based on code analysis and are detached from a human user’s perception of the web user interface (WUI). Our paper is dedicated to approaches that instead focus on analysis of webpage’s visual representation – the way it is rendered in different browsers and environments and delivered to the end user. Specifically, we describe the software tool that we built, which takes a WUI screenshot and produces structured and machine-readable representation (JSON) of interface elements as made out by a human user. The implementation is based on OpenCV (image recognition functions), dlib (trained detector for the elements’ classification), and Tesseract (label and content text recognition). To demonstrate feasibility of the approach, we describe application of our analyzer tool to auto-calculate certain measures for a WUI and to predict users’ subjective impressions. Particularly, we assess UI visual complexity, which is known to significantly influence both cognitive and affective aspects of interaction. The results suggest the analyzer’s output is mostly characteristic of the users’ visual perception and can be useful for auto-assessing and comparing WUIs.


Web design mining HCI vision Image recognition Human factors Visual complexity 



The reported study was funded by RFBR according to the research project No. 16-37-60060 mol_a_dk.


  1. 1.
    Grigera, J., Garrido, A., Rivero, J.M., Rossi, G.: Automatic detection of usability smells in web applications. Int. J. Hum. Comput. Stud. 97, 129–148 (2017)CrossRefGoogle Scholar
  2. 2.
    Bakaev, M., Mamysheva, T., Gaedke, M.: Current trends in automating usability evaluation of websites: can you manage what you can’t measure? In: Proceedings of IEEE 11th International Forum on Strategic Technology (IFOST), pp. 510–514 (2016)Google Scholar
  3. 3.
    Cai, D., Yu, S., Wen, J.-R., Ma, W.-Y.: Extracting content structure for web pages based on visual representation. In: Zhou, X., Orlowska, Maria E., Zhang, Y. (eds.) APWeb 2003. LNCS, vol. 2642, pp. 406–417. Springer, Heidelberg (2003). Scholar
  4. 4.
    Kumar, R., Talton, J.O., Ahmad, S., Klemmer, S.R.: Bricolage: example-based retargeting for web design. In: Proceedings of the ACM SIGCHI Conference on Human Factors in Computing Systems, pp. 2197–2206 (2011)Google Scholar
  5. 5.
    Sanoja, A., Gançarski, S.: Block-o-matic: a web page segmentation framework. In: Proceedings of IEEE International Conference on Multimedia Computing and Systems (ICMCS), pp. 595–600 (2014)Google Scholar
  6. 6.
    Pu, J., Liu, J., Wang, J.: A vision-based approach for deep web form extraction. In: Park, James J.(Jong Hyuk), Chen, S.-C., Raymond Choo, K.-K. (eds.) MUE/FutureTech -2017. LNEE, vol. 448, pp. 696–702. Springer, Singapore (2017). Scholar
  7. 7.
    Kong, J., et al.: Web interface interpretation using graph grammars. IEEE Trans. Syst. Man Cybern. Part C (Appl. Rev.) 42(4), 590–602 (2012)CrossRefGoogle Scholar
  8. 8.
    Reinecke, K., et al.: Predicting users’ first impressions of website aesthetics with a quantification of perceived visual complexity and colorfulness. In: Proceedings of the ACM SIGCHI Conference on Human Factors in Computing Systems, pp. 2049–2058 (2013)Google Scholar
  9. 9.
    Wu, O., Hu, W., Shi, L.: Measuring the visual complexities of web pages. ACM Trans. Web (TWEB) 7(1), 1 (2013)CrossRefGoogle Scholar
  10. 10.
    Bakaev, M., Razumnikova, O.: Opredeleine slozhnosti zadach dlya zritelno-prostranstvennoi pamyati i propustkoi spospobnosti cheloveka-operatora. Upravlenie bol’shimi sistemami = Large-Scale Systems Control 70, 25–57 (2017). (in Russian)Google Scholar

Copyright information

© Springer International Publishing AG, part of Springer Nature 2018

Authors and Affiliations

  • Maxim Bakaev
    • 1
    Email author
  • Sebastian Heil
    • 2
  • Vladimir Khvorostov
    • 1
  • Martin Gaedke
    • 2
  1. 1.Novosibirsk State Technical UniversityNovosibirskRussia
  2. 2.Technische Universität ChemnitzChemnitzGermany

Personalised recommendations