Improving Success/Completion Ratio in Large Surveys: A Proposal Based on Usability and Engagement

  • Juan Cruz-BenitoEmail author
  • Roberto Therón
  • Francisco J. García-Peñalvo
  • José Carlos Sánchez-Prieto
  • Andrea Vázquez-Ingelmo
  • Martín Martín-González
  • Jorge M. Martínez
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10296)


This paper presents a research focused on improving the success/completion ratio in large surveys. In our case, the large survey is a questionnaire produced by the Spanish Observatory for University Employability and Employment (OEEU in the Spanish acronym). This questionnaire is composed by around 32 and 60 questions and between 86 and 181 variables to be measured. The research is based on the previous experience of a past questionnaire proposed by the OEEU composed also by a large amount of questions and variables to be measured (63–92 questions and 176–279 variables). After analyzing the target population of the questionnaire (with the target population of the previous questionnaire as reference) and reviewing the literature, we have designed 11 proposals for changes in the questionnaire that could improve users’ completion and success ratios (changes that could improve the users’ trust in the questionnaire, the questionnaire usability and user experience or the users’ engagement to the questionnaire). These changes are planned to be applied in the questionnaire in two main different experiments based on A/B test methodologies that will allow researchers to measure the effect of the changes in different populations and in an incremental way. The proposed changes have been assessed by five experts through an evaluation questionnaire. In this questionnaire, researchers gathered the score of each expert regarding to the pertinence, relevance and clarity of each change proposed. Regarding the results of this evaluation questionnaire, the reviewers fully supported 8 out of the 11 changes proposals, so they could be introduced in the questionnaire with no variation. On the other hand, 3 of the proposed changes or improvements are not fully supported by the experts (they have not received a score in the top first quartile of the 1–7 Likert scale). These changes will not be discarded immediately, because despite they have not received a Q1 score, they received a score within the second quartile, so could be reviewed to be enhanced to fit the OEEU’s context.


Human-Computer Interaction HCI Online survey Online questionnaire Usability User experience Engagement Trust A/B test 



The research leading to these results has received funding from “la Caixa” Foundation. Also, the author Juan Cruz-Benito would like to thank the European Social Fund and the Consejería de Educación of the Junta de Castilla y León (Spain) for funding his predoctoral fellow contract. This work has been partially funded by the Spanish Government Ministry of Economy and Competitiveness throughout the DEFINES project (Ref. TIN2016-80172-R).


  1. 1.
    Jarrett, C., Gaffney, G.: Forms that Work: Designing Web Forms for Usability. Morgan Kaufmann, Boston (2009)Google Scholar
  2. 2.
    Dillman, D.A.: Mail and Internet Surveys: The Tailored Design Method. Wiley, New York (2000)zbMATHGoogle Scholar
  3. 3.
    Michavila, F., Martín-González, M., Martínez, J.M., García-Peñalvo, F.J., Cruz-Benito, J.: Analyzing the employability and employment factors of graduate students in Spain: the OEEU information system. In: Third International Conference on Technological Ecosystems for Enhancing Multiculturality (TEEM 2015). ACM Inc., Porto, Portugal (2015)Google Scholar
  4. 4.
    Michavila, F., Martínez, J.M., Martín-González, M., García-Peñalvo, F.J., Cruz-Benito, J.: Barómetro de Empleabilidad y Empleo de los Universitarios en España, 2015 (Primer informe de resultados) (2016)Google Scholar
  5. 5.
    Cruz-Benito, J., Therón, R., García-Peñalvo, F.J., Martín-González, M.: Herramienta para la validación de elementos de mejora UX/Engagement para los cuestionarios de recogida de información de egresados en el contexto del Observatorio de Empleabilidad y Empleo Universitarios (OEEU). GRIAL Research Group, University of Salamanca (2017)Google Scholar
  6. 6.
    Anderson, S.P.: Seductive Interaction Design: Creating Playful, Fun, and Effective User Experiences, Portable Document. Pearson Education (2011)Google Scholar
  7. 7.
    Kveton, P., Jelínek, M., Klimusová, H., Voboril, D.: Data collection on the internet: evaluation of web-based questionnaires. Studia Psychologica 49, 81 (2007)Google Scholar
  8. 8.
    Flavián, C., Gurrea, R., Orús, C.: The effect of product presentation mode on the perceived content and continent quality of web sites. Online Inf. Rev. 33, 1103–1128 (2009)CrossRefGoogle Scholar
  9. 9.
    Huber, W., Vitouch, P.: Usability and accessibility on the internet: effects of accessible web design on usability. In: Miesenberger, K., Klaus, J., Zagler, W., Karshmer, A. (eds.) ICCHP 2008. LNCS, vol. 5105, pp. 482–489. Springer, Heidelberg (2008). doi: 10.1007/978-3-540-70540-6_69 CrossRefGoogle Scholar
  10. 10.
    Seckler, M., Tuch, A.N., Opwis, K., Bargas-Avila, J.A.: User-friendly locations of error messages in web forms: put them on the right side of the erroneous input field. Interact. Comput. 24, 107–118 (2012)CrossRefGoogle Scholar
  11. 11.
    Shneiderman, B.: Designing trust into online experiences. Commun. ACM 43, 57–59 (2000)CrossRefGoogle Scholar
  12. 12.
    Recabarren, M., Nussbaum, M.: Exploring the feasibility of web form adaptation to users’ cultural dimension scores. User Model. User-Adap. Inter. 20, 87–108 (2010)CrossRefGoogle Scholar
  13. 13.
    Pengnate, S.F., Sarathy, R.: An experimental investigation of the influence of website emotional design features on trust in unfamiliar online vendors. Comput. Hum. Behav. 67, 49–60 (2017)CrossRefGoogle Scholar
  14. 14.
    Sayago, S., Blat, J.: Some Aspects of Designing Accessible Online Forms for the Young Elderly. WEBIST 2, 13–17 (2007)Google Scholar
  15. 15.
    Stieger, S., Reips, U.-D.: What are participants doing while filling in an online questionnaire: a paradata collection tool and an empirical study. Comput. Hum. Behav. 26, 1488–1495 (2010)CrossRefGoogle Scholar
  16. 16.
    ISO/DIS: Draft BS ENISO 9241-220 Ergonomics of human-computer interaction. Part 220: processes for enabling, executing and assessing human-centred design within organizations (2016)Google Scholar
  17. 17.
    ISO/DIS: Ergonomics of human-system interaction — Part 125: guidance on visual presentation of information (2013)Google Scholar
  18. 18.
    Bevan, N., Carter, J., Harker, S.: ISO 9241-11 revised: what have we learnt about usability since 1998? In: Kurosu, M. (ed.) HCI 2015. LNCS, vol. 9169, pp. 143–151. Springer, Cham (2015). doi: 10.1007/978-3-319-20901-2_13 CrossRefGoogle Scholar
  19. 19.
    Bargas-Avila, J., Brenzikofer, O.: Simple but crucial user interfaces in the world wide web: introducing 20 guidelines for usable web form design (2011)Google Scholar
  20. 20.
    Bevan, N., Spinhof, L.: Are guidelines and standards for web usability comprehensive? In: Jacko, Julie A. (ed.) HCI 2007. LNCS, vol. 4550, pp. 407–419. Springer, Heidelberg (2007). doi: 10.1007/978-3-540-73105-4_45 CrossRefGoogle Scholar
  21. 21.
    Seckler, M., Heinz, S., Forde, S., Tuch, A.N., Opwis, K.: Trust and distrust on the web: user experiences and website characteristics. Comput. Hum. Behav. 45, 39–50 (2015)CrossRefGoogle Scholar
  22. 22.
    Bargas-Avila, J.A., Brenzikofer, O., Tuch, A.N., Roth, S.P., Opwis, K.: Working towards usable forms on the worldwide web: optimizing multiple selection interface elements. Adv. Hum.-Comput. Interact. 2011, 4 (2011)Google Scholar
  23. 23.
    Couper, M.P., Tourangeau, R., Conrad, F.G., Crawford, S.D.: What they see is what we get response options for web surveys. Soc. Sci. Comput. Rev. 22, 111–127 (2004)CrossRefGoogle Scholar
  24. 24.
    Sánchez-Prieto, J.C., Olmos-Migueláñez, S., García-Peñalvo, F.J.: Informal tools in formal contexts: development of a model to assess the acceptance of mobile technologies among teachers. Comput. Hum. Behav. 55, 519–528 (2016)CrossRefGoogle Scholar

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  • Juan Cruz-Benito
    • 1
    Email author
  • Roberto Therón
    • 1
  • Francisco J. García-Peñalvo
    • 1
  • José Carlos Sánchez-Prieto
    • 1
  • Andrea Vázquez-Ingelmo
    • 1
  • Martín Martín-González
    • 2
  • Jorge M. Martínez
    • 2
  1. 1.GRIAL Research Group, Department of Computers and AutomaticsUniversity of SalamancaSalamancaSpain
  2. 2.UNESCO Chair in University Management and PolicyTechnical University of MadridMadridSpain

Personalised recommendations