How to Assess the Usability Metrics of E-Voting Schemes

  • Karola MarkyEmail author
  • Marie-Laure Zollinger
  • Markus Funk
  • Peter Y. A. Ryan
  • Max Mühlhäuser
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11599)


Voters play an important role in end-to-end verifiable e-voting schemes because the schemes encourage them to carry out several security-critical tasks by themselves. If the voters cannot complete the tasks by themselves or experience bad usability while executing them, vote manipulations by either a faulty software or deliberate attacks cannot be detected which renders verification useless. Therefore, the scheme’s usability is of crucial importance and demands an early investigation of human factors when implementing e-voting systems. In this paper, we give an overview of user study design challenges when investigating end-to-end verifiable e-voting schemes. We provide guidelines that address these challenges and support researchers in the design of user studies. The guidelines are based on the literature and the authors’ experiences.


E-voting Usability evaluation End-to-end verifiability 



This work has been co-funded by the DFG as part of project “Area D.1” within the RTG 2050 “Privacy and Trust for Mobile Users” and by the Horst Görtz Foundation. We acknowledge support from the Luxembourg National Research Fund (FNR) for funding, in particular Marie-Laure Zollinger was supported by the FNR-INTER-VoteVerif project and FNR-INTER-SeVoTe project.


  1. 1.
    Acemyan, C.Z., Kortum, P., Byrne, M.D., Wallach, D.S.: Usability of voter verifiable, end-to-end voting systems: Baseline data for helios, prêt à voter, and scantegrity ii. USENIX J. Election Technol. Syst. (JETS) 2(3), 26–56 (2014)Google Scholar
  2. 2.
    Acemyan, C.Z., Kortum, P., Byrne, M.D., Wallach, D.S.: Summative usability assessments of STAR-Vote: a cryptographically secure e2e voting system that has been empirically proven to be easy to use. Hum. factors, 1–24 (2018) Google Scholar
  3. 3.
    Albert, W., Tullis, T.: Measuring the User Experience: Collecting, Analyzing, and Presenting Usability Metrics. Newnes, Boston (2013)Google Scholar
  4. 4.
    Ali, S.T., Murray, J.: An overview of end-to-end verifiable voting systems. In: Real-World Electronic Voting: Design, Analysis and Deployment, pp. 171–218. CRC Press (2016)Google Scholar
  5. 5.
    Bederson, B.B., Lee, B., Sherman, R.M., Herrnson, P.S., Niemi, R.G.: Electronic voting system usability issues. In: SIGCHI Conference on Human Factors in Computing Systems (CHI), pp. 145–152. ACM (2003)Google Scholar
  6. 6.
    Benaloh, J., Rivest, R., Ryan, P.Y., Stark, P., Teague, V., Vora, P.: End-to-end verifiability, pp. 1–7 (2015).
  7. 7.
    Boren, T., Ramey, J.: Thinking aloud: reconciling theory and practice. IEEE Trans. Prof. Commun. 43(3), 261–278 (2000)CrossRefGoogle Scholar
  8. 8.
    Brooke, J.: SUS - a quick and dirty usability scale. Usability Eval. Ind. 189(194), 4–7 (1996)Google Scholar
  9. 9.
    Budurushi, J., Renaud, K., Volkamer, M., Woide, M.: An investigation into the usability of electronic voting systems for complex elections. Ann. Telecommun. 71(7), 309–322 (2016). Scholar
  10. 10.
    Byrne, M.D., Greene, K.K., Everett, S.P.: Usability of voting systems: baseline data for paper, punch cards, and lever machines. In: SIGCHI Conference on Human Factors in Computing Systems (CHI), pp. 171–180. ACM (2007)Google Scholar
  11. 11.
    Campbell, B.A., Byrne, M.D.: Now do voters notice review screen anomalies? A look at voting system usability. In: Conference on Electronic Voting Technology/Workshop on Trustworthy Elections (EVT/WOTE). USENIX Association (2009)Google Scholar
  12. 12.
    Chaum, D.: Secret-ballot receipts: true voter-verifiable elections. IEEE Secur. Priv. 2(1), 38–47 (2004)CrossRefGoogle Scholar
  13. 13.
    Conrad, F.G., et al.: Electronic voting eliminates hanging chads but introduces new usability challenges. Int. J. Hum.-Comput. Stud. 67(1), 111–124 (2009)CrossRefGoogle Scholar
  14. 14.
    Cortier, V., Galindo, D., Küsters, R., Mueller, J., Truderung, T.: SoK: verifiability notions for e-voting protocols. In: Symposium on Security and Privacy (S&P), pp. 779–798. IEEE (2016)Google Scholar
  15. 15.
    Culnane, C., Teague, V.: Strategies for voter-initiated election audits. In: Zhu, Q., Alpcan, T., Panaousis, E., Tambe, M., Casey, W. (eds.) GameSec 2016. LNCS, vol. 9996, pp. 235–247. Springer, Cham (2016). Scholar
  16. 16.
    Distler, V., Zollinger, M.L., Lallemand, C., Rønne, P.B., Ryan, P.Y., Koenig, V.: Security-visible, yet unseen? How displaying security mechanisms impacts user experience and perceived security. In: CHI Conference on Human Factors in Computing Systems, CHI 2019. ACM (2019)Google Scholar
  17. 17.
    Escala, A., Guasch, S., Herranz, J., Morillo, P.: Universal cast-as-intended verifiability. In: Clark, J., Meiklejohn, S., Ryan, P.Y.A., Wallach, D., Brenner, M., Rohloff, K. (eds.) FC 2016. LNCS, vol. 9604, pp. 233–250. Springer, Heidelberg (2016). Scholar
  18. 18.
    Everett, S.P., Byrne, M.D., Greene, K.K.: Measuring the usability of paper ballots: efficiency, effectiveness, and satisfaction. In: Proceedings of the Human Factors and Ergonomics Society Annual Meeting, vol. 50, no. 24, pp. 2547–2551 (2006)Google Scholar
  19. 19.
    Everett, S.P., et al.: Electronic voting machines versus traditional methods: improved preference, similar performance. In: SIGCHI Conference on Human Factors in Computing Systems (CHI), pp. 883–892. ACM (2008)Google Scholar
  20. 20.
    Fuglerud, K.S., Røssvoll, T.H.: An evaluation of web-based voting usability and accessibility. Univ. Access Inf. Soc. 11(4), 359–373 (2012). Scholar
  21. 21.
    Gibson, J.P., MacNamara, D., Oakley, K.: Just like paper and the 3-colour protocol: a voting interface requirements engineering case study. In: International Workshop on Requirements Engineering for Electronic Voting Systems, pp. 66–75. IEEE (2011)Google Scholar
  22. 22.
    Gjøsteen, K.: The Norwegian internet voting protocol. In: Kiayias, A., Lipmaa, H. (eds.) Vote-ID 2011. LNCS, vol. 7187, pp. 1–18. Springer, Heidelberg (2012). Scholar
  23. 23.
    Greene, K.K., Byrne, M.D., Everett, S.P.: A comparison of usability between voting methods. In: Electronic Voting Technology Workshop (EVT). USENIX Association (2006)Google Scholar
  24. 24.
    Grimm, P.: Social desirability bias. Wiley International Encyclopedia of Marketing. Wiley, Hoboken (2010)CrossRefGoogle Scholar
  25. 25.
    Herrnson, P.S., Niemi, R.G., Hanmer, M.J., Bederson, B.B., Conrad, F.G., Traugott, M.: The importance of usability testing of voting systems. In: Electronic Voting Technology Workshop (EVT) (2006)Google Scholar
  26. 26.
    de Jong, M., van Hoof, J., Gosselt, J.: User research of a voting machine: Preliminary findings and experiences. J. Usability Stud. 2(4), 180–189 (2007)Google Scholar
  27. 27.
    Karayumak, F., Kauer, M., Olembo, M.M., Volk, T., Volkamer, M.: User study of the improved Helios voting system interfaces. In: Workshop on Socio-Technical Aspects in Security and Trust (STAST), pp. 37–44. IEEE (2011)Google Scholar
  28. 28.
    Kulyk, O., Neumann, S., Budurushi, J., Volkamer, M.: Nothing comes for free: How much usability can you sacrifice for security? IEEE Secur. Priv. 15(3), 24–29 (2017)CrossRefGoogle Scholar
  29. 29.
    Laskowski, S.J., Autry, M., Cugini, J., Killam, W., Yen, J.: Improving the usability and accessibility of voting systems and products. NIST Spec. Publ. 500, 256 (2004)Google Scholar
  30. 30.
    Laugwitz, B., Held, T., Schrepp, M.: Construction and evaluation of a user experience questionnaire. In: Holzinger, A. (ed.) USAB 2008. LNCS, vol. 5298, pp. 63–76. Springer, Heidelberg (2008). Scholar
  31. 31.
    Mac Namara, D., Scully, T., Gibson, P.: Dualvote addressing usability and verifiability issues in electronic voting systems (2011).
  32. 32.
    MacNamara, D., Carmody, F., Scully, T., Oakley, K., Quane, E., Gibson, J.P.: Dual vote: a novel user interface for e-voting systems. In: International Conference on Interfaces and Human Computer Interaction, pp. 129–138. IADIS (2010)Google Scholar
  33. 33.
    MacNamara, D., Gibson, P., Oakley, K.: A preliminary study on a dualvote and Prêt à Voter hybrid system. In: Conference for E-Democracy and Open Government, p. 77. Edition-Donau-Univ. Krems (2012)Google Scholar
  34. 34.
    Marky, K., Kulyk, O., Renaud, K., Volkamer, M.: What did I really vote for? On the usability of verifiable e-voting schemes. In: CHI Conference on Human Factors in Computing Systems (CHI), pp. 176:1–176:13. ACM (2018)Google Scholar
  35. 35.
    Marky, K., Kulyk, O., Volkamer, M.: Comparative usability evaluation of cast-as-intended verification approaches in internet voting. In: SICHERHEIT 2018, pp. 197–208. Lecture Notes in Informatics (LNI), Gesellschaft für Informatik (2018)Google Scholar
  36. 36.
    Marky, K., Schmitz, M., Lange, F., Mühlhäuser, M.: Usability of code voting modalities. In: Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems, CHI EA 2019, pp. LBW2221:1–LBW2221:6. ACM (2019)Google Scholar
  37. 37.
    Nancarrow, C., Brace, I.: Saying the “right thing”: coping with social desirability bias in marketing research. Bristol Bus. Sch. Teach. Res. Rev. 3(11), 1–11 (2000)Google Scholar
  38. 38.
    Neumann, S.: Evaluation and improvement of internet voting schemes based on legally-founded security requirements. Ph.D. thesis, Technische Universität Darmstadt (2016)Google Scholar
  39. 39.
    Olembo, M.M., Volkamer, M.: E-voting system usability: Lessons for interface design, user studies, and usability criteria. In: Human-Centered System Design for Electronic Governance, pp. 172–201. IGI Global (2013)Google Scholar
  40. 40.
    Patrick, A.: Ecological validity in studies of security and human behaviour. In: Symposium on Usable Privacy and Security (SOUPS) (2009)Google Scholar
  41. 41.
    Realpe-Muñoz, P., Collazos, C.A., Hurtado, J., Granollers, T., Muñoz-Arteaga, J., Velasco-Medina, J.: Eye tracking-based behavioral study of users using e-voting systems. Comput. Stan. Interfaces 55, 182–195 (2017)CrossRefGoogle Scholar
  42. 42.
    Selker, T., Rosenzweig, E., Pandolfo, A.: A methodology for testing voting systems. J. Usability Stud. 2(1), 7–21 (2006)Google Scholar
  43. 43.
    Sherman, A.T., et al.: Scantegrity mock election at Takoma Park. In: International Conference on Electronic Voting (EVOTE), pp. 45–61. LNI, Gesellschaft für Informatik (2010)Google Scholar
  44. 44.
    Standardization, I.O.F.: ISO 9241–11: Ergonomics of human system interaction - part 11: Guidance on usability (1998)Google Scholar
  45. 45.
    Standardization, I.O.F.: ISO 9241–210: Part 210: Human-centred design for interactive systems (2015)Google Scholar
  46. 46.
    Strafgesetzbuch (StGB): §107c Verletzung des Wahlgeheimnisses.
  47. 47.
    Van Hoof, J.J., Gosselt, J.F., de Jong, M.D.: The reliability and usability of the NEDAP voting machine: a pilot study. University of Twente Faculty of Behavioural Sciences Department of Technical and Professional Communication (2007)Google Scholar
  48. 48.
    Weber, J.L., Hengartner, U.: Usability study of the open audit voting system Helios. (2009)
  49. 49.
    Winckler, M., et al.: Assessing the usability of open verifiable e-voting systems: a trial with the system prêt à voter. In: International Conference on Theory and Practice of Electronic Governance (ICEGOV), pp. 281–296. ACM (2009)Google Scholar

Copyright information

© International Financial Cryptography Association 2020

Authors and Affiliations

  1. 1.Telecooperation LabTechnische Universität DarmstadtDarmstadtGermany
  2. 2.University of LuxembourgLuxembourg CityLuxembourg

Personalised recommendations