Why Johnny the Developer Can’t Work with Public Key Certificates

An Experimental Study of OpenSSL Usability
  • Martin Ukrop
  • Vashek Matyas
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10808)


There have been many studies exposing poor usability of security software for the common end user. However, only a few inspect the usability challenges faced by more knowledgeable users. We conducted an experiment to empirically assess usability of the command line interface of OpenSSL, a well known and widely used cryptographic library. Based on the results, we try to propose specific improvements that would encourage more secure behavior. We observed 87 developers/administrators at two certificate-related tasks in a controlled environment. Furthermore, we collected participant opinions on both the tool interface and available documentation. Based on the overall results, we deem the OpenSSL usability insufficient according to both user opinions and standardized measures. Moreover, the perceived usability seems to be correlated with previous experience and used resources. There was a great disproportion between the participant views of a successful task accomplishment and the reality. A general dissatisfaction with both OpenSSL interface and its manual page was shared among the majority of the participants. As hinted by a participant, OpenSSL gradually “turned into a complicated set of sharp kitchen knives” – it can perform various jobs very well, but laymen risk stabbing themselves in the process. This highlights the necessity of a usable design even for tools targeted at experienced users (Supplementary material available at



This work has been supported by Red Hat Czech and done in collaboration with Red Hat crypto team. We are particularly grateful to Nikos Mavrogiannopoulos and Jan Pazdziora for insightful ideas, to Lenka Horáková, Vlasta Št’avová and Agáta Dařbujánová for their help with the experiment and to Lujo Bauer and Martin Preisler for comments on the paper draft. Vashek Matyas thanks Red Hat Czech and CyLab, Carnegie Mellon University for a supportive sabbatical environment and the Czech Science Foundation project GBP202/12/G061 for partial funding. We also thank all experiment participants.


  1. 1.
    Internet Archive: Wayback Machine.
  2. 2.
  3. 3.
    Man page search on Gentoo.
  4. 4.
  5. 5.
    OpenSSL: Cryptography and SSL/TLS Toolkit.
  6. 6.
    The GnuTLS Transport Layer Security Library.
  7. 7.
    Acar, Y., Backes, M., Fahl, S., Garfinkel, S., Kim, D., Mazurek, M., Stransky, C.: Comparing the usability of cryptographic APIs. In: 2017 IEEE Symposium on Security and Privacy. IEEE (2017)Google Scholar
  8. 8.
    Acar, Y., Backes, M., Fahl, S., Kim, D., Mazurek, M., Stransky, C.: You get where you’re looking for: the impact of information sources on code security. In: 2016 IEEE Symposium on Security and Privacy, pp. 289–305. IEEE (2016)Google Scholar
  9. 9.
    Barker, E., Dang, Q.: NIST SP 800–57 recommendation for key management part 3: application-specific key management guidance. Technical report (2015)Google Scholar
  10. 10.
    Bernstein, D.J., Lange, T., Schwabe, P.: The security impact of a new cryptographic library. In: Hevia, A., Neven, G. (eds.) LATINCRYPT 2012. LNCS, vol. 7533, pp. 159–176. Springer, Heidelberg (2012). CrossRefGoogle Scholar
  11. 11.
    Brooke, J.: SUS - a quick and dirty usability scale. Usability Eval. Indus. 189(194), 4–7 (1996)Google Scholar
  12. 12.
    Egele, M., Brumley, D., Fratantonio, Y., Kruegel, C.: An empirical study of cryptographic misuse in android applications. In: Proceedings of the 2013 ACM Conference on Computer and Communications Security, pp. 73–84. ACM Press (2013)Google Scholar
  13. 13.
    Fischer, F., Bottinger, K., Xiao, H., Stransky, C., Acar, Y., Backes, M., Fahl, S.: Stack overflow considered harmful? The impact of copy & paste on android application security. In: 2017 IEEE Symposium on Security and Privacy. IEEE (2017)Google Scholar
  14. 14.
    Garfinkel, S., Miller, R.: Johnny 2: a user test of key continuity management with S/MIME and outlook express. In: Proceedings of the 2005 Symposium on Usable Privacy and Security, pp. 13–24. ACM Press (2005)Google Scholar
  15. 15.
    Georgiev, M., Iyengar, S., Jana, S., Anubhai, R., Boneh, D., Shmatikov, V.: The most dangerous code in the world: validating SSL certificates in non-browser software. In: Proceedings of the 2012 ACM Conference on Computer and Communications Security, pp. 38–49. ACM Press (2012)Google Scholar
  16. 16.
    Horáková, L.: User interface design for certificate operations with network securityservices. Master thesis. Masaryk University (2017)Google Scholar
  17. 17.
    Krombholz, K., Mayer, W., Schmiedecker, M., Weippl, E.: “I Have No Idea What I’m Doing” - on the usability of deploying HTTPS. In: Proceedings of the 26th USENIX Security Symposium. USENIX Association (2017)Google Scholar
  18. 18.
    Lazar, D., Chen, H., Wang, X., Zeldovich, N.: Why does cryptographic software fail? In: Proceedings of 5th Asia-Pacific Workshop on Systems, pp. 7:1–7:7. ACM Press (2014)Google Scholar
  19. 19.
    Lethbridge, T., Singer, J., Forward, A.: How software engineers use documentation: the state of the practice. IEEE Softw. 20(6), 35–39 (2003)CrossRefGoogle Scholar
  20. 20.
    McLellan, S., Muddimer, A., Peres, C.: The effect of experience on system usability scale ratings. J. Usability Stud. 7(2), 56–67 (2012)Google Scholar
  21. 21.
    Nemec, M., Klinec, D., Svenda, P., Sekan, P., Matyas, V.: Measuring popularity of cryptographic libraries in internet-wide scans. In: Proceedings of the 33rd Annual Computer Security Applications Conference, ACSAC 2017, pp. 162–175. ACM (2017)Google Scholar
  22. 22.
    Robillard, M.: What makes APIs hard to learn? Answers from developers. IEEE Softw. 26(6), 27–34 (2009)CrossRefGoogle Scholar
  23. 23.
    Sheng, S., Broderick, L., Koranda, C., Hyland, J.: Why johnny still can’t encrypt: evaluating the usability of email encryption software. In: Proceedings of the 2006 Symposium on Usable Privacy and Security, pp. 3–4. ACM Press (2006)Google Scholar
  24. 24.
    Sheskin, D.: Handbook of Parametric and Nonparametric Statistical Procedures, 4th edn. Chapman and Hall/CRC, Boca Raton (2007)zbMATHGoogle Scholar
  25. 25.
    Subramanian, S., Inozemtseva, L., Holmes, R.: Live API documentation. In: Proceedings of the 36th International Conference on Software Engineering, pp. 643–652. ACM Press (2014)Google Scholar
  26. 26.
    Uddin, G., Robillard, M.P.: How API documentation fails. IEEE Softw. 32(4), 68–75 (2015)CrossRefGoogle Scholar
  27. 27.
    Whitten, A., Tygar, J.: Why johnny can’t encrypt: a usability evaluation of PGP 5.0. In: Proceedings of the 8th USENIX Security Symposium, vol. 8, pp. 169–184. USENIX Association (1999)Google Scholar

Copyright information

© Springer International Publishing AG, part of Springer Nature 2018

Authors and Affiliations

  1. 1.Centre for Research on Cryptography and Security, Faculty of InformaticsMasaryk UniversityBrnoCzechia

Personalised recommendations