Redesign Based on Card Sorting: How Universally Applicable are Card Sort Results?

  • Jobke WentzelEmail author
  • Nienke Beerlage de Jong
  • Thea van der Geest
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9745)


Card sort studies can facilitate developers to create an information structure for their website or application. In addition, this human-centered design method provides researchers with insights into the target group’s mental models regarding the information domain under study. In this method, participants sort cards, with excerpts of the website’s or information source’s information on them, into piles or groups. Even though the method lends itself for large numbers of participants, it can be difficult to include sufficient participants in a study to ensure generalizability among large user groups. Especially when the potential user group is heterogeneous, basing the information structure on a limited participant group may not always be valid. In this study, we investigate if card-sort results among one user group (nurses) are comparable to the results of a second (potential) user group (physicians/residents).

The results of a formative card sort study that were used to create an antibiotic information application are compared to the results of a second card sort study. This second study was conducted with the aim of redesigning the nurse-aimed information application to meet the (overlapping) needs of physicians. During the first card sort study, 10 nurses participated. In the second card sort study, 8 residents participated. The same set of 43 cards were used in both setups. These cards contain fragments of antibiotic protocols and reference documents that nurses and physicians use to be informed about the use and administration of antibiotics. The participants sorted the cards in individual sessions, into as many categories as they liked. The sorts of both user groups were analyzed separately. Dendrograms and similarity matrices were generated using the Optimal Sort online program. Based on the matrices, clusters were identified by two independent researchers. On these resulting clusters of cards, overlap scores were calculated (between nurse and resident clusters). Differences are compared.

The results show that overall, residents reached higher agreement than the nurses. Some overlap between categories is observed in both card sort data matrices. Based on the nurses’ data, more and more specific clusters were created (which in part were observed in the larger residents’ clusters).

Based on our findings we conclude that a redesign may not be necessary. Especially when the target group with the lowest prior knowledge levels of the information domain is included in the card sort study, the results can be translated to other groups as well. However, groups with little knowledge will more likely result in lower agreement in the card sorts. Therefore, a larger sample and/or including participants with low and high knowledge of the information domain is advisable.


Card sorting Re-design Human-centered design Information architecture 



We thank the nurses and medical residents of the Medisch Spectrum Twente hospital, in Enschede, the Netherlands, for participating in this study. This study was conducted within the Interreg IVa-funded project EurSafety Heath-net (III-1-02  =  73). This is a Dutch-German cross-border network supported by the European Commission, the Federal States of Nordrhein-Westfalen and Niedersachsen(Germany), and the Dutch provinces of Overijssel, Gelderland, and Limburg. Part of the study was executed within a project (Google Glass For VIPS), funded as a Tech4People 2015 grant by the faculty BMS of the University of Twente.


  1. 1.
    Maguire, M.: Methods to support human-centred design. Int. J. Hum Comput Stud. 55, 587–634 (2001)CrossRefzbMATHGoogle Scholar
  2. 2.
    Morville, P., O’reilly, T.: Information Architecture for the World Wide Web. O’Reilly, Sebastopol (2007)Google Scholar
  3. 3.
    Van Velsen, L., Wentzel, J., Van Gemert-Pijnen, J.E.W.C.: Designing eHealth that matters via a multidisciplinary requirements development approach. JMIR Res. Protoc. 2(1), e21 (2013)CrossRefGoogle Scholar
  4. 4.
    Verhoeven, F., Karreman, J., Bosma. A., Hendrix, H.R.M., van Gemert-Pijnen, L.E.W.C.: Toward improved education of the public about methicillin-resistant Staphylococcus aureus: a mental models approach. Inter. J. Infect. Control 6(1) (2010)Google Scholar
  5. 5.
    Verhoeven, F., Hendrix, R.M., Daniels-Haardt, I., Friedrich, A.W., Steehouder, M.F., van Gemert-Pijnen, J.E.: The development of a web-based information tool for cross-border prevention and control of Methicillin Resistant Staphylococcus aureus. Inter. J. Infect. Control 4(1), 11 (2008)CrossRefGoogle Scholar
  6. 6.
    Wentzel, J., van Velsen, L., van Limburg, M., de Jong, N., Karreman, J., Hendrix, R., van Gemert, J.E.W.C.: Participatory eHealth development to support nurses in antimicrobial stewardship. BMC Med. Inf. Decis. Making 14(1), 45 (2014)CrossRefGoogle Scholar
  7. 7.
    Wentzel, J., van Drie-Pierik, R., Nijdam, L., Geesing, J., Sanderman, R., van Gemert-Pijnen, J.: Antibiotic information application offers nurses quick support. Am. J. Inf. Control (2016). (in press)Google Scholar
  8. 8.
    Optimal Workshop - Optimal sort.
  9. 9.
    Capra, M.G.: Factor analysis of card sort data: an alternative to hierarchical cluster analysis. Proc. Hum. Factors Ergon. Soc. Ann. Meet. 49(5), 691–695 (2005). SAGE PublicationsCrossRefGoogle Scholar
  10. 10.
  11. 11.
  12. 12.
    Wentzel, J., Müller, F., Beerlage-de Jong, N., van Gemert-Pijnen, J.: Card sorting to evaluate the robustness of the information architecture of a protocol website. Inter. J. Med. Inf. 86, 71–81 (2016)CrossRefGoogle Scholar
  13. 13.
    Nielsen, J.: Card Sorting: How Many Users to Test (2004). Accessed 2 March 2015
  14. 14.
    Coxon, A.P.M.: Sorting data: Collection and analysis, vol. 127. Sage Publications, Thousand Oaks (1999)CrossRefzbMATHGoogle Scholar

Copyright information

© Springer International Publishing Switzerland 2016

Authors and Affiliations

  • Jobke Wentzel
    • 1
    Email author
  • Nienke Beerlage de Jong
    • 2
  • Thea van der Geest
    • 1
  1. 1.Department of Media, Communication and OrganisationUniversity of TwenteEnschedeThe Netherlands
  2. 2.Department of Psychology, Health and TechnologyUniversity of TwenteEnschedeThe Netherlands

Personalised recommendations