A Grounded Procedure for Managing Data and Sample Size of a Home Medical Device Assessment

  • Simone Borsci
  • Jennifer L. Martin
  • Julie Barnett
Part of the Lecture Notes in Computer Science book series (LNCS, volume 8004)


The selection of participants for usability assessment, together with the minimum number of subjects required to obtain a set of reliable data, is a hot topic in Human Computer Interaction (HCI). Albeit, prominent contributions through the application of different p estimation models argued that five users provide a good benchmark when seeking to discover interaction problems a lot of studies have complained this five-user assumption. The sample size topic is today a central issue for the assessment of critical-systems, such as medical devices, because lacks in usability and, moreover, in the safety in use of these kind of products may seriously damage the final users. We argue that rely on one-size-fits-all solutions, such as the five-user assumption (for websites) or the mandated size of 15 users for major group (for medical device) lead manufactures to release unsafe product. Nevertheless, albeit there are no magic numbers for determining “a priori” the cohort size, by using a specific procedure it is possible to monitoring the sample discovery likelihood after the first five users in order to obtain reliable information about the gathered data and determine whether the problems discovered by the sample have a certain level of representativeness (i.e., reliability). We call this approach “Grounded Procedure” (GP).The goal of this study is to present the GP assumptions and steps, by exemplifying its application in the assessment of a home medical device.


discovery likelihood medical device sample size usability testing 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Streitz, N.: From cognitive compatibility to the disappearing computer: experience design for smart environments. In: Proceedings of the 15th European Conference on Cognitive Ergonomics: the Ergonomics of Cool Interaction, pp. 1–2. ACM, Funchal (2008)CrossRefGoogle Scholar
  2. 2.
    Soylu, A., Causmaecker, P.D., Desmet, P.: Context and Adaptivity in Pervasive Computing Environments: Links with Software Engineering and Ontological Engineering. Journal of Software 4, 992–1013 (2009)CrossRefGoogle Scholar
  3. 3.
    Herman, W.A., Devey, G.B.: Future Trends in Medical Device Technologies: A Ten-Year Forecast. Food and Drug Administration, Center for Devices and Radiological Health (2011)Google Scholar
  4. 4.
    IEC: IEC 62366: 2007 Medical devices – Application of usability engineering to medical devices. CEN, Brussels, BE (2007) Google Scholar
  5. 5.
    ISO: ISO 14971:2000 Medical devices – Application of risk management to medical devices. CEN, Brussels, BE (2000) Google Scholar
  6. 6.
    ISO: ISO 13485:2003 Medical devices – Quality management systems – Requirements for regulatory purposes. CEN, Brussels, BE (2003) Google Scholar
  7. 7.
    ISO: ISO 14971:2007 Medical devices – Application of risk management to medical devices. CEN, Brussels, BE (2007) Google Scholar
  8. 8.
    ISO: ISO 15223-1:2012 Medical devices – Symbols to be used with medical device labels, labelling and information to be supplied – Part 1: General requirements. CEN, Brussels, BE (2012) Google Scholar
  9. 9.
    ANSI/AAMI: HE75: Human factors engineering-Design of medical devices. Association for the Advancement of Medical Instrumentation, Arlington, VA (2009) Google Scholar
  10. 10.
    Martin, J., Norris, B.J., Murphy, E., Crowe, J.A.: Medical device development: The challenge for ergonomics. Applied Ergonomics 39, 271–283 (2008)CrossRefGoogle Scholar
  11. 11.
    Money, A., Barnett, J., Kuljis, J., Craven, M., Martin, J., Young, T.: The role of the user within the medical device design and development process: medical device manufacturers’ perspectives. BMC Medical Informatics and Decision Making 11, 15 (2011)CrossRefGoogle Scholar
  12. 12.
    Martin, J., Barnett, J.: Integrating the results of user research into medical device development: insights from a case study. BMC Medical Informatics and Decision Making 12, 74 (2012)CrossRefGoogle Scholar
  13. 13.
    Heneghan, C., Thompson, M., Billingsley, M., Cohen, D.: Medical-device recalls in the UK and the device-regulation process: retrospective review of safety notices and alerts. BMJ Open 1 (2011)Google Scholar
  14. 14.
    Food and Drug Administration (FDA): Draft Guidance for Industry and Food and Drug Administration Staff - Applying Human Factors and Usability Engineering to Optimize Medical Device Design. U.S. Food and Drug Administration, Silver Spring, MD (2011) Google Scholar
  15. 15.
    Faulkner, L.: Beyond the five-user assumption: Benefits of increased sample sizes in usability testing. Behavior Research Methods 35, 379–383 (2003)CrossRefGoogle Scholar
  16. 16.
  17. 17.
  18. 18.
    Nielsen, J., Landauer, T.K.: A mathematical model of the finding of usability problems. In: Proceedings of the INTERACT 1993 and CHI 1993 Conference on Human Factors in Computing Systems, pp. 206–213. ACM, Amsterdam (1993)Google Scholar
  19. 19.
    Virzi, R.A.: Streamlining the Design Process: Running Fewer Subjects. In: Proceedings of the Human Factors Society 34th Annual Meeting, vol. 34, pp. 291–294. ACM, Santa Monica (1990)Google Scholar
  20. 20.
    Virzi, R.A.: Refining the test phase of usability evaluation: how many subjects is enough? Human Factors 34, 457–468 (1992)Google Scholar
  21. 21.
    Borsci, S., Londei, A., Federici, S.: The Bootstrap Discovery Behaviour (BDB): a new outlook on usability evaluation. Cognitive Processing 12, 23–31 (2011)CrossRefGoogle Scholar
  22. 22.
    Caulton, D.A.: Relaxing the homogeneity assumption in usability testing. Behaviour & Information Technology 20, 1–7 (2001)CrossRefGoogle Scholar
  23. 23.
    Lewis, J.R.: Evaluation of Procedures for Adjusting Problem-Discovery Rates Estimated From Small Samples. International Journal of Human-Computer Interaction 13, 445–479 (2001)CrossRefGoogle Scholar
  24. 24.
    Schmettow, M.: Heterogeneity in the usability evaluation process. In: Proceedings of the 22nd British HCI Group Annual Conference on People and Computers: Culture, Creativity, Interaction, vol. 1, pp. 89–98. British Computer Society, Liverpool (2008)Google Scholar
  25. 25.
    Schmettow, M.: Sample size in usability studies. Communications of the ACM 55, 64–70 (2012)CrossRefGoogle Scholar
  26. 26.
    Turner, C.W., Lewis, J.R., Nielsen, J.: Determining Usability Test Sample Size. In: Karwowski, W. (ed.) International Encyclopedia of Ergonomics and Human Factors, vol. 2, pp. 3084–3088. CRC Press, Boca Raton (2006)Google Scholar
  27. 27.
    Lewis, J.R.: Validation of Monte Carlo estimation of problem discovery likelihood (Tech. Rep. No. 29.3357). IBM (2000) Google Scholar
  28. 28.
    Borsci, S., Macredie, R.D., Barnett, J., Martin, J., Kuljis, J., Young, T.: Reviewing and Extending the Five–user Assumption: A Grounded Procedure for Interaction Evaluation (manuscript submitted for publication, 2013)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2013

Authors and Affiliations

  • Simone Borsci
    • 1
  • Jennifer L. Martin
    • 2
  • Julie Barnett
    • 1
  1. 1.School of Information Systems, Computing and MathematicsBrunel UniversityUxbridgeUK
  2. 2.Department of Electrical and Electronic EngineeringUniversity of NottinghamUniversity ParkUK

Personalised recommendations