Skip to main content

Consigliere Evaluation: Evaluating Complex Interactive Systems with Users with Disabilities

  • 1224 Accesses

Part of the Lecture Notes in Computer Science book series (LNISA,volume 12376)


Conducting accessibility evaluations with users with disabilities is an important part of developing accessible interactive systems. Conducting such evaluations of systems which require complex domain knowledge is often impossible, as disabled users do not exist or are very rare. This paper presents a user evaluation method to address this problem, consigliere evaluation. A consigliere evaluation has the disabled user as the main participant, but they are assisted by an advisor, or consigliere, who understands the complex domain; the consigliere is in turn, monitored by an accessibility expert, who acts as an enforcer. As in all user evaluations, the disabled participant undertakes a series of tasks. But in a consigliere evaluation, if the task requires some particular domain knowledge or skill, the role of the consigliere is to provide appropriate information. However, it is easy for the consigliere, who usually does not have knowledge of the accessibility domain, to provide information not specifically about the domain, but about how to do the task in general. So the role of the enforcer, who is an accessibility expert, is to ensure this does not happen, and also to provide assistance and explanation if accessibility issues arise that the disabled participant cannot solve. The paper illustrates the consigliere method with a case study, the evaluation of Skillsforge, an online system used by a number of universities to manage progress of postgraduate students. This system requires considerable domain knowledge of the terminology and progression requirements for these students. It is used by university administrative staff, academic staff who supervise postgraduate students or are involved in monitoring students and the students themselves. The case study illustrates how the consigliere evaluation method works and some of the things which need to be considered in conducting the evaluation appropriately.


  • Evaluation methods
  • Disabled users
  • Complex systems
  • User evaluations

This is a preview of subscription content, access via your institution.

Buying options

USD   29.95
Price excludes VAT (USA)
  • DOI: 10.1007/978-3-030-58796-3_5
  • Chapter length: 8 pages
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
USD   84.99
Price excludes VAT (USA)
  • ISBN: 978-3-030-58796-3
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
Softcover Book
USD   109.99
Price excludes VAT (USA)


  1. 1.

    The use of this term in no way implies admiration for the mafia, but we were inspired by the Godfather film in which the consigliere whispers advice in the ear of the mafia boss.


  1. World Wide Web Consortium – Web Accessibility Initiative (W3C WAI): Web Content accessibility Guidelines (WCAG) 2.1 (2018). Accessed 12 Apr 2020

  2. Android Developers. Make apps more accessible. Accessed 12 Apr 2020

  3. Apple Human Interface Guidelines. Accessibility. Accessed 12 Apr 2020

  4. Centre for Excellence in Universal Design. Public access terminals, 1 July 2019 Accessed 12 Apr 2020

  5. Centre for Excellence in Universal Design. Telecoms. Accessed 12 Apr 2020

  6. AbleGamers. Design patterns for APX. Accessed 12 Apr 2020

  7. Sani, Z.H.A., Petrie, H.: User evaluation of an app for liquid monitoring by older adults. In: Antona, M., Stephanidis, C. (eds.) UAHCI 2017. LNCS, vol. 10279, pp. 86–97. Springer, Cham (2017).

    CrossRef  Google Scholar 

  8. Power, C., Freire, A., Petrie, H., Swallow, D.: Guidelines are only half the story: accessibility problems encountered by blind users on the web. In: Proceedings of the 30th International Conference on Human Factors in Computing (CHI 2012). ACM Press, New York (2012)

    Google Scholar 

  9. Petrie, H., Bevan, N.: The evaluation of accessibility, usability and user experience. In: Stephanidis, C. (ed.) The Universal Access Handbook. Taylor and Francis, London (2009)

    Google Scholar 

  10. Sears, A., Hanson, V.L.: Representing users in accessibility research. ACM Trans. Access. Comput. 4(2), 7 (2012)

    CrossRef  Google Scholar 

  11. Aizpurua, A., Arrue, M., Harper, S., Vigo, M.: Are users the gold standard for accessibility evaluation? In: W4A: Proceedings of the 11th Web for All Conference (2014). Article 13

    Google Scholar 

  12. Power, C., Petrie, H.: Working with participants. In: Yesilada, Y., Harper, S. (eds.) Web Accessibility. HIS, pp. 153–168. Springer, London (2019).

    CrossRef  Google Scholar 


Download references


We would like to thank the participants who took part in this evaluation, they helped us clarify the consigliere evaluation method. We would also like to thank Skillsforge for the opportunity to work with them and the Engineering and Physical Sciences Research Council (EPSRC) and the University of York for funding the research through the Innovation Voucher Programme.

Author information

Authors and Affiliations


Corresponding author

Correspondence to Helen Petrie .

Editor information

Editors and Affiliations

Rights and permissions

Reprints and Permissions

Copyright information

© 2020 Springer Nature Switzerland AG

About this paper

Verify currency and authenticity via CrossMark

Cite this paper

Petrie, H., Samaddar, S., Power, C., Merdenyan, B. (2020). Consigliere Evaluation: Evaluating Complex Interactive Systems with Users with Disabilities. In: Miesenberger, K., Manduchi, R., Covarrubias Rodriguez, M., Peňáz, P. (eds) Computers Helping People with Special Needs. ICCHP 2020. Lecture Notes in Computer Science(), vol 12376. Springer, Cham.

Download citation

  • DOI:

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-58795-6

  • Online ISBN: 978-3-030-58796-3

  • eBook Packages: Computer ScienceComputer Science (R0)