What Difference Do Guidelines Make? An Observational Study of Online-questionnaire Design Guidelines Put to Practical Use

  • Jo Lumsden
  • Scott Flinn
  • Michelle Anderson
  • Wendy Morgan
Conference paper


As a new medium for questionnaire delivery, the Internet has the potential to revolutionize the survey process. Online-questionnaires can provide many capabilities not found in traditional paper-based questionnaires. Despite this, and the introduction of a plethora of tools to support online-questionnaire creation, current electronic survey design typically replicates the look-and-feel of paper-based questionnaires, thus failing to harness the full power of the electronic delivery medium. A recent environmental scan of online-questionnaire design tools found that little, if any, support is incorporated within these tools to guide questionnaire designers according to best-practice [Lumsden & Morgan 2005]. This paper briefly introduces a comprehensive set of guidelines for the design of online-questionnaires. Drawn from relevant disparate sources, all the guidelines incorporated within the set are proven in their own right; as an initial assessment of the value of the set of guidelines as a practical reference guide, we undertook an informal study to observe the effect of introducing the guidelines into the design process for a complex online-questionnaire. The paper discusses the qualitative findings — which are encouraging for the role of the guidelines in the ‘bigger picture’ of online survey delivery across many domains such as e-government, e-business, and e-health — of this case study.


online-questionnaire design guidelines evaluative case study 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. American Statistical Association [1999], American Statistical Association Series: What is a Survey?, (retrieved 2003-06-07).Google Scholar
  2. Badre, A. N. [2002], Shaping Web Usability: Interaction Design in Context, Pearson Education.Google Scholar
  3. Bandilla, W., Bosnjak, M. & Altdorfer, P. [2003], Self Administration Effects? A Comparison of Web-Based and Traditional Written Self-Administered Surveys Using the ISSP Environment Module, Social Science Computing Review 21(2), 235–43.CrossRefGoogle Scholar
  4. Belson, W. A. [1981], The Design and Understanding of Survey Questions, Gower Publishing.Google Scholar
  5. Brewer, J. [2001], How People with Disabilities Use the Web, W3C Working Draft, W3C. See current and previous versions.Google Scholar
  6. CASRO [1998], Guidelines for Survey Research Quality, (retrieved 2003-06-07). Council of American Survey Research Organization.Google Scholar
  7. Coyne, K. P. & Nielsen, J. [2001], Beyond ALT Text: Making the Web Easy to Use for Users with Disabilities, Technical Report, Nielsen Norman Group.Google Scholar
  8. Coyne, K. P. & Nielsen, J. [2002], Web Usability for Senior Citizens, Technical Report, Nielsen Norman Group.Google Scholar
  9. Crawford, S. D., Couper, M. P. & Lamias, M. J. [2001], Web Surveys: Perceptions of Burden, Social Science Computing Review 19(2), 146–62.CrossRefGoogle Scholar
  10. Dillman, D. A. [1978], Mail and Telephone Surveys: The Total Design Method, John Wiley & Sons.Google Scholar
  11. Dillman, D. A. [2000], Mail and Internet Surveys: The Tailored Design Method, second edition, John Wiley & Sons.Google Scholar
  12. Fink, A. [1995], How to Ask Survey Questions, Sage Publications.Google Scholar
  13. Hinton, S. M. [1998], From Home Page to Home Site: Effective Web Resource Discovery at the ANU, in H. Ashman & P. Thistlewaite (eds.), Proceedings of the Seventh International World Wide Web Conference (WWW7), Vol. 30(17–7) of Computer Networks and ISDN Systems, Elsevier Science, pp.309–16. See also Scholar
  14. Jackson, W. [1988], Research Methods: Rules for Survey Design and Analysis, Prentice-Hall.Google Scholar
  15. Kothari, R. & Basak, J. [2002], Perceptually Automated Evaluation of Web Page Layouts, in Paper Presented in an Alternative Track of the Eleventh International World Wide Web Conference (WWW2002). Scholar
  16. Kwak, N. & Radler, B. [2002], A Comparison Between Mail and Web Surveys: Response Pattern, Respondent Profile and Data Quality, Journal of Official Statistics 18(2), 257–74.Google Scholar
  17. Lindgaard, G. [1994], Usability Testing and System Evaluation: A Guide for Designing Useful Computer Systems, Chapman & Hall.Google Scholar
  18. Lumsden, J. & Morgan, W. [2005], Online Questionnaire Design: Establishing Guidelines and Evaluating Existing Support, in M. Khosrow-Pour (ed.), Managing Modern Organizations with Information Technology: Proceedings of the 16th Information Resources Management Association International Conference (IRMA 2005), IRM Press, pp.407–10.Google Scholar
  19. Lynch, P. J. & Horton, S. [1997], Web Style Guide, Yale University Press. See also Scholar
  20. National Cancer Institute [2002], National Cancer Institute’s Research Based Web Design & Usability Guidelines, (retrieved 2003-06-10).Google Scholar
  21. National Institute on Aging & National Library of Medicine [2001], Making Your Web Site Senior Friendly, (retrieved 2003-06-19).Google Scholar
  22. Norman, K. L., Lee, S., Moore, P., Murry, G. C., Rivadeneira, W., Smith, B. K. & Verdines, P. [2003], Online Survey Design Guide, (retrieved 2003-06-17).Google Scholar
  23. OECD [2001], Bridging the “Digital Divide”: Issues and Policies in OECD Countries, http://www.oecd.Org/dataoecd/10/0/27128723.pdf (retrieved 2003-06-03).Google Scholar
  24. Oppenheim, A. N. [1992], Questionnaire Design, Interviewing and Attitude Measurement, Pinter Publishers.Google Scholar
  25. Schonlau, M., Fricker, R. D. & Elliott, M. N. [2001], Conducting Research via E-mail and the Web, 2003-06-16).Google Scholar
  26. Stover, A., Coyne, K. P. & Nielsen, J. [2002], Designing Usable Site Maps for Websites, Technical Report, Nielsen Norman Group.Google Scholar
  27. Stover, A. & Nielsen, J. [2002], Accessibility and Usability of Flash for Users with Disabilities, Technical Report, Nielsen Norman Group.Google Scholar
  28. Taylor-Powell, E. [1998], Questionnaire Design: Asking Questions with a Purpose, Technical Report G3658-2, University of Wisconsin.Google Scholar
  29. W3C [1999], Web Content Accessibility Guidelines 1.0, (retrieved 2003-06-08).Google Scholar

Copyright information

© Springer-Verlag London Limited 2006

Authors and Affiliations

  • Jo Lumsden
    • 1
  • Scott Flinn
    • 1
  • Michelle Anderson
    • 1
  • Wendy Morgan
    • 1
  1. 1.National Research Council of CanadaFrederictonCanada

Personalised recommendations