Skip to main content
SpringerLink
Log in

Using cognitive task analysis to explore issues in the procurement of intelligent decision support systems

  • Original Article
  • Published:
Cognition, Technology & Work Aims and scope Submit manuscript

Abstract

Government statistics and various news reports suggest that upwards of half of all large-scale information technology (IT) development projects fail to meet expectations for facilitating cognitive work. Many of the failures point to the neglect of human-centering considerations during the development of sociotechnical systems. The groups of people who create IT themselves constitute a sociotechnical system. Therefore, laws of cognitive work apply to the cognitive work of IT development, and these laws include the “reductive tendency” for people to form simplified understandings when confronted with domains of dynamics and complexity. In this article, we report a study in which we “turned the tables” on IT systems development. Rather than using cognitive task analysis to study some work domain for which an envisioned IT system would be developed, we used cognitive task analysis to study the work domain of IT systems development itself. Through documentation analysis and critical decision method procedures, we sought to reveal specific challenges with regard to human-centering, and ways in which principles, methods, and tools of ergonomics (human factors, cognitive systems engineering) might help the developers of IT systems better address the human and social aspects of cognitive work. The findings highlight the outstanding challenges and barriers to the procurement and development of usable, useful, and understandable IT for sociotechnical systems. Challenges include the following: the need for better coordination mechanisms; the need to locate cognitive systems engineers, as advocates for workers, in key leadership roles; the need to reconceive concepts and methods of requirements and requirements specification; and the need for better negotiation of the trade-offs of cost/schedule considerations with human-centering considerations.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

Notes

  1. Throughout this article we use the single phrase “cognitive task analysis” (CTA) to refer to methods and methodologies of work analysis, ethnography, sociotechnics, and cognitive work analysis as these methods are known in various communities of practice.

References

  • Bar-Yam Y (2003). When systems engineering fails—toward complex systems engineering. In: IEEE international conference on systems, man, and cybernetics, vol 2. IEEE Press, Piscataway, pp 2021–2028

  • Boehm B (1988) A spiral model of software development and enhancement. IEEE Comput 21:61–62

    Google Scholar 

  • Boehm B (2006) Some future trends and implications for systems and software engineering processes. Syst Eng 9:1–19

    Article  Google Scholar 

  • Boehm B, Hansen WJ (2001) The spiral model as a tool for evolutionary acquisition. Crosstalk 14(5):4–9. Retrieved from http://www.stsc.hill.af.mil/crossTalk/

    Google Scholar 

  • Bush M (1997) NIMROD case study. Retrieved from http://www.scism.sbu.ac.uk/~bushm/units/NIMROD_Case_Study.html

  • Carr DF, Cone E (2002) Can FAA salvage its IT disaster? Baseline, 8 April 2001. Retrieved from http://www.baselinemag.com/article2/0,1540,656862,00.asp

  • Cockburn A (2002) Learning from agile software development. Crosstalk 15:10–14. Retrieved from http://www.stsc.hill.af.mil/crosstalk/

    Google Scholar 

  • Coyle PE (1999). Simulation Based acquisition for information technology. Paper presented at the 1999 Academia, Industry, Government Crosstalk Conference, Washington, DC, May 1999

  • Crandall B, Klein G, Hoffman RR (2006) Working minds: a practitioner’s handbook of cognitive task analysis. MIT Press, Cambridge

    Google Scholar 

  • Eggen D (2005) FBI pushed ahead with troubled software. Washingtonpost.com, 6 June 2005

  • Eggleston RG, Gualtieri J, Burns C, Linern G, Zachary W, Wiggins S (2004) Tightening the linkage of CSE and software systems engineering. In: Proceedings of the Human Factors and Ergonomics Society 48th annual meeting. Santa Monica, CA: HFES. [CDROM]

  • Feltovich PJ, Hoffman RR, Woods D (2004). Keeping it too simple: how the reductive tendency affects cognitive engineering. IEEE Intell Syst 90–95

  • Finkelstein A, Dowell J (1996) A comedy of errors: the London Ambulance Service case study. In: Proceedings of the 8th international workshop on software specification and design IWSSD-8, pp 2–4

  • Goguen J (1994) Requirements engineering as the reconciliation of technical and social issues. In: Goguen J, Jirotka M (eds) Requirements engineering: social and technical issues. Academic Press, Burlington, pp 165–199

  • Goguen J (1997) Towards a social, ethical theory of information. In: Bowker G, Gasser L, Star L, Turner W (eds) Social science research: technical systems and cooperative work: beyond the great divide. Erlbaum, Mahwah, pp 27–56

    Google Scholar 

  • Goguen J (1999) Tossing algebraic flowers down the great divide. In: Calude CS (ed) People and ideas in theoretical computer science. Springer, New York, pp 93–129

    Google Scholar 

  • Hoffman RR, Crandall B, Shadbolt N (1998) Use of the critical decision method to elicit expert knowledge: A case study in the method of cognitive task analysis. Hum Factors 40:254–276

    Article  Google Scholar 

  • Hoffman RR, Elm WC (2006) HCC implications for the procurement process. IEEE: Intelligent Systems, January/February 2006, pp 74–81

  • Hoffman RR, Ford KM, Coffey JW (2000) The handbook of human-centered computing (deliverable on the contract, human-centered system prototype). National Technology Alliance

  • Hoffman RR, Roesler A, Moon BM (2004) What is design in the context of human-centered computing? IEEE: Intelligent Systems, July/August 2004, pp 89–95

  • Hoffman RR, Shadbolt N, Burton AM, Klein GA (1995) Eliciting knowledge from experts: a methodological analysis. Org Behav Hum Decis Process 62:129–158

    Article  Google Scholar 

  • Hoffman RR, Woods DD (2005) Steps toward a theory of complex and cognitive SYSTEMS. IEEE Intelligent Systems, January/February 2005, pp 76–79. Retrieved from http://www.ihmc.us/research/projects/EssaysOnHCC/

  • Hollnagel E, Woods DD (2006) Joint cognitive systems: foundations of cognitive systems engineering. Taylor and Francis, Boca Raton

  • Koopman P, Hoffman RR (2003) Work-arounds, make-work, and kludges. IEEE: Intelligent Systems, November/December 2003, pp 70–75

  • Kraut RE, Streeter LA (1995) Coordination in software development. Commun ACM 38(3):69–81. Retrieved from http://www.cs.cmu.edu/~kraut/RKraut.site.files/articles/kraut95-CoordiantionInSoftwareDevelopment.pdf

    Google Scholar 

  • Legros LA, Weston HC (1926) On the design of machinery in relation to the operator. Report no. 36, Industrial Fatigue Research Board. His Majesty’s Stationery Office, London

    Google Scholar 

  • Leishman TR, Cook DA (2002) Requirements risks can drown software projects. Crosstalk 15:4–8. Retrieved from http://www.stsc.hill.af.mil/crosstalk/

    Google Scholar 

  • Marketplace (1997) Marketplace for January 31st, 1997. January 1997, National Public Radio

  • Mattice JJ (2005) Hubble space telescope systems engineering case study. Air Force Center for Systems Engineering, Dayton, OH. Retrieved from http://cse.afit.edu

  • McConnell S (1996) Rapid development: taming wild software schedules. Microsoft Press, Redmond

    Google Scholar 

  • Norman DO, Kuras ML (2004) Engineering complex systems (MITRE technical report no. 040043). MITRE. Retrieved from http://www.mitre.org/work/tech_papers/tech_papers_04/

  • Nuseibeh BA (2001) Weaving together requirements and architecture. IEEE Comput 34:115–117

    Google Scholar 

  • Nuseibeh BA, Easterbrook SM (2000) Requirements engineering: a roadmap. In: Finkelstein ACW (ed) The future of software engineering (companion volume to the proceedings of the 22nd International conference on software engineering, ICSE’00). IEEE Computer Society Press, Los Alamitos

  • Rajanen M (2003) Usability cost-benefit models—different approaches to usability benefit analysis. In: Proceedings of the 26th information systems research seminar in Scandinavia (IRIS26), Haikko, Finland.

  • Richey GK (2005) F-111 systems engineering case study. Air Force Center for Systems Engineering, Dayton. Retrieved from http://cse.afit.edu

  • Royce WW (1970) Managing the development of large software systems: concepts and techniques. In: Proceedings of IEEE WestCon. IEEE, New York, pp. 328–338

  • Surdu J, Parsons DJ (2006) Army simulation program balances agile and traditional methods with success. Crosstalk 19(4):4–7. Retrieved from http://www.stsc.hill.af.mil/crosstalk/2006/04/index.html

    Google Scholar 

  • Ward PT, Mellor SJ (1985) Structured development for real-time systems, vol I: introduction and tools. Yourdon Press, Englewood Cliffs

    Google Scholar 

  • Wiegers KE (2000) When telepathy won’t do: requirements engineering key practices. Cutter IT Journal, 13(5). Retrieved from http://www.processimpact.com/pubs.shtml#requirements

  • Woods DD (2005) Generic support requirements for cognitive work: laws that govern cognitive work in action. In: Proceedings of the Human Factors and Ergonomics Society 49th Annual Meeting. HFES, Santa Monica. Retrieved from http://csel.eng.ohio-state.edu/blog/woods/archives/000113.html

  • Woods DD, Dekker S (2000) Anticipating the effects of technological change: a new era of dynamics for human factors. Theor Issues Ergonom Sci 1:272–282

    Google Scholar 

  • Zachary W, Neville K, Fowlkes J, Hoffman RR (2007) Human total cost of ownership: the penny foolish principle at work. IEEE Intelligent Systems, March/April 2007, pp 88–92

Download references

Acknowledgments

The contributions of the authors was made possible by a Phase II contract, “Joint Systems Engineering Methodology” to CHI Systems, Inc. under the Small Business Innovative Research Program contract FA8650-06-C-6637, sponsored by the US Air Force Research Laboratory.

Author information

Authors and Affiliations

  1. Florida Institute for Human and Machine Cognition, 40 South Alcaniz St., Pensacola, FL, 32502-6008, USA

    Robert R. Hoffman

  2. Chi Systems, Inc., Orlando, FL, USA

    Kelly Neville & Jennifer Fowlkes

Authors
  1. Robert R. Hoffman
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Kelly Neville
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jennifer Fowlkes
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Robert R. Hoffman.

Appendix: representative sampling of documents

Appendix: representative sampling of documents

  • Brooks FP (1995) The mythical man-month. Addison-Wesley, Boston.

  • Bush M (1997) NIMROD case study. Retrieved from http://www.scism.sbu.ac.uk/~bushm/units/NIMROD_Case_Study.html.

  • Carnegie Mellon University (2000) Results of the USC-SEI workshop on spiral development. Retrieved from http://www.sei.cmu.edu/cbs/spiral2000/february2000/.

  • Gaitros DA (March 2004) Common errors in large software development projects. Crosstalk 17(3):21–25. Retrieved from http://www.stsc.hill.af.mil/crosstalk/.

  • Goguen J (1999) Tossing algebraic flowers down the great divide. In: Calude CS (ed) People and ideas in theoretical computer science. Springer, New York, pp 93–129.

  • Goguen J (1997) Towards a social, ethical theory of information. In: Bowker G, Gasser L, Star L, Turner W (eds) Social science research: technical systems and cooperative work: beyond the great divide. Erlbaum, Mahwah, pp 27–56.

  • Hollnagel E, Woods DD (2006) Joint cognitive systems: foundations of cognitive systems engineering. Taylor and Francis, Boca Raton.

  • Jokela T (2005) The ISO 9241-11 definition of usability in usability requirements determination: lessons learnt from case studies. Proceedings of the 2005 Human Computer Interaction International Conference, Las Vegas, NV.

  • Leishman TR, Cook DA (April 2002) Requirements risks can drown software projects. Crosstalk 15(4):4–8. Retrieved from http://www.stsc.hill.af.mil/crosstalk/.

  • McConnell S (1996). Rapid development: taming wild software schedules. Microsoft Press, Redmond.

  • Norman DO, Kuras ML (2004) Engineering complex systems (MITRE technical report no. 040043). MITRE. Retrieved from http://www.mitre.org/work/tech_papers/tech_papers_04/.

  • Richey GK (2005) F-111 systems engineering case study. Air Force Center for Systems Engineering, Dayton. Retrieved from http://cse.afit.edu/page.cfm?page=49.

  • SysML Partners (2006). Systems modeling language (SysML). Retrieved from http://www.sysml.org/specs.htm.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Hoffman, R.R., Neville, K. & Fowlkes, J. Using cognitive task analysis to explore issues in the procurement of intelligent decision support systems. Cogn Tech Work 11, 57–70 (2009). https://doi.org/10.1007/s10111-008-0120-5

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10111-008-0120-5

Keywords

Search

Navigation