Personal and Ubiquitous Computing

, Volume 17, Issue 2, pp 351–358 | Cite as

A kinect-based vocational task prompting system for individuals with cognitive impairments

  • Yao-Jen Chang
  • Li-Der Chou
  • Frank Tsen-Yung Wang
  • Shu-Fang Chen
Original Article


Difficulties in executing daily living tasks hamper the quality of life of many individuals with cognitive impairments who are otherwise physically mobile. With sufficient and appropriate support on the job, many people with developmental disabilities and cognitive impairments are capable of participating in the world of work to various levels. Kinect is used as assistive technology for individuals with cognitive impairments to achieve the goal of performing task steps independently. In a community-based rehabilitation program under the guidance of three job coaches, a task prompting system called Kinempt was designed to assist four participants involving pre-service food preparation training. The study assessed the effectiveness of Kinempt in terms of precision and recall. A follow-up comparative study then evaluated a baseline method and the system of least prompts against the Kinempt system. Results indicate that for participants with cognitive disabilities, acquisition of job skills may be facilitated by use of Kinempt in conjunction with operant conditioning strategies. Our findings suggest that the image recognition technology may be able to facilitate task prompts needed by people with cognitive impairments. Therefore, the system may be helpful for pre-service training while increasing independence in the process of community integration.


Cognitive Impairment Assistive Technology Kinect Sensor Occupational Rehabilitation Task Step 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.



This research was granted and supported by National Research Program for the Department of Industrial Technology (DoIT) of the Ministry of Economic Affairs (MOEA), Taiwan, R.O.C (99-EC-17-A-19-S1-163) and National Science Council, Taiwan, R.O.C (NSC 100-2221-E-033 -062 -).


  1. 1.
    Braddock D, Hemp RR, Parish S (2002) The state of the states in developmental disabilities: Study summary. In Disability at the Dawn of the 21st Century and the State of the States. Braddock, D. (ed). American Association in Mental Retardation, Washington DC, p 124Google Scholar
  2. 2.
    The Aphasia Institute. What is Aphasia? (2005) [Online]. Available: September 2005 at
  3. 3.
    Goodwin S (1997) Comparative mental health policy: From institutional to community care. Sage, Thousand OaksGoogle Scholar
  4. 4.
    Bond GR, Drake RE, et al. (1996) Supported employment for people with severe mental illness: a review. Psychiatric ServicesGoogle Scholar
  5. 5.
    Bond GR, Liberman RP (1992) Vocational rehabilitation. Handbook of psychiatric rehabilitation, New YorkGoogle Scholar
  6. 6.
    Mechling LC, Ortega-Hurndon F (2007) Computer-based video instruction to teach young adults with moderate intellectual disabilities to perform multiple step, job tasks in a generalized setting. Educ Train Develop Disabil 42:24–37Google Scholar
  7. 7.
    Van Laarhoven T, Van Laarhoven-Myers T (2006) Comparison of three video-based instructional procedures for teaching daily living skills to persons with developmental disabilities. Educ Train Develop Disabil 41:365–381Google Scholar
  8. 8.
    Cannella-Malone H, Sigafoos J, O’Reilly M, de la Cruz B, Lancioni GE (2006) Comparing video prompting to video modeling for teaching daily living skills to six adults with developmental disabilities. Educ Train Develop Disabil 41:344–356Google Scholar
  9. 9.
    Sohlberg MM, Fickas S, Hung PF, Fortier A (2007) A Comparison of four prompt modes for route finding for community travelers with severe cognitive impairments. Brain Inj 21(5):531–538CrossRefGoogle Scholar
  10. 10.
    Wu M, Baecker R, Richards B (2005) Participatory design of an orientation aid for amnesics. In: Proceedings of the SIGCHI conference on human factors in computing systems, pp 511–520Google Scholar
  11. 11.
    Carmien S, Dawe M, Fischer G, Gorman A, Kintsch A, Sullivan JF (2005) Socio-technical environments supporting people with cognitive disabilities using public transportation. ACM Trans Comp Human Interact 12(2):233–262CrossRefGoogle Scholar
  12. 12.
    Cihaka DF, Kesslerb KB, Alberto PA (2007) Generalized use of a handheld prompting system. Res Dev Disabil 28:397–408CrossRefGoogle Scholar
  13. 13.
    Cihak DF, Kessler K, Alberto PA (2008) Use of a handheld prompting system to transition independently through vocational tasks for student with moderate and severe intellectual disabilities. Educ Train Develop Disabil 43:102–110Google Scholar
  14. 14.
    Liu L, Hile H, Kautz H, Borriello G, Brown PA, Harniss M, Johnson K (2006) Indoor Wayfinding: developing a functional interface for individuals with cognitive impairments. In: Proceedings of computers & accessibility, ASSETS 2006, pp 95–102Google Scholar
  15. 15.
    Chang YJ, Tsai SK, Chang YS, Wang TY (2007) A novel wayfinding system based on Geo-coded QR codes for individuals with cognitive impairments, 9th international ACM SIGACCESS conference on computers and accessibility, Tempe, Oct 15–17Google Scholar
  16. 16.
    Gentry T, Wallace J, Kvarfordt C, Lynch KB (2007) Personal digital assistants as cognitive aids for individuals with severe traumatic brain injury: a community-based trial, Brain Inj, December 19–24Google Scholar
  17. 17.
    Liu AL, Hile H, Kautz H, Borriello G, Brown PA, Mark Harniss, Johnson K (2007) Indoor wayfinding: developing a functional interface for individuals with cognitive impairments. Disabil Rehabil Assist Technol 3(1):69–81CrossRefGoogle Scholar
  18. 18.
    Davies DK, Stock SE, Wehmeyer M (2002) Enhancing independent task performance for individuals with mental retardation through use of a handheld self-directed visual and audio prompting. Educ Train Mental Retard Develop Disabil 37:209–218Google Scholar
  19. 19.
    Bohonos S, Lee A, Malik A, Thai C, Manduchi R (2007) Universal real–time navigational assistance (URNA): an urban Bluetooth beacon for the blind, In: Proceedings of the 1st ACM SIGMOBILE international workshop on systems and networking support for healthcare and assisted living environments, pp 83–88Google Scholar
  20. 20.
    Yao-Jen Chang, Shih-Kai Tsai, Tseng-Yun Wang (2008) A context aware handheld wayfinding system for individuals with cognitive impairments. 10th international ACM SIGACCESS conference on computers and accessibility, Halifax, Oct 13–15, pp 27–34Google Scholar
  21. 21.
    NASA Task Load Index (TLX) [Online]. Available:
  22. 22.
    Taber-Doughty T (2005) Considering student choice when selecting instructional strategies: a comparison of three prompting systems. Res Dev Disabil 26:411–432CrossRefGoogle Scholar
  23. 23.
    Cook AM, Polgar JM, Hussey SM (2008) Assistive technology principles and practice, 3rd edn. Mosby-ElsevierGoogle Scholar
  24. 24.
    Brodwin MG, Star T, Cardoso E (2004) Computer assistive technology for people who have disabilities: computer adaptations and modifications. J Rehabil 70:28–33Google Scholar
  25. 25.
    Wessels R, Dijcks B, Soede M, Gelderblom GJ, De Witte L (2003) Non-use of provided assistive technology devices, a literature overview. Technol Disabil 15:231–238Google Scholar
  26. 26.
    Chang YJ, Wang TY (2010) Comparing picture and video prompting in autonomous indoor wayfinding for individuals with cognitive impairments. Pers Ubiquit Comput 14:737–747CrossRefGoogle Scholar
  27. 27.
    Chang YJ, Wang TY (2010) Mobile location-based social networking in supported employment for people with cognitive impairments. Cybernet Syst 41(3):245–261CrossRefGoogle Scholar

Copyright information

© Springer-Verlag London Limited 2011

Authors and Affiliations

  • Yao-Jen Chang
    • 1
  • Li-Der Chou
    • 2
  • Frank Tsen-Yung Wang
    • 3
  • Shu-Fang Chen
    • 4
  1. 1.Department of Electronic EngineeringChung Yuan Christian UniversityChung LiTaiwan
  2. 2.Department of Computer Science and Information EngineeringNational Central UniversityChung LiTaiwan
  3. 3.Institute of Social WorkNational Cheng Chi UniversityTaipeiTaiwan
  4. 4.Center for Occupational TherapyTaipei Municipal HospitalTaipeiTaiwan

Personalised recommendations