Personal and Ubiquitous Computing

, Volume 16, Issue 2, pp 169–191 | Cite as

Monitoring children’s developmental progress using augmented toys and activity recognition

  • Tracy L. Westeyn
  • Gregory D. Abowd
  • Thad E. Starner
  • Jeremy M. Johnson
  • Peter W. Presti
  • Kimberly A. Weaver
Original Paper


Previous research has established the connection between the way in which children interact with objects and the potential early identification of children with autism. Those findings motivate our own work to develop "smart toys," objects embedded with wireless sensors that are safe and enjoyable for very small children, that allow detailed interaction data to be easily recorded. These sensor-enabled toys provide opportunities for autism research by reducing the effort required to collect and analyze a child’s interactions with objects. In the future, such toys may be a useful part of clinical and in-home assessment tools. In this paper, we discuss the design of a collection of smart toys that can be used to automatically characterize the way in which a child is playing. We use statistical models to provide objective, quantitative measures of object play interactions. We also developed a tool to view rich forms of annotated play data for later analysis. We report the results of recognition experiments on more than fifty play sessions conducted with adults and children as well as discuss the opportunities for using this approach to support video annotation and other applications.


Content analysis Automatic indexing Toy design Object-play Multimodal wireless sensing Pattern recognition 


  1. 1.
    Adamson L, Bakeman R, Deckner D (2004) The development of symbol-infused joint engagement. Child Dev 75(4):1171–1187CrossRefGoogle Scholar
  2. 2.
    Agata Rozga P (2009) Personal CommunicationGoogle Scholar
  3. 3.
    Aylward R, Lovell SD, Paradiso JA (2006) A compact, wireless, wearable sensor network for interactive dance ensembles. In: International workshop on wearable and implantable body sensor networks (BSN 2006), 3–5 Apr 2006, Cambridge, Massachusetts, USA, IEEE Computer Society, pp 65–70Google Scholar
  4. 4.
    Baranek GT, Barnett C, Adams E, Wolcott N, Watson L, Crais E (2005) Object play in infants with autism: methodological issues in retrospective video analysis. Am J Occup Ther 59(1):20–30CrossRefGoogle Scholar
  5. 5.
    Baranek GT, David FJ, Poe MD, Stone WL, Watson LR (2005) Sensory experiences questionnaire: discriminating sensory features in young children with autism, developmental delays, and typical development. J Child Psychol Psychiatry 47(6):591–601CrossRefGoogle Scholar
  6. 6.
    Blasco PA (1991) Pitfalls in developmental diagnosis. Pediatr Clin North Am 38:1425–1438Google Scholar
  7. 7.
    Chang CC, Lin CJ (2001) LIBSVM: a library for support vector machines. Software available at
  8. 8.
    Cohen J (1960) A coefficient of agreement for nominal scales. Educ Psychol Meas 20(1):37–46CrossRefGoogle Scholar
  9. 9.
    First L, Palfrey J (1994) The infant or young child with developmental delay. New Engl J Med 330:478–483CrossRefGoogle Scholar
  10. 10.
    Ganesan M, Russell NW, Rajan R, Welch N, Westeyn TL, Abowd GD (2010) Grip sensing in smart toys: a formative design method for user categorization. In: CHI EA ’10: Proceedings of the 28th of the international conference extended abstracts on human factors in computing systems. ACM, pp 3745–3750Google Scholar
  11. 11.
    Gorbet MG, Orth M, Ishii H (1998) Triangles: tangible interface for manipulation and exploration of digital information topography. In: Proceedings of CHI ’98. ACM, pp 49–56,
  12. 12.
    Jurafsky D, Martin JH (2000) Speech and language processing: an introduction to natural language processing, computational linguistics, and speech recognition. Prentice Hall PTR, Upper Saddle RiverGoogle Scholar
  13. 13.
    Kehoe C, Cassell J, Goldman S, Dai J, Gouldstone I, MacLeod S, O’Day T, Pandolfo A, Ryokai K, Wang A (2004) Sam goes to school: story listening systems in the classroom. In: ICLS ’04: Proceedings of the 6th international conference on learning sciences, International Society of the Learning Sciences, pp 613–613Google Scholar
  14. 14.
    Kernberg PF, Chazan SE, Normandin L (1998) The children’s play therapy instrument (cpti): description, development, and reliability studies. J Psychother Pract Res 7:196–207Google Scholar
  15. 15.
    Kientz JA (2008) Decision support for caregivers through embedded capture and access. PhD thesis, College of Computing, School of Interactive Computing, Georgia Institute of Technology, Atlanta, GA, USAGoogle Scholar
  16. 16.
    Kientz JA, Arriaga RI, Chetty M, Hayes GR, Richardson J, Patel SN, Abowd GD (2007) Grow and know: understanding record-keeping needs for tracking the development of young children. In: CHI ’07: Proceedings of the SIGCHI conference on human factors in computing systems. ACM Press, pp 1351–1360Google Scholar
  17. 17.
    Kitamura Y, Itoh Y, Kishino F (2001) Real-time 3d interaction with active cube. In: CHI ’01: CHI ’01 extended abstracts on human factors in computing systems. ACM Press, New York, pp 355–356Google Scholar
  18. 18.
    Landis JR, Koch GG (1977) The measurement of observer agreement for categorical data. Biometrics 33:159–174MathSciNetzbMATHCrossRefGoogle Scholar
  19. 19.
    Lester J, Hannaford B, Borriello G (2004) ‘Are you with me?’—using accelerometers to determine if two devices are carried by the same person. In: Proceedings of the second international conference on pervasive computing, pp 33–50Google Scholar
  20. 20.
    Mayrhofer R, Gellersen H (2007) Shake well before use: Authentication based on accelerometer data. In: Proceedings of 5th international conference of pervasive computing. Lecture notes in computer science, vol 4480. Springer, pp 144–161Google Scholar
  21. 21.
    Minnen D, Westeyn T, Starner T, Ward J, Lukowicz P (2006) Performance metrics and evaluation issues for continuous activity recognition. In: Performance metrics for intelligent systems. NIST, Gaithersburg, pp 141–148Google Scholar
  22. 22.
    Ozonoff S, Macari S, Young GS, Goldring S, Thompson M, Rogers SJ (2008) Atypical object exploration at 12 months of age is associated with autism in a prospective sample. Autism 12(5):457–472CrossRefGoogle Scholar
  23. 23.
    Presti P (2006) Bluesense—a wireless interface prototyping system. Master’s thesis, College of Computing, Georgia Institute of Technology, Atlanta, GAGoogle Scholar
  24. 24.
    Rijsbergen CJV (1979) Information retrieval, 2nd edn. Butterworth Scientific Ltd, LondonGoogle Scholar
  25. 25.
    Rosa Arriaga P (2007) Personal CommunicationGoogle Scholar
  26. 26.
    Shannon CE (1949) Communication in the presence of noise. Proc IRE 37(1):10–21MathSciNetCrossRefGoogle Scholar
  27. 27.
    Sharlin E, Itoh Y, Watson B, Kitamura Y, Sutphen S, Liu L (2002) Cognitive cubes: a tangible user interface for cognitive assessment. In: Proceedings of the SIGCHI conference on human factors in computing systems. ACM Press, pp 347–354Google Scholar
  28. 28.
    Shevell M, Ashwal S, Donley D, Flint J, Gingold M, Hirtz D, Majnemer A, Noetzel M, Sheth R (2003) Practice parameter: evaluation of the child with global developmental delay: report of the quality standards subcommittee of the American Academy of Neurology and the Practice Committee of the Child Neurology Society. Neurology 60:367–380Google Scholar
  29. 29.
    Technologies C (2010) Caring technologies website: Retrieved 20 Aug 2010. World Wide Web electronic publication
  30. 30.
    Thelen E (2000) Motor development as foundation and future of development psychology. J Behav Dev 24:385–397CrossRefGoogle Scholar
  31. 31.
    Wang P, Abowd GD, Rehg JM (2009) Quasi-periodic event analysis for social game retrieval. In: Proceedings of IEEE international conference on computer vision, IEEEGoogle Scholar
  32. 32.
    Westeyn TL (2010) Child’s play: activity recognition for monitoring children’s developmental progress with augmented toys. PhD thesis, Georgia Institute of TechnologyGoogle Scholar

Copyright information

© Springer-Verlag London Limited 2011

Authors and Affiliations

  • Tracy L. Westeyn
    • 1
  • Gregory D. Abowd
    • 1
  • Thad E. Starner
    • 1
  • Jeremy M. Johnson
    • 2
  • Peter W. Presti
    • 2
  • Kimberly A. Weaver
    • 1
  1. 1.School of Interactive ComputingGeorgia Institute of TechnologyAtlantaUSA
  2. 2.Interactive Media Technology CenterGeorgia Institute of TechnologyAtlantaUSA

Personalised recommendations