Multimodal Classification of Activities of Daily Living Inside Smart Homes
Smart homes for the aging population have recently started attracting the attention of the research community. One of the problems of interest is this of monitoring the activities of daily living (ADLs) of the elderly aiming at their protection and well-being. In this work, we present our initial efforts to automatically recognize ADLs using multimodal input from audio-visual sensors. For this purpose, and as part of Integrated Project Netcarity, far-field microphones and cameras have been installed inside an apartment and used to collect a corpus of ADLs, acted by multiple subjects. The resulting data streams are processed to generate perception-based acoustic features, as well as human location coordinates that are employed as visual features. The extracted features are then presented to Gaussian mixture models for their classification into a set of predefined ADLs. Our experimental results show that both acoustic and visual features are useful in ADL classification, but performance of the latter deteriorates when subject tracking becomes inaccurate. Furthermore, joint audio-visual classification by simple concatenative feature fusion significantly outperforms both unimodal classifiers.
KeywordsVisual Feature Gaussian Mixture Model Smart Home Visual Tracking Microphone Array
Unable to display preview. Download preview PDF.
- 1.Netcarity – Ambient Technology to Support Older People at Home, http://www.netcarity.org
- 4.Grassi, M., Lombardi, A., Rescio, G., Malcovati, P., Leone, A., Diraco, G., Distante, C., Siciliano, P., Malfatti, M., Gonzo, L., Libal, V., Huang, J., Potamianos, G.: A hardware-software framework for high-reliability people fall detection. In: Proc. IEEE Conf. on Sensors, Lecce, Italy, pp. 1328–1331 (2008)Google Scholar
- 5.Fleury, A., Vacher, M., Glasson, H., Serignat, J.-F., Noury, N.: Data fusion in health smart home: Preliminary individual evaluation of two families of sensors. In: Proc. Int. Conf. of the Int. Soc. for Gerontechnology, Pisa, Italy (2008)Google Scholar
- 6.Huang, J., Zhuang, X., Libal, V., Potamianos, G.: Long-time span acoustic activity analysis from far-field sensors in smart homes. In: Proc. Int. Conf. Acoustics, Speech, and Signal Process (ICASSP), Taipei, Taiwan (2009)Google Scholar
- 7.Wojek, C., Nickel, K., Stiefelhagen, R.: Activity recognition and room level tracking in an office environment. In: Proc. IEEE Int. Conf. on Multisensor Fusion and Integration for Intelligent Systems (MFI), Heidelberg, Germany (2006)Google Scholar
- 8.Cappelletti, A., Lepri, B., Mana, N., Pianesi, F., Zancanaro, M.: A multimodal data collection of daily activities in a real instrumented apartment. In: Proc. Works. on Multimodal Corpora: From Models of Natural Interaction to Systems and Applications – Held in Conjunction with the 6th Language Resources and Evaluation Conf. (LREC), Marrakech, Morocco (2008)Google Scholar