STARS: Soft Multi-Task Learning for Activity Recognition from Multi-Modal Sensor Data

  • Xi LiuEmail author
  • Pang-Ning Tan
  • Lei Liu
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10938)


Human activity recognition from ubiquitous sensor data is an important but challenging classification problem for applications such as assisted living, energy management, and security monitoring of smart homes. In this paper, we present a soft probabilistic classification model for human activity recognition from multi-modal sensors in a smart home environment. The model employs a softmax multi-task learning approach to fit a joint model for all the rooms in the smart home, taking into account the diverse types of sensors available in different rooms. The model also learns the transitional dependencies between activities to improve its prediction accuracy. Experimental results on a real-world dataset showed that the proposed approach outperforms several baseline methods, including k-nearest neighbors, conditional random field, and standard multinomial logistic regression.



This research is supported in part by the U.S. National Science Foundation through grant NSF III-1615612. Any use of trade, firm, or product names is for descriptive purposes only and does not imply endorsement by the U.S. Government.


  1. 1.
    The sphere challenge: activity recognition with multimodal sensor data.
  2. 2.
    Agarwal, A., Rakhlin, A., Bartlett, P.: Matrix regularization techniques for online multitask learning. Technical report UCB/EECS-2008-138, EECS Department, University of California, Berkeley (2008)Google Scholar
  3. 3.
    Bishop, C.M.: Pattern Recognition and Machine Learning. Springer, New York (2006)zbMATHGoogle Scholar
  4. 4.
    Boyd, S., Vandenberghe, L.: Convex Optimization. Cambridge University Press, New York (2004)CrossRefGoogle Scholar
  5. 5.
    Carós, J.S., Chételat, O., Celka, P., Dasen, S., CmÃral, J.: Very low complexity algorithm for ambulatory activity classification. In: Proceedings of the 3rd European Medical and Biological Conference, EMBEC, pp. 16–20 (2005)Google Scholar
  6. 6.
    Cavallanti, G., Cesa-Bianchi, N., Gentile, C.: Linear algorithms for online multitask classification. J. Mach. Learn. Res. 11, 2901–2934 (2010)MathSciNetzbMATHGoogle Scholar
  7. 7.
    Dougherty, J., Kohavi, R., Sahami, M., et al.: Supervised and unsupervised discretization of continuous features. In: Proceedings of the 12th International Conference of Machine Learning, vol. 12, pp. 194–202 (1995)CrossRefGoogle Scholar
  8. 8.
    Gao, Q., Doering, M., Yang, S., Chai, J.: Physical causality of action verbs in grounded language understanding. In: Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, vol. 1, pp. 1814–1824 (2016)Google Scholar
  9. 9.
    Kim, E., Helal, S., Cook, D.: Human activity recognition and pattern discovery. IEEE Pervasive Comput. 9(1) (2010)CrossRefGoogle Scholar
  10. 10.
    Lafferty, J., McCallum, A., Pereira, F.C.: Conditional random fields: probabilistic models for segmenting and labeling sequence data (2001)Google Scholar
  11. 11.
    Lester, J., Choudhury, T., Borriello, G.: A practical approach to recognizing physical activities. In: Fishkin, K.P., Schiele, B., Nixon, P., Quigley, A. (eds.) Pervasive 2006. LNCS, vol. 3968, pp. 1–16. Springer, Heidelberg (2006). Scholar
  12. 12.
    Liu, X., Liu, L., Simske, S.J., Liu, J.: Human daily activity recognition for healthcare using wearable and visual sensing data. In: 2016 IEEE International Conference on Healthcare Informatics (ICHI), pp. 24–31. IEEE (2016)Google Scholar
  13. 13.
    Nesterov, Y.: A method of solving a convex programming problem with convergence rate o(1/k2). Sov. Math. Dokl. 27, 372–376 (1983)zbMATHGoogle Scholar
  14. 14.
    Sun, X., Kashima, H., Ueda, N.: Large-scale personalized human activity recognition using online multitask learning. IEEE Trans. Knowl. Data Eng. 25(11), 2551–2563 (2013)CrossRefGoogle Scholar
  15. 15.
    Sutton, C., McCallum, A.: An introduction to conditional random fields for relational learning. In: Sutton, C., McCallum, A. (eds.) Introduction to Statistical Relational Learning, pp. 93–128. MIT Press, Cambridge (2006)Google Scholar
  16. 16.
    Twomey, N., Diethe, T., Kull, M., Song, H., Camplani, M., Hannuna, S., Fafoutis, X., Zhu, N., Woznowski, P., Flach, P., Craddock, I.: The SPHERE challenge: Activity recognition with multimodal sensor data. arXiv:1603.00797 (2016)
  17. 17.
    Vail, D.L., Veloso, M.M., Lafferty, J.D.: Conditional random fields for activity recognition. In: Proceedings of the 6th International Joint Conference on Autonomous Agents and Multiagent Systems, p. 235. ACM (2007)Google Scholar
  18. 18.
    Wang, S., Yang, J., Chen, N., Chen, X., Zhang, Q.: Human activity recognition with user-free accelerometers in the sensor networks. In: Proceedings of the International Conference on Neural Networks and Brain, vol. 2, pp. 1212–1217. IEEE (2005)Google Scholar
  19. 19.
    Yan, Y., Ricci, E., Liu, G., Sebe, N.: Egocentric daily activity recognition via multitask clustering. IEEE Trans. Image Process. 24(10), 2984–2995 (2015)MathSciNetCrossRefGoogle Scholar
  20. 20.
    Yang, H., King, I., Lyu, M.R.: Online learning for multi-task feature selection. In: Proceedings of the 19th ACM International Conference on Information and Knowledge Management, pp. 1693–1696. ACM (2010)Google Scholar
  21. 21.
    Yang, S., Gao, Q., Liu, C., Xiong, C., Zhu, S.C., Chai, J.Y.: Grounded semantic role labeling. In: Proceedings of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 149–159 (2016)Google Scholar

Copyright information

© Springer International Publishing AG, part of Springer Nature 2018

Authors and Affiliations

  1. 1.Michigan State UniversityLansingUSA
  2. 2.Huawei TechnologiesSanta ClaraUSA

Personalised recommendations