Advertisement

Springer Nature is making Coronavirus research free. View research | View latest news | Sign up for updates

Exploiting visual quasi-periodicity for real-time chewing event detection using active appearance models and support vector machines

Abstract

Steady increases in healthcare costs and obesity have inspired recent studies into cost-effective, assistive systems capable of monitoring dietary habits. Few researchers, though, have investigated the use of video as a means of monitoring dietary activities. Video possesses several inherent qualities, such as passive acquisition, that merits its analysis as an input modality for such an application. To this end, we propose a method to automatically detect chewing events in surveillance video of a subject. Firstly, an Active Appearance Model (AAM) is used to track a subject’s face across the video sequence. It is observed that the variations in the AAM parameters across chewing events demonstrate a distinct periodicity. We utilize this property to discriminate between chewing and non-chewing facial actions such as talking. A feature representation is constructed by applying spectral analysis to a temporal window of model parameter values. The estimated power spectra subsequently undergo non-linear dimensionality reduction. The low-dimensional embedding of the power spectra are employed to train a binary Support Vector Machine classifier to detect chewing events. To emulate the gradual onset and offset of chewing, smoothness is imposed over the class predictions of neighboring video frames in order to deter abrupt changes in the class labels. Experiments are conducted on a dataset consisting of 37 subjects performing each of five actions, namely, open- and closed-mouth chewing, clutter faces, talking, and still face. Experimental results yielded a cross-validated percentage agreement of 93.0%, indicating that the proposed system provides an efficient approach to automated chewing detection.

This is a preview of subscription content, log in to check access.

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Notes

  1. 1.

    Experiments of the proposed approach are conducted in part on the dataset employed in [16]. A description of the dataset is provided in Sect. 7.

References

  1. 1.

    Amft O, Tröster G (2008) Recognition of dietary activity events using on-body sensors. Artif Intell Med 42(2):121–136

  2. 2.

    Cai D, He X, Han J (2005) Document clustering using locality preserving indexing. IEEE Trans Knowl Data Eng 17(12):1624–1637

  3. 3.

    Cai D, He X, Han J, Zhang H (2006) Orthogonal laplacianfaces for face recognition. IEEE Trans Image Process 15(11):3608–3614

  4. 4.

    Cai D, He X, Zhang WV, Han J (2007) Regularized locality preserving indexing via spectral regression. In: Proceeding of the ACM conference on conference on information and knowledge management, pp 741–750

  5. 5.

    Cootes TF, Edwards GJ, Taylor CJ (1998) Active appearance models. In: Proceedings of the European conference on computer vision 2, pp 484–498

  6. 6.

    Guo G, Fu Y, Dyer CR, Huang TS (2008) Image-based human age estimation by manifold learning and locally adjusted robust regression. IEEE Trans Image Process 17(7):1178–1188

  7. 7.

    Hayes MH (1996) Statistical digital signal processing and modeling. Wiley, New York

  8. 8.

    He X, Yan S, Hu Y, Niyogi P, Zhang H (2005) Face recognition using laplacianfaces. IEEE Trans Pattern Anal Mach Intell 27(3):328–340

  9. 9.

    Mahoor MH, Cadavid S, Messinger DS, Cohn JF (2009) A framework for automated measurement of the intensity of non-posed facial action units. In: CVPR workshop on human communicative behavior analysis (CVPR4HB)

  10. 10.

    Nishimura J, Kuroda T (2008) Eating habits monitoring using wireless wearable in-ear microphone. In: Third international symposium on wireless pervasive computing. ISWPC 2008, Santorini, pp 130–132. doi:10.1109/ISWPC.2008.4556181. ISBN 978-1-4244-1652-3

  11. 11.

    Niyogi P (2003) Laplacian eigenmaps for dimensionality reduction and data representation. Neural Comput Appl 15:1373–1396

  12. 12.

    Pogalin E, Smeulders A, Thean A (2008) Visual quasi-periodicity. In: Proceedings of the conference on computer vision and pattern recognition, pp 1–8

  13. 13.

    Reis HT, Judd CM (2000) Handbook of research methods in social and personality psychology. Cambridge University Press, Cambridge

  14. 14.

    Roweis S, Saul L (2000) Nonlinear dimensionality reduction by locally linear embedding. Sci Agric 290(5500):2323–2326

  15. 15.

    Sazonov E, Schuckers S, Lopez-Meyer P, Makeyev O, Sazonova N, Melanson EL, Neuman M (2008) Non-invasive monitoring of chewing and swallowing for objective quantification of ingestive behavior. Physiol Meas 29(5):525

  16. 16.

    Schmalz M, Mendez-Vazquez A, Helal A (2009) Algorithms for the detection of chewing behavior in dietary monitoring applications. In: Proceedings of SPIE technical symposium: mathematics of data/image coding, compression, and encryption with applications XII, vol 7444A

  17. 17.

    Tanigawa S, Nishihara H, Kaneda S, Haga H (2008) Detecting mastication by using microwave doppler sensor. In: PETRA ’08: Proceedings of the 1st international conference on PErvasive Technologies Related to Assistive Environments, New York, pp 1–7

  18. 18.

    Tenenbaum J, de Silva V, Langford J (2000) A global geometric framework for nonlinear dimensionality reduction. Sci Agric 290(5500):2319–2323

  19. 19.

    Tolstikov A, Biswas J, Tham CK, Yap P (2008) Eating activity primitives detection—a step towards adl recognition. In: e-health networking, applications and services, 2008. HealthCom 2008. 10th International Conference on, pp 35–41

  20. 20.

    Turk M, Pentland A (1991) Eigenfaces for recognition. J Cogn Neurosci 3(1):71–86

  21. 21.

    Vapnik VN (1995) The nature of statistical learning theory. Springer, Berlin

  22. 22.

    Viola P, Jones MJ (2004) Robust real-time face detection. Int J Comput Vis 57(2):137–154

  23. 23.

    Wang Y, Lucey S, Cohn J (2008) Enforcing convexity for improved alignment with constrained local models. In: IEEE international conference on computer vision and pattern recognition

  24. 24.

    Zhang S, Ang MH, Xiao W, Tham CK (2009) Detection of activities by wireless sensors for daily life surveillance: eating and drinking. Sensors 9(3):1499–1517

Download references

Acknowledgments

This research was supported in part by a National Institute of Health (NIH) grant number 5R21DA024294.

Author information

Correspondence to Steven Cadavid.

Rights and permissions

Reprints and Permissions

About this article

Cite this article

Cadavid, S., Abdel-Mottaleb, M. & Helal, A. Exploiting visual quasi-periodicity for real-time chewing event detection using active appearance models and support vector machines. Pers Ubiquit Comput 16, 729–739 (2012). https://doi.org/10.1007/s00779-011-0425-x

Download citation

Keywords

  • Dietary monitoring
  • Behavior detection
  • Active appearance models
  • Manifold learning
  • Support vector machines