Advertisement

Context Change Detection for an Ultra-Low Power Low-Resolution Ego-Vision Imager

  • Francesco PaciEmail author
  • Lorenzo Baraldi
  • Giuseppe Serra
  • Rita Cucchiara
  • Luca Benini
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9913)

Abstract

With the increasing popularity of wearable cameras, such as GoPro or Narrative Clip, research on continuous activity monitoring from egocentric cameras has received a lot of attention. Research in hardware and software is devoted to find new efficient, stable and long-time running solutions; however, devices are too power-hungry for truly always-on operation, and are aggressively duty-cycled to achieve acceptable lifetimes. In this paper we present a wearable system for context change detection based on an egocentric camera with ultra-low power consumption that can collect data 24/7. Although the resolution of the captured images is low, experimental results in real scenarios demonstrate how our approach, based on Siamese Neural Networks, can achieve visual context awareness. In particular, we compare our solution with hand-crafted features and with state of art technique and propose a novel and challenging dataset composed of roughly 30000 low-resolution images.

Keywords

Egocentric vision ULP camera Low-resolution Deep learning 

Notes

Acknowledgments

This work was partially supported by the Swiss National Foundation under grant 162524 (MicroLearn: Micropower Deep Learning), the ERC MultiTherman project (ERC-AdG-291125) and the Vision for Augmented Experiences through the Fondazione CRMO Project.

References

  1. 1.
    Alletto, S., Serra, G., Calderara, S., Cucchiara, R.: Understanding social relationships in egocentric vision. Pattern Recogn. 48(12), 4082–4096 (2015)CrossRefGoogle Scholar
  2. 2.
    Andri, R., Cavigelli, L., Rossi, D., Benini, L.: Yodann: an ultra-low power convolutional neural network accelerator based on binary weights. arXiv preprint arXiv:1606.05487 (2016)
  3. 3.
    Baraldi, L., Grana, C., Cucchiara, R.: A deep siamese network for scene detection in broadcast videos. In: Proceedings of the 23rd ACM International Conference on Multimedia, pp. 1199–1202. ACM (2015)Google Scholar
  4. 4.
    Bifet, A., Gavalda, R.: Learning from time-changing data with adaptive windowing. In: Proceedings of International Conference on Data Mining (2007)Google Scholar
  5. 5.
    Castro, D., Hickson, S., Bettadapura, V., Thomaz, E., Abowd, G., Christensen, H., Essa, I.: Predicting daily activities from egocentric images using deep learning. In: Proceedings of the ACM International symposium on Wearable Computers (2015)Google Scholar
  6. 6.
    Centeye: http://www.centeye.com/. Accessed 15 July 2016
  7. 7.
    Conti, F., Benini, L.: A ultra-low-energy convolution engine for fast brain-inspired vision in multicore clusters. In: DATE 2015 (2015)Google Scholar
  8. 8.
    Dalal, N., Triggs, B.: Histograms of oriented gradients for human detection. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2005)Google Scholar
  9. 9.
    Dimiccoli, M., Bolaños, M., Talavera, E., Aghaei, M., Nikolov, S.G., Radeva, P.: Sr-clustering: Semantic regularized clustering for egocentric photo streams segmentation. arXiv preprint arXiv:1512.07143 (2015)
  10. 10.
    Donahue, J., Jia, Y., Vinyals, O., Hoffman, J., Zhang, N., Tzeng, E., Darrell, T.: Decaf: A deep convolutional activation feature for generic visual recognition. In: Proceedings of the International Conference on Machine Learning (2014)Google Scholar
  11. 11.
    Guan, D., Yuan, W., Lee, Y.K., Gavrilov, A., Lee, S.: Activity recognition based on semi-supervised learning. In: Proceedings of the International Conference on Embedded and Real-Time Computing Systems and Applications (2007)Google Scholar
  12. 12.
    Hadsell, R., Chopra, S., LeCun, Y.: Dimensionality reduction by learning an invariant mapping. In: IEEE International Conference on Computer Vision and Pattern Recognition, vol. 2, pp. 1735–1742. IEEE (2006)Google Scholar
  13. 13.
    Hodges, S., Williams, L., Berry, E., Izadi, S., Srinivasan, J., Butler, A., Smyth, G., Kapur, N., Wood, K.: SenseCam: a retrospective memory aid. In: Dourish, P., Friday, A. (eds.) UbiComp 2006. LNCS, vol. 4206, pp. 177–193. Springer, Heidelberg (2006). doi: 10.1007/11853565_11 CrossRefGoogle Scholar
  14. 14.
    Kelly, P., Doherty, A., Berry, E., Hodges, S., Batterham, A.M., Foster, C.: Can we use digital life-log images to investigate active and sedentary travel behaviour? Results from a pilot study. Int. J. Behav. Nutr. Phys. Act. 8(1), 1 (2011)CrossRefGoogle Scholar
  15. 15.
    Khan, A.M., Tufail, A., Khattak, A.M., Laine, T.H.: Activity recognition on smartphones via sensor-fusion and kda-based svms. Int. J. Distrib. Sensor Netw. 2014, 14 (2014)Google Scholar
  16. 16.
    Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. In: Advances in Neural Information Processing Systems, pp. 1097–1105 (2012)Google Scholar
  17. 17.
    Kwapisz, J.R., Weiss, G.M., Moore, S.A.: Activity recognition using cell phone accelerometers. ACM SigKDD Explor. Newslett. 12(2), 74–82 (2011)CrossRefGoogle Scholar
  18. 18.
    Li, Y., Fathi, A., Rehg, J.M.: Learning to predict gaze in egocentric video. In: Proceedings of the International Conference on Computer Vision (2013)Google Scholar
  19. 19.
    Long, J., Shelhamer, E., Darrell, T.: Fully convolutional networks for semantic segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2015)Google Scholar
  20. 20.
    Lu, Z., Grauman, K.: Story-driven summarization for egocentric video. In: Proceedings of the Conference on Computer Vision and Pattern Recognition (2013)Google Scholar
  21. 21.
    Magno, M., Brunelli, D., Sigrist, L., Andri, R., Cavigelli, L., Gomez, A., Benini, L.: Infinitime: multi-sensor wearable bracelet with human body harvesting. Sustainable Computing: Informatics and Systems (2016). http://www.sciencedirect.com/science/article/pii/S2210537916300816
  22. 22.
    Mannini, A., Sabatini, A.M.: Machine learning methods for classifying human physical activity from on-body accelerometers. Sensors 10(2), 1154–1175 (2010)CrossRefGoogle Scholar
  23. 23.
    MSP430: Texas instruments. http://www.ti.com/ww/it/msp.430.html. Accessed 15 July 2016
  24. 24.
    Narrative: http://getnarrative.com/. Accessed 15 July 2016
  25. 25.
    Patel, S., Park, H., Bonato, P., Chan, L., Rodgers, M.: A review of wearable sensors and systems with application in rehabilitation. J. Neuroeng. Rehabil. 9(1), 1 (2012)CrossRefGoogle Scholar
  26. 26.
    Poleg, Y., Arora, C., Peleg, S.: Temporal segmentation of egocentric videos. In: Proceedings of the Conference on Computer Vision and Pattern Recognition (2014)Google Scholar
  27. 27.
    Ryoo, M.S., Rothrock, B., Matthies, L.: Pooled motion features for first-person videos. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 896–904 (2015)Google Scholar
  28. 28.
    Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014)
  29. 29.
    Singh, K.K., Fatahalian, K., Efros, A.A.: Krishnacam: Using a longitudinal, single-person, egocentric dataset for scene understanding tasks. In: Proceedings of the IEEE Winter Conference on Applications of Computer Vision (2016)Google Scholar
  30. 30.
    Spadaro, L., Magno, M., Benini, L.: Kinetisee: A perpetual wearable camera acquisition system with a kinetic harvester: poster abstract. In: Proceedings of the 15th International Conference on Information Processing in Sensor Networks, IPSN 2016, pp. 68: 1–68: 2. IEEE Press, Piscataway (2016). http://dl.acm.org/citation.cfm?id=2959355.2959423
  31. 31.
    Su, Y.C., Grauman, K.: Detecting engagement in egocentric video. In: Proceedings of the European Conference on Computer Vision (2016)Google Scholar
  32. 32.
    Zhang, H., Li, L., Jia, W., Fernstrom, J.D., Sclabassi, R.J., Sun, M.: Recognizing physical activity from ego-motion of a camera. In: Proceedings of the IEEE Conference Engineering in Medicine and Biology (2010)Google Scholar

Copyright information

© Springer International Publishing Switzerland 2016

Authors and Affiliations

  • Francesco Paci
    • 1
    Email author
  • Lorenzo Baraldi
    • 2
  • Giuseppe Serra
    • 2
  • Rita Cucchiara
    • 2
  • Luca Benini
    • 1
    • 3
  1. 1.Univeristà di BolognaBolognaItaly
  2. 2.Università di Modena e Reggio EmiliaModenaItaly
  3. 3.ETH ZürichZürichSwitzerland

Personalised recommendations