Advertisement

Chasing Feet in the Wild: A Proposed Egocentric Motion-Aware Gait Assessment Tool

  • Mina Nouredanesh
  • Aaron W. Li
  • Alan Godfrey
  • Jesse Hoey
  • James TungEmail author
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11134)

Abstract

Despite advances in gait analysis tools, including optical motion capture and wireless electrophysiology, our understanding of human mobility is largely limited to controlled conditions in a clinic and/or laboratory. In order to examine human mobility under natural conditions, or the ‘wild’, this paper presents a novel markerless model to obtain gait patterns by localizing feet in the egocentric video data. Based on a belt-mounted camera feed, the proposed hybrid FootChaser model consists of: (1) the FootRegionProposer, a ConvNet that proposes regions with high probability of containing feet in RGB frames (global appearance of feet), and (2) LocomoNet, which is sensitive to the periodic gait patterns, and further examines the temporal content in the stacks of optical flow corresponding to the proposed region. The LocomoNet significantly boosted the overall model’s result by filtering out the false positives proposed by the FootRegionProposer. This work advances our long-term objective to develop novel markerless models to extract spatiotemporal gait parameters, particularly step width, to complement existing inertial measurement unit (IMU) based methods.

Keywords

Ambulatory gait analysis Wearable sensors Deep convolutional neural networks Egocentric vision Optical flow 

Notes

Acknowledgments

Research supported by National Sciences and Engineering Research Council of Canada (NSERC), and by AGE-WELL NCE Inc. M. Nouredanesh was funded by an AGE-WELL Inc. (Canada’s technology and aging network) Graduate Scholarship.

References

  1. 1.
    Abebe, G., Cavallaro, A.: Inertial-vision: cross-domain knowledge transfer for wearable sensors. In: Proceedings of International Conference on Computer Vision Workshops (ICCVW), Venice, Italy, vol. 7 (2017)Google Scholar
  2. 2.
    Abebe, G., Cavallaro, A.: A long short-term memory convolutional neural network for first-person vision activity recognition. In: Proceedings of International Conference on Computer Vision Workshops (ICCVW) (2017)Google Scholar
  3. 3.
    Auvinet, E., Multon, F., Manning, V., Meunier, J., Cobb, J.: Validity and sensitivity of the longitudinal asymmetry index to detect gait asymmetry using microsoft kinect data. Gait Posture 51, 162–168 (2017)CrossRefGoogle Scholar
  4. 4.
    Taati, B., Mihailidis, A.: Vision-based approach for long-term mobility monitoring: single case study following total hip replacement. J. Rehabil. Res. Dev. 51(7), 1165 (2014)CrossRefGoogle Scholar
  5. 5.
    Bigy, A.A.M., Banitsas, K., Badii, A., Cosmas, J.: Recognition of postures and freezing of gait in Parkinson’s disease patients using microsoft kinect sensor. In: 2015 7th International IEEE/EMBS Conference on Neural Engineering (NER), pp. 731–734. IEEE (2015)Google Scholar
  6. 6.
    Brach, J.S., Berlin, J.E., VanSwearingen, J.M., Newman, A.B., Studenski, S.A.: Too much or too little step width variability is associated with a fall history in older persons who walk at or near normal gait speed. J. Neuroeng. Rehabil. 2(1), 21 (2005)CrossRefGoogle Scholar
  7. 7.
    Brattoli, B., Büchler, U., Wahl, A.S., Schwab, M.E., Ommer, B.: LSTM self-supervision for detailed behavior analysis. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), vol. 2 (2017)Google Scholar
  8. 8.
    Brodie, M.A., Lord, S.R., Coppens, M.J., Annegarn, J., Delbaere, K.: Eight-week remote monitoring using a freely worn device reveals unstable gait patterns in older fallers. IEEE Trans. Biomed. Eng. 62(11), 2588–2594 (2015)CrossRefGoogle Scholar
  9. 9.
    Charles, J., Pfister, T., Magee, D., Hogg, D., Zisserman, A.: Personalizing human video pose estimation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3063–3072 (2016)Google Scholar
  10. 10.
    Cippitelli, E., Gasparrini, S., Spinsante, S., Gambi, E.: Kinect as a tool for gait analysis: validation of a real-time joint extraction algorithm working in side view. Sensors 15(1), 1417–1434 (2015)CrossRefGoogle Scholar
  11. 11.
    Dai, J., Li, Y., He, K., Sun, J.: R-FCN: object detection via region-based fully convolutional networks. In: Advances in Neural Information Processing Systems, pp. 379–387 (2016)Google Scholar
  12. 12.
    Elhayek, A., et al.: Marconi-convnet-based marker-less motion capture in outdoor and indoor scenes. IEEE Trans. Pattern Anal. Mach. Intell. 39(3), 501–514 (2017)CrossRefGoogle Scholar
  13. 13.
    Everingham, M., Van Gool, L., Williams, C.K., Winn, J., Zisserman, A.: The pascal visual object classes (VOC) challenge. Int. J. Comput. Vis. 88(2), 303–338 (2010)CrossRefGoogle Scholar
  14. 14.
    Gabel, M., Gilad-Bachrach, R., Renshaw, E., Schuster, A.: Full body gait analysis with kinect. In: 2012 Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), pp. 1964–1967. IEEE (2012)Google Scholar
  15. 15.
    Güler, R.A., Neverova, N., Kokkinos, I.: DensePose: dense human pose estimation in the wild. arXiv preprint arXiv:1802.00434 (2018)
  16. 16.
    He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)Google Scholar
  17. 17.
    Hickey, A., Del Din, S., Rochester, L., Godfrey, A.: Detecting free-living steps and walking bouts: validating an algorithm for macro gait analysis. Physiol. Meas. 38(1), N1 (2016)CrossRefGoogle Scholar
  18. 18.
    Hu, R.Z.L., Hartfiel, A., Tung, J., Fakih, A., Hoey, J., Poupart, P.: 3D pose tracking of walker users’ lower limb with a structured-light camera on a moving platform. In: 2011 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 29–36. IEEE (2011)Google Scholar
  19. 19.
    Huang, J., et al.: Speed/accuracy trade-offs for modern convolutional object detectors. In: IEEE CVPR, vol. 4 (2017)Google Scholar
  20. 20.
    Huang, Y.: Towards accurate marker-less human shape and pose estimation over time. In: 2017 International Conference on 3D Vision (3DV), pp. 421–430. IEEE (2017)Google Scholar
  21. 21.
    Ihlen, E.A., Weiss, A., Bourke, A., Helbostad, J.L., Hausdorff, J.M.: The complexity of daily life walking in older adult community-dwelling fallers and non-fallers. J. Biomech. 49(9), 1420–1428 (2016)CrossRefGoogle Scholar
  22. 22.
    Iluz, T., et al.: Automated detection of missteps during community ambulation in patients with Parkinson’s disease: a new approach for quantifying fall risk in the community setting. J. Neuroeng. Rehabil. 11(1), 48 (2014)CrossRefGoogle Scholar
  23. 23.
    Jain, A., Tompson, J., LeCun, Y., Bregler, C.: MoDeep: a deep learning framework using motion features for human pose estimation. In: Cremers, D., Reid, I., Saito, H., Yang, M.-H. (eds.) ACCV 2014. LNCS, vol. 9004, pp. 302–315. Springer, Cham (2015).  https://doi.org/10.1007/978-3-319-16808-1_21CrossRefGoogle Scholar
  24. 24.
    Jiang, H., Grauman, K.: Seeing invisible poses: estimating 3D body pose from egocentric video. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3501–3509. IEEE (2017)Google Scholar
  25. 25.
    Kim, A., Kim, J., Rietdyk, S., Ziaie, B.: A wearable smartphone-enabled camera-based system for gait assessment. Gait Posture 42(2), 138–144 (2015)CrossRefGoogle Scholar
  26. 26.
    Liu, W., et al.: SSD: single shot MultiBox detector. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9905, pp. 21–37. Springer, Cham (2016).  https://doi.org/10.1007/978-3-319-46448-0_2CrossRefGoogle Scholar
  27. 27.
    Lord, S., Galna, B., Verghese, J., Coleman, S., Burn, D., Rochester, L.: Independent domains of gait in older adults and associated motor and nonmotor attributes: validation of a factor analysis approach. J. Gerontol. Series A Biomed. Sci. Med. Sci. 68(7), 820–827 (2012)CrossRefGoogle Scholar
  28. 28.
    Ma, M., Fan, H., Kitani, K.M.: Going deeper into first-person activity recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1894–1903 (2016)Google Scholar
  29. 29.
    von Marcard, T., Rosenhahn, B., Black, M.J., Pons-Moll, G.: Sparse inertial poser: automatic 3D human pose estimation from sparse IMUS. In: Computer Graphics Forum, vol. 36, pp. 349–360. Wiley Online Library (2017)Google Scholar
  30. 30.
    Mueller, F., Mehta, D., Sotnychenko, O., Sridhar, S., Casas, D., Theobalt, C.: Real-time hand tracking under occlusion from an egocentric RGB-D sensor. In: Proceedings of International Conference on Computer Vision (ICCV), vol. 10 (2017)Google Scholar
  31. 31.
    Ng, S., Fakih, A., Fourney, A., Poupart, P., Zelek, J.: Towards a mobility diagnostic tool: tracking rollator users’ leg pose with a monocular vision system. In: International Conference of IEEE Engineering in Medicine and Biology Society (EMBC), vol. 1, pp. 662–666 (2009)Google Scholar
  32. 32.
    Nouredanesh, M., McCormick, A., Kukreja, S.L., Tung, J.: Wearable vision detection of environmental fall risk using Gabor Barcodes. In: 2016 6th IEEE International Conference on Biomedical Robotics and Biomechatronics (BioRob), pp. 956–956. IEEE (2016)Google Scholar
  33. 33.
    Nouredanesh, M., McCormick, A., Kukreja, S.L., Tung, J.: Wearable vision detection of environmental fall risks using convolutional neural networks. arXiv preprint arXiv:1611.00684 (2016)
  34. 34.
    Oquab, M., Bottou, L., Laptev, I., Sivic, J.: Learning and transferring mid-level image representations using convolutional neural networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1717–1724 (2014)Google Scholar
  35. 35.
    Page, S., Martins, M.M., Saint-Bauzel, L., Santos, C.P., Pasqui, V.: Fast embedded feet pose estimation based on a depth camera for smart walker. In: 2015 IEEE International Conference on Robotics and Automation (ICRA), pp. 4224–4229. IEEE (2015)Google Scholar
  36. 36.
    Phillips, L.J., et al.: Using embedded sensors in independent living to predict gait changes and falls. West. J. Nurs. Res. 39(1), 78–94 (2017)CrossRefGoogle Scholar
  37. 37.
    Redmon, J., Farhadi, A.: Yolo9000: better, faster, stronger. arXiv preprint arXiv:1612.08242 (2017)
  38. 38.
    Ren, S., He, K., Girshick, R., Sun, J.: Faster R-CNN: towards real-time object detection with region proposal networks. In: Advances in Neural Information Processing Systems, pp. 91–99 (2015)Google Scholar
  39. 39.
    Rhodin, H., et al.: EgoCap: egocentric marker-less motion capture with two fisheye cameras. ACM Trans. Graph. (TOG) 35(6), 162 (2016)CrossRefGoogle Scholar
  40. 40.
    Rogez, G., Supancic, J.S., Ramanan, D.: First-person pose recognition using egocentric workspaces. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4325–4333 (2015)Google Scholar
  41. 41.
    Rozantsev, A., Lepetit, V., Fua, P.: Flying objects detection from a single moving camera. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4128–4136 (2015)Google Scholar
  42. 42.
    Russakovsky, O., et al.: Imagenet large scale visual recognition challenge. Int. J. Comput. Vis. 115(3), 211–252 (2015)MathSciNetCrossRefGoogle Scholar
  43. 43.
    Shiratori, T., Park, H.S., Sheikh, Y., Hodgins, J.K., et al.: Motion capture from body mounted cameras. US Patent 8,786,680, 22 July 2014Google Scholar
  44. 44.
    Simonyan, K., Zisserman, A.: Two-stream convolutional networks for action recognition in videos. In: Advances in Neural Information Processing Systems, pp. 568–576 (2014)Google Scholar
  45. 45.
    Song, S., et al.: Multimodal multi-stream deep learning for egocentric activity recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 24–31 (2016)Google Scholar
  46. 46.
    Taylor, K., et al.: Context focused older adult mobility and gait assessment. In: 2015 37th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), pp. 6943–6946. IEEE (2015)Google Scholar
  47. 47.
    Tekin, B., Rozantsev, A., Lepetit, V., Fua, P.: Direct prediction of 3D body poses from motion compensated sequences. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 991–1000 (2016)Google Scholar
  48. 48.
    Tzutalin: Labelimg. Git code (2015). https://github.com/tzutalin/labelImg
  49. 49.
    Wang, L., et al.: Temporal segment networks: towards good practices for deep action recognition. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9912, pp. 20–36. Springer, Cham (2016).  https://doi.org/10.1007/978-3-319-46484-8_2CrossRefGoogle Scholar
  50. 50.
    Wang, Y., Liu, Y., Tong, X., Dai, Q., Tan, P.: Outdoor markerless motion capture with sparse handheld video cameras. IEEE Trans. Vis. Comput. Graph. 24(5), 1856–1866 (2018)CrossRefGoogle Scholar
  51. 51.
    Xu, W., et al.: Mo2cap2: real-time mobile 3d motion capture with a cap-mounted fisheye camera. arXiv preprint arXiv:1803.05959 (2018)
  52. 52.
    Yu, A., Grauman, K.: Fine-grained visual comparisons with local learning. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 192–199 (2014)Google Scholar
  53. 53.
    Zach, C., Pock, T., Bischof, H.: A duality based approach for realtime TV-L1 optical flow. In: Hamprecht, F.A., Schnörr, C., Jähne, B. (eds.) DAGM 2007. LNCS, vol. 4713, pp. 214–223. Springer, Heidelberg (2007).  https://doi.org/10.1007/978-3-540-74936-3_22CrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  1. 1.Department of Mechanical and Mechatronics EngineeringUniversity of WaterlooWaterlooCanada
  2. 2.David R. Cheriton School of Computer ScienceUniversity of WaterlooWaterlooCanada
  3. 3.Department of Computer and Information ScienceNorthumbria UniversityNewcastle upon TyneUK

Personalised recommendations