Skip to main content

Towards Automated Performance Status Assessment: Temporal Alignment of Motion Skeleton Time Series

  • Chapter
  • First Online:
Explainable AI in Healthcare and Medicine

Part of the book series: Studies in Computational Intelligence ((SCI,volume 914))

Abstract

Patient Performance Status (PS) is used in cancer medicine to predict prognosis and prescribe treatment. Today, PS assessments rely on assessor’s observation, which is susceptible to biases. A motion tracking system can be used to supplement PS assessments, by recording and analyzing patient’s movement as they perform a standardized mobility task e.g. getting up from office chair to sit on examination table. A temporal alignment of the extracted motion skeleton time series is then needed to enable comparison of corresponding motions in mobility task across recordings. In this paper, we apply existing state-of-the-art temporal alignment algorithms to the extracted time series and evaluate their performance in aligning the keyframes that separate corresponding motions. We then identify key characteristics of these time series that the existing algorithms are not able to exploit correctly: task left-right invariance and vertical-horizontal relative importance. We thus propose Invariant Weighted Dynamic Time Warping (IW-DTW), which takes advantage of these key characteristics. In an evaluation against state-of-the-art algorithms, IW-DTW outperforms them in aligning the keyframes where these key characteristics are present.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 149.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 199.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 199.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Abid, A., Zou, J.: Autowarp: learning a warping distance from unlabeled time series using sequence autoencoders. In: Advances in Neural Information Processing Systems (NeurIPS), October 2018

    Google Scholar 

  2. Broderick, J.E., May, M., Schwartz, J.E., Li, M., Mejia, A., Nocera, L., Kolatkar, A., Ueno, N.T., Yennu, S., Lee, J.S.H., Hanlon, S.E., Cozzens Philips, F.A., Shahabi, C., Kuhn, P., Nieva, J.: Patient reported outcomes can improve performance status assessment: a pilot study. J. Patient-Reported Outcomes 3(1), 41 (2019)

    Article  Google Scholar 

  3. Hasnain, Z., Li, M., Dorff, T., Quinn, D., Ueno, N.T., Yennu, S., Kolatkar, A., Shahabi, C., Nocera, L., Nieva, J., Kuhn, P., Newton, P.K.: Low-dimensional dynamical characterization of human performance of cancer patients using motion data. Clinical Biomech. 56(December 2017), 61–69 (2018)

    Google Scholar 

  4. Kao, J.Y., Nguyen, M., Nocera, L., Shahabi, C., Ortega, A., Winstein, C., Sorkhoh, I., Chung, Y.C., Chen, Y.A., Bacon, H.: Validation of Automated Mobility Assessment Using a Single 3D Sensor. In: Hua, G., Jégou, H. (eds.) European Conference on Computer Vision (ECCV) Workshops, vol. 3, pp. 162–177. Springer, Cham (2016)

    Google Scholar 

  5. Nguyen, M.N.B., Hasnain, Z., Li, M., Dorff, T., Quinn, D., Purushotham, S., Nocera, L., Newton, P.K., Kuhn, P., Nieva, J., Shahabi, C.: Mining Human Mobility to Quantify Performance Status. In: IEEE International Conference on Data Mining (ICDM) Workshop (2017)

    Google Scholar 

  6. Ofli, F., Chaudhry, R., Kurillo, G.: Berkeley Multimodal Human Action Database (MHAD). In: IEEE Workshop on Applications of Computer Vision (WACV), pp. 53–60 (2013)

    Google Scholar 

  7. Oken, M.M., Creech, R.H., Tormey, D.C., Horton, J., Davis, T.E., McFadden, E.T., Carbone, P.P.: Toxicity and response criteria of the Eastern Cooperative Oncology Group. Am. J. Clin. Oncol. 5(6), 649–656 (1982)

    Article  Google Scholar 

  8. Rabiner, L., Juang, B.H.: Fundamentals of Speech Recognition. Prentice-Hall Inc., Upper Saddle River (1993)

    Google Scholar 

  9. Schnadig, I.D., Fromme, E.K., Loprinzi, C.L., Sloan, J.A., Mori, M., Li, H., Beer, T.M.: Patient-physician disagreement regarding performance status is associated with worse survivorship in patients with advanced cancer. Cancer 113(8), 2205–2214 (2008)

    Article  Google Scholar 

  10. Trigeorgis, G., Nicolaou, M.A., Zafeiriou, S., Schuller, B.W.: Deep canonical time warping. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5110–5118 (2016)

    Google Scholar 

  11. Vu, H.T., Carey, C.J., Mahadevan, S.: Manifold warping: Manifold alignment over time. In: Proceedings of the National Conference on Artificial Intelligence, vol. 2, pp. 1155–1161 (2012)

    Google Scholar 

  12. Wang, R., Medioni, G., Winstein, C.J., Blanco, C.: Home monitoring musculo-skeletal disorders with a single 3D sensor. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops, pp. 521–528 (2013)

    Google Scholar 

  13. Zhou, F., De La Torre, F.: Generalized canonical time warping. IEEE Trans. Pattern Anal. Mach. Intell. 38(2), 279–294 (2016)

    Article  Google Scholar 

  14. Zhou, F., de la Torre, F.: Canonical time warping for alignment of human behavior. In: Advances in Neural Information Processing Systems (NeurIPS), pp. 1–9 (2009)

    Google Scholar 

Download references

Acknowledgements

This research has been funded in part by US National Cancer Institute under award number P30CA014089, USC Integrated Media Systems Center (IMSC), and unrestricted cash gifts from Oracle, Microsoft, and Google. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of any of the sponsors. T. Nilanon was also supported in part by DPST, IPST, Thailand.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Tanachat Nilanon .

Editor information

Editors and Affiliations

Rights and permissions

Reprints and permissions

Copyright information

© 2021 The Editor(s) (if applicable) and The Author(s), under exclusive license to Springer Nature Switzerland AG

About this chapter

Check for updates. Verify currency and authenticity via CrossMark

Cite this chapter

Nilanon, T., Nocera, L.P., Nieva, J.J., Shahabi, C. (2021). Towards Automated Performance Status Assessment: Temporal Alignment of Motion Skeleton Time Series. In: Shaban-Nejad, A., Michalowski, M., Buckeridge, D.L. (eds) Explainable AI in Healthcare and Medicine. Studies in Computational Intelligence, vol 914. Springer, Cham. https://doi.org/10.1007/978-3-030-53352-6_32

Download citation

Publish with us

Policies and ethics