Skip to main content

Attention to Warp: Deep Metric Learning for Multivariate Time Series

Part of the Lecture Notes in Computer Science book series (LNIP,volume 12823)

Abstract

Deep time series metric learning is challenging due to the difficult trade-off between temporal invariance to nonlinear distortion and discriminative power in identifying non-matching sequences. This paper proposes a novel neural network-based approach for robust yet discriminative time series classification and verification. This approach adapts a parameterized attention model to time warping for greater and more adaptive temporal invariance. It is robust against not only local but also large global distortions, so that even matching pairs that do not satisfy the monotonicity, continuity, and boundary conditions can still be successfully identified. Learning of this model is further guided by dynamic time warping to impose temporal constraints for stabilized training and higher discriminative power. It can learn to augment the inter-class variation through warping, so that similar but different classes can be effectively distinguished. We experimentally demonstrate the superiority of the proposed approach over previous non-parametric and deep models by combining it with a deep online signature verification framework, after confirming its promising behavior in single-letter handwriting classification on the Unipen dataset.

Keywords

  • Attention model
  • Dynamic time warping
  • Metric learning
  • Signature verification

This is a preview of subscription content, access via your institution.

Buying options

Chapter
USD   29.95
Price excludes VAT (USA)
  • DOI: 10.1007/978-3-030-86334-0_23
  • Chapter length: 16 pages
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
eBook
USD   89.00
Price excludes VAT (USA)
  • ISBN: 978-3-030-86334-0
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
Softcover Book
USD   119.99
Price excludes VAT (USA)
Fig. 1.
Fig. 2.
Fig. 3.
Fig. 4.
Fig. 5.
Fig. 6.
Fig. 7.
Fig. 8.

Notes

  1. 1.

    http://human.ait.kyushu-u.ac.jp/~matsuo/ICDAR2021_Appendix.pdf.

References

  1. Ahrabian, K., BabaAli, B.: Usage of autoencoders and Siamese networks for online handwritten signature verification. Neural Comput. Appl. 31(12), 9321–9334 (2019)

    CrossRef  Google Scholar 

  2. Bagnall, A.J., Lines, J., Bostrom, A., Large, J., Keogh, E.J.: The great time series classification bake off: a review and experimental evaluation of recent algorithmic advances. Data Min. Knowl. Discov. 31(3), 606–660 (2017)

    MathSciNet  CrossRef  Google Scholar 

  3. Bahdanau, D., Cho, K., Bengio, Y.: Neural machine translation by jointly learning to align and translate. In: ICLR (2015)

    Google Scholar 

  4. Bromley, J., Guyon, I., LeCun, Y., Säckinger, E., Shah, R.: Signature verification using a Siamese time delay neural network. In: NIPS, pp. 737–744 (1993)

    Google Scholar 

  5. Che, Z., He, X., Xu, K., Liu, Y.: DECADE: a deep metric learning model for multivariate time series. In: Workshop on Mining and Learning from Time Series (2017)

    Google Scholar 

  6. Coskun, H., Tan, D.J., Conjeti, S., Navab, N., Tombari, F.: Human motion analysis with deep metric learning. In: ECCV, pp. 693–710 (2018)

    Google Scholar 

  7. Cpalka, K., Zalasinski, M., Rutkowski, L.: A new algorithm for identity verification based on the analysis of a handwritten dynamic signature. Appl. Soft Comput. 43, 47–56 (2016)

    CrossRef  Google Scholar 

  8. Faúndez-Zanuy, M.: On-line signature recognition based on VQ-DTW. Pattern Recognit. 40(3), 981–992 (2007)

    CrossRef  Google Scholar 

  9. Fawaz, H.I., Forestier, G., Weber, J., Idoumghar, L., Muller, P.: Deep learning for time series classification: a review. Data Min. Knowl. Discov. 33(4), 917–963 (2019)

    MathSciNet  CrossRef  Google Scholar 

  10. Fiérrez-Aguilar, J., Ortega-Garcia, J., Ramos, D., Gonzalez-Rodriguez, J.: HMM-based on-line signature verification: feature extraction and signature modeling. Pattern Recognit. Lett. 28(16), 2325–2334 (2007)

    CrossRef  Google Scholar 

  11. Ge, X., Smyth, P.: Deformable Markov model templates for time-series pattern matching. In: KDD, pp. 81–90 (2000)

    Google Scholar 

  12. Grabocka, J., Schmidt-Thieme, L.: NeuralWarp: time-series similarity with warping networks. CoRR (2018)

    Google Scholar 

  13. Guyon, I., Schomaker, L., Plamondon, R., Liberman, M., Janet, S.: UNIPEN project of on-line data exchange and recognizer benchmarks. In: ICPR, pp. 29–33 (1994)

    Google Scholar 

  14. Hadsell, R., Chopra, S., LeCun, Y.: Dimensionality reduction by learning an invariant mapping. In: CVPR, pp. 1735–1742 (2006)

    Google Scholar 

  15. He, K., Zhang, X., Ren, S., Sun, J.: Delving deep into rectifiers: Surpassing human-level performance on ImageNet classification. In: ICCV, pp. 1026–1034 (2015)

    Google Scholar 

  16. Kholmatov, A., Yanikoglu, B.A.: Identity authentication using improved online signature verification method. Pattern Recognit. Lett. 26(15), 2400–2408 (2005)

    CrossRef  Google Scholar 

  17. Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. In: ICLR (2015)

    Google Scholar 

  18. Lai, S., Jin, L., Lin, L., Zhu, Y., Mao, H.: SynSig2Vec: learning representations from synthetic dynamic signatures for real-world verification. In: AAAI, pp. 735–742 (2020)

    Google Scholar 

  19. Li, C., et al.: A stroke-based RNN for writer-independent online signature verification. In: ICDAR, pp. 526–532 (2019)

    Google Scholar 

  20. Lines, J., Bagnall, A.J.: Time series classification with ensembles of elastic distance measures. Data Min. Knowl. Discov. 29(3), 565–592 (2015)

    MathSciNet  CrossRef  Google Scholar 

  21. Luong, T., Pham, H., Manning, C.D.: Effective approaches to attention-based neural machine translation. In: EMNLP, pp. 1412–1421 (2015)

    Google Scholar 

  22. Marteau, P.: Time warp edit distance with stiffness adjustment for time series matching. IEEE Trans. Pattern Anal. Mach. Intell. 31(2), 306–318 (2009)

    CrossRef  Google Scholar 

  23. Martinez-Diaz, M., Fiérrez, J., Krish, R.P., Galbally, J.: Mobile signature verification: Feature robustness and performance comparison. IET Biom. 3(4), 267–277 (2014)

    CrossRef  Google Scholar 

  24. Mueller, J., Thyagarajan, A.: Siamese recurrent architectures for learning sentence similarity. In: AAAI, pp. 2786–2792 (2016)

    Google Scholar 

  25. Nanni, L., Lumini, A.: A novel local on-line signature verification system. Pattern Recognit. Lett. 29(5), 559–568 (2008)

    CrossRef  Google Scholar 

  26. Ortega-Garcia, J., et al.: MCYT baseline corpus: a bimodal biometric database. IEE Proc. Vis. Image Sig. Process. 150(6), 395–401 (2003)

    CrossRef  Google Scholar 

  27. Rakthanmanon, T., et al.: Searching and mining trillions of time series subsequences under dynamic time warping. In: KDD, pp. 262–270 (2012)

    Google Scholar 

  28. Ronneberger, O., Fischer, P., Brox, T.: U-net: convolutional networks for biomedical image segmentation. In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (eds.) MICCAI 2015. LNCS, vol. 9351, pp. 234–241. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-24574-4_28

    CrossRef  Google Scholar 

  29. Roy, D., Mohan, C.K., Murty, K.S.R.: Action recognition based on discriminative embedding of actions using Siamese networks. In: ICIP, pp. 3473–3477 (2018)

    Google Scholar 

  30. Sae-Bae, N., Memon, N.D.: Online signature verification on mobile devices. IEEE Trans. Inf. Forensics Secur. 9(6), 933–947 (2014)

    CrossRef  Google Scholar 

  31. Sakoe, H., Chiba, S.: Dynamic programming algorithm optimization for spoken word recognition. IEEE Trans. Acoust. Speech Signal Process. 26(1), 43–49 (1978)

    CrossRef  Google Scholar 

  32. Tallec, C., Ollivier, Y.: Can recurrent neural networks warp time? In: ICLR (2018)

    Google Scholar 

  33. Tang, L., Kang, W., Fang, Y.: Information divergence-based matching strategy for online signature verification. IEEE Trans. Inf. Forensics Secur. 13(4), 861–873 (2018)

    CrossRef  Google Scholar 

  34. Tolosana, R., Vera-Rodríguez, R., Fiérrez, J., Ortega-Garcia, J.: Exploring recurrent neural networks for on-line handwritten signature biometrics. IEEE Access 6, 5128–5138 (2018)

    CrossRef  Google Scholar 

  35. Tolosana, R., Vera-Rodríguez, R., Fiérrez, J., Ortega-Garcia, J.: DeepSign: deep on-line signature verification. IEEE Trans. Biom. Behav. Identity Sci. 3(2), 229–239 (2021)

    CrossRef  Google Scholar 

  36. Vaswani, A., et al.: Attention is all you need. In: NIPS, pp. 5998–6008 (2017)

    Google Scholar 

  37. Vivaracho-Pascual, C., Faúndez-Zanuy, M., Pascual, J.M.: An efficient low cost approach for on-line signature recognition based on length normalization and fractional distances. Pattern Recognit. 42(1), 183–193 (2009)

    CrossRef  Google Scholar 

  38. Vorugunti, C.S., Guru, D.S., Mukherjee, P., Pulabaigari, V.: OSVNet: convolutional Siamese network for writer independent online signature verification. In: ICDAR, pp. 1470–1475 (2019)

    Google Scholar 

  39. Wu, X., Kimura, A., Iwana, B.K., Uchida, S., Kashino, K.: Deep dynamic time warping: end-to-end local representation learning for online signature verification. In: ICDAR, pp. 1103–1110 (2019)

    Google Scholar 

  40. Wu, X., Kimura, A., Uchida, S., Kashino, K.: Prewarping Siamese network: Learning local representations for online signature verification. In: ICASSP, pp. 2467–2471 (2019)

    Google Scholar 

  41. Yanikoglu, B.A., Kholmatov, A.: Online signature verification using Fourier descriptors. EURASIP J. Adv. Sig. Process. (2009)

    Google Scholar 

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Shinnosuke Matsuo .

Editor information

Editors and Affiliations

Rights and permissions

Reprints and Permissions

Copyright information

© 2021 Springer Nature Switzerland AG

About this paper

Verify currency and authenticity via CrossMark

Cite this paper

Matsuo, S. et al. (2021). Attention to Warp: Deep Metric Learning for Multivariate Time Series. In: Lladós, J., Lopresti, D., Uchida, S. (eds) Document Analysis and Recognition – ICDAR 2021. ICDAR 2021. Lecture Notes in Computer Science(), vol 12823. Springer, Cham. https://doi.org/10.1007/978-3-030-86334-0_23

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-86334-0_23

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-86333-3

  • Online ISBN: 978-3-030-86334-0

  • eBook Packages: Computer ScienceComputer Science (R0)