Modelling Combined Handwriting and Speech Modalities

  • Andreas Humm
  • Jean Hennebert
  • Rolf Ingold
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 4642)


We are reporting on consolidated results obtained with a new user authentication system based on combined acquisition of online handwriting and speech signals. In our approach, signals are recorded by asking the user to say what she or he is simultaneously writing. This methodology has the clear advantage of acquiring two sources of biometric information at no extra cost in terms of time or inconvenience. We are proposing here two scenarios of use: spoken signature where the user signs and speaks at the same time and spoken handwriting where the user writes and says what is written. These two scenarios are implemented and fully evaluated using a verification system based on Gaussian Mixture Models (GMMs). The evaluation is performed on MyIdea, a realistic multimodal biometric database. Results show that the use of both speech and handwriting modalities outperforms significantly these modalities used alone, for both scenarios. Comparisons between the spoken signature and spoken handwriting scenarios are also drawn.


Speech Signal Gaussian Mixture Model Time Variability Equal Error Rate Biometric System 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


  1. 1.
    Reynolds, D.: An overview of automatic speaker recognition technology. In: Proc. IEEE ICASSP, vol. 4, pp. 4072–4075 (2002)Google Scholar
  2. 2.
    Plamondon, R., Lorette, G.: Automatic signature verification and writer identification - the state of the art. Pattern Recognition 22(2), 107–131 (1989)CrossRefGoogle Scholar
  3. 3.
    Leclerc, F., Plamondon, R.: Automatic signature verification: the state of the art–1989-1993. Int’l J. Pattern Rec. and Art. Intelligence 8(3), 643–660 (1994)CrossRefGoogle Scholar
  4. 4.
    Liwicki, M., Schlapbach, A., Bunke, H., Bengio, S., Mariéthoz, J., Richiardi, J.: Writer identification for smart meeting room systems. In: Proceedings of the 7th International Workshop on Document Analysis Systems, pp. 186–195 (2006)Google Scholar
  5. 5.
    Nakamura, Y., Kidode, M.: Online writer verification using kanji handwriting. In: Gunsel, B., Jain, A.K., Tekalp, A.M., Sankur, B. (eds.) MRCS 2006. LNCS, vol. 4105, pp. 207–214. Springer, Heidelberg (2006)CrossRefGoogle Scholar
  6. 6.
    Krawczyk, S., Jain, A.K.: Securing electronic medical records using biometric authentication. In: Kanade, T., Jain, A., Ratha, N.K. (eds.) AVBPA 2005. LNCS, vol. 3546, pp. 1110–1119. Springer, Heidelberg (2005)Google Scholar
  7. 7.
    Fuentes, M., et al.: Identity verification by fusion of biometric data: On-line signature and speech. In: Proc. COST 275 Workshop on The Advent of Biometrics on the Internet, Rome, Italy, November 2002, pp. 83–86 (2002)Google Scholar
  8. 8.
    Ly-Van, B., et al.: Signature with text-dependent and text-independent speech for robust identity verification. In: Proc. Workshop MMUA, pp. 13–18 (2003)Google Scholar
  9. 9.
    Koreman, J., et al.: Multi-modal biometric authentication on the securephone pda. In: Proc. Workshop MMUA, Toulouse (2006)Google Scholar
  10. 10.
    Humm, A., Hennebert, J., Ingold, R.: Gaussian mixture models for chasm signature verification. In: 3rd Joint Workshop on Multimodal Interaction and Related Machine Learning Algorithms, Washington (2006)Google Scholar
  11. 11.
    Hennebert, J., Humm, A., Ingold, R.: Modelling spoken signatures with gaussian mixture model adaptation. In: 32nd ICASSP, Honolulu (2007)Google Scholar
  12. 12.
    Humm, A., Ingold, R., Hennebert, J.: Spoken handwriting verification using statistical models. In: ICDAR (accepted for publication, 2007)Google Scholar
  13. 13.
    Dumas, B., et al.: Myidea - multimodal biometrics database, description of acquisition protocols. In: Proc. of Third COST 275 Workshop (COST 275), Hatfield (UK) (October 27-28 2005), pp. 59–62 (2005)Google Scholar
  14. 14.
    Hennebert, J., et al.: Myidea database (2005),
  15. 15.
    Humm, A., Hennebert, J., Ingold, R.: Scenario and survey of combined handwriting and speech modalities for user authentication. In: RASC 2006. 6th Int’l. Conf. on Recent Advances in Soft Computing, Canterburry, Kent, UK, pp. 496–501 (2006)Google Scholar
  16. 16.
    Humm, A., Hennebert, J., Ingold, R.: Combined handwriting and speech modalities for user authentication. Technical Report 06-05, University of Fribourg, Department of Informatics (2006)Google Scholar
  17. 17.
    Van Ly, B., Garcia-Salicetti, S., Dorizzi, B.: Fusion of hmm’s likelihood and viterbi path for on-line signature verification. In: Biometrics Authentication Workshop, Prague (May 15, 2004)Google Scholar
  18. 18.
    Reynolds, D., Quatieri, T., Dunn, R.: Speaker verification using adapted gaussian mixture models. Digital Signal Processing 10, 19–41 (2000)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2007

Authors and Affiliations

  • Andreas Humm
    • 1
  • Jean Hennebert
    • 1
  • Rolf Ingold
    • 1
  1. 1.Université de Fribourg, Boulevard de Pérolles 90, 1700 FribourgSwitzerland

Personalised recommendations