Modelling and Recognition of Signed Expressions Using Subunits Obtained by Data–Driven Approach
The paper considers automatic vision based modelling and recognition of sign language expressions using smaller units than words. Modelling gestures with subunits is similar to modelling speech by means of phonemes. To define the subunits a data–driven procedure is proposed. The procedure consists in partitioning time series of feature vectors obtained from video material into subsequences which form homogeneous clusters. The cut points are determined by an optimisation procedure based on quality assessment of the resulting clusters. Then subunits are selected in two ways: as clusters’ representatives or as hidden Markov models of clusters. These two approaches result in differences in classifier design. Details of the solution and results of experiments on a database of 101 Polish words and 35 sentences used at the doctor’s and in the post office are given. Our subunit–based classifiers outperform their whole–word–based counterpart, which is particularly evident when new expressions are recognised on the basis of a small number of examples.
KeywordsSign language recognition Time series segmentation Time series clustering Evolutionary optimisation Hidden Markov models Computer vision Data mining
Unable to display preview. Download preview PDF.
- 3.Hendzel, J.K.: Dictionary of Polish Sign Language. OFFER, Olsztyn (1985) (in Polish)Google Scholar
- 4.Kong, W.W., Ranganath, S.: Automatic Hand Trajectory Segmentation and Phoneme Transcription for Sign Language. In: 8th IEEE International Conference on Automatic Face and Gesture Recognition, FG 2008, pp. 1–6 (2008)Google Scholar
- 9.Ratanamahatana, C.A., Keogh, E.: Three myths about dynamic time warping data mining. In: SIAM Int. Conf. on Data Mining, pp. 506–510 (2005)Google Scholar
- 11.Theodorakis, S., Pitsikalis, V., Maragos, P.: Model–level data-driven sub–units for signs in videos of continuous sign language. In: IEEE International Conference on Acoustics Speech and Signal Processing (ICASSP), pp. 2262–2265 (2010)Google Scholar
- 12.Theodoridis, A., Kontroumbas, K.: Pattern Recognition. Acad. Press, London (1999)Google Scholar
- 16.Xu, R., Wunsch, D.C.: Clustering. J. Wiley and Sons, Inc., Hoboken (2009)Google Scholar
- 17.Young, S., Evermann, G., Gales, M., Hain, T., Kershaw, D., Liu, X., Moore, G., Odell, J., Ollason, D., Povey, D., Valtchev, V., Woodland, P.: The HTK Book. Cambridge University (2006)Google Scholar