Workflow Activity Monitoring Using Dynamics of Pair-Wise Qualitative Spatial Relations

Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 7131)


We present a method for real-time monitoring of workflows in a constrained environment. The monitoring system should not only be able to recognise the current step but also provide instructions about the possible next steps in an ongoing workflow. In this paper, we address this issue by using a robust approach (HMM-pLSA) which relies on a Hidden Markov Model (HMM) and generative model such as probabilistic Latent Semantic Analysis (pLSA). The proposed method exploits the dynamics of the qualitative spatial relation between pairs of objects involved in a workflow. The novel view-invariant relational feature is based on distance and its rate of change in 3D space. The multiple pair-wise relational features are represented in a multi-dimensional relational state space using an HMM. The workflow monitoring task is inferred from the relational state space using pLSA on datasets, which consist of workflow activities such as ‘hammering nails’ and ‘driving screws’. The proposed approach is evaluated for both ‘off-line’ (complete observation) and ‘on-line’ (partial observation). The evaluation of the novel approach justifies the robustness of the technique in overcoming issues of noise evolving from object tracking and occlusions.


Qualitative Spatio-temporal Relations Workers Instructions Activity Recognition Hidden Markov Model (HMM) Probabilistic Latent Scemantic Analysis (pLSA) 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Ivanov, Y., Bobick, A.F.: of visual acticities and interactions by stochastic parsing. IEEE Trans. on PAMI 22(8), 852–872 (2000)CrossRefGoogle Scholar
  2. 2.
    Allen, J.F.: Maintaining knowledge about temporal intervals. Communications of the ACM 26(11), 832–843 (1983)CrossRefzbMATHGoogle Scholar
  3. 3.
    Baum, L.E., Petrie, T., Soules, G., Weiss, N.: A maximization technique occurring in the statistical analysis of probabilistic functions of Markov chains. Annals of Mathematical Statistics 41(1), 164–171 (1970)CrossRefzbMATHGoogle Scholar
  4. 4.
    Blei, D.M., Ng, A., Jordan, M.: Latent dirichlet allocation. J. Mach. Learn. Res. 3, 993–1022 (2003)zbMATHGoogle Scholar
  5. 5.
    Brand, M., Oliver, N., Pentland, A.: Coupled Hidden Markov models for complex action recognition. In: Proc. of IEEE CVPR, pp. 994–999 (1997)Google Scholar
  6. 6.
    Bui, H., Venkatesh, S., West, G.: Policy recognition in the abstract hidden markov model. Journal of Artificial Intelligence Research 17, 451–499 (2002)zbMATHGoogle Scholar
  7. 7.
    Chang, C.C., Lin, C.J.: LIBSVM: a library for support vector machines (2001),
  8. 8.
    Dempster, A.P., Laird, N.M., Rubin, D.B.: Maximum likelihood from incomplete data via the em algorithm. Royal Statistical Society 39(1), 1–38 (1977)zbMATHGoogle Scholar
  9. 9.
    Grimson, W.E.L., Stauffer, C., Romano, R., Lee, L.: Using adaptive tracking to classify and monitor activities in a site. In: Proc. of IEEE CVPR, pp. 22–29 (1998)Google Scholar
  10. 10.
    Hofmann, T.: Probabilistic latent semantic analysis. In: Proc. of Uncertainty in Artificial Intelligence, UAI 1999, pp. 289–296 (1999)Google Scholar
  11. 11.
    Hongeng, S., Nevatia, R.: Multi-agent event recognition. In: Proc. of ICCV, vol. 2, pp. 84–91 (2001)Google Scholar
  12. 12.
    Intille, S.S., Bobick, A.F.: A framework for recognizing multi-agent action from visual evidence. In: AAAI 1999, pp. 518–525 (1999)Google Scholar
  13. 13.
    Jaeger, H.: The “echo state” approach to analysing and training recurrent neural networks. Tech. rep., Fraunhofer Institute for Autonomous Intelligent Systems (December 2001)Google Scholar
  14. 14.
    Johnson, N., Hogg, D.C.: Learning the distribution of object trajectories for event recognition. Image Vision Comput. 14(8), 609–615 (1996)CrossRefGoogle Scholar
  15. 15.
    Moore, D., Essa, I.: Recognizing multitasked activities from video using stochastic context-free grammar. In: Proc. AAAI National Conf. on AI, pp. 770–776 (2002)Google Scholar
  16. 16.
    Nguyen, N., Phung, D., Venkatesh, S., Bui, H.: Learning and detecting activities from movement trajectories using the hierarchical hidden Markov model. In: Proc. of IEEE CVPR, vol. 2, pp. 955–960 (2005)Google Scholar
  17. 17.
    Oliver, N., Garg, A., Horvitz, E.: Layered representations for learning and inferring office activity from multiple sensory channels. Computer Vision and Image Understanding 96(2), 163–180 (2004)CrossRefGoogle Scholar
  18. 18.
    Padoy, N., Weinl, D.M.D., Berger, M.O., Navab, N.: Workflow monitoring based on 3D motion features. In: Proc. of ICCV Workshop on Video-oriented Object and Event Classification (2009)Google Scholar
  19. 19.
    Pinhanez, C., Bobick, A.: Human action detection using pnf propagation of temporal constraints. In: Proc. of IEEE CVPR (1998)Google Scholar
  20. 20.
    Schüldt, C., Laptev, I., Caputo, B.: Recognizing human actions: A local svm approach. In: Proc. ICPR, pp. 32–36 (2004)Google Scholar
  21. 21.
    Shi, Y., Huang, Y., Minnen, D., Bobick, A., Essa, I.: Propagation networks for recognition of partially ordered sequential action. In: Proc. of CVPR, vol. 2, pp. 862–869 (2004)Google Scholar
  22. 22.
    Veres, G., Grabner, H., Middleton, L., Van Gool, L.: Automatic Workflow Monitoring in Industrial Environments. In: Kimmel, R., Klette, R., Sugimoto, A. (eds.) ACCV 2010, Part I. LNCS, vol. 6492, pp. 200–213. Springer, Heidelberg (2011)CrossRefGoogle Scholar
  23. 23.
    Vicon Systems,
  24. 24.
    Viterbi, A.: Error bounds for convolutional codes and an asymptotically optimum decoding algorithm. IEEE Trans. on Information Theory 13(2), 260–269 (1967)CrossRefzbMATHGoogle Scholar
  25. 25.
    Vogler, C., Metaxas, D.: Parallel Hidden Markov Models for American sign language recognition. In: Proc. of IEEE ICCV, vol. 1, pp. 116–122 (1999)Google Scholar
  26. 26.
    Wang, X., Ma, X., Grimson, W.E.L.: Unsupervised activity perception in crowded and complicated scenes using hierarchical Bayesian models. IEEE Trans. PAMI 31, 539–555 (2009)CrossRefGoogle Scholar
  27. 27.
    Wanstall, B.: HUD on the Head for Combat Pilots. Interavia 44, 334–338 (1989)Google Scholar
  28. 28.
    Young, S.J.: The htk hidden Markov model toolkit: Design and philosophy. Tech. rep., Cambridge University Engineering Department (September 1994)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2012

Authors and Affiliations

  1. 1.School of ComputingUniversity of LeedsLeedsUK

Personalised recommendations