Discovering activity patterns in office environment using a network of low-resolution visual sensors

  • Mohamed Eldib
  • Francis Deboeverie
  • Wilfried Philips
  • Hamid Aghajan
Original Research


Understanding activity patterns in office environments is important in order to increase workers’ comfort and productivity. This paper proposes an automated system for discovering activity patterns of multiple persons in a work environment using a network of cheap low-resolution visual sensors (900 pixels). Firstly, the users’ locations are obtained from a robust people tracker based on recursive maximum likelihood principles. Secondly, based on the users’ mobility tracks, the high density positions are found using a bivariate kernel density estimation. Then, the hotspots are detected using a confidence region estimation. Thirdly, we analyze the individual’s tracks to find the starting and ending hotspots. The starting and ending hotspots form an observation sequence, where the user’s presence and absence are detected using three powerful Probabilistic Graphical Models (PGMs). We describe two approaches to identify the user’s status: a single model approach and a two-model mining approach. We evaluate both approaches on video sequences captured in a real work environment, where the persons’ daily routines are recorded over 5 months. We show how the second approach achieves a better performance than the first approach. Routines dominating the entire group’s activities are identified with a methodology based on the Latent Dirichlet Allocation topic model. We also detect routines which are characteristic of persons. More specifically, we perform various analysis to determine regions with high variations, which may correspond to specific events.


Visual sensor network Supervised learning Probabilistic graphical models Topic models Sequence mining 

List of symbols

\(\left( x',y'\right)\)

Tracked position


Probability density function of tracked positions


Kernel function in \(f'\)

\(\mathbf {H}\)

Bandwidth matrix in \(f'\)


Position in cluster m


Cluster index


Number of clusters


Number of positions in cluster m

\(\mathbf {U}^{(m)}\)

Positions matrix

\(\mathbf {C}^{(m)}\)

Covariance matrix of clustered positions

\(\lambda _{1}^{(m)}\), \(\lambda _{2}^{(m)}\)

Eigenvalues of the covariance matrix \(\mathbf {C}^{(m)}\)


Correlation coefficient

\(\sigma _{{x'}^{(m)}}\)

Standard deviation of clustered \({x'}^{(m)}\) position

\(\sigma _{{y'}^{(m)}}\)

Standard deviation of clustered \({y'}^{(m)}\) position


Ellipse confidence level

\(\mathbf {V}^{(m)}\)

Eigenvectors matrix

\(\mathbf {D}^{(m)}\)

Diagonal matrix

\(\theta ^{(m)}\)

Rotation angel of the ellipse


Start of hotspot


End of hotspot


Track with start hotspot S and end hotspot E


Number of positions in track T


Center of hotspot (cluster) m


Hotspot threshold

\(\mathbf {x}_t\)

Feature vector at time instant t

\(\mathbf {x}_{1:T}\)

Sequence of observations

\(\mathbf {y}_{1:T}\)

Sequence of states


Number of time steps


Number of persons


Number of states

\(p(\mathbf {x}_{1:T},y)\)

Joint distribution of \(\mathbf {x}_{1:T}\) and y

\(\gamma\), \(\delta\)

Potential functions


Actual potential


Feature function

\(p(\mathbf {y} \mid \mathbf {x})\)

Conditional probability of \(\mathbf {y}\) given \(\mathbf {x}\)


Normalization function


Pattern length


Word in a document (presence status)


Document of words (presence status sequence)


Topic (activity pattern)


Number of topics


Number of constructed words


Number of documents


Vocabulary of words


Matrix of day-specific mixture weights


Matrix of word-specific mixture weights


Per-document activity pattern distributions hyperparameter


Per-activity word distribution hyperparameter


Mean absolute error


Estimated presence duration for hour r


Actual presence duration for hour r


Number of hours


Relative absolute error


Training corpus model


Length of the document m



This research has been financed by the Belgian National Fund for Scientific Research (FWO Flanders) and Ghent University through the FWO project G.0.398.11.N.10. The evaluation was performed in the context of the projects “LittleSister” and the European AAL project “SONOPA,” financed by the agency for Innovation by Science and Technology (IWT), imec and the EU Ambient Assisted Living programme.


  1. Aztiria A (2010) Learning frequent behaviours of the users in intelligent environments. J Ambient Intell Smart Environ 2(4):435–436. doi: 10.3233/AIS-2010-0084 Google Scholar
  2. Baggenstoss PM (2001) A modified baum-welch algorithm for hidden markov models with multiple observation spaces. IEEE Trans Speech Audio Process 9(4):411–416. doi: 10.1109/89.917686 CrossRefGoogle Scholar
  3. Bickford M (2005) Stress in the workplace: a general overview of the causes, the effects, and the solutions. Canadian Mental Health Association Newfoundland and Labrador Division, pp 1–3Google Scholar
  4. Blei DM, Ng AY, Jordan MI (2003) Latent dirichlet allocation. J Mach Learn Res 3:993–1022zbMATHGoogle Scholar
  5. Bo NB, Deboeverie F, Eldib M, Guan J, Xie X, Niño J, Van Haerenborgh D, Slembrouck M, Van de Velde S, Steendam H et al (2014) Human mobility monitoring in very low resolution visual sensor network. Sensors 14(11):20,800–20,824. doi: 10.3390/s141120800
  6. Camilli M, Kleihorst R (2011) Demo: Mouse sensor networks, the smart camera. In: Fifth ACM/IEEE International Conference on Distributed Smart Cameras, pp 1–3. doi: 10.1109/ICDSC.2011.6042944
  7. Castanedo F, de Ipia DL, Aghajan HK, Kleihorst R (2014) Learning routines over long-term sensor data using topic models. Expert Syst 31(4):365–377. doi: 10.1111/exsy.12033 CrossRefGoogle Scholar
  8. Chen CW, Aghajan H (2011) Multiview social behavior analysis in work environments. In: Fifth ACM/IEEE International Conference on Distributed Smart Cameras, pp 1–6. doi: 10.1109/ICDSC.2011.6042910
  9. Chen CW, Aztiria A, Aghajan H (2011a) Learning human behaviour patterns in work environments. In: CVPR 2011 WORKSHOPS, pp 47–52. doi: 10.1109/CVPRW.2011.5981696
  10. Chen CW, Aztiria A, Allouch SB, Aghajan H (2011b) Understanding the influence of social interactions on individuals behavior pattern in a work environment. In: International Workshop on Human Behavior Understanding, Springer, pp 146–157. doi: 10.1007/978-3-642-25446-8_16
  11. Chen CW, Ugarte RC, Wu C, Aghajan H (2011) Discovering social interactions in real work environments. Face Gesture 2011:933–938. doi: 10.1109/FG.2011.5771376 Google Scholar
  12. Cheng CC, Lee D (2014) Smart sensors enable smart air conditioning control. Sensors 14(6):11,179–11,203. doi: 10.3390/s140611179
  13. Cinaz B, Arnrich B, Marca R, Tröster G (2013) Monitoring of mental workload levels during an everyday life office-work scenario. Person Ubiquitous Comput 17(2):229–239. doi: 10.1007/s00779-011-0466-1 CrossRefGoogle Scholar
  14. Cosemans B, Cosmar M, Gründler R, Flemming D, Van den Broek K (2014) Calculating the cost of work-related stress and psychosocial risks. In: Tech. rep., European Agency for Safety and Health at Work, Luxembourg. doi: 10.2802/20493
  15. Docobo (2013) Sonopa:social networks for older adults to promote an active life. (Online). Accessed 12 May 2016
  16. Duong T, Hazelton ML (2005) Cross-validation bandwidth matrices for multivariate kernel density estimation. Scand J Stat 32(3):485–506. doi: 10.1111/j.1467-9469.2005.00445.x MathSciNetCrossRefzbMATHGoogle Scholar
  17. Eijckelhof BH, Huysmans MA, Blatter BM, Leider PC, Johnson PW, van Dien JH, Dennerlein JT, van der Beek AJ (2014) Office workers’ computer use patterns are associated with workplace stressors. Appl Ergon 45(6):1660–1667. doi: 10.1016/j.apergo.2014.05.013 CrossRefGoogle Scholar
  18. Eldib M, Bo NB, Deboeverie F, Nino J, Guan J, Van de Velde S, Steendam H, Aghajan H, Philips W (2014a) A low resolution multi-camera system for person tracking. In: IEEE International Conference on Image Processing (ICIP), IEEE, pp 378–382. doi: 10.1109/ICIP.2014.7025075
  19. Eldib M, Bo NB, Deboeverie F, Xie X, Philips W, Aghajan H (2014b) Behavior analysis for aging-in-place using similarity heatmaps. In: Proceedings of the International Conference on Distributed Smart Cameras, ACM, ICDSC ’14, vol 6, pp 1–34. doi: 10.1145/2659021.2659038
  20. Eldib M, Deboeverie F, Haerenborgh DV, Philips W, Aghajan H (2015a) Detection of visitors in elderly care using a low-resolution visual sensor network. In: Proceedings of the 9th International Conference on Distributed Smart Cameras, ACM, pp 56–61. doi: 10.1145/2789116.2789137
  21. Eldib M, Deboeverie F, Philips W, Aghajan H (2015b) Sleep analysis for elderly care using a low-resolution visual sensor network. In: Human Behavior Understanding, Springer, pp 26–38. doi: 10.1007/978-3-319-24195-1_3
  22. Eldib M, Deboeverie F, Philips W, Aghajan H (2016a) Behavior analysis for elderly care using a network of low-resolution visual sensors. J Electron Imaging 25(4):041,003–041,003. doi: 10.1117/1.JEI.25.4.041003
  23. Eldib M, Deboeverie F, Philips W, Aghajan H (2016b) Towards more efficient use of office space. In: Proceedings of the 10th International Conference on Distributed Smart Camera, ACM, pp 37–43. doi: 10.1145/2967413.2967424
  24. Eldib M, Zhang T, Deboeverie F, Philips W, Aghajan H (2016c) A data fusion approach for identifying lifestyle patterns in elderly care. In: Active and Assisted Living: Technologies and Applications, Healthcare Technologies, Institution of Engineering and Technology, pp 81–102. doi: 10.1049/PBHE006E_ch5
  25. EU-OSHA (2013a) Campaign guide managing stress and psychosocial risks at work. (Online). Accessed 12 May 2016
  26. EU-OSHA (2013b) European opinion poll on occupational safety and health. In: Tech. rep. doi: 10.2802/55505
  27. Farrahi K, Gatica-Perez D (2011) Discovering routines from large-scale human locations using probabilistic topic models. ACM Trans Intell Syst Technol 2(1):3:1–3:27. doi: 10.1145/1889681.1889684
  28. Griffiths TL, Steyvers M (2004) Finding scientific topics. Proc Natl Acad Sci 101(suppl 1):5228–5235. doi: 10.1073/pnas.0307752101 CrossRefGoogle Scholar
  29. Hamid R, Maddi S, Johnson A, Bobick A, Essa I, Isbell C (2009) A novel sequence representation for unsupervised analysis of human activities. Artif Intell 173(14):1221–1244. doi: 10.1016/j.artint.2009.05.002 MathSciNetCrossRefGoogle Scholar
  30. Healey JA, Picard RW (2005) Detecting stress during real-world driving tasks using physiological sensors. IEEE Trans Intell Transport Syst 6(2):156–166. doi: 10.1109/TITS.2005.848368 CrossRefGoogle Scholar
  31. Huynh T, Fritz M, Schiele B (2008) Discovery of activity patterns using topic models. In: Proceedings of the 10th International Conference on Ubiquitous Computing, ACM, UbiComp ’08, pp 10–19. doi: 10.1145/1409635.1409638
  32. iMinds (2013) Little sister: low-cost monitoring for care and retail. (Online). Accessed 12 May 2016
  33. Jaramillo P, Amft O (2013) Improving energy efficiency through activity-aware control of office appliances using proximity sensing-a real-life study. In: IEEE International Conference on Pervasive Computing and Communications Workshops (PERCOM Workshops), IEEE, pp 664–669. doi: 10.1109/PerComW.2013.6529576
  34. Kim E, Helal S, Cook D (2010) Human activity recognition and pattern discovery. IEEE Pervasive Comput 9(1):48–53. doi: 10.1109/MPRV.2010.7 CrossRefGoogle Scholar
  35. Lafferty J, McCallum A, Pereira F et al (2001) Conditional random fields: probabilistic models for segmenting and labeling sequence data. In: Proceedings of the eighteenth international conference on machine learning, ICML. Morgan Kaufmann Publishers Inc., vol 1, pp 282–289Google Scholar
  36. Lancaster HO, Seneta E (1969) Chi-square distribution. Wiley Online Library. doi: 10.1002/0470011815.b2a15018
  37. Liao W, Zhang W, Zhu Z, Ji Q (2005) A real-time human stress monitoring system using dynamic bayesian network. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’05)—Workshops, pp 70–70. doi: 10.1109/CVPR.2005.394
  38. Liu DC, Nocedal J (1989) On the limited memory bfgs method for large scale optimization. Math Prog 45(1):503–528. doi: 10.1007/BF01589116 MathSciNetCrossRefzbMATHGoogle Scholar
  39. Milczarek M, Rial-González E, Schneider E (2009) OSH [Occupational safety and health] in figures: stress at work-facts and figures. Office for Official Publications of the European CommunitiesGoogle Scholar
  40. Milenkovic M, Amft O (2013) An opportunistic activity-sensing approach to save energy in office buildings. In: Proceedings of the fourth international conference on Future energy systems, ACM, pp 247–258. doi: 10.1145/2487166.2487194
  41. Mrazovac B, Bjelica MZ, Teslic N, Papp I (2011) Towards ubiquitous smart outlets for safety and energetic efficiency of home electric appliances. In: 2011 IEEE International Conference on Consumer Electronics—Berlin (ICCE-Berlin), pp 322–326. doi: 10.1109/ICCE-Berlin.2011.6031795
  42. Okada Y, Yoto TY, Suzuki T, Sakuragawa S, Sugiura T (2013) Wearable ecg recorder with acceleration sensors for monitoring daily stress: Office work simulation study. In: 35th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), pp 4718–4721. doi: 10.1109/EMBC.2013.6610601
  43. Oliver N, Horvitz E (2005) A comparison of hmms and dynamic bayesian networks for recognizing office activities. In: Proceedings of the 10th International Conference on User Modeling, UM’05, pp 199–209. doi: 10.1007/11527886_26
  44. Oliver N, Horvitz E, Garg A (2002) Layered representations for human activity recognition. In: Multimodal Interfaces, 2002. Proceedings. Fourth IEEE International Conference on, IEEE, pp 3–8. doi: 10.1109/ICMI.2002.1166960
  45. Oliver N, Garg A, Horvitz E (2004) Layered representations for learning and inferring office activity from multiple sensory channels. Comput Vis Image Underst 96(2):163–180. doi: 10.1016/j.cviu.2004.02.004 CrossRefGoogle Scholar
  46. Rabiner L (1989) A tutorial on hidden markov models and selected applications in speech recognition. Proc IEEE 77(2):257–286. doi: 10.1109/5.18626 CrossRefGoogle Scholar
  47. Rish I (2001) An empirical study of the naive bayes classifier. In: IJCAI 2001 workshop on empirical methods in artificial intelligence. IBM New York, vol 3, pp 41–46Google Scholar
  48. Salamone F, Belussi L, Danza L, Ghellere M, Meroni I (2016) An open source smart lamp for the optimization of plant systems and thermal comfort of offices. Sensors 16(3):338. doi: 10.3390/s16030338 CrossRefGoogle Scholar
  49. Sheather SJ, Jones MC (1991) A reliable data-based bandwidth selection method for kernel density estimation. J R Stat Soc Ser B (Methodol). doi: 10.2307/2345597
  50. Si Z, Pei M, Yao B, Zhu SC (2011) Unsupervised learning of event and-or grammar and semantics from video. In: International Conference on Computer Vision, IEEE, pp 41–48. doi: 10.1109/ICCV.2011.6126223
  51. Silverman BW (1986) Density estimation for statistics and data analysis, vol 26. CRC Press. doi: 10.1007/978-1-4899-3324-9
  52. Simonoff J (1996) Smoothing methods in statistics. Springer, BerlinGoogle Scholar
  53. Sutton C, McCallum A (2012) An introduction to conditional random fields. Found Trends Mach Learn 4(4):267373. doi: 10.1561/2200000013 CrossRefGoogle Scholar
  54. Tao S, Kudo M, Nonaka H, Toyama J (2011) Person authentication and activities analysis in an office environment using a sensor network. In: International Joint Conference on Ambient Intelligence, Springer, pp 119–127. doi: 10.1007/978-3-642-31479-7_19
  55. Teixeira T, Dublon G, Savvides A (2010) A survey of human-sensing: methods for detecting presence, count, location, track, and identity. ENALAB technical reportGoogle Scholar
  56. Varadarajan J, Emonet R, Odobez JM (2013) A sequential topic model for mining recurrent activities from long term video logs. Int J Comp Vis 103(1):100–126. doi: 10.1007/s11263-012-0596-6 MathSciNetCrossRefzbMATHGoogle Scholar
  57. Wojek C, Nickel K, Stiefelhagen R (2006) Activity recognition and room-level tracking in an office environment. In: IEEE International Conference on Multisensor Fusion and Integration for Intelligent Systems, pp 25–30. doi: 10.1109/MFI.2006.265608
  58. Xie X, Deboeverie F, Eldib M, Philips W, Aghajan H (2014) Phd forum: Analyzing behaviors patterns of the elderly from low-precision trajectories. In: Proceedings of the International Conference on Distributed Smart Cameras, ACM, ICDSC ’14, vol 2, pp 1–47. doi: 10.1145/2659021.2675057
  59. Ziefle M, Rocker C, Holzinger A (2011) Medical technology in smart homes: exploring the user’s perspective on privacy, intimacy and trust. In: IEEE 35th Annual Computer Software and Applications Conference Workshops, pp 410–415. doi: 10.1109/COMPSACW.2011.75
  60. Zimmermann P, Guttormsen S, Danuser B, Gomez P (2003) Affective computinga rationale for measuring mood with mouse and keyboard. Int J Occup Saf Ergon 9(4):539–551. doi: 10.1080/10803548.2003.11076589 CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2017

Authors and Affiliations

  • Mohamed Eldib
    • 1
  • Francis Deboeverie
    • 1
  • Wilfried Philips
    • 1
  • Hamid Aghajan
    • 1
    • 2
  1. 1.Image Processing and InterpretationTELIN, Ghent University/imecGentBelgium
  2. 2.Ambient Intelligence Research LabStanfordUSA

Personalised recommendations