Abstract
Aging in place has garnered a lot of interest in the past decade among researchers and politicians. It presents itself as a humane and cost-efficient solution to the worsening problem related to the financing and staffing of our fragile healthcare ecosystem. In that regard, smart environments could help monitor the daily activities of the person and provide information regarding the health status and any potential problem necessitating immediate assistance. To do so, many teams, including ours, have been working on human activity recognition from distributed sensors. Nevertheless, it is still very challenging due to the difficulty of amassing enough data to learn activity models that generalizes well across different residents and different smart environments. Moreover, whenever one wants to add a new activity to the set of recognizable activities, it requires to gather additional data with label. The whole process is prohibitively costly and time consuming. Therefore, in this paper, our team proposed to explore web mining in order to build activity model in an unsupervised fashion. More specifically, using the results of two popular image search engines with automated queries and with the help of object detection/classification models, we learned the set of key objects involved in the realization of activities of daily life. A total of 108 configurations were tested. The experiments showed that the key objects related to activities can be easily extracted with a good accuracy.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Notes
References
Aggarwal J, Ryoo M (2011) Human activity analysis. ACM Comput Surv 43(3):1–43
Alemdar H, Durmaz Incel O, Ertan H, Ersoy C (2013) ARAS human activity datasets in multiple homes with multiple residents. In: Proceedings of the ICTs for improving patients rehabilitation research techniques, IEEE, pp 232–235, https://doi.org/10.4108/icst.pervasivehealth.2013.252120
Ann OC, Theng LB (2014) Human activity recognition: a review. In: Proceedings—4th IEEE International Conference on control system, computing and engineering, ICCSCE 2014, pp 389–393, https://doi.org/10.1109/ICCSCE.2014.7072750
Cao J, Guo F, Lai X, Zhou Q, Dai J (2020) A Tree-structure convolutional neural network for temporal features exaction on sensor-based multi-resident activity recognition. In: Communications in computer and information science, vol 1265 CCIS. Springer Singapore, pp 513–525, https://doi.org/10.1007/978-981-15-7670-6_43
Colledanchise M, Ögren P (2014) How behavior trees modularize robustness and safety in hybrid systems. In: 2014 IEEE/RSJ International Conference on intelligent robots and systems, pp 1482–1488, https://doi.org/10.1109/IROS.2014.6942752
Contributors to Wikimedia projects (2020) Riemann sum. https://en.wikipedia.org/w/index.php?title=Riemann_sum&oldid=969091575. Accessed 1 July 2020
Cook DJ, Crandall AS, Thomas BL, Krishnan NC (2013) CASAS: a smart home in a box. Computer 46(7):62–69. https://doi.org/10.1109/MC.2012.328
Deng J, Dong W, Socher R, Li LJ, Kai Li, Li F-F (2009) ImageNet: a large-scale hierarchical image database. In: 2009 IEEE Conference on computer vision and pattern recognition, Ieee, IEEE, pp 248–255, https://doi.org/10.1109/CVPR.2009.5206848, https://ieeexplore.ieee.org/document/5206848/
DomuSafe (2019) aront. https://sites.unica.it/domusafe/aront. Accessed 18 Jan 2020
Ducatel K, Bogdanowicz M, Scapolo F, Leijten J, Burgelman JC (2003) Ambient intelligence: from vision to reality. Tech. rep, IST Advisory Group Draft Report, European Commission
DuckDuckGo (2021) DuckDuckGo Traffic. https://duckduckgo.com/traffic. Accessed 1 Jan 2021
Etimberg, Benmccann, Tannerlinsley, Simonbrunel (2020) Chart.js. https://github.com/chartjs/Chart.js. Accessed 1 July 2020
Fishkin R (2019) Less than half of google searches now result in a click. https://sparktoro.com/blog/less-than-half-of-google-searches-now-result-in-a-click. Accessed 15 Nov 2019
Francillette Y, Boucher E, Bouzouane A, Gaboury S (2017) The virtual environment for rapid prototyping of the intelligent environment. Sensors 17(11):2562
Francillette Y, Boucher E, Bier N, Lussier M, Bouchard K, Belchior P, Gaboury S (2020) Modeling the behavior of persons with mild cognitive impairment or Alzheimer's for intelligent environment simulation. User Model User-Adapt Interact 15:10. https://doi.org/10.1007/s11257-020-09266-4 (0123456789)
Hu C, Chen Y, Hu L, Peng X (2018) A novel random forests based class incremental learning method for activity recognition. Pattern Recognit 78:277–290
Hui J (2019) Real-time object detection with YOLO, YOLOv2 and now YOLOv3. Medium https://medium.com/@jonathan_hui/real-time-object-detection-with-yolo-yolov2-28b1b93e2088. Accessed 15 Jan 2019
Ihianle IK, Naeem U, Tawil AR, Azam MA (2016) Recognizing activities of daily living from patterns and extraction of web knowledge. In: Proceedings of the 2016 ACM International Joint Conference on pervasive and ubiquitous computing adjunct—UbiComp ’16, ACM, pp 1255–1262, https://doi.org/10.1145/2968219.2968440, http://dl.acm.org/citation.cfm?doid=2968219.2968440. Accessed 1 Jan 2019
Järvelin K, Kekäläinen J (2017) IR evaluation methods for retrieving highly relevant documents. In: ACM SIGIR Forum, ACM New York, NY, USA, vol 51, pp 243–250
Lara ÓD, Labrador MA (2013) A survey on human activity recognition using wearable sensors. IEEE Commun Surv Tutor 15(3):1192–1209. https://doi.org/10.1109/SURV.2012.110112.00192
Liu C, Zoph B, Neumann M, Shlens J, Hua W, Li LJ, Fei-Fei L, Yuille A, Huang J, Murphy K (2017) Progressive neural architecture search. arXiv: 1712.00559
Lu L, Qing-ling C, Yi-Ju Z (2017) Activity recognition in smart homes. Multimed Tools Appl 76(22):24203–24220
Manning CD, Raghavan P, Schutze H (2008) Chap 8: evaluation in information retrieval. In: Introduction to information retrieval. Cambridge University Press, Cambridge, pp 151–175, https://doi.org/10.1017/CBO9780511809071, http://ebooks.cambridge.org/ref/id/CBO9780511809071. Accessed 1 Jan 2019
Ng A (2020) Non-max suppression—object detection. https://www.coursera.org/lecture/convolutional-neural-networks/non-max-suppression-dvrjH. Accessed 26 July 2020
Palmes P, Pung HK, Gu T, Xue W, Chen S (2010) Object relevance weight pattern mining for activity recognition and segmentation. Pervasive and Mob Comput 6(1):43–57. https://doi.org/10.1016/j.pmcj.2009.10.004
Perkowitz M, Philipose M, Fishkin K, Patterson DJ (2004) Mining models of human activities from the web. In: Proceedings of the 13th Conference on World Wide Web—WWW ’04, ACM, p 573, https://doi.org/10.1145/988672.988750, http://portal.acm.org/citation.cfm?doid=988672.988750. Accessed 1 Feb 2019
Redmon J, Farhadi A (2017) YOLO9000: better, faster, stronger. In: 2017 IEEE Conference on computer vision and pattern recognition (CVPR), IEEE, pp 6517–6525, https://doi.org/10.1109/CVPR.2017.690, http://ieeexplore.ieee.org/document/8100173/
Redmon J, Farhadi A (2018) YOLOv3: an incremental improvement. CoRR http://arxiv.org/abs/1804.02767
Riboni D, Murtas M (2017) Web mining & computer vision: new partners for object-based activity recognition. In: Proceedings—2017 IEEE 26th International Conference on enabling technologies: infrastructure for collaborative enterprises, WETICE 2017, pp 158–163, https://doi.org/10.1109/WETICE.2017.38, https://ieeexplore.ieee.org/document/8003808/. Accessed 1 Nov 2018
Riboni D, Murtas M (2019) Sensor-based activity recognition: One picture is worth a thousand words. Future Gener Comput Syst, https://doi.org/10.1016/j.future.2019.07.020, https://linkinghub.elsevier.com/retrieve/pii/S0167739X19303863. Accessed 1 Jan 2020
Russell S, Norvig P (2009) Artificial intelligence: a modern approach, 3rd edn. Prentice Hall Press
Sandler M, Howard A, Zhu M, Zhmoginov A, Chen LC (2018) MobileNetV2: inverted residuals and linear bottlenecks. In: Proceedings of the IEEE Computer Society Conference on computer vision and pattern recognition, pp 4510–4520, https://doi.org/10.1109/CVPR.2018.00474
Sung-Hyun Y, Thapa K, Kabir MH, Hee-Chan L (2018) Log-Viterbi algorithm applied on second-order hidden Markov model for human activity recognition. Int J Distrib Sens Netw 14(4):155014771877254. https://doi.org/10.1177/1550147718772541
Szegedy C, Ioffe S, Vanhoucke V, Alemi AA (2017) Inception-v4, inception-resnet and the impact of residual connections on learning. In: Proceedings of the thirty-first AAAI conference on artificial intelligence, San Francisco, California, USA. AAAI Press, pp 4278–4284. https://doi.org/10.5555/3298023.3298188
Tapia EM, Choudhury T, Philipose M (2006) Building reliable activity models using hierarchical shrinkage and mined ontology. In: Fishkin KP, Schiele B, Nixon P, Quigley A (eds) Lecture notes in computer science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol 3968 LNCS. Springer Berlin Heidelberg, Berlin, pp 17–32. https://doi.org/10.1007/11748625_2
Tsang SH (2019) Review: YOLOv2 & YOLO9000 You Only Look Once (Object Detection). Medium https://towardsdatascience.com/review-yolov2-yolo9000-you-only-look-once-object-detection-7883d2b02a65. Accessed 1 Apr 2019
United Nations Department of Economic and Social Affairs | Popular Division (2017) World Population Ageing 2017. Tech. Rep. ST/ESA/SER.A/408, United Nations
Wimo PA, Guerchet DM, Ali MGC, Wu DYT, Prina DM, Alzheimer’s Disease International (2015) World Alzheimer Report 2015. Tech. rep., Alzheimer’s Disease International
Wyatt D, Philipose M, Choudhury T (2005) Unsupervised activity recognition using automatically mined common sense. In: Proceedings of the 20th National Conference on artificial intelligence, vol 20, pp 21–27, https://doi.org/10.1093/bmb/ldx041
Xiao Q, Song R (2018) Action recognition based on hierarchical dynamic Bayesian network. Multimed Tools Appl 77(6):6955–6968. https://doi.org/10.1145/956750.956785
Yi L, Liu B, Li X (2003) Eliminating noisy information in Web pages for data mining. Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining pp 296–305, https://doi.org/10.1145/956750.956785
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
The authors declare that they have no conflict of interest.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Cousyn, C., Bouchard, K. & Gaboury, S. Web-based objects detection to discover key objects in human activities. J Ambient Intell Human Comput 14, 3041–3056 (2023). https://doi.org/10.1007/s12652-021-03433-0
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s12652-021-03433-0