Abstract
With the rapid development and wide proliferation of sensor devices and the Internet of Things (IoT), machine learning algorithms processing and analysing one or more modalities of sensory signals have become an active research field given its numerous applications, particularly in the domestic intelligent environment (DIE). In the past decades, the research on sensing and interactive devices of DIE and deep learning (DL) based methods have become strikingly popular. Several missions, such as the pro- cessing and analysis of sensing signals related to domestic instruments and the control of certain devices to act upon the results, comprise the main working targets in DIE. The goal of this review is to provide a brief overview of the aforementioned sensors, their related DL algorithms and their applications. To comprehend the ideas behind the use of various devices found in domestic intelligent instruments, we first summarize the available information. Then, to quantify and adapt the residents’ knowledge of the household environment, we review data-driven learning techniques based on the aforementioned sensor-based devices and introduce robotic applications that provide helpers and action outputs in the environment. Finally, we investigate the commonly utilized datasets relevant to DIE and human activ- ity recognition (HAR) and explore the challenges and prospects of their applications in the DIE field.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Ab Wahab R (2007) Smart home security sys- tem(design of magnetic switch sensor and phone dialer). universiti teknologi petronas
Alshammari T, Alshammari N, Sedky M et al (2018) Simadl: simulated activities of daily liv- ing dataset. Data 3(2):11
Alsheikh MA, Selim A, Niyato D, et al (2016) Deep activity recognition models with triaxial accelerometers. In: Workshops at the Thirtieth AAAI Conference on Artificial Intelligence
Amarasinghe R, Dao DV, Toriyama T et al (2006) Simulation, fabrication and characterization of a three-axis piezoresistive accelerometer. Smart Mater Struct 15(6):1691
Anguita D, Ghio A, Oneto L, et al (2013) A public domain dataset for human activity recognition using smartphones. In: Esann, p 3
Arriany AA, Musbah MS (2016) Applying voice recognition technology for smart home net- works. In: 2016 International Conference on Engineering & MIS (ICEMIS), IEEE, pp 1–6
Atzori L, Iera A, Morabito G (2010) The inter- net of things: a survey. Comput Netw 54(15):2787–2805
Bakar U, Ghayvat H, Hasanm S et al (2016) Activity and anomaly detection in smart home: a survey. In: Mukhopadhyay SC (ed) Next generation sensors and systems. Springer, Cham, pp 191–220
Balasubramaniam S, Kangasharju J (2013) Real- izing the internet of nano things: challenges, solutions, and applications. Computer 46(2):62–68. https://doi.org/10.1109/MC.2012.389
Bangali J, Shaligram A (2013) Design and imple- mentation of security systems for smart home based on gsm technology. Int J Smart Home 7(6):201–208
Banos O, Villalonga C, Garcia R et al (2015) Design, implementation and validation of a novel open framework for agile development of mobile health applications. Biomed Eng-Neering Online 14(2):1–20
Berrezueta-Guzman J, Pau I, Martin-Ruiz ML et al (2020) Smart-home environment to sup- port homework activities for children. IEEE Access 8:160–267
Boric-Lubecke O, Lubecke VM, Mostafanezhad I et al (2009) Doppler radar architectures and signal processing for heart rate extraction. Mikrotalasna Rev 15(2):12–17
Bouchard K, Bilodeau JS, Fortin-Simard D, et al (2014) Human activity recognition in smart homes based on passive rfid localization. In: Proceedings of the 7th international conference on PErvasive technologies related to assistive environments, pp 1–8
Chavarriaga R, Sagha H, Calatroni A et al (2013) The opportunity challenge: A bench- mark database for on-body sensor-based activ- ity recognition. Patt Recogn Lett 34(15):2033–2042
Chen K, Yao L, Zhang D et al (2020) A semisupervised recurrent convolutional atten- tion model for human activity recognition. IEEE Trans Neural Netw Learn Syst 31(5):1747–1756. https://doi.org/10.1109/TNNLS.2019.2927224
Chernbumroong S, Cang S, Atkins A et al (2013) Elderly activities recognition and classification for applications in assisted living. Expert Sys-Tems Appl 40:1662–1674. https://doi.org/10.1016/j.eswa.2012.09.004
Chernbumroong S, Cang S, Yu H (2014a) Genetic algorithm-based classifiers fusion for multisen- sor activity recognition of elderly people. IEEE J Biomed Health Inform 19(1):282–289
Chernbumroong S, Cang S, Yu H (2014b) A prac- tical multi-sensor activity recognition system for home-based care. Decis Support Syst 66:61–70
Chikhaoui B, Gouineau F (2017) Towards auto- matic feature extraction for activity recogni- tion from wearable sensors: a deep learning approach. In: 2017 IEEE International Confer- ence on Data Mining Workshops (ICDMW), IEEE, pp 693–702
Cho H, Yoon SM (2018) Divide and conquer-based 1d cnn human activity recognition using test data sharpening. Sensors 18(4):1055
Chong G, Zhihao L, Yifeng Y (2011) The research and implement of smart home system based on internet of things. In: 2011 International Con- ference on Electronics, Communications and Control (ICECC), IEEE, pp 2944–2947
Cleland I, Donnelly MP, Nugent CD, et al (2018) Collection of a diverse, realistic and annotated dataset for wearable activity recognition. In: 2018 IEEE International Conference on Per- vasive Computing and Communications Work- shops (PerCom Workshops), IEEE, pp 555–560
Cook DJ (2010) Learning setting-generalized activity models for smart spaces. IEEE Intelli-Gent Syst 2010(99):1
Cook DJ, Crandall AS, Thomas BL et al (2012) Casas: a smart home in a box. Computer 46(7):62–69
Cook aSEMDiane, (2009) Assessing the quality of activities in a smart environment. Methods Inf Med 48(05):480–485
Dar JA, Srivastava KK, Lone SA (2022a) Spectral features and optimal hierarchical attention networks for pulmonary abnormal- ity detection from the respiratory sound sig- nals. Biomed Signal Process Control 78(103):905. https://doi.org/10.1016/j.bspc.2022.103905
Dar JA, Srivastava KK, Sajaad, (2022b) Design and development of hybrid optimization enabled deep learning model for covid-19 detec- tion with comparative analysis with dcnn, biat-gru, xgboost. Comput Biol Med 150:106–123
Das R, Munkhdalai T, Yuan X, et al (2018) Build- ing dynamic knowledge graphs from text using machine reading comprehension. arXiv preprint arXiv:181005682
Dawadi PN, Cook DJ, Schmitter-Edgecombe M (2013) Automated cognitive health assessment using smart home monitoring of complex tasks. IEEE Trans Syst Man Cyber-Netics: Syst 43(6):1302–1313
Dawadi P, Cook D, Parsey C, et al (2011) An approach to cognitive assessment in smart home. In: Proceedings of the 2011 workshop on Data mining for medicine and healthcare, pp 56–59
Devlin J, Chang MW, Lee K, et al (2018) Bert: Pre-training of deep bidirectional transform- ers for language understanding. arXiv preprint arXiv:181004805
Do HM, Pham M, Sheng W et al (2018) Rish: a robot-integrated smart home for elderly care. Robot Auton Syst 101:74–92
Du Y, Lim Y, Tan Y (2019) Rf-arp: Rfid-based activity recognition and prediction in smart home. In: 2019 IEEE 25th International Con- ference on Parallel and Distributed Systems (ICPADS), IEEE, pp 618–624
El-Basioni BMM, El-Kader S, Abdelmonim M (2013) Smart home design using wireless sensor network and biometric technologies. Inform Technol 2(1):2
Espinilla M, De-La-Hoz-Franco E, Medina Quero J, et al (2021) Uja human activity recogni- tion multi-occupancy dataset. In: Proceedings of the 54th Hawaii International Conference on System Sciences, p 1938
Feki MA, Kawsar F, Boussard M et al (2013) The internet of things: The next technological revo- lution. Computer 46(2):24–25. https://doi.org/10.1109/MC.2013.63
Fleury A, Vacher M, Noury N (2009) Svm- based multimodal classification of activities of daily living in health smart homes: sen- sors, algorithms, and first experimental results. IEEE Trans Inf Technol Biomed 14(2):274–283
Fortin-Simard D, Bilodeau JS, Bouchard K et al (2015) Exploiting passive rfid technology for activity recognition in smart homes. IEEE Intel-Ligent Syst 30(4):7–15
Freitas DJ, Marcondes TB, Nakamura LH, et al (2015) A health smart home system to report incidents for disabled people. In: 2015 Interna- tional Conference on Distributed Computing in Sensor Systems, IEEE, pp 210–211
Galinina O, Mikhaylov K, Andreev S et al (2015) Smart home gateway system over bluetooth low energy with wireless energy transfer capability. EURASIP J Wirel Commun Netw 1:1–18
Gao W, Zhang L, Teng Q et al (2021) Danhar: dual attention network for multimodal human activity recognition using wearable sensors. Appl Soft Comput 111(107):728. https://doi.org/10.1016/j.asoc.2021.107728
Gochoo M, Tan TH, Huang SC, et al (2017) Dcnn-based elderly activity recognition using binary sensors. In: 2017 International Confer- ence on Electrical and Computing Technologies and Applications (ICECTA), IEEE, pp 1–5
Gong X, Wu WJ, Liao WH (2020) A low-noise three-axis piezoelectric mems accelerometer for condition monitoring. In: Sensors and Smart Structures Technologies for Civil, Mechani- cal, and Aerospace Systems 2020, International Society for Optics and Photonics, p 113790U
Guan Y, Ploetz T (2017) Ensembles of deep lstm learners for activity recognition using wearables. Proceed ACM Interact Mob Wearable Ubiquitous Technol. https://doi.org/10.1145/3090076
Gunawan TS, Yaldi IRH, Kartiwi M et al (2017) Prototype design of smart home system using internet of things. IJEECS 7(1):107–115
Gurunath R, Agarwal M, Nandi A, et al (2018) An overview: Security issue in iot network. In: 2018 2nd International Conference on I-SMAC (IoT in Social, Mobile, Analytics and Cloud) (I-SMAC)I-SMAC (IoT in Social, Mobile, Ana- lytics and Cloud) (I-SMAC), 2018 2nd Interna- tional Conference on, pp 104–107, https://doi.org/10.1109/I-SMAC.2018.8653728
Guth J, Breitenbu¨cher U, Falkenthal M, et al (2018) A detailed analysis of iot platform archi- tectures : Concepts, imilarities, and differences. In: Internet of Everything. Internet of Things, Springer Singapore, p 81–101, https://doi.org/10.1007/978-981-10-5861-5 4
H¨oller J, Tsiatsis V, Mulligan C, et al (2014) From machine-to-machine to the internet of things. Elsevier, 10.1016/ C2012-0-03263-2
Ha S, Choi S (2016) Convolutional neural net- works for human activity recognition using mul- tiple accelerometer and gyroscope sensors. In: 2016 International Joint Conference on Neu- ral Networks (IJCNN), pp 381–388, https://doi.org/10.1109/IJCNN.2016.7727224
Hamad RA, Hidalgo AS, Bouguelia MR et al (2019) Efficient activity recognition in smart homes using delayed fuzzy temporal windows on binary sensors. IEEE J Biomed Health Inform 24(2):387–395
Hammerla NY, Halloran S, Pl¨otz T (2016) Deep, convolutional, and recurrent models for human activity recognition using wearables. arXiv preprint arXiv:160408880
Henaff M, Weston J, Szlam A, et al (2016) Tracking the world state with recurrent entity networks. arXiv preprint arXiv:161203969
Hill F, Bordes A, Chopra S, et al (2015) The goldilocks principle: Reading children’s books with explicit memory representations. arXiv preprint arXiv:151102301
Hong S, Lee Y, Park H et al (2015) Stretch- able active matrix temperature sensor array of polyaniline nanofibers for electronic skin. Adv Mater. https://doi.org/10.1002/adma.201504659
Hu Y, Tilke D, Adams T et al (2016) Smart home in a box: usability study for a large scale self- installation of smart home technologies. J Reliab Intell Environ 2(2):93–106
Huang J, Lin S, Wang N et al (2019) Tse-cnn: A two-stage end-to-end cnn for human activ- ity recognition. IEEE J Biomed Health Inform 24(1):292–299
Hussein M, Torki M, Gowayyed M, et al (2013) Human action recognition using a temporal hierarchy of covariance descriptors on 3d joint locations
Jiang L, Liu DY, Yang B (2004) Smart home research. In: Proceedings of 2004 international conference on machine learning and cybernetics (IEEE Cat. No. 04EX826), IEEE, pp 659–663
Jie Y, Pei JY, Jun L, et al (2013) Smart home system based on iot technologies. In: 2013 International conference on computational and information sciences, IEEE, pp 1789–1791
Jokinen K, Wilcock G (2014) Multimodal open- domain conversations with the nao robot. In: Natural Interaction with Robots, Knowbots and Smartphones. Springer, p 213–224
Juds S (1988) Photoelectric sensors and controls: selection and application, vol 63. CRC Press
Jung Y (2017) Hybrid-aware model for senior well- ness service in smart home. Sensors 17(5):1182
Kashimoto Y, Fujiwara M, Fujimoto M, et al (2017) Alpas: Analog-pir-sensor-based activ- ity recognition system in smarthome. In: 2017 IEEE 31st International Conference on Advanced Information Networking and Appli- cations (AINA), IEEE, pp 880–885
Kaushik AR, et al (2006) Characterization of pas- sive infrared sensors for monitoring occupancy pattern. In: 2006 International Conference of the IEEE Engineering in Medicine and Biology Society, IEEE, pp 5257–5260
Kelly SDT, Suryadevara NK, Mukhopadhyay SC (2013) Towards the implementation of iot for environmental condition monitoring in homes. IEEE Sens J 13(10):3846–3853. https://doi.org/10.1109/JSEN.2013.2263379
Kim JK, Kim YB (2018) Supervised domain enablement atten-tion for personalized domain classification. arXiv preprint arXiv:181207546
Krose B, Van Kasteren T, Gibson C et al (2008) Care: Context awareness in residences for elderly. International Conference of the International Society for Gerontechnology. Pisa, Tuscany, Italy, Citeseer, pp 101–105
Kumar A, Irsoy O, Ondruska P, et al (2016) Ask me anything: Dynamic memory networks for natural language processing. In: Interna- tional conference on machine learning, PMLR, pp 1378–1387
Lago P, Lang F, Roncancio C et al (2017) The contextact@ a4h real-life dataset of daily- living activities. In: Brézillon P, Turner R, Penco C (eds) International and inter- disciplinary conference on modeling and using context. Springer, Cham, pp 175–188
Law T, de Leeuw J, Long JH (2020) How move- ments of a non-humanoid robot affect emotional perceptions and trust. International Journal of Social Robotics pp 1–12
Lee SM, Yoon SM, Cho H (2017) Human activ- ity recognition from accelerometer data using convolutional neural network. In: 2017 ieee international conference on big data and smart computing (bigcomp), IEEE, pp 131–134
Li Y, Shi D, Ding B, et al (2014) Unsupervised feature learning for human activity recognition using smartphone sensors. In: Mining intelli- gence and knowledge exploration. Springer, p 99–107
Lin C, Chen M (2017) Design and implementa- tion of a smart home energy saving system with active loading feature identification and power management. In: 2017 ieee 3rd international future energy electronics conference and ecce asia (ifeec 2017-ecce asia), IEEE, pp 739–742
Liu K, Zhang W, Chen W et al (2009) The devel- opment of micro-gyroscope technology. J Micromech Microeng 19(11):113001
Mahmud S, Tonmoy MTH, Bhaumik K, et al (2020) Human activity recognition from wear- able sensor data using self-attention. https://doi.org/10.3233/FAIA200236
Makinwa K (2010) Smart temperature sensors in standard cmos. Proced Eng 5:930–939
Malche MPTimothy (2017) Internet of things (iot) for building smart home system. In: 2017 Inter- national Conference on I-SMAC (IoT in Social, Mobile, Analytics and Cloud)(I-SMAC), IEEE, pp 65–70
Mano L, Faical B, Nakamura LHV et al (2016) Exploiting iot technologies for enhancing health smart homes through patient identification and emotion recognition. Comput Commun. https://doi.org/10.1016/j.comcom.2016.03.010
Micucci D, Mobilio M, Napoletano P (2017) Unimib shar: A dataset for human activity recognition using acceleration data from smart- phones. Appl Sci 7(10):1101
Mikolov T, Sutskever I, Chen K, et al (2013) Dis- tributed representations of words and phrases and their compositionality. In: Advances in neu- ral information processing systems, pp 3111– 3119
Morbiducci U, Scalise L, De Melis M et al (2007) Optical vibrocardiography: a novel tool for the optical monitoring of cardiac activity. Ann Biomed Eng 35(1):45–58
Munir A, Ehsan SK, Raza SM, et al (2019) Face and speech recognition based smart home. In: 2019 International Conference on Engineering and Emerging Technologies (ICEET), IEEE, pp 1–5
Naser A, Lotfi A, Zhong J (2020) Adaptive ther- mal sensor array placement for human segmen- tation and occupancy estimation. IEEE Sens J 21(2):1993–2002
Naser A, Lotfi A, Zhong J (2021) Towards human distance estimation using a thermal sensor array. Neural Computing and Applications pp 1–11
Ng WW, Xu S, Wang T et al (2020) Radial basis function neural network with localized stochastic-sensitive autoencoder for home-based activity recognition. Sensors 20(5):1479
Noguchi H, Mori T, Sato T (2004) Network mid- dleware for flexible integration of sensor pro- cessing in home environment. In: 2004 IEEE International Conference on Systems, Man and Cybernetics (IEEE Cat. No. 04CH37583), IEEE, pp 3845–3851
Noor MHM (2021) Feature learning using convolu- tional denoising autoencoder for activity recog- nition. Neural Computing and Applications pp 1–14
Ordonez FJ, Roggen D (2016) Deep convolutional and lstm recurrent neural networks for mul- timodal wearable activity recognition. Sensors 16(1):115
Pandharipande A, Li S (2013) Light-harvesting wireless sensors for indoor lighting control. Sen-s j, IEEE 13:4599–4606. https://doi.org/10.1109/JSEN.2013.2272073
Park J, Jang K, Yang SB (2018) Deep neural net- works for activity recognition with multi-sensor data in a smart home. In: 2018 IEEE 4th World Forum on Internet of Things (WF-IoT), IEEE, pp 155–160
Passaro V, Cuccovillo A, Vaiani L et al (2017) Gyroscope technology and applications: a review in the industrial perspective. Sensors 17(10):2284
Pennington J, Socher R, Manning CD (2014) Glove: Global vectors for word representation. In: Proceedings of the 2014 conference on empir- ical methods in natural language processing (EMNLP), pp 1532–1543
Peters ME, Neumann M, Iyyer M, et al (2018) Deep contextualized word representa- tions. arXiv preprint arXiv:180205365
Pujolle G (2006) An autonomic-oriented architec- ture for the internet of things. In: IEEE John Vincent Atanasoff 2006 International Sympo- sium on Modern Computing JVA’06. IEEE, pp 163–168, https://doi.org/10.1109/JVA.2006.6
Qiao T, Dong J, Xu D (2018) Exploring human- like attention supervision in visual question answering. In: Thirty-Second AAAI Conference on Artificial Intelligence
Radford A, Narasimhan K, Salimans T, et al (2018) Improving language understanding by generative pre-training
Ranjit SSS, Ibrahim AFT, Salim SI, et al (2009) Door sensors for automatic light switching sys- tem. In: 2009 Third UKSim European Sympo- sium on Computer Modeling and Simulation, IEEE, pp 574–578
Ransing RS, Rajput M (2015) Smart home for elderly care, based on wireless sensor network. In: 2015 International Conference on Nascent Technologies in the Engineering Field (ICNTE), IEEE, pp 1–5
Reed MJ, Robertson C, Addison P (2005) Heart rate variability measurements and the predic- tion of ventricular arrhythmias. QJM 98(2):87–95
Retto J (2017) Sophia, first citizen robot of the world. ResearchGate
Robotics S (2016) Thirteen advanced humanoid robots for sale today. Smashing Robotics, April 16
Ronao CA, Cho SB (2015) Deep convolutional neural networks for human activity recogni- tion with smartphone sensors. In: International Conference on Neural Information Processing, Springer, pp 46–53
Ronao CA, Cho SB (2016) Human activity recog- nition with smartphone sensors using deep learning neural networks. Expert Syst Appl 59:235–244
Saeed A, Ozcelebi T, Lukkien J (2019) Multi- task self-supervised learning for human activity detection. Proc ACM Interact Mob Wear Ubiquitous Technol 3(2):30. https://doi.org/10.1145/3328932
Sanabria AR, Ye J (2020) Unsupervised domain adaptation for activity recognition across het- erogeneous datasets. Pervasive Mob Comput 64(101):147
Seelye AM, Schmitter-Edgecombe M, Cook DJ et al (2013) Naturalistic assessment of every- day activities and prompting technologies in mild cognitive impairment. J Int Neuropsychol Soc 19(4):442–452
Shamsuddin S, Ismail LI, Yussof H, et al (2011) Humanoid robot nao: Review of control and motion exploration. In: 2011 IEEE international conference on Control System, Computing and Engineering, IEEE, pp 511–516
Singh D, Merdivan E, Hanke S, et al (2017a) Convolutional and recurrent neural networks for activity recognition in smart environment. In: Towards integrative machine learning and knowledge extraction. Springer, p 194–205
Singh D, Merdivan E, Psychoula I, et al (2017b) Human activity recognition using recurrent neu- ral networks. In: International cross-domain conference for machine learning and knowledge extraction, Springer, pp 267–274
Singla G, Cook DJ, Schmitter-Edgecombe M (2010) Recognizing independent and joint activ- ities among multiple residents in smart envi- ronments. J Ambient Intell Humaniz Comput 1(1):57–63
Stojkoska B, Trivodaliev KV (2017) A review of internet of things for smart home: challenges and solutions. J Clean Prod 140:1454–1464
Su T, Sun H, Ma C, et al (2019) Hdl: Hierarchi- cal deep learning model based human activity recognition using smartphone sensors. In: 2019 International Joint Conference on Neural Net- works (IJCNN), IEEE, pp 1–8
Su H, Shen X, Xiao Z, et al (2020) Moviechats: Chat like humans in a closed domain. In: Pro- ceedings of the 2020 Conference on Empiri- cal Methods in Natural Language Processing (EMNLP), pp 6605–6619
Suh C, Ko YB (2008) Design and implementa- tion of intelligent home control systems based on active sensor networks. IEEE Trans Consum Electron 54(3):1177–1184
Sukhbaatar S, Szlam A, Weston J, et al (2015) End-to-end memory networks. arXiv preprint arXiv:150308895
Sutjarittham T, Habibi Gharakheili H, Kanhere SS et al (2019) Experiences with iot and ai in a smart campus for optimizing class- room usage. IEEE Internet Things J 6(5):7595–7607. https://doi.org/10.1109/JIOT.2902410
Tan TH, Gochoo M, Huang SC et al (2018) Multi-resident activity recognition in a smart home using rgb activity image and dcnn. IEEE Sens J 18(23):9718–9727
Tan J, Koo SG (2014) A survey of technologies in internet of things. In: 2014 IEEE International Conference on Distributed Computing in Sensor Systems, pp 269–274, https://doi.org/10.1109/DCOSS.2014.45
Tang Y, Teng Q, Zhang L et al (2020b) Efficient convolutional neural networks with smaller fil- ters for human activity recognition using wear- able sensors. IEEE Sens J PP. https://doi.org/10.1109/JSEN.2020.3015521
Tang J, Feng Y, Zhao D (2020a) Understanding procedural text using interactive entity net- works. In: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp 7281–7290
Tapia EM, Intille SS, Larson K (2004) Activity recognition in the home using simple and ubiq- uitous sensors. In: International conference on pervasive computing, Springer, pp 158–175
Teng Q, Wang K, Zhang L et al (2020) The layer-wise training convolutional neural networks using local loss for sensor-based human activity recognition. IEEE Sens J 20(13):7265–7274. https://doi.org/10.1109/JSEN.2020.2978772
Tez S, Aykutlu U, Torunbalci MM et al (2015) A bulk-micromachined three-axis capacitive mems accelerometer on a single die. J Micro- Electromech Syst 24(5):1264–1274
Trivedi R, Mathur G, Mathur A (2011) A survey on platinum temperature sensor. Int J Soft Comput Eng 1
Tsai SM, Wu SS, Sun SS et al (2000) Integrated home service network on intelligent intranet. IEEE Trans Consum Electron 46(3):499–504
Uddin MZ, Hassan M, Alsanad A et al (2019) A body sensor data fusion and deep recur- rent neural network-based behavior recogni- tion approach for robust healthcare. Inform Fus. https://doi.org/10.1016/j.inffus.2019.08.004
Van KT, Noulas A, Englebienne G, et al (2008) Accurate activity recognition in a home set- ting. In: Proceedings of the 10th international conference on Ubiquitous computing, pp 1–9
Vaswani A, Shazeer N, Parmar N, et al (2017) Attention is all you need. In: Advances in neural information processing systems, pp 5998–6008
Virone G, Alwan M, Dalal S et al (2008) Behav- ioral patterns of older adults in assisted living. IEEE Trans Inf Technol Biomed 12(3):387–398
Walse KH, Dharaskar RV, Thakare VM (2016) Pca based optimal ann classifiers for human activity recognition using mobile sensors data. In: Satapathy SC (ed) In: Proceedings of First International Con-ference on Information and Communication Technology for Intelligent Systems. Springer, Cham, pp 429–436
Wang W, Pan SJ, Dahlmeier D et al (2017) Cou- pled multi-layer attentions for co-extraction of aspect and opinion terms. Proceed AAAI Conf Artif Intell. https://doi.org/10.1609/aaai.v31i1.10974
Wang A, Chen G, Shang C, et al (2016) Human activity recognition in a smart home environ- ment with stacked denoising autoencoders. In: International conference on web-age informa- tion management, Springer, pp 29–40
Weiss GM (2019) Wisdm smartphone and smart- watch activity and biometrics dataset. UCI Machine Learning Repository: WISDM Smart- phone and Smartwatch Activity and Biometrics Dataset Data Set
Weston J, Chopra S, Bordes A (2014) Memory networks. arXiv preprint arXiv:14103916
Wilson C, Hargreaves T, Hauxwell-Baldwin R (2017) Benefits and risks of smart home tech- nologies. Energy Policy 103:72–83
Wu J, Osuntogun A, Choudhury T, et al (2007) A scalable approach to activity recognition based on object use. pp 1–8, https://doi.org/10.1109/ICCV.2007.4408865
Xi R, Hou M, Fu M, et al (2018) Deep dilated convolution on multimodality time series for human activity recognition. In: 2018 Interna- tional Joint Conference on Neural Networks (IJCNN), IEEE, pp 1–8
Xia K, Huang J, Wang H (2020) Lstm-cnn architecture for human activity recognition. IEEE Access 8(56):855–856. https://doi.org/10.1109/ACCESS.2020.2982225
Xu C, Chai D, He J et al (2019) Innohar: A deep neural network for complex human activity recognition. Ieee Access 7:9893–9902
Yang J, Nguyen MN, San PP, et al (2015) Deep convolutional neural networks on multichannel time series for human activity recognition. In: Twenty-fourth international joint conference on artificial intelligence
Youssef A, Aerts JM, Vanrumste B, et al (2020) A localised learning approach applied to human activity recognition. Intelligent Sys- tems, IEEE PP:1–1. https://doi.org/10.1109/MIS.2020.2964738
Zhang Y, Zhang Z, Zhang Y et al (2019b) Human activity recognition based on motion sensor using u-net. IEEE Access 7:75–226. https://doi.org/10.1109/ACCESS.2019.2920969
Zhang M, Sawchuk AA (2012) Usc-had: a daily activity dataset for ubiquitous activity recog- nition using wearable sensors. In: Proceedings of the 2012 ACM conference on ubiquitous computing, pp 1036–1043
Zhang T, Ser W, Daniel GYT, et al (2010) Sound Based Heart Rate Monitoring for Wearable Systems. In: 2010 International Conference on Body Sensor Networks. IEEE, pp 139–143, https://doi.org/10.1109/BSN.2010.25, URL http://ieeexplore.ieee.org/document/5504744/
Zhang L, Wu X, Luo D (2015) Human activ- ity recognition with hmm-dnn model. In: 2015 IEEE 14th International Conference on Cogni- tive Informatics & Cognitive Computing (ICCI* CC), IEEE, pp 192–197
Zhang Y, Marshall I, Wallace BC (2016) Rationale-augmented convolutional neural net- works for text classification. In: Proceedings of the Conference on Empirical Methods in Natu- ral Language Processing. Conference on Empir- ical Methods in Natural Language Processing, NIH Public Access, p 795
Zhang X, Wong Y, Kankanhalli M et al (2019a) Hierarchical multi-view aggregation network for sensor-based human activity recognition. PLoS ONE. https://doi.org/10.1371/journal.pone.0221390
Zhao R, Wang K, Su H et al (2019) Bayesian graph convolution lstm for skeleton based action recognition. IEEE/CVF Int Conf Comput vis (ICCV). https://doi.org/10.1109/ICCV.2019.00698
Zheng W, Yan L, Gou C et al (2021) Meta- learning meets the internet of things: Graph prototypical models for sensor-based human activity recognition. Inform Fusion. https://doi.org/10.1016/j.inffus.2021.10.009
Zhong J, Ling C, Cangelosi A et al (2021) On the gap between domestic robotic applica- tions and computational intelligence. Electron- Ics 10(7):793
Zhong JJ, Lotfi A (2020) Sensor2vec: An embed- ding learning for heterogeneous sensors for activity classification. In: 2020 International Symposium on Community-Centric Systems, CcS 2020, Institute of Electrical and Electronics Engineers Inc., p 9231478
Zhong J, Han T, Lotfi A, et al (2019) Bridg- ing the gap between robotic applications and computational intelligence in domestic robotics. In: 2019 IEEE Symposium Series on Computa- tional Intelligence (SSCI), IEEE, pp 1445–1452
Funding
Hong Kong Polytechnic University, ZVUY-P0035417, Junpei Zhong,CD5E-P0043422, Junpei Zhong, WZ09-P0043123, Junpei Zhong, National Natural Science Foundation of China, 42177440, Hongjie Jiang,
Author information
Authors and Affiliations
Contributions
Conceptualization, JY and JZ and XW; methodology, JY and XW; formal analysis, JZ and XW; investigation, JY and JZ and XW; writing—original draft preparation, JY and XW; writing—review and editing, JY, JZ and HJ and XW; super- vision, JZ and HJ; project administration, JZ and HJ. All authors have read and agreed to the published version of the manuscript.
Corresponding authors
Ethics declarations
Conflict of interest
The authors declare no conflict of interest.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Appendix
Appendix
See Figs.
The usage of object sensor based on RFID tech- nology in kitchen setup (red dashed ellipse) Wu et al (2007).
1,
The usage of temperature sensor for electronic skin Hong et al (2015).
2,
The humanoid robot NAO with 5 key poses: standing, speaking, start presentation, emphasis, surprise Zhong et al (2021).
3,
The humanoid robot chating with human Zhong et al (2021)
4,
Figure on the left shows a basic intelligent home system based on active multi-sensor networks, which contains various software components such as a sensing component, decision component, control component, etc. We can realize remotely intelligent control through this software just by a smartphone Suh and Ko (2008). Figure on the right is designed for elders to protect their health situation and monitor their physical changes in real-time Jung (2017)
5,
The utilization of switch sensor made of data collection circuit board and magnetic sensor Tapia et al (2004).
6,
The usage of motion sensor based on analog PIR technology in ALPAS IDE area Kashimoto et al (2017).
7,
The usage of light sensor in DIE ( black dashed ellipse ) Pandharipande and Li (2013).
8,
The basic electrical structure and fabrication of three-axis MEMS accelerometer and three-dimensional signal of the x-axis, y-axis, and z-axis Tez et al (2015)
9,
The proposed activity recognition method to detect elderly ADLs using wrist-worn multi-sensors, which is one typical wearable heart rate sensor ( red dashed ellipse ) Chernbumroong et al (2013); Zhang et al (2010)
10,
The basic concept of Internet of Thing Gurunath et al (2018).
We introduce a general process of CNN used in the HAR field. In this implementation, the CNN is used to extract features from input data (some photos from human activities), through the flattened and fully connected layer, to realize the recognition mission
A basic human attention network (HAN) based on the CNN and GRU units for predicting the attention map, which is also a general procedure for HAR application Qiao et al. (2018)
See Tables 1
1 and
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Ye, J., Wang, M., Zhong, J. et al. A review on devices and learning techniques in domestic intelligent environment. J Ambient Intell Human Comput 15, 2361–2380 (2024). https://doi.org/10.1007/s12652-024-04759-1
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s12652-024-04759-1