Students’ behavior mining in e-learning environment using cognitive processes with information technologies

  • Ahmad JalalEmail author
  • Maria Mahmood


Rapid growth and recent developments in education sector and information technologies have promoted E-learning and collaborative sessions among the learning communities and business incubator centers. Traditional practices are being replaced with webinars (live online classes) E-Quizes (online testing) and video lectures for effective learning and performance evaluation. These E-learning methods use sensors and multimedia tools to contribute in resource sharing, social networking, interactivity and corporate trainings. While, artificial intelligence tools are also being integrated into various industries and organizations for students’ engagement and adaptability towards the digital world. Predicting students’ behaviors and providing intelligent feedbacks is an important parameter in the E-learning domain. To optimize students’ behaviors in virtual environments, we have proposed an idea of embedding cognitive processes into information technologies. This paper presents hybrid spatio-temporal features for student behavior recognition (SBR) system that recognizes student-student behaviors from sequences of digital images. The proposed SBR system segments student silhouettes using neighboring data points observation and extracts co-occurring robust spatio-temporal features having full body and key body points techniques. Then, artificial neural network is used to measure student interactions taken from UT-Interaction and classroom behaviors datasets. Finally a survey is performed to evaluate the effectiveness of video based interactive learning using proposed SBR system.


Artificial intelligence Behavior mining E-learning Student behavior recognition 


Compliance with Ethical Standards

Conflict of interests

The authors declare that they have no conflict of interest.


  1. Buys, K., Cagniart, C., Baksheev, A., Laet, T.-D., Schutter, J.D., Pantofaru, C. (2014). An adaptable system for RGB-D based human body detection and pose estimation. Journal of visual communication and image representation, 25, 39–52.CrossRefGoogle Scholar
  2. Oberg, J., Eguro, K., Bittner, R., Forin, A. (2012). Random decision tree body part recognition using FPGAS. In: Proceedings of international conference on field programmable logic and applications, pp. 330–337.Google Scholar
  3. Jalal, A., & Zeb, M.A. (2008). Security enhancement for e-learning portal. International Journal of Computer Science and Network Security, 8(3), 41–45.Google Scholar
  4. Kanungo, T., Mount, D.M., Netanyahu, N.S., Piatko, C.D., Silverman, R., Wu, A.Y. (2002). An efficient k-means clustering algorithm: analysis and implementation. IEEE Transaction on Pattern Analysis and Machine Intelligence, 24(7), 881–892.CrossRefzbMATHGoogle Scholar
  5. Yang, X., & Tian, Y. (2014). Super normal vector for activity recognition using depth sequences. In: Proceedings of CVPR conference, Columbus, pp. 804–811.Google Scholar
  6. Sehar, R., Mahmood, M., Yousaf, S., Khatoon, H., Khan, S., Moqurrab, S.A. (2018). An Investigation on Students Speculation towards Online Evaluation. In: Proceedings of 11th International Conference on Assessments and Evaluation on global south.Google Scholar
  7. Yang, X., & Tian, Y. (2012). Eigenjoints-based action recognition using naive-bayes-neartest-neighbor. In: Proceedings of CVPR conference, Providence, RI, pp 14–19.Google Scholar
  8. Jalal, A., Kim, Y., Kim, D. (2014). Ridge body parts features for human pose estimation and recognition from RGB-D video data. In: Proceedings of the IEEE international conference on computing, communication and networking technologies.Google Scholar
  9. Muller, M., & Roder, T. (2006). Motion templates for automatic classification and retrieval of motion capture data. In: Proceedings of ACM symposium on computer animation, Austria, pp. 137–146.Google Scholar
  10. Mahmood, M., Jalal, A., Evans, H.A. In press. (2018). Facial expression recognition in image sequences using 1D transform and gabor wavelet transform. In: Proceedings of international conference on applied and engineering mathematics.Google Scholar
  11. Fatahi, S., Shabanali-Fami, F., Moradi, H. (2018). An empirical study of using sequential behavior pattern mining approach to predict learning styles. Journal of Education and Information Technologies, 23(4), 1427–1445.CrossRefGoogle Scholar
  12. Aissaoui, O., Madani, Y., Oughdir, L., Allioui, Y. (2018). A fuzzy classification approach for learning style prediction based on web mining technique in e-learning environments. Journal of Education and Information Technologies, pp. 1–17.Google Scholar
  13. Zhao, X., Li, X., Pang, C., Wang, S. (2013). Human action recognition based on semi-supervised discriminant analysis with global constraints. Neurocomputing, 105, 45–50.CrossRefGoogle Scholar
  14. Jalal, A., Sharif, N., Kim, J.T., Kim, T.S. (2013). Human Activity Recognition via Recognized Body Parts of Human Depth Silhouettes for Residents Monitoring Services at Smart Home. Indoor and Built Environment, 22, 271–279.CrossRefGoogle Scholar
  15. Houda, K., & Yannick, F. (2014). Human interaction recognition based on the co-occurrence of visual words. In: Proceedings of CVPR conference, pp. 455–460.Google Scholar
  16. Ryoo, M.S., & Aggarwal, J.K. (2009). Spatio-temporal relationship match: video structure comparison for recognition of complex human activities. In: Proceedings of ICCV, pp. 1593–1600.Google Scholar
  17. Berlin, S.J., & John, M. (2016). Human interaction recognition through Deep Learning Network. In: Proceedings of IEEE International Carnahan conference on security technology.Google Scholar
  18. Chattopadhyay, C., & Das, S. (2016). Supervised framework for automatic recognition and retreival of interaction: a framework for classification and retrieving videos with similar human interactions. IET Computer Vision, 10, 220–227.CrossRefGoogle Scholar
  19. Zhan, S., & Chang, I. (2014). Pictorial structures model based human interaction recognition. In: Proceedings of ICMLC, pp. 862–866.Google Scholar
  20. Hwang, G-J., Yang, T-C., Tsai, C-C., Yang, J.H. (2009). A context-aware ubiquitous learning environment for conducting complex science experiments. In: Computers and Education, Volume 53 (2).Google Scholar
  21. Lu, T., Zhang, S., Hao, Q., Yang, J.H. (2012). Activity Recognition in Ubiquitous Learning Environment. In: Journal of advances in information technology, Volume 3 (1).Google Scholar
  22. Chen, K., Yue, G., Yu, F., Shen, Y., Zhu, A. (2007). Research on speech emotion recognition system in E-learning. In Lecture notes in computer science, Vol. 4489. Berlin: Springer.Google Scholar
  23. Kowalewski, W., Koodziejczak, B., Roszak, M., Ren-Kurc, A. (2013). Gesture recognition technology in education. In:Distance learning, simulation and communication, pp. 113–120.Google Scholar
  24. Zaletelj, J., & Košir, A. (2017). Predicting students’ attention in the classroom from Kinect facial and body features. In: EURASIP journal on image and video processing.Google Scholar
  25. Sabanc, O., & Bulut, S. (2018). The Recognition and Behavior Management of Students With Talented and Gifted in an Inclusive Education Environment. In:Journal of Education and Training Studies, Volume 6 (6).Google Scholar
  26. Gaschler, A., Jentzsch, S., Giuliani, M., Huth, K., Ruiter, J., Knoll, A. (2012). Social behavior recognition using body posture and head pose for human-robot interaction. In: Proceedings of IEEE/RSJ international conference on intelligent robots and systems, pp. 2128–2133.Google Scholar
  27. Fujii, T., Lee, J., Okamoto, S. (2014). Gesture Recognition System for Human-Robot Interaction and its application to robotic service task. In: Proceedings of international multiconference of engineers and computer scientists, pp. 63–68.Google Scholar
  28. Babiker, M., Khalifa, O., Htyke, K., Hassan, A., Zaharadeen, M. (2017). Automated daily human activity recognition for video surveillance using neural network. In: Proceedings of IEEE 4th International Conference on Smart Instrumentation, Measurement and Application, pp. 1–5.Google Scholar
  29. Gkioxari, G., Girshick, R., Dollár, P., He, K. (2018). Detecting and recognizing human-object interactions. In: Proceedings of computer vision and pattern recognition.Google Scholar
  30. Shen, L., Yeung, S., Hoffman, J., Mori, G., Fei, L. (2018). Scaling human-object interaction recognition through zero-shot learning. In: Proceedings of IEEE winter conference on applications of computer vision, pp. 1568–1576.Google Scholar
  31. Cho, N., Park, S., Park, J., Park, U., Lee, S. (2017). Compositional interaction descriptor for human interaction recognition. Neurocomputing, pp. 169–181.Google Scholar
  32. Kong, Y., Liang, W., Dong, Z., Jia, Y. (2014). Recognizing human interactions from videos by a discriminative model. IET Computer Vision, 8, 277–286.CrossRefGoogle Scholar
  33. Ma, L., Liu, J., Wang, J. (2009). A improved silhouette tracking approach integrating particle filter with graph cuts. In: Proceedings of ICCV, pp.1593–1600.Google Scholar
  34. Jalal, A., Kim, J.T., Kim, T.-S. (2012). Human activity recognition using the labeled depth body parts information of depth silhouettes. In: Proceedings of the 6th international symposium on sustainable healthy buildings, pp. 1–8.Google Scholar
  35. Milanfar, P. (2012). A tour of modern image filtering: New imsights and methods, both practical and theoretical. IEEE signal processing magazine, 30, 106–128.CrossRefGoogle Scholar
  36. Jalal, A., & Kim, Y. (2014). Dense depth maps-based human pose tracking and recognition in dynamic scenes using ridge data. In: Proceedings of the IEEE international conference on advanced video and signal-based surveillance, pp. 119–124.Google Scholar
  37. Jalal, A., Kim, Y.-H., Kim, Y.-J., Kamal, S., Kim, D. (2017). Robust human activity recognition from depth video using spatiotemporal multi-fused features. Pattern recognition, 61, 295–308.CrossRefGoogle Scholar
  38. Enyedi, B., Konyha, L., Fazekas, K. (2005). Threshold procedures and image segmentation. In: Proceedings of the IEEE international symposium ELMAR, pp. 119–124.Google Scholar
  39. Kwang-Kyo, H.-S. (2011). Distance-based formation control using euclidean dstance dynamics matrix: Three-agent case. In: Proceedings of american control conference, pp. 4810–4815.Google Scholar
  40. Javed, J., Yasin, H., Ali, S. (2010). Human movement recognition using euclidean distance: A tricky approach. In: Proceedings of 3rd international congress on image and signal processing.Google Scholar
  41. Sony, A., Ajith, K., Thomas, K., Thomas, T., Deepa, P.L. (2011). Video summarization by clustering using euclidean distance. In: Proceedings of international conference on signal processing, communication, Computing and Networking Technologies.Google Scholar
  42. Jalal, A., Kim, J.T., Kim, T.-S. (2012). Development of a life logging system via depth imaging-based human activity recognition for smart homes. In: Proceedings of the international symposium on sustainable healthy buildings, pp. 91–95.Google Scholar
  43. Li, Q., & Lu, W. (2009). A histogram descriptor based on co-occurrence matrix and its application in cloud image indexing and retrieval. In: Proceedings of 5th international conference on intelligent information hiding and multimedia signal processing.Google Scholar
  44. Jalal, A., Kamal, S., Kim, D. (2014). A depth video sensor-based life-logging human activity recognition system for elderly care in smart indoor environments. Sensors, 14, 11735–11759.CrossRefGoogle Scholar
  45. Walker, R.F., Jackway, P.T., Longstaff, I.D. (2002). Recent developments in the use of the co-occurrence matrix for texture recognition. In: Proceedings of 13th international conference on digital signal processing.Google Scholar
  46. Fan, B., & Wang, Z. (2004). Pose estimation of human body based on silhouette images. In: Proceedings of international conference on information acquisition.Google Scholar
  47. Barnard, M., Matilainen, M., Heikkila, J. (2008). Body part segmentation of noisy human silhouette images. In Proceedings of IEEE international conference on multimedia and expo.Google Scholar
  48. Maric, S.V., & Kolarov, A. (2002). Threshold based admission policies for multi-rate services In: the DECT system. In: Proceedings of 6th international symposium on personal, indoor and mobile radio communications.Google Scholar
  49. Wang, W., Qin, Z., Rong, S., Xingfu, S.R. (2008). A kind of method for selection of optimum threshold for segmentation of digital color plane image. In: Proceedings of 9th international conference on computer-aided industrial design and conceptual design.Google Scholar
  50. Tang, X., Pang, Y., Zhang, H., Zhu, W. (2008). Fast image segmentation method based on threshold. In: Proceedings of Chinese control and decision conference.Google Scholar
  51. Lynch, R., & Willett, P. (2002). Classification with a combined information test. In: Proceedings of IEEE international conference on acoustics, speech, and signal processing.Google Scholar
  52. Wang, J., Wang, S., Cui, Q., Wang, Q. (2016). Local-based active classification of test report to assist crowdsourced testing. In: Proceedings of IEEE international conference on automated software engineering, pp. 190–201.Google Scholar
  53. Zhang, J., Chen, C., Xiang, Y., Zhou, W. (2012). Semi-supervised and compound classification of network traffic. In: Proceedings of international conference on distributed computing systems workshops, pp. 617–62.Google Scholar
  54. Siswanto, A., Nugroho, A., Galinium, M. (2015). Implementation of face recognition algorithm for biometrics based time attendance system. In: Proceedings of International Conference on ICT for Smart Society.Google Scholar
  55. Xu, G., & Lei, Y. (2008). A new image recognition algorithm based on skeleton. In: Proceedings of IEEE world congress on computational intelligence.Google Scholar
  56. Huang, H. (2010). A simplified image recognition algorithm based on simple scenarios. In: Proceedings of international conference on computational intelligence and software engineering.Google Scholar
  57. Turcanik, M. (2010). Network routing by artificial neural network. Military communications and information systems conference.Google Scholar
  58. Maa, C.Y., & Schanblatt, M.A. (1992). A two-phase optimization neural network. IEEE Transactions on Neural Networks, vol. 3.Google Scholar
  59. Lavalle, M., & Rodriguez, G. (2007). Feature selection with interactions for continuous attributed and discrete class. In: Proceedings of electronics, robotics and automative mechanics conference.Google Scholar
  60. Tsang, E.C.C, Huang, D.M., Yeung, D.S., Lee, J.W.T., Wang, X.Z. (2003). A weighted fuzzy reasoning and its corresponding neural network. In: Proceedings of IEEE international conference on systems, man and cybernetics.Google Scholar
  61. Sima, J. (2017). Neural networks between integer and rational weights. In: Proceedings of international joint conference on neural networks.Google Scholar
  62. Ryoo, M.S., & Aggarwal, J.K. (2009). Spatio-temporal relationship match: video structure comparison for recognition of complex human activities. In: Proceedings of IEEE international conference on computer vision.Google Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2019

Authors and Affiliations

  1. 1.Department of Computer Science and EngineeringAir UniversityIslamabadPakistan

Personalised recommendations