Springer Nature is making Coronavirus research free. View research | View latest news | Sign up for updates

Personalized gesture interactions for cyber-physical smart-home environments


  • 229 Accesses

  • 9 Citations


A gesture-based interaction system for smart homes is a part of a complex cyber-physical environment, for which researchers and developers need to address major challenges in providing personalized gesture interactions. However, current research efforts have not tackled the problem of personalized gesture recognition that often involves user identification. To address this problem, we propose in this work a new event-driven service-oriented framework called gesture services for cyber-physical environments (GS-CPE) that extends the architecture of our previous work gesture profile for web services (GPWS). To provide user identification functionality, GS-CPE introduces a two-phase cascading gesture password recognition algorithm for gesture-based user identification using a two-phase cascading classifier with the hidden Markov model and the Golden Section Search, which achieves an accuracy rate of 96.2% with a small training dataset. To support personalized gesture interaction, an enhanced version of the Dynamic Time Warping algorithm with multiple gestural input sources and dynamic template adaptation support is implemented. Our experimental results demonstrate the performance of the algorithm can achieve an average accuracy rate of 98.5% in practical scenarios. Comparison results reveal that GS-CPE has faster response time and higher accuracy rate than other gesture interaction systems designed for smart-home environments.


1. 论文设计了一个支持智能家庭环境中个性化手势交互的集成框架GS-CPE,该框架同时集成了用户身份识别与个性化手势识别,通过服务的形式提供了一组简单易用的API。 2. 论文提出了一种可用于用户身份识别的手势识别算法。通过结合基于统计模型的HMM分类器及基于模板匹配的GSS分类器,该算法可在训练数据集有限的条件下较大幅度地提高手势密码的识别准确率。 3. 论文提出了一种融合多数据源的个性化手势识别算法,通过对比来自不同数据源的手势识别结果来消除对拒绝似然比的依赖,并针对实际环境设计了手势模板动态调节方法,从而提高了该算法的识别准确率。

This is a preview of subscription content, log in to check access.


  1. 1

    Bernhaupt R, Obrist M, Weiss A, et al. Trends in the living room and beyond: results from ethnographic studies using creative and playful probing. ACM CIE, 2008, 6: 5

  2. 2

    Panger G. Kinect in the kitchen: testing depth camera interactions in practical home environments. In: Proceedings of the CHI Extended Abstracts on Human Factors in Computing Systems. New York: ACM, 2012. 1985–1990

  3. 3

    Pan G, Wu J H, Zhang D Q, et al. GeeAir: a universal multimodal remote control device for home appliances. Pers Ubiquitous Comput, 2010, 14: 723–735

  4. 4

    Vatavu R D. Point & click mediated interactions for large home entertainment displays. Multimed Tools Appl, 2012, 59: 113–128

  5. 5

    Kühnel C, Westermann T, Hemmert F, et al. I’m home: defining and evaluating a gesture set for smart-home control. Int J Hum-Comput Stud, 2011, 69: 693–704

  6. 6

    Vatavu R D. A comparative study of user-defined handheld vs. freehand gestures for home entertainment environments. J Ambient Intell Smart Environ, 2013, 5: 187–211

  7. 7

    Vatavu R D, Zaiti I A. Leap gestures for TV: insights from an elicitation study. In: Proceedings of the ACM International Conference on Interactive Experiences for TV and Online Video. New York: ACM, 2014. 131–138

  8. 8

    Li W, Lee Y H, Tsai W T, et al. Service-oriented smart home applications: composition, code generation, deployment, and execution. Serv Oriented Comput Appl, 2012, 6: 65–79

  9. 9

    Vatavu R D, Chera C M, Tsai W T. Gesture profile for web services: an event-driven architecture to support gestural interfaces for smart environments. In: Ambient Intelligence. Berlin: Springer-Verlag, 2012. 161–176

  10. 10

    Lou Y H, Wu W J. A real-time personalized gesture interaction system using Wii remote and Kinect for tiled-display environment. In: Proceedings of the International Conference on Software Engineering and Knowledge Engineering. Skokie: KSI, 2013. 131–136

  11. 11

    Zhang H K, Wu W J, Lou Y H. A personalized gesture interaction system with user identification using Kinect. In: PRICAI 2014: Trends in Artificial Intelligence. Berlin: Springer, 2014. 614–626

  12. 12

    Vatavu R D. User-defined gestures for free-hand TV control. In: Proceedings of the 10th European Conference on Interactive TV and Video. New York: ACM, 2012. 45–48

  13. 13

    Wobbrock J O, Morris M R, Wilson A D, et al. User-defined gestures for surface computing. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. New York: ACM, 2009. 1083–1092

  14. 14

    Vatavu R D, Wobbrock J O. Formalizing agreement analysis for elicitation studies: new measures, significance test, and toolkit. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. New York: ACM, 2015. 1325–1334

  15. 15

    Wobbrock J O, Aung H H, Brandon R, et al. Maximizing the guessability of symbolic input. In: Proceedings of the CHI Extended Abstracts on Human Factors in Computing Systems. New York: ACM, 2005. 1869–1872

  16. 16

    Lou Y H, Yao T, Chen Y Q, et al. A novel scheme of ROI detection and transcoding for mobile devices in high-definition videoconferencing. In: Proceedings of the 5th Workshop on Mobile Video. New York: ACM, 2013. 31–36

  17. 17

    Wang Y W, Yang C, Wu X, et al. Kinect based dynamic hand gesture recognition algorithm research. In: Proceedings of the 4th International Conference on Intelligent Human-Machine Systems and Cybernetics, Nanchang, 2012. 274–279

  18. 18

    Zhu H M, Pun C M. Real-time hand gesture recognition form depth image sequences. In: Proceedings of the 9th International Conference on Computer Graphics, Imaging and Visualization, Hsinchu, 2012. 49–52

  19. 19

    Moni M A, Shawkat Ali A B M. HMM based hand gesture recognition: a review on techniques and approaches. In: Proceedings of the 2nd IEEE International Conference on Computer Science and Information Technology, Beijing, 2009. 433–437

  20. 20

    Kiefer J. Sequential minimax search for a maximum. Proc American Math Soc, 1953, 4: 502–506

  21. 21

    Vatavu R D, Anthony L, Wobbrock J O. Gestures as point clouds: a $P recognizer for user interface prototypes. In: Proceedings of the 14th ACM International Conference on Multimodal Interaction. New York: ACM, 2012. 273–280

  22. 22

    Myers C S, Rabiner L R. A comparative study of several dynamic time-warping algorithms for connected word recognition. Bell Syst Tech J, 1981, 60: 1389–1409

  23. 23

    Wobbrock J O, Wilson A D, Li Y. Gestures without libraries, toolkits or training: a $1 recognizer for user interface prototypes. In: Proceedings of the 20th Annual ACM Symposium on User Interface Software and Technology. New York: ACM, 2007. 159–168

  24. 24

    Carmona J M, Climent J. A performance evaluation of HMM and DTW for gesture recognition. In: Progress in Pattern Recognition, Image Analysis, Computer Vision, and Applications. Berlin: Springer-Verlag, 2012. 236–243

  25. 25

    Liu J Y, Zhong L, Wickramasuriya J, et al. uWave: accelerometer-based personalized gesture recognition and its applications. Pervasive Mob Comput, 2009, 5: 657–675

  26. 26

    Reyes M, Dominguez G, Escalera S. Feature weighting in dynamic time warping for gesture recognition in depth data. In: Proceedings of the IEEE International Conference on Computer Vision Workshops. Piscataway: IEEE, 2011. 1182–1188

  27. 27

    Liu X, Mu Y, Zhang D, et al. Large-scale unsupervised hashing with shared structure learning. IEEE Trans Cybern, 2015, 45: 1811–1822

  28. 28

    Liu X, Deng C, Lang B, et al. Query-adaptive reciprocal hash tables for nearest neighbor search. IEEE Trans Image Process, 2015, 25: 907–919

  29. 29

    Chen M Y, AlRegib G, Juang B H. 6DMG: a new 6D motion gesture database. In: Proceedings of the 3rd Multimedia Systems Conference. New York: ACM, 2012. 83–88

  30. 30

    Mitra S, Acharya T. Gesture recognition: a survey. IEEE Trans Syst Man Cybern Part C-Appl Rev, 2007, 37: 311–324

  31. 31

    Ruiz J, Li Y, Lank E. User-defined motion gestures for mobile interaction. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. New York: ACM, 2011. 197–206

  32. 32

    Schlömer T, Poppinga B, Henze N, et al. Gesture recognition with a Wii controller. In: Proceedings of the 2nd International Conference on Tangible and Embedded Interaction. New York: ACM, 2008. 11–14

  33. 33

    Vatavu R D. Nomadic gestures: a technique for reusing gesture commands for frequent ambient interactions. J Ambient Intell Smart Environ, 2012, 4: 79–93

  34. 34

    Pu Q F, Gupta S, Gollakota S, et al. Whole-home gesture recognition using wireless signals. In: Proceedings of the 19th Annual International Conference on Mobile Computing & Networking. New York: ACM, 2013. 27–38

  35. 35

    van Seghbroeck G, Verstichel S, de Truck F, et al. WS-Gesture: a gesture-based state-aware control framework. In: Proceedings of the IEEE International Conference on Service-Oriented Computing and Applications. Piscataway: IEEE, 2010. 1–8

  36. 36

    Zheng Y W, Sheng H, Zhang B C, et al. Weight-based sparse coding for multi-shot person re-identification. Sci China Inf Sci, 2015, 58: 100104

  37. 37

    Hayashi E, Maas M, Hong J I. Wave to me: user identification using body lengths and natural gestures. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. New York: ACM, 2014. 3453–3462

  38. 38

    Wang D X, Xiong Z H, Zhang M. An application oriented and shape feature based multi-touch gesture description and recognition method. Multimed Tools Appl, 2012, 58: 497–519

  39. 39

    Chera C M, Tsai W T, Vatavu R D. Gesture ontology for informing Service-oriented architecture. In: Proceedings of IEEE International Symposium on Intelligent Control. Piscataway: IEEE, 2012. 1184–1189

Download references

Author information

Correspondence to Wenjun Wu.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Lou, Y., Wu, W., Vatavu, R. et al. Personalized gesture interactions for cyber-physical smart-home environments. Sci. China Inf. Sci. 60, 072104 (2017). https://doi.org/10.1007/s11432-015-1014-7

Download citation


  • gesture interaction
  • personalization
  • gesture recognition
  • user identification
  • event-driven architecture


  • 手势交互
  • 个性化
  • 手势识别
  • 用户识别
  • 事件驱动架构


  • 072104