Abstract
With advances in microelectromechanical systems (MEMS), researchers have now become interested in the systems operating based on inertial signals. In fact, inertial signals have proven useful in different areas due to advances in their manufacturing technology, availability, and inexpensiveness as well as the development of powerful processing methods such as deep learning techniques. Handwritten character recognition (HCR) is among such areas. This paper aimed to design, implement, and evaluate a novel system for the recognition of handwritten Farsi characters extracted from an inertial pen. For this purpose, a wireless inertial pen was designed. Its motion trajectory was then determined by combining the signals of its angular velocity and acceleration and using the concepts of navigation systems such as quaternion in order to estimate the position signals of characters. A convolutional neural network (CNN) was also employed to facilitate the extraction of high-level features and the classification of characters. The position signal was also extracted as an image used for model learning to enhance the classifier efficiency. The experimental results indicated the CNN-6 architecture outperformed the other CNN-n architectures in terms of character classification accuracy. According to the evaluation of the proposed method through test data, character recognition accuracies of Farsi letters and numbers were reported 91.06% and 94.52%, respectively. In comparison with the previous systems, the proposed method managed to improve the recognition of handwritten Farsi characters.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Akbarisanto R, Danar W, Purwarianti A (2016) Analyzing bandung public mood using Twitter data. In: 2016 4th international conference on information and communication technology (ICoICT), 25–27 May 2016, pp 1–6
Breuel TM, Ul-Hasan A, Al-Azawi MA, Shafait F (2013) High-performance OCR for printed English and Fraktur using LSTM networks. In: 2013 12th international conference on document analysis and recognition, 25–28 Aug 2013, pp 683–687
Choi S, Lee S (2012) 3D stroke reconstruction and cursive script recognition with magnetometer-aided inertial measurement unit. IEEE Trans Consum Electron 58(2):661–669. https://doi.org/10.1109/TCE.2012.6227474
Christian M, Uyanik C, Erdemir E, Kaplanoglu E, Bhattacharya S, Bailey R et al (2019) Application of deep learning to IMU sensor motion. In: 2019 SoutheastCon. 11–14 April 2019, pp 1–6
Euston M, Coote P, Mahony R, Kim J, Hamel T (2008) A complementary filter for attitude estimation of a fixed-wing UAV. In: 2008 IEEE/RSJ international conference on intelligent robots and systems, 22–26 Sept. 2008, pp 340–345
Gałka J, Mąsior M, Zaborski M, Barczewska K (2016) Inertial motion sensing glove for sign language gesture acquisition and recognition. IEEE Sens J 16(16):6310–6316. https://doi.org/10.1109/JSEN.2016.2583542
Hidalgo-Carrió J, Arnold S, Poulakis P (2016) On the design of attitude-heading reference systems using the allan variance. IEEE Trans Ultrason Ferroelectr Freq Control 63(4):656–665. https://doi.org/10.1109/TUFFC.2016.2519268
Hsu Y-L, Yang S-C, Chang H-C, Lai H-C (2018) Human daily and sport activity recognition using a wearable inertial sensor network. IEEE Access 6:31715–31728. https://doi.org/10.1109/access.2018.2839766
Jing L, Dai Z, Zhou Y (2017) wearable handwriting recognition with an inertial sensor on a finger nail. In: 2017 14th IAPR international conference on document analysis and recognition (ICDAR), 9–15 Nov. 2017, pp 1330–1337
Kamilaris A, Prenafeta-Boldú FX (2018) Deep learning in agriculture: a survey. Comput Electron Agric 147:70–90. https://doi.org/10.1016/j.compag.2018.02.016
Khomami SA, Shamekhi S (2021) Persian sign language recognition using IMU and surface EMG sensors. Measurement 168:108471. https://doi.org/10.1016/j.measurement.2020.108471
Kim M, Cho J, Lee S, Jung Y (2019) IMU sensor-based hand gesture recognition for human-machine interfaces. Sensors (basel). https://doi.org/10.3390/s19183827
Krizhevsky A, Sutskever I, Hinton GE (2012) ImageNet classification with deep convolutional neural networks. In: Paper presented at the proceedings of the 25th international conference on neural information processing systems, Volume 1, Lake Tahoe, Nevada
Lecun Y, Bottou L, Bengio Y, Haffner P (1998) Gradient-based learning applied to document recognition. Proc IEEE 86(11):2278–2324. https://doi.org/10.1109/5.726791
LeCun Y, Bengio Y, Hinton G (2015) Deep learning. Nature 521(7553):436–444. https://doi.org/10.1038/nature14539
Liu ZT, Wong DPY, Chou PH (2020) An Imu-based wearable ring for on-surface handwriting recognition. In: 2020 international symposium on VLSI design, automation and test (VLSI-DAT), 10–13 Aug. 2020, pp 1–4
Lu C, Uchiyama H, Thomas D, Shimada A, Taniguchi R-I (2019) Indoor positioning system based on chest-mounted IMU. Sensors. https://doi.org/10.3390/s19020420
Mueller PA, Oppenheimer DM (2014) The pen is mightier than the keyboard: advantages of longhand over laptop note taking. Psychol Sci 25(6):1159–1168. https://doi.org/10.1177/0956797614524581
Pan T-Y, Kuo C-H, Liu H-T, Hu M-C (2018) Handwriting trajectory reconstruction using low-cost IMU. IEEE Trans Emerg Top Comput Intell. https://doi.org/10.1109/tetci.2018.2803777
Rangasamy K, As’ari MA, Rahmad NA, Ghazali NF (2020) Hockey activity recognition using pre-trained deep learning model. ICT Express 6(3):170–174. https://doi.org/10.1016/j.icte.2020.04.013
Schrapel M, Stadler M-L, Rohs M (2018) Pentelligence: combining pen tip motion and writing sounds for handwritten digit recognition. In: Proceedings of the 2018 CHI conference on human factors in computing systems, p Paper 131. Association for Computing Machinery. https://doi.org/10.1145/3173574.3173705
Sepahvand M, Abdali-Mohammadi F (2021a) A novel multi-lead ECG personal recognition based on signals functional and structural dependencies using time-frequency representation and evolutionary morphological CNN. Biomed Signal Process Control 68:102766. https://doi.org/10.1016/j.bspc.2021.102766
Sepahvand M, Abdali-Mohammadi F (2021b) A novel representation in genetic programming for ensemble classification of human motions based on inertial signals. Expert Syst Appl. https://doi.org/10.1016/j.eswa.2021.115624
Sepahvand M, Abdali-Mohammadi F, Mardukhi F (2017) Evolutionary metric-learning-based recognition algorithm for online isolated Persian/Arabic characters, reconstructed using inertial pen signals. IEEE Trans Cybern 47(9):2872–2884. https://doi.org/10.1109/TCYB.2016.2633318
Serrano JI, Lambrecht S, del Castillo MD, Romero JP, Benito-León J, Rocon E (2017) Identification of activities of daily living in tremorous patients using inertial sensors. Expert Syst Appl 83:40–48. https://doi.org/10.1016/j.eswa.2017.04.032
Shengli Z, Zhuxin D, Li WJ, Chung Ping K (2008) Hand-written character recognition using MEMS motion sensing technology. In: Paper presented at the 2008 IEEE/ASME international conference on advanced intelligent mechatronics
Tick DF, Satici AC, Shen J, Gans N (2012) Tracking control of mobile robots localized via chained fusion of discrete and continuous epipolar geometry, IMU and odometry, pp 2168–2275 (Electronic)
Wang J-S, Chuang F-C (2012) An accelerometer-based digital pen with a trajectory recognition algorithm for handwritten digit and gesture recognition. IEEE Trans Ind Electron 59(7):2998–3007. https://doi.org/10.1109/tie.2011.2167895
Weng Y, Xia C (2020) A new deep learning-based handwritten character recognition system on mobile computing devices. Mobile Netw Appl 25(2):402–411. https://doi.org/10.1007/s11036-019-01243-5
Yu-Liang H, Cheng-Ling C, Yi-Ju T, Jeen-Shing W (2015) An inertial pen with dynamic time warping recognizer for handwriting and gesture recognition. IEEE Sens J 15(1):154–163. https://doi.org/10.1109/jsen.2014.2339843
Zeiler MD, Fergus R (2014) Visualizing and understanding convolutional networks. Springer International Publishing, Cham, pp 818–833
Zhang X, Yin F, Zhang Y, Liu C, Bengio Y (2018) Drawing and recognizing chinese characters with recurrent neural network. IEEE Trans Pattern Anal Mach Intell 40(4):849–862. https://doi.org/10.1109/TPAMI.2017.2695539
Zhang J, Bi H, Chen Y, Wang M, Han L, Cai L (2020) SmartHandwriting: handwritten chinese character recognition with smartwatch. IEEE Internet Things J 7(2):960–970. https://doi.org/10.1109/JIOT.2019.2947448
Zhao Y (2017) Applying time-differenced carrier phase in nondifferential GPS/IMU tightly coupled navigation systems to improve the positioning performance. IEEE Trans Veh Technol 66(2):992–1003. https://doi.org/10.1109/TVT.2016.2558206
Žumer J, Reynaerts D, Boltežar M (2012) An advanced nonlinear model of a low-g MEMS accelerometer for a computer pen. Measurement 45(3):459–468. https://doi.org/10.1016/j.measurement.2011.10.027
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Meshkat, F., Abdali-Mohammadi, F. A new in-air handwritten persian characters recognition method based on inertial sensor position estimation and convolutional neural network. J Ambient Intell Human Comput 14, 13097–13112 (2023). https://doi.org/10.1007/s12652-022-03770-8
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s12652-022-03770-8