The achievement of higher flexibility in multiple-choice-based tests using image classification techniques

  • Mahmoud AfifiEmail author
  • Khaled F. Hussain
Original Paper


In spite of the high accuracy of the existing optical mark reading (OMR) systems and devices, a few restrictions remain existent. In this work, we aim to reduce the restrictions of multiple-choice questions (MCQ) within tests. We use an image registration technique to extract the answer boxes from answer sheets. Unlike other systems that rely on simple image processing steps to recognize the extracted answer boxes, we address the problem from another perspective by training a machine learning classifier to recognize the class of each answer box (i.e., confirmed, crossed out or blank answer). This gives us the ability to deal with a variety of shading and mark patterns, and distinguish between chosen (i.e., confirmed) and canceled answers (i.e., crossed out). All existing machine learning techniques require a large number of examples in order to train a model for classification; therefore, we present a dataset including six real MCQ assessments with different answer sheet templates. We evaluate two strategies of classification: a straightforward approach and a two-stage classifier approach. We test two handcrafted feature methods and a convolutional neural network. At the end, we present an easy-to-use graphical user interface of the proposed system. Compared with existing OMR systems, the proposed system has the least constraints and achieves a high accuracy. We believe that the presented work will further direct the development of OMR systems toward reducing the restrictions of the MCQ tests.


MCQ Optical mark reading OMR CNN BoVW Dataset of examinations 



  1. 1.
    Gronlund, N.E.: Assessment of student achievement. ERIC (1998)Google Scholar
  2. 2.
    McCoubrie, P.: Improving the fairness of multiple-choice questions: a literature review. Med. Teach. 26(8), 709–712 (2004)Google Scholar
  3. 3.
    Aldabe, I., Maritxalar, M.: Semantic similarity measures for the generation of science tests in basque. IEEE Trans. Learn. Technol. 7(4), 375–387 (2014)Google Scholar
  4. 4.
    Liu, M., Rus, V., Liu, L.: Automatic chinese multiple choice question generation using mixed similarity strategy. IEEE Trans. Learn. Technol. 11(2), 193–202 (2017)Google Scholar
  5. 5.
    Spadaccini, A., Rizzo, V.: A multiple-choice test recognition system based on the Gamera framework (2011). arXiv preprint arXiv:1105.3834
  6. 6.
    Chai, D.: Automated marking of printed multiple choice answer sheets. In: IEEE International Conference on Teaching, Assessment, and Learning for Engineering (TALE), pp. 145–149 (2016)Google Scholar
  7. 7.
    Fisteus, J.A., Pardo, A., García, N.F.: Grading multiple choice exams with low-cost and portable computer–vision techniques. J. Sci. Educ. Technol. 22(4), 560–571 (2013)Google Scholar
  8. 8.
    Ahmed, A.H, Afifi, M., Korashy, M., William, E.K, El-sattar, M.A., Hafez, Z.: OCR system for poor quality images using chain–code representation. In: The 1st International Conference on Advanced Intelligent System and Informatics, pp. 151–161 (2016)Google Scholar
  9. 9.
    Wang, T., Wu, D.J, Coates, A., Ng, A.Y: End-to-end text recognition with convolutional neural networks. In: 21st International Conference on Pattern Recognition (ICPR), pp. 3304–3308 (2012)Google Scholar
  10. 10.
    Jaderberg, M., Simonyan, K., Vedaldi, A., Zisserman, A.: Reading text in the wild with convolutional neural networks. Int. J. Comput. Vis. 116(1), 1–20 (2016)MathSciNetGoogle Scholar
  11. 11.
    Chinnasarn, K., Rangsanseri, Y.: An image-processing oriented optical mark reader. J. Soc. Photo Opt. Instrum. Eng. 3808, 702–709 (1999)Google Scholar
  12. 12.
    Nguyen, T.D., Manh, Q.H., Minh, P.B., Thanh, L.N., Hoang, T.M.: Efficient and reliable camera based multiple-choice test grading system. In: International Conference on Advanced Technologies for Communications (ATC), pp. 268–271 (2011)Google Scholar
  13. 13.
    Hussmann, S., Deng, P.W.: A high-speed optical mark reader hardware implementation at low cost using programmable logic. Real Time Imaging 11(1), 19–30 (2005)Google Scholar
  14. 14.
    Deng, H., Wang, F., Liang, B.: A low-cost OMR solution for educational applications. In: International Symposium on Parallel and Distributed Processing with Applications, pp. 967–970 (2008)Google Scholar
  15. 15.
    Levi, J.A., Solewicz, Y.A., Dvir, Y., Steinberg, Y.: Method of verifying declared identity in optical answer sheets. Soft Comput. 15(3), 461–468 (2011)Google Scholar
  16. 16.
    Chouvatut, V., Prathan, S.: The flexible and adaptive x-mark detection for the simple answer sheets. In: International Computer Science and Engineering Conference (ICSEC), pp. 433–439 (2014)Google Scholar
  17. 17.
    van der Maaten, L., Hinton, G.: Visualizing data using t-SNE. J. Mach. Learn. Res. 9, 2579–2605 (2008)zbMATHGoogle Scholar
  18. 18.
    Otsu, N.: A threshold selection method from gray-level histograms. IEEE Trans. Syst. Man Cybern. 9(1), 62–66 (1979)Google Scholar
  19. 19.
    Lopresti, D., Nagy, G., Smith, E.B.: A document analysis system for supporting electronic voting research. In: The Eighth IAPR International Workshop on Document Analysis Systems, pp. 167–174 (2008)Google Scholar
  20. 20.
    Barney-Smith, E.H., Nagy, G., Lopresti, D.: Mark detection from scanned ballots. In: Document Recognition and Retrieval XVI, vol. 7247, pp. 72470P-1–72470P-10 (2009)Google Scholar
  21. 21.
    Smith, E.H.B., Lopresti, D., Nagy, G., Wu, Z.: Towards improved paper-based election technology. In: International Conference on Document Analysis and Recognition (ICDAR), pp. 1255–1259 (2011)Google Scholar
  22. 22.
    Nagy, G., Lopresti, Dv: The role of document image analysis in trustworthy elections. In: Advances in Digital Document Processing and Retrieval, pp. 51–81 (2014)Google Scholar
  23. 23.
    Smith, A.M: Optical mark reading-making it easy for users. In: Proceedings of the 9th annual ACM SIGUCCS conference on User services, pp. 257–263 (1981)Google Scholar
  24. 24.
    Lopresti, D., Nagy, G., Smith, E.B.: A document analysis system for supporting electronic voting research. In: The Eighth International Workshop on Document Analysis Systems, pp. 167–174 (2008)Google Scholar
  25. 25.
    Sanguansat, P.: Robust and low-cost optical mark recognition for automated data entry. In: International Conference on Electrical Engineering/Electronics, Computer, Telecommunications and Information Technology (ECTI-CON), pp. 1–5 (2015)Google Scholar
  26. 26.
    China, R.T., de Assis Zampirolli, F., de Oliveira Neves, R.P., Quilici-Gonzalez, J.A.: An application for automatic multiple-choice test grading on android. Rev. Bras. Iniciaç. Cient. 3(2), 4–25 (2016)Google Scholar
  27. 27.
    Bay, H., Tuytelaars, T., Van Gool, L.: Surf: Speeded up robust features. In :ECCV, pp. 404–417 (2006)Google Scholar
  28. 28.
    Muja, M., Lowe, D.G: Fast matching of binary features. In: Ninth Conference on Computer and Robot Vision (CRV), pp. 404–410 (2012)Google Scholar
  29. 29.
    Torr, P.H.S., Murray, D.W.: The development and comparison of robust methods for estimating the fundamental matrix. Int. J. Comput. Vis. 24(3), 271–300 (1997)Google Scholar
  30. 30.
    Lewis, D.D: Naive (Bayes) at forty: the independence assumption in information retrieval. In: European Conference on Machine Learning, pp. 4–15 (1998)Google Scholar
  31. 31.
    Csurka, G., Dance, C., Fan, L., Willamowski, J., Bray, C.: Visual categorization with bags of keypoints. In: Workshop on Statistical Learning in Computer Vision, ECCV, pp. 1–2 (2004)Google Scholar
  32. 32.
    Krizhevsky, A., Sutskever, I., Hinton, G.E: Imagenet classification with deep convolutional neural networks. In: Advances in Neural Information Processing Systems, pp. 1097–1105 (2012)Google Scholar
  33. 33.
    Dalal, N., Triggs, B.: Histograms of oriented gradients for human detection. In: CVPR, pp. 886–893 (2005)Google Scholar
  34. 34.
    Vapnik, V.N., Vapnik, V.: Statistical Learning Theory. Wiley, New York (1998)zbMATHGoogle Scholar
  35. 35.
    Mikolajczyk, K., Schmid, C.: An affine invariant interest point detector. In: ECCV, pp. 128–142 (2002)Google Scholar
  36. 36.
    Duda, R.O., Hart, P.E., Stork, D.G.: Pattern Classification. Wiley, New York (2012)zbMATHGoogle Scholar
  37. 37.
    LeCun, Y., Bengio, Y., Hinton, G.: Deep learning. Nature 521(7553), 436–444 (2015)Google Scholar
  38. 38.
    Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition (2014). arXiv preprint arXiv:1409.1556
  39. 39.
    He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: CVPR, pp. 770–778 (2016)Google Scholar
  40. 40.
    Canziani, A., Paszke, A., Culurciello, E.: An analysis of deep neural network models for practical applications (2016). arXiv preprint arXiv:1605.07678
  41. 41.
    Murphy, K.P.: Machine Learning: A Probabilistic Perspective. MIT Press, Cambridge (2012)zbMATHGoogle Scholar
  42. 42.
    Storkey, A.: When training and test sets are different: characterizing learning transfer. In: Quiñonero-Candela, J., Sugiyama, M., Schwaighofer, A., Lawrence N.D. (eds.) Dataset Shift in Machine Learning, The MIT Press (2009)Google Scholar

Copyright information

© Springer-Verlag GmbH Germany, part of Springer Nature 2019

Authors and Affiliations

  1. 1.Electrical Engineering and Computer Science Department, Lassonde School of EngineeringYork UniversityTorontoCanada
  2. 2.Information Technology Department, Faculty of Computers and InformationAssiut University, AssiutAssiutEgypt
  3. 3.Computer Science Department, Faculty of Computers and InformationAssiut University, AssiutAssiutEgypt

Personalised recommendations