Skip to main content
SpringerLink
Log in

The achievement of higher flexibility in multiple-choice-based tests using image classification techniques

  • Original Paper
  • Published:
International Journal on Document Analysis and Recognition (IJDAR) Aims and scope Submit manuscript

Abstract

In spite of the high accuracy of the existing optical mark reading (OMR) systems and devices, a few restrictions remain existent. In this work, we aim to reduce the restrictions of multiple-choice questions (MCQ) within tests. We use an image registration technique to extract the answer boxes from answer sheets. Unlike other systems that rely on simple image processing steps to recognize the extracted answer boxes, we address the problem from another perspective by training a machine learning classifier to recognize the class of each answer box (i.e., confirmed, crossed out or blank answer). This gives us the ability to deal with a variety of shading and mark patterns, and distinguish between chosen (i.e., confirmed) and canceled answers (i.e., crossed out). All existing machine learning techniques require a large number of examples in order to train a model for classification; therefore, we present a dataset including six real MCQ assessments with different answer sheet templates. We evaluate two strategies of classification: a straightforward approach and a two-stage classifier approach. We test two handcrafted feature methods and a convolutional neural network. At the end, we present an easy-to-use graphical user interface of the proposed system. Compared with existing OMR systems, the proposed system has the least constraints and achieves a high accuracy. We believe that the presented work will further direct the development of OMR systems toward reducing the restrictions of the MCQ tests.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

Notes

  1. (1) MCTest Corrector. (2) ZipGrade. (3) MCScanner. (4) Exam Reader (ER).

References

  1. Gronlund, N.E.: Assessment of student achievement. ERIC (1998)

  2. McCoubrie, P.: Improving the fairness of multiple-choice questions: a literature review. Med. Teach. 26(8), 709–712 (2004)

    Article  Google Scholar 

  3. Aldabe, I., Maritxalar, M.: Semantic similarity measures for the generation of science tests in basque. IEEE Trans. Learn. Technol. 7(4), 375–387 (2014)

    Article  Google Scholar 

  4. Liu, M., Rus, V., Liu, L.: Automatic chinese multiple choice question generation using mixed similarity strategy. IEEE Trans. Learn. Technol. 11(2), 193–202 (2017)

    Article  Google Scholar 

  5. Spadaccini, A., Rizzo, V.: A multiple-choice test recognition system based on the Gamera framework (2011). arXiv preprint arXiv:1105.3834

  6. Chai, D.: Automated marking of printed multiple choice answer sheets. In: IEEE International Conference on Teaching, Assessment, and Learning for Engineering (TALE), pp. 145–149 (2016)

  7. Fisteus, J.A., Pardo, A., García, N.F.: Grading multiple choice exams with low-cost and portable computer–vision techniques. J. Sci. Educ. Technol. 22(4), 560–571 (2013)

    Article  Google Scholar 

  8. Ahmed, A.H, Afifi, M., Korashy, M., William, E.K, El-sattar, M.A., Hafez, Z.: OCR system for poor quality images using chain–code representation. In: The 1st International Conference on Advanced Intelligent System and Informatics, pp. 151–161 (2016)

  9. Wang, T., Wu, D.J, Coates, A., Ng, A.Y: End-to-end text recognition with convolutional neural networks. In: 21st International Conference on Pattern Recognition (ICPR), pp. 3304–3308 (2012)

  10. Jaderberg, M., Simonyan, K., Vedaldi, A., Zisserman, A.: Reading text in the wild with convolutional neural networks. Int. J. Comput. Vis. 116(1), 1–20 (2016)

    Article  MathSciNet  Google Scholar 

  11. Chinnasarn, K., Rangsanseri, Y.: An image-processing oriented optical mark reader. J. Soc. Photo Opt. Instrum. Eng. 3808, 702–709 (1999)

    Google Scholar 

  12. Nguyen, T.D., Manh, Q.H., Minh, P.B., Thanh, L.N., Hoang, T.M.: Efficient and reliable camera based multiple-choice test grading system. In: International Conference on Advanced Technologies for Communications (ATC), pp. 268–271 (2011)

  13. Hussmann, S., Deng, P.W.: A high-speed optical mark reader hardware implementation at low cost using programmable logic. Real Time Imaging 11(1), 19–30 (2005)

    Article  Google Scholar 

  14. Deng, H., Wang, F., Liang, B.: A low-cost OMR solution for educational applications. In: International Symposium on Parallel and Distributed Processing with Applications, pp. 967–970 (2008)

  15. Levi, J.A., Solewicz, Y.A., Dvir, Y., Steinberg, Y.: Method of verifying declared identity in optical answer sheets. Soft Comput. 15(3), 461–468 (2011)

    Article  Google Scholar 

  16. Chouvatut, V., Prathan, S.: The flexible and adaptive x-mark detection for the simple answer sheets. In: International Computer Science and Engineering Conference (ICSEC), pp. 433–439 (2014)

  17. van der Maaten, L., Hinton, G.: Visualizing data using t-SNE. J. Mach. Learn. Res. 9, 2579–2605 (2008)

    MATH  Google Scholar 

  18. Otsu, N.: A threshold selection method from gray-level histograms. IEEE Trans. Syst. Man Cybern. 9(1), 62–66 (1979)

    Article  Google Scholar 

  19. Lopresti, D., Nagy, G., Smith, E.B.: A document analysis system for supporting electronic voting research. In: The Eighth IAPR International Workshop on Document Analysis Systems, pp. 167–174 (2008)

  20. Barney-Smith, E.H., Nagy, G., Lopresti, D.: Mark detection from scanned ballots. In: Document Recognition and Retrieval XVI, vol. 7247, pp. 72470P-1–72470P-10 (2009)

  21. Smith, E.H.B., Lopresti, D., Nagy, G., Wu, Z.: Towards improved paper-based election technology. In: International Conference on Document Analysis and Recognition (ICDAR), pp. 1255–1259 (2011)

  22. Nagy, G., Lopresti, Dv: The role of document image analysis in trustworthy elections. In: Advances in Digital Document Processing and Retrieval, pp. 51–81 (2014)

  23. Smith, A.M: Optical mark reading-making it easy for users. In: Proceedings of the 9th annual ACM SIGUCCS conference on User services, pp. 257–263 (1981)

  24. Lopresti, D., Nagy, G., Smith, E.B.: A document analysis system for supporting electronic voting research. In: The Eighth International Workshop on Document Analysis Systems, pp. 167–174 (2008)

  25. Sanguansat, P.: Robust and low-cost optical mark recognition for automated data entry. In: International Conference on Electrical Engineering/Electronics, Computer, Telecommunications and Information Technology (ECTI-CON), pp. 1–5 (2015)

  26. China, R.T., de Assis Zampirolli, F., de Oliveira Neves, R.P., Quilici-Gonzalez, J.A.: An application for automatic multiple-choice test grading on android. Rev. Bras. Iniciaç. Cient. 3(2), 4–25 (2016)

    Google Scholar 

  27. Bay, H., Tuytelaars, T., Van Gool, L.: Surf: Speeded up robust features. In :ECCV, pp. 404–417 (2006)

  28. Muja, M., Lowe, D.G: Fast matching of binary features. In: Ninth Conference on Computer and Robot Vision (CRV), pp. 404–410 (2012)

  29. Torr, P.H.S., Murray, D.W.: The development and comparison of robust methods for estimating the fundamental matrix. Int. J. Comput. Vis. 24(3), 271–300 (1997)

    Article  Google Scholar 

  30. Lewis, D.D: Naive (Bayes) at forty: the independence assumption in information retrieval. In: European Conference on Machine Learning, pp. 4–15 (1998)

  31. Csurka, G., Dance, C., Fan, L., Willamowski, J., Bray, C.: Visual categorization with bags of keypoints. In: Workshop on Statistical Learning in Computer Vision, ECCV, pp. 1–2 (2004)

  32. Krizhevsky, A., Sutskever, I., Hinton, G.E: Imagenet classification with deep convolutional neural networks. In: Advances in Neural Information Processing Systems, pp. 1097–1105 (2012)

  33. Dalal, N., Triggs, B.: Histograms of oriented gradients for human detection. In: CVPR, pp. 886–893 (2005)

  34. Vapnik, V.N., Vapnik, V.: Statistical Learning Theory. Wiley, New York (1998)

    MATH  Google Scholar 

  35. Mikolajczyk, K., Schmid, C.: An affine invariant interest point detector. In: ECCV, pp. 128–142 (2002)

  36. Duda, R.O., Hart, P.E., Stork, D.G.: Pattern Classification. Wiley, New York (2012)

    MATH  Google Scholar 

  37. LeCun, Y., Bengio, Y., Hinton, G.: Deep learning. Nature 521(7553), 436–444 (2015)

    Article  Google Scholar 

  38. Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition (2014). arXiv preprint arXiv:1409.1556

  39. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: CVPR, pp. 770–778 (2016)

  40. Canziani, A., Paszke, A., Culurciello, E.: An analysis of deep neural network models for practical applications (2016). arXiv preprint arXiv:1605.07678

  41. Murphy, K.P.: Machine Learning: A Probabilistic Perspective. MIT Press, Cambridge (2012)

    MATH  Google Scholar 

  42. Storkey, A.: When training and test sets are different: characterizing learning transfer. In: Quiñonero-Candela, J., Sugiyama, M., Schwaighofer, A., Lawrence N.D. (eds.) Dataset Shift in Machine Learning, The MIT Press (2009)

Download references

Author information

Authors and Affiliations

  1. Electrical Engineering and Computer Science Department, Lassonde School of Engineering, York University, Toronto, ON, Canada

    Mahmoud Afifi

  2. Information Technology Department, Faculty of Computers and Information, Assiut University, Assiut, Assiut, Egypt

    Mahmoud Afifi

  3. Computer Science Department, Faculty of Computers and Information, Assiut University, Assiut, Assiut, Egypt

    Khaled F. Hussain

Authors
  1. Mahmoud Afifi
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Khaled F. Hussain
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Mahmoud Afifi.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Afifi, M., Hussain, K.F. The achievement of higher flexibility in multiple-choice-based tests using image classification techniques. IJDAR 22, 127–142 (2019). https://doi.org/10.1007/s10032-019-00322-3

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10032-019-00322-3

Keywords

Search

Navigation