Skip to main content
View expanded cover

International Conference on Artificial Intelligence in Education

AIED 2021: Artificial Intelligence in Education pp 433–445Cite as

Towards Blooms Taxonomy Classification Without Labels

Part of the Lecture Notes in Computer Science book series (LNAI,volume 12748)

Abstract

In this work, we explore weakly supervised machine learning for classifying questions into distinct Bloom’s Taxonomy levels. Bloom’s levels provide important information that guides teachers and adaptive learning algorithms in selecting appropriate questions for their students. However, manually providing Bloom labels is expensive and labor-intensive, which motivates a machine learning approach. Current automated Bloom’s level classification methods employ supervised learning that relies on large labeled datasets that are difficult and costly to construct. In this paper, we propose a weakly supervised learning method that performs binary Bloom’s level labeling without any a priori known Bloom’s taxonomy labels. The key idea behind BLACBOARD (for Bloom’s Level clAssifiCation Based On weAkly supeRviseD learning) is to appropriately incorporate human domain knowledge into the modeling process to produce a weakly labeled dataset on which discriminative models can then be trained. We compare BLACBOARD to fully supervised learning methods and show that it achieves little to no performance compromise while using entirely unlabeled data.

Keywords

  • Bloom’s level classification
  • Weakly supervised learning

This is a preview of subscription content, access via your institution.

Chapter
USD   29.95
Price excludes VAT (USA)
  • DOI: 10.1007/978-3-030-78292-4_35
  • Chapter length: 13 pages
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
eBook
USD   79.99
Price excludes VAT (USA)
  • ISBN: 978-3-030-78292-4
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
Softcover Book
USD   99.99
Price excludes VAT (USA)
Learn about institutional subscriptions
Fig. 1.
Fig. 2.
Fig. 3.
Fig. 4.

Notes

  1. 1.

    A demonstration and associated code of BLACBOARD are available at https://github.com/manningkyle304/edu-research-demo.

  2. 2.

    https://openstax.org.

  3. 3.

    We also experimented with other featurization methods, but the results were similar to TF-IDF. We thus use TF-IDF for all experiments in this paper.

References

  1. Swart, A.J.: Evaluation of final examination papers in engineering: a case study using Bloom’s taxonomy. IEEE Trans. Educ. 53(2), 257–264 (2010)

    CrossRef  Google Scholar 

  2. Blumberg, P.: Maximizing learning through course alignment and experience with different types of knowledge. Innovative High. Educ. 34(2), 93–103 (2009)

    CrossRef  Google Scholar 

  3. Bloom, B., Engelhart, M., Furst, E., Hill, W., Krathwohl, D.: Taxonomy of educational objectives: the classification of educational goals. Handbook 1: cognitive domain. David McKay, New York (1956)

    Google Scholar 

  4. Krathwohl, D.R.: A revision of bloom’s taxonomy : an overview. Theory Pract. 41(4), 212–218 (2002)

    CrossRef  Google Scholar 

  5. Abduljabbar, D.A., Omar, N.: Exam questions classification based on Bloom’s taxonomy cognitive level using classifiers combination. J. Theor. Appl. Inf. Technol. 78(3), 447 (2015)

    Google Scholar 

  6. Osadi, K., Fernando, M., Welgama, W., et al.: Ensemble classifier based approach for classification of examination questions into Bloom’s taxonomy cognitive levels. Int. J. Comput. Appl. 162(4), 1–6 (2017)

    Google Scholar 

  7. Yahya, A.A., Osman, A.: Automatic classification of questions into Bloom’s cognitive levels using support vector machines. In: International Arab Conference on Information Technology (2011)

    Google Scholar 

  8. Jayakodi, K., Bandara, M., Perera, I.: An automatic classifier for exam questions in engineering: a process for Bloom’s taxonomy. In: IEEE International Conference on Teaching, Assessment, and Learning for Engineering (TALE), pp. 195–202 (2015)

    Google Scholar 

  9. Mohamed, O.J., Zakar, N.A., Alshaikhdeeb, B.: A combination method of syntactic and semantic approaches for classifying examination questions into bloom’s taxonomy cognitive. J. Eng. Sci. Technol. 14(2), 935–950 (2019)

    Google Scholar 

  10. Mohammed, M., Omar, N.: Question classification based on Bloom’s taxonomy using enhanced TF-IDF. Int. J. Adv. Sci. Eng. Inf. Technol. 8, 1679–1685 (2018)

    CrossRef  Google Scholar 

  11. Osman, A., Yahya, A.A.: Classifications of exam questions using natural language syntatic features: a case study based on Bloom’s taxonomy. In: Proceedings of the International Arab Conference on Quality Assurance in Higher Education (2016)

    Google Scholar 

  12. Sangodiah, A., Ahmad, R., Ahmad, W.F.W.: A review in feature extraction approach in question classification using support vector machine. In: IEEE International Conference on Control System, Computing and Engineering, pp. 536–541 (2014)

    Google Scholar 

  13. Pota, M., Esposito, M., De Pietro, G.: A forward-selection algorithm for SVM-based question classification in cognitive systems. In: De Pietro, G., Gallo, L., Howlett, R.J., Jain, L.C. (eds.) Intelligent Interactive Multimedia Systems and Services 2016. SIST, vol. 55, pp. 587–598. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-39345-2_52

    CrossRef  Google Scholar 

  14. Supriyanto, C., Yusof, N., Nurhadiono, B., et al.: Two-level feature selection for Naive Bayes with kernel density estimation in question classification based on Bloom’s cognitive levels. In: International Conference on Information Technology and Electrical Engineering, pp. 237–241 (2013)

    Google Scholar 

  15. Diab, S., Sartawi, B.: Classification of questions and learning outcome statements (LOS) into Blooms taxonomy (BT) by similarity measurements towards extracting of learning outcome from learning material. arXiv preprint (2017)

    Google Scholar 

  16. Zoller, U.: Are lecture and learning compatible? Maybe for LOCS: unlikely for HOCS. J. Chem. Edu. 70(3), 195 (1993)

    CrossRef  Google Scholar 

  17. Karamustafaoğlu, S., Sevim, S., Karamustafaoğlu, O., Cepni, S.: Analysis of Turkish high-school chemistry-examination questions according to bloom’s taxonomy. Chem. Educ. Res. Pract. 4(1), 25–30 (2003)

    CrossRef  Google Scholar 

  18. Crowe, A., Dirks, C., Wenderoth, M.P.: Biology in Bloom: implementing bloom’s taxonomy to enhance student learning in biology. CBE-Life Sci. Educ. 7(4), 368–381 (2008)

    CrossRef  Google Scholar 

  19. Baker, E.L.: Developing comprehensive assessments of higher order thinking. Assess. High. Order Think. Math. 7, 20 (1990)

    Google Scholar 

  20. Herrington, J., Oliver, R.: Using situated learning and multimedia to investigate higher-order thinking. J. Interact. Learn. Res. 10(1), 3–24 (1999)

    Google Scholar 

  21. Ratner, A., et al.: Snorkel: rapid training data creation with weak supervision. VLDB J. 29(2), 709–730 (2019). https://doi.org/10.1007/s00778-019-00552-1

  22. Varma, P., Ré, C.: Snuba: automating weak supervision to label training data. Proc. VLDB Endow. 12(3), 223–236 (2018)

    CrossRef  Google Scholar 

  23. Varma, P., He, B.D., Bajaj, P., Khandwala, N., Banerjee, I., Rubin, D., Ré, C.: Inferring generative model structure with static analysis. Adv. Neural. Inf. Process. Syst. 30, 240–250 (2017)

    Google Scholar 

  24. Bach, S.H., He, B., Ratner, A., Ré, C.: Learning the structure of generative models without labeled data. In: Proceedings of the International Conference on Machine Learning, pp. 273–282 (2017)

    Google Scholar 

  25. Fries, J.A., et al.: Trove: ontology-driven weak supervision for medical entity classification. arXiv e-prints, August 2020

    Google Scholar 

  26. Fries, J.A., et al.: Weakly supervised classification of aortic valve malformations using unlabeled cardiac MRI sequences. Nature Commun. 10(1), 3111 (2019)

    CrossRef  Google Scholar 

  27. Callahan, A., Fries, J.A., Ré, C., Huddleston, J.I., Giori, N.J., Delp, S., Shah, N.H.: Medical device surveillance with electronic health records. NPJ Digit. Med. 2(1), 94 (2019)

    Google Scholar 

  28. Kuleshov, V., et al.: A machine-compiled database of genome-wide association studies. Nature Commun. 10(1), 3341 (2019)

    CrossRef  Google Scholar 

  29. Datta, D., Phillips, M., Chiu, J., Watson, G.S., Bywater, J.P., Barnes, L., Brown, D.: Improving classification through Weak supervision in context-specific conversational agent development for teacher education. arXiv e-prints, October 2020

    Google Scholar 

  30. Salton, G., McGill, M.J.: Introduction to Modern Information Retrieval. McGraw-Hill Inc, New York, USA (1986)

    MATH  Google Scholar 

  31. Hastie, T., Tibshirani, R., Friedman, J.: The Elements of Statistical Learning. Springer New York Inc., New York, USA (2001)

    CrossRef  Google Scholar 

  32. Bishop, C.M.: Pattern Recognition and Machine Learning (information science and statistics). Springer-Verlag, Berlin, Heidelberg (2006)

    MATH  Google Scholar 

  33. Wang, Y., Zheng, J., Liu, Q., Zhao, Z., Xiao, J., Zhuang, Y.: Weak supervision enhanced generative network for question generation. arXiv preprint (2019)

    Google Scholar 

  34. Datta, D., et al.: Improving classification through weak supervision in context-specific conversational agent development for teacher education. arXiv preprint (2020)

    Google Scholar 

  35. Ramesh, A., Kumar, S.H., Foulds, J., Getoor, L.: Weakly supervised models of aspect-sentiment for online course discussion forums. In: Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing, pp. 74–83 (2015)

    Google Scholar 

  36. Qiu, Y., Wang, Y., Jin, X., Zhang, K.: Stepwise reasoning for multi-relation question answering over knowledge graph with weak supervision. In: Proceedings of the 13th International Conference on Web Search and Data Mining, pp. 474–482 (2020)

    Google Scholar 

Download references

Acknowledgements

This work was supported by NSF grants 1842378 and 1937134 and by ONR grant N0014-20-1-2534. We thank Prof. Colleen Countryman (Ithaca College), Prof. Lauren Rast (The University of Alabama at Birmingham), Joyce Spangler (Six Red Marbles), and Andrew Giannakakis (OpenStax) for helpful discussions on the Labeling Functions. Thanks to Fred Sala for insights on WSL. Thanks to anonymous reviewers and CJ Barberan for suggestions on the manuscript.

Author information

Authors and Affiliations

  1. Rice University, Houston, TX, USA

    Zichao Wang, Kyle Manning & Richard G. Baraniuk

  2. OpenStax, Houston, TX, USA

    Debshila Basu Mallick & Richard G. Baraniuk

Authors
  1. Zichao Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Kyle Manning
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Debshila Basu Mallick
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Richard G. Baraniuk
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Zichao Wang .

Editor information

Editors and Affiliations

  1. Technion – Israel Institute of Technology, Haifa, Israel

    Ido Roll

  2. Arizona State University, Tempe, AZ, USA

    Danielle McNamara

  3. Utrecht University, Utrecht, The Netherlands

    Sergey Sosnovsky

  4. London Knowledge Lab, London, UK

    Rose Luckin

  5. University of Leeds, Leeds, UK

    Vania Dimitrova

Rights and permissions

Reprints and Permissions

Copyright information

© 2021 Springer Nature Switzerland AG

About this paper

Verify currency and authenticity via CrossMark

Cite this paper

Wang, Z., Manning, K., Mallick, D.B., Baraniuk, R.G. (2021). Towards Blooms Taxonomy Classification Without Labels. In: Roll, I., McNamara, D., Sosnovsky, S., Luckin, R., Dimitrova, V. (eds) Artificial Intelligence in Education. AIED 2021. Lecture Notes in Computer Science(), vol 12748. Springer, Cham. https://doi.org/10.1007/978-3-030-78292-4_35

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-78292-4_35

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-78291-7

  • Online ISBN: 978-3-030-78292-4

  • eBook Packages: Computer ScienceComputer Science (R0)