Advertisement

Adaptive Testing in Programming Courses Based on Educational Data Mining Techniques

  • Vladimir Ivančević
  • Marko Knežević
  • Bojan Pušić
  • Ivan Luković
Chapter
Part of the Studies in Computational Intelligence book series (SCI, volume 524)

Abstract

Designers of student tests, often teachers, primarily rely on their experience and subjective perception of students when selecting test items, while devoting little time to analyse factual data about both students and test items. As a practical solution to this common issue, we propose an approach to automatic test generation that acknowledges required areas of competence and matches the overall competence level of target students. The proposed approach, which is tailored to the testing practice in an introductory university course on programming, is based on the use of educational data mining. Data about students and test items are first evaluated using the predictive techniques of regression and classification, respectively, and then used to guide the test creation process. Besides a genetic algorithm that selects a test most suitable to the aforementioned criteria, we present a concept map of programming competencies and a method of estimating the test item difficulty.

Keywords

Programming competencies Concept maps Test creation Classification of test items Genetic algorithms 

Abbreviations

ARC

Area coverage

C1

Criterion1

C2

Criterion2

CAT

Computerized adaptive testing

CBA

Computer-based assessment

CR

Correct ratio

DF

Difficulty

DM

Data mining

EDM

Educational data mining

FIT

Fitness

FTS

Faculty of Technical Sciences

GA

Genetic algorithm

GC

Generation count

IC

Item count

IRT

Item response theory

M

Mean

MAX

Maximum

MDF

Mean difficulty

MF

Mean fitness

MGC

Max generation count

MH

Math

MIN

Minimum

MT

Mean completion time

NDF

Natural difficulty

NDFC

Natural difficulty category

OWL

Web ontology language

PAS

Past assignment

PLADS

Programming languages and data structures

PS

Population size

PTS

Past test

RA

Random approach

RDF

Resource description framework

SC

Student capacity

SCR

Student capacity rank

SD

Standard deviation

SDF

Standard deviation for fitness

SGC

Student group capacity

SK

Skewness

SPDF

Specified difficulty

SVM

Support vector machine

TPS

Test pool size

TS

Test

TSR

Test ratio

WRST

Wilcoxon rank sum test

Notes

Acknowledgments

The research was supported by the Ministry of Education, Science and Technological Development of the Republic of Serbia, Grant III-44010, Title: Intelligent Systems for Software Product Development and Business Support based on Models. The authors are very grateful to their colleagues from the Chair of Applied Computer Science at the Faculty of Technical Sciences (University of Novi Sad, Serbia), who have contributed to the Otisak testing system and participated in the organization of the Programming Language and Data Structures course, thus making the presented study possible.

References

  1. 1.
    Romero, C., Ventura, S.: Educational data mining: a review of the state of the art. IEEE Trans. Syst. Man Cybern. Part C Appl. Rev. 40(6), 601–618 (2010)CrossRefGoogle Scholar
  2. 2.
    Živanov, Ž., Rakić, P., Stričević, L., Pušić, B., Suvajdžin, Z., Hajduković, M.: Computer aided student examination. Info M 7(25), 45–53 (2008)Google Scholar
  3. 3.
    Wauters, K., Desmet, P., Noortgate, W.V.D.: Acquiring item difficulty estimates: a collaborative effort of data and judgment. In: Pechenizkiy, M., Calders, T., Conati, C., Ventura, S., Romero, C., Stamper, J. (eds.) 4th International Conference on Educational Data Mining, pp. 121–127. International Educational Data Mining Society, Eindhoven (2011)Google Scholar
  4. 4.
    Peña-Ayala, A., Sossa-Azuela, H., Cervantes-Pérez, F.: Predictive student model supported by fuzzy-causal knowledge and inference. Expert Syst. Appl. 39, 4690–4709 (2012)CrossRefGoogle Scholar
  5. 5.
    Holland, J.H.: Adaptation in Natural and Artificial Systems. MIT Press, Cambrigde (1992)Google Scholar
  6. 6.
    Barla, M., Bieliková, M., Ezzeddinne, A.B., Kramár, T., Šimko, M., Vozár, O.: On the impact of adaptive test question selection for learning efficiency. Comput. Educ. 55(2), 846–857 (2010)CrossRefGoogle Scholar
  7. 7.
    Feng, M., Heffernan, N.: Can we get better assessment from a tutoring system compared to traditional paper testing? Can we have our cake (Better Assessment) and eat it too (Student Learning During the Test)? In: Alven, V., Kay, J., Mostow, J. (eds.) Intelligent Tutoring Systems. LNCS, vol. 6095, pp. 309–311. Springer, Heidelberg (2010)Google Scholar
  8. 8.
    Thelwall, M.: Computer-based assessment: a versatile educational tool. Comput. Educ. 34(1), 37–49 (2000)CrossRefGoogle Scholar
  9. 9.
    Daly, C., Waldron, J.: Assessing the assessment of programming ability. ACM SIGCSE Bull. 36(1), 210–213 (2004)CrossRefGoogle Scholar
  10. 10.
    Douce, C., Livingstone, D., Orwell, J.: Automatic test-based assessment of programing: a review. J Educ. Resour. Comput. 5(3), 4 (2005)CrossRefGoogle Scholar
  11. 11.
    Ihantola, P., Ahoniemi, T., Karavirta, V., Seppälä, O.: Review of recent systems for automatic assessment of programming assignments. In: 10th Koli Calling International Conference on Computing Education Research, pp. 86–93. ACM, New York (2010)Google Scholar
  12. 12.
    Baker, F.B.: The Basics of Item Response Theory. ERIC Clearinghouse on Assessment and Evaluation, Washington (2001)Google Scholar
  13. 13.
    Sosnovsky, S., Gavrilova, T.: Development of educational ontology for C-programming. Int. J. Inf. Theor. Appl. 13, 303–308 (2006)Google Scholar
  14. 14.
    Sosnovsky, S.: C Programming Language Ontology, http://www.sis.pitt.edu/~paws/ont/c_programming.rdfs
  15. 15.
    Zhou, M., Xu, Y., Nesbit, J.C., Winne, P.H.: Sequential pattern analysis of learning logs: methodology and applications. In: Romero, C., Ventura, S., Pechenizkiy, M., Baker, R.S.J.d. (eds.). Handbook of Educational Data Mining, Chapman & Hall/CRC Data Mining and Knowledge Discovery Series, pp. 107–121. CRC Press, Boca Raton (2010)Google Scholar
  16. 16.
    Faculty of Technical Sciences in Novi Sad, Accreditation of the Study Programme; Computing and Control Engineering, http://www.ftn.uns.ac.rs/_data/planovi/2012/engleski/osnovne/ftn_e2.pdf
  17. 17.
    Rakić, P., Stričević, L., Živanov, Ž., Suvajdžin, Z., Hajduković, M.: Computer classroom: deployment and exploitation. Info M 6(21), 9–13 (2007)Google Scholar
  18. 18.
    Klyne, G., Carroll, J.J., McBride, B.: Resource description framework (RDF): concepts and abstract syntax. W3C Recommendation 10 (2004)Google Scholar
  19. 19.
    McGuinness, D.L., Van Harmelen, F.: OWL web ontology language overview. W3C Recommendation 10 (2004)Google Scholar
  20. 20.
    Motik, B., Patel-Schneider, P.F., Parsia, B., Bock, C., Fokoue, A., Haase, P., Smith, M.: OWL 2 Web ontology language: structural specification and functional-style syntax. W3C Recommendation 27, 17 (2009)Google Scholar
  21. 21.
    Grau, B.C., Horrocks, I., Motik, B., Parsia, B., Patel-Schneider, P., Sattler, U.: OWL 2: the next step for OWL. Web Semant.: Sci., Serv. Agents World Wide Web 6(4), 309–322 (2008)CrossRefGoogle Scholar
  22. 22.
    Falconer, S.: OntoGraf, http://protegewiki.stanford.edu/wiki/OntoGraf (2010)
  23. 23.
  24. 24.
    Krivov, S., Williams, R., Villa, F.: GrOWL: a tool for visualization and editing of OWL ontologies. Web Semant.: Sci., Serv. Agents World Wide Web 5(2), 54–57 (2007)CrossRefGoogle Scholar
  25. 25.
    Novak, J.D., Cañas, A.J.: The theory underlying concept maps and how to construct and use them. Technical report, Florida Institute for Human and Machine Cognition (2008)Google Scholar
  26. 26.
    Novak, J.D.: Learning, Creating, and Using Knowledge: Concept Maps as Facilitative Tools in Schools and Corporations. Taylor & Francis, New York (2010)Google Scholar
  27. 27.
    Ruiz-Primo, M.A., Shavelson, R.J.: Problems and issues in the use of concept maps in science assessment. J. Res. Sci. Teach. 33(6), 569–600 (1996)CrossRefGoogle Scholar
  28. 28.
    Cañas, A.J., Hill, G., Carff, R., Suri, N., Lott, J., Eskridge, T., Carvajal, R.: CmapTools: a knowledge modeling and sharing environment. In: Concept maps: Theory, Methodology, Technology 1st International Conference on Concept Mapping, vol. 1, pp. 125–133. Universidad Pública de Navarra, Pamplona (2004)Google Scholar
  29. 29.
    Bivand, R.: ClassInt: Choose Univariate Class Intervals. R package version 0.1-19 (2012), http://CRAN.R-project.org/package=classInt
  30. 30.
    R Core Team.: R: A Language and Environment for Statistical Computing, Manual. R Foundation for Statistical Computing (2013)Google Scholar
  31. 31.
    Karatzoglou, A., Smola, A., Hornik, K.: Achim Zeileis, A.: Kernlab—an S4 package for kernel methods in R. J. Stat. Softw. 11(9), 1–20 (2004)Google Scholar

Copyright information

© Springer International Publishing Switzerland 2014

Authors and Affiliations

  • Vladimir Ivančević
    • 1
  • Marko Knežević
    • 1
  • Bojan Pušić
    • 1
  • Ivan Luković
    • 1
  1. 1.University of Novi Sad, Faculty of Technical SciencesNovi SadSerbia

Personalised recommendations