3.1 University and Course Management
While Automata Tutor can be used for independent online practice, one of the main advantages is its infrastructure for large university courses. To this end, it is organized in courses. A course is created and supervised by one or more teachers. Together, they can create, test and edit exercises. The students cannot immediately see the problems, but only after the teachers have decided to pose them. This involves setting the maximum number of points, the number of allowed attempts as well as the start and end date.
To use Automata Tutor, students must have an account. One can either register by email or, in case the university supports it, login with an external login service like LDAP or Oauth. When using the login service of their university, teachers get a certified mapping from users to students and enabling teachers to use Automata Tutor v3 for grading homework or exams.
Students can enroll in a course using a password. Enrolled students see all posed problems and can solve them (using the allowed number of attempts). The final grade can be accessed by the teachers in the grade overview.
3.2 New Problem Types
In this section, we list the problem types newly added to Automata Tutor v3. They are all part of the course
[10] and a detailed description of each problem can be found in
[5, Appendix A], including the basic theoretical concept, how a student can solve such a problem, what a teacher has to provide to create a problem, the idea of the grading algorithm, and what feedback the tool gives.
-
RE/CFG/PDA Words: Finding words in or not in the language of a regular expression, context free grammar or pushdown automaton.
-
RE/CFG/PDA Construction: Given a description of a language, construct a regular expression, context free grammar or pushdown automaton.
-
RE to NFA: Given a regular expression, construct a nondeterministic-finite automaton.
-
Myhill-Nerode Equivalence Classes: There are two subtypes: either, given a regular expression and two words, find out whether they are equivalent w.r.t. the language, or, given a regular expression and a word, find further words in the same equivalence class.
-
Pumping-Lemma Game: Given a language, the student has to guess whether it is regular or not and then plays the game as one of the quantifiers.
-
Find Derivation: Given a context free grammar and a word, the student has to specify a derivation of that word.
-
CNF: Given a context free grammar, the student has to transform it into Chomsky Normal Form.
-
CYK: Given a context free grammar in CNF and a word, the student has to decide whether the word is in the language of the grammar by using the Cocke–Younger–Kasami algorithm.
-
While to TM: Given a while-program (a Turing-complete programming language with very restricted syntax), construct a (multi-tape) Turing machine with the same input-output behaviour.
3.3 Automatic Problem Generation
Automatic Problem Generation (APG) allows one to generate new exercises of a requested difficulty level and problem type. This allows students to practice independently and supports teachers when creating new exercises. While APG is currently implemented for four CFG problem types and for the problem type “While to TM”, it can be easily extended to other problem types by providing the following components:
-
Procedure for generating exercises at random either from given basic building blocks or from scratch.
-
A “quality” metric \( qual (E)\) for assessing the quality of the generated exercise E, ranging from trivial or infeasible to realistic.
-
A “difficulty” metric \( diff (E)\) for assessing the difficulty of E.
Given these components, Automata Tutor generates a new problem with a given minimum difficulty \(d_{\min }\) and maximum difficulty \(d_{\max }\) as follows. Firstly, 100 random exercises are generated. Secondly, Automata Tutor chooses exercises E with the best quality such that \(d_{\min } \le diff (E) \le d_{\max }\).
Concretely, for the CFG problem types, CFGs with random productions are generated and sanitized. Resulting CFGs that do not accept any words or have too few productions are excluded using the quality metric. The difficulty metric always depends on the number of productions; additionally, depending on the exact problem type, further criteria are taken into account.
For the problem type “While to TM” we use an approach similar to the one suggested in existing tools for automatic problem generation
[15, 18]: We handcrafted several base programs which are of different difficulty level. In the generation process, the syntax tree of such a base program is abstracted and certain modifying operations are executed; these change the program without affecting the difficulty too much. E.g. we choose different variables, switch the order of if-else branches or change arithmetic operators. Then several programs are generated and those of bad quality are filtered out. A program is of bad quality if its language is trivially small or if it contains infinite loops; since detecting these properties is undecidable, we employ heuristics such as checking that the loops terminate for all inputs up to a certain size with a certain timeout.