Keywords

1 Introduction

Augmented Cognition (AugCog) focuses on the use of technology to enhance cognitive function across a set of users during the execution of a particular problem or task. From the perspective of building Intelligent Tutoring Systems (ITS) to support personalized instruction, AugCog is focused on the use of technology to enhance computer-based learning experiences by augmenting instruction to optimize outcomes. These instructional augmentations are based on principles established within the tutoring literature on how people learn and the human computer-interaction literature on how people apply technology to perform tasks and solve problems. These principles are then used to inform strategies that adapt learning experiences around defined state representations of a given learner.

The state representations of interest are based on both historical and realtime captured data, including but not limited to: (1) performance-based metrics inferred upon through user inputs and actions in a learning environment; (2) affective and cognitive state-based metrics derived from interaction sequences and the use of physiological and behavioral sensors; (3) and trait-based metrics linked with historical data found to be more persistent in terms of its effect on learning, such as an individual’s prior knowledge, motivation, and grit [1, 2]. While a large portion of the research and development across ITSs is dedicated to assessment techniques for measuring the state spaces listed above, an area of research that requires considerably more attention to optimize ITS development is the field of pedagogical modeling and instructional management.

In this paper, we will discuss the tools and methods constructed by the U.S. Army Research Laboratory’s Human Research and Engineering Directorate to support instructional management focused research within the ITS and adaptive training communities of practice. The specific tools of interest are the Generalized Intelligent Framework for Tutoring (GIFT) and the Engine for Management of Adaptive Pedagogy (EMAP). An in-depth review of the EMAP and how it supports pedagogical principles grounded in AugCog theory and practice will be provided. The types of augmentations supported by the EMAP will be presented along with the authoring tools developed to enable the implementation of adaptive instruction for any technology-driven educational product.

2 Adaptation in the Generalized Intelligent Framework for Tutoring

In an effort to promote the advancement of building and researching ITSs and educational technologies, GIFT was developed as a modular, service-oriented architecture built upon standardized tools and methods to support personalized instruction across an array of computer-based training applications [3]. With an emphasis on personalization, GIFT requires a domain-agnostic pedagogical structure to support the authoring and delivery of varying instructional techniques that can be executed within any learning environment [4]. From an implementation standpoint, this pedagogical framework requires authoring functions that enable a system developer to configure strategy types to specific domain calls and a runtime component that supports strategy execution in real-time based on learner model inputs. During an initial market survey to address this requirement, it was recognized that there is no open-source solution that provides adaptive course-flow capabilities based on both individual differences and historical and real-time performance metrics.

To facilitate the recognized need, we developed the EMAP. The EMAP is composed of a set of tools and methods housed within GIFT. It is designed to enhance the architecture by extending the authoring capabilities to support highly adaptive courseware materials. In the following sections we will review the multiple components that make up the EMAP and how the various tools are used to build customized learning experiences based on learner data linked with the varying state spaces described above. Further, we will explore how the EMAP can be leveraged by the Human Computer Interaction (HCI) community to support robust research associated with education and training with technology.

2.1 Augmenting Instructional Practice in GIFT Using the EMAP

The primary functions made available in the EMAP to support AugCog based practices are focused on the implementation of sound instructional design principles that account for individual differences related with learning, retention, and transfer [5]. The goal is to select and adapt strategies so as to optimize a learning experience to promote efficient training outcomes. The variables of interest are defined in terms of an individual’s knowledge, skills, and abilities (KSAs) associated with a domain. The established KSAs serve as moderators in determining what information to deliver and how best to guide the learning event in relation to sequence of instruction and feedback [2].

The pedagogical framework is intended to support the various interactions that occur when an individual is learning a new topic or skill. This includes the design of actionable logic that determines for every domain of instruction (1) what material to present, (2) how best to assess knowledge and skill linked to a task and set of competencies, and (3) what guidance and feedback practices most reliably impact subsequent performance outcomes. The goal is to establish a data-driven pedagogical model that adapts strategy execution based on the type of learner the interventions moderate most appropriately. To make this a manageable problem space, the first steps associated with the EMAP design was identifying a generalized framework to support its function that is grounded in sound instructional system design (ISD) theory.

Following an extensive review of literature focused on ISD and pedagogical strategy implementations within computer-based learning environments, we selected David Merrill’s Component Display Theory (CDT) to serve as the guiding framework to formalize the EMAP requirements around [4]. The CDT was originally constructed as a way to simplify theories and models of instruction around a set of core interactions a learner engages in when mastering a new domain [7]. For our efforts, the CDT served as a simplistic framework to organize instructional strategy research that ultimately informs the development of the EMAP’s domain independent structure.

The CDT breaks learning down into four fundamental forms of instruction that focus on content and presentation modes. These instructional conditions, known as CDT’s Presentation Forms, provide the basic building blocks for the instructional strategies present in the EMAP. CDT indicates two paths when it comes to content: it can be presented (expository) or the instructor asks the student to remember or use the content (inquisitory). The content can represent a general case (generality) or it can represent a specific case (instance). Therefore, instruction can be divided into four categories: Expository generality – present a general case (Rule); Expository instance – present a specific case (Example); Inquisitory generality – ask the student to remember or apply the general case (Recall); and Inquisitory instance – ask the student to remember or apply the specific case (Practice).

These four categories can be used as high-level descriptors to associate training content with, assuming each category applies different pedagogical practices inherent to the learning process. Therefore, instructional strategies can be explicitly defined and categorized within each component of the CDT. This association allows an instructional designer to understand what a piece of content is intended to provide in a lesson context (i.e., this video provides an example for enabling objective x), and further instructional strategies can be defined to inform when this piece of material is most suitable for use.

With the CDT serving as a generalized pedagogical framework for course construction, the next task was building empirically driven condition statements for the delivery of instructional materials that account for individual differences across a set of learners. The initial construction was facilitated through the creation of an algorithm in the form of a decision tree that informs adaptation based on general learner characteristics [4]. Specifically, the decision tree informs the selection of instructional strategies based on known information about the learner (e.g., learner motivation, learning style, previous experience, etc.) and the logic is established using GIFT’s Pedagogical Configuration Authoring Tool (PCAT), which will be described in detail below. The resulting strategies were identified through an extensive literature review of empirically based research in an attempt to produce a list of commonly applied techniques found to reliably impact learning outcomes.

While many strategies were investigated across the learning sciences community, and many themes recognized, the studies often were limited to single domains and learning environments. The summary of this work can be seen across two publications [2, 4]. As a result, establishing a truly generalized pedagogical manager requires future research in an attempt to study the effect of strategies on outcomes and how those specific instructional techniques transfer to different learning environments. In this vein, the EMAP was designed to support both the creation of customized learning applications in addition to providing tools to support robust instructional strategy focused research. In the next section, we highlight the various authoring tools developed to support EMAP functions and how the tools are applied in AugCog based research formats.

3 GIFT’s Engine for Management of Adaptive Pedagogy

The goal of the EMAP is to provide a means for instructors and course developers to build highly individualized lesson interactions that manage the student experience based on configured learning paths and real-time performance. It is intended to serve as an automated lesson manager when a learning event is executed in a self-regulated environment. It is currently being built to manage the delivery of information and course materials, perform assessment related practices to determine an individual’s knowledge and skill for a set of concepts linked with a domain, manage the delivery of real-time guidance and interventions found to support learning outcomes, and directs remediation paths based on reported outcomes.

Identified augcog based instructional principles managed by the EMAP are configured by two levels of operational data: (1) enumerated metadata values used to describe content and interaction characteristics on a domain-agnostic level and (2) learner attribute data that categorizes education-centric individual differences on KSAs across a sample of learners. The learner attribute data is used as a moderator in selecting appropriate metadata values when adapting a lesson for a given learners knowledge and abilities. This structure is critical for maintaining GIFT’s generalizability across disparate domains, while providing a means to effectively support adaptive practices through descriptive tags that characterize lesson materials and pedagogical strategy instantiations. To further enhance standardization, our current enumerated metadata values are mapped with standards outlined in IEEE’s Learning Object Metadata [8]. This is intended to support reuse of existing content and for easy integration of external materials never implemented using GIFT. Next, we present the authoring tools and processes associated with building an EMAP driven lesson.

3.1 Configuring the EMAP Using GIFT Authoring Tools

A number of authoring processes have been established to support the various pedagogical functions made available by the EMAP. In this subsection we will introduce the environments currently in place to implement specific configurations that the EMAP operates on. While work is underway to create an intuitive web-based interface to guide the authoring process of EMAP functions, it is important to understand the underlying tools and methods created and how they impact course construction and delivery. The tools to be reviewed include (1) the Pedagogical Configuration Authoring Tool, (2) the Metadata Authoring Tool, (3) the Survey Authoring System, and (4) the Course Authoring Tool. Each tool serves a different function in building out the adaptive logic associated with a lesson using Merrill’s CDT to structure interaction and assessment types.

Pedagogical Configuration Authoring Tool. The Pedagogical Configuration Authoring Tool (PCAT) is an interface component designed for linking learner relevant information with metadata that drives pedagogical decisions in GIFT. One important note to mention is the domain-independent nature of what is authored in the PCAT. More often than not, systems tightly couple pedagogy with the domain, providing little reuse for future developmental efforts. The EMAP differs from this description in that the authoring process is based on generalized tools and methods that translate across domain applications, including the pedagogical strategies the system acts upon [2]. A developer interacting with the PCAT defines learner model attributes they want GIFT to use for moderating adaptive practices. This is accomplished by linking a set of content descriptors with a given attribute across each of the established CDT quadrants represented in GIFT. These descriptors are a collection of metadata tags that are machine actionable and provide generalized information associated with content and guidance that can be used to configure system interactions in real-time. The metadata currently in use is based on the LOM standards put in place by the Institute for Electrical and Electronics Engineers (IEEE) [8]. This provides a set of high level categories (e.g., interactivity type, difficulty, skill level, coverage, etc.) and value ranges (i.e., skill level is broken down into novice, journeyman, and expert) that inform characteristics for a type of interaction.

While ‘Rules’ and ‘Examples’ primarily focus on information delivery, the ‘Recall’ and ‘Practice’ quadrants are focused on assessment, guidance, and remediation. The ‘Recall’ quadrant is unique because it focuses on knowledge elicitation through an automatically generated quiz that serves as a ‘check on learning’ function. To enable this capability, there are two additional authoring processes that take place. First, in the Survey Authoring System (SAS) you build a bank of concept questions to generate a recall assessment from and second, you use the Course Authoring Tool (CAT) to configure assessment scoring outcomes that determine if you advance to practice or if a remediation loop is triggered. Each process will be covered in more detail below.

In the ‘Practice’ quadrant, an author establishes adaptive configurations associated with the difficulty and complexity of a scenario, along with timing and specificity configurations linked to strategies executed as the result of concept assessments being managed during run-time. These specific ‘Practice’ pedagogical capabilities are still being addressed in the EMAP baseline. Much of this work is being based around Vygotsky’s Zone of Proximal Development [6]. This theory of instruction focuses on the use of guidance to assist a learner in attaining knowledge and skills to complete a task outside of their ability level in conjunction with adapting the complexity of a problem to maintain an appropriate challenge level of a given learner. These tradeoffs must be fully understood to implement sound pedagogical structure when handling practice events in a generalized fashion. The initial configurations will require numerous studies to determine their efficacy.

Metadata Authoring Tool. The Metadata Authoring Tool (MAT) is an interface component in GIFT that provides a simple function; it allows you to tag existing training content with concept-dependent metadata that is acted upon by the EMAP. While the PCAT enables an ITS developer to build configurations between learner model data and metadata descriptors linked with pedagogical techniques, the MAT is established to build files that link metadata with actual content that can be delivered for learning purposes. When in an EMAP managed lesson, a learner is directed through defined branch points, called Merrill’s Branching, that associate with the four quadrants of the CDT. When a learner starts instruction in the ‘Rules’ quadrant, the PCAT produced file is referenced for determining the type of content to search for based on a learner’s individual profile and the type of metadata their models inform. Next, the EMAP searches through the various files generated in the MAT, for a given lesson, and looks for the closest match with respect to the number of descriptors for a tagged learning content and the ideal match informed by the PCAT (see GIFT documentation provided with software download for algorithm that manages content selection based on metadata attributes). As such, the MAT is a very important tool that allows an author and system developer to label their content in a controlled fashion that is then moderated autonomously by the EMAP during run-time.

Survey Authoring System. The Survey Authoring System (SAS) was developed for two primary purposes; (1) to collect learner relevant information through surveys that are used to update learner model attributes for the purpose of adaptation and (2) to deliver knowledge assessments in the form of questions that can be automatically scored for updating competency states and reporting performance across a set of learning objectives and concepts. In terms of the EMAP, both functions are highly relevant. Firstly, if you define configurations in the PCAT that act on learner trait information such as motivation or grit, the SAS allows a developer to present validated questionnaires and surveys that can be used to classify that individual learner.

For the purpose of this paper, we will focus on how the SAS is used to support the ‘Recall’ quadrant in the EMAP by assembling and delivering a set of questions for determining competency and informing remediation paths if required. The SAS was modified to support the EMAP by providing a ‘check on learning’ function through the assessment of knowledge states. A course developer would start interaction in the SAS by building out a bank of questions that are of relevance to the topic of instruction. GIFT’s SAS provides a web interface to author the question, the answer type, and scoring weights that are used in determining knowledge states. A new field added to the question building process is defining a set of associated concepts the specific item informs. For EMAP purposes, this field is very important as it is used in defining the specific concept question bank that will be referenced within the ‘Recall’ quadrant interactions. You can also define the difficulty level of a question in the properties field, as a course developer can configure the number of questions to be administered for a difficulty level across a specified concept.

Following question generation, a course developer is then required to build the context for which the generated items will be acted upon. This is achieved by establishing the specified concept question bank the EMAP will operate on within the SAS’s Survey Context interface window. The survey context is referenced by GIFT’s Course Authoring Tool, which will be described next, and is used to identify the questionnaires in the database that are referenced during course construction, as well as establishing question banks that are used for ‘Recall’ assessment purposes. In the SAS survey context interface, the author has the ability to add concept question banks by selecting a specific concept that was established during question generation. If you are using the EMAP to deliver a lesson on three concepts, this is where you organize separate questions banks for each concept, thus providing a granular approach to assessment that directs personalized and performance driven remediation.

Course Authoring Tool. The GIFT Course Authoring Tool (CAT) is the final authoring interface a researcher interacts with once all of their materials and assessments have been appropriately configured for EMAP run-time. It is in this environment that an author designates the sequence of interaction a learner will experience for a specific lesson. The lesson is composed of defined transitions that dictate what will be presented next (See GIFT documentation for a full list of available transitions and their associated descriptions). The transitions of interest for the EMAP are ‘Present Survey’ and ‘Merrills Branch Point’. The ‘Present Survey’ transition enables an author to select a questionnaire present in the SAS that will be delivered to a learner at any point in the course flow. For EMAP purposes, a course developer might call for surveys upfront to collect information on learner model attributes, such as measuring an individual’s motivation for a topic. This requires an author to designate a survey context present within the SAS that determines what surveys are made available.

With defined surveys used to inform learner attributes, the next step for an author is establishing Merrill Branch Points. These branch point transitions are managed by the configurations established in the PCAT and MAT, as described above. Within the CAT, the author selects specific concepts used to build the quadrants of the CDT for a given branch point. Within a single course you can define multiple branch point transitions. Once you select the concepts a branch point associates with, the author then has the opportunity to configure interactions across the quadrants of the CDT. The CAT configurations allow a user to deselect certain quadrants that would be bypassed when a learner enters that branch transition.

The last configurations an author can make within the CAT is specifying what will be delivered to a learner within the Recall quadrant of the CDT. Once a user specifies the survey context referenced in the SAS and the concepts a Merrills Branch Point is used to manage, the next step is defining the number of questions to deliver for knowledge state assessment purposes and the scoring rules that determine the state value to communicate out to the learner module. When defining the number of questions to deliver, the inputs are used to identify matches within the established question banks that are built in the survey context interface within the SAS. It first looks for questions linked to a concept and then it looks for metadata values used to classify a difficulty ranking.

With the ‘Recall’ quadrant specified, an author then configures what is to be experienced with the ‘Practice’ quadrant. This interaction takes the form of a traditional ‘Training Application’ transition in the CAT where the user designates what application is being used and what Gateway is being applied for communication purposes. In addition, the author also defines specific scenario files to load based on metadata and the associated Domain Knowledge Files (DKFs) built for assessment purposes. These DKFs are essential components of all GIFT practice-based interactions. It is in the DKF where conditions are created for real-time assessment purposes, as well as where specific feedback strategies are translated into tactics that are delivered to a learner in real-time. The DKF is being modified to support the personalized strategy configurations that link to timing and specificity of feedback along with available scenario adaptations. See the GIFT documentation for all materials related to the DKF. With these final touches in place, an author is now ready to deliver an EMAP driven course. While the tools were established to build courses and lessons to be managed by GIFT, their intended function may differ. Depending on how a developed lesson is used will determine the specific authoring interfaces a user needs to interact within.

4 AugCog Research Using GIFT and the EMAP

When designing the EMAP, two primary end users were considered. These included traditional educators and training developers that would use the EMAP toolsets to author adaptive courseware materials they could distribute across classes and training organizations, and it included researchers and scientists within an AugCog focused research community that has interest in adaptive instruction, individual differences, and pedagogical heuristics. Making the distinctions between these two user groups is important because it will dictate the primary authoring tools they will interact with.

When considering the authoring experience across the EMAP tools for the AugCog community, an assumption is that this user group is interested in the creation of experimental treatments for running studies associated with cognitive intervention based research. The main difference between these individuals and educators/training developers is that the AugCog group will interact heavily with the PCAT to build out specific configurations in support of their research questions. This involves modifying current attributes available in the EMAP as well as adding additional variables in the learner model with the goal of assessing its effectiveness in informing adaptive intervention approaches. Within the PCAT, an author references a learner model attribute as well as metadata descriptors associated with the ideal type of content to deliver to that specific learner. For research purposes, the AugCog user group will be responsible for adding and removing available attribute tags for both the learner model and metadata variables and associated values. Each of the available tags are referenced in an enumeration file in GIFT’s source code that can be modified to support the type of learner attribute and metadata a researcher desires the EMAP to act on. These files can be located at the following directory off of GIFT root: GIFT\src\mil\arl\gift\common\enums\MetadataAttributeEnum.java and GIFT\src\mil\arl\gift\common\enums\LearnerStateAttributeNameEnum.java. For each of the attributes represented in the learner state attribute enumerations file, a user will need to author an additional.java file that specifies the available tags made available for that variable when manipulating the PCAT authoring tool. Many examples are available to work from in the GIFT\src\mil\arl\gift\common\enums\ folder.

In implementing the EMAP for support in the AugCog research community, there are a couple assumed dependencies associated with the authoring tools. While an author will be required to modify enumeration files in the source code to introduce variables of interest not currently supported in GIFT’s baseline version, an author will also be required to make additions in the SAS for the purpose of collecting individual differences metrics used to inform variables that dictate strategy selection. Further, GIFT also supports tracking and inferring upon affective based data for the purpose of diagnosing cognitive and emotional states experienced during a learning event. None of the current EMAP configurations are built to support acting on this type of assessment. The PCAT needs to be modified for this very purpose, along with establishing classifiers within the learner and sensor modules of GIFT that would inform state.

In addition, the EMAP’s current baseline provides an automated lesson manager for a GIFT course with some caveats. For example, the course author has but only one option for a ‘check on learning’ as part of the Recall quadrant and that is a test composed of questions selected at runtime from a pre-established bank. Other forms of knowledge assessment could be given either with a static pre-authored survey or using AutoTutor which is also integrated with GIFT [9]. Using AutoTutor allows GIFT to hold a conversation in natural language between one or more agents and the learner. This interaction can be used to elicit comprehension of the concepts covered in prior Rule and Example quadrants that goes beyond just recalling information. Implementing this approach allows a learner to explain knowledge in their own words, thus providing evidence of higher order cognitive understanding of the material.

5 Conclusion

The EMAP provides a suite of authoring interfaces to build highly individualized and adaptive courseware materials. In this paper, we review the authoring tools developed in support of the EMAP functions that relate to AugCog based principles associated with training and education. A description of each tool is provided, along with the data used to inform its logic. The AugCog community can leverage these tools to create highly robust testbeds to evaluate intervention techniques, as well as examining new domain of application, such as robotics and human-machine interaction.