Evaluating intelligent knowledge systems: experiences with a user-adaptive assistant agent

Abstract

This article examines experiences in evaluating a user-adaptive personal assistant agent designed to assist a busy knowledge worker in time management. We examine the managerial and technical challenges of designing adequate evaluation and the tension of collecting adequate data without a fully functional, deployed system. The CALO project was a seminal multi-institution effort to develop a personalized cognitive assistant. It included a significant attempt to rigorously quantify learning capability, which this article discusses for the first time, and ultimately the project led to multiple spin-outs including Siri. Retrospection on negative and positive experiences over the 6 years of the project underscores best practice in evaluating user-adaptive systems. Lessons for knowledge system evaluation include: the interests of multiple stakeholders, early consideration of evaluation and deployment, layered evaluation at system and component levels, characteristics of technology and domains that determine the appropriateness of controlled evaluations, implications of ‘in-the-wild’ versus variations of ‘in-the-lab’ evaluation, and the effect of technology-enabled functionality and its impact upon existing tools and work practices. In the conclusion, we discuss—through the lessons illustrated from this case study of intelligent knowledge system evaluation—how development and infusion of innovative technology must be supported by adequate evaluation of its efficacy.

Appendices

Appendix 1: CALO Test scoring method

The annual CALO Test proceeded as follows, for project Years 2–4 (Sects. 4.1, 4.2).

Methods. The CALO Test process consisted of five parts, as follows for PTIME.

• 1. The independent evaluator (IE) defined a set of parameterized questions (PQs). These templates were made known to the PTIME team, who worked to develop the system’s capabilities towards them. There were some 60 PQs relevant to time management. For example:

  Rank the following times in terms of their suitability for [MEETING-ID], given the schedules and locations of anticipated participants.

  Each PQ was supplemented by an agreed interpretation (i.e., what the PQ means) and an ablation process (i.e., how to remove any learning in LCALO, to yield BCALO), both approved by the IE.

• 2. Data was collected during the week-long critical learning period (CLP).

• 3. The IE selected a subset of the PQs for evaluation. In Year 2, nine PTIME-relevant PQs were selected. In Year 3, two additional questions were selected.

• 4. The IE instantiated each PQ with three instantiations relevant to the data set. For example, one instantiation of the above PQ is:

  Rank the following times in terms of their suitability for MTG-CALO-0133, given the schedules and locations of anticipated participants: (1) 7 am, (2) 10 am, (3) 10:30 am, (4) 3 pm, (5) 8 pm.

• 5. The IE scored LCALO and BCALO on each such instantiated question (IQ) instance and produced the overall results. First, the IE determined the ‘gold-standard’ answer for each IQ. For each PQ, the process for determining the answer key was documented prior to the Test. For example, for the above PQ:

  Since this PQ is not asked from a single user’s perspective but from a global perspective (what is best considering all invitees), the Test evaluators will select an arbitrator who will be given access to all calendars and user preferences. The arbitrator may also ask any user for any information that may help the arbitrator identify the best answer. For example, the arbitrator may ask how important the meeting is to a user. The arbitrator will come up with the correct answer.

  While some PTIME IQs had objective answers, others (such as the above) had subjective answers. The IE followed the answer determination process to derive the answer key for each IQ. If necessary, the IE elicited information from the CLP participants, and if further warranted, made subjective final decisions.

  Second, the IE scored LCALO and BCALO against the answer for each IQ. Scores were between 0 (worst) and 4 (best). Again, for each PQ the process for determining the score was documented prior to the Test. For our example PQ, the process was to compare the ordered list generated by PTIME with the ordered list of the answer key by (quoting verbatim):

  Kendall rank correlation coefficient (also called Kendall Tau) with a shifted and scaled value: Kendall Tau generates numbers between −1.0 and 1.0 that indicate the correlation between two different rankings of the same items. 1.0 indicates the rankings are identical. −1.0 indicates that they are the reverse of each other. Kendall Tau accommodates ties in the ranking. To get values that range from 0 to 4 (rather than −1.0 to 1.0), we use the following adjustment: Score $=$ ( Kendall Tau $+ 1$ ) $\times \,2$

  This scoring process was encoded programmatically so that scores could be computed automatically for LCALO and BCALO.

Tasks. Participants used CALO as part of their regular work activities during the period of the CLP. The participants pursued realistic tasks, in a semi-artificial scenario, i.e., in a dedicated physical location rather than their usual workplaces (Lessons 3 and 4). Participants were given guidance about activities to try to include in their work, for instance, to schedule and attend a certain number of meetings; the independent evaluator approved the guidance. Participants were informed that the CALO system was being evaluated (and not their work) and that they might encounter bugs in the system, due to it being in on-going development.

Appendix 2: Specific critique of the CALO Test

Further to the discussion of Sect. 4.1—the CALO Test aimed for objectivity, as far as could be attained, in providing a quantitative measure of the effects of learning on CALO’s performance. However, the nature of the scoring process of the CALO Test introduced unintended artefacts.

First, instantiated questions (IQs) were instantiated (from the parameterized questions (PQs)) with a range of ‘difficulty’, determined by what the Independent Evaluator (IE) considered easy/difficult for a human office assistant. What is easy or difficult for an intelligent assistant can differ from what is easy or difficult for a human.

Second, as described in “Appendix 1”, some IQs had subjective ‘gold-standard’ answers that required ex-post (i.e., after the activity) elicitation from subjects by the IE, and a partially subjective human decision on the answer key. More generally, a difficulty in evaluation is in defining successful completion of a task. It is worth noting how the PQs were defined by the IE to scope the information required to determine the answer key. For instance, it was not necessary to determine whether users had chosen the best schedules for their requirements out of all possible schedules, but only from the multiple choices of the IQ answers.

Third, for PQs asked as a multi-choice question, a chance effect could unintentionally favour BCALO. For example, consider a multiple choice PQ with two possible answers, A or B, and its three instantiations to IQs. Suppose BCALO has a naive strategy of always returning answer A. There is a $\frac{3}{8}$ probability that for two of the three instantiations, A is the correct answer. In this case, BCALO scores 67% (2.67 / 4.0), which is higher than the LCALO target for the question!

Fourth, the scoring process for some PQs created artefacts. For example, consider the PQ of “Appendix 1”, which was scored using a shifted Kendall rank correlation coefficient. If CALO’s answer showed no correlation with the correct answer, it still gets 2 out of 4 points; thus BCALO scored at least 2. Only failing to pick some answer from the given list of choices would score 0 points.

Fifth—a point which we recognized with PTIME, when for instance memory usage of other components slowed CALO’s responsiveness—even though the Test was intended to measure CALO’s learning ability, it could not do so unless the other parts required by the Test process were in good working order, so that learning data could be collected for LCALO. Architecture, documentation, pretesting, debugging, usability, and user behaviour were therefore all important to scoring well, as much as learning algorithms (Lesson 6).

As a rule, it is difficult in any evaluation to eliminate effects such as selection bias, experimenter bias, learning effects, and the Hawthorne effect— although proper experimental design can minimize them or at least make their effects measurable. The CALO Test was in part deliberately not structured and conducted to eliminate such effects as much as it otherwise might have been, since the CLP was more a data-gathering exercise on the system than a regular user study. For example, whether it was impactful or not upon the data collected, there was selection bias from using subjects from our own institution (which was required for legal reasons). The Test was overseen by the IE, and monitors from the project sponsor were present. Both were satisfied by the validity of the Test results.