Abstract
In this paper we describe a method that can be used in the analysis and evaluation of interactive systems. This method is a combination of three techniques: logging, playback, and verbal protocols. We have employed the method in the evaluation of an interactive program verification system. This evaluation involved users on two levels of expertise: on one hand, novices taking a course on program verification; and on the other hand, subjects already familiar with verification. The system stored the commands of the test subjects into a log file while the test subjects were working on a given problem. Directly after the first session, a second one followed where the previous session was shown in playback on the terminal to the test subjects and they were asked to verbalize their work retrospectively. According to our tests the above procedure turned out to be a convenient way of obtaining data for system evaluation as well as data on the mental models formed by the subjects of the system. It is especially useful in early design phases eliciting user opinions, e.g., on prototype versions which then can be used for iterative design. In our case, the collected information has been utilized in the design of a new version of the verification system. In addition, the combined method can be useful in teaching, which is also demonstrated through our experiments.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsPreview
Unable to display preview. Download preview PDF.
References
Back, R. J. R., 1983, “Invariant Based Programs and Their Correctness”, Automatic Program Construction Techniques, Biermann, Guiho, and Kodratoff, eds., MacMillan, Section IV, pp. 1–22.
Back, R. J. R., and Hietala, P., 1984, “A Simple User Interface for Interactive Program Verification”, Proceedings of INTERACT’84: First IFIP Conference on Human-Computer Interaction, B. Shackel, ed., London, September 4–7, 1984, 1, pp. 114–118.
Bainbridge, L., 1979, “Verbal Reports as Evidence of the Process Operator’s Knowledge”, International Journal of Man-Machine Studies, 11, pp. 411–436.
Brown, M. H., and Sedgewick, R., 1984, “Progress Report: Brown University Instructional Computing Laboratory”, Proceedings of the Fifteenth SIGCSE Technical Symposium on Computer Science Education, Philadelphia, February 16–17, 1984. ACM SIGCSE Bulletin 16, 1, February, 1984, pp. 91–101.
Ericsson, K. A., and Simon, H. A., 1980, “Verbal Reports as Data”, Psychological Review, 87, pp. 215–251.
Ericsson, K. A., and Simon, H. A., 1984, Protocol Analysis: Verbal Reports as Data, MIT Press, Cambridge, Mass.
Gaines, B. R., 1981, “The Technology of Interaction-Dialogue Programming Rules”, International Journal of Man-Machine Studies, 14, pp. 137–150.
Goldberg, A., 1984, Smalltalk-80: The Interactive Programming Environment, Addison-Wesley, Reading, Mass.
Good, M., 1985, “The Use of Logging Data in the Design of a New Text Editor”, Human Factors in Computing Systems, L. Borman and B. Curtis, eds., CHI’85 Conference Proceedings, San Francisco, CA, April 14–18, 1985, pp. 93–98.
Gould, J. D., and Lewis, C., 1985, “Designing for Usability: Key Principles and What Designers Think”, Comm. ACM, 28 (3), pp. 300–311.
Hietala, P., 1985, A Course on Program Verification, University of Tampere, Department of Mathematical Sciences, Report A137.
Hietala, P., 1985, “Teaching Program Verification With the Help of an Interactive Program Verification”, Proceedings of WCCE/85: World Conference on Computers in Education, K. Duncan and D. Harris, eds., Norfolk, Virginia, July 29 -August 2, 1985, pp. 561–566.
Hietala, P., 1985, “On the Evaluation of a Support System for Interactive Program Verification”, Preprints of the Second IFAC/IFIP/IFORS/IEA Conference on Analysis, Design, and Evaluation of Man-Machine Systems, G. Johannsen, G. Mancini and L. Martensson, eds., Varese, Italy, September 10–12, 1985, pp. 198–203.
Hietala, P., Järvinen, P., Mäkinen, E.., and Tyllilä, P., 1985, “On Two Approaches to Computer Science Research and on Their Symbiosis”, 20th Anniversary Book of the Faculty of Economics and Administration, University of Tampere, Acta Universitatis Tamperensis, Ser. A, 190, pp. 317–338.
Hoc, J-M., and Leplat, J., 1983, “Evaluation of Different Modalities of Verbalization in a Sorting Task”, International Journal on Man-Machine Studies, 18, pp. 286–306.
Lewis, C., 1982, Using the “Thinking Aloud” Method in Cognitive Interface Design, IBM, T. J. Watson Research Center, Research Report RC 9265.
Neal, A. S., and Simons, R. M., 1983, “Playback: A Method for Evaluating the Usability of Software and its Documentation”, Human Factors in Computing Systems, A. Janda, ed., CHI’83 Conference Proceedings, Boston, December 12–15, 1983, pp. 78–82.
O’Malley, C., Draper, S., and Riley, M., 1984, “Constructive Interaction: A Method for Studying User-Computer-User Interaction”, Proceedings of INTERACT’84: First IFIP Conference on Human-Computer Interaction, B. Shackel, ed., London, September 4–7, 1984, 2, pp. 1–5.
Root, R. W., and Draper, S., “Questionnaires as a Software Evaluation Tool”, Human Factors in Computing Systems, A. Janda, ed., CHI’83 Conference Proceedings, Boston, December 12–15, 1983, pp. 83–87.
Shneiderman, B., 1980, Software Psychology: Human Factors in Computer and Information Systems, Winthrop, Cambridge, Mass.
Yoder, E., MacCracken, D., and Akscyn, R., 1984, “Instrumenting a Human-Computer Interface for Development and Evaluation”, Proceedings of INTERACT’84: First IFIP Conference on Human-Computer Interaction, B. Shackel, ed., London, September 4–7, 1984, 2, pp. 309–314.
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 1987 Plenum Press, New York
About this chapter
Cite this chapter
Hietala, P. (1987). Combining Logging, Playback and Verbal Protocols: A Method for Analyzing and Evaluating Interactive Systems. In: Rasmussen, J., Zunde, P. (eds) Empirical Foundations of Information and Software Science III. Springer, Boston, MA. https://doi.org/10.1007/978-1-4613-1895-8_9
Download citation
DOI: https://doi.org/10.1007/978-1-4613-1895-8_9
Publisher Name: Springer, Boston, MA
Print ISBN: 978-1-4612-9055-1
Online ISBN: 978-1-4613-1895-8
eBook Packages: Springer Book Archive