Present day applications demand that behavioral measurements are performed in natural environments, where the measuring devices are thoroughly integrated into everyday objects and activities. Understanding users’ behavior in different contexts could be a goal of measuring human behavior, as well as a means for designing user experiences that utilize on pervasive measuring technology.

The aim of this thematic issue on ‘Measuring human behavior and interaction’ is to summarize emerging trends and common problems across different branches of social, behavioral sciences, and interaction design that involve measurements of human behavior. It is inspired by the Measuring Behavior 2010 conference [1], which provided a broad and interdisciplinary forum for novel methods to define, measure, and analyze human, animal, and machine behavior. The papers in this thematic issue of Personal and Ubiquitous Computing are a selection of work presented at or related to the scope of the conference, but also relevant for the readers of PUC journal.

One increasingly important issue in the measurement of behavior in computer (or machine)–human interaction is that of the ecological relevance of a study. If a test participant is asked to perform a task in a laboratory environment, then the behavior might be measured almost 100% correctly, but have limited meaning for real-life settings. On the other hand, measurements carried out in the subject’s natural environment are often hampered by the lack of controlled conditions, inadequate replication, and all sorts of technical issues caused by the difficulties of mobile and active subjects. Several of the papers in this thematic issue address that problem in various ways. Maly et al. show how the problem can be addressed by integrating and visualizing several different sources of data recorded from mobile subjects, giving much better power of interpretation than a narrow range of measurements. Their experiment measured navigation of visually impaired subjects in large-scale complex real-world environments. Navigation in such environments is qualitatively different from that of artificial small-scale environments, so laboratory or virtual reality studies have a large risk that their results will be without explanatory power in the real world. Kukka et al. present a study measured in an entirely natural situation, where the use of information services provided to people in a city center was measured, and despite the lack of laboratory conditions, information about the strategies of information seekers could be derived. As in many behavioral studies, that one also proved that the behavior of the people studies was actually quite different from what the subjects themselves said that their behavior would be. Studies in natural environments do not have to be on a large city-wide scale. Zillmer’s paper shows the insights, which can be gained in an oral care campaign by the automated measurement of people brushing their teeth. Such studies are often carried out purely by questionnaire and as such do not quantify what the subjects actually do. Wijn also describes a study where the ecological context was the focus; he shows that the expertise of security camera operators is largely governed by their knowledge of the location of the scene rather than their knowledge of the behavior of suspects.

The true potential of the emerging branches of research, such as Ubiquitous Computing, Ambient Intelligence, and Social robotics, is to be achieved only if a true synergy between multiple disciplines from both: exact and social sciences is achieved. The envisioned systems can only be realized if human behavior can be analyzed automatically beyond the level of its conscious expressions, including emotions, intentions, and mental states [2]. This thematic issue illustrates the trend in two ways; measurement of the emotional experience of the subjects as an augmentation of additional measures, and as an affective feedback loop in product interaction. Terzis presents data validating the measurement of human emotions expressed in facial expression using computer vision technology (FaceReader software). The emotions were measured while the subjects were doing computer-based assessment, and the author shows that the performance of the examinees could be influenced by their emotional reactions as well as their knowledge and aptitude for the subject under examination. Burns also measured emotional response, but using a special sort of pen that sensed behaviors related to restlessness. In his study, the device not only measured the emotion, but gave feedback designed to influence the emotional experience. Van den Broek focuses on methods for the measurement of emotions.

Another research trend that is apparent in this thematic issue is the increasing use of multiple data streams including physiological data to measure the totality of the subjects’ behavior. Van der Broek shows that combined measurements of heart rate variability, self-assessment, and analysis of the speech give powerful insights into the emotions of the subjects. Three variables in the speech (pitch variability, intensity, and energy) enabled discrimination of the 6 basic emotions and arousal. Maly et al. present a tool for the visualization of multiple data sources including galvanic skin response and heart rate variability (both of which provide a good indication of stress levels) together with video and a log of observed behavior.

It is also clear that traditional observations and video filming retain a central position in most studies. For example, Gamberini et al.’s study using The Observer [3] to video code detailed user interaction shows how the subjects’ behaviors can be broken down into a progressively finer level of granularity.

Several of the papers, such as the one above and Terzis’s paper, focus on the development of methods of usability studies (in the broad sense), but the papers in this thematic issue also illustrate the upcoming importance of two areas; health and security. Zillmer’s research shows more effective ways of collecting data in the context of dental health care, and Maly et al.’s research is in the context of assisting visually impaired people. The increasing importance of the study of human–machine interaction in the security domain is illustrated by Wijn’s paper. Nevertheless, it is also clear that IT still stands for information technology, and the papers of Kukka et al. Huang, Wijn, and Maly et al. all relate to how information is conveyed to users from various information systems. Huang’s study uses eye tracking to measure how subjects extract information from graphs. The best way to convey information through graphs is traditionally based on little more than common sense and intuition, and Huang shows that the measurement of how people actually look at the graphs can be a tool for really understanding the underlying mechanisms affecting what is apparently more of an esthetic issue.

In general, all the papers in this thematic issue illustrate that the measurement of the behavior of users interacting with information systems has come a long way in the past years, and that the time when the only information we had available was derived from a subjective questionnaire or a simplistic test is long behind us. Furthermore, today’s methods and tools for measuring behavior are finding their way outside of the research assessment context and are becoming an integral part of intelligent products and services that can lead to a breakthrough in domains such as healthcare, security, education, and entertainment.