Keywords

1 Designing Line Graphs for Accessibility

1.1 Accessibility for Visualizing Data: Haptic Line Graphs

Line graphs are means for data visualization, which aim at presenting or representing statistical data. By interpreting graphical constituents together with the graph-schema knowledge, users gain access to information about domain-specific events. For example, a line graph may represent population trends or change in monthly temperature in time-series. In the recent state of technology, graphs are frequently used tools that facilitate communication, thinking, reasoning. The immediate visual interpretation of the readers and easy design of line graphs by end-users result in their widespread use both in printed and digital communication settings.

Visually impaired users may explore graphs haptically, by hand-controlling a stylus of a force-feedback device, such as a Phantom Omni® (recently Geomagic® TouchTM, see Fig. 1.a). The haptic exploration of a virtual graph using a stylus has to allow extracting information about geometrical properties of the graph lines. The user interprets the geometric forms as domain-specific events, in a similar way as users of visual graph displays do. However, in contrast to the ease of design as a visual representation, the design of haptic line graphs faces a set of challenges. These challenges are pertinent to the efficiency of haptic graphs in communication, thinking and reasoning. First and foremost, a drawback of haptic graphs (compared to visual graphs) is the restriction of the haptic sense in simultaneous perception of spatially distributed information [1]. In particular, the information extraction through haptic line graphs is based on specific explorations processes. For instance, line-tracing hand-movements aim at detecting shape properties during the course of haptic exploration. Moreover, the recognition of concavities and convexities, as well as the recognition of maxima and minima, are of major importance (see Fig. 1 for sample line graphs).

Fig. 1.
figure 1

(a) Sample haptic graph, (b) Exploration of a physical haptic map and (c) Phantom Omni® device and visualization in the domain of geometry.

Full size image

The challenges for designing haptic line graphs apply to both simple line graphs that may be expected to be subject to straightforward user exploration, as well as to complex ones. For instance, the haptic exploration of a line graph with a single (global) maximum is only moderately difficult, whereas the haptic exploration of a line graph with several local extrema requires assistance in another modality. Those general observations about the user experience in haptic graphs suggest that multimodal exploration may facilitate graph comprehension by visually impaired individuals.

Multimodal exploration of haptic line graphs has been the topic of research since the past decade. The research has shown that providing additional information facilitates haptic-graph exploration. For instance, supplementing haptic exploration by aural assistance through the auditory channel (cf. Sonification) leads to better comprehension of haptic graphs [2]. In a series of prior studies, we have shown that appropriate spoken utterances (i.e. verbal assistance) facilitate haptic graph exploration [3–5]. For instance, verbal assistance improves detection and specification of local and global extrema in haptic graph lines. Therefore, as a novel, multimodal human-computer interaction method, a synchronous use of force-feedback exploration of haptic graphs and verbal assistance make statistical graphs accessible to visually impaired users.

In previous studies we have mainly focused on the information content of the verbal assistance that is to be delivered to the user of the haptic interface. In other words, this information has been about “what to say” to the user to improve his/her comprehension during the course of haptic exploration. The findings have shown that appropriate information content lead to a successful conceptualization of the events that are represented by graph lines. An equally important aspect in designing haptic graphs is to identify “when to say”, as well as “what-to-say”. In the present study, we aim at developing an automated mechanism, which is able to detect and predict whether the user needs verbal assistance, by analyzing his/her haptic exploration patterns.

1.2 Haptic Exploration as a Joint Activity: An Experimental Paradigm

The design of our model is based on the data collected by employing a specific experiment setting that was developed for studying the role of verbal assistance in haptic exploration [3–5]. The setting allowed us recording the information content of verbal assistances, as well as users’ haptic exploration patterns before, during and after an explorer’s help-request. In this experimental paradigm, verbally assisted haptic graph exploration is conceived as a task-oriented collaborative activity between two human partners, a (visually impaired) explorer (E) of a haptic graph and an observing assistant (A) providing verbal assistance (see Fig. 2).

Fig. 2.
figure 2

Assisted haptic graph exploration as a joint activity

Full size image

The alignment between the interlocutor’s internal models (cf. Building common ground in [6]) and the availability of appropriate verbal assistance lead to a successful haptic comprehension of graphs. In this experimental paradigm, A observes E’s exploration movements and provides appropriate verbal assistance upon requirements of E. During E’s exploration, A analyzes the ongoing exploration event. In the long run, A has to take into account the history of exploration events, as well as the history of the verbal assistance, to be able to produce appropriate verbal assistance.

The experimental paradigm involved two human partners. An automatic verbal assistance system, which is able to predict when to produce an utterance for verbal assistance, should aim at modeling the alignment between the interlocutors and making communication between E and A efficient and effective. In the first prototype, called the ObservingAssistant, we employed a rule-based approach. The system analyzes users’ exploration patterns and triggers reactively canned text, as realized by the MARY text-to-speech system [7]. In the present study, we employ a machine-learning and classification perspective by means of analyzing the current position of exploration, the history of exploration movements, and the referring utterances (the referred locations and how these regions were referred). The following section provides the technical background for the model by introducing the shape concepts basic for graph-line descriptions.

1.3 Shape Concepts in Graph Line Descriptions

Qualitative approaches to shape representation usually focus on the shape of contours, e.g. [8, 9], and the curvature landmarks of contours [10]. These landmarks involve visually salient regions, such as positive maxima and negative minima, depending on the concepts of convexity and concavity of contours, and inflection points. A qualitative analysis of graph lines requires additional shape representations and shape cognition characteristics, which go beyond that of contours. This is because the interpretation of graphs lines requires graph-schema knowledge [11, 12]. The left-to-right orientation of graph lines, which usually relies on the writing direction, leads to conventionalized forms of interpretation. Graphs can also be interpreted with respect to orthogonal two-axes-systems. In haptic graphs, we also use a rectangular frame that induces an orthogonal system of axes. Accordingly, the functional characteristics of graph lines lead to the prominence of value extrema in graph lines, in contrast to curvature extrema of contours. Figure 3 exemplifies a set of basic geometric shape concepts for describing graph lines. Table 1 exemplifies some qualitative representations for selected shape landmarks and induced line segments (see [5] for details). Since we user smoothed graphs in the experiments, we classify extrema as smooth points (sp).

Fig. 3.
figure 3

Qualitative shape landmark ascription for a sample graph (augmented with orthogonal axes for making the reference frame in Table 1 explicit) [‘ep’ stands for end point].

Full size image
Table 1. Qualitatively described shape landmarks and shape segments
Full size table

In the previous studies, we focused on the qualitative ascriptions of each landmark and segment for the analysis of generation of referring expressions by an automated system. In the present study, our focus is to use this analysis together with exploration patterns to automatically predict when-to-say verbal assistance.

2 Data Collection: A Joint-Activity Experiment on Haptic Line Graph Exploration

In this section, we give a brief description of the experiment (participants, stimuli and procedure) that provides the data for haptic exploration patterns presented in Sect. 3 (see [5] for details of the experiment and the analysis of verbal utterances). Twenty-six participants (13 pairs of sighted and blindfolded university students) participated in the experiment. Each pair was composed of a haptic explorer (E) and a verbal assistant (A), as described in the previous section. The participants were located in separate rooms so that they communicated through speakers without visual contact. During the experiment session, E explored the graph haptically and A was able to display the graph and the current location of E’s exploration, which was represented by an animated point marker on the visual graph presented at A’s screen. However, haptic pointing was possible only for E. The haptic explorer E was instructed to explore the graph and ask for verbal assistance when needed, whereas the observer assistant A was instructed to provide verbal assistance shortly and plainly, when requested by E.

The participants were informed that the graphs represented populations of bird species in a lagoon and also about post-exploration tasks detailed below. The graphs employed in this study were taken from a publicly available consensus report [13]. Each graph had a different pattern in terms of the number and polarity of curvature landmarks, length and direction of line segments. In the experiment session, each participant was presented five haptic line graphs in random order. Haptic graph exploration was performed by moving the stylus of the haptic device, which can be moved in all three spatial dimensions (with six degree-of-freedom). The haptic graph proper (i.e., the line of the line graph) was represented by engraved concavities on a horizontal plane; therefore haptic explorers perceived the line as deeper than the other regions of the haptic surface. The numerical labels were not represented. The participants did not have time limitation. After the experiment session, both participants (E and A) were asked independently to present single-sentence verbal descriptions of the graphs to a hypothetical audience. Not all haptic exploration protocols had dialogue or request for verbal assistance therefore 26 protocols from 7 pairs were chosen from the empirical data for the feature set construction mentioned in the next section.

3 Haptic Exploration Patterns

Our goal in the present study is to design a classifier for the exploration patterns, which is able to characterize the need for verbal assistance during the course of exploration. The raw data set consisted of the recorded locations of the stylus of the haptic device on the 2D horizontal plane (horizontal and vertical coordinates). In this section, feature construction and transformation from 2D raw data into a 14D feature set, cleansing and chunking procedures and classification of feature sets are presented.

3.1 Feature Construction and Transformation

As the first step in the model design the raw data points (i.e., the coordinates of the stylus) were segmented with respect to the landmarks of the graph lines (see Fig. 4). The segments were then labeled with their corresponding landmarks and line segments (Feature#3: Temporal Order in Table 2). For example the data point (15,7) was labeled “SP3-SP4” and the data point (23,2) was labeled “SP4” depending on the landmarks that the data points belonged to. After the labeling step, the features were identified within the range from basic (low-level) features to complex (high-level) features. The first feature set (SET I) in Table 2 contains a set of basic features that can be derived from haptic explorations. The second set (SET II) involves higher-level features, such as the direction change during the exploration of the same unit and the visit history (time sensitivity). Two low-level graphical features, namely the Graph ID and the size of the graphical unit, were added in the third set (SET III). The last set of features (Set IV) contains conceptual features.

Fig. 4.
figure 4

Five different haptic graphs

Full size image
Table 2. The list of features
Full size table

Low-Level User Action Features. After the labeling of each data point, the Euclidean distance (Feature#4 in Table 2) between each consecutive data point was calculated. Time and speed were also fundamental features included in this data set. Afterwards, the direction of the action between each consecutive data point was labeled as shown in Fig. 5a. The Direction feature was not put directly into the final feature sets; instead it was used for calculating the number of direction changes in the same unit during the same action, as explained below. Data cleansing procedure was applied subsequently. As Grunwald et al. stated, “Human haptic perception is accompanied by movement pauses of the exploring fingers and hands in healthy humans” [14]. The raw data contained instances of “no-movement” between the continuous actions, as well, such as a Left-Left-NoMovement-Left-Left-Left pattern. Therefore the data points, which preceded and followed by an action in the same direction, were labeled with their temporal neighbors’ direction.

Fig. 5.
figure 5

(a) Labels for direction mapping and (b) simple user actions in 2-pixel-width haptic line

Full size image

High-Level User Action Features. The Direction change (Feature#7) between the two consecutive data points with different directions was also labeled with respect to the two axes. In other words, the action with a Rightward-Increase, when followed by leftward actions or when followed by downward actions assumed to exhibit a direction change (see Fig. 5a). One should note that both the NoMovement instances and the actions with the change in the main direction need special attention. This is because the width of the haptic graph line is two-pixels in size. Therefore the data need to be corrected in order to eliminate the miscalculation of direction changes. Consider the following example to clarify the correction procedure: Let’s assume that the dots in Fig. 5b correspond to pixels. When we look at just two consecutive actions, we can label the action presented in Fig. 5b.1 as Right-Up-Right Right-Right. However, the upward action in that case does not have a meaning in terms of the user’s intended action, which is “to move right”. Therefore the actions with main directions (Left, Right, Up and Down) were checked with respect to their previous and next directions. If the previous and next actions were in the same direction, then the value of this data point was updated. On the other hand, if they were different (Fig. 5b.2), then they were marked as a direction change (Feature#7).

It is a common practice to handle data by chunking with respect to the units (segments or landmarks). Therefore the next step in the analysis was to merge the data points that belonged to the same graph unit by taking their temporal order into account. Figure 6 illustrates a sample for the step-by-step chunking procedure. As a result of this procedure, the consecutive data points on the same region were merged, and their features (i.e. distance, time, direction count) were also merged.

Fig. 6.
figure 6

Sample for the chunking procedure (d:distance, t = duration for each action)

Full size image

Statistical line graphs are spatio-temporal representations when they represent time-series data. The features that are presented above address spatial properties of the graphs. To keep track of the frequency of the visit in a time-sensitive way, a history of visit count was addressed by Feature#8 in Table 2. Since the haptic exploration is sequential, the haptic explorer has to pass all the units on the way to that particular unit, in order to reach to an attended/intended location at a certain time. Therefore instead of counting the number of visits during the whole exploration, looking for the back-and-forth movement in a pre-defined temporal window among the units seems more efficient. In the present study, two versions of the visit history were calculated: The visit count of a particular data point (i) in the last 6-steps or (ii) in the last 10-steps. Figure 7 shows a simple illustration of the calculation.

Fig. 7.
figure 7

A simple example for the calculation of the visit count

Full size image

Low-Level Graphical Features (Perceptual). In the analysis of the features presented above, only haptic user actions were taken into account. On the other hand, empirical studies indicate that perceptual characteristics of the graphs might have a crucial role on haptic exploration. Those characteristics were addressed in two features: The first feature (Feature#9) identifies the graph with a unique graph ID. The actual size of the graphical elements (the length of the units in pixels) comprised the second feature (Feature#10). The second feature highlights the perceptual saliency of the units (i.e., a long segment vs. a short segment).

High-Level Graphical Features (Conceptual). In the last feature set (SET IV), we used a set of conceptual features that were derived from the raw coordinate data. First, the units were split into two categories: intermediate units and end points (Feature#11). Empirical results indicated that the explorers might request assistance for descriptional instructions (i.e. the graph domain content) or navigational instructions about positioning (being at the start point, or being on the line, etc.). The navigational instructions were mostly asked for end points of the graph lines. Therefore the data points were also classified with respect to their location on the graph. After then, each data point was categorized into two, according to whether they belonged to a segment or they belonged to a landmark (Feature#12), e.g. “SP1”: landmark, and “SP1-SP2”: segment. The data points were the categorized with respect to their qualitative ascriptions, as described in Sect. 1.2 and Table 2 (i.e. “Ep1-SP1”: steep, and “SP5”: Global Max). Finally, the last feature represents a target feature, which is nominal, and it addresses the binary classification problem itself.

As a result of the feature construction and chunking procedures described above, 69,574 instances (data points with horizontal and vertical coordinates) were reduced to 8015 instances (units), which composed of the units without verbal assistance (6953 VA–) and the units with verbal assistance (1062 VA+).

3.2 Results

The goal of the present study has been to develop a feature set useful for detecting the need of assistance during haptic exploration and to test it, and not to make a comparative analysis of classification algorithms). The features were tested by using the J48 decision tree algorithm. This algorithm was selected, because it is fast, simple and it can be easily converted to a set of production rules. Also it does not have any priori assumptions about the nature of the data. Weka (Waikato Envirionment for Knowledge Analysis) environment was used to apply the classification algorithm on the dataset [15].

We employed 10-Folds Cross Validation. Table 3 presents the results for each combination of the feature sets. In the first trial, we tested the low-level user action feature set (Set I), the overall accuracy (ACC, the rate of the correctly classified items) was %91.79. Then, we added the features of the SET II. To reveal the contribution of the history window (Feature#8 in Table 2), we used two versions of the feature: (1) the visit count in the last 6 chunks, (2) the visit count in the last 10 chunks. The results showed that adding the high level user actions slightly improved the performance, and increasing the size of the history window was more contributive. In the next run, the low-level graphical features (perceptual) were added (SET III). It improved the overall accuracy of the model by approximately 2 %. As the last test, high-level graphical features (conceptual, SET IV) were added to the model, and this also slightly improved the overall accuracy.

Table 3. The results of J48 algorithm for the feature sets (VA: Verbal Assistance, TP: True Positive, FP: False Positive).
Full size table

Although we obtained high accuracy results, even with SET I only, it should be noted that it is more important for the verbal assistance system to provide verbal assistance when it is needed, therefore correct classification of VA+ (specificity) cases have higher importance. As expected, the results are lower for VA+ cases than VA– cases due to the unbalanced distribution of the data set. Figure 8 illustrates the precision and recall values (presented in Table 3) for VA+ class. The results indicated that J48 performed better with the inclusion of graphical features, resulting improvement from 77 % to 86 % in the precision and from 54 % to 71 % in the recall rates for VA+ cases.

Fig. 8.
figure 8

Precision and recall values for VA+ Class

Full size image

4 Conclusion

In this paper, we introduced a classifier that predicts whether verbal assistance is needed or not by analyzing haptic exploration patterns. The motivation of the study was that in addition to deciding “what to say” as a message carried by verbal assistance system, providing this content on the appropriate time (“when to say”) is also crucial in the implementation of automated verbal assistance system. The feature sets were derived from raw data (the coordinates of the stylus of the haptic device) obtained in an empirical study [5], which studied the production of referring expressions in a joint activity of a haptic explorer and a verbal assistant, namely the exploration of a haptic comprehension by a blind-folded person assisted verbally by an observer.

We employed a (supervised) classification algorithm, namely the J48 decision tree. All feature set combinations (starting from the low level user actions) achieved high accuracy level (> 91.5 %). The best results in terms of sensitivity (true positive rate) and specificity (true negative rate) were achieved by adding the low-level graphical features. On the other hand, adding the high-level graphical features slightly improved the classification. The results are in line with the experimental study [5], which showed that haptic explorers might have difficulty to grasp those conceptual features due to low bandwidth and sequentiality of the haptic exploration movement. Future work should address a detailed understanding the contribution of those conceptual features.

In the experimental study [5], the haptic explorers requested assistance when they needed, and as a response to the request, verbal assistance was provided by the assistant. Therefore both the content and the timing of the assistance were up to the haptic explorer. This resulted in an unbalanced distribution of data points in our data set: The instances without verbal assistance outnumbered the instances with verbal assistance. Therefore in our results, we focus more on the specificity over sensitivity.

Finally, the empirical data showed that 72.4 % of the verbal assistance was requested regarding start and end points. As a consequence, the majority of the data points, where verbal assistance was requested, belonged to a limited region. In future work, we plan to provide sonification to alert the haptic explorer about the start point and the end point of the graph line so that he/she may focus on communicating about the intermediate regions and the changes of the trend without worrying about navigational complexities. That would provide suitable testing environment, in more general, to assess the applicability of the feature set proposed here on the new data-set, and in more specific, to evaluate the contribution of the conceptual features.