The omni-viscosity string is a visual thin thread-like shape displayed on a touch-based tablet device (Fig. 6). The string is designed to produce a type of haptic illusion, where the user would feel stiffness or weight in interacting with displayed objects (Nakakoji et al. 2010, 2012; Yamamoto and Nakakoji 2013).
This section describes the mechanism of the omni-viscosity string and a preliminary user study that uses the string to investigate whether the displayed string communicates weight with users. It then presents the anatomy of the string as a shikake based on our observations from the study.
The mechanism of the omni-viscosity string
As a user touches any part of the omni-viscosity string displayed on a tablet device and drags it with the fingertip, the touched part of the string follows, which then changes the string shape (Fig. 6). The string is assigned a variety of “visual” stiffnesses. When the string is set to the hardest stiffness, the dragged string behaves like an iron wire. When the string is set to the softest stiffness, the dragged string behaves like a silk thread. This is the origin of the name of the string as omni-viscosity.
The design of the string is composed of hundreds of line segments of equal length, each of which connects two adjacent points of a sequence of points. When a user touches and drags a part of the string with the fingertip, the closest point touched by the user follows the location of the fingertip. The locations of the rest of the points are then calculated based on the strength parameter, which is used to express the variety of stiffnesses of the string.
Figure 7 illustrates the algorithm. Let us call the point on the string closest to the touched part P
, the controlled point. Let the adjacent point be P
n+1, and its adjacent point be P
n+2, and so on.
When the user drags the controlled point P
(x,y) to P′
(x,y), its adjacent point P
n+1(x,y) moves to P′
n+1(x,y). The distance and the direction of the movement of P
n+1 are determined by the movement of P
(x,y)) and the strength parameters, and those of P
n+2 are determined by those of P
n+1 and s. The movement of each point is thus propagated from the controlled point toward the two terminal points of the string on the two end points while the distance between each two adjacent points on the string remains constant.
When the strength parameter s is set to 1.0, the distance and direction of the movement of the controlled point P
(x,y) are propagated equally to P
n+2(x,y), and all the other points. Thus, the string keeps the exact shape as the user drags a part of the string on a touch screen. The string feels very hard, similar to an iron wire.
When s is set to 0.0, the propagated movement to adjacent points is a minimum while keeping the distance between the two adjacent points constant. The visual transformation of the string shape then becomes very smooth, and the string feels very soft, similar to a silk thread.
By dynamically changing the strength parameter while the user drags the string, the algorithm demonstrates different degrees of visual viscosity of the string.
A preliminary user study using the omni-viscosity string
We have conducted a preliminary user study to investigate whether a user feels a different weight according to the different visual stiffnesses the omni-viscosity string demonstrates on the tablet display.
Figure 8 shows screen images of the experimental setting we have built and used. In this environment, an omni-viscosity string is initially vertically drawn like a stick in the middle of the display. The bottom of the string is attached to a small area where we asked a study participant to softly or firmly hold it with a fingertip.
Each trial task consisted of two experimental sessions, the base session and the comparison session. In the base session, we asked the participant to firmly touch the small area at the bottom of the string, and in the comparison session, we asked the participant to softly touch the same area.
When each experimental session starts, a gray box slowly moves down from the top of the screen toward the bottom. When the bottom of this box starts to touch the top of the stick-like omni-viscosity string, the string starts to bend as though the box is pushing the string and the string is trying to hold the box up. In about 10 s, the box stops moving down. In the base session, the gray box always stops at the base position (level 3, which is the middle one in the bottom row of Fig. 8). In the comparison session, the gray box stops at a randomly selected level of string bending (levels 1 through 5). Note that how the box moves down on the touch screen is not affected by how firmly or softly the participant holds the bottom part of the string. The box moves down and stops constantly for the base session and randomly for the comparison session, and how the participant holds the string is due only to the instruction we gave to the participant at the beginning of the session. This user study is designed to see whether the changing shape of the held string affects how people behave depending on how firmly or softly they are instructed to hold the string.
We asked participants to engage in 50 trial tasks in a row. In the first session of each trial task, the user is instructed as to how to hold the string (firmly if it is for the base session or softly if it is for the comparison session). As soon as the participant touches and holds the bottom of the string, the gray box starts to come down from the top of the screen and then stops. In the second session, the participant is instructed to hold the string the opposite way (softly or firmly). After the two consecutive sessions, we asked the participant to choose which one was heavier (the first session or the second session). The right-most picture on the upper row of Fig. 8 shows a snapshot of one session. The order of the base and comparison sessions was randomized and balanced for each participant. Each study took about 45–60 min per participant, depending on how quickly a participant made choices.
We collected results on which of the two sessions the participant chose as heavier in each of the 50 trial tasks—the base setting or the comparison setting—and then analyzed the distribution of the levels of the comparison setting.
If there were no visual or behavioral effect at all, the result would be that each comparison setting chosen as heavier would be 50 %, as though they are chosen randomly.
If there were a visual effect (i.e., the different amount of bending of the string) but no effect of the holding strength (i.e., firmly or softly), the result would be that levels 1 and 2 would be chosen more (or less), levels 4 and 5 would be chosen less (or more), and that level 3 would be chosen 50 % because the base session was level 3.
Figure 9 shows the results from the two study runs, where 100 % for each level indicates that the participant said that the base session was heavier in all ten trials in which the comparison session had that particular level. It is interesting to see that in both cases, firmly or softly held, the results show that not only levels 4 and 5, but also levels 2 and 3 are chosen as always or almost always heavier, that is, whether the participant held the bottom of the string firmly or softly affected their choice, in addition to the visual differences.
The result indicates that when the participant sees that a string bends in a certain manner, the participant finds the falling box heavier if the participant is holding the string firmly, and the falling box lighter if the participant is holding the string softly. Note that whether holding the string firmly or softly does not affect the system behavior, and it is merely a matter of feedforward information to the participants (Nakakoji et al. 2011).
With the omni-viscosity string implementation, we have found that visual interactivity has led people to behave differently in making choices regarding which is heavier. This effect is a type of pseudo-haptics, a type of tactile illusion (Lecuyer et al. 2004; Lecuyer 2009). The illusion in haptics has been studied for quite a long time, and the size–weight illusion is well known: if two objects have the same material look and the same weight, people tend to perceive the smaller one as heavier (Ross 1969). With the virtual reality implementation for testing the size–weight illusion, it has been found by using electromyography (EMG) that people in fact not only say that they think the smaller object is heavier, but they use more force on their forearm (Koike et al. 2006). They are changing their behavior through the illusion.
The anatomy of the omni-viscosity string as a shikake
Based on our observation and analysis of how people interact with the omni-viscosity string in the preliminary user study, we have analyzed the string as a shikake from the four shikake elements (Fig. 10).
A large box appears in the upper part of the display of a touch-based tablet device, and one end of a vertical string, the length of which is about one-third of the length of the area, is anchored at the bottom of the display. The box slowly moves down toward the top of the string, and as the bottom of the box touches the tip of the string, the string starts bending as though it is being pushed by the box and is pushing back. A person is instructed to keep touching the bottom of the string either firmly or softly to support the string. How the person touches the string is actually ignored and not taken into account for what is displayed, but the person is not told that. The box stops falling at a certain amount of bending of the string.
The person who is instructed to keep holding the bottom of the bending string firmly or softly as the box falls down feels the weight. The more the string bends, the more weight the person feels. The person who keeps touching the bottom of the string firmly feels more weight, even if the amount of bending is not so much.
The person, who is asked to keep holding the string, feels more weight with the increased bending of the string, which is displayed as though the string is being pushed by the falling box and is pushing back. The person feels more weight if instructed to hold the string more firmly, although the amount of bending of the string remains the same.
The display is of a box falling from the top and a short vertical string from the bottom appearing to be supporting the box with different amounts of falling and bending. The instruction to the person is to keep touching the bottom of the string either firmly or softly to hold the string, which makes the person feel different weights as the result of pseudo-haptics, a type of tactile illusion. The person feels weight differently by changing the visual stiffness of the string, as well as by changing the instruction to keep touching the bottom of the string either firmly or softly. The pseudo-haptics comes from the nature of human perception, in which vision dominates touch, and the perception of weight depends on the feedforward mechanism.