Finding Objects Faster in Dense Environments Using a Projection Augmented Robotic Arm

  • Hind GacemEmail author
  • Gilles Bailly
  • James Eagan
  • Eric Lecolinet
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9298)


Locating an object in an unfamiliar and dense physical environment, such as a control room, supermarket, or warehouse, can be challenging. In this paper, we present the Projection-Augmented Arm (PAA), a motorized robotic arm augmented with a pico-projector to help users to localize targets in such environments. The arm moves and displays a projected spotlight on the target. We present the results of a study that shows that the PAA helps users to more quickly locate target objects in a dense environment. We further study the influence of the visibility of the projected spotlight while moving versus that of the physical movement of the projection arm on user performance and search strategy, finding that (1) information about the orientation of the arm has a stronger impact on performance than moving spotlight projected on the search space; (2) the orientation of the arm is useful (24 % improvement) and especially when the target is behind the user (26 % improvement); and (3) users’ strategies relied mainly on the arm when it is visible.


Guidance techniques Augmented arm Steerable pico-projector 

1 Introduction

Finding and locating physical objects can be challenging, especially in dense environments such as control rooms, supermarkets, warehouses, etc. These environments may contain several hundreds or even thousands of objects, which may look similar and be spread about the space, including behind the user. Even when the user knows what the target looks like (e.g. color, shape, size, etc.) and has a general idea of the organization of the space, finding an object in such an environment is already difficult. Even in such situations, one might look several times over several areas before finding the target, such as when looking for, say, a yoghurt pot in the refrigerator. For unfamiliar targets or unfamiliar environments, locating an object can be even more challenging. Generally, one must first orient oneself to find the general vicinity of the target and then perform a visual search on this area to pick out the target.

Although this localization task occurs frequently in many application areas (maintenance, training in control rooms, libraries, warehouses, etc.), proposed guidance techniques for such environments [2, 10, 19, 20, 21] focus separately on how to help the user to orient himself or how to show him precisely the location of the target, but not both.

Existing work mainly focuses on outdoor way-finding and guidance techniques for people with visual impairments [4, 5], for small environments [14, 18, 22, 23] and/or guidance techniques requiring instrumenting users (e.g. with a head-mounted display (HMD)) [12, 29], which may be cumbersome, expensive, and sometimes incompatible with other constraints. Our goal is to provide guidance support to help users to orient themselves and to precisely locate targets in dense environments, such as controls in a control room, by augmenting only the environment and not the user.

In this paper, we introduce the Projection Augmented Arm (PAA), illustrated in Fig. 1b. It consists of a motorized arm with a pico-projector mounted on the end to project a spotlight on a given target. It can be mounted to a cart (e.g. a shopping cart in a supermarket or to a delivery cart in a warehouse) in front of the user (Fig. 1b) or handled by the user to allow multiple usages. The design of the PAA draws inspiration from the way that humans gesture toward targets by pointing. An observer can easily recognize the general direction of the target without looking directly at the arm of the person pointing it out. However, because pointing at an object is insufficiently precise, we extend the arm with a projected spotlight, as if the person were carrying a flashlight in her hand (Fig. 1a).
Fig. 1.

(a) The metaphor of the PAA: a user points to a specific component on the wall, which contains several similar targets. The observer follows the direction of the user’s pointing arm, which is augmented with a projector to precisely highlight the target. (b) The PAA prototype. A user fills out a form on a rolling table, on which an automated arm is fixed and projects toward the desired target on top right of the wall.

The design of the motorized arm (1) offers a large projection surface: the projector can target anywhere in area from 0 to 300-degrees horizontally and vertically. Moreover, by placing the arm within the user’s field of view, (2) the general direction of the arm remains visible in the user’s peripheral vision, thus (3) reducing the search area by orienting the user toward the desired object while (4) the spotlight precisely indicates the location of the target. Moreover, the projector could display additional information around the target, using the same device. It is straightforward to attach the system to a cart, making it (5) mobile. Further, it (6) alleviates occlusion problems and (7) can be less intrusive than guidance techniques requiring users to wear additional devices such as glasses or HMDs. Finally, (8) PAA may be visible by several users simultaneously when performing a collaborative task.

We conducted a user study to better tease out the impact of having a physical pointer perceivable in the periphery on the performance on object localization tasks. More precisely, we investigated the influence of the projected spotlight versus the physical movement of the projection arm on user performance and search strategy. Results show (1) information about the orientation of the arm has a stronger impact on performance than moving spotlight projected on the search space; (2) the orientation of the arm is useful (24 % improvement) and especially when the target is behind the user (26 % improvement); and (3) users mainly relied on the arm for guidance in conditions where it was visible.

Our primary contributions are:
  • An application of the metaphor of the way humans use pointing to orient each other for improving guidance techniques.

  • The design and implementation of PAA, a mechanical, motorized pico-projector for localizing physical targets in dense environments.

  • Findings of a user study investigating the influence of the projected light while moving versus the physical movement of the projection arm on user performance.

2 Informative Study

Our primary context of interaction is control rooms of nuclear plants. To better understand the specificities of this environment, we conducted both observations as well as interviews with five domain experts.

We learned that these control rooms are quite large and contain numerous panels densely filled with buttons, gauges, and alarm indicators that are difficult to quickly distinguish, especially for novice users (typically, people performing maintenance or when training novice or experienced operators on a new type of control room). The environment can be noisy and is manned by several personnel who move about the room. Moreover, operators generally bring with them a wheeled high-top table to transport documents.

Regarding the task, we learned that operators typically follow a paper procedure document, which references the appropriate controls. Paper is used rather than an electronic device for safety and verifiability reasons. This document is generally placed on a high-top table cart, which also serves to store auxiliary documents. For each step of the procedure, operators first move with the cart to the position of a specific element in the room and then take notes according to the state of this element on the paper procedure (e.g. to mark that a button has been pressed, report the value of a sensor, etc.).

We thus derive several requirements for our guidance system:
  1. 1.

    Users should be able to move about freely. Not only must they roam around the control room, but their hands must also be free to manipulate pens and various documents (they may have to find supporting information in other documents, including large books with complex diagrams).

  2. 2.

    Users’ senses should not be obstructed. In particular, audio does not seem an appropriate modality in such a noisy environment and in which operators frequently communicate.

  3. 3.

    Glasses and HMDs are problematic for the same reason. The operator must maintain an environmental awareness and be able to rapidly detect any visual alarm, regardless of the panel it is displayed on. Moreover, operators often work in pairs, and, occasionally, in larger groups. It is thus convenient and advantageous to use equipment that several people can see simultaneously. This is also less constraining for the people working permanently in the control room (they do not have to wear a device all day long) and for those working in several rooms (they do not have to put on and take off a device multiple times). Augmenting a cart or wheeled table offers a straightforward solution to these problems.

  4. 4.

    The guidance system should accommodate multiple operators. In particular, some operators may temporarily occlude some elements with their body. Moreover, the guidance system should minimize the disruption of other operators. Hence, on the one hand, people working together should all be able to see the system, but, on the other hand, other people should not be bothered by the system.

  5. 5.

    Due to the size of the room and the density of the elements, the guidance system should be precise enough to unambiguously identify a single 4 cm × 3.5 cm control in a control panel.


Because of this specific context of use, we focused the design of the PAA for command and control rooms. Nonetheless, we believe our guidance approach can be readily generalizable to other contexts such as a supermarket, a warehouse or a library. As in our case, these are dense environments where it is difficult to augment objects and where users often use carts (either a shopping cart, a library shelving cart, or a warehouse delivery cart).

3 Related Work

3.1 Guidance Techniques

Guidance techniques have been proposed for a variety of purposes. Generally, they can be categorized in terms of the granularity of the guidance they provide: from helping the user to navigate within a space (e.g. by moving one’s feet) to helping her to orient herself (e.g. turning in place) [12] to precisely locating a target in front of the user (e.g. visual search) [2, 27]. The PAA focuses on these last two categories.

Body/Head Orientation. Several techniques have been proposed for orienting the user toward a given direction using different modalities. For example, Yamano et al. [32] rely on sound (frequency or amplitude panned sound), while Erp et al. [8] rely on haptics by using a vibrating waist. As our technique should work in noisy areas and allow for precise localization, we focus on visual guidance techniques (e.g. [12, 20]).

Precise Guidance. Techniques generally help the user to find a target that is already in his/her field of view. Again, the use of audio [27] and haptics [18] has been considered, but vision is more appropriate in our case as in [2] where a steerable projected spotlight highlights a target. The projector is, however, ceiling-mounted, outside the view of the user. As such, it provides precise guidance, but does not provide orientation guidance to help a user who might be facing in the wrong direction.

Few techniques support both of these guidance granularities. Henderson and Feiner [12] use left/right arrows displayed on augmented reality goggles to help orient the user. Once the target is in the field of view of the user, a 3D virtual arrow directly points to the target in the glasses, providing for precise guidance. This technique is promising, but requires the user to wear additional equipment, notably augmented reality glasses. Gröhn et al. [9] combine 3D sound to help orient the user with a spotlight to precisely locate a target, but the use of sound precludes this approach from noisy environments in which the user cannot wear headphones. For the reasons explained in Sect. 2, our goal is to provide guidance without requiring the user to wear special equipment. Control rooms may have strict safety requirements that prevent the use of such equipment, as in our direct case. In supermarkets or libraries, providing clients with a cart is feasible whereas requiring them to share wearable devices may not be.

3.2 Projector-Based Augmented Reality

Based on these considerations, we now focus on projector-based techniques that do not require the user to wear any special equipment such as augmented reality glasses or Google glasses. From the literature review, we have identified two main dimensions: whether the projection system is motorized or not, and whether it is fixed (the environment is augmented) or not fixed (the user or environment are augmented) (Table 1).
Table 1.

Motorized vs. Fixed projection systems.


Motorized projection system



Fixed projection system


[2, 7, 25, 26, 31]

[6, 15]


PAA, [3, 16]

[11, 24, 30]

Fixed Projectors Without a Motorization System. Single-projector solutions can only project on a limited surface area. They may also be susceptible to occlusion if a person or object passes between the projector and the projection surface. To overcome these limitations, Jones et al. [15] enlarge the projection area and minimize occlusion problem by deploying several fixed projectors in a room with shared projection areas. Ehnes and Hirose [6] proposed to minimize the degradation of projection quality by selecting the projector that is closest to the projection surface and that has the best orientation. However this approach may be costly and hard to implement for complex and dense environments composed of many projection surfaces such as in supermarkets or control rooms.

Fixed Projectors with a Motorization System. Pinhanez [25] proposed using a steerable mirror mounted on a fixed projector to enlarge the projectable surface of a single projector. Some other systems [2, 26, 31] can also augment moving objects, possibly in real-time [7]. However, these solutions suffer from possible occlusion problems because they rely on a single projector at a predefined position. They may also introduce degradation in the projection quality [6] because of the distance and the orientation (for non-perpendicular angles) between the projector and the projection surface. Moreover, because of their fixed location, these platforms will not be always directly visible to the user if he moves freely. The user must then follow a projected spotlight to be aware of its location. Keeping track of the moving spotlight may be problematic (1) if another task occupies the attention of the user (he may be doing something else while the spotlight moves), (2) the projection passes over a light-absorbing surface (such as a computer screen) and becomes barely visible, or (3) the projection moves too rapidly, especially if the target is outside the user’s field of view.

Mobile Projectors Without a Motorization System. When fixed on the user’s body [11], such techniques have the advantages to support user mobility using a single and inexpensive projector. However, because their orientations and height are fixed, they have a limited available projection area. Ota et al. propose using several projectors to solve this problem [24], but this solution tends to be cumbersome and expensive and still does not allow projecting in all directions. Hand held mobile projectors are another solution [30], but using them may be tiring, they suffer from unsteady projection (especially when the users move), and they may disturb the user’s attention (who must worry about how to keep the projection steady). However, such solutions may improve projection quality because the projector can be moved close to projection surface.

Mobile Projectors with a Motorization System. In contrast with the previous category, motorized projectors allow projecting anywhere around the user. Either the user adjusts the projector direction [3], or it is automatically controlled in real time by a computer to improve steadiness [16], such as by mounting the motorized camera-projector on the backpack of the user and above the shoulder height. Hence, contrary to our own system, the projector is not visible by the user because it is located behind her head. As will be seen in the following sections, this is a major difference in the approach. Our key insight is to exploit the physical movement of a motorized projection arm to help guide the user to the target. We thus study the impact of using a projected spotlight from an out-of-view movable projector versus the physical movement of a projection arm on user performance and search strategy. Obviously, the user cannot benefit from this alternate modality (the projection arm) without seeing the device.

4 PAA: Projection Augmented Arm

We present PAA, a Projection Augmented Arm consisting of a pico-projector fixed to a motorized arm that is attached to a wheeled high-top table (Fig. 1). The user selects the name of an object on a piece of paper (such as by pointing with an Anoto pen) to indicate to the system which object he is looking for. The system extracts the physical location of this object in the room from a database and (1) moves the motorized arm to make it point towards this object and (2) highlights this object by a spotlight emitted from the pico-projector. In the current implementation, arm movement time is under 1 s, with a standard deviation of pointing error of approximately 4 mm at a 2.5 m distance. As explained later, the spotlight can either be shown while the arm is moving or only once it is oriented to the proper direction. The orientation of the arm (hence the projector) is automatically updated when the table is moving.

4.1 Properties

PAA has several advantages:

Projection Area. PAA offers a large surface of projection thanks to its capability to orient itself. In practice, the projector can target anywhere in an area of 0 to 300-degrees horizontally and vertically.

Precision. Standard motorized projectors attached at a fixed position (typically, the ceiling) cannot provide a high resolution when the targets are far from the projector. Moreover, a small error in the orientation of the projector can shift the spotlight from a couple of centimeters from the target and highlights another target. In contrast, PAA is movable: when the operator is in the vicinity of the target, the projector is also in the vicinity of the target: this increases the resolution of the projected surface and reduces the risk of shift. Finally, PAA reduces occlusion problems that can occur with fixed projectors.

Pointing Metaphor. We further use the metaphor of a human arm pointing towards a target. Although imprecise, people have a lifetime of experience following such gestures. With PAA, users can look at the system to get an idea of the location of the target. This is especially useful when users do not know where the target or the spotlight are located (e.g. when the spotlight is behind users). As the augmented arm is in the close vicinity of the users (see Fig. 1b), users can get an idea of the location of the target by looking at the direction of the arm. In fact, they can even use their peripheral vision without explicitly looking at the arm. This differs from fixed projectors, which are not easily visible, especially if fixed to the ceiling. Users can thus maintain their attention on their primary task. Moreover, a physical 3D pointer should be easier to follow than a 2D one, such as an arrow displayed on the top of the cart. The PAA thus provides general and precise guidance with the same device and therefore it is less expensive compared to a system using several display devices.

Spotlight. The spotlight is a useful visual cue to help users to localize the targets not only when focused on the target but also when the arm is moving. A large, dynamic and colorful spotlight can help users to follow it when the arm is moving. However, a moving spotlight can also disturb the other people located in the room. As said in the Informative Study section, the guidance system should be both easy to follow and minimally intrusive so as to minimize the disruption of other operators. Augmenting the arm with projection addresses this problem by providing an alternate modality. We considered a simpler approach of using a laser pointer, but laser pointers are restricted to a limited size. With a projector, we can vary the size of the spotlight, to make the spotlight visible when it is projected on distant targets, or show additional information around the target. It can even be switched off while the arm is being moved.

Our user study aims at better understanding the respective impact of the arm and the spotlight on visual search performance. One potential advantage of the arm compared to a moving spotlight is that the user does not necessarily need to follow it while it is moving: The user can be doing another task while the arm is moving and only look at its final position when she is ready. Thanks to the pointing metaphor, she will still know where to search the object.

Attachment. We primarily designed PAA to be attached on a table/cart. Using a table to carry the device frees the user from having to wear any special equipment that may cover the eyes or ears. Such a system could also provide sound feedback or haptic feedback. However, we have also imagined other scenarios involving a handheld version of the system that could provide haptic force feedback resulting from the acceleration of the PAA, subtly suggesting the direction of movement of the arm. In the future, when the servomotors and projectors become small enough to fit in the hand, the users could sense the orientation of the arm just by touching the arm.

4.2 Implementation

Hardware and Software. The PAA prototype hardware consists of five primary components: two Dynamixel AX-12A robotic actuators, mounted in series, a Philips PicoPix 3610 projector, an Arduino-based controller, a laptop computer and an ARTTrack 3D motion tracking [1]. The ARTTrack is composed of 8 IR cameras connected to a controller, which estimate the coordinates of tracked objects at 60 fps with a precision of about 0.05 mm on the position and 0.0015 rad on the orientation.

Additionally, the prototype is target-aware, meaning that the system maintains a list of all of the possible targets and their coordinates and a calibrated model of the room. This model is composed of six equations of the walls of the room. To compute these equations we measured manually the dimensions of the walls and their distance from the origin of an absolute referential (R0) (Fig. 2a). In more complex environments, it is possible to use approaches based on depth cameras to build the 3D model of the projection surfaces that compose the environment [13]. The problem with such approaches is the limited precision of the obtained models.
Fig. 2.

(a) The distorted projection is corrected from S to and target Pt position is computed relative to the referential related to . (b) The actual PAA prototype and its components.

For a given target, a controller running on the laptop looks up the target’s position in this model and queries the ARTTrack [1] tracker for the position and orientation of the projector. From these positions, it calculates the necessary movements and relays the appropriate heading and tilt angles to the Arduino control board, which drives the robotic actuators.

The ARTTrack follows the position of the projector during movement, creating a feedback loop to ensure that the projector is properly aimed at the desired target in real time. Finally, the laptop calculates the appropriate affine transform to compensate for any keystone effects introduced by projecting at a non-perpendicular angle. Using the current prototype, the projector can target an object within 300-degrees horizontally and vertically, in under a second.

Operating of PAA. By using a pre-calibrated model of the room and its targets (P t/R0 ), the PAA can dynamically adapt its keystone correction to project an undistorted spotlight on a given target, even while the PAA is moving. This pre-calibration allows us to use a simpler tracking model that does not require dynamic recalibration [17] or a separate camera to correct the distortion [28, 31]. The coordinates of the projector, the target, and the equation of projection plane it is on suffice.

When selecting a new target, the PAA performs up to two simultaneous angular movements (heading and tilt) to bring the target within the projection range. Closed-loop tracking enables the system to dynamically compensate if the PAA over- or undershoots the target (e.g. due to inertia).

Control of the Arm Angular Movement. In order to determine the necessary angular movements to update the arm’s position, we first use the measurements of the coordinates of the projector expressed in terms of an absolute room referential, R 0 . A transformation, 0 T L , maps between R 0 and the projector lens referential, R L . Then we compute the position of the target in lens-coordinates, P t/RL . Then, we compute the spherical coordinates of the target and thus correspond to the angular deviations between the PAA and the targets, ρ, Δϕ, Δθ. In our setup, the roll angle in R 0 is fixed to zero for the projector and for the targets. Finally, we compute the input controls to the servomotors, C h , C t . Two constants, k h and k t represent gain for both tilt and heading control loops.
$$ P_{{t/R_{L} }} = ({}_{ }^{0} T_{L} )^{ - 1}. P_{{t/R_{0} }} $$
$$ C_{h} = k_{h}. \Delta \theta $$
$$ C_{t} = k_{t}. \Delta \phi $$
Computation of Target Location in the Projection Area. We dynamically compute the position of the target, P t/S’ , in the referential of the undistorted projection area, S’, where l T S’ is the transformation between the projector lens referential and S’. Each time, the target comes within S’, we compute its position \( P_{I'} \) in the undistorted OpenGL frame.
$$ P_{t/S '} = ({}_{ }^{0} T_{L}. {}_{ }^{L} T_{S '} )^{ - 1}. P_{{t/R_{0} }} $$

Distortion Correction. To provide precise guidance, the PAA projects a spotlight, which has the same shape as the target (rectangular) and with fixed size regardless of the orientation and distance between the target and the projector.

To correct any keystone distortion, we first estimate S, to do that we compute the intersection points P = (p 1, p 2, p 3, p 4) between projection plan equation and unit vectors (\( \overrightarrow {{z_{1} }} \), \( \overrightarrow {{z_{2} }} \), \( \overrightarrow {{z_{3} }} \), \( \overrightarrow {{z_{4} }} \)) pointing in the direction of the light beam emitted from the projector (Fig. 2a), the components of these vectors are function of the orientation of the projector (Φ, θ) and the projector characteristics (α, β) (Fig. 3). Then we compute Sʹ, the maximal rectangular area included in S. We then compute homographic transformation H between S and the original image I, using A and B: the matrix based on P that delimits S, and the coordinates of the corners of I:
$$ H = A^{ - 1} . B $$
Fig. 3.

Left: side view of the projector with tilt angle ϕ and vertical focal length β. Right: top view with heading angle θ and horizontal focal length α.

Finally to compute the new location of a given point P I in the undistorted image, we apply the following formula, using the (x, y) coordinates in the referential of the undistorted projection area:
$$ P_{I '} = \left[ {\begin{array}{*{20}c} {H_{1,1} - H_{3,1} .x} & {H_{1,2} - H_{3,2} .x} \\ {H_{2,1} - H_{3,1} .y} & {H_{2,2} - H_{3,2} .y} \\ \end{array} } \right]^{ - 1} . \left[ {\begin{array}{*{20}c} {x - H_{3,1} } \\ {y - H_{2,3} } \\ \end{array} } \right] $$

H i,j is the element of H 3 × 3 matrix at the i line and j column.

5 User Study

Our intuition is that, by making the motorized projection arm visible, we can help the user to more quickly narrow down the search space for the target. In this user study, we try to identify the relative contributions of being able to see the arm versus a moving spotlight that also guides the user to the target. As mentioned earlier, the PAA provides several cues to guide the user’s search, including the movement of the spot or the direction of the arm. In this laboratory study, we wanted to investigate two of these properties by controlling (1) the visibility of the arm and (2) the visibility of the spot while moving the arm. The four resulting techniques are illustrated on Table 2.
Table 2.

The four guidance techniques we explore in this study. They are organized according to two dimensions: the visibility of the arm and the visibility of the spotlight while moving the arm.


Visibility of the spot while moving



Visibility of the arm






Arm & spot

Arm & Spot is the PAA technique: Users can look at the direction of the arm and follow the spotlight to help locate the target. The Arm and Spot techniques derive from PAA by removing one property: Arm does not let users follow the spotlight (which only appears once the final direction of the arm has been reached) while Spot does not let users look at the direction of the arm. The Baseline condition completes our design space so that we can precisely understand the relative contributions of the arm and the spotlight.

5.1 Participants and Apparatus

12 participants (6 female) aged from 20 to 38 (M = 27.83, SD = 4.8) were recruited from our institution. They were compensated for their participation with candy.

Room. The study was conducted in a room (5 × 5m2) under standard overhead lighting condition. The room was equipped with an ARTTrack tracking system [0] to track the position and orientation of the pico-projector and the user. Figure 1b shows the room.

Targets. Each wall contained 8 paper panels of 20 targets (total = 480) to mimic the real-world control dials in a control room. We used the three walls to compare three conditions: Targets located in front, behind, or on the side of the participant (Fig. 4). Each target was a rectangle (4 × 3.5 cm2) containing an id of two characters (letter + digit). The font size was 64 to ensure that each participant could easily read them. The distance between targets of 4 cm was superior to the precision of the system.
Fig. 4.

Left: map of the room; Right: example of a target part on a panel

Techniques. Each of the four tested techniques relies on the same hardware presented above to avoid possible confounded variables (e.g. inertia of the arm): PAA was installed in the center of the room as shown on Fig. 4, with the participant located on its left side. The spotlight has the same shape as a target. We fixed its size to be big enough to be visible from across the room yet not overlap the objects in the vicinity of the target. We also wanted to maintain a small enough size to avoid being overly intrusive while moving. We used the same size and shape of spotlight in all conditions to ensure the same level of difficulty and to avoid to biasing the results. This included correcting for distortion and resizing the spotlight during movement to compensate for the effects of non-perpendicular projection angles on the wall (Φ ≠ 0 or θ ≠ 0) (see Fig. 3). In the Arm condition (spot not visible), the spotlight was switched off during arm movement. In the Spot condition (arm not visible), participants wore basketball dribble glasses that prevented them from looking down at the arm. Finally, participants wore a hat to track head position as well as earphones with a gentle white noise to ensure that they could not hear the sound of the servomotors, which could serve as a hint about relative distance to the target. This was done to prevent confounding factors and to be in accordance with our primary context (control rooms) which is noisy and where users cannot rely on auditory feedback.

5.2 Task and Procedure

The task consisted of finding a target as quickly and accurately as possible. Participants started a trial by pressing button. The arm then moved from its current position (that of the last target) to highlight the next target. Participants then searched for the target on the three walls by using the arm or the moving spot depending on the conditions. As soon as the participants identified the target, they pressed the same button to stop the trial, at which point the spot disappeared and participants orally indicated the ID of the target to the experimenter. Participants could take a short break between each trial and between each block. Participants were videotaped during the experiment. After the experiment, they answered a questionnaire and were debriefed via a semi-structured interview.

5.3 Design

We used a within-participants design: each participant tested the four techniques. The order of techniques was counter-balanced using a Latin square design. Participants performed two blocks per technique. Each block contained 25 different target selections (from 460 targets). The location of the targets was randomized in order to appear either on the wall in front of the user, on a lateral wall on his side or on the wall behind him. In summary, the design of this study was: 12 participants × 4 techniques (baseline; arm; spot; arm&spot) × 2 blocks × 25 targets = 2400 selections.

6 Results and Observations

6.1 Completion Time Analysis

We assumed the normal distribution of our data (Shapiro-Wilk test) and we used a three way ANOVA which showed a significant effect on completion time of the location of the target relative to the initial orientation of the participant (F2,22 = 225.17, P < 0.001). Post hoc Tukey tests revealed that this effect is significant for each pair of relative locations between the user and the target. Indeed, the search time was faster if the target was: on the wall in front of the participant (1.6 s) rather than on the wall behind him (2.6 s, 38 % improvement); on the front wall rather than on the lateral wall (2.4 s, 33 % improvement); and on the lateral wall (2.4 s) rather than on wall behind (2.6 s, 8 % improvement).

ANOVA also showed a significant effect of the visibility of the arm on completion time (F1,11 = 91.79, P < 0.001) (see Fig. 5). Indeed, participants were faster when they could see the arm (1.9 s) than when it was not visible (2.5 s).
Fig. 5.

Mean completion time (ms) according to the visibility of the arm and the visibility of the spot while moving.

ANOVA also revealed a significant interaction effect between the visibility of the arm and the relative location of the target from to the initial orientation of the user (F2,22 = 5.20, P < 0.05) with a highest improvement (26 %) when the target is behind the user (see Fig. 6). Post hoc Tukey tests confirmed a significant effect for all possible interactions between visibility and relative location, except in two cases: (a) when the arm is visible, for lateral (2.0 s) vs. behind (2.2 s), locations; (b) for lateral locations when the arm is visible (2.0 s) vs. for front locations when the arm is not visible (1.9 s). For the remaining cases, we found significant decreasing completion time with increasing location distance whether the arm is visible or not.
Fig. 6.

Completion time according to the visibility of the arm and the relative location of the target from the orientation of the user (behind, close to (on the lateral wall), and in front of the user).

Finally, there was no significant effect of the visibility of the spotlight while moving (See Fig. 5).

6.2 Observations and Questionnaire

The participants rated the ease of the search task for each technique. They preferred the two conditions where the Arm was visible. Then, for each case (Arm visible and Arm not visible), they preferred having a spotlight (Fig. 7).
Fig. 7.

Subjective feeling of easiness for the four conditions (7 corresponds to very easy /1 to uneasy).

We also made the following observations:

When the arm is visible, users seem to orient themselves faster from the beginning to the end of the search task and stop on the correct direction without exceeding the target. When the arm is hidden and the spot visible, users start moving rapidly but become slower at the end.

Unsurprisingly, when the arm and the spotlight are not visible, users seem to be slower than in any other condition at all stages of the searching task. They also occasionally go on the wrong direction, which never happened when both the arm and the spot were visible.

All participants stated that they were aware about the movement of the arm in their peripheral field of view. Some also said they just followed the arm automatically, without thinking.

The interviews show that users rely on different strategies depending on whether the arm and the moving spot are visible or not. Surprisingly, more than half of the participants (55 %) reported that, when the arm was visible, they did not care about the spotlight while it was moving. However, they all tried to follow the spotlight when the arm was not available.

7 Discussion

Moving Arm. Results confirm our hypothesis that the visibility of the arm reduces completion time. However, the results are surprisingly good with a decrease of 24 % in average, regardless of the visibility of the moving spotlight and relative location of the target. The visibility of the arm is especially useful when the task is complex (i.e. the target is not in front of the user). Participants provided several possible explanations: The arm provides an efficient visual cue for localizing the general direction of the targets, presumably because the movement could easily be perceived in the peripheral field of view. This explains why users relied on the arm even when the spotlight was visible. When the arm was visible, participants always oriented their body in the correct direction without overshooting the target. As a result, they did not perform unnecessary movements, which could be time consuming and tiring. This effect could explain the completion time gain. Moreover, several participants mentioned that they were aware of the direction of the arm even without looking at it thanks to the use of their peripheral vision. This is especially useful in scenarios where users have to share their attention (for instance between a paper procedure and verifying and activating controls).

Moving Spotlight. We expected that the introduction of the moving spotlight would reduce the completion time. However, our results did not reveal a significant effect regardless the conditions (visibility of the arm and location of the targets). Participants explained that the speed of the spotlight was too fast and that they had some difficulties to track it. It results that some participants sometimes tried to anticipate the movement of the spotlight and overshoot the target. As a consequence, the spotlight was not as useful as expected. It would have been possible to use a larger spotlight or reduce its speed. However, both have some drawbacks. Increasing the size of the spotlight can disturb other users (operators in nuclear plants; consumer in a supermarket) working in the same environment. Reducing the speed of the spotlight would also reduce the speed of the arm, which was not judged too fast. When the moving spotlight was combined with the moving arm, participants explained that they primarily used the arm because they found it more useful.

Finally, in light of the obtained results, it would be interesting to explore several potential optimizations: depending on the density of the projection area and whether it is shared by other users or not, the speed of the arm and the size of the spot may be controlled dynamically during the movement of the arm. For example, if it is possible to model shared and non-shared spaces, the moving spotlight could be made bigger when moving the arm so as to allow the user to more easily following it, and to reduce its size in shared areas. Also, the motors could adjust their speed based on the difficulty of following the spotlight on the particular projection surface. In the future when the servomotors and projectors become smaller and the PAA could fit in the hand, we could explore the impact of haptic feedback on localization performance.

8 Conclusion and Future Work

We have designed, implemented and evaluated a novel guidance technique, PAA, which is inspired from the way humans perform in air pointing gestures with their arms to orient each other. Moreover, this technique also relies on a projection system to allow precise indication of the target. Our user study provides two main findings: (1) the moving arm helps users to localize the target especially when the task is complex (i.e. the target is not in front of the users) because users can quickly get a raw idea of the direction of the target: the arm is always localized at the same location, in the peripheral vision of the user. In contrast, (2) the moving spot does not provide real benefits because it requires a lot of attention to track it. A challenge would be to design a spotlight easier to track without disturbing other operators and impairing the performance of the arm.

In the present study, we focused on a fully automated PAA: the system controls the direction of the arm. As a future work, we plan to investigate scenarios where users (partially) control the arm and how shared control will impact visual search performance. Moreover, we plan to evaluate PAA with multiple operators (and possibly multiple PAAs) to better understand the impact of our system on cognitive load, global awareness and concentration. We also plan to investigate the impact of 3D pointing using the arm vs. 2D pointing using a display to indicate a direction in 3D space.



This work is supported by the French ANR/Investissement d’Avenir “Cluster Connexion” and by the ÉQUIPEX DigiScope ANR-10-EQPX-0026.


  1. 1.
  2. 2.
    Butz, A., Schneider, M., Spassova, M.: SearchLight – a lightweight search function for pervasive environments. In: Ferscha, A., Mattern, F. (eds.) PERVASIVE 2004. LNCS, vol. 3001, pp. 351–356. Springer, Heidelberg (2004)CrossRefGoogle Scholar
  3. 3.
    Cauchard, J.R., Fraser, M., Han, T., Subramanian, S.: Steerable projection: exploring alignment in interactive mobile displays. Pers. Ubiquitous Comput. 16, 27–37 (2012)CrossRefGoogle Scholar
  4. 4.
    Cosgun, A., Sisbot, E.A., Christensen, H.I.: Evaluation of rotational and directional vibration patterns on a tactile belt for guiding visually impaired people. In: 2014 IEEE Haptics Symposium (HAPTICS), pp. 367−370 (2014)Google Scholar
  5. 5.
    Dramas, F., Oriola, B., Katz, B.G., Thorpe, S.J., Jouffrais, C.: Designing an assistive device for the blind based on object localization and augmented auditory reality. In: Proceedings of the 10th International ACM SIGACCESS Conference on Computers and accessibility, pp. 263−264. ACM (2008)Google Scholar
  6. 6.
    Ehnes, J., Hirose, M.: Projected Reality - Enhancing Projected Augmentations by Dynamically Choosing the Best Among Several Projection Systems. In: IEEE Conference on Virtual Reality, pp. 283−284 (2006)Google Scholar
  7. 7.
    Ehnes, J., Hirota, K., Hirose, M.: Projected augmentation - augmented reality using rotatable video projectors. In: IEEE & ACM ISMAR 2004, pp. 26−35 (2004)Google Scholar
  8. 8.
    Van Erp, J.B.F., Van Veen, H.A.H.C., Jansen, C., Dobbins, T.: Waypoint navigation with a vibrotactile waist belt. ACM Trans. Appl. Percept. 2(2), ACM (2005)Google Scholar
  9. 9.
    Gröhn, M., Lokki, T., Takala, T.: Comparison of auditory, visual, and audiovisual navigation in a 3D space. ACM Trans. Appl. Percept. 2(4), 564–570 (2005)CrossRefzbMATHGoogle Scholar
  10. 10.
    Harada, S., Takagi, H., Asakawa, C.: On the audio representation of radial direction. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 2779−2788. ACM (2011)Google Scholar
  11. 11.
    Harrison, C., Benko, H., Wilson, A.D.: OmniTouch: wearable multitouch interaction everywhere. In: UIST 2011, pp. 441–450. ACM (2011)Google Scholar
  12. 12.
    Henderson, S.J., Feiner, S.: Evaluating the benefits of augmented reality for task localization in maintenance of an armored personnel carrier turret. In: IEEE ISMAR 2009, pp. 135–144 (2009)Google Scholar
  13. 13.
    Henry, P., Krainin, M., Herbst, E., Ren, X., Fox, D.: RGB-D maping:using kinect-style depth cameras for dense 3D modeling of indoorenvironments. IEEE Int. J. Rob. Res. (IJRR) 31(5), 647–663 (2012)CrossRefGoogle Scholar
  14. 14.
    Ishii, K., Yamamoto, Y., Imai, M., Nakadai, K.: A navigation system using ultrasonic directional speaker with rotating base. In: Smith, M.J., Salvendy, G. (eds.) HCII 2007. LNCS, vol. 4558, pp. 526–535. Springer, Heidelberg (2007)Google Scholar
  15. 15.
    Jones, B., Sodhi, R., Murdock, M., Mehra, R., Benko, H., Wilson, A., Shapira, L.: RoomAlive: magical experiences enabled by scalable, adaptive projector-camera units. In: UIST 2014, pp. 637–644. ACM (2014)Google Scholar
  16. 16.
    Kratz, S., Rohs, M., Reitberger, F., Moldenhauer, J.: Attjector: an attention-following wearable projector. In: Kinect Workshop at Pervasive (2012)Google Scholar
  17. 17.
    Lee, J.C., Dietz, P.H., Maynes-Aminzade, D., Raskar, R., Hudson, S.E.: Automatic projector calibration with embedded light sensors. In: UIST 2004, pp. 123–126. ACM (2004)Google Scholar
  18. 18.
    Lehtinen, V., Oulasvirta, A., Salovaara, A., Nurmi, P.: Dynamic tactile guidance for visual search tasks. In: UIST 2012, pp. 445–452. ACM (2012)Google Scholar
  19. 19.
    Li, M., et al.: ProFi: design and evaluation of a product finder in a supermarket scenario. In: UbiComp 2013, pp. 977–984. Adjunct (2013)Google Scholar
  20. 20.
    Li, M., Arning, K., Sack, O., Park, J., Kim, M.-H., Ziefle, M., Kobbelt, L.: Evaluation of a mobile projector-based indoor navigation interface. Interact. Comput. 26(6), 595–613 (2013)CrossRefGoogle Scholar
  21. 21.
    Lindeman, R.W., Sibert, J.L., Mendez-Mendez, E., Patil, S., Phifer, D.: Effectiveness of directional vibrotactile cuing on a building-clearing task. In: CHI 2005, pp. 271–280. ACM (2005)Google Scholar
  22. 22.
    Ngo, M., Spence, C.: Auditory, tactile, and multisensory cues facilitate search for dynamic visual stimuli. Attention Percept. Psychophys. 72, 1654–1665 (2010)CrossRefGoogle Scholar
  23. 23.
    Ogata, K., Seya, Y., Watanabe, K., Ifukube, T.: Effects of visual cues on the complicated search task. pp. 478–485. ACM (2012)Google Scholar
  24. 24.
    Ota, S., Takegawa, Y., Terada, T., Tsukamoto, M.: A method for wearable projector selection that considers the viewability of projected images. Comput. Entertain. 8, 17:1–17:16 (2010)CrossRefGoogle Scholar
  25. 25.
    Pinhanez, C.: The everywhere displays projector: a device to create. In: UbiComp 2001, pp. 315–331. ACM (2001)Google Scholar
  26. 26.
    Pinhanez, C., Kjeldsen, R., Levas, A., Pingali, G., Podlaseck, M., Sukaviriya, N.: Applications of steerable projector-camera systems. In: ICCV Workshop on Projector-Camera Systems, IEEE (2003)Google Scholar
  27. 27.
    Pulkki, V.: Virtual sound source positioning using vector base amplitude panning. J. Audio Eng. Soc. 45(6), 456–466 (1997)Google Scholar
  28. 28.
    Sukthankar, R., Stockton, R.G., Mullin, M.D.: Smarter presentations: exploiting homography in camera-projector systems. In: ICCV 2001, (1) pp. 247–253 (2001)Google Scholar
  29. 29.
    Umlauf, E.J., Piringer, H., Reitmayr, G., Schmalstieg, D.: ARLib: the augmented library. In: The First IEEE International Workshop on Augmented Reality Toolkit, p. 2 (2002)Google Scholar
  30. 30.
    Willis, K.D.D., Poupyrev, I., Hudson, S.E., Mahler, M.: SideBySide: Ad hoc multi-user interaction with handheld projectors. In: UIST 2011, pp. 431–440 (2011)Google Scholar
  31. 31.
    Wilson, A., Benko, H., Izadi, S., Hilliges, O.: Steerable augmented reality with the beamatron. In: UIST 2012, pp. 413–422. ACM (2012)Google Scholar
  32. 32.
    Yamano, S., Hamajo, T., Takahashi, S., Higuchi, K.: EyeSound: single-modal mobile navigation using directionally annotated music. In: Augmented Human, p. 1. ACM (2012)Google Scholar

Copyright information

© IFIP International Federation for Information Processing 2015

Authors and Affiliations

  • Hind Gacem
    • 1
    • 2
    Email author
  • Gilles Bailly
    • 1
    • 2
  • James Eagan
    • 1
    • 2
  • Eric Lecolinet
    • 1
    • 2
  1. 1.Télécom ParisTechParisFrance
  2. 2.CNRS LTCI UMR 5141ParisFrance

Personalised recommendations