Keywords

1 Introduction

Above-device interfaces allow users to interact with their mobile phones using in-air hand gestures over the device. Gestures can be used when touchscreen input is unavailable or inconvenient; for example, users could gesture to follow a recipe on their phone without getting the touchscreen messy or having to wash their hands between using the phone and handling food. Gestures also allow more expressive forms of input than touch, allowing users to do more than simply touch the screen.

Users need helpful feedback when gesturing to be able to interact confidently and effectively. However, mobile phones have limited display capabilities because of their small size. Giving visual feedback about gesture interaction would affect on-screen content and may require redesigning application interfaces for gesture and non-gesture use. Audio and tactile feedback have been considered instead, although lacks the expressive, spatial characteristics of visual feedback. We propose giving visual feedback in the space surrounding the device, using light to illuminate surrounding table surfaces while users gesture (as in Fig. 1). We call this interactive light feedback.

Mobile phones have already used light to illuminate table surfaces for other purposes. For example, Samsung’s Galaxy S6 Edge [1] has a curved screen which illuminates the table surface when placed face-down, glowing different colours to show who is calling. Some of Sony’s Xperia phones feature an Illumination Bar [2] along one edge, which is used for visualisations during media playback. Research prototypes have also used lights embedded within the device bezel [8] and case [9] to visualise off-screen content. We build on these examples by looking at how interactive light can be used for dynamic gesture feedback instead.

In this paper we look at the design space for interactive light feedback. We share ideas about how light can be used to create gesture feedback and our demonstration allows others to experience these designs.

Fig. 1.
figure 1

Interactive light feedback around a mobile phone. Here, brightness changes dynamically to show how users how well their hands can be sensed.

Full size image

2 Related Work

Users require plentiful feedback when gesturing. However, mobile phones have small displays which limits the amount of feedback they can give. Others have suggested using other modalities instead for gesture feedback. Freeman et al. [4] looked at tactile feedback for in-air gestures. They evaluated a variety of methods of giving tactile feedback, including ultrasound haptics and vibration from smart-watches and other wearables. They found tactile feedback effective at enhancing visual feedback, although it lacked necessary bandwidth to replace it.

Others have used audio feedback for in-air interactions while phones are in a pocket, where visual feedback would go unnoticed. Imaginary Phone [6] and Nenya [3] read selected menu items aloud as users gestured, either in front of their body (for Imaginary Phone) or by moving a ring on their finger (for Nenya). In these cases, audio feedback came after gestures; users received no cues during interaction to help them. Our research looks at continuous interactive light feedback during gesture interaction, giving users cues as they interact.

In similar work to ours, Qin et al. [9] presented a prototype mobile phone which had LEDs embedded in its case. They presented an example application where users could respond to phone calls by touching the table on the left (illuminated red) or right (illuminated green) side of the phone. We build on this idea by using interactive light for dynamic feedback during gestures, rather than static feedforward before interaction. Rainbowfish [5] used a grid of LEDs to give feedback about gestures over a proximity-sensing surface, acting as a display. In our work, light is used to illuminate the area around a display, instead.

3 Interactive Light Feedback

Earlier we defined interactive light feedback as visual feedback given by illuminating the space surrounding a device. In our prototype (Fig. 1), we use LEDs around the device edge to present interactive light feedback, although light sources could be enclosed within the bezel itself, as in Sparkle [8]. We use LEDs as they are small, cheap and have low power requirements. Our prototype consists of a flexible strip of 60 LEDsFootnote 1 (approximately 7 mm apart) affixed to a mobile phone and controlled by an Arduino microcontroller. Each LED is independently controllable and can vary its own hue and brightness.

Each LED has two design properties: brightness, which is always variable, and hue, which is fixed in some LEDs. These basic properties of light become more expressive when varied over time and location. Harrison et al. [7] showed how expressive and informative a single LED can be when change in brightness is animated over time. They presented a set of “light behaviours” which could be used to communicate information from a single light.

We have found dynamic change in brightness to be especially effective for gesture feedback, as users immediately see a response to their hand movements. One of our feedback designs uses brightness to show how well users’ hands and gestures can be sensed, using a visibility metaphor: when light is easier to see (that is, brightness is greater), it is because hands can be more easily seen (Fig. 1). If light is less visible, it is because users are not as easily sensed; for example, their hands may be too far from the sensor.

Fig. 2.
figure 2

Varying brightness at different locations to show hand position.

Full size image

Brightness can also be varied at different locations to create spatial patterns. For example, lights could be illuminated (that is, brightness greater than zero) to show hand position relative to the mobile phone (as in Fig. 2). This is similar to the “shadow” feedback design in Rainbowfish [5], although feedback does not occlude on-screen content, since it is around the device instead. Spatial and temporal changes in brightness could also be combined to create rich feedback metaphors; for example, a progress bar could “fill up” over time as users dwell to make a selection (as with the gestures in [4]).

Hue can also be changed by some LEDs, allowing colourful feedback designs. Like brightness, hue can be varied over time and location to create expressive and informative feedback. Colours often have iconic meanings, making them useful for presenting state or static information. For example, green and red light could be used to show if a gesture was recognised or not. Hue could also be used for application-specific feedback metaphors; for example, using blue and red light to show temperature for thermostat control.

4 Summary

In this paper we described interactive light feedback, a technique which allows phones to give visual feedback about above-device gestures without affecting on-screen content. We discussed how brightness and hue allow rich information presentation and described how these could be used for gesture feedback. Our demonstration allows others to interact with a gesture interface and experience these designs themselves, to see how interactive light feedback can be used to informatively illuminate in-air interfaces.