Keywords

1 Introduction

Large interactive surfaces, such as interactive vertical displays or tabletops, have become commonplace in many public settings. These displays often react to the presence and proximity of people [1] or support other interaction modalities such as mid-air gestures, body posture or touch [2].

Several studies, however, report on problems encountered by users while interacting with these large surfaces. People can be prevented from engaging in interactions with displays because they tend to ignore them (display blindness [3, 4]) or fail to recognize that they are interactive (interaction blindness [5]). Additionally, people could be uncertain about the available possibilities for interaction [4, 6], or be hesitant to interact due to social embarrassment [2, 5, 7]. Finally, people are often unaware of ways to recover from mistakes such as accidental interactions [5, 7]. It is difficult for people to know how to address (or how not to address) displays and what they can do when the surface is reacting to their input, as these displays typically rely on implicit interaction using sensors [8]. These problems—users abandoning the surface or not interacting with it because of confusion, frustration or fear of being embarrassed—pose important challenges for designers of applications on large interactive surfaces.

In this paper, we explore the design of dynamic, peripheral floor visualizations to help address interaction challenges with large interactive surfaces. The core of our work is combining vertical interactive displays with a secondary, peripheral floor display. We apply the term ‘peripheral’ similar to its use for public ambient displays [9], as defined by Weiser and Brown [10] to describe, “what we are attuned to without attending to explicitly”. Information shown in the periphery can seamlessly become the center of the attention and move back to the periphery in fluent transitions [10]. Extending the foundations of existing work on interactive illuminated floors (e.g., [11, 12]), the floor display is not used as a primary interaction space, but serves a secondary, assisting role to aid users in interacting with the main display. The floor display can be used to inform users about the tracking status, indicate action possibilities, show interaction zones, and invite and guide users throughout their interaction with the primary interactive display. In the future, we imagine these floor displays being integrated in public spaces to accompany vertical public displays. Due to their relatively low cost, LED floor displays can be frequently found in urban spaces, such as the flashing LED lights in Washington DC metro stations to show arriving trains (Fig. 1).

Fig. 1.
figure 1

An LED floor display used in the Washington metro (YouTube ID: DppgBi0ZMc8).

We illustrate how floor displays can provide additional feedback and guidance through a series of visualizations on the illuminated floor of the Proxemic Flow system. As shown in Fig. 2, we focus on leveraging floor visualizations to mitigate interactions with a proxemic-aware vertical display [1] that reacts to people’s presence, approach, and movements (inspired by Ballendat et al. [1]). For example, the floor shows personal halos indicating when people are sensed by the system (circles in Fig. 2) and provides information about the quality of tracking. Furthermore, the floor reveals the boundaries of the interaction area (red line at the bottom of Fig. 2) and invites moving closer to interact (footsteps at the top of Fig. 2).

Fig. 2.
figure 2

Proxemic Flow providing awareness of tracking and fidelity, zones of interaction, and invitations for interactions (Color figure online).

Our aim with this work is a design space exploration, and in the remainder of this paper we provide the following contributions:

  • We propose an assistive role for floor displays, where they serve as secondary, peripheral displays that help users in interacting with the primary proxemic-aware surface.

  • We demonstrate the expressive power of this new design space with a vocabulary of in situ floor visualization strategies, and explain in detail how we implemented these visualizations for a low-resolution large floor display.

2 Related Work

2.1 Feedback, Discoverability and Guidance for Large Interactive Surfaces

Earlier work has explored techniques to address interaction challenges for large interactive surfaces. We provide a brief overview of related techniques, categorized by the challenges that they address.

Attracting Attention to Overcome Display Blindness and Interaction Blindness.

Displays can be designed to attract attention and motivate users to interact with a display, and thus overcome display blindness [4] and interaction blindness [5]. Earlier work identified barriers that prevent people from engaging with a large display when passing by (e.g., [7, 13]). With a few exceptions (e.g., [14]), most techniques attempt to convey interactivity and attract users by using visualizations on the interactive display (e.g., [13, 15, 16]). In contrast, with Proxemic Flow we primarily focus on the floor display to convey interactivity. One of the advantages of providing visualizations on a secondary display is that they do not occlude or distract from existing content on the primary display. The floor can reveal the interaction area through borders and zones, and show halos when people are recognized by the display. Our floor visualizations are designed to indicate to passersby that they can interact, making them aware that they are tracked, and also allowing people to avoid interacting with the display. This addresses a common issue with interactive surfaces in semi-public settings, i.e., that they lack opt-in and opt-out choices [17].

Revealing Action Possibilities and Providing Guidance.

Commonly used interaction modalities for public displays (e.g., proximity, body posture, mid-air gestures) are often hard to understand by passersby at first glance [2]. A number of systems suggest action possibilities and input gestures by visualizing sensor data, such as depth camera images, detected user skeletons [18, 19] or mirrored images [9, 20]. Early work by Vogel and Balakrishnan [9] included a self-revealing help feature using a mirror image video sequence. Walter et al. [6] studied different visualizations to reveal a ‘teapot gesture’ that allows users to indicate that they would like to start interacting with the display.

These techniques work well in particular for revealing action possibilities (e.g., mid-air gestures and body postures) on the display itself. Our in situ techniques introduce a new vocabulary of visualizations potentially more appropriate for proxemic-aware surfaces that often use people’s and devices’ spatial movements as implicit input. Such implicit application input might be surprising and possibly disturbing in walk-up-and use scenarios, as in the Proxemic Media Player [1] where videos are automatically paused when users are not facing the display. Techniques that reveal action possibilities on the primary display often focus on particular proxemic dimensions [1]. For example, cross-device interaction techniques typically use orientation to show possible targets around the user’s device, as in the Gradual Engagement pattern [21] or the RELATE interaction model [22].

Providing Tracking Feedback.

A common problem users experience while interacting with public displays is a lack of feedback about how the system is currently recognizing and interpreting their input, and also how reliably this input is being sensed. In crowded spaces, people can be unsure about the level of control they have over the display [19]. A number of systems reveal tracking feedback for proxemic interactions to convey what the system sees of the user, e.g., by visualizing detected skeletons [18] or mirror images [20]. With their interactive whiteboard, Ju et al. [8] showed a dot pattern to indicate in which proximity zone the user was recognized. In the Medusa proximity-aware tabletop [23], the user’s proximity is shown using an orb visualization. Both in the Proxemic Media Player [1] and the Gradual Engagement pattern [21], tracked devices are visualized on the large surface with their relative size mapped to their proximity to the large display.

With Proxemic Flow we provide in-place tracking feedback on the floor, in the space where users are tracked. A number of earlier systems provided projected tracking feedback on the floor in the form of halos such as the Solstice LAMP [24] and Proximity Lab [25].

In Situ Feedback and Guidance.

Proxemic-aware systems typically take different actions based on the interaction zone in which the user is located [1, 8, 9], which may be unintelligible to users. Rehman et al. [26] used augmented reality to visualize interaction zones in-place, but this technique required users to wear head-mounted displays. Researchers have also explored the possibilities of providing in situ feedback and guidance using spatial augmented reality, which uses a combination of projectors and depth cameras and thus eliminates the need for users to wear additional apparel. For example, LightSpace [27] shows when users are tracked by the system by projecting colored higlights on the user’s body. LightGuide [28] uses a projector and depth cameras to project visualizations on the user’s body that provide movement guidance. In a public art context, Ozturk et al. [29] explored people’s reactions to projections of their ‘future footsteps’ in an airport terminal.

Although the use of projectors allows for high-resolution visualizations and more flexibility, projectors often require low-lighting conditions, which makes these techniques less suitable for large interactive surfaces in urban spaces (especially during daytime). Regarding guidance using LED floors, Rogers et al. [30] explored the use of LEDs embedded in carpet tiles to motivate people to use the stairs more often. They observed that the LED lights had a significant effect on people’s behavior, which illustrates the power of in situ visualizations and guidance.

2.2 Related Work on Interactive Illuminated Floors

With Proxemic Flow, we propose the use of graphical information shown directly on the floor of the interactive space, around the people who are engaging in the interaction, for providing feedback about the system status or informing users of action possibilities and consequences.

Interactive illuminated floors have been used in different contexts, such as interactive art [24, 25] or games [31], and have recently seen increasing exploration as a primary interaction space [11, 12]. A variety of input and output technologies have been used for these interactive floors, such as tracking users through computer vision techniques [31] or pressure sensing [12], and showing output using projectors [12, 31], LED illumination [32] or vibrotactile feedback [33].

Our work extends this earlier research by (a) proposing the use of the floor as a peripheral/secondary output device that can help to mediate interactions with a different, primary interaction device, and (b) providing a vocabulary of strategies to provide in situ feedback about current and future interactions with the system.

3 In Situ Floor Visualization Strategies

In order to mitigate the previously mentioned interaction challenges, we introduce a series of interaction techniques and in situ visualizations on the floor. These are categorized into three phases, progressing from:

  • (a) in situ tracking feedback (answering the questions: What does the system see? How well does the tracking work?), and

  • (b) revealing interaction possibilities (answering: What possible interactions are available?), to

  • (c) inviting for and guiding interactions (answering: What can I do next?).

3.1 Walkthrough with Photo Gallery Application

We will illustrate all our in situ visualization strategies with a running example application, inspired by the design of the Proxemic Media Player [1]. Our photo gallery application shows photos collections on a large public display. A series of interactions are possible with this gallery application: it shows photo thumbnails when in idle mode (Fig. 3a), reveals more content when a person approaches the display (Fig. 3b), shows full screen photos when people stand directly in front of the display (or sit down), and allows mid-air gestures to navigate the photo collection (e.g., waving left or right to browse through the timeline of photos). While limited in scope, we believe this example application captures the essence of many proxemic interactions applications and works best for demonstrating our in situ floor visualization strategies. Throughout this paper, however, we will also refer to the use of our visualization strategies in other application contexts.

Fig. 3.
figure 3

Walkthrough photo gallery application: photo thumbnails at a distance (a), revealing more content when moving closer (b).

3.2 Design Space for Floor Visualization Strategies

Our visualization strategies can be categorized in a design space for in situ floor visualizations. Table 1 shows an overview of the different strategies and indicates to which of the three phases (tracking feedback, action possibilities or guidance) they correspond. The table compares our floor visualization strategies based on three different aspects: perspective, position, and temporal relevance. Regarding perspective, we distinguish between egocentric and exocentric visualizations. For example, tracking halos are targeted towards being viewed from the user’s own perspective (egocentric), while zones and borders are mostly useful from an external perspective (exocentric). Additionally, a few visualizations have a static position on the floor, while others can move dynamically (e.g., together with the user). Finally, visualizations can be relevant to the user’s current (or present) interactions with the primary display (e.g., quality of tracking), or can alternatively provide clues about past or future actions.

Table 1. An overview of the design space for in situ floor visualization strategies.

We see this design space as a starting point for characterizing in situ floor visualizations for mediating large surface interactions, and foresee possible future extensions. It also functions as an analytical tool to reflect on the set of floor visualization strategies that we propose in this paper, and can help to further explore alternative floor visualizations. We will now go over the three phases, and will later come back to these different aspects while discussing our floor visualization strategies.

3.3 Phase 1. In Situ Personal Tracking Feedback with Halos

A fundamental challenge for interaction with large surfaces is providing a person with immediate feedback about how the system is currently recognizing and interpreting gestures or other input from the user. In this section, we introduce visualization strategies to provide this feedback directly in the physical space where the person is moving in front of the display.

Personal Halos.

The personal halo provides immediate feedback on the floor display about the tracking of a person in space. When the person enters the area in front of the public display, a green halo (an area of approximately 1 m diameter) appears underneath the person’s feet (Fig. 4a). The halo moves with them when moving in the tracking area, and therefore gives continuous feedback about the fact that the person is being recognized and tracked by the system.

Fig. 4.
figure 4

Halos: (a) providing feedback about active tracking and (b) the tracking quality (Color figure online).

Another important part of information (besides information about the fact that a person is tracked) is the actual quality of tracking. Most computer vision based tracking systems (RGB, depth, or other tracking) have situations where tracking works well, where it does not work well, or where it does not work at all (e.g., due to lighting conditions, occlusion, limited field of view). Therefore, our personal halo visualization encodes the quality of tracking in the color of the halo. To indicate tracking quality, we use three different colors (Fig. 4b). A green halo indicates optimal tracking of the person in space. Its color changes to yellow when the quality of tracking decreases, for example when the person moves to the limits of the field of view or when partially occluded by another person or furniture. Finally, a red halo color is shown when the tracking of the person is lost, such as when moving too far away from the camera, or if the occlusion is hiding the person completely. For this last case, since the person is now not tracked anymore, the red halo visualization remains static at the last known location of the person, fades in and out twice, and then disappears (the duration of that animation is approximately 4 s). If the person moves back into the field of view of the camera and the tracked region, the halo color changes back accordingly to green or yellow.

The immediate feedback of tracking through halos can provide people with more control over their interaction with the system. For example, when noticing that they are being tracked, the user could decide to opt out of interaction with the system by moving back out of the active tracking area.

Alternative Halo Visualization Strategies.

Although this is a crude mapping of tracking accuracy to different colors, we found it to be an effective in situ form of feedback about tracking activity and fidelity. Other applications might require different levels of granularity. For example, for interactive proxemic game experiences [34], tracking accuracy per body part could be helpful information for players, which could be visualized using more fine-grained halos. Tracking accuracy of different body parts could be mapped to different areas of the halo; e.g., front left corresponds to left arm, back left to the left leg. Alternatively, halos could change their size depending on the area covered by the player. There is a limit to the amount of information that can be conveyed using our low-resolution floor display. Revealing precise details about the tracking quality for different body parts (or showing text for instructions) would require higher-resolution floor displays. For the remainder of this paper, however, we focus on the expressive potential of low-resolution peripheral floor visualizations.

Multi-User Halos.

Interaction around interactive surfaces is often not limited to a single person, but can involve multiple people present in the space and interacting with the display. With multiple people, information about active tracking and its fidelity becomes even more important, because tracking problems increase with the likelihood for occlusions.

If multiple people are present in front of the screen, each person’s individual position that the system currently tracks is shown with a colored halo (Fig. 5a). Color changes indicate a change in how well the user is tracked. For example, in case another person walking in interrupts the tracking camera’s view of a person, the changing color of the halo from yellow to red tells the person that they are not tracked anymore (Fig. 5b). Similarly, if two people stand very close to another, making it difficult for the computer vision algorithm to separate the two, the halo color changes to yellow.

Fig. 5.
figure 5

Halos for multi-user interaction: (a) both people are visible to the system; (b) one person is occluding the camera’s view of the other person, indicated by the red halo (Color figure online).

Trails: Revealing Interaction History.

As a variation of the halo technique, the spatial trail feedback visualizes the past spatial movements of a person in the interaction area. The trails are shown as illuminated lines on the floor that light up when a person passes that particular area (Fig. 6). The illumination fades out after a given time (in our application after five seconds), thus giving the impression of a comet-like trail. The colors that are used to light up the floor are identical to those of the person’s halo (i.e., green, yellow, red), and therefore still provide information about the tracking quality. Because the trail visualization remains visible for a longer time, it provides information about the past movements of the people interacting with the system. Potentially, the trails could help to amplify the honeypot effect [7] by showing the past trails of other people moving towards the interactive display, and thus inviting other bystanders and passersby to approach the display as well—which is why they are categorized in both phase 1 and 3 (Table 1).

Fig. 6.
figure 6

Trails, visualizing the history of spatial movements of a person (Color figure online).

Reflecting on Phase 1 in the Design Space.

Halos are an example of an egocentric strategy (Table 1). They are primarily designed to be viewed from the user’s perspective, providing feedback about the tracking status. The trails variation, however, is a mostly exocentric technique that shows information about past interactions from the perspective of other users. However, since the trails are still shown underneath the user’s feet, and change color depending on the user’s tracking accuracy, they are simultaneously egocentric and inform the user about their present interactions (Table 1a). In addition, as they potentially invite bystanders to interact with the display, the trails can serve as an invitation for future interactions (Table 1b). We can also imagine other exocentric halo visualizations. For example, pulsating exocentric halos could indicate open spots where users could move towards, e.g., to form teams in proxemic gaming scenarios [34].

3.4 Phase 2. Zones and Borders: Entries and Exits for Interaction

As mentioned earlier, people often have difficulties knowing when and how they can interact with a large public display [2, 19]. To mitigate this problem and to reveal interaction possibilities, we explicitly visualize the spatial zones for interaction and the borders of the interaction space (Fig. 7).

Fig. 7.
figure 7

The interaction areas in front of the display represented as (a) red and (b) blue rectangular zones; (c) borders indicate thresholds to cross for (d) leaving the interaction space in front of the display (Color figure online).

Opting-in: Proxemic Interaction Zones.

Many designs of large interactive displays use spatial zones around the display to allow different kinds of interaction [9] or change the displayed content depending on which zone a person is currently in. These zones, however, are not always immediately understandable or perceivable by a person interacting with the display. Our floor-visualizations explicitly reveal zones of interaction, allowing a person to see where interaction is possible, and make deliberate decisions about opting in for an interaction with the display by entering any of the zones.

We demonstrate the use of zone visualizations with the Proxemic Flow photo gallery application. Similar to earlier examples of proxemic-aware displays [1, 9], our application uses discrete spatial zones around the display that are mapped to the interactive behavior of the application on the large display. When no users are interacting with the system, a large red rectangular zone indicates the area furthest away from the display that triggers the initial interaction with the display (Fig. 7a). This serves as an entry zone for interaction, i.e., an area to opt-in for interaction with the system. In our current implementation, we use a 3 s pulsating luminosity animation, fading the color in and out, in our approach of balancing the goal of attracting attention while not being too intrusive. While a static color would be possible, identifying it as part of an interactive system is potentially more difficult. Once a person enters this zone, the large display recognizes the presence of the person, tracks the person’s movement, and their halo is shown. The first zone now disappears and a second zone appears—an area to interact with the display when in front of is (visible as the blue rectangle in Fig. 7b). When the person begins approaching the display, the content gradually reveals more of the photo collection on the display. The closer the person gets, the more images become revealed (this is a behavior identical to the Proxemic Media Player [1]). Once entering the second zone, the person can now use hand gestures in front of the display to more precisely navigate the temporally ordered photo gallery (e.g., grabbing photos, sliding left or right to move forward or back in time). Again, once the person entered that close-interaction zone in front of the display, the visualization disappears.

Opting-out and Exit Interaction: Borders.

While we envision zones primarily as explicit visualizations of the zones to interact, and for allowing a person to deliberately engage and “opt-in” for an interaction with the system, we can also consider visualizations that help a person to leave the interaction area (i.e., opting out). We illustrate this concept with borders shown in the Proxemic Flow application. In continuation of the application example from before, once the person entered the interaction zone (blue) directly in front of the display and interacts with the display content through explicit gestures, a red border around the actively tracked interaction area surrounding the display is shown to make the boundaries of that interaction space explicit and visible (Fig. 7c). While we decided to dynamically show the border only in situations when a person engaged with the system, alternatively it could remain a fixed feature of the visualizations shown on the floor. A reason for showing a fixed visualization of the interaction boundaries with borders could be to always clearly indicate where a person can both enter but also leave the interaction area (Fig. 7d).

Using Zones and Borders with Multiple Users.

We can consider alternative design aspects when using zone and border visualizations with multiple users. For example, we can consider whether area visualizations are only shown to the first user entering the space and disappear once that person entered the zone, or whether the visualizations remain persistent. Showing visualizations for the first person entering a space seem most critical, and hiding the zone visualizations after the person enters a particular zone has the advantage of a floor that is less visually cluttered and therefore can help emphasizing certain parts of the visualizations (for example, make the halos stand out).

Reflecting on Phase 2 in the Design Space.

In contrast to halos and trails, zones and borders are static visualizations. They are fixed at a certain position, and although they might only be shown at certain times, they do not follow the user. Zones and borders are also mostly exocentric, as they are designed for observations from an external point of view. Nevertheless, zones can also be used from an egocentric perspective, when the user is inside the interaction zone (Table 1c). Finally, they convey cues relevant to the user’s current interactions (present), such as borders around the actively tracked interaction area. However, zones and borders can also provide cues for future interactions, such as possible next areas to move to, or where to go to opt-out of the interaction (Table 1d–e).

3.5 Phase 3. Waves and Footsteps: Inviting for Approach, Spatial Movement, or Next Interaction Steps

The last set of floor visualization strategies we introduce is designed to invite for approach, encourage a person’s movement to a new location, and suggest possible next interaction steps. In particular, in this category of visualizations we introduce two strategies: waves and footsteps.

Waves: Encouraging Approach.

Our first strategy is intended for inviting people to move closer to the large display for interaction. Several strategies for encouraging approach of people have been proposed in the past, including showing text labels, animations, graphic icon representations or using sound (e.g., strategies in [15]). With our waves technique, we leverage the output capabilities of the illuminated floor for showing looped animations of lights fading in and out, with the effect of a wave of light going towards the large screen (Fig. 8a). Alternatively, different visual designs of the wave effect are possible, for example a circular wave effect with the large display at the center, starting with circles having a large radius and continuously decreasing the circle radius.

Fig. 8.
figure 8

(a) Waves inviting for interaction and (b) footsteps suggesting action possibilities (Color figure online).

Footsteps: Suggesting Next Action Possibilities.

The footsteps visualization is designed to offer a person clues about possible next interaction steps (directly addressing discoverability), in particular for encouraging spatial movements in the environment. The visualization shows animated footsteps (in our case these are represented through glowing circles) beginning at one location on the floor and leading to another location. This technique is inspired by earlier work of the Follow-the-light [30] design that uses animated patterns of lights embedded in a carpet to encourage different movement behaviors by luring people away from an elevator towards the stairs.

To illustrate this technique, we again revisit our Proxemic Flow example application with the large display photo gallery viewer. When a person entered the interactive (i.e., tracked) space in front of the display and stands still for over 5 s, the floor begins the footstep animation (Fig. 8b) to invite the person to move closer to the display—in particular, moving to the interaction zone in front of the display allowing the person to use mid-air gestures to further explore the image collection. The footstep animation begins directly in front of the person and leads towards the blue rectangular area highlighted in front of the display (Fig. 8b). The footsteps visualization strategy can be used to reveal interaction possibilities—in particular those involving spatial movements of the person. The strategy can be used in many other contexts for guiding or directing a user in the environment, and for encouraging movements in space.

Reflecting on Phase 3 in the Design Space.

The visualization strategies for phase three provide cues that invite users to future interactions. The waves strategy is exocentric, as it invites bystanders to interact with the primary display. It is a static visualization, as people’s movements do not influence its position. The waves pattern could be shown across the full floor display or be centralized around the primary display. The steps strategy, on the other hand, is a dynamic and egocentric visualization that starts from underneath the person’s feet, and guides them towards a certain position.

3.6 Reflection on In Situ Visualization Strategies

We discussed and demonstrated a set of in situ floor visualizations that provide peripheral tracking and fidelity information with personal halos, make interaction zones and borders explicit for easy opt-in and opt-out, and provide cues inviting for spatial movement or possible next interaction steps through wave, trail, and footstep animations. This set of floor visualization strategies targets important interaction issues with large interactive surfaces that were identified in earlier research. During informal observations of people interacting with our floor display, we noticed that essential concepts such as halos and zones were easy to understand. Future studies are necessary, however, to confirm these early observations. The strategies we presented here are a starting point for a collection of building blocks for how to provide in situ visual feedback on the floor to mediate spatial interactions. In the next section, we present the Proxemic Flow software architecture and explain how we implemented the floor visualizations.

4 Implementation

The Proxemic Flow architecture consists of three major technical components: (1) the hardware setup of the illuminated floor, (2) the user tracker and (3) the floor renderer. The user tracker is responsible for tracking users in the space in front of the display, and for mapping these positions to positions on the floor. The floor renderer consists of a .NET client that draws visuals to a bitmap and sends display updates over the network to a Processing sketch connected to the Arduino board controlling the different light units. We will now explain these components in more detail.

4.1 Hardware Setup of the Interactive Floor Display

The foyer floor that we use for our setup comprises 288 light wells set in concrete, of which 216 of these wells are fitted with a custom light unit [35]. The custom light units in each of the 216 light wells consist of four RGB LEDs cut from an LPD8806 LED strip, joined together and mounted onto a plastic cap which fits neatly into the concrete surface from the floor below. The light units are connected in series, with three modified ATX power supplies providing power. A single Arduino Mega with the ATmega1280 microcontroller is controlling the floor display. Each of the light units can be set to one of around 2 million colors and the whole array can be updated at a rate of up to 25 fps—effectively turning the floor into a large display with a resolution of 12 × 18 pixels.

4.2 Tracking Users

Users are tracked across the floor using a single Microsoft Kinect depth camera and the Kinect SDK, which allows us to track up to six simultaneous users (with skeleton data available for two users). As users positions can be represented in a 2D (x, z) plane (we ignore the user’s vertical position), a simple affine matrix transformation suffices to map the (x, z) coordinates as given by the Kinect camera to a position on the floor. To set up the system, a four-point calibration is performed to map positions seen by the Kinect to the corresponding floor positions, after which the corresponding transformation matrix is calculated.

Each user’s tracking accuracy—as used for determining the color of their personal halo—is specified as a value in the range [0,1]. We calculate the tracking accuracy by performing an arithmetic average over the accuracy of the skeleton joints. Skeleton joints have one of three states: tracked, inferred, or not tracked. We currently assign the value 1.0 to tracked joints, 0.3 to inferred joints and 0.0 to joints that are not tracked. Green halos are shown for accuracies over 0.7; yellow halos for accuracies between 0.3 and 0.7; and halos turn red when the accuracy drops below 0.3. These specific thresholds have been selected based on empirical observations, but can be easily changed.

4.3 Floor Renderer

The Proxemic Flow renderer provides a set of reusable rendering primitives that react to user tracking updates. All graphics and animations can be translated into a floor bitmap, which allows the rendering pipeline to be agnostic to the specifics of the graphics being shown on the floor. This rendering pipeline is the central hub of the architecture and handles updates to the floor display (Fig. 9).

Fig. 9.
figure 9

The Proxemic Flow rendering pipeline: Visualizations on the floor display are abstracted in a floor scene (a). This floor scene is processed by the floor renderer (b), resulting in (c) a floor bitmap (an abstraction of a floor display update) that is sent over the network to the connected floor displays that implement the IFloor interface (d). We also implemented a projected floor display (f) (Color figure online).

The floor renderer (Fig. 9b) uses a timer to allow sending update messages to the floor at a fixed rate. A floor update message is represented by a FloorBitmap object (Fig. 9c), which is an 18 × 12 grid of color values for each of the light wells in the grid. Every tick, the rendering pipeline sends a floor update message to the connected instances of the IFloor interface (Fig. 9d). The default IFloor implementation (Fig. 9e) sends messages to a Processing sketch that is connected to the Arduino board that runs the floor, which then renders the floor bitmap to the physical floor display.

4.4 Alternative Implementations

The concept of Proxemic Flow goes beyond the specifics of our floor setup. The illuminated floor could be implemented using different floor displays (e.g., using projectors or FTIR floor displays [11]) and tracking solutions (e.g., 2D cameras with markers, other depth cameras, or optical trackers such as VICON). Figure 10 shows an alternative rendering solution we implemented in order to show visuals on arbitrary surfaces, based on an overhead projector mounted to the ceiling. It connects another IFloor instance to the same rendering pipeline (Fig. 9f), so that applications written once run without modification. In this implementation, a separate Windows Presentation Foundation (WPF) window that renders the floor grid is projected onto the floor. Our projection-based floor responds to floor update messages by changing an internal model of the floor grid, which is then also updated in the WPF view. Higher resolution floor displays (e.g., with a circular halo visualization) would be possible by making rendering primitives adapt to different resolution floor bitmaps, specific to each IFloor instance.

Fig. 10.
figure 10

Alternative floor display using a ceiling-mounted short-throw projector (Color figure online).

5 Discussion

We presented Proxemic Flow, dynamic in situ floor visualizations for revealing and mediating large surface interactions. Based on previously identified interaction challenges with large interactive surfaces, we demonstrate the expressive potential of the floor as a peripheral/secondary output device for showing in situ feedback using three categories of visualizations: (1) personal halos and trails that provide peripheral information about current tracking and tracking fidelity; (2) interaction zones and borders for easy opt-in and opt-out; and (3) wave and footstep cues that invite users for movement across the space or possible next interaction steps.

Our approach is intentionally minimalistic: we reduced the visualizations to essential cues that require minimal visual bandwidth. These can be extended—for example with more fine-grained spatial movements cues—but we believe it is important to avoid a visually cluttered floor with (perhaps even animated) visualizations that distract the user. Ideally, the visualizations should be shown when needed, but not unnecessarily draw the user’s attention and detract from interacting with the primary display, as the floor serves a secondary, assisting role. We plan studies further investigating the balancing of showing information while avoiding distractions.

During initial observations, we noticed that people became aware of the floor being a display as they approached the tracking zone. Users noticed their personal tracking halos when they entered space in front of the display. Due to their low visual complexity, a quick glance at the visualizations is often sufficient, e.g., when users are unsure about action possibilities. An interesting opportunity for future work is to investigate how user’s peripheral view, which is very sensitive to motion [36], can be used to draw their attention when needed.