Keywords

1 Introduction

People interact with an evergrowing number of mobile computing devices in everyday life. Nowadays, these devices have become ubiquitous and are commonly used in various places such as buses, trains, airports, coffee shops, and restaurants [3, 14]. As a result of the continuous growth, privacy and security challenges of these devices are becoming increasingly pressing. For example, smartphones hold sensitive information about users, including business records, financial interactions, personal details, and many more that should be kept hidden from others. Nevertheless, finding privacy-preserving solutions is not restricted to smartphones only. These solutions need to consider a variety of personal devices (e.g., smartwatches and tablets) as well as public or shared devices (e.g., ATMs and ticket machines).

All these devices are subject to various types of attacks. For instance, thermal attacks, where intruders use thermal cameras to analyze the heat traces of the entered authentication [1] or attacks that analyze the smudges on the screen for password reconstruction and gaining illegitimate access [49, 52]. However, smudge attacks are mainly focused on the authentication period, and thermal attacks require technical support and proper planning for a person to take a photo, feed it to a recognizer, and gain unauthorized access. On the other hand, observation attacks, commonly known as shoulder surfing attacks, are directly performed by humans and usually do not require additional hardware to be successfully completed. Despite a large body of work on these observation attacks, shoulder surfing remains a significant unresolved problem that requires more attention.

Observation attacks are not limited to a specific device, location, or acquaintance level. Shoulder surfer can gaze at a person interacting with their personal phone or at someone’s PIN, while they authenticate themselves after getting the phone out of the pocket. They do not need an extra device and can quickly memorize entered PINs or passwords. They could be standing in a train [46], or sitting next to the victim in an office [2] (see Fig. 1). The incident could occur between two closely tied people or with total strangers. Previous works confirm that observation attacks are widespread and highly likely to occur [14].

Fig. 1
A chart contains 2 cartoon characters. A cartoon lady is the victim and she is using a phone which is the target device. The person standing next to her is the attacker.

Sketched example of a spontaneous shoulder surfing attack taking place during daily commute

Full size image

With this pervasiveness, nearly everyone is both attacker and victim. Albeit, recent studies showed that shoulder surfing incidents often take place opportunistically, and without malicious intent. To this end, we consider a person looking at the user’s interaction as an observer, as we are not sure of their motives. Many researchers focused on understanding the occurrence of the observation attacks. However, regardless of the intentions of the observers, researchers also worked on various approaches to mitigate the risk of being observed, either by detection of the observer, or by providing novel solutions to prevent the looker from perceiving the content displayed.

Chapter Overview

In the next section, we define the term shoulder surfing, describe different dimensions relevant for shoulder surfing attacks, and present key findings from previous research. Thereafter, we look at proposed strategies to mitigate shoulder surfing attacks. Here, we start by looking at threat models and algorithmic detection of shoulder surfers. Finally, we outline challenges and future research directions for shoulder surfing research.

2 Investigating the Phenomenon

In this section, we first define shoulder surfing to set the scope for this chapter. After that, we describe different methods with which researchers have investigated the phenomenon and discuss their advantages and disadvantages. Finally, we highlight the key findings from studies investigating shoulder surfing behavior.

2.1 Defining Shoulder Surfing (Attacks)

Observation attacks, commonly known as shoulder surfing attacks, are directly performed by humans and usually do not require additional technology to be successful. Farzand et al. [16] define shoulder surfing as observing someone’s device screen without their consent. There are technology-based approaches to investigate observation attacks using machine vision, commonly referred to as recording attacks or video-based observation attacks (e.g., [30, 61]). Nonetheless, this chapter primarily focuses on shoulder surfing attacks performed by humans.

To be classified as shoulder surfing, it does not matter if the motivation to shoulder surf is simply curiosity or a deliberate attempt to steal information [9]. In fact, shoulder surfing mainly occurs in an opportunistic, non-malicious way [14]. Nonetheless, failing to prevent bystanders from observing sensitive information can lead to negative consequences such as financial loss, public exposure, and embarrassment [3]. An example of a shoulder surfing attack is shown in Fig. 1.

In the following, we provide an overview of different dimensions that help describe and classify shoulder surfing. The goal is not to present a complete overview of all dimensions relevant to shoulder surfing but rather to discuss different aspects that should be considered:

Motivation of Attack::

Shoulder surfing attacks can be either intentional or unintentional, whereas unintentional means in an opportunistic, non-malicious way [9]. In most cases, shoulder surfing is unintentional and does not have serious consequences [14]. Nonetheless, it can evoke negative feelings for both parties and result in various coping strategies.

Attack Pattern::

Shoulder surfing attacks can follow different attack patterns. Abdrabou et al. [2] found three different patterns: continuous attacks, cautious attacks, and repeated attack. While continuous attacks are characterized by bystanders looking at the target device for an extended period with few or no gaze shifts, cautious and repeated attacks alternate between observing the target device and looking away. For the latter two, the difference is the victim’s behavior, who either looks up from the target device (from time to time) or shows high engagement. Friends, family, or colleagues at work may repeatedly observe their peers and thereby combine multiple partial observations to form a hypothesis of a target device’s secret [37, 57].

Number of Attackers::

In theory, a shoulder surfing attack can be performed by multiple attackers. While some research considers threat models with more than one attacker [24], many studies simplify this aspect and study 1:1 relationships between victim and attacker.

Relationship Between Victim and Attacker::

Besides the number of attackers, the type of relationship (family, friend, colleague, stranger) is important as well. Muslukhov et al. [37] conducted surveys and interviews to investigate users’ concerns about unauthorized access by insiders and strangers. They concluded that observing unlock attempts, memorizing it, and thus gaining unauthorized access by insiders are highly likely to occur. That is directly linked to insiders’ ability to observe interactions closely and repeatedly. Farzand et al. [16] showed that the type of relationship impacts the choice of mitigation behavior. Moreover, depending on the relationship with the attacker, victims often do not want them to know they were caught.

Victim–Attacker Relative Pose::

To successfully shoulder surf, the content on the target device must be directly visible to the attacker (unless we reconstruct the screen content from visual reflections with machine learning [60]). Thus, the relative pose between victim and attacker is important, as the used term shoulder surfing illustrates. A sitting pose, for example, enables shoulder surfing more than a standing pose [46]. Furthermore, viewing angle and distance play an important role as well [6]. However, tilting the device away from the observer, a widely adopted defense strategy, provides limited protection from shoulder surfing attacks [25].

Type of Device::

Different devices can be the target of a shoulder surfing attack, including but not limited to notebooks, tablets, smartphones, and smartwatches. However, shoulder surfing can also occur when using shared devices or accessing private information on public devices [9]. The main prerequisite for shoulder surfing is that a bystander can observe the user’s screen. Hence, smartglasses are unaffected and can be used as a mitigation strategy [58].

Type of Content::

Mainly, two different types exist: (1) authentication-based and (2) content-based shoulder surfing [18]. The primary focus of many shoulder surfing studies is to investigate secure password or PIN entry [8]. While authentication is, of course, important and prone to observational attacks, other types of content can also be observed. Moreover, content-based shoulder surfing is more frequently experienced than authentication-based shoulder surfing [18]. Previous work has examined different content types such as notifications, texts, photos, social media, and gaming [6, 46]. Nevertheless, while different types of content are affected by shoulder surfing, there are differences in their perceived sensitivity [17].

Type of Environment::

Shoulder surfing can take place in different environments such as buses, trains, airports, coffee shops, and restaurants [3]. These environments can be classified in two different ways. One can either distinguish private, semi-public (work), or public contexts [45] or differentiate between personal and professional contexts [62]. Independent of the classification choice, the location cannot be neglected when studying shoulder surfing attacks as it influences victim and attacker behavior [48].

2.2 Research Methods

As outlined in the chapter “Empirical Research Methods in Usable Privacy and Security” , privacy and security research has applied various methods. In this section, we highlight the methods that were previously used to study shoulder surfing. In summary, we classify these methods into four categories: (1) surveys and interviews, (2) lab studies, (3) field/in-the-wild studies, and (4) studies in extended reality. The following subsection describes the different methods and highlights their advantages and disadvantages. Our goal is to provide an overview of the different methods to support researchers and practitioners (new to the field) in deciding which method to apply in their research.

Surveys and Interviews

Surveys and interviews are helpful tools for privacy researchers to gather valuable insights into a broader population or specific user groups [36]. The difference between surveys and interviews is that in interviews, a researcher takes an active role and directs questions to the interviewee (cf., Lazar et al.[27, 28]), while in surveys, a set of predefined questions is presented to the participants. With surveys and interviews, it is possible to achieve various objectives. On one side, researchers can use them to gather evidence for shoulder surfing attacks in the real world and get insights into personal experiences with the phenomenon from both victims and attackers of shoulder surfing incidents (e.g., [14]). On the other side, they help to understand preliminary performance metrics of authentication techniques against observation attacks (e.g., robustness [4]) and can even be used to quantify which parameters of these techniques help to make them less observable (e.g., [54]). Different approaches to constructing surveys exist. Noticeable is the inclusion of video material to present recreations of shoulder surfing attacks to participants [4]. Aviv et al. [5] show that these videos embedded in surveys can achieve results comparable to user studies in the lab.

Compared to other research methods, surveys allow larger sample sizes as researchers can reach and recruit more participants. Nevertheless, sample sizes vary enormously for shoulder surfing research. Previous work has reported studies with more than 1000 participants (n = 1173) [4] to smaller numbers that remain in the hundreds (e.g., n = 298 [54] or n = 174 [14]). Compared with other research methods, surveys often report higher numbers of participants. Recently, crowdsourcing platforms have entered the stage of privacy research and provide researchers with access to different user groups (that can be specified concerning various dimensions) [23]. Nowadays, researchers can more easily recruit a diverse set of participants.

In addition to surveys, in-depth interviews can be a sensible next step that allows scientists to understand the reasons behind the observed data [14]. Nonetheless, interviews can also be applied as a standalone method. For interviews, the more active participation of a researcher asking questions can lead to more detailed responses [28]. Moreover, interviews allow the live demonstration of specific techniques under controlled conditions. For example, the interviewer can present different shoulder surfing mitigation strategies to participants during the interview [16].

Finally, there has been a recent study that explored shoulder surfing through a longitudinal investigation, meaning they performed a diary study with 23 participants over one month [18]. They found that content-based shoulder surfing takes place more frequently than authentication-based shoulder surfing.

While we presented different methods in this part, they all have in common that they rely on self-reporting. While self-reporting is frequently deployed in privacy research, it has a few noteworthy drawbacks. As researchers do not directly observe a phenomenon, factor, or effect, they rely on the subjective perception of the participant, which can include a recall bias [43]. Moreover, not every type of information can be gathered with self-reporting; however, asking indirect and anonymity-preserving questions can minimize social desirability bias [33, 53].

Lab Studies

Scientists often conduct experiments to answer their research questions concerning shoulder surfing. In experiments, it is often necessary that researchers can observe a shoulder surfing situation taking place. Due to the challenges of researching the phenomenon during field or in-the-wild studies (see below), these studies are primarily carried out in the lab. Moreover, compared to surveys and interviews, recruiting participants is more difficult, and conducting the experiment is often more workload-intense. As a result, experiments generally report smaller sample sizes. Nevertheless, a lab study also has certain advantages, for example, compared to field or in-the-wild studies. The most significant benefit (compared to other study types) is the high degree of control over the experimental conditions. Moreover, a lab study allows gathering consent from all involved parties before the experiment.

When conducting a lab study to research different dimensions of a shoulder surfing attack (e.g., the resilience of authentication techniques against human shoulder surfers), a challenge is to replicate these attacks for the study [56]. In lab studies, participants often take over the role of the attacker (e.g., [46]). Nevertheless, it remains challenging to replicate realistic attacks, as often they are performed out of boredom in opportunistic moments [14]. Simply instructing participants to perform a shoulder surfing attack would broadly differ from the behavior observable during an actual attack. To overcome this challenge, researchers have designed studies that inform participants about the study’s goals toward the end (e.g., [46]). These studies partially deceive participants by leaving out specific study details not to influence their behavior. However, it should be noted that deceiving participants in a user study can be problematic and not justified. Hence, it is strongly encouraged to balance ethical implications and knowledge gain and act cautiously when deceiving participants.

A different approach is to research factors and effects that are not related to the timing, occurrence, or behavior of shoulder surfing attacks but instead focus on aspects that can be researched with the research goal out in the open. For example, a previous study has investigated the effect viewing angle and distance have on the success of shoulder surfing attacks [6]. Here, a lab study can offer control to isolate research factors from others that would introduce too much complexity to the experiment.

In-the-Wild or Field Studies

Researching the phenomenon of shoulder surfing with in-the-wild or field studies sheds more light on the contexts in which these attacks take place and could provide insights into the behavior of attackers and victims. However, performing these studies is very challenging and, thus, rarely conducted. One of these studies was a two-week in-the-wild study conducted by Schneegass et al. [48], where they investigated the likelihood of shoulder surfing attacks occurrence during unlock events. Nonetheless, shoulder surfing is socially unacceptable and privacy-invasive. Hence, observing these attacks requires consent, potentially biasing participants and making it very difficult to observe authentic interactions. Moreover, outside the lab, bystanders get involved quickly; when that happens, their consent is also necessary (e.g., when recording video for eye tracking). In the past, researchers have primarily relied on surveys and interviews to assess in-the-wild experiences [14], relying on self-assessment as the most frequent research method. To encompass both the benefits of a study in the lab (such as its associated high degree of control) and to enable researching more realistic (in situ) shoulder surfing scenarios, researchers have applied eXtended Reality as a study method.

Studies in Extended Realities

Recently, eXtended Reality (XR) [42] entered human–computer interaction (HCI) as a means to conduct user studies that are not directly related to XR but use XR as a modality to conduct user studies instead (e.g., [31]). This is particularly the case for user studies that are taking place in virtual reality (VR) in a virtual environment (VE), whereas XR could implicate “augmented reality” (AR) or “mixed reality” (MR) as well. The trend of using XR as a research method got amplified with the ongoing Covid-19 pandemic as different frameworks appeared [19, 40].

Using VR to research the shoulder surfing phenomenon has several inherent benefits. First, a virtual environment allows a more believable recreation of a real-life situation, which would otherwise be hard to recreate in the lab (e.g., a bus stop or office environment with different people present [2]; see Fig. 2). In addition to the realistic recreated scenes, VR allows maintaining the consistency among study participants, avoiding external uncontrolled situations. With eye trackers embedded in the head-mounted displays (HMD), researchers are able to capture and analyze the gaze of the participants. Accordingly, they are able to profoundly understand the observation attacks cycles and expect what triggers the observers’ attention. As VR is associated with a high degree of immersion, it allows placing the subject in a simulated, virtual environment, where they can experience the situation as intended by the researchers. Here, the degree of presence can be assessed through the usage of presence questionnaires [50, 51, 59].

Fig. 2
2 virtual reality images. The image on the left exhibits an open office space. A location is plotted on the image. The image on the right exhibits a bus stop with 2 people and a location is plotted.

Example taken from a previous paper that studied shoulder surfing in virtual reality [2]. The figure shows two virtual scenes that were used to investigate observing others’ displays in an open office space (left) and a bus stop (right). The read markers indicate the participants’ initial position

Full size image

Potentially, such studies can also run outside the lab on HMDs owned by participants [40], and they were validated for usable security evaluations [35]. Additionally, user studies in XR allow fulfilling particular requirements specific for shoulder surfing studies. One is privacy, as conducting a user study in a real-world environment with real victims can be considered ethically challenging, whereas shoulder surfing a virtual avatar in a virtual environment (VE) is less likely an issue. Furthermore, conducting a user study in a VE allows for a very high degree of control since the environment is simulated by a computer, often exceeding the capability of control that an experimenter has over a real-world situation, even if it takes place in a lab. The high degree of control allows for replicability of such user studies between participants, as the experienced situation can be made to be precisely always the same.

2.3 Key Findings on Shoulder Surfing Behavior

With the growing number of studies investigating shoulder surfing events, we highlight the key findings on observers behaviors that we believe are of high relevance.

Observations Are Often More Random Than Planned

In the survey by Eiband et al. [14], the main findings showed that despite the fact that observations are frequently conducted on an opportunistic basis, they go beyond exposing the authentication. Several participants reported negative feelings when other content such as personal photos or texts are exposed.

Victim–Attacker Pose Relationships Are Unalike

In 2021, Saad et al. [46] explored the tendency of bystanders to shoulder surf in a scenario within an underground train. To that end, they varied the point of view of the attacker (standing vs. sitting) and the position of the victim (again standing vs. sitting) and used a 360∘ camera to obtain a photorealistic recording of this setting, where several actors played either the role of the victim or became extras to simulate other people on the train. This recording then was played back to participants in a user study on an HMD that was equipped with an eye tracker in a lab study, and the point of view of the participants is seen in Fig. 3. Through the eye-tracking data, it was apparent that participants gazed at the object of interest, a smartphone held by the victim, and 11.16% of the time they were nearby.

Fig. 3
4 photos of the passengers using mobile phones on the metro train.

User study conducted in virtual reality to investigate shoulder surfing attacks with prerecorded 360∘ videos [46]. Left to right: viewpoints of the participants with four different relative poses to the (virtual) victim: standing to standing, standing to sitting, sitting to standing, and sitting to sitting

Full size image

VR Reflects Genuine Behavior…

In 2022, Abdrabou et al. [2] conducted another project on the understanding of shoulder surfer behavior and the associated attack patterns. Here, they created a simulation in virtual reality with virtual, human-like avatars who were either located at a bus stop scene or within an office. The human participant of this study then was placed inside this VE through a VR HMD, which was again equipped with an eye tracker. The experimenters then recorded the participants’ gaze and their walking patterns in VR and found that participants looked at several objects of interest (e.g., smartphones in the bus stop scene or monitors in the office scene) 5.7 times on average, whereas the average eye contact duration was 1.61 s.

…but Immersion Is Needed.

Also in 2022, Mathis et al. [34] considered the differences between non-immersive and immersive VR for shoulder surfing research and conducted a user study to explore the characteristics of both settings. They considered shoulder surfing attacks on automated teller machines, smartphone personal identification numbers (PIN), and smartphone pattern unlock mechanisms. They compare three scenarios, 2D video observations, 3D observations, and VR observations. The first scenario, 2D video observations, consists of the study participants watching a video of the shoulder surfing situation that they cannot influence on a traditional computer monitor, whereas in 3D observations, they could use the keyboard and mouse to walk around. These two conditions then were compared against each other and VR observations, where participants were wearing a VR headset and could freely move around and adjust their observation perspective. The authors found that VR observations lead to a significantly higher sense of presence and involvement and that VR observations also lead to the most accurate shoulder surfing observations.

There Is More than Smartphones

There are other devices that are becoming more ubiquitous nowadays, smartwatches for instance. Recently, more studies are proposing authentication approaches for smartwatches, with resilience against shoulder surfing as a key metric for robustness [38, 39].

In conclusion, we can observe that there is an increasing number of publications that utilize XR, particularly VR, as a research method for shoulder surfing research. The high degree of immersion lets the participants of a user study easily take the role of the attacker, while such a lab study setting allows for an efficient resolution of the problematic aspects connected to ethics in this kind of research. Furthermore, VR allows the study to be exactly the same for each subject, as the computer-driven simulation creates an easily repeatable environment. Thereby, realistic scenarios can effectively be replicated in the lab.

3 Mitigating Shoulder Surfing Attacks

For the mitigation of shoulder surfing attacks, it is important to note that not every shoulder surfing incident is equally problematic. One important aspect to consider is the type of content visible. For content-based shoulder surfing, we need to understand what is considered sensitive content as it plays an important role in selecting a suitable mitigation strategy. To tackle this challenge, Farzand et al. [17] present a typology of perceived sensitivity that can help to understand the content sensitivity. Furthermore, one needs to take into account that the perception of shoulder surfing is different between cultures [47]. As a consequence, it also differs what is considered sensitive content.

In the following section, we look at research that aims to find solutions to mitigate shoulder surfing attacks. Therefore, we start by looking at different threat models against which researchers and practitioners can evaluate their mitigation strategies. After that, we briefly describe technical approaches to detect shoulder surfing and their current limitations. Finally, we present an overview of different mitigation strategies.

3.1 Threat Models

Threat models provide a systematic approach to investigate potential weaknesses to privacy and security [32]. For shoulder surfing, different threat models have been considered in the literature. Below, we provide a selection of these models and describe them briefly. It should be noted that also mixes of these are possible (e.g., a repeated attack that is technology-supported [7]):

Weak Attacks::

A shoulder surfing attack is considered a weak attack if it is performed by a human observer without the help of any technology and with only limited practice [11].

Trained Shoulder Surfers.:

Compared to weak attacks, trained shoulder surfers are more effective by training themselves. They often employ cognitive strategies that help to reach higher success rates [26]. Please note that trained shoulder surfers manage to be more effective without using recording devices.

Repeated Attacks::

The repeated attacks threat model assumes that an attacker can repeatedly observe the target device of the victim. Moreover, this threat model often considers the attacker to be at close range—the attacker quite literally looks over the victims’ shoulder [7].

Insider Attacks::

Quite similar to the repeated attacks threat model are the insider attacks. The main difference is that for this type of attack, family, friends, or colleagues perform them. They may repeatedly observe the victim, and by combining these partial observations, it is easier to form a hypothesis on the victim’s secret [57].

Multiple Attackers::

The shoulder surfing attacks become more threatening when multiple attackers try to observe the target device. In this case, attackers can coordinate by either focusing on specific parts or organizing distraction and information stealing roles between attackers [24].

Technology-Supported Attacks::

The probably strongest form of shoulder surfing attacks are technology-supported ones. In these cases, an attacker is recording the victim’s interactions, for example, when drawing money from an ATM [10]. With recent technology advances, camera-based sensors can be manufactured in very tiny proportions, allowing attackers to seamlessly integrate them in their clothing or accessories. When analyzing the recorded data with machine learning, breaches of privacy are possible even when the attacker is not direct line of sight because reflections on glasses are sufficient for reconstruction of screen content [60].

3.2 Algorithmic Detection of Attacks

To mitigate shoulder surfing attacks, they first need to be detected. In previous research, detecting shoulder surfing attacks is primarily achieved by focusing on the human attackers. Here, algorithmic approaches oftentimes rely on visual sensor data (i.e., monochrome and RGB cameras). As shoulder surfing is frequently researched for mobile devices, the built-in camera is a good source for visual information to detect attackers. For example, Ali et al. [3] investigated the use of the built-in camera on mobile devices to detect if an unauthorized person tries to gain access to the device. Here, to detect an observer, face detection is applied to the incoming video feed. Interestingly, popular operating systems such as Android come with real-time face detection capabilities that can be used for detecting shoulder surfers [7]. Nonetheless, not every detected face is necessarily a potential attacker as other factors play an important role as well, such as gaze direction and context, among others. In a recent study, different angles and distances have been investigated to understand which of them are most critical as they provide a good position for shoulder surfing [6]. The threat model was also based on evaluating people’s perception on the displayed content that varied between visual, textual, and authentication, as seen in Fig. 4.

Fig. 4
4 photos of the smartphones placed on a phone stand rotated between 0, 30, and 60 degrees.

Study apparatus to investigate the influence of distance and viewing angle on shoulder surfing success rate, figure taken from Bâce et al. [6] licensed under CC BY-NC-ND 4.0. The subfigures show examples of different content types on the phone display: (left to right): text, PIN, photo, and no content visible. The mechanical prototype visible rotated the smartphone between 0, 30, and 60∘

Full size image

Nevertheless, visual detection of potential shoulder surfing also comes with a few downsides. First, they require the camera to be active and to record the scene. This scene likely involves the users of the device as well and, thereby, introduces another privacy risk. Furthermore, not only the privacy of a user may be violated, but also that of bystanders (as it continuously records the scene). Another issue is that the continuous recording and processing of the video feed drains the battery more quickly [7]. Hence, researchers have explored other options as well. For example, Lian et al. [29] used “multiple sensors, i.e., video camera module, ultrasonic distance module, light sensor module, to detect screen peeping, user distance and environmental lightness.” Here, future studies should compare the different sensor technologies and develop adaptive strategies that take the context into consideration. For example, when a user is logged in to their wireless network at home and no other Bluetooth signatures are around, continuous monitoring via the built-in camera to detect shoulder surfing may not be necessary.

3.3 Prevention Strategies

Oftentimes, a detection algorithm proposed by researchers goes hand in hand with an implementation of a mitigation strategy (cf. [44]). In the following, we discuss two different strategy types into which proposed systems can be classified. On one side, there are strategies that try to be generalizable toward every kind of content, and on the other side, there are strategies that focus on mitigating attacks against specific types of contents. These two strategies are in line with how we categorize shoulder surfing attacks into authentication-based and content-based shoulder surfing. Here, it is important to note that while authentication-based shoulder surfing is perceived as more problematic, content-based shoulder surfing is occurring more frequently [18].

Strategies Independent of Content

Often times, researchers propose systems that mitigate shoulder surfing attacks independent of the content shown by the target device. Different systems have been proposed that try to create awareness for an actively ongoing shoulder surfing attack. For example, Ali et al. [3] proposed a system that informs users whether text on the screen could be read by an attacker. To better understand, in which way users want to be alerted, researchers have conducted a user study to compare four different methods: vibro-tactile, front LED, on-screen icons, and video feedback, finding that vibro-tactile feedback works best, as seen in Fig. 5. Their findings showed that vibration feedback allowed for a faster response time, in comparison to the other three methods [44]. Moreover, it has been examined how additional parameters such as distance and orientation can benefit victims in applying appropriate actions [62].

Fig. 5
4 photos of the screenshot of a Facebook post on a smartphone. The tab Post a status update is exhibited on the photos.

Different feedback conditions to communicate a shoulder surfing incident investigated in previous work [44]. The different feedback conditions are (from left to right): (1) front LED, (2) video preview, (3) vibro-tactile, and (4) on-screen icon. The authors found that vibro-tactile feedback results in the lowest reaction time

Full size image

While awareness-based systems leave it to the user to decide on how they want to react, researchers have proposed different strategies that help users in their actions [9] or automatically react to shoulder surfing attacks [29]. Here, users can either move or hide information presented on the screen by performing explicit interactions [9] or information is automatically masked [9, 29] (e.g., with the help of eye tracking [41]). Lian et al. [29] found that with limited brightness or contrast, only the user could read the screen, while others have trouble reading it [29].

Furthermore, different strategies have been proposed that do not rely upon detecting a shoulder surfer at first, but rather are applied constantly. For example, Chen et al. [12] developed Hide Screen, which utilizes human vision characteristics to preserve privacy. Simplified, the approach allows changing the readability of information based on the viewing angle. Instead of hiding the information from an attacker, Watanabe et al. [55] suggest adding additional information that is designed to throw an attacker off. They suggest showing multiple cursors on the screen and, thereby, effectively hiding the real cursor for an observer. Finally, it has been proposed to extend an observable screen with a second screen that is not observable and can be used to show private information. For example, Winkler et al. [58] are using smartglasses to show private information that would have otherwise be shown on the smartphone display.

Strategies Focused on Specific Types of Content

Because not every type of content requires the same level of protection, many proposed strategies that are highly dependent on the type of content that they protect. In particular, authentication approaches need high protection against shoulder surfing attacks. Hence, researchers have suggested a variety of authentication techniques that are more resilient against observational attacks.

Bianchi et al. [7] proposed to use a composition of non-visual cues (i.e., audio and haptic cues) to enter a password. As a result, an observational attack cannot rely on visual information only to decipher the password. Furthermore, others have suggested to use gaze as an input modality in combination with graphical passwords [10]. Thereby, an attacker would need to observe the eye gaze of the victim additionally to the phone screen, making it very challenging to reconstruct the password. Another strategy is to extend the input surface for the authentication scheme toward the backside of the smartphone, which is more difficult to observe [13].

Besides authentication approaches, researchers have focused on other types of content. For example, Eiband et al. [15] have investigated how text can be presented in a way that is readable to the user but unreadable to an observer. In essence, they propose to display text in the user’s own handwriting. While this does not prevent an attacker from reading the text, it significantly slows them down.

4 Challenges and Future Research Directions

In the following, we present challenges and research directions concerning the methodology of researching shoulder surfing and the phenomenon itself. These are particularly related to the methodology of shoulder surfing research and the attacker’s behavior.

Research Methods to Investigate Shoulder Surfing

While conducting research on shoulder surfing in the wild, several challenges regarding the methodology became apparent. First of all, a central element is an ethical dilemma associated with the necessity of obtaining the shoulder surfer’s consent. When researchers ethically design an experiment on shoulder surfing that involves participants, participants usually have to get into the role of either the victim or attacker. However, shoulder surfing usually is an interaction that is very affective by its nature [14], hence instructing participants on the roles that they should get into highly inflects their behavior, and thus, results elicited from the study. Consequently, there is a dichotomy between asking for consent and subjects’ unchanged behavior that needs to be weighed individually for each study, taking the objectives of the study into account.

Another argument on shoulder surfing studies is to simultaneously consider both roles of the attacker and the victim. Considering only the role of the observer and not the victim could leave out vital parts of the shoulder surfing incident, such as the occlusion of the phone display by the victim [6].

Virtual Reality for User Studies

To overcome some of the challenges related to this ethical dichotomy, several research projects utilized virtual reality to simulate the shoulder surfing interaction with virtual avatars [2, 34, 46]. Although it is not necessary to obtain consent from a virtual avatar that has the role of the victim, it, however, still is necessary to obtain consent from a participant that gets into the role of the attacker. Furthermore, virtual reality allows for a simulation of the environment; hence, the interaction can be explored in different settings that would be hard to replicate in a physical lab.

However, virtual reality is also only a limited solution, as there are certain aspects impacted by the simulation of the environment. For example, today’s head-mounted displays can influence people’s behavior such as their movement [20] or also their social comfort distance that is less in virtual reality than in reality [22]. They can, however, help in recreating scenarios from the real-world by simulating them in a lab, as conducting field studies or in-the-wild experiments is particularly challenging due to the ethical aspects, particularly, when uninvolved third parties become part of the investigation. The same applies to other methodologies such as the usage of recording videos outside the lab, the so-called “lifelogs”, as using cameras impacts the protection of private information of both the wearer and potential bystanders [21].

Identifying Sensitive Content

In general, two types of shoulder surfing are distinguished: authentication-based and content-based shoulder surfing. While authentication-based shoulder surfing is inherently problematic as it exposes sensitive information (e.g., PIN or password), it is more complicated for content-based shoulder surfing that happens more frequently [18]. Privacy is an individual concept. Hence, what one person considers sensitive information may not be considered sensitive by someone else. This makes it very difficult to have an overall solution that equally protects all users. As a consequence, we need to investigate what content is considered sensitive (e.g., [17]). Furthermore, we need to examine different factors that can influence the perception of what is considered sensitive content such as cultural differences [47].

Understanding the Attacks and Behavior

Another open research direction is to create an understanding of the shoulder surfing interaction itself, by, for instance, creating models of it. Here, Abdrabou et al. have created one of the first works in creating a model of attack patterns [2]. Their study took place in virtual reality; hence, creating a model-based understanding of the phenomenon, in reality, is still an open research opportunity nowadays. It is therefore necessary to conduct further studies to determine more substance to derive models about behavior within more contexts of the interaction. This includes, but is not limited to, in-the-wild studies as well as long-term studies to understand, whether the behavior changes over time.

Additionally, recent studies focus on password attacks but do not have a strong focus on understanding shoulder surfing behavior in general [8]. However, when considering only the attacks on passwords, such as android pattern locks, models were already created that predict the grade of observability [54]. This also opens up the opportunity to further explore the type of content that is particularly attracting shoulder surfing attacks, which partly has been covered by recent studies [2, 46].

5 Conclusion

In this chapter, we presented lessons learned from research on the shoulder surfing phenomenon and attack mitigation strategies. We started with a definition of shoulder surfing and an introduction of different types of attacks. After that, we present different research methods that have been applied in the past and discussed key findings related to shoulder surfing behavior. Next, we gave an overview of different threat models for shoulder surfing and discussed algorithmic detection of these attacks and different mitigation strategies. We concluded the chapter with an outlook on persistent challenges and future research directions. We believe that this book chapter offers a great starting point for new researchers and practitioners in the field. Moreover, we see great potential for eXtended Reality to overcome the limitations that field and in-the-wild studies introduce.