Keywords

1 Introduction

Sketching is an important ancient human skill stimulating creative, visual and spatial thinking. Computer systems that support sketching have been studied since the early days of computer science [5]. Through the development of mobile computing devices, such as smartphones, Magic Lens (ML) became a popular interface to support user sketching (e.g. [2–4, 6, 7]). ML acts as filter augmenting the scene with additional digital content, for example, by adding 3D models based on a recognized paper sketch [2] or by allowing the creation of new 3D content [4, 7]. In this paper we explore how novice users can be supported in creating physical sketches through virtual tracing, i.e. creating a physical sketch on paper given a virtual image on the handheld device.

The core challenge for this (and other sketching) tasks, which involve a physical pen and paper, is to provide accurate and robust pose tracking without disrupting the sketching process. So far, AR sketching systems focused on authoring digital objects and relied on external tracking systems (e.g. [4]) or on marker based tracking (e.g. [6]). Here we focus on utilizing AR as a crafting tool for curtain of real objects (e.g. pen drawings). While marker based tracking would allow in-situ sketching on physical paper in otherwise unprepared environments (Fig. 1b), it limits the sketching experience in a fundamental way: the marker has to be in the camera view taking away valuable space for sketching. While approaches, such as contour tracking [1, 2] circumvent the use of a marker, they are prone to failure as they cannot be occluded during interaction.

Fig. 1.
figure 1

(a)–(c) Mobile device sketching aid tools on the study; (d) D-ML transformations

Full size image

An alternative is to eliminate the need for pose tracking by placing the device and drawing surface at fixed position, such as in the case of a virtual mirrorFootnote 1 or camera sketcher.Footnote 2 In both situations the user has to manually position the graphical content into the real world using traditional handheld interfaces such as static peephole (SP). Hence, if the drawing format does not fit into camera’s field-of-view (FOV), the user is required to manually realign the graphic every time the device is moved.

In this paper we address the challenge of pose tracking while mitigating the effects on in-situ sketching experiences. Specifically, we evaluate how utilizing both front and back facing cameras concurrently could improve the utility of ML as a sketching aid tool. In order to do so, we: (i) design and build a Dual Camera Magic Lens (D-ML) system utilizing the front camera for pose tracking and the back camera for scene capture and rendering; (ii) evaluate the proposed solution on a commercially available handheld device by conducting a preliminary user study with 6 participants performing virtual tracing task. We compare 3 interaction methods: Static Peephole (SP), Magic Lens (ML) with fiducial marker and Dual Camera Magic Lens (D-ML).

2 Dual Camera Magic Lens

In contrast to using the back facing camera as in standard handheld AR applications we propose to utilize the front facing camera for pose tracking while providing the view of the scene through the back facing camera. We do this in order to mitigate the effects on in-situ sketching experiences.

In order to enable front camera pose tracking, a marker is placed above the drawing surface in parallel orientation (in our case ca. 60 cm, see Fig. 1c). As the tracking and rendering camera are not the same, a set of additional transformations (Fig. 1d) needs to be added to the tracked pose result (on Fig. 2a denoted as RTFC->TM). However, as the two cameras on the phone are fixed and the top marker is rigidly attached above the drawing area, the only two transformations that change are RTFC->TM and RTBC->DS. Hence, as long as the front camera tracking is successful and the two constant transformations are known, RTBC->DS can be calculated (Fig. 1d).

Fig. 2.
figure 2

(a) Drawing quality ranking results (smaller is better); (b) “Overall reactions” part of QUIS scores [1–9]. (c) Preference ranking results (smaller is better); (d) Task time in minutes

Full size image

3 User Study

The preliminary user study asked participants to perform a virtual tracing task on A3 paper. Participants were instructed to sit at a table and draw a cartoon character as quickly and as accurately as possible. To estimate participants’ perceived satisfaction we are utilizing the “overall reactions” part of the Questionnaire for User Interaction Satisfaction (QUIS). In addition, we asked participants to rank interaction modes and to justify their choice. As objective measure we recorded task completion time.

We used a within-subjects design. Each participant drew three different contours with each interaction method: SP, ML, D-ML (see Fig. 1). In SP manual alignment is required each time the phone is moved. In ML the drawing is possible whilst holding the phone in hand, whereas in SP it is mandatory to place the phone on the stand. In case of the D-ML, the stand was included because contrary to the tracker used in ML implementation, the tracker used in D-ML did not provide sufficiently robust and accurate orientation tracking results. By placing the phone on stand, we locked two degrees of freedom (Rx and Ry) improving tracking quality. We recognize this as a limitation. However, this decision was mandatory as poor tracking quality is bound to undermine performance of the proposed interaction paradigm. Additionally, as it should be possible to improve tracking performance of future D-ML systems, this does not undermine the proposed interaction concept per say, but rather affects the direct comparability of captured results. Yet, within the context of this study which is of exploratory nature and predominantly based on qualitative data, we consider our study design as appropriate.

4 Results

The study was completed by six participants. All were male, aged between 24 and 45 years. Due to the small number of participants null-hypothesis significance testing would result in poor statistical power. Hence, we present solely descriptive statistics. By overlaying drawn contours with template contours, two researchers independently and subjectively compared the quality of all three drawings for each participant and ranked drawings from best to worst. As shown in Fig. 2e, the comparison did not highlight any obvious deviations in obtained rankings. The results in Fig. 2 also suggest that D-ML has the potential to (i) be the fastest mode; (ii) have the highest QUIS score across all properties; and (iii) have best rank. Again those results could not be reliably tested for statistical difference. Five participants that ranked D-ML as the best method justified their ranking choice by highlighting the advantage of automatic alignment and the fact that marker was not in their way.

5 Discussion and Conclusion

The results show that the utility of handheld AR as an in-situ sketching tool has the potential to be improved utilizing the D-ML approach. The designed and implemented system demonstrates such a solution is feasible on commercially available mobile devices. The preliminary study indicated that whilst achieving comparable quality of drawing, compared to SP and ML, D-ML is: potentially faster, users perceived higher satisfaction, and is preferred by participants. We believe the main reason for such a results is the fact the camera tracking did not interfere with user sketching. As underlined by participants themselves, the main benefit is automatic alignment of virtual image with the real word. Even though, until the user moves the marker, automatic alignment is also present in case of ML, in D-ML the user did not have to put up with the marker and avoid occluding the marker whilst trace drawing onto the paper. Although one could argue that the stand ambiguity increased the divide between the ML and D-ML, none of participants highlighted the stand as the factor influencing their ranking choice, suggesting the importance of the stand might be limited. However, in future work we will explore the effects of stand vs. handheld mode in more detail. Additionally, due to the small sample size our results are of preliminary nature and hence should be verified with a larger number of users. Finally, future research should look at less intrusive ways of placing a marker above the user.