Keywords

1 Introduction

Color is a salient visual attribute for both paintings [1] and our daily life. When searching paintings on the keyword-based image search engine (e.g. Google), we found that most of queries describe the expected paintings by use of color names such as ‘golden lady in green’, ‘yellowish hat’ etc. However, the huge ‘semantic gap’ [4] makes search results unsatisfactory. Moreover, it is difficult for users to describe some paintings and pictures in several keywords, such as abstract paintings.

With the proliferation of the touch-screen mobile devices, people are able to express their intention more freely. Zhang et al. [6] proposed that most users could draw out rough color-sketches of previously-seen paintings from their memory. As one of the image retrieval paradigms, Banfi [5] concluded that Query-by-Drawing (QbD) was the most flexible way for users to present their search ideas. There are many studies on QbD algorithms [7, 8] which focused on the search accuracy and efficiency, however, little is known about the design guidelines of the natural and intuitive user interfaces that facilitate user’s drawing.

For the small-screen mobile phone, drawing color patches [9] roughly is more reasonable [10]. One of the most important tasks for drawing is color selection. There are a variety of ways to select colors, and the most common method is to directly manipulate a color palette [3]. The color palette usually includes a color model and an interface to represent colors to users. Several studies were conducted to explore the effects of different color models on color matching task. Schwarz et al. [11] compared five color models: RGB, HSV, LAB, YIQ and Opponent, and they found that RGB color model was the fastest to use yet least accurate, while HSV color model was the most accurate to use. However, these arguments were only concluded from conducting the color matching experiment, which ignored the color selection interface for drawing purpose.

This work aims to focus on the user interface of color palette, by focusing on a specific color model and a current Query-by-Drawing algorithm, to explore the influence of different color palette interfaces on color selection for drawing query on mobile phone. Specifically, an experiment was conducted to collect task performance and users’ attitudes on five different color palette interfaces.

2 Related Color Palettes Study

The color palette has been widely explored by the scientific visualization researchers [12], with aims of displaying information efficiently by combining proper colors. Looking through current mobile application (referred to ‘App’ in the remainder of this paper) market and computer software, including mobile drawing Apps, coloring games for kids, note-taking Apps, and photo editing software and Apps, we found that a plethora of different color palettes were adopted by different developers according to their individual preference. To the best of our knowledge, there is no design guideline about making use of color palettes for drawing on touch-screen mobile devices, especially for image search by drawing mobile App.

As mentioned earlier, some previous work has investigated the effects of color model on the speed and accuracy of color selection in color matching experiment [11, 13, 14, 18], in which participants choose a closest color to a given color shown on the computer screen by use of the color palettes with different color models. Besides Schwarz et al.’s study [11], Everly and Mason [14] also assessed the usability and accuracy of four color selectors via exploring participants’ performance on color matching. Dogulas and Kirkpatrick [18] found that the color space had little impact on color selection. However, the color matching task is out of context of the real-world drawing scenario in which choosing one color might be influenced by the surrounding or background colors.

There are a few studies on the color palettes used in Content-based Image Retrieval (CBIR). For example, Broek et al. [15, 16] reviewed ten color selection interfaces for CBIR and categorized them into three groups: slider-bars, discrete color matrix, and continuous color square. However, their work was conducted on the computer-based CBIR systems almost 20 years ago. Their limitations include: (1) non-intuitive interaction of color selection by using mouse; (2) out-of-date color palette interfaces; (3) not representative of the current mobile device capability. Zhang et al. [17] compared two discrete color palettes with two continuous color palettes (HSV square and HSV color wheel) on mobile phone and concluded that participants preferred the color palette with full-choice of colors during drawing search query. Inspired by Zhang et al.’s work [17], our purpose is to explore which kind(s) of continuous color palette interface work best for a Drawing-to-Search mobile App. We also adopt HSV color model in this work, because it is perceptually intuitive and is used by lots of current mobile applications.

Current color research all define a color in a triplet of three parameters, hence selecting a color means finding a location in a three-dimensional (3D) space. We divide the continuous color palette interfaces into ‘1D-1D-1D’ and ‘1D-2D’. In ‘1D-1D-1D’ interface, each one of three parameters is controlled by one motion (such as a trio of 1D sliders); while in ‘1D-2D’ interface, one motion simultaneously change two parameters and one for another parameter, e.g. color palettes shown in Fig. 1. In this paper, we focus on comparing the ‘1D-2D’ color palette interfaces based on the human-oriented HSV (Hue-Saturation-Value) color model, because this pair of interface and color model is frequently used in both current color selection tools and the graphics literature.

3 Query-by-Drawing Application

We built five Client/Server-based drawing Apps on Android phone, and each App provided the same basic drawing tools except the color picker. The basic drawing tools include brushes with five different sizes (10, 20, 30, 40 and 60 pixels), ‘Brush color picker’, erasers with five different sizes (10, 30, 40, 50 and 70 pixels), ‘Canvas color picker’, ‘Undo’, ‘Redo’, and ‘Trash’ on the top of canvas (see Fig. 1, left). We separated brush color and canvas color to facilitate user to create and refine drawings easily, and the same color selection interface is used for ‘Brush color picker’ and ‘Canvas color picker’ of each App. Each color palette interface could keep 20 recently used colors.

Fig. 1.
figure 1

The screenshots of drawing and result-viewing interface of our Query-by-Drawing App.

Full size image

When users complete one drawing, they tap the ‘Search’ button to submit their drawings to our server, which will extract drawing’s color features and match them to that of database images by using Fast Multi-resolution Image Querying algorithm [2]. Top fifty-one matched images, three image thumbnails on a line and 17 lines in total, will be returned from server to user’s mobile phone and displayed in a descending order of visual similarity. That is, the most similar painting is at the top-left (e.g. Fig. 1, right). The reason behind showing fifty-one results is that users usually view results on the first few pages and fifty-one image thumbnails are on the first three screens. The database contained 400 portrait paintings downloaded from BBC Your Paintings websiteFootnote 1.

4 Experiment

The ‘1D-2D’ color palette interfaces could be designed in a variety of ways in terms of the combinations of three axes of HSV color model and the shapes to display these combinations. We are interested in (1) if the combinations of three parameters of color model have measurably effects on users’ color selection; and we also want to see (2) how visual representations of parameter(s) influence users’ performance on drawing.

4.1 Experimental Design

Based on the results of reviewing 140 drawing Apps, 22 coloring Apps, and 18 photo-editing Apps available on Google Play store, we adopted four commonly used shapes of color palette interface, including square, wheel, ring and bar. The square and wheel were used for displaying the ‘2D’ combinations, and ring and bar were for the ‘1D’.

The bar-square interface is explored to answer the first research question. For ‘1D-2D’ interfaces, there are three combinations of HSV color model: H-SV, S-HV, and V-HS. The ‘2D’ is displayed as a square, and ‘1D’ is controlled by a vertical slider bar. These three combinations were named as H-SVsquare, S-HVsquare, and V-HSsquare, as shown in Fig. 2, and the comparison of these three interfaces was comparison ‘C1’.

Fig. 2.
figure 2

Five color palette interfaces: (a) H-SVsquare, (b) S-HVsquare, (c) V-HSsquare, (d) V-HSwheel, (e) Vring-HSwheel

Full size image

As for the second research question, firstly, we built a slider bar and a ring of V (Value) with HS color wheel, which were formed into V-HSwheel and Vring-HSwheel as shown in Fig. 2. We defined comparison ‘C2’ as V - HSwheel vs. Vring - HSwheel. Secondly, we were also interested in looking at two different representations of HS with slider bar: square vs. wheel, that is, comparison ‘C3’: V - HSsquare vs. V - HSwheel.

The experiment adopted a within-subjects design. The independent variable is ‘Interface Type’ (five Apps), and the dependent variables include performance measures and self-rated satisfaction and interface usability obtained from questionnaires.

Participants.

Forty-one college students (mean age 19 years, s.d. 1.88) have volunteered to take part in the experiment. All participants reported having experience of using touchscreen mobile phone and none of them self-indicated as drawing-expert or artist. All participants were novice users of our Apps and right-handed.

Material and Stimuli.

Two Google Nexus 4 (133.9 mm × 68.7 mm) mobile phones with Android 4.4 operating system were used: One mobile phone was used to present the reference painting to participants; and five Apps and SCR Screen Recorder Pro were installed on the other phone on which participants would draw. Both phones were adjusted to the same level of brightness to reduce the device difference effects.

The choice of painting as stimulus followed the following rationales: (1) The painting is not well-known; (2) The painting has at least six rough color blocks, which contains the color widely used in literature and agreed as the basic colors [18], e.g., blue, red, and green etc. We believe the painting Old man (shown in Fig. 3) is sufficient to evaluate user’s color selection performance with at least seven gist color blocks.

Fig. 3.
figure 3

The reference painting Old man used for copying task.

Full size image

4.2 Procedure

Only one participant involved in study each time. Participant was briefly informed of the aim and procedure of our study, then he/she was instructed to complete the pre-task questionnaire before a tutorial on how to use our mobile Apps. After a practice session (for eliminating individual difference effects), each participant was asked to complete the copying task (copying painting Old man shown in Fig. 4) by use of five Apps in a random order to minimize the practice effects. One drawing was required on each App. We encouraged participants to draw the rough color sketches with accurately tracing color and proportion of the original painting. At the end of each drawing session, participant was asked to rate on the usability with 7-point Likert scale in terms of the drawing speed, the level of ease-of-use, the similarity of his/her drawing compared to the reference painting, and the overall preference on the color selection interface.

Fig. 4.
figure 4

The Percentage of drawings that get the Old man on top 51 result lists.

Full size image

After completing all tasks participants were interviewed to specify which interface they believed to be their favorite one among five Apps, and also the preferred square-based color palette and wheel-based color palette. The time spent on drawing was captured by SCR Screen Recorder Pro. And the position of Old man in search results for each query was recorded by the experimenter. All procedures were video-recorded.

5 Results and Discussions

206 drawn-queries were produced with relevant retrieval results and drawing-videos recorded. The time taken to complete a drawing, the retrieval result, and the self-rated satisfaction with both drawing process and color selection interfaces were analyzed.

Participants spent the shortest time on V - HSwheel (mean = 4 m 31 s) to complete one drawing, followed by Vring - HSwheel (mean = 4 m 47 s), and the longest time taken on V - HSsquare (mean = 5 m 46 s). A paired samples t-test revealed that the difference between V - HSwheel and V - HSsquare was highly significant (t(40) = −2.786, p = .008), whereas no significant difference was found between V - HSwheel and Vring - HSwheel (t(40) = −1.089, p = .283). A repeated measures ANOVA with a Greenhouse-Geisser correction indicated that there were no significant differences (F(1.620, 64.816) = 0.197, p = .775) among the time taken to complete one drawing on H-SVsquare, S-HVsquare and V-HSsquare. We also found that the time spent on drawing a query gradually decreased as participants’ experience gained.

Figure 4 illustrates the proportion of drawings on each App that got the Old man in top 51 results. The V - HSsquare worked best (71.4 % drawings succeeded) among five Apps, and V - HSwheel (64.5 %) was better than Vring - HSwheel (58.8 %).

Regarding to the usability of color palette interfaces and the user preference, user’s self-rated Likert-type scales were collected from questionnaire, were statistically analyzed by Friedman tests and Wilcoxon tests with a significant level of 0.01. The Friedman tests were carried out for the comparison C1. There were no statistically significant differences in the interface usability (first seven feature rows illustrated in Table 1) and the preference among three HSV axes combination interfaces (H-SVsquare vs. S-HVsquare vs. V-HSsquare), as shown by column C1 in Table 1.

Table 1. The statistics analysis of self-rated usability, user preference on color palette interface.
Full size table

For participants’ attitudes to drawing process and preference, the Wilcoxon’s singed rank tests indicated that V - HSwheel was highly significantly better than Vring - HSwheel in terms of participants’ satisfaction rankings on Relaxing-to-draw (Z = −3.123, p = .002), Quick-to-draw (Z = −2.701, p = .007), Easy-to-draw (Z = −3.257, p = .001), similarity of drawing (Z = −2.853, p = .004), and easy-to-pick wanted color (Z = −2.755, p = .006). While comparing V - HSwheel and V - HSsquare, a highly significant difference (Z = −2.644, p = .008) was only found in the ratings of enough color-choices, that is, participants thought that V - HSwheel provided more color choices than V - HSsquare. But in fact, both interfaces supply the same number of color choices.

There were no statistically significant differences in participants’ ratings on interface preference. However, from answers to question about which interface was preferred in each group of bar-square and wheel-based color palettes, most people chose H - SVsquare as their favorite bar-square color picker, and Vring - HSwheel as their favorite wheel-based color palette. And they preferred H - SVsquare color interface overall.

It is found that each drawing contains 6 ~ 8 colors on average. A paired samples t-test revealed that the difference between the number of colors used for drawing on V_HSwheel (mean 6.87) and V_HSsquare (mean 7.46) was highly significant (t(40) = −3.438, p = .001).

It is observed that how participants draw varied significantly, however, several common issues of drawing behavior emerged. First, participants with high-level drawing skills, e.g. sketching or oil painting, focused on small details of painting during the first drawing session. For example, Participant #16 and #20 drew both outline of the coat and face details e.g. eyes, eyeglasses, and mouth etc., as shown in Fig. 5. Second, most of users followed the same drawing style as that of using pencil-and-paper, that is, (a) sketching an outline firstly and filling colors afterwards, e.g. the screenshot of the first sketching process of Participant #32 as shown in Fig. 5, right; (b) Rubbing out the mistakes and then drawing the correct one rather than drawing over the mistakes directly.

Fig. 5.
figure 5

First drawings from Participant #16, #20, #32.

Full size image

6 Conclusions

In this paper, we proposed a method of categorizing the continuous color palette interfaces into two classes: ‘1D-1D-1D’ interface and ‘1D-2D’. By focusing on ‘1D-2D’ interface with HSV color model, an experiment (with 41 participants) was designed and carried out to explore the influence of both combinations of the three axes of HSV color model and shape representing these combinations on users’ query-drawing performance. Specifically, we compared three groups of color selection interfaces: three bar-square color palettes, bar-wheel vs. ring-wheel, and bar-wheel vs. bar-square. The experimental results suggest that shape has more effects on users’ performance than color model axes combinations. In general, wheel-based interface leads to faster drawing than the square-based one; and the usability rating on ring-wheel is higher than that on bar-wheel. Most of participants preferred the H-SV square-bar colour selection interface.

One limitation of our study is conducting the copying-task only. However, the fact that our reference painting contains 7 different color blocks makes this preliminary study reliable. We will do more studies on other types of tasks, e.g. landscape drawing.