Image-guided color mapping for categorical data visualization

Appropriate color mapping for categorical data visualization can significantly facilitate the discovery of underlying data patterns and effectively bring out visual aesthetics. Some systems suggest predefined palettes for this task. However, a predefined color mapping is not always optimal, failing to consider users’ needs for customization. Given an input categorical data visualization and a reference image, we present an effective method to automatically generate a coloring that resembles the reference while allowing classes to be easily distinguished. We extract a color palette with high perceptual distance between the colors by sampling dominant and discriminable colors from the image’s color space. These colors are assigned to given classes by solving an integer quadratic program to optimize point distinctness of the given chart while preserving the color spatial relations in the source image. We show results on various coloring tasks, with a diverse set of new coloring appearances for the input data. We also compare our approach to state-of-the-art palettes in a controlled user study, which shows that our method achieves comparable performance in class discrimination, while being more similar to the source image. User feedback after using our system verifies its efficiency in automatically generating desirable colorings that meet the user’s expectations when choosing a reference.


Introduction
Color is one of the most important visual channels in data visualization. It is widely used to encode different data attributes and facilitate visual search. In particular, when presenting categorical data, a good color strategy can generate a visually attractive representation, clearly discriminate between different data categories, and help users effectively gain understanding and insight from the data [1,2]. Nonetheless, it is often difficult for even professional designers to design appropriate color mappings for data classes, especially when they have specific requirements, e.g., tuning color for a certain design atmosphere or personality. Designers usually go through a process of trial and error to find suitable colors for tasks.
Generating a suitable color mapping for particular data often has two steps: designing or choosing a good color palette, and assigning the selected colors to different classes. Existing strategies mainly focus on one of the two steps. For example, ColorBrewer [3] and Colorgorical [4] provide predefined color palettes that ensure discriminability and aesthetics. Color assignment methods automatically assign the colors within a given palette to the data classes in the visualization chart according to semantics [5], visual appeal [6], or class structure visibility [7]. As color appearance often depends on the relationship to surrounding regions [8], combining the two steps can potentially improve class structure visibility [9].
Informal interviews with designers indicate that they often use images as references for color mapping. From a reference image, designers can extract its prominent colors to generate a visually pleasing color palette, as well as use it to assign selected colors to the classes. If the distribution of main colors, i.e., their positions, proportions, and harmonious spatial arrangement, can be kept in the target, we assume that aesthetic perception of the reference image is thus transferred to the target.
Inspired by this, we propose to generate imageguided colorings of a given categorical visualization. A good coloring should allow easy perception of class structure and resemble the reference image in order to meet the user's expectations; see Fig. 1.
To achieve this goal, the selected colors need to be discriminable, easy to identify, and representative of the reference image. Furthermore, Wang et al. [7] suggested that the colors assigned to overlapping classes should have large perceptual differences. Differing from them, we additionally wish to retain the spatial distribution of colors of the source image, preserving both their adjacency relationships and the locations of dominant colors. Recently, Lu et al. [9] proposed to solve palette extraction and color assignment from the full RGB space in a unified optimization framework. However, an image's limited colors can cause the optimization to get stuck in a local optimum. Hence we solve palette extraction and color assignment in two stages; see Figs. 2(c) and 2(e).
To extract distinct colors, we look at palette selection as a farthest-point sampling problem [10] Fig. 1 A given categorical data visualization is colored to follow colors in different images. Each coloring is diverse with good visual separability of class structures, and is similar to its reference image. Class separation scores are 18,472, 15,382, and 17,149, respectively. in the color space of the image, considering linear separation [11], the minimal color CIEDE2000 distance between any two colors, and their frequency of occurrence in the image. We iteratively update the color samples after initialization.
We solve the color assignment problem by finding a consistent one-to-one correspondence between the colors and class labels. This is formulated as an integer quadratic program containing unary and binary terms. The unary term evaluates each classcolor assignment based on spatial position. The binary term assesses a pair of assignments based on the distance of two classes, the perceptual distance of the two colors, and their adjacency in the reference. An approximately optimal solution is found efficiently for this NP-hard problem.
In summary, the main contributions of this paper include: • a color mapping method to extract the color palette of an image and coherently apply it to a categorical data visualization to generate distinct and aesthetic color appearances, • a new way to extract a categorical color scheme representative of the source image, which can create a more distinct color palette than existing methods, and • an optimization method to produce the colorclass mapping, considering both spatial relations and color discriminablity, with comparable performance in class discrimination to prior works [7], yet being similar to the source image.

Related work
Existing related work on color design can be divided into three categories: color palette design, palette extraction from images, and color suggestion. Given an input categorical data visualization with M classes (a) and a reference image (b), we extract a distinct M -color palette from the image, indicated in RGB space by the green spheres (c). The radius of each sphere encodes the color's occurrences. After analyzing the color spatial distribution in the image (d) and the class distinctness of the given chart, we assign these colors to the classes (e). While making classes easy to discriminate, our mapping also follows colors in the source image. It keeps the major color adjacencies of the source, highlighted by the green box. It further retains the position of the light pink, a concentrated dominant color in the source.

Color palette design
Creating discriminable and aesthetically pleasing color palettes is a demanding task when visualizing categorical data. Guidelines for manually designing color palettes have been provided by several researchers. Colors in a palette design should meet the criterion of separability from each other [12] or overall compatibility [13]. Healey [11] proposed a method to choose effective colors allowing rapid and accurate identification of color targets. Recently, Bartram et al. [14] suggested that even a palette of a small set of colors could be manipulated to communicate emotions. These guidelines clarify the considerations for color design, but still, designing a palette from scratch is usually a cumbersome task for most users. There are many online toolkits for color palette selection. Adobe Color (https://color.adobe.com) and COLOURLovers (www.colourlovers.com) are two websites commonly used to select color themes. ColorBrewer, developed by Harrower and Brewer [3], provides a large number of predefined, recognizable palettes for thematic maps. Recently, Gramazio et al. [4] proposed Colorgorical, a tool that allows users to create customized color palettes by balancing discriminability and aesthetic preference.
Methods for rating, expanding, or optimizing a given color palette have also been proposed. O'Donovan et al. [15] studied color compatibility theories using large datasets and offered a learned model to evaluate a coloring combination of five colors. Later, Kita and Miyata [16] extended this model to rate arbitrary-sized color combinations considering human aesthetic preferences, and extended color palettes while retaining color harmony. Fang et al. [17] optimized a given color palette by maximizing the perceptual distances while meeting a set of userdefined constraints.
However, choosing from pre-generated color palettes is not always intuitive, as color appearance depends on the relationships to surrounding regions and the spatial arrangement of classes varies according to the data.

Palette extraction from images
A possible route for generating color palettes is to capture the prominent colors from an input image. When representing the image, the discriminability of the extracted colors is not significant. Lin and Hanrahan [18] created five-color themes from natural images based on a study considering the human perception. Their results closely match human-extracted themes. Poco et al. [19] developed a method to semi-automatically recover a color encoding from a bitmap visualization image by extracting the color and text information from its legend, enabling recoloring of the static image.
Recently, researchers in image processing looked at color palette extraction as a means to allow nonexperts to recolor an input image by editing the colors in the extracted palette. In 2015, Chang et al. [20] proposed a variant of the k-means algorithm to generate a suitable palette of a user-chosen size. Tan et al. [21] proposed a geometric method, identifying the color palette as the vertices of a simplified convex hull of image pixels in RGB-space. They also proposed a simple scheme for automatic palette size selection in following work [22]. The picked colors do not necessarily exist in the image. A different approach proposed by Aksoy et al. [23] automatically estimates a statistical color model from an image based on an energy formulation. It automatically decides the sizes of prominent colors and exclusively selects colors that exist in the image. Zhang et al. [24] presented an approach to edit colors of an image so as to preserve inherent color characteristics of the source by extracting and adjusting a compact color palette.
Some approaches deal with image collections. The group color theme extraction method of Nguyen et al. [25] derives the group color theme from input images' individual color palettes, for recoloring multiple images consistently. To capture color styles of a fine art collection, Phan et al. [26] introduced a method to order extracted color palettes, enabling palette prediction and interpolation. These approaches focus on image collections with a consistent color style or color theme.

Color suggestion
Many methods take the characteristics and content of the data into consideration when performing coloring.
Some methods combine colors with semantics, so that the color mapping matches typical colors for items defining the categorical data. Lin et al. [5] presented an algorithm to automatically select semantically-resonant colors from the 20 common colors suggested by Tableau to represent data. These resonant colors for each category were computed by finding relevant images using Google Image Search. Later Setlur and Stone [27] explored linguistic information to generate semantic coloring. However, not all classes shown in a visual chart have explicit semantics.
Some works aim to generate aesthetically pleasing color suggestions for uncolored pattern templates. Kim et al. [6] proposed a perceptually driven method for automatically coloring a 2D pattern template with a given color palette, guided by commonly accepted rules in color theories. They considered color harmony and luminance contrast of adjacent segments. Lin et al. [8] presented a probabilistic factor graph model for automatically generating color suggestions for an input pattern. The model was trained on example patterns made by experts. Due to the significant differences between patterns and visualization charts, their trained model does not work well on visualizations. Furthermore, the color variation between available example visualization charts is much smaller, making it hard to train the model.
Other works aim to generate color suggestions that make class structures more noticeable. Lee et al. [28] introduced the concept of class visibility to measure the utility of a color palette to present a categorical structure. Based on this concept, they proposed a method to optimize the luminance and saturation of assigned colors, to better display all class structures. Recently, Wang et al. [7] proposed a color assignment method with a given color palette for multi-category scatterplots, by modeling class separation considering spatial relationships, density, and point clusters. In following work, Lu et al. [9] introduced a data-aware color palette framework for both creating and assigning color palettes with maximal discriminability. While these systems are similar to our approach in considering color relationships as well as the spatial configuration of data, our method also takes the spatial distribution of colors in an image into consideration, making the colored visualization similar to the reference, to meet users' expectations.
Recently, Bohra and Gandhi [29] proposed a system for automatic colorization of graphic arts. Itautomatically selects a relevant reference image for any input pattern template from a built-in dataset, and then propagates the reference's colors to the input. The color propagation considers the composition of colors and the adjacency of colors, as we do. However, they do not encode color differences within the color propagation. Instead, our approach incorporates both the color spatial relations of the reference and class visibility of data for an optimal solution.

Palette extraction
If C is a distinct color palette, the minimal color distance of any two colors in the palette, denoted by should be large. Here, E(c a , c b ) represents the difference of two colors c a and c b in the CIEDE2000 color distance metric [30], which is commonly used for measuring the difference of colors within a color palette [4,9]. To form a distinct palette C of a given size M , the selected colors should be prominent, exist in the image, and approximate the image's color space. This is a farthest-point sampling problem in the color space and is hard to solve directly. To quickly generate an approximate optimal solution, we first reduce the sampling space, create a good initialization, and then iteratively update the color samples; see Fig. 3.
Though an image's distribution ( Fig. 2(b)) only covers a subset of the whole color space, it still has thousands of different colors. A minimum CIELAB interval, indicated by the noticeable difference function, is required to discriminate the colors of two graphical marks [31]. Thus, we perform color quantization by dividing the CIELAB color space into voxels in steps of 5 units along each axis. For each voxel, we record its frequency n a (the number of pixels whose color falls inside it), and compute its mean color c a . We then exclude colors that are too light (L > 85) or too dark (L < 20) [4]. To remove outliers of low frequency, we successively exclude the color with the lowest frequency until the frequency sum of all removed colors exceeds 3% of the input image. Figure 3(c) shows colors by spheres of different sizes according to frequency.
We use the initialization of Zhang et al. [24] to select M dominant colors. We successively choose the color with the highest frequency, and update the frequencies of remaining colors by different factors to penalize colors similar to it: where τ represents the iteration index, n 0 a = n a , and σ is set to 80 by default. We repeat this until M colors have been selected. However, some chosen colors may still be too similar to distinguish; see for example the light and dark gray marked by red dots in the leftmost palette in Fig. 3

(d).
We gradually update the palette C to increase d C by farthest-point sampling [10]. In each iteration, we successively move each color in the current palette to a new position that maximizes the minimum distance and its linear separation to the rest of the colors in the palette, in ascending order of frequency. To replace a color, we first select the k = 3 farthest colors from the rest of the palette as candidates using CIEDE2000 color distance. Then we evaluate each candidate c a by considering its linear separation s a , color frequency n a , and minimal distance: The score of c a is a weighted combination of the three terms: We approximate s a in the u-v plane of the CIELUV color space as the minimal distance to the convex hull of the remaining colors of the palette [11]. Weights w s and w n balance the three terms. After analyzing these terms' ranges, we set w s equal to 0.15 and w n equal to the inverse of 0.03% of the image's pixel number by trial and error. The minimal distance d C increases in each iteration. This scheme usually converges after several iterations; we stop after 20. Figure 3 shows an example. The reference image is then approximated by the color palette C. A pixel is either deleted or represented by the most similar color in C if their CIEDE2000 color distance is smaller than a threshold (15 by default); see Fig. 2(d).

The given visualization contains
The points with label l i are denoted by Q i . The aim of color assignment is to find a bijective mapping between the colors in palette C and the data labels, represented by a set of M pairs (i, a), where l i ∈ L and c a ∈ C.
We formulate color assignment as an integer quadratic program, taking into consideration both unary and binary terms: for a candidate assignment (i, a), we measure how well label l i matches color c a ; we also evaluate how compatible each pair of assignments (i, a) and (j, b) is. From a list of candidates-in our case, all possible assignmentswe compute an optimal subset to form a one-toone correspondence mapping that maximizes the quadratic score function; see Figs. 4(a) and 4(b).

Pairwise term
Classes with close-by points should be assigned with colors having large differences and adjacent in the reference if possible. Based on this intuition, we define the compatibility score of a pair of assignments (i, a) and (j, b) using three terms: while D(l i , l j ) represents the closeness of two point sets, E(c a , c b ) is the color difference in the CIEDE2000 metric, and A(c a , c b ) measures the adjacency of two colors in the reference image. Let q i denote a point in Q i . We define D(Q i , Q j ) as the summed closeness between q i and Q j : Fig. 4 Color assignment. (a) From a list of candidates, we aim to compute an optimal bijective mapping. Each symbol represents a class. (b) For clarity, the point set is repeated below; the virtual boundary of each class is shown by dashed lines. We measure the quality of a class-color assignment (the unary term) by comparing the locations of the class and the color; the bar chart on the top left corner shows the quality scores of two assignments. We also measure the quality of a pair of assignments (the binary term), considering the closeness of the two classes, the difference of their colors, and the adjacency of the two colors in the image; the bar chart on the top right shows the quality scores of two pairs. (c) The result using both terms is most similar to the reference.
This is inspired by point distinctness which measures the distinctness of a data point from its neighbours [7].
Here, |N (i )| represents the number of points in q i 's nearest neighbors N (i ). Within them, the points with label l j (the intersection of Q j and N (i )) is denoted Ω i . We define the nearest neighbors N (i ) using the α-shape graph as suggested by Lu et al. [9], which selects the closest neighbors of a given point within a certain radius. g( q i − q j ) is a distancebased function that varies inversely with Euclidean distance between point q i and point q j ; we use The spatial adjacency of color c a and c b is computed based on the closeness score of their associated pixel sets. We assign large values to close-by colors in the image and small ones to faraway colors: where P a and |P a | are the pixel group represented by color c a and its number of pixels, respectively. We measure how much a color surrounds other colors, and divide the value by its number of pixels. As A(c a , c b ) is not symmetric, we define A(c a , c b ) by linearly mapping max(Ã,Ã T ) to the range of [1,2]. We choose a narrow range for color adjacency as it is less important than point distinctness. When they conflict, the latter should dominate. Figure 5 shows an example of assigning colors to a six-class pie chart when only considering the binary term. The closeness score of any two pie sectors D(Q i , Q j ) is either set to 1 if they are adjacent or 0 otherwise. When the color difference score is removed from Eq. (3), some adjacent colors with low discriminability are given to close-by sectors, which diminishes the chart's distinctness: see Fig. 5(b).

Unary term
The unary term measures the quality of a class-color assignment. We give a high score to an assignment (i, a) if the locations of points Q i in the visualization are similar to the locations of color c a in the reference image. We align the image to the visualization by their boundaries using scaling and translation, and then measure the distance between the point set Q i and the pixel set P a : whered H (Q i , P a ) is the one-sided Hausdorff distance from Q i to P a , and f (d) is inversely related to distance. The one-sided Hausdorff distance is preferable, as in our case we desire partial matching of the point set to the pixel set. We use f (d) = exp(−8d 2 ) so that f (d) declines rapidly from 1 to 0.1 as d grows from 0 to 0.5. Note that image size is normalized to [0, 1]. To speed up the computation ofd H , we apply the Euclidean distance transform to binary images, each of which represents the pixels of a color. Figure 4(b) shows an example on the left.

Solution
Given a candidate set F of |F| assignments, a chosen subset is represented by an indicator vector x ∈ {0, 1} n where chosen elements have a flag value of 1. The optimal subset is the solution of the objective function as in Eq. (7): x * = argmax x T M x such that Bx = 1 (7) where Bx = 1 represents the constraint that x is a bijective mapping. M is an |F| × n matrix. The k-th and m-th assignments are denoted by (i, a) and (j, b), and we have where w u is a weight to balance the binary and unary terms. We use w u = 0.2 max k,m,k =m M k,m in our setting. Though this is an NP-hard problem, an approximate solution can be found efficiently. We use the integer projected fixed point (IPFP) method [32] to solve it. Algorithm 1 shows the key steps for color assignment. Figure 4(c) shows how the result considering both unary and binary terms is most similar to the reference. Note that if we ignore the color spatial information in the reference image, i.e., exclude the unary term Algorithm 1 Color assignment Input: A reference image, a color palette C, and a visualization containing a point set with labels L. Output: A bijective mapping between C and L, represented by a set of pairs (i, a) with L i ∈ L and ca ∈ C. Initialization: Generate the set F of all possible assignments. for all assignment (i, c) ∈ F do Compute its quality S ((i, a)) by Eq. (6); for all assignment (j, b) ∈ F\(i, a) do Compute the compatibility score of two assignments S ((i, a), (j, b)) by Eq. (3). end for end for Solve for the optimal indicator vector x * using Eqs. (7) and (8).
Return the chosen subset of F whose flags from x * = 1. and the color adjacency score (A(c a , c b )) in the binary term, the objective function is the same as that of Wang et al. [7]. Instead of using a genetic algorithm, we solve the problem by graph matching, which can quickly return an approximately optimal solution. By respecting both the spatial positions of colors and their spatial relationships, we can transfer colors' spatial information to the visualization.

Results and applications
Here we first describe implementation details. After that, we show coloring results for different types of charts. Lastly, we present two extensions in which user-specified constraints are added.

Implementation details
Although the core algorithm requires a point set with class information as input, we can easily apply the approach to other categorical visualizations, such as bar charts and pie charts, by sampling points inside the region of each class. We use Poisson disk sampling to generate samples from a blue noise distribution [33]. We sample about 300 points in each example; see the leftmost of Fig. 6. The randomness in generated samples may affect the class closeness score D(Q i , Q j ), so we sample 10 times and use the mean closeness score.
We implemented the algorithm using MATLAB. All experiments were run on a computer with an Intel Xeon Gold 5118 processor with 64 GB memory, and took less than 2 s. We designed an interactive interface using JavaScript to invoke the core algorithm; see Fig. 7. The interface works with SVG files as inputs and outputs. Users can select or upload an input visualization, and then upload or choose an image to set the visualization's color scheme. For more information, please see the video in the Electronic Supplementary Material (ESM).

Results
The class separation score for a coloring is (c a , c b ). This measurement is also dependent on the points' distributions Q, so it is only meaningful to compare these scores for different color mappings of the same data. The spatial consistency between colors in reference images and those in the coloring is measured by two components. The position score of a color assignment  is defined by the unary term: ((i, a)). The adjacency score is defined by the binary term:  All the input scatterplots except in Fig. 8 were generated from simulated categorical data. We randomly set each cluster's center, and then sampled points for each cluster using a Gaussian distribution, following Wang et al. [7]. Figure 9 shows examples of two scatterplots of 6 and 8 classes colored according to five different images, together with their extracted palettes. Note that an image's 6-color palette is not a subset of its 8-color palette, for increased distinctness. We can see that classes are easy to spot even though all points are represented by quite small dots. Although the  two scatterplots have different class spatial structures, both still look similar to the corresponding reference images. For example, in the first column of Fig. 9, the yellow color is placed at the bottom right corner in both scatterplots, the same as its position in the reference; the adjacency of light and dark green colors in the image is also present in both scatterplots. Note the best among the five reference images in terms of class separation changes when the input alters.
We generated a scatterplot by projecting the MNIST dataset [34], which contains thousands of handwritten images of 10 digits, to the 2D plane using t-SNE [35]. Figure 8 shows the color mappings of this 10-class scatterplot using colorful images as guidance.
Our method can also suggest color mappings for webpage layouts and infographics. In webpage design, designers often use an image as the header and use its color scheme for the webpage. Figure 10 shows six suggested color mappings of a webpage layout according to different header images. The input is a web layout pattern specifying which connected components can be colored and which components must map to the same color.
In infographic design, images related to the infographic's topic are often used as references for the color setting. Figure 11 shows four suggested color mappings of an infographic according to different background images, generated using the interface in Fig. 7. Our method identifies all colors used in the input infographic and asks the users to mark the colors to replace. Then new color mappings are automatically generated according to the reference images. However, our method is limited to infographics without gradient colors. Although the first source image only has two dominant colors Fig. 10 Color mappings of a web layout according to six images. In each case, part of the corresponding image is the page header. Fig. 11 Color mappings of an infographic according to four images. In each case, the corresponding image in the center is also the background. Note that the light gray color in the second image is not selected because of its similarity to the infographic's background. The infographic is from Freepik.
(red-brown and green), our method still extracts a distinct 4-color palette, and the two different versions of red-brown are adjacent in the infographic, just as in the source. The presented results are able to nicely combine the given image with the webpage or infographic in a coherent and harmonious way.

Extensions
It is easy to extend the method to optimize the color mapping to any user-specified background color. Without modifying other part of the method, we simply let the minimal color distance of a palette C take into account the background color c 0 : The colorization result is thus tailored according to c 0 . Three colorization results with black, white, and gray backgrounds are presented in Fig. 12. Note that the source image is not very colorful, but we are still able to extract colors distinct from each other and from the background in each case. It is also easy to add user-defined constraints to the color assignment step. We only need to change the candidate set according to the given constraints. Figure 13 shows three color mappings of a treemap, in which only the 2 most dominant colors of each reference image are allowed to be assigned to the largest rectangle.

Evaluation
We first evaluate our palette extraction step and  compare it with existing methods. Then we evaluate our class assignment for visualization tasks by a controlled user study. A limitation is shown later.

Palette extraction evaluation
To evaluate the extracted palettes, we visually compare results of our approach with those from previous palette extraction methods [15,18,20,24,36]: see Fig. 14. The three test images have commonly been used in previous works for comparison, and palettes of other methods are taken from Zhang et al. [24]. Our model tends to extract palettes with large distances between colors, which are often easyto-spot and vivid. Our results are most similar to  [20] (e), Zhang et al. [24], and our method (g). Our results provide a more discriminable palette. For example, in the middle photo, our method chooses a brighter blue, which is more separable from the dark blue. Testing images and palettes for other methods are taken from Zhang et al. [24]. those of Zhang et al. [24] as we use their initialization to select seed colors. However, our method produces palettes with lower similarities between different color components.
We next consider palette extraction for varying sizes of palette. The minimal distance of all color pairs in the extracted palette (d C ) decreases quite slowly when the number of colors increases; see the solid gray line for the left image and the dashed black one for the right in Fig. 16. Although the values of d C depend on the given images, the changing trends are similar. Thus, we assume we can generate over 12 discriminable colors if they exist in the image.

Quantitative comparison
We quantitatively compare our method with two other palette extraction methods that automatically decide an optimal number of prominent colors in an image. One is a color extraction Web API provided by TinEye based on k-means clustering [37], chosen for its popularity. The other was proposed by Aksoy et al. [23] and exclusively selects colors that exist in the image. We chose this state-of-the-art method as it provides 100 testing images, as well as the extracted palettes. According to the two methods, 17 out of the 100 images include fewer than six distinct colors. We removed these and compared six-color palettes extracted from the remaining images. When the palettes extracted by other methods have over six colors, we selected six colors by greedily iteratively removing the color that most increases the minimal color distance of the remaining colors. Fig. 15 Comparison with two common color palette extraction methods, TinEye [37] and Aksoy et al. [23]. Sometimes the number of colors in a palette from other methods is larger than six. When this happens, we select 6 colors by greedily removing the color that can best increase the minimal color distance of the remaining colors. We evaluate each palette using two measures: its minimal color distance d C , and the average distortion between the quantized image and the original. Four example images and their corresponding palettes are shown. The minimal color distances of our results ( green) are larger than for the others. Testing images are taken from Aksoy et al. [23]. We evaluated each palette by two measures: its minimal color distance and the average distortion (MSE) between the quantized image and the original. Figure 15 shows these values for four examples. It can be seen that for most images, our palettes have similar average distortion but larger minimal color distances than others. Though we only compare sixcolor palettes, our method is more advantageous when the number of colors in the extracted palette increases, since others do not explicitly penalize similar colors. Thus, as the number of required colors increases, they are more likely to choose similar colors than our method.
We conducted a paired sample t-test to compare the minimal color distance scores. Results found a significant difference in the distance score comparing our method (M = 23.06, SD = 7.46) to TinEye (M = 18.84, SD = 4.32, t(72) = 7.24, p < 0.0001), and Aksoy's method (M = 16.33, SD = 7.25, t(72) = 9.50, p < 0.0001). This indicates that our method can create a more distinct color palette than these others.

Background
In general, it is difficult to define any ground truth for a color mapping task. The choice often depends on users' preference for the reference image, the aesthetic preferences of the user, and more. Thus, in our user evaluation, we do not examine subjective preferences, but rather focus on two main issues: • Discrimination. In terms of discrimination, are the color mappings generated by our method comparable to state-of-the-art methods? To answer this, we performed several discrimination tasks in which we examined participants' perception of the number of classes in a given scatterplot. We hypothesized that the our method will be comparable in these discrimination tasks. • Similarity. Is the color mapping generated by our method based on an image indeed more similar to that image than color mappings generated by non-spatial methods? To answer this, we performed several similarity tasks, in which we asked participants to judge which of two scatterplots is more similar to a source image in terms of color configuration. A neutral choice option was also provided. We hypothesized that our method will be perceived as more similar to the source image in the similarity tasks. As far as we know, no existing method explicitly uses the spatial relationship between colors in an image for color mapping of a data visualization chart. Therefore we chose the recent color assignment method for optimizing perception of class separability [7] (referred to as SepOpt) for comparison. To ensure a fair comparison, the results of SepOpt were also solved by graph matching after removing the color spatial information from the formulation, instead of the original genetic algorithm. As our core algorithm deals with point sets, we performed the user studies on scatterplots.

Participants
We recruited 20 participants, out of which 14 were male and 6 were female, 18 were students and 2 were designers. Their ages ranged from 22 to 36 years old. None of them reported color vision deficiency. We paid each participant $6 for the two tasks. The entire study took about 20-30 min. Each participant first provided demographic information, and then performed the discrimination tasks followed by the similarity tasks.

Discrimination task
In each trial, each participant was first shown a gray dot in the middle of the screen to calibrate the eye gaze, followed by a scatterplot. They were instructed to press a key when they knew the number of classes. When the key was pressed, the scatterplot disappeared. Then the participant chose the number of classes from five options (5)(6)(7)(8)(9). We recorded the time taken for each trial and counted the difference between the actual class number and the participant response as the error.
We used five multi-class scatterplots (1 of 6 classes, two of 7 classes, and 2 of 8 classes), generating them using the SepOpt method. First, a synthetic scatterplot of 307 points with 6 classes was created. Each class was placed according to a Gaussian distribution with strong overlaps between classes. Then we created other scatterplots by randomly selecting some points to form new classes. As the same points were used in all scatterplots, we rotated each by a randomized angle for variety.
We selected the top 20 from images used for Fig. 15 according to the minimal color distances d C of their palettes. To avoid unwanted learning effects, we allowed each participant to see each color palette only once, using a between-subject layout. After shuffling the chosen images, we subdivided them into five equalsized subsets, each assigned to a scatterplot. Thus there were two participant groups; every participant was assigned to either group G1 or G2 and completed 20 trials. For the first 10 images G1 saw SepOpt results and G2 saw our results. For the next 10 images, the groups were swapped. The corresponding colorized scatterplots were shown at half of their original resolution of 1048 × 1048 pixels with a small point size of 4 pixels.
The results are summarized in Fig. 17. The reported error is the sum of all errors made by a participant on a given method. Figure 17 shows the performance of participants on the two methods are similar in terms of both error and time. An independent samples t-test was conducted to compare error based on our method (M = 2.95, SD = 2.84) and SepOpt (M = 2.80, SD = 2.31). We observed  Figure 18(c) shows that the class separation of our results is slightly lower, but it does not affect the actual performance in counting tasks. A possible reason is that the small change in class separation score is not easy to spot when the value is high.

Similarity tasks
This task checked whether the color mappings generated by our method are perceived as more similar to the corresponding source images. We used six palettes extracted from six images, three of which are shown in Fig. 1. We used four different scatterplots with 6 or 8 classes, two of which were from Fig. 9 and the others were from SepOpt. Hence, the total number of trials was 4 (scatterplots) × 6 (palettes) = 24. In each trial, two scatterplots generated by different methods using the same image were shown in pairs. To avoid bias, we randomised the locations for the two methods. An example is shown in Fig. 19(top).
We asked participants to compare them with the reference image on a 5-point Likert scale: The left is much more similar, The left is slightly more similar, No difference, The right is slightly more similar, and The right is much more similar. The  Top: example of similarity tasks. Participants were instructed to compare the similarity of two colorings related to a reference image. In this example, our result is shown on the left. Bottom: result summarised as a bar. Most participants thought our results resemble the source images better. colored scatterplots were shown at a resolution of 1048 × 1048 pixels, with a larger point size of 6 pixels, as similarity tasks should be insensitive to the point size.
The participant's choices are summarized in Fig. 19(bottom). Overall, in 64% of the trials, participants thought our results were more similar to the source image, while only in 17% of the trials did participants think SepOpt was more similar. We conclude that our method manages to bring the spatial relationship of the source image to the target visualization. The scores of the spatial consistency between colors in reference images and in the scatterplots, shown in Fig. 20, also indicate our color mappings can better preserve the color spatial relations.

Limitations
Our method is built on an input reference image, and can thus fail to produce a good color mapping when the source image is unsuitable: more specifically, when it lacks sufficient distinct colors. Figure 21(left) shows a case in which some classes are difficult to discriminate. Another color mapping which makes  classes distinguishable and has a much larger class separation score is shown on the right. We could easily detect such failures by examining the minimal color distance d C of extracted palettes and the class separation scores. When d C is smaller than a threshold such as 10, or the class separation score is significantly lower than the scores of other color mappings, we could warn users and suggest changing the reference.
Furthermore, some image features, such as saliency, foreground, and background, are ignored during color palette extraction. Our algorithm also has several parameters that need to be set: weights for linear separation, color frequency, and minimal distance in the palette extraction, and weights to balance the binary and unary terms. Changing these weights may slightly change the results. However, in all presented experiments and results, we used constant values.
Finally, we evaluated the similarity between colored visualizations and their corresponding references only in terms of color spatial consistency.

Conclusions
We have presented an automatic image-guided approach to produce color mappings for categorical visualizations that are visually similar to source images while allowing visual discrimination of classes.
The key idea is to extract a discriminable and representative color palette from the reference image using farthest-point sampling, and assign its colors to classes in the visualization considering both class discrimination and spatial distribution of colors in the image. We show various results, evaluate the approach numerically, and perform a controlled user study to examine the perceptual effect of the method.
Some image features, such as the saliency, foreground, and background, are ignored during color palette extraction. There is a lot left for further exploration. Now we exploit images to give graphical marks suitable colors. In the future, we could explore image information to set other channels of graphical marks, such as gradients, non-linear color triads [38], outline colors, and even mark shapes. In addition, we would also like to extend our method to support the coloring of other visualizations such as maps.
[ She is currently a senior member of IEEE/ACM/CSIG, a distinguished member of CCF, and is on the editorial boards of ACM TOG and IEEE TVCG.
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made.
The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.
Other papers from this open access journal are available free of charge from http://www.springer.com/journal/41095. To submit a manuscript, please go to https://www. editorialmanager.com/cvmj.