1 Introduction

Ancient Chinese inscriptions flourished during the Tang Dynasty and were used to record history and spread culture. Through the inscriptions, we can not only read the contents of the inscriptions but also appreciate the handwriting of the calligrapher and the ornamentation of the inscriptions. They provide original written and pictorial records for subsequent generations to study ancient Chinese culture [1]. Meanwhile, the style of Chinese calligraphy has evolved from the its birth to the Six Dynasties Period, and it can be classified into categories such as regular scripts, semi-cursive scripts, and cursive scripts [2]. However, as calligraphic works have eroded over time, fewer works have been preserved. Therefore, the inscriptions of the Tang Dynasty preserve an extremely rich historical material of Chinese calligraphy.

Wang Xizhi’s calligraphy occupies an elevated position in the history of Chinese calligraphy [3]. His work “Preface to the Poems Composed at the Orchid Pavilion” is also known as “the world’s No. 1 semi-cursive script”. However, in the study of the history of Chinese calligraphy, there is not much literature about how the Tang people learned calligraphy from Wang Xizhi. Thus, we will investigate the evolution of Wang Xizhi’s calligraphic style by comparing the Tang inscriptions with his authentic works or works that collect inscriptions of Wang Xizhi. In this work, we attempt to explore the evolutionary path of Wang Xizhi’s calligraphy and hope to help the public learn about the calligraphic style of Wang Xizhi.

However, there are three challenges in the extraction and analysis of the calligraphy feature. Because of the complex structure of Chinese characters and the existence of a large number of similar characters [4], the first challenge is the difficulty in identifying and processing calligraphic characters. Second, feature extraction is a key problem in the research of calligraphy style analysis. The work [5] in feature extraction generally consists of two categories: stroke extraction approaches and global feature extraction approaches. However, recent works lack a workflow for local-to-whole feature extraction of calligraphy to support downstream applications. Therefore, the second challenge is how to extract the calligraphy features from the local details to the whole picture. Additionally, due to the specialization of the art of inscriptions and calligraphy, Chinese calligraphy has not been popularized. Thus, how to make the professional knowledge of calligraphy available to the public is the third challenge in this work.

Current work on Chinese calligraphy involves many aspects, such as the transmission and promotion of the calligraphy. Some works [6–8] study ancient calligraphic works, including the conservation of works, digital display of works and the study of calligraphic styles. Others [9–12] have studied the transmission and promotion of the art of calligraphy, including the generation of calligraphic characters and systems of practising digital calligraphy. The rapid development of image processing technology and neural networks provides solutions for such applications. Recently, some works [13–15] have used visualization techniques to spread Chinese culture, which can help users understand the culture intuitively and provide an engaging and interactive approach. Thus, we want to tell a data story from calligraphic characters to calligraphic culture in an innovative and experiential way and bring the calligraphy into the public’s eye through data visualization.

To address the above mentioned challenges, we present a visual analysis system that allows the general public to understand the evolutionary path of the calligraphic style of Wang Xizhi from multiple dimensions. We first complete the segmentation of text by pre-processing images of Chinese inscriptions. Then, we extract multi-dimensional features such as skeleton and thickness of strokes from Chinese characters. Furthermore, we calculate the similarity between works by features to determine the evolutionary path of calligraphic styles. As shown in Fig. 1, this system supports users in intuitively understanding the characteristics of calligraphic styles starting with seven dimensions and guides users step by step through the overview and seven detail views to explore the evolutionary path of Wang Xizhi’s calligraphic style in inscriptions of the Tang Dynasty. The main contributions of this paper are summarized as follows:

Figure 1
figure 1

The visual analysis system consists of six components: (a) Geographical distribution of inscription works; (b) Users click on the graph or timeline to view the detailed information and original images of inscriptions; (c) Temporal distribution of inscription works for the exploration of the rise and fall of dynasties in relation between the works; (d) Word cloud display of words with high frequency in different calligraphy works; (e) Calligraphic evolution of the same characters displayed in the rice grid in different works according to the timeline based on the calligraphic strokes of the conjoined and thickness features; (f) Calculation of similarity between different works based on four weighted characteristics

Full size image

(1) A workflow for the processing of calligraphic images and the extraction of calligraphy features. It provides a paradigm of extracting calligraphy features for other calligraphic analysis tasks.

(2) A visual analysis system that provides a set of intuitive visual designs to support interactive exploratory analysis of ancient Chinese calligraphy in Tang Dynasty inscriptions.

(3) Two case studies provide profound insights into Wang Xizhi’s evolutionary path of calligraphy through multiple dimensions.

2 Related work

This section reviews analytical techniques that are most relevant to our work regarding two main problems, namely, image processing of inscriptions and digital literary visualization.

2.1 Image processing of inscriptions

In recent years, digital image processing technology has become increasingly mature, and the method of preserving the images of inscriptions using digitization has been widely used. According to the relevant work [16–20], the general processing process is as follows. First, electronic scanning technology and camera technology are used to obtain digitized inscription images. After the image pre-processing operation, the image is subjected to text segmentation, character recognition and feature extraction.

Similar to other image processing tasks [21–23], it is necessary to perform pre-processing operations such as binarization and noise elimination on the inscription images. Wang et al. [24] applied the dual-binarization method to calligraphy works and used the anisotropic diffusion technique to blur the calligraphy image and reduce noise, but the method does not remove the ant-like block noise and the background patterns in the calligraphy works well. Therefore, Ge et al. [25] proposed using \(L_{0}\) gradient minimization to smooth the noise and repair the Chinese character strokes by guided filtering, and finally the noise of the ant-like blocks was well removed by using the eight-connected region technique.

In terms of Chinese character segmentation, the prevailing methods can be roughly divided into the following three categories: segmentation based on statistical methods [26], stroke and structure-based methods [27] and horizontal and vertical projection methods [28]. Statistical methods are commonly used for occasions where the width of characters is relatively uniform and the methods have worse effects on the cutting of characters with overlap and adhesion. Stroke and structure-based methods used the relationship between strokes and the structure of Chinese characters to discover the intrinsic features between texts, but the extraction of strokes is more complicated and demanding. Horizontal and vertical projection methods are suitable for texts with neat distributions of character rows and columns, but for Chinese characters, it was easy to cause incorrect segmentations of Chinese character components and punctuators. In addition to using a single segmentation algorithm, combining multiple complementary algorithms using a multi-step tessellation method is also an effective way to solve the image segmentation problem. Sahare et al. [29] first used vertical projection contours on the image to find the main segmentation path and then used distance-related criteria to obtain the fine segmentation path.

Regarding character recognition, Chinese characters are more complex than other scripts, and the current mainstream character recognition methods, such as convolutional recurrent neural network(CRNN) [30], recurrent neural network(RNN) [31], are difficult to be applied well to Chinese character recognition. Therefore, this paper adopts a tool and manual correction approach, using the Chinese OCR tool CnOCRFootnote 1 for the initial recognition of characters and then performing manual correction to re-label the wrong images.

Feature extraction is the key to calligraphic style recognition, and the research mainly includes calligraphic contour extraction, calligraphic character skeleton extraction, and calligraphic character stroke extraction. Zhang et al. [32] optimized and improved the extraction effects based on the rotation invariant refinement algorithm proposed by Liu et al. [33], reducing the skeleton to the width of a single pixel and using the maximum circle method to merge the bifurcation points. Zhang et al. [34] extracted the horizontal strokes, the vertical strokes and the characters to quantify and calculate the calligraphic style.

In this paper, the pre-processing work is similar to others, and the image pre-processing is completed by image graying, binarization processing and median filtering. Unlike the previous work, we propose a recursive cutting algorithm using the adaptive adjustment of the structure threshold based on the previous work of horizontal-vertical projection segmentation, which can further improve the correct segmentation rate of Chinese characters. Furthermore, we perform word aggregation on the recognition results of each inscription work and extract features from six dimensions, involving outer and inner contour features, thickness of strokes, skeleton features, rice grid area features and conjoined stroke features. Following the above characteristics, we calculate the similarity among inscriptions.

2.2 Digital literary visualization

Currently, the cross-border integration of visualization and digital literary has sparked widespread attention in the field of social humanities and history [35–37]. Through cutting-edge information technology and digital design tools, the value of humanities and social sciences is deeply explored and creatively presented in the novel form of digital humanities works, which enriches the research means of digital humanities, and new digital humanities research projects have been implemented in multiple disciplines such as history, archaeology, literature, linguistics and sociology.

For example, China Biographical Database Project (CBDB)Footnote 2, developed under the auspices of Harvard University, contains biographical data of hundreds of thousands of people. The CBDB centers on Chinese historical figures, digitizes important tool book entries and biographical data, extracts characters and social relationships among them and provides a large amount of data for historical visualization and analysis and visualization of social network relationships. Song et al. [13] introduced the data narrative in visualization analysis of Tibetan calligraphy, explored the data of writing movement and textual dimensions in Tibetan calligraphy and interpreted Tibetan culture from a novel perspective. Taking Dunhuang murals as an example, Luo et al. [14] constructed a semantic enrichment framework to aggregate and present image semantic information. The visualization platform based on the point of interest (POI) perspective is used to help users better understand cultural heritage images.

Combined with previous work, we start by digitizing images of inscriptions, deciphering semantic information in images of cultural heritage, visualizing the presentation of inscription works in terms of spatial and temporal information, stylistic similarity, stroke characteristics and font structure, and exploring the evolutionary path of Wang Xizhi’s calligraphic style in Tang Dynasty inscriptions, contributing to both visual analysis and calligraphy research.

3 Design rationale

This section presents the definition of the terms and the design rationale of this work, including the requirement analysis and design goals. We first introduce the terms about the calligraphy domain and then analyze the requirements summarized from domain issues and user surveys. Finally, we derive design goals that can fulfil the requirements.

3.1 Term definition

Before introducing the design rationale of this work, we first briefly introduce the terms about calligraphy domain to help the reader better understand this paper. As shown in Fig. 2(b), the Six Dynasties Period (222–589) [38] is a collective term for six Han-ruled Chinese dynasties that existed from the early 3rd century AD to the late 6th century AD. The style of Chinese calligraphy has evolved from its birth to the Six Dynasties Period, and it can be classified into categories such as regular scripts and, semi-cursive scripts [39]. During the Tang Dynasty (618–907), inscriptions flourished and Wang Xizhi [40] was held in high esteem by Emperor Taizong of Tang [41], leading to the appearance of many inscriptions in Wang Xizhi’s calligraphic works.

Figure 2
figure 2

Example diagram for introducing the terms about the calligraphy domain. (a) shows the comparison between the semi-cursive script and the regular script, and demonstrates examples of the work, character, radical and rice grid in the calligraphy. (b) presents the timeline of Chinese history to give readers some insight into the Six Dynasties and the Tang Dynasty. Note that the timeline cannot show the exact beginning year of Han Dynasty, Song Dynasty, Yuan Dynasty and Qing Dynasty, because these dynasties were established when their previous dynasty did not end

Full size image

This work uses images of inscriptions as datasets for data extraction and analysis, and the image of the inscription shown in Fig. 2(a) is called a piece of work. Each work consists of multiple Chinese characters. Radical is a semantic and phonetic component of Chinese characters, which plays an important role in modelling character semantics. For characters with the same radical, the characters generally have similar semantic meaning and grammatical usage [42]. The rice grid is a square grid used to help beginners determine how to write the characters, i.e., for the structure of the characters, when they learn calligraphy [43].

3.2 Requirement analysis

Our system is mainly designed for the general public, who may have no experience in calligraphy or have some interest and knowledge of calligraphy. Therefore, we conducted two weeks of interviews with five calligraphy practitioners with over five years of experience and two experts (one studied calligraphy and the other studied inscription culture). We analyzed the calligraphy features in the dataset to ensure that the extracted data were correct in terms of domain knowledge. Based on the above expertise, we conducted a questionnaire survey and communicated with 15 target users (five of whom were interested in or had experience in calligraphy and ten of whom had no experience in calligraphy). We identified the following requirement around the questions “What would users like to know about calligraphy?” and “How do users want to appreciate calligraphy?”

R1 How to explore the distribution of calligraphy works in the spatial-temporal dimensions? The rise and fall of times and changes in the current situation affect the development of calligraphy, and the geographical distribution reflects the origins of different calligraphic styles. Thus, users are inclined to gain a macro preliminary impression of calligraphy and analyze the correlations from the perspective of time and region.

R2 How to distinguish and compare the calligraphic styles from different works? Users can subjectively distinguish different styles of calligraphy by comparing the calligraphic handwriting of different works but lack expert technical guidance. Therefore, we extracted quantitative features that can represent calligraphic styles to help users understand the metrics of calligraphy and explore the similarity between different works.

R3 How to discover the evolutionary path of calligraphy style? Users are able to enjoy calligraphy works along the timeline, but there is a problem that the evolution of calligraphy style does not strictly follow the linear change in the timeline. Thus, we need to provide a way for users to clearly understand the path by comparing the same Chinese characters in different works.

R4 How to understand the characteristics of calligraphy in a more tangible way? Users may lack knowledge of calligraphy, and if we directly introduce the calligraphy features with the text, users’ interest will be reduced. Therefore, users prefer to map calligraphy features to specific things with which they are familiar and which may help them better appreciate calligraphy. For example, users tend to compare different styles of calligraphy by the thickness of strokes.

3.3 Design goals

Based on the above requirements, we derive several design goals as follows:

G1 Visualizing the spatial and temporal distribution of calligraphic works. Users tend to understand the relationship between the development of calligraphic works and their temporal and geographic distribution (R1). Thus, we map the rise and fall of the Tang Dynasty and the distribution of works by chronology. We aim to guide users to explore the relationship between the development of calligraphy and the rise and fall of the Tang Dynasty. We visualize the spatial distribution of works to discover the relationship between geography and calligraphic works.

G2 Exploring the similarity between works from different levels of granularity. Through literature surveys and interviews with experts, we define the calligraphic characteristics and use image processing techniques to extract calligraphy features from the granularity of words and works. To allow users to compare the calligraphic styles of different works (R2), we calculate and visualize the similarity between the works through the above features.

G3 Discovering the evolutionary path of Wang Xizhi’s calligraphic style. The evolutionary path is found through the skeleton and layout features of the same character in different works. Thus, we design the visualization method in chronological order (R3) with the help of the rice grids. Furthermore, the evolution of Wang Xizhi’s calligraphic style is explored by comparing different works with the calligraphy work of Wang Xizhi “Preface to the Poems Composed at the Orchid Pavilion”.

G4 Interactively understanding the calligraphy features using a metaphorical approach. For users to understand the characteristics in calligraphy more graphically (R4), we analogize the momentum of mountains with the rhythm of calligraphy. This metaphor compares each character to a mountain, using the height of the mountain to represent the thickness of the calligraphic strokes. It combines the strokes in sequence to the high and low peaks of a mountain, which reflects the sense of continuous and everlasting calligraphy.

4 Data processing

In this section, we start with a brief introduction of the dataset (see Sect. 4.1). Subsequently, we provide a detailed illustration of the data processing pipeline (see Fig. 3). We first pre-process the image to eliminate image noise (see Sect. 4.2), and then use image processing techniques to extract the calligraphic characteristics for subsequent analysis (see Sect. 4.3).

Figure 3
figure 3

The workflow of the system is from the inscription image input (a) to the calligraphy visualization output (c). (b) We pre-process the dataset and extract six Chinese characters for subsequent visualization, and then propose one overview and seven detailed views (c) to help users understand the inscription and appreciate the calligraphy

Full size image

4.1 Data description

As shown in Fig. 3(a), the dataset contains 27 precious ancient samples of Tang Dynasty inscriptions, which are provided by https://chinavis.org/2022/challenge.html and are licensed by the Ancient Books Library of Peking University. It includes 21 works by 12 authors, including 19 from the Wang Xizhi Semi-cursive Script Series and 3 from the Wang Xizhi Characters Series collection, and obtains 2978 Chinese character images after pre-processing. Wang Xizhi Semi-cursive Script Series is a collection of works by authors in different periods of the Tang Dynasty who have followed the Wang Xizhi style of calligraphy. Wang Xizhi Characters Series is a collection of calligraphic works that integrate the handwriting of Wang Xizhi. These works record calligraphic information from different authors at different times and provide effective characteristics for the downstream task.

4.2 Data pre-processing

As ancient inscriptions have a long history, the quality of acquired images is mediocre. In addition, the variety of Chinese characters, the complex character structure of Chinese characters and the large number of similar characters lead to difficulties in extracting characteristics of Chinese calligraphy. To efficiently extract Chinese calligraphic characteristics, the image datasets are pre-processed to remove the noisy images and improve the effect of font segmentation and text recognition of the inscription images.

To improve the image quality, emphasize the text brush, reduce the background color and the noise impact, the image is grayed out. By comparing the gray scale effect of the component method [44], such as the maximum method and the weighted average method, we use the maximum method to set the image gray scale and then binarize the image. Furthermore, by setting the gray value of each pixel point to the median of the gray values of all pixel points in a \(3\times 3\) sliding window at that point, the pretzel-like noise in the inscription images can be efficiently removed and the edge characteristics of the images can be maintained.

Regarding the subsequent extraction of Chinese characters, we split the inscription images by Chinese characters. Since the structures of Chinese characters are more complex than English words, there are independent structures such as the radical structure, so Chinese characters are prone to incorrect segmentation. At the same time, the problems of uneven alignment and different character spacing variations in multiple inscriptions make the traditional methods’ segmentation effect insufficient, such as the horizontal and vertical projection methods [45]. To improve the accurate segmentation rate of Chinese characters and the adaptability of segmentation algorithms in inscription images, we improve the horizontal and vertical projection methods and propose a recursive splitting algorithm based on the adaptive adjustment of structural thresholds, as it is shown in Algorithm 1. The algorithm adaptively calculates the position \(\mathrm{adj}_{hi}, \mathrm{adj}_{vi}\) to be segmented by traversing the number of pixels in the horizontal and vertical directions of the image, and recursively segments until the segmentation result no longer changes or reaches the maximum recurrence depth \(\mathrm{deep}_{\mathrm{max}}\).

Algorithm 1
figure a

IterSplitImage(arr, n, step)

Full size image

4.3 Data feature extraction

Calligraphy is composed of lines as the basic elements, formed by organic combinations that create the structure and shape of beauty [46]. Thus, we consider the shape and power of calligraphy to compare the characteristics of different calligraphic styles. Based on the segmented images obtained in Sect. 4.2, the features of Chinese calligraphic characters are extracted from both the granularity of Chinese characters and works (Chinese character structure and the calligraphy works’ style).

In terms of Chinese character structure, we extract the skeleton feature, outer and inner contour features and rice grid structure features (see Fig. 3(b)). The skeleton features represent the morphological features of the characters, which stress the composition and layout of characters’ strokes. The outer and inner contour features indicate the projection features [5], and they emphasize the balance of the character strokes in the horizontal direction and vertical direction. The rice grid structure features express the position features [5], which emphasize the balance of the whole character. In terms of the style of the works, we approximate the stroke thickness and conjoined features (see Fig. 3(b)) to indicate the calligraphic style. The thick/thin strokes show whether the work is graceful/bold, and the conjoined features demonstrate whether the calligraphy is smooth [47]. These features are consistent with the evaluation of the aesthetics of calligraphy—the coordination, balance and distribution [11, 46].

Granularity of Chinese characters These features provide information on the structure and morphology of Chinese characters, such as the skeleton features, outer and inner contour features and rice grid structure features (see Fig. 3(b)). Skeleton features [48] are used to extract structural information of Chinese calligraphic characters, and we adopt the binary image refinement method to obtain the skeleton features of Chinese characters by refining a region connected to the width of one pixel.

The outer contour feature is designed to describe the external contour information of the Chinese characters. We scan the segmented binary image processed in Sect. 4.2 from the four directions of the image: top, bottom, left and right, and count the area of each direction that first encounters the strokes. When performing the traversal operation from the top direction of the image to calculate the outer contour feature as an example, as illustrated in Fig. 3(b), the top direction is divided into four equal regions and the area of the four regions (four shaded sections) \(C_{ t1 }, C_{ t2 }, C_{ t3 }\) and \(C_{ t4 }\) is calculated to obtain the vector \([C_{ t1 }, C_{ t2 }, C_{ t3 }, C_{ t4 }]\). Finally, the characteristic vector in the four directions obtains \(4\times 4\) dimension characteristics. The internal contour feature is used to describe the internal structure information of the Chinese characters. We scan the refined skeleton images of Chinese characters from four directions and calculate the area between for the first time we encounter the stroke (the pixel point is 1) and for the second time we encounter the stroke again in the pixel data traversal, as shown in Fig. 3(b). Similar to the acquisition of external contour features, we finally obtain four dimension vectors to represent the inner contour features.

Moreover, the rice grid (see Fig. 2(a)) is generally an auxiliary tool for beginners in calligraphy [43]. It helps users to learn the layout of the Chinese characters and divides a square grid into eight triangles. Inspired by the function of the rice grid and position features in reference [5], we place the segmented binary image in the rice grid. To compare the differences in the structure of the same Chinese characters from different works, we calculate the percentage of the Chinese characters in the eight triangular areas of the rice grid. Specifically, we first scale down the longest side of the segmented Chinese character image to 255 pixels, and the other side to an equal proportion. Next, we place the deflated Chinese characters in the center of the \(255\times 255\) pixel rice grid (the character’s centroid is the center of the rice grid). Finally, we divide the characters into eight pieces according to the rice grid and calculate the proportion of each triangle area (see Fig. 3(b) Rice Grid Area).

Granularity of the works These features include the stroke thickness and calligraphy conjoined features (see Fig. 3(b)). The calligraphic style is generally represented by such features as the stroke thickness of the calligraphic works, so we compare the calligraphic styles by calculating the thickness of the strokes of works. In this work, we adopt the Stroke Width Transform (SWT) [49] to calculate the thickness of strokes, which starts from a point on an edge in an image, and finds a point on another edge parallel to it by traversing pixels in the direction perpendicular to the edge. Then, we form a stroke cross-section from the points of these two edges, and produce a completed stroke by connecting multiple stroke cross-sections of similar width.

Furthermore, as displayed in Fig. 2, a comparison between semi-cursive script and the regular script shows that there is a significant difference in the characteristics of the continuous strokes. The regular script with pauses between strokes is more regular than the semi-cursive script, while the semi-cursive script has the continuity between strokes, making the characters look smoother [39, 50]. In this work, we approach closer to the calligraphic conjoined feature by calculating the number of connected domains of Chinese character images. We implement the search around the domain with the seed filling algorithm to determine the connected regions and calligraphic conjoined feature.

Calculation of feature similarity According to the above extracted characteristics, this work calculates the similarity of the various characteristics of the works to explore the characteristics and evolutionary pattern of Wang Xizhi’s calligraphic style. We select the same Chinese characters in different works for similarity calculation, and the various features are weighted to represent the general similarity between the works.

Since the skeleton features of Chinese character are input as images, we calculate the similarity by image processing methods, such as the root mean squared error (RMSE) [51], peak signal to noise ratio (PSNR) [52] and structural similarity (SSIM) [53] algorithms. The RMSE algorithm calculates the square root error of the image pixel values, which is sensitive to scaling and cropping operations. It is generally suitable for comparing the difference between the before and after frames in the videos. Both PSNR and SSIM algorithms are suitable for aligned images with low complexity. Thus, they are suitable for comparing different skeleton features of the same Chinese character. However, the SSIM algorithm is more consistent with the human eye’s judgement of image similarity than the PSNR algorithm in terms of image similarity measurement [54]. Therefore, we use the SSIM algorithm to measure the similarity of the skeleton features of Chinese characters in this paper, and the calculation formula is as follows:

$$\begin{aligned} \begin{aligned} \mathcal{SSIM}(x,y) &= \bigl[l(x,y) \bigr]^{\alpha}\bigl[c(x,y)\bigr]^{\beta}\bigl[s(x,y) \bigr]^{ \gamma } \\ &= \frac{(2\mu _{x}\mu _{y} + C_{1})(2\sigma _{x}\sigma _{y} + C_{2})}{(\mu _{x}^{2} + \mu _{y}^{2} + C_{1})(\sigma _{x}^{2}\sigma _{y}^{2} + C_{2})}, \end{aligned} \end{aligned}$$
(1)

where \(x, y\) represent the skeleton image of any two Chinese characters, \(l(x, y), c(x, y), s(x, y)\) are used to compare the brightness, contrast and structure of two images, respectively, \(\alpha, \beta, \gamma \) are power indices to adjust the importance of the three factors and we set them all to 1 here. Further simplification, \(\mu _{x}, \mu _{y}\) is the mean values of the \(x, y\), respectively, and \(\sigma _{x}, \sigma _{y}\) is the variance of the \(x, y\). \(C_{1}, C_{2}\) are constant values to prevent the formula from being divided by zero, the value of \(C_{1}\) is 6.5025 and the value of \(C_{2}\) is 58.5225.

In terms of stroke thickness features, outer and inner contour features, conjoined stroke features and rice grid region features are all in the form of vectors or matrices, so we first flatten the vectors or matrices and then calculate the Euclidean distance between vectors. The calculation equation is described as below:

$$\begin{aligned} \mathrm{Similarity}(x,y) = \frac{1}{1+\sqrt{\sum_{k=1}^{n}(x_{k} - y_{k})^{2}}}, \end{aligned}$$
(2)

where \(x, y\) represent the feature vectors of two Chinese characters, and k represents the dimension of the feature vectors. Then, the higher the similarity value, the higher the similarity between the two characters.

5 Visual design

Following the design goals, we design and implement the visual interface to support the public in understanding and appreciating calligraphic works, and further explore the evolution of Wang Xizhi’s calligraphic style in the inscriptions of Tang Dynasty. The system consists of three modules: (1) Spatial-temporal Correlation Analysis Module to explore the development of calligraphic works in relation with the time and region (G1); (2) Similarity Analysis between Works Module to provide users with various calligraphy metrics for understanding the calligraphy style (G2, G4); (3) Calligraphic Style Evolution Analysis Module to present the evolutionary path of calligraphy of Wang Xizhi(G3). Each module contains the detailed views to help the public further explore the characteristics of the calligraphy. Users can first explore the spatial-temporal distribution of the work, the character structure and the style of the work through the overview. Furthermore, with the help of textual explanation and interactive operations, users can explore seven detailed views step by step in the order of the spatial-temporal distribution, similarity analysis and evolution of Chinese calligraphic style. It is ensured that users have thoroughly used the core functions and explored each part of the system.

5.1 Spatial-temporal correlation analysis module

This module explores the trends in the development of Tang Dynasty inscription calligraphy in two dimensions: temporal distribution and geographical distribution. Figure 1(c) maps the rise and fall of the Tang Dynasty with the height of the mountains, and distributes the works in chronological order to help users explore the relationship between the rise and fall of the Tang Dynasty and the number of inscriptions. Furthermore, in the detailed view (Fig. 4(c-1)), the number of inscriptions is totaled by the reign title, and the number of distributions is reflected by the size of the dots. Users can click on different reign titles to view the information of the works. Figure 1(a) uses a bubble chart to visualize the geographical distribution of the works, with the size of the circles indicating the number of works in different regions. Furthermore, users can click on a region to view the distribution of works and learn about the information of the inscriptions in Fig. 1(b). Combining the chronology with the geographic distribution, the users are guided to explore the influence of the rise and fall of the Tang Dynasty and the geographic distribution on the development of calligraphy.

Figure 4
figure 4

Spatial-temporal Correlation Analysis Module explores the trends in the development of Tang Dynasty inscription calligraphy in temporal and geographical distribution. (a) and (c) are the part of the overview, (a-1) is a detailed view of the distribution and (c-1) is a detailed view of the timeline. (a-1) introduces the relationship between the spatial distribution in Chinese at the upper right part of (a-1), presenting the number of works in relation to the geographical distribution. (c-1) introduces the relationship between the temporal distribution and lists the information of the works in Chinese at the bottom of (c-1). It shows the number of the works in different years in relation to the rise and fall of the Tang Dynasty. The size of the dots on the timeline or on the graph indicates the number of the works and the statistical graph of quantities at the lower right part of (a-1)

Full size image

5.2 Similarity analysis between works module

Calligraphers perpetuate the calligraphy style of Wang Xizhi while developing their individuality, so users can see the similarity between calligraphic styles of calligraphers while appreciating calligraphy. This work extracts calligraphic strokes of the conjoined and thickness features as the style features of the works and clusters them according to the above features. Figure 5(f-1) uses ink dots as the mask to display various calligraphy works, and the distance between the works indicates the degree of similarity in style. Users can click on the works to see the comparison of the work style with the style of Wang Xizhi in “Preface to the Poems Composed at the Orchid Pavilion”.

Figure 5
figure 5

Similarity Analysis between Works Module presents the similarity between calligraphic styles from multiple dimensions. (d) and (f) are the part of the overview, while (f-1) and (f-2) present the similarity between the works. Based on the calligraphic strokes of the conjoined and thickness features, (f-1) is a detailed view to compare the works style with the calligraphic style of Wang Xizhi. The circle diagram represents the comparison of different works on the above two features. (f-2) calculates the similarity between the works by weighting the four characteristics of Chinese characters and presents the characteristics at the lower right part of (f-2). It introduces the relationship between the works in Chinese on the upper right part of (f-2). The left part of (f-2) presents the similarity of the works by the thickness of lines between the works. (f-3) compares calligraphy to mountains and presents a brief introduction in Chinese in the lower left part of (f-3). It uses the height of the mountain to indicate the thickness of the strokes and the order of the strokes with a rolling mountain range

Full size image

Both momentum and rhythm, as well as brushwork, structure and layout are indispensable to calligraphy. This paper extracts four characteristics: skeleton feature, outer and inner contour features and thickness of stroke. Then, we calculate the similarity between the two works. Following the above features, Fig. 5(f-2) reflects the similarity between different works, with thicker lines between different works representing higher similarity and thinner lines representing lower similarity. Users can click on the lines to see the feature values of different works, which makes it intuitive for users to see the similarity among works.

Chinese calligraphy consists of a series of strokes, which is known as “the art of strokes” [46]. Therefore, we extract the strokes of Chinese characters and the stroke thickness feature as data for visualization, while the thick/thin strokes show whether the work is graceful/bold. We can generally present the stroke order of each character by enumeration and compare the stroke thickness characteristics among different works by bar diagram. Inspired by the art of Chinese landscape painting [55], different shapes of mountains affect the style of landscape painting, so we want to link the mountains with calligraphy. We attempt to use the momentum of different mountains to indicate different calligraphic styles and map the thickness of the strokes to the height of the mountains. Therefore, we enumerate the strokes of a Chinese character by the horizontal axis to form a rolling mountain, while a pair of works form a group of mountains in Fig. 5(f-3). We map the stroke thickness to the height of the mountain. This allows users to intuitively perceive different styles of different works by comparing the height of the mountains.

5.3 Calligraphic style evolution analysis module

The art of calligraphy has lasted for thousands of years, showing its artistic charm with its unique form. Therefore, we compare the morphologies of the same characters in different works to help users understand the style of different calligraphy works. We segment the images from the inscriptions and extract high-frequency characters to build a word cloud using the Orchid Pavilion [56]. As demonstrated in Fig. 6(e-1), users can view the original image and information of the corresponding character by clicking on a character in the word cloud, and the historical evaluation of the corresponding work of the character is hovered around. This interaction helps the users clearly see the comparison of the same Chinese character shapes in various works.

Figure 6
figure 6

Calligraphic Style Evolution Analysis Module helps users explore the evolutionary path of the calligraphic style in Tang Dynasty inscriptions. (e) is the part of the overview and the circle diagram indicates the stroke thickness and continuous stroke characteristics of the work. (e-1) is a detailed view of the word cloud and shows the same Chinese characters and history comments from the various inscription works. The central bottom part of the (e-1) presents the characteristics of the various styles of calligraphy in Chinese. (e-2) presents the structure and layout features of the same characters and displays them in chronological order. The right part of the (e-2) shows the evolution of calligraphy in Chinese

Full size image

On this basis, in order to help users explore the evolutionary path of calligraphic style of Wang Xizhi in Tang Dynasty inscriptions, this paper classifies the same Chinese characters from different works according to their authors. Then, we analyze the structure and layout features of the characters, map the characters in the rice grids, categorize the similar structural drawings and display characters in chronological order (see Fig. 6(e-2)).

6 Case study

We first provide a brief introduction of the system implementation and then demonstrate the functionality and usefulness of the system through two case studies.

We chose two users to participate in the experiment and presented insights in the cases. One of them is Huang, a university student with five years of experience in calligraphy, and the other is Lu, a university student with no experience in calligraphy at all. The first case shows how to obtain an initial overview of the overall comparison of calligraphic styles of Tang Dynasty inscriptions, while the second case shows how to have a deeper insight into a particular work from a detailed view.

6.1 System implementation

We design a visualization system for visual explorations of calligraphy from Tang Dynasty inscriptions’ image dataset. The system helps users to understand the characteristics and evolution of the calligraphic style of Tang Dynasty inscriptions from different periods. The system consists of one overview view and seven detail views, which presents a horizontal scrolling format to explore the characteristics and evolution of Wang Xizhi’s calligraphic style from a macroscopic to a microscopic perspective.

The system is a web-based system with two components: a back-end for processing inscription images, extracting Chinese character features and calculating the similarity between works, and the front-end for visualizing the calligraphy metrics. We use OpenCV to implement the back-end and the D3.js library to implement the front-end.

6.2 Case 1: an overview of the inscription

In this session, we present the insights that the users discovered during the exploration (see Fig. 1).

First, they observed the chronological mountain graph in Fig. 1(c), and learned the frequency distribution of all works of inscriptions on the timeline. They found that the mountains were higher in the regions where the works were denser and relatively lower in the regions where the works were sparser. After learning that there was a link between the rise and fall of the mountains and the Tang dynasty, Huang suggested that calligraphic culture would have been influenced by the politics and economy of the time. Therefore, she considered this presentation to be very plausible. They found the geographical distribution in Fig. 1(a) and clicked on the largest dot to enter the local map of Shaanxi Province. Then, they saw the exact location of the works in that province. Furthermore, they clicked on the name of the work and knew the original image and related information in Fig. 1(b).

Huang was interested in the word cloud distribution in Fig. 1(d), where she discovered that the characters “zhi”, “guo” and “shen” could be clicked on for interaction. At the same time, the character structure diagrams in Fig. 1(e) would change correspondingly. Huang found that the distribution of calligraphic fonts could be displayed in the meter grid well, which helped her recognize the similarities and differences of different authors’ strokes. Lu suggested that this approach could present the beauty of calligraphy in a new structural system. She thought that even though users have no basic knowledge about calligraphy, they can still observe the difference between calligraphic characters directly.

Finally, they found that the chord chart in Fig. 1(f) showed the similarity between the eight works. The similarity result was statistically extracted from the four features below, which were visually reflected in the thickness of the chords. Lu thought that using data visualization to present the similarity of calligraphic works clearly. This can help people like him who have no knowledge of calligraphy understand the differences and connections between calligraphic works. After looking at the eight original drawings of the works, Huang agreed that the similarity results of the chord diagram were more in line with her knowledge of the field. For example, some works were more similar to her in terms of brushwork, structure and character shape.

6.3 Case 2: exploring calligraphy in detail

In Case 1, Huang and Lu were initially acquainted with the effectiveness of the system. However, they wanted to learn more about a particular work through the detailed views. They chose the work “Tombstone of Gao Fu” to explore the relevant content of the work step by step.

First, they wanted to know the comparison between this work and “Preface to the Poems Composed at the Orchid Pavilion”, so they explored the clustering chart in Fig. 5(f-1). They found that the contiguous feature of “Preface to the Poems Composed at the Orchid Pavilion” was 0.71 and “Tombstone of Gao Fu” was 0.72, the stroke feature of “Preface to the Poems Composed at the Orchid Pavilion” was 51 and “Tombstone of Gao Fu” was 50. In both respects, the two works were relatively similar. Thus, Huang surmised that the author of “Tombstone of Gao Fu” probably imitated Wang Xizhi’s works.

Next, they explored the strokes of the mountain range diagram in Fig. 5(f-3) by the “Tombstone of Gao Fu”. They observed that the overall changes is relatively gentle, which was consistent with the state of the work’s thin strokes. In addition to this, Lu appreciated the beauty of the mountain peaks constructed by the strokes of Chinese calligraphy.

Finally, they browsed the different writing styles of the same characters in Fig. 5(f-2). By comparing them horizontally in time, Lu visually observed the different ways in which the characters “zhi” and “wei” were written in the “Tombstone of Gao Fu”. Huang found that the “Tombstone of Gao Fu” belonged to the early Tang Dynasty. Its script tends to be slender and fluid, which is more similar to Wang Xizhi’s calligraphy. In addition, some of the post-mid-Tang works tend to be thicker and more atmospheric. Although the shadow of learning Wang Xizhi’s calligraphy can also be found, it has more of its own characteristics.

7 Evaluation

To evaluate the functionality and usefulness of the system, 20 users were interviewed and consulted about their feelings after using the system.

7.1 Participants

Twenty college students (age mean: 22.4; female: 10) volunteered to participate in our interviews. Twelve of the users came from humanities and arts disciplines, and the other eight were from computer science disciplines. Five of the users had three or more years of experience practising calligraphy.

7.2 Procedures

The user interviews were structured in three parts. First, participants watched a five-minute demonstration video describing the functions in detail. They could pause the video at any point to ask questions or reflect. After the video, participants interacted with the system on their own. The process lasted for 10–30 minutes. Finally, participants completed a questionnaire to assess the functionality of the system, which took approximately 20 minutes on average. Each question in the questionnaire contained a 5-point Likert scale (from 1 = strongly disagree to 5 = strongly agree), and participants needed to give their reasons after rating. The entire experiment took approximately 50 minutes on average. Figure 7 lists the questions in the questionnaire.

Figure 7
figure 7

The questions in our questionnaire and their corresponding score average and distribution

Full size image

7.3 Results and feedback

Based on users’ questionnaire ratings and free comments, we analyze the three aspects of view design, system interaction and overall evaluation as follows.

View design In terms of the view design, most of the participants agreed that the chronological mountain graph and the geographical distribution could reflect the spatial and temporal distribution of the inscriptions well \((4.20/5)\). A small number of participants remarked that the amount of data was too small and the information was limited. The participants agreed that the cluster and chord diagram could be used to compare the similarity of works \((4.80/5)\), which could be intuitively understood from the visual guidance of distance, color, thickness, etc. Participants showed polarized evaluations of the stroke of mountains graph \((3.87/5)\). Some commented that the design view used the intention of mountains to reflect the overall thickness of strokes between the different works well. However, some expressed doubts about whether the view could truly reflect the thickness of strokes. Participants were most satisfied with the functional design of the word cloud view and the character structure view \((4.75/5)\). “Placing the font in a rice grid will allow users to better understand the layout of strokes.” “It is possible to discover the similarities and differences between works.”

System interaction In terms of system interaction, most participants agreed that the overall design of the system is beautiful and artistic \((4.70/5)\), “Intentional calligraphy as mountains, the change of strokes shaping the high and low trend.” “The word cloud graph is constructed with the intention of ‘Lanting’, which fits the theme well.” Some participants felt there was room for improvement in the interactive format \((4.25/5)\), “The overall visual cue guidance is clearer, but some of the interactive features are vague, and there are fewer illustrations.” “Easier to operate, but too few interactive features.”

Overall evaluation In terms of overall evaluation, participants remarked that this system stimulated their interest in calligraphy \((4.50/5)\). “The calligraphy was presented in a relatively novel way and system interaction is interesting” “Even people who do not know much about calligraphy can appreciate the artistry and knowledge of Chinese calligraphy.” Participants were very supportive of presenting calligraphy works in a visual way \((4.40/5)\) and agreed that the system was popularity of science \((4.50/5)\), “Presenting calligraphy works with data visualization is more scientifically useful than just showing the original calligraphy images.” “Although it is only a visual analysis of the works of Wang Xizhi’s style, it provides an idea of how to understand other works of art.” For the calligraphy learners among the participants, they commented that the system could better help them learn calligraphy \((4.20/5)\). “Arranging the works by chronological history can give learners a clear direction to imitate, and they can appreciate Wang Xizhi’s style from different dimensions.”

Suggestions for improvement Some participants prefer to have more annotated information and interactive guidance. For example, some users would like to introduce features beyond the calligraphic strokes of the conjoined and thickness features as a measure of similarity in calligraphic style, such as the way in which the strokes start and the strength of the strokes. Moreover, the users tended to add annotations explaining the effect of different features on the style of the calligraphic works. Some people also suggested adding more textual introduction or image display about the works and authors. This would make the system more popular in science. These suggestions help us optimize the system and provide future directions for development.

8 Discussion

Although case studies and evaluation can verify the functionality and utility of our system, they inevitably have some limitations. In this section, we summarize the lessons learned from this study and explore future directions for development.

Calligraphic evolution We focused on the study of the Tang Dynasty monuments and posters in Wang Xizhi’s calligraphy style. Due to the insufficient dataset, the pattern information obtained through data visualization is not rich enough, resulting in a thin main story line. The evolutionary relationship between Wang Xizhi’s calligraphy style needs to be further explored. In addition, some of the view interaction information is not coherent enough. The logic is not clear enough, and some of the patterns are more far-fetched and improbable. It is necessary to further expand the exploration by acquiring more works of Tang Dynasty inscriptions and Wang Xizhi’s calligraphy.

Generalizability The framework of the system can be extended to other similar analysis tasks of ancient calligraphic images or even other cultural heritage. The system can be deployed in museums, libraries and other places. It can provide novel ideas for the preservation and dissemination of traditional Chinese culture. Further, the system can be extended to the field of calligraphy evaluation systems, but the work is currently difficult. It requires upgrading of image processing techniques and calligraphy style extraction algorithms.

Evaluation In the user evaluation phase, we collected many positive comments about the system from 20 college students. This proves to some extent the success of the system’s visual analysis direction for calligraphy. However, the interviewees all came from humanities, computer science, and art disciplines. They have a good professional foundation and understanding of learning ability. It is difficult to fully represent the general public users that we need to target. We also need to be more accessible in the interactive guidance and view explanation of the system.

9 Conclusion and future work

In this work, we propose an interactive visualization system for analyzing the calligraphy style of the inscriptions of the Tang Dynasty. Based on image processing techniques to complete extraction of the calligraphy features from multiple dimensions, such as layout and structural features, we calculate similarity between different works. Following the above extracted features, we develop a visual analysis system that allows the ordinary users to interactively explore the calligraphy features and the evolution path of Wang Xizhi’s calligraphic style. Two in-depth case studies and a user interview demonstrate the usability and effectiveness of the system in facilitating users’ understanding of calligraphic characteristics. Feedback from the users’ interviews provides us with some valuable suggestions for future work.

In future work, we intend to expand the scope of image processing and enhance the extraction of calligraphy features on a more refined level. In addition, we provide a more intelligent workflow to expand to the exploration of calligraphy works other than inscription works. Based on the above techniques, we further explore the connections of calligraphy features and enrich the pattern information to introduce users in a more vibrant way about calligraphy styles and evolution. Moreover, we hope the system framework is extended to other cultural heritage analysis tasks and is displayed in museums or other venues, which provide a novel approach to the preservation and dissemination of traditional Chinese culture.