1 Introduction

Chinese calligraphy is a fascinating and ancient way of expressing Chinese characters as a form of art. It is a highly artistic practice with a long history and is an important part of Chinese art [1]. The soft hairy brush is an essential tool for the preservation of traditional Chinese culture and has held a precious position in the culture of Chinese calligraphy since its creation [2]. As industrial robots have developed, their control accuracy has reached a level where they are fully capable of reproducing precise hand movements. As a result, recent research has focused on using robotic arms to simulate human calligraphy behavior [3, 4].

Chinese characters are formed by combining writing characteristics and writing trajectory. Writing characteristics relate to brush strokes, which reflect the shape of Chinese characters created with different brush position and posture. The brush strokes are superimposed along the trajectory points to generate the stroke models. These brush strokes are a crucial component of Chinese calligraphy and are viewed as calligraphers’ handwriting. A stroke is a specific graphic that is generated when a brush is pressed on paper without moving, with a certain depth and angle of inclination. The simulation of the brush stroke model is key to robotic Chinese calligraphy. The construction of brush stroke models provides the foundation for robotic Chinese calligraphy as a stroke is a basic unit of a character’s trace. Thus, finding a way to simulate stroke models with physical parameters and training a stroke renderer based on parameters that are similar to real writing samples is an important challenge that needs to be addressed.

In recent years, some researchers have focused on generating and simulating trajectories by training and optimizing trajectory models using reinforcement learning [5, 6] and generative adversarial networks (GANs) [7,8,9,10]. However, they have ignored the stroke point as the basic unit of the stroke trace and the actual writing process. Furthermore, a large number of repeated writing iterations can quickly wear out robotic arms. Many studies have explored robotic arms writing with a hard pen and generating the trace of strokes as brush pen strokes [11, 12], but they have not considered the brush stroke model, especially the correspondence between the control parameters of the calligraphy robot and the generated brush stroke models. Only the stroke width of the writing trajectory has been considered for intelligent Chinese calligraphy using robotic writing [13, 14]. The width of the stroke is related to the height of the vertical pressure of the brush pen. However, when writing calligraphy, the brush tip tilts. Therefore, it is necessary to analyze the relationship between the brush stroke model and the height and tilt angle of the robotic arm.

Some brush stroke models have been synthesized based on stroke graphic generation techniques [15,16,17,18]. Among the existing models, Bézier curve-based stroke simulated methods [15] have achieved better performance. The methods use two symmetrical cubic Bézier curves to simulate the shape of the stroke graphic, taking morphology into account, and then estimate parameters through image samples and learning algorithms. Recently, neural renderers based on brush strokes have been presented in deep reinforcement learning [19,20,21]; however, they were used for painting not writing. So far, the end-to-end method involves building a generative model based on neural networks, which requires a large number of samples for training. The ideal training samples are the control parameters and corresponding images of the robotic arm, which allows the trained model to establish the relationship between parameters and images. This practice is not realistic at present. Therefore, it is necessary to solve this issue directly or only use samples for the parameter estimation model to approximate the real samples. Subsequently, the end-to-end generator can be trained using reinforcement learning or deep learning methods.

To address these problems, this paper proposes a Bézier brush stroke model-based generator (B-BSMG) for robotic Chinese calligraphy. This approach involves obtaining the mathematical relationship between the brush stroke image and the control parameters of the robotic arm based on a large number of experiments. The brush stroke model is then simulated and used as training samples for the B-BSMG, which trains the end-to-end generator and generates images using a neural network. The results of experiments show that the B-BSMG has a high similarity to real writing samples. This is demonstrated in Fig. 1, which shows a stroke written using a real brush pen and the effect of the proposed B-BSMG.

Fig. 1
figure 1

Stroke written using a real brush pen and B-BSMG

Full size image

In summary, the contributions of this paper mainly include the following:

  1. (1)

    The simulation brush stroke model is mathematically defined. The mathematical relationship is established between the parameters of the simulation brush stroke model and the robotic arm control parameters h, α, β.

  2. (2)

    The simulation brush stroke model generator is built that uses a differentiable neural network. This generator offers increased flexibility and is trained for optimal performance.

  3. (3)

    Several experiments show that our proposed generator, which is based on a stroke model, outperforms some existing neural renderers that also rely on stroke-based rendering.

The rest of this paper is organized as follows. Section 2 introduces related work. Section 3 outlines the main idea and method used to generate the stroke model in our proposed B-BSMG. Section 4 presents the results of the experiments conducted to verify the performance of the proposed method along with a discussion of the results. Finally, Sect. 5 concludes this paper and suggests some possible avenues for future research.

2 Related Work

2.1 Empirical Models Based on Virtual Brushes

In their work, Wong and Ip [16] focused on the design of the brush bundle. Their model set the radius of the brush holder, the length of the brush bundle, and the number of brush bundles as fixed parameters. They generated an elliptical brush stroke shape by creating a perpendicular intersection between the paper and the brush bundle. However, the fixed model proposed by Wong and Ip does not adhere to realistic writing practices. When using a brush pen to write on paper, it is not possible for the pen to penetrate the surface of the paper, rather it leaves a mark in the form of a brush stroke on the surface of the paper. Wang and Min [15] collected a large number of brush stroke shapes produced by a robotic arm and used these data to develop a multiple linear regression model using the least-squares method. They used statistical methods to simulate the brush stroke shape by enclosing a closed area with two symmetrical third Bézier curves and applied linear regression analysis to determine the brush stroke model parameters. The model established the mathematical relationship between the length of the brush bundle tip, the length of the brush bundle base, the length of the brush bundle belly, and the descending height and tilt angle of the brush. Joshi [17] and Wang et al. [18] proposed fitting the brush stroke using a circle. They proposed a method to model the formation of brush strokes by continuously superimposing circular brush strokes along the trajectory of the strokes, by combining trajectory and brush strokes.

Recently, various brushstroke renderers (BSR) for painting have been proposed in deep reinforcement learning [19,20,21]. Some researchers have described the generated graphics using Bézier curves with thickness, position coordinates, and opacity at both the beginning and end points on a given canvas. Based on these features, an agent can paint using deep reinforcement learning. Stroke-based rendering, which incorporates brushstrokes, has also been used to create an artistic style where the renderer’s input image can be converted to a series of brushstrokes [21]. However, the brushstrokes used in these BSR techniques are not suitable for Chinese calligraphy, especially for robotic Chinese calligraphy. Therefore, our proposed brush stroke neural generator was simulated using parameters suitable for robotic end effectors.

2.2 Trajectory Optimization Based on Deep Learning

Chao et al. [7] regarded robot writing as a confrontation process and used a GAN to generate the probability distribution of stroke trajectories. In [22], Chao et al. generated the writing trajectory sequence of Chinese character strokes using an LSTM-GAN. The LSTM-GAN [22] converted the stroke image pixels into a vector trajectory sequence controlled by a robotic arm. The robotic Chinese calligraphy with human preference framework proposed in [23] used a random sample as input for a G network to determine the position points of the stroke trajectory. The output was the probability distribution of the stroke trajectories of various Chinese characters. Wu et al. [5] developed a model that takes a set of Gaussian noise as input and generates stroke images as output. Their model combined a GAN with the actor-critic model of deep reinforcement learning. The deep deterministic policy gradient algorithm was used to train the sequence written by a robotic arm and simulate the writing process of a human. The G network designed by Wu et al. [24] used a U-Net structure. The D network was composed of convolutional layers that reduce the output dimension to a two-dimensional map. Each element in the map represented the possibility of whether the input was true or false. These studies have all used GANs to optimize processes for robotic writing tasks. GANCCRobot [8] improved traditional GAN-based frameworks by adding stroke types and latent code information to represent the characteristics of different strokes. Liang et al. [13] proposed a robot writing method based on deep learning by training a GAN to generate target fonts and realize style transfer. Gao et al. [9] proposed a new learning framework for writing tasks with human esthetic preferences. An autonomous robotic writing system for Chinese calligraphy was proposed by Gan [10]. On the other hand, trajectory optimization-based methods [25,26,27] have used simple sets of disorder points as basic writing units. Therefore, there is a need to model the relationship between generative strokes and robotic end effectors.

3 Proposed Method

Due to the lack of brush stroke samples required for neural network renderer training, the proposed method is divided into two parts. First, a parameter representation of the simulated brush stroke model is created. Then, this model is used as the ground truth to train a brush stroke-based neural network generator, which will learn to produce brush stroke images. An overview of the proposed framework is shown in Fig. 2.

Fig. 2
figure 2

Overview of the proposed framework for robotic Chinese calligraphy

Full size image

3.1 Proposed Brush Stroke Model

To create an appropriate mathematical model that matches real brush strokes, we collected real brush strokes and located the four vertex coordinates. We then used the multiple linear regression method to analyze and build a multiple linear regression model of the simulated brush strokes, which allowed us to obtain the model parameters. We chose the linear regression method, because the simulation model parameters and the control parameters of the robotic arm have a consistent linear relationship. That is, the deeper the brush pen descends and the larger the tilt angle, the larger the length of the brush stroke left on the paper, which satisfies a positive correlation. This has been verified through numerous experiments. Furthermore, each sample obeys independent and identical distribution, and all of them satisfy a uniform distribution.

3.1.1 Tuple Representation of Brush Stroke Model

The real brush stroke model is represented as a tuple with three elements: \(F_{\text{R}} = (h,\alpha ,\beta )\). Where \(h\) denotes the descending height of the robotic manipulator and \(\alpha\) denotes the tilt angle between the hairy brush bundle and the Z axis. β denotes the rotated angle. When the descending height of the brush pen is between 11 and 20 mm, the brush stroke model consists of a closed area enclosed by two symmetrical Bézier curves. The Bézier curve model is denoted by \(S_{\text{B}}\) and is also represented as a tuple with three elements: \(S_{\text{B}} = (L_t ,L_h ,L_r )\). Where \(L_t\), \(L_h\), and \(L_r\) represent bifeng, bigen, and bidu of the collected brush strokes, respectively. The purpose is to build the mathematical relationship between \(F_{\text{R}}\) and \(S_{\text{B}}\).

3.1.2 Multivariate Linear Regression of Model Parameters

The parameters of the brush pen are defined as \(B = \{ L,R\}\), where \(L\) denotes the length of the hairy brush bundle and \(R\) denotes the maximum diameter of the hairy brush bundle. Based on our experience using the brush pen, the pressure value is linearly associated with the \(L\) of the same brush when the brush tip was not deformed and the decreasing height was not more than half the length of the brush bundle \(L\) [28]. This paper assumes that the pressure value depends on \(h,\alpha ,\beta\). Moreover, the values of \(h,\alpha ,\beta\) are directly related to \(L_t\) and \(L_h\). Therefore, there is a linear correlation between the descending height \(h\), tilt angle \(\alpha\), rotation angle \(\beta\) of the robotic arm, and \(L_t + L_h\), as shown in Eq. (1). The ratio relationship between \(L_t\) and \(L_h\), \(L_t\) and \(L_r\) is obtained by matching the real brush stroke model and the simulated \(F_{\text{B}}\) model, as shown in Eq. (2)

$$L_t + L_h = (k_{\rm{1}} *h + k_{\rm{2}} *\alpha + k_{\rm{3}} *\beta + \varepsilon )*pix2len$$
(1)
$$L_h = k_{\rm{4}} *L_t ,L_r = k_{\rm{5}} *L_t .$$
(2)

Here, pix2len denotes the conversion coefficient of pixel unit into length (mm) unit. The pix2len is set to 0.05. \(\alpha ,\beta\) represent the radian.

By substituting Eq. (2) into Eq. (1), we obtain Eqs. (3)–(5)

$$L_t = (k_{\rm{1}} *h + k_{\rm{2}} *\alpha + k_{\rm{3}} *\beta + \varepsilon )*pix2len/(1 + k_{\rm{4}} )$$
(3)
$$L_h = k_{\rm{4}} /(1 + k_{\rm{4}} )*(k_{\rm{1}} *h + k_{\rm{2}} *\alpha + k_{\rm{3}} *\beta + \varepsilon )*pix2len$$
(4)
$$L_r = k_{\rm{5}} /(1 + k_{\rm{4}} )*(k_{\rm{1}} *h + k_{\rm{2}} *\alpha + k_{\rm{3}} *\beta + \varepsilon )*pix2len.$$
(5)

Formula (6) is used to represent \(L_t ,L_h ,L_r\)

$$S_{\text{B}} = (L_t ,L_h ,L_r ) = K_{\rm{1}} *h + K_{\rm{2}} *\alpha + K_{\rm{3}} *\beta + \varepsilon .$$
(6)

3.1.3 Standard Brush Stroke Design

We have designed a cubic Bézier curve (CBC) to simulate the graphics created by a brush pen written by a robotic arm. The simulated brush stroke is defined as the following tuple:

$$S_{\text{B}} = (x_{\rm{0}} ,y_{\rm{0}} ,L_t ,L_h ,L_r ),$$
(7)

where (x0, y0) are the starting coordinates of the CBC. The SB model based on the CBC is used by the network architecture of the BSMG to implement the generator for creating CBC brush stroke designs.

The formula of the standard CBC is

$$B(t) = P_0 (1 - t)^3 + 3P_1 t(1 - t)^2 + 3P_2 t^2 (1 - t) + P_3 t^3 \quad t \in [0,1],$$
(8)

where P0, P1, P2, and P3 denote four control points of the CBC. The shape of the Bézier curve is specified by the coordinates of control points. P0 and P3 control the starting and ending points of the CBC. Figure 3 shows the simulated SB model of real brush strokes. We assume that P1, P2 is a horizontal line segment. The tangent of the CBC at the highest point is also horizontal. According to the parallel line theorem and the definition of the CBC, the CBC reaches its highest point when t = 0.5 in the curve equation. The control point coordinates P1, P2 can be calculated using Eq. (9) to determine the standard model parameters

$$\left\{ \begin{gathered} P_{{\rm{1}}x} = \frac{{\rm{1}}}{{\rm{3}}} \ast (L_t - {\rm{4}}L_h ) \hfill \\ P_{{\rm{2}}x} = L_h \hfill \\ P_{{\rm{1}}y} = P_{{\rm{2}}y} = \frac{{\rm{4}}}{{\rm{3}}} \ast L_r \hfill \\ \end{gathered} \right..$$
(9)
Fig. 3
figure 3

Simulated brush stroke

Full size image

3.2 B-BSMG

The B-BSMG based on Bézier curves is built. They are used to train the agent to produce brush strokes that are closer to the simulated Bézier brush strokes, which serve as the ground truth. In addition, the B-BSMG can be extended to work with other brush stroke models. All of the models can use the same network architecture structure. Figure 4 shows the B-BSMG based on the simulated Bézier brush stroke models.

Fig. 4
figure 4

B-BSMG based on simulated brush strokes

Full size image

3.2.1 Neural Generator BSMG

The input of BSMG is a set of B-BSM parameters \(S_{\text{B}} = (L_t ,L_h ,L_r )\) and the starting coordinates (x0, y0). The outputs are the brush stroke images. B-BSM is used to generate training samples. The neural network generator is then learned and trained with supervised learning. The premise of learning and training is that the geometric curve of the simulated brush stroke model needs to have a closed-form gradient. The SB-based models proposed in this paper are all continuously differentiable. To train the agent successfully, it is necessary to continuously approximate the discrete pixel position and the pixel value of two-dimensional images when deriving the gradient, so that the error can be back propagated.

There are two main advantages to using neural networks to generate brush strokes. First, the neural generator can flexibly generate the brush strokes of any given simulated model, and it is more efficient due to the powerful computing capabilities of GPUs. Second, the neural generator is differentiable, allowing for end-to-end training to improve the performance of the agent.

3.2.2 Network Architectures

The network architecture of the B-BSMG consists of several fully connected neural networks (FCN/MLP) and convolutional neural networks (CNN). An FCN is used to take the model parameters (h, α, β) of the robotic Chinese calligraphy as the input and map them into two-dimensional matrix brush stroke images that are output by the CNN. Figure 5 shows the network architecture of the BSMG.

Fig. 5
figure 5

Network architecture of B-BSMG

Full size image

3.2.3 Loss Function

The loss function used for the B-BSMG is the mean squared error (MSE) between the input data \(y_i\) and the generated data \(\hat{y}_i\), which is expressed in Eq. (10). All of the parameters of B-BSMG can be easily optimized through a back-propagation algorithm

$${\text{MSE}} = \frac{{\rm{1}}}{n}\sum_{i = {\rm{1}}}^m {w_i } (y_i - \hat{y}_i )^{^{\rm{2}} } ,$$
(10)

where n denotes the size of the mini-batch training samples, and \(y_i\) and \(\hat{y}_i\) denote the input sample and generated sample, respectively.

3.3 Stroke Trace Generation with B-BSMG

For any Chinese characters and strokes with different styles, shapes, and sizes, the B-BSMG method can handle, and then, the robot can write along the trajectory. It is defined as 5T-dimensional vector, given by a set of 5D track Ti = (Hi, αi, xi, yi, βi), where i = {0, 1, …., n − 1} denotes the index of a trajectory point in total n points. Hi and αi denote the pressing depth and inclined angle, respectively, and the parameters will be changed with the movement of the brush pen controlled by the robotic arm. \((x_i ,y_i )\) is the intersection point of the brush pen axis and the paper surface. βi indicates the rotation angle of the stroke graphic on the paper plane. The parameters of the B-BSMG \(\{ H_i ,\alpha_i \}\) determine the shape and size of each stroke graphic of the trajectory Ti. The parameters \(\{ x_i ,y_i ,\beta_i \}\) determine the position and direction of each stroke graphic.

The graph search algorithm is applied to find the coordinate points \((x_i ,y_i )\) of the stroke trajectory on each pixel, and the corresponding width at this location can be determined. According to the width and trajectory, we can obtain pressing depth \(H_i\) and inclined angle \(\alpha_i\) of B-BSMG for robotic arm. The rotation angle \(\beta_i\) at coordinate points \((x_i ,y_i )\) can also be calculated by trajectory. Thus the brush stroke can be generated with all of the parameters \(\{ H_i ,\alpha_i ,x_i ,y_i ,\beta_i \}\). Based on this, the coordinates and postures of the trajectory control points of robotic end effector can be converted from the \(\{ H_i ,\alpha_i ,x_i ,y_i ,\beta_i \}\) by coordinate transformation from plane coordinate of the paper to be written on to the robotic base coordinate.

4 Experiments and Analysis

4.1 Experimental Hardware System

The configuration of our experimental hardware system is shown in Fig. 6. The specifications of the system are as follows. (1) Collaborative robotic arm: Aubo-I5, which has a repeatability of ±0.05 mm. (2) Hairy brush pen: Langhao brush pen with brush radius of R = 6 mm and brush bundle length of L = 48 mm. (3) Xuan paper: specially used for hairy brush pen.

Fig. 6
figure 6

Hardware architecture of the robotic calligraphy system

Full size image

4.2 Simulated Brush Stroke Model B-BSM

4.2.1 Data Collection

Aubo-I5 script programming was used to collect real brush strokes. By moving the brush down along the Z axis to the position where the tip of the brush pen just touches the paper as the initial position, we set the initial height value at H = 0. As the brush pen drops, H continually increases. In the experiment, we first collected samples with H = {11 mm, 13 mm, 15 mm, 17 mm, 20 mm} and α = {0°, 5°, 10°}, β = {0°, 5°}, since the parameters H and α, β are controllable in highly precise robotic writing systems. For each combination, at least ten stroke images were collected.

The entire process of collecting brush strokes is shown in Fig. 7. First, the robotic arm moved to the initial position with a perpendicular posture. Second, it adjusted the posture to the value of α and β. In the third step, the robotic arm dropped to H height. When reaching the H, the brush pen pressed down and remained in that position for t = 0.5 s before the last step which was picking up the pen. In Fig. 7d, you can see a brush stroke was left on the surface of the xuan paper. In the experiment, t = 0.5 s ensures that the brush pen was in contact with the paper surface long enough to form a brush stroke while effectively preventing the ink from being over-rendered on the xuan paper. To obtain the brush stroke under the same conditions, the robotic arm had to be re-dipped in the ink and pressed down for the same interval on the xuan paper each time.

Fig. 7
figure 7

Four steps of data collection: a initial stage, b tilt and rotate brush pen, c drop and press brush pen, and d pick up brush pen

Full size image

4.2.2 Brush Stroke Contours Matching

The real brush stroke images underwent several processes including size normalization, cropping, binarization, and contour drawing. In particular, the length values of Lt, Lh, and Lr were converted to the pixel values for both the real collected images and generated images. To achieve normalization, a transformation coefficient λpix2len = 0.05 was defined, which meant that one pixel in the image was equivalent to the length of 20 mm in reality. Thus, all images were resized to a consistent resolution of length based on λpix2len. The simulated SB model parameters \(L_t ,L_h ,L_r\) of \(S_{\text{B}} = (L_t ,L_h ,L_r )\) were set to fit the contour of real brush stroke images. The parameters were adjusted until the two coincided on the whole. Then, we recorded model parameters \(L_t ,L_h ,L_r\). Figure 8 shows the contour matching between a real image and simulated image.

Fig. 8
figure 8

Contour matching a the real brush stroke image, b standard simulated brush stroke, and c contour matching between a and b

Full size image

4.2.3 Model Parameters’ Acquisition

Table 1 shows the regression coefficients between the simulated model parameters \(S_{\text{B}} = (L_t ,L_h ,L_r )\) and \(F_{\text{R}} = (h,\alpha ,\beta )\) collected by the robotic arm. Figure 9 shows the regression relationship between Lt, Lh, Lr and h, α, β.

Table 1 Regression coefficients of \(S_{\text{B}} = (L_t ,L_h ,L_r )\)
Full size table
Fig. 9
figure 9

Regression relationship between Lt, Lh, and Lr and h, α: a,d regression relationship between Lt and h, α; b,e regression relationship between Lh and h, α, and c,f regression relationship between Lr and h, α

Full size image

4.3 B-BSMG’s Training and Test Datasets

4.3.1 Data Preprocessing

To compare the real and simulated images using the same set of parameters, the collected images were preprocessed by resizing them. The images were then cropped and aligned. Finally, a comparison and similarity calculation were carried out for all binarized images. Figure 10 shows the collected stroke images after preprocessing.

Fig. 10
figure 10

Samples of stroke images

Full size image

4.3.2 Training Dataset and Testing Dataset

Following the data collection and preprocessing steps, we generated multiple simulated B-BSM images in the range of H = (11–20 mm) and α = {0°–10°}, β = {0°, 5°}. H and α, β corresponded to the range of samples collected by the robotic arm. The simulated B-BSM images were used as the dataset for training and testing. Some of the training and testing images are shown in Fig. 11.

Fig. 11
figure 11

Examples of training and testing images

Full size image

4.4 Training of B-BSMG

Prior to being used for training, all of the images were resized to a resolution of 128 × 128 pixels. The entire process was trained on an NVIDIA GeForce RTX 2080 Ti GPU, with a batch size initially set to 64 for each step.

The B-BSMG training process involved using the MSE loss function to update the network parameters. During each iteration of the training procedure, B-BSM with parameters H and α, β were used as ground truth and were fed into the generator. We used the Adam algorithm to iterate until the objective was optimized or the maximum number of iterations was reached. The training loss and validating loss results are shown in Fig. 12.

Fig. 12
figure 12

Results of a training loss and b validating loss

Full size image

4.5 Results and Discussion

4.5.1 Comparison with B-BSM

We compared our proposed B-BSMG with other classic models, especially Bézier-based models. The Bézier-based model used two symmetrical cube curves to form a stroke model [15, 28]. Other models included the circular stroke proposed by Joshi [17] and Wang et al. [18] and the elliptical model by Wong and Ip [16] to simulate the hairy brush stroke. Figure 13 shows real images and the corresponding simulated models. To ensure a fair comparison, we used the same parameters for all models. In terms of morphology, our proposed B-BSMG and the Bézier curve-based model performed better in simulating the real stroke graphic compared to other models.

Fig. 13
figure 13

Comparison of proposed B-BSMG and other models

Full size image

We evaluated and compared different methods quantitatively using two metrics, namely CSIM and SSIM. The evaluation results were recorded in the best case, worst case, and average of all images for each method. The comparison values of CSIM and SSIM are shown in Table 2. According to the results, B-BSMG outperformed the other methods with the highest similarity score, reaching a maximum similarity of 94.59%. The comparison values showed that the order of similarity scores was B-BSMG > Bézier > Ellipse > Circle, indicating that B-BSMG performed significantly better than the other methods.

Table 2 Performance (%) of different approaches for simulated brush stroke models
Full size table

4.5.2 Ablation Study

We conducted an ablation study to verify the effectiveness of the introduced network structure in our proposed B-BSMG. Since different stroke representations only required changing the input parameters of the network, we used neural generators with the same network architecture to realize different stroke designs. The Bézier curves-based renderers were used to paint in deep reinforcement learning [19, 21]. Besides the B-BSMG, the quadratic Bézier curve (QBC) strokes were designed with eight parameters, which included the coordinates of three control points and the thickness of two endpoints of the QBC. With these parameters, as shown in Fig. 14, the QBC-based renderers can be trained and generated.

Fig. 14
figure 14

Comparison of our B-BSMG and QBC: a our generated images and b the images generated by QBC

Full size image

4.5.3 Comparison with the Effects of Different Factors

For the brush stroke generation and simulation, the different factors which may affect the realism and quality of the brush stroke images produced by the B-BSMG method contain the effects of ink diffusion, paper texture, and brush deformation.

  1. (1)

    The effects of ink diffusion

    During the writing process, we found that when the hairy brush pen is lack of ink, the penetration of ink is weak, and then, the width of strokes and Chinese characters written by robotic arm with the same parameters is narrow; When the ink is moderate, the brush stroke images are more uniform; When the brush pen contains more ink, the penetration of ink is strong and the stroke is wide. The analysis of gray value of brush strokes was carried out on the images of ink diffusion in three situations and the histogram of value change of brush stroke images is shown in Fig. 15. The brush strokes with moderate ink conform to the model in this paper. Therefore, to obtain better writing results, the experiments were carried out under the condition of moderate ink.

    Fig. 15
    figure 15

    The effects of ink diffusion with ink: a few ink, b moderate ink, and c more ink

    Full size image
  2. (2)

    The effects of paper texture

    Xuan paper is divided into three categories, namely raw xuan paper, semi-raw semi-cooked xuan paper, and cooked xuan paper. According to the characteristic of ink penetration, semi-raw semi-cooked xuan papers were used in the experiment for robotic calligraphy to gain better writing effects.

  3. (3)

    The effects of brush deformation

    After each brush stroke was collected, the robotic end effector performed the operation of dipping ink to avoid the situation of the poor brush strokes collected caused by deformed hairy brushes, split hairy brushes, curved hairy brushes, and the lacking of ink.

4.5.4 Comparison with Other Deep Learning or Reinforcement Learning Methods

In addition to comparing with classical models, we compared the B-BSMG method with other deep learning or reinforcement learning methods. The Bézier-based renderers used for painting were generated in deep reinforcement learning [19, 21]. By training the generator for QBC and the Bézier-based renderer (BBR) was also generated. The work of [29] considered the straight line brush stroke to train stroke-based neural network, the straight line was considered to represent shape parameters of strokes. Figure 16 shows real images and the corresponding deep learning or reinforcement learning models.

Fig. 16
figure 16

Comparison of proposed B-BSMG and other deep learning or reinforcement learning methods

Full size image

We also calculated the two evaluations CSIM and SSIM for each method based on our proposed dataset, to quantitatively evaluate and compare different methods. The results were recorded in best case, worst case, and average of all images for each method. The comparison values of CSIM and SSIM are shown in Table 3. From Table 3, compared with other simulated stroke models, the B-BSMG exhibited the higher similarity with the real stroke images. According to the average of CSIM and SSIM, B-BSMG > BBR > Straight Line. This fully shows that our proposed B-BSMG based on B-BSM outperformed the others significantly.

Table 3 The performance (%) of different approaches for reinforcement learning methods
Full size table

4.5.5 Comparison with Chinese Characters

To verify the robustness and efficacy of the proposed model, complete Chinese characters were formed based on the superimposition of a series of generated stroke graphics B-BSMG along the trajectory. Thus, for any Chinese character, we can control the robot to write with brush pen. A simulation image can be generated by a series of stroke graphics in the trajectory points and a real image can be obtained through the robotic writing. Take Chinese character “zi” for example, Fig. 17 shows the reference “yanti” image of “zi”, the simulated image of B-BSMG and real character written by robotic arm. For Chinese character “zi”, more than 100 writing tests were used, and the quantitative evaluating results were recorded in Table 4.

Fig. 17
figure 17

Comparison of Chinese character “zi”: a shows the reference simulated images, b shows the simulated images, and c shows the image written by the robot

Full size image
Table 4 The performance (%) of generated image and the corresponding real image of Chinese character “zi”
Full size table

As shown in Table 4, both the generated images of B-BSMG and the robot writing images can obtain a good performance with the Chinese character “zi”. Our proposed brush stroke model generator based on B-BSM exhibited the highest similarity with the corresponding real image, with a maximum similarity of 96.83%. The average value of CSIM written by the robot was more than 90%, which proved that the parameters Hi and αi obtained from B-BSMG are consistent with expectations. This especially shows the robustness and generalization of B-BSMG with physical parameters of Chinese calligraphy robot.

4.5.6 Comparison with Different Brush Pens

To verify the B-BSMG method can adapt to different types and parameters of hairy brushes and robotic arms, we simulated and generated Chinese characters with brush pens of different specifications (pen1 and pen2). The length and the radius of the brush bundle of pen1 were 48 mm and 6 mm respectively, that is, L = 48 mm, R = 6 mm for pen 1. For pen 2, L = 32 mm, R = 4.5 mm. obviously, the pen 2 is smaller than the pen 1. With sampling and parameter optimization, the simulated images of B-BSMG under position and posture parameters of robotic arms were different using pens with different sizes. With these obtained parameters, the characters can be written by the robotic arm.

Take reference “kaiti” calligraphy of “qu” as an example, the hairy brush pen No.1 and No. 2 were used by the calligraphy robot to write the character “qu” respectively. Figures 18 and 19 show the actual writing result of the character “qu”. In terms of the structure, the Chinese characters written with the No. 1 brush pen and No. 2 brush pen are all the same as the reference Chinese characters in outline. From Figs. 18 and 19, the generated graphics in B-BSMG are consistent with the change of real stroke images. This demonstrates the B-BSMG method can adapt to different types and parameters of hairy brushes and robotic arms.

Fig. 18
figure 18

Writing result of character “qu” (No. 1 brush): a reference calligraphy image and b the actual writing image

Full size image
Fig. 19
figure 19

Writing result of character “qu” (No. 2 brush): a reference calligraphy image and b the actual writing image

Full size image

To verify the generalization and robustness of the calligraphy robot under the condition of different hairy brush pens, the No. 1 brush pen and No. 2 brush pen were, respectively, used to conduct 100 writing experiments, and the writing results were quantitatively evaluated based on the CSIM and SSIM, as shown in Table 5.

Table 5 The Performance (%) of different brush pens and the corresponding real image of Chinese character “qu”
Full size table

5 Conclusions

In this study, we introduced a neural network generator called B-BSMG that simulates brush strokes and is differentiable. To generate a large number of simulated images for training purposes, this paper first defined the mathematical relationship between the multivariate parameters of the real brush stroke samples and the simulated stroke models. With this information, the B-BSMG can train the agent to generate corresponding brush stroke images based on the curves or geometries of the simulated stroke model.

Simulated brush stroke images provide a source of training datasets and serve as a source of training datasets for deep learning. Thus, we can obtain a large number of images similar to the reference images effectively and rapidly. Using the B-BSMG neural generator, the input graphic can be converted into a series of brush strokes based on Bézier curves. We train stroke-based neural network to generate brushstrokes and images that can be given by the parametric representation. The network can follow human-like strokes and be implemented using these differentiable neural brush strokes. The network’s convergence speed is faster compared to other generation models based on unordered pixels. Our renderer was based on parameterized brush strokes, which offers a better foundation for painting and writing in deep learning or other areas of artistic creation. This work also has the potential to modify existing brush strokes further and improve their effectiveness.

As an important application of intelligent robot, robotic Chinese calligraphy can provide helpful ideas for solving other similar problems, which will promote the further research and development of intelligent robot. The application of calligraphy robots is not only employed in writing, but also can be extended into other operation fields, such as robot welding, robot painting, robot spraying, and so on.