DualSmoke: Sketch-Based Smoke Illustration Design with Two-Stage Generative Model

The dynamic effects of smoke are impressive in illustration design, but it is a troublesome and challenging issue for common users to design the smoke effect without domain knowledge of fluid simulations. In this work, we propose DualSmoke, two stage global-to-local generation framework for the interactive smoke illustration design. For the global stage, the proposed approach utilizes fluid patterns to generate Lagrangian coherent structure from the user's hand-drawn sketches. For the local stage, the detailed flow patterns are obtained from the generated coherent structure. Finally, we apply the guiding force field to the smoke simulator to design the desired smoke illustration. To construct the training dataset, DualSmoke generates flow patterns using the finite-time Lyapunov exponents of the velocity fields. The synthetic sketch data is generated from the flow patterns by skeleton extraction. From our user study, it is verified that the proposed design interface can provide various smoke illustration designs with good user usability. Our code is available at: https://github.com/shasph/DualSmoke


Introduction
Nowadays, due to the rapid prevalence of social media and personal computing devices, it becomes an emerging and growing topic to help amateur users achieve highquality content creation.The dynamic effects of 2D illustration play an important role to increase the attraction of various media such as e-books, education materials, and art design.Especially, the design of complex flow design is difficult for common users without the specific domain knowledge and design skills, which are required in commercial design tools, such as PhotoShop and Blender.Previous works adopted kinetic textures [23] and elemental dynamics [43] to author the dynamical effects, and sketch-based approaches to design fluid motions [18,46].However, these approaches may fail to provide the turbulent details in flow motions, especially for smoke design.To solve this issue, we aim to provide an efficient design tool for smoke illustration design.
The conventional approaches for fluid control mostly focused on the shape constraints [29].However, a timeconsuming process and tedious parameter adjustments are required by users with expert knowledge due to the nonlinear flows.For the keyframe-based approach [40], the optimization process was designed to explore the appropriate external force, which was normally computationally heavy.In addition, it is not practical to adjust parameters and boundary conditions interactively for smoke simulations.To efficiently design global motions of the fluid, lowresolution simulations were used as input for controlling high-resolution simulation [27,32].Low-resolution simulations can be executed at low costs, so users can easily repeat the simulation multiple times to design the motions.However, designing operations of fluid motions are not still intuitive, because users need to adjust simulation parameters.For more intuitive design, paths or sketches were introduced as inputs to generate the fluid flows [24,25].We follow this idea and our smoke design system is based on sketching operations.
In this work, we propose DualSmoke, a two-stage generation framework with conditional generative adversarial networks (cGAN) for smoke illustration design to meet the users' design intention, as shown in Figure 1.For the phase of data training, the proposed framework utilizes Lagrangian coherent structure (LCS) for a given velocity field to represent the main flow in velocity fields, which is obtained as the ridge of the finite-time Lyapunov exponent (FTLE) field.We then extract the synthetic sketch from the LCS using a heat equation.For the inference phase, we can generate the LCS domain and the velocity field from the input of the user's freehand sketch.Then, we calculate the guiding force from the generated LCS and the velocity field and add the force into the smoke simulations.Note that we set the flow direction as upward in the simulations as common design scenarios.In order to evaluate the proposed system, we conduct an evaluation experiment to verify the system usability and subjective assessment.In the evaluation experiment, the participants were asked to perform tasks using the proposed system with and without design constraints.
The main contributions of this work are as follows.
• We propose DualSmoke, a two-stage generative model for smoke design considering the flow patterns as structure information in the learning process.
• We constructed the paired training dataset with a pattern-guided smoke simulation where the FTLE and LCS fields are calculated to obtain the synthetic sketches.
• We develop a user-friendly design interface with the proposed system for various design scenarios and verify the proposed user interface with subjective assessment and user experience.

Sketch-based Content Creation
Freehand sketching is a naive and efficient way to describe objects and motions for content creation in computer graphics and user interface [4].For geometry modelling, sketch-based interfaces have been proposed for freeform modeling [21], terrain authoring [15], and character modeling [5].For physical simulations, sketch input has been adopted in crystal [30] and character animations [14].For more intuitive design, paths or sketches were introduced as inputs to generate the fluid flows [25,41].In this work, we follow this sketch-based idea and adopt the deep learning based approach to guide the smoke design with the rough sketch input to achieve high-quality flow details.
Recently, the deep learning based approaches have been explored largely in sketch-based content creation.Hu et al. [18] introduced an interactive user interface for fluid design and generated fluid simulations based on the velocity field obtained by cGAN.A similar approach has been used in terrain modeling [15].Yan et al. [44] used a 3D sketch in an immersive virtual reality environment to generate a splash representation of a liquid with cGAN.
For these previous works, it is difficult to achieve highquality design results.A recent work adopted volumetric density fields in 3D smoke reconstruction using CNN networks [24].However, the sketch input may require fine details of viewpoint and density information, rather than rough hand-drawn sketches.To address these issues, we adopt the global-to-local two-stage deep generative modeling for smoke design.A similar strategy has been utilized in facial image synthesis [20] and character motion planning [28].This work is the first trial to apply the strategy in smoke design with pattern-guided flow simulations.With the pre-trained generative models, the proposed deep learning based framework can achieve an interactive system without the expensive computation on the fluid simulations.

Control of Fluid and Smoke Simulations
For fluid simulation, a stable fluid solver [36] was proposed to be unconditionally stable and has been widely used in research and product communities.Since then, the solver has been improved by other researchers [10,34].However, the user needs to repeatedly adjust the parameters and run the simulation to generate the desired flow.
To address this problem, many fluid control methods have been proposed.Treuille et al. [40] used the quasi-Newton method to compute the optimal external forces to achieve user-specified keyframes.Rasmussen et al. [29] proposed a method to control fluid velocity through control particles.Dobashi et al. [9] controlled the cloud simulation to fit the cloud shape into an input curve.Simulation parameters are automatically adjusted according to the difference between the curve and the cloud shape.Wu et al. [41] introduced the pattern image and computed the driving force based on the image for locally controlling the fluid shape.Tang et al. [39] addressed the lack of parameter space constraints and high dimensionality in the optimization of control forces on high-resolution simulations.In these methods, users can specify the target shape of the fluid via input curves, images, or 3D meshes, but the flow motion cannot be designed intuitively.
To efficiently design the fluid flow, some researchers used velocity fields obtained from a low-resolution fluid simulation as an input.Nielsen et al. [27] proposed an optimization method to guide the high-resolution simulation according to the low-resolution input.Yuan et al. [45] proposed a method to control the flow of a high-resolution simulation based on the low-resolution flow by extracting the LCS from the low-resolution flow.Huang et al. [19] also proposed a control method using the low-resolution simulation as a preview.This method can control any properties including velocity fields as well as densities, temperatures, and so on.Sato et al. [32] formulated the optimization problem in stream function space to reduce the problem size of Nielsen's method [27].Our system also uses a velocity field and LCS to control the smoke simulation [45], but the velocity field and LCS are obtained via user sketches.

Deep Learning based Fluid Simulation
Deep learning based approaches can efficiently create fluid flows, and several methods have recently been proposed.Ladicky et al. [26] achieved a speedup of one order of magnitude compared to position-based simulations at the time by estimating the acceleration of each frame.Chu et al. [7] learned relations between low-and high-resolution velocity fields using CNN (Convolutional Neural Network) for enhancing details of the smoke flow.Xie et al. [42] used cGAN to generate high-resolution frames from a single frame of a coarse smoke simulation.Chu et al. [8] proposed a cGAN based model to obtain velocity fields from a single frame of a density field.In this study, two cGAN models are trained to generate the LCS with the input of the sketch and the velocity field with the input of the LCS.

System Overview
The objective of DualSmoke is to design the velocity field of smoke simulation from the input of user sketches, and obtain a designed smoke flow as system output.To achieve this goal, the proposed system framework of the interactive smoke design adopts a two-stage generative model with the generated LCS and VF fields as shown in Figure 2.
For the two-stage generative model, two generative model networks are adopted to learn the relationships between the LCS and the synthetic sketch data denoted as LCS Generator in Figure 2, and between the velocity field and the LCS denoted as Velocity Field (VF) Generator.After generating the velocity field and the LCS flow patterns from this model, we calculate the guiding force field from the generated LCS field and the velocity field from the input of the user sketch.By using the conventional fluid simulator, the smoke simulation can be generated with the guiding force fields in real time.Finally, we develop the design interface for the smoke simulation using the proposed framework.For the design interface, we adopt both smoke control lines and obstacle lines in the smoke design: the control lines rep-resent the user sketch in the fluid domain for the proposed generative models; the obstacle lines represent the rigid objects which are utilized in boundary conditions of the smoke simulation.

Training Preprocess
For the training preprocess, we collect the training dataset for the deep learning-based generative models.From a smoke simulation with the given velocity fields, we calculate the FTLE structure and then obtain the LCS fields using a Gaussian mixture model (GMM).Based on the obtained LCS fields, we calculate the synthetic sketches using the skeleton extraction method with a heat equation.

Smoke Simulation
For a smoke simulation at any timestep t, the materials are advected by the velocity field u and the pressure field p.Assuming that u and p at the initial time t = 0 are known, the variation of these quantities can be solved using Navier-Stokes equations, as follows: where ν is the kinematic viscosity, ρ is the flow density, and f is the external force.The solid wall boundary condition is adopted for the smoke flow around the obstacle domain in Equation 3, where n denotes the normal vector on the solid boundary.In this work, we used the semi-Lagrangian method [36] for advection term, and the Mac-Cormack method [33] to suppress the numerical diffusion on grids.

Finite-Time Lyapunov Exponent
The FTLE is calculated by tracing a particle placed at the computation point in the velocity field.We obtain the FTLE within the integration interval based on the change in the particle position at the end [35].The trajectory of a particle advanced from its initial position t 0 by an integration time T is defined by the following flow map Φ.
where p(t) is the position of the particle at time t and u is the velocity.For the flow map, the local deformations are computed using the Cauchy-Green deformation tensor as follows: where M * is the transpose of M and ∆ is a 2 × 2 matrix for a 2D flow in this work.The FTLE value is defined as a time-dependent scalar quantity using the largest eigenvalue of the matrix, λ max as follows.
where ln is the natural logarithm.Note that the FTLE fields are different according to the Lagrangian particles' movements.When T > 0, the particle moves forward and the flow is repulsive (Figure 3

GMM-based Lagrangian Coherent Structure
The LCS reveals the flow pattern is obtained by extracting the ridge of the maxima of the calculated FTLE field.Figure 4 shows the LCS extracted image using a Hessian matrix and threshold-based approaches [11,45].A certain threshold value may achieve good extraction results from FTLE fields.However, the extraction with the same threshold value caused artifacts in dataset construction because the optimal threshold values were varied for different FTLE fields.Therefore, we propose a GMM to obtain the optimal threshold value for FTLE fields in this study.
A GMM is used to extract the LCS from the computed FTLE field which classifies the given data into multiple normal distributions, as shown in Figure 4(f).In this work, the FTLE field is divided into two classifiers, and the larger mean values of each class are used as the threshold values.We applied a Gaussian filter before clustering due to  the noise generated in the integration calculation of FTLE fields.

Synthetic Sketch
Skeleton extraction is proposed to generate a synthetic sketch from the extracted LCS (Figure 5).The important issue in skeleton extraction is to preserve the topological structure of the input 2D shape and the geometric features of the original shape.To avoid artificial lines and the postprocess, such as pruning of the generated branches [2,38], we use the heat equation [12] to extract skeletons.We first place a heat source in the contour of the 2D LCS image and block its propagation outside the extraction region.Finally, we can extract the ridge of the temperature map obtained by proceeding with the thermal simulation.

Generative Model
For our two-stage generative model, we first discuss a two-stage strategy with paired training data.The network structures of both generative models are then described.

Two-Stage Strategy
For a conditional generative model with a deep learning approach, it is challenging to reveal the structural information in the training process.Especially for complex domains such as smoke simulations, there are detailed structures in the velocity fields such as turbulent details.To solve this issue, we explicitly adopt a two-stage strategy in the generation of a smoke simulation.The global features of LCS skeletons are generated from initial sketch inputs, and the local details of velocity fields are generated from the LCS fields.
Figure 6 shows the network structure of the proposed DualSmoke global-to-local strategy for generative models.In our training dataset, we collect the paired synthetic sketch and the LCS data for the LCS generator, and the LCS and velocity fields for the VF generator.The training data consists of the velocity fields generated by the smoke simulation, the LCS, and the synthetic sketches, as illustrated in Figure 7.The sketch data in the training data is obtained by extracting the skeleton from the LCS.

Network Structure
In this study, we use the cGAN-based generative structures [22] for LCS generator G LCS from hand-drawn sketches S, and VF generator G V F from the generated LCS field L (Figure 2).The generators are based on the U-Net structure with skip connections between the decoder and encoder networks [31].The discriminators D LCS and D V F where E (S,L) and E S denote the expected values over sketch S and generated LCS L fields, respectively.The loss function of the VF generator with the generated veloc-ity field V is given as follows. min where E (L,V ) and E L denote the expected values over LCS L and generated velocity field V fields, respectively.The deep-learning model was trained with a batch size of 64 and 100 epochs.The optimization algorithm used was Adam for both the generator and the discriminator, with a learning rate of 0.0002.

Smoke Design
For the runtime smoke design, we calculate the force fields with the generated results from the aforementioned generative model that are used in the conventional fluid solver for simulation control.In addition, we developed a design interface for the interactive smoke design with the proposed system.

Guiding Force Field
In the smoke simulations, the flow behavior can change significantly with the change in simulation resolution and the addition of turbulence.To solve this issue, we adopted the guided smoke simulation using the velocity and LCS fields generated from the generative models [45].In the Navier-Stokes equations, we represent the external force f in Equation 2 as follows: where the first item is the buoyancy force, z = (0, 1) and α is a positive constant; α = 0.025 was used for most cases in this work.The second item F(p) denotes the guiding force, which can be calculated from the generated LCS and velocity field.The region of the velocity field to be guided is determined based on LCS region Ω as force guidance.The guiding force F(p) at point p is calculated as follows.
where u G is the generated velocity field and u S is the velocity field in the simulation.The constant c is a scaling parameter for the guiding force and indicates the degree to which the simulation follows the guiding force.Due to the trade-off between high-fidelity to freehand sketches and the naturalness of simulation results for sketch-based design, we leave the design freedom to the users to adjust c value (the default value is 1.0).When c is small, the simulation is less constrained by the velocity field and is more sensitive to external forces, such as buoyancy.However, as c becomes

User Interface
The design interface can support user drawing and control with interactive simulation feedback, as shown in Figure 9.Our interface provides the main canvas and three functional panels.For the drawing panel, a radio button is provided to toggle between smoke control (smoke) and an obstacle (obstacle) line.The control line is black and the obstacle is red.Note that the control line is used as the sketch input of our proposed two-stage generative model, and the solid obstacle is handled in the solid wall boundary condition in Equation 3.
For the parameter panel, we provide the scaling parameter modification to change the scales of the guiding force.For the operation panel, the operational functions are provided, such as backward, forward, and reset functions.The user can load the previously saved sketches for editing purposes and can start the smoke simulation by clicking the "Enter" button to confirm the designed results in real time.

Implementation Details
The proposed DualSmoke interface is implemented by PyQt, and the smoke simulation is driven by the mantaflow framework [1].The generative models were trained on RTX3090 and Intel Xeon W-2223@3.60GHz.In our system prototype, all smoke simulations were conducted on 256 × 256 grids.For the generation of the LCS and the velocity field using the pre-trained model, the generation time from user sketch to LCS was 8 ms, and the generation time from the LCS to the velocity field was about 3 ms.The data learning process spent around 6 hours for LCS generator, and 18 hours for VF generator.

Training Data
We adopted the grid-based solver for smoke simulations to obtain the velocity fields for training data.The semi-Lagrangian method was used for advection calculation.Simulations for dataset construction were run in an environment where the outflow position was determined by random x-axis and fixed y-axis coordinates, and a square area of a uniform randomly oriented velocity field was placed in the center.The velocity field and the locations of the sources made the flow in each simulation more random, and the data became diversified.The simulation interval is ∆t = 0.1, that is 1000 frames per 100 seconds.The velocity field used as training data was that of the 1000-frame point, and the FTLE was calculated as the backward motion from the 1000th frame, with an integration interval of T = 2.5, or 25 frames.For the standard deviation of the mixed Gaussian model, σ = 1.0 was used.The dataset consisted of 1374 paired training data and 344 paired test data.

LCS Calculation
To calculate the LCS from the given velocity field, we first compute the FTLE by tracing a Lagrangian particle placed with the velocity field during the integration interval T .For the 2D fluid domain in grid point (i, j), the four particles are placed at p lef t = (i−τ, j), p right = (i+τ, j), p down = (i, j − τ ), and p up = (i, j + τ ), where τ is the infinitesimal distance (in this work, we use 0.1 for a unit grid length of 1).The four particles are traced in the integration interval T using the velocity field of each frame.Equation 5is computed numerically, with the traced particles as p lef t , p right , p up , and p down , respectively.
(11) Finally, the FTLE is obtained by calculating Equation 5from the maximum eigenvalue in Equation 6.The fourth order Runge-Kutta method was used for tracing, and linear interpolation was used to interpolate the particle positions.The calculated FTLE fields are classified into two normal distributions by a GMM, and LCS is calculated based on the threshold values.

Ablation Study
To verify the proposed network structure, we compared the generative model with and without LCS flow patterns.Figure 10 shows the comparison results among our twostage approach, the one-stage generative model without LCS.For the network verification, we adopted the extracted skeleton from the ground truth as the sketch input, rather than the user sketches.Therefore, the sketch complexity has no apparent influence on the network structure.For the one-stage generative model, we trained the model with paired synthetic sketches and velocity fields in cGAN network.We found that the color distribution in our proposed approach is closer to ground truth, which indicates that the global flow features are captured well.For turbulent details, we calculated L1 and L2 distances and the cosine similarity of the normalized velocity fields in the one-stage model and our two-stage generative model, as shown in Table 1.Our method has smaller L1 and L2 distances of errors for whole and LCS domains.Regarding the cosine similarity, our approach achieved better performance in the LCS domain as indicated by the global features.Therefore, we verified that the proposed method can achieve better both global fluid patterns and local turbulent details than the one-stage model.

Application Design
The proposed DualSmoke can be applied in various applications and has various practical uses.Figure 11 shows a smoke illustration with the designed "SMOKE" text shapes.The scaling parameters used are c = 1.3 for the S-shape, c = 0.8 for the M, c = 1.8 for the O, c = 0.85 for the K, and c = 1.47 for the E. We can observe that the smoke may accumulate in the curved lines because the smoke can gain momentum due to the buoyancy and guiding force.Figure 12 shows the illustration design in which smoke from a chimney (here, c = 0.8) .It is difficult for a designer to control complex smoke performance.With the help of the proposed system, users can simply draw the smoke flow with a single stroke.The simulated result is physically correct with turbulent details, which cannot be achieved by previous sketch-based flow design [18,46].The proposed design framework enables the smoke generation in not only single stroke characters but also a variety of characters by designing the flow patterns.
The proposed approach can be utilized for education and demonstration purposes.For example, it is crucial to ensure the user safety in fire accidents with dense smoke.Figure 13 illustrates the smoke movement in fire to help the user seek the escape route in the building.With the help of the proposed design interface, the user can draw freely to control the smoke situation in real-time feedback.We believe the  proposed system would be useful for dynamic illustrations and smoke simulations in various applications.

System Evaluation
The main objective of the proposed system is to help users in designing complex smoke simulations using the design interface.To verify the effectiveness of the proposed system, we evaluate the developed user interface using the designed user study and system evaluation.

User Study
A user study was conducted to evaluate the system usability and user experience of the proposed system.The experiment was divided into the following Task A and Task B to measure the system usability and the design diversity.
For Task A, the participants were asked to watch a smoke simulation that was created in advance.This smoke simulation could be viewed at any time during the experiment.The participants were asked to create a similar smoke simulation.Figure 14 shows the sketch image and one frame image of the smoke simulation used in Task A. For Task B, the participants were asked to create a smoke simulation freely.
We asked 20 graduate students to join the user study with the diverse majors in computer graphics, industry design, human-computer interaction, biology and chemistry (10 males and 10 females, aged [20][21][22][23][24][25][26][27][28][29][30][31][32][33][34][35].The experimental procedure was organized as follows.We introduced the NASA-TLX tasks in 2 minutes because the participants were not familiar with the evaluation.We then explained the proposed system and allowed system practice for 5 minutes.After that, the participants completed Task A and Task B in turn with a time limit of 3-10 minutes for each task (20 minutes in total).The first 3 minutes were set as trialand-error time to prevent the participants from completing the task without engaging.After the task experiment, we conducted a questionnaire survey using the system usability tool called the System Usability Scale (SUS) and the subjective assessment tool called NASA-TLX [6,16].In addition, we collected user feedback as free comments in 5 minutes.

Evaluation Results
In our user study, we verified the proposed design interface with both system usability and workload assessments.sically, a sketch drawn from the bottom to the top can generate the smoke flows without the disturbed velocity field.

System Usability
The results of the SUS evaluation are shown in Figure 16, where the mean score is 80.5 (60 for min value, 95 for max).Because the average SUS score for a user interface was 76.2 [3].Therefore, this indicates that the proposed interface has excellent system usability.
For the SUS questionnaire, each question was answered using 5-point Likert scale (1 for strongly disagree, 5 for strongly agree).We had low scores for complexity, need technical support, cumbersomeness, and need prior knowledge.This indicated that the proposed interface is easy to use without the requirement of any expert knowledge.We had high scores for frequent use, ease of use, integrity, learnability, and confidence.This indicated that the users felt confident and comfortable using the proposed system.Note that we had a lower score for inconsistency which may be caused by the separated design and simulation windows.We plan to implement the integrated simulation window as part of future work.In terms of the user feedback, one participant reported that "I did not understand why the smoke control is necessary after drawing obstacle".This suggests that the naturalness of the smoke design may be related to both the environment design and smoke control.

Subjective Assessment
For the subjective NASA-TLX assessment, the highest rated item was mental demand as shown in Figure 17.Our user study involved the creation of a specified task and a free design task using the proposed system, and this may be the reason that the mental load of thinking was high.The physical demand and the temporal demand were rated as low, which indicates that the proposed sketch-based interface is easy to use.The average time costs for the two assigned tasks were around 6.1 minutes for Task A and 6 minutes for Task B.
In terms of the degree of performance and the effort required for the tasks, the weighted average score is not high due to the clarity of the proposed editing interface.Some participants commented that the degree of performance was related to the mental load.We thought that the interactive design interface allowed the participants to easily judge the quality of the drawing performance.Meanwhile, the low frustration level indicated the proposed interface is user friendly.

Conclusions
In this work, we proposed DualSmoke, a sketch-based design method for smoke illustration design with a twostage generative model.By extracting synthetic sketch data from flow patterns, the proposed system can generate the guiding force from the hand-drawn sketches for patternguided smoke simulation.The proposed method can allow common users without expert knowledge to edit fluid simulations easily.In ablation study, we compared the proposed approach with the one-stage generative model to verify the proposed network structure, and the proposed system can obtain better global flow patterns and local turbulent details in the generated velocity fields.Our user study and SUS evaluation verified the proposed editing interface has good system usability, and the NASA-TLX evaluation verified the workload is low.In the current system, the user sketches were limited to an upward flow direction.It is possible to design a flexible design system with a training dataset for flows in multiple directions.In terms of geometry learning, normal maps from 2D sketches can be generated with generative models [17,37].A similar approach can be used to estimate the geometry features of smoke simulations.This work calculated the guiding force from the generated results, and it is feasible to apply the guiding force fields to high-resolution simulation results [45].
Although this work aimed 2D smoke illustration design for common users without professional skills and domain knowledge, we consider that an extension to 3D simulations is a promising research direction for visual effects.With the training dataset for 3D smoke simulations, 3D FTLE [13] is a feasible solution, and 3D sketching would be useful for authoring this task, such as in an immersive virtual reality environment.

Figure 1 .
Figure 1.The proposed DualSmoke design framework utilizes fluid patterns to generate velocity field in the generative model.

Figure 2 .
Figure 2. The system framework of the proposed two-stage generative model.

Figure 3 .
Figure 3. Flow patterns with forward FTLE (b) and backward FTLE (c) with a velocity field (a).
(b)); when T < 0, the particle moves backward and the flow converges (Figure3(c)).In this work, we use backward FTLE to find the convergence point of the flow and extract the flow patterns.

Figure 5 .
Figure 5. Synthetic sketch generation using the proposed skeleton extraction.

Figure 6 .
Figure 6.Network structure of our proposed two-stage generative model.The training data includes velocity field, LCS, and the synthetic sketch on the left hand.There are two generators (GLCS and GV F ) with U-Net structure and two discriminators (DLCS and DV F ) with PatchGAN structure.The numbers on the left of blocks represent the spatial size of the network layer and the ones below layer blocks for the number of layers.

Figure 7 .
Figure 7. Examples of the paired training data used in our work.From left to right: the velocity field, the LCS, and the synthetic sketch.

Figure 8 .
Figure 8. Smoke simulation results with different scaling parameters.

Figure 9 .
Figure 9. User interface for interactive smoke design.

Figure 10 .
Figure 10.Comparison results for the one-stage generative model (2), our proposed framework (3), and the ground truth (4).Note that the sketch input (1) is the synthetic sketch extracted from the ground truth.The lines and colors indicate the streamline and magnitude of the velocity field, respectively.

Figure 11 .
Figure 11.Smoke illustration design of steam locomotive in "SMOKE" shapes.The bottom-right sub-figure is the input of user sketches.

Figure 12 .
Figure 12.Smoke design from a chimney from user sketches.The red lines in the bottom-right figure denote the obstacle as a chimney.The black line was designed for smoke control.

Figure 13 .
Figure 13.Fire simulation of smoke movement inside a building.The bottom-left figure illustrates the user sketches with red lines for the building and black lines for smoke control.

Figure 14 .
Figure 14.The assigned design task A in our user study.

Figure 15 .
Figure 15.The designed smoke simulations from user sketches (bottom right in each figure) in both simple and complex cases from our user study.

Table 1 .
Comparison results for one-stage and two-stage frameworks.