Tracking droplets in soft granular flows with deep learning techniques

The state-of-the-art deep learning-based object recognition YOLO algorithm and object tracking DeepSORT algorithm are combined to analyze digital images from fluid dynamic simulations of multi-core emulsions and soft flowing crystals and to track moving droplets within these complex flows. The YOLO network was trained to recognize the droplets with synthetically prepared data, thereby bypassing the labor-intensive data acquisition process. In both applications, the trained YOLO + DeepSORT procedure performs with high accuracy on the real data from the fluid simulations, with low error levels in the inferred trajectories of the droplets and independently computed ground truth. Moreover, using commonly used desktop GPUs, the developed application is capable of analyzing data at speeds that exceed the typical image acquisition rates of digital cameras (30 fps), opening the interesting prospect of realizing a low-cost and practical tool to study systems with many moving objects, mostly but not exclusively, biological ones. Besides its practical applications, the procedure presented here marks the first step towards the automatic extraction of effective equations of motion of many-body soft flowing systems.


Introduction
Over the last decade, machine learning has taken the scientific world by storm and even more so the commercial one. Even though it is not all old gold that glitters [1], the potential of machine learning to gain new insights into a variety of complex systems in science and society is unquestionably tantalizing [2,3].
Deep learning is a sub-field of machine learning specifically inspired by the structure and function of the human brain [4]. Its core consists of artificial neural networks that imitate the operation of the human brain in data processing and decision making. In the actual digital era, which generates an unprecedented amount of data (Big Data) in the form of images, text, and audio-video clips, deep learning techniques have proved capable of extracting relevant information and correlations [4], sometimes dramatically abating the amount of time required a e-mail: mihirdurve@gmail.com (corresponding author)

Lattice Boltzmann method and simulations
We employed two variants of the Lattice Boltzmann approach for the multicomponent flows simulations, namely the color-gradient approach with near-contact interactions and the free energy model for multicomponent fluids. In Appendices A and B, we briefly outline the two models, while in the following sections, we report the simulation details for the two physical systems under consideration.

Simulation details for translocation of a soft granular material within narrow channels
This system is studied using the color gradient LB. The simulation setup consists of a twodimensional microfluidic channel made of an inlet reservoir followed by a thinner channel connected to a further downstream reservoir. The height of the chambers is h = 600 lattice units and the one of the constriction is h s = 120 lattice units while its length is l s = 240 lattice units. We impose a bounce-back rule for the distribution functions at the walls, while at the outlet, we employ absorbing (zero gradient) [30].
The soft granular material is formed by droplets (component A, white region in Fig. 1) immersed within an inter-droplet continuous phase (component B, blue lines) surrounded by an external bulk phase (component C, black region outside the emulsion). Its structure closely resembles that of a high internal phase double emulsion with multi-core morphology [31,32]. In Fig. 1a we show a snapshot of the domain with the relevant dimension of the geometrical setup and in Fig. 1b a translocation sequence of the soft material made of N d = 49 internal droplets occupying an area fraction φ 0.9.
In our numerical experiment, a uniform velocity profile of speed U in = 2×10 −3 (in simulation units) is imposed at the inlet, which pushes the emulsion within the narrow constriction. Finally, the value of the velocity ensures that Capillary and Reynolds numbers remain well within the typical range of microfluidic experiments (Ca ∼ O(10 −3 ) and Re ∼ O(1)).
2.2 Simulation details for a flow of a multi-core emulsions within a microchannel.
Here we report the numerical details of the multi-core emulsion simulated using the freeenergy LB. The droplet is set between two flat walls placed at distance L y , where no-slip conditions hold for the velocity field v and neutral wetting for the fluid density ψ. The former means that the velocity is zero at the boundaries, while the latter that no mass flux crosses the walls where droplet interfaces are perpendicular.
In Fig. 2 we show two examples of multi-core emulsions whose design is inspired by experimental realizations. They are made of monodisperse immiscible droplets arranged in a symmetric configuration. In Fig. 2a, three cores (white) of diameter D i = 30 lattice sites are accommodated within a larger drop (black) of diameter D O = 100 lattice sites, in turn, surrounded by a further fluid (white). An analogous setup is used for the four-core emulsion (see Fig. 2b). Starting from these configurations, the emulsions are then driven out of equilibrium by applying a symmetric shear flow, where the top wall moves rightwards (along the positive x axis) at constant speed v w and the bottom wall in the opposite direction at speed −v w . This sets a shear rate ofγ = 2L y /v w . In previous works [33,34], it has been shown that, once the shear is turned on, the external shell elongates and aligns along the direction imposed by the flow, while the internal cores acquire a periodic rotational motion triggered by a fluid vortex formed within the emulsion. These dynamics persist over long periods of time provided that the shear is on. In Fig. 2c, d we show two instantaneous configurations of a three and a four-core emulsion at late times, whereγ 10 −4 with v w = 0.01 and L y = 170 in simulation units.
This system approximately corresponds to a real multi-core emulsion with a shell of diameter 100μm and cores of diameter 30μm, having a surface tension ranging between 1 − 10mN/m, a viscosity 10 −1 Pa·s and a shear rates varying between 0.1 − 1/s. Like in the previous method, Capillary and Reynolds numbers here range between 0.1 and 1 as well, as in typical microfluidic experiments.
The output of the simulations of the physical systems described above are saved as video files and later given as input to the deep learning-based application for automatic extraction of the trajectories of the moving droplets. In the supplementary material [35], see video1.avi file for the simulation output of translocation of triple emulsions within narrow channels and video2.avi for a flow of multi-core emulsions within a microchannel.

Technique
In this work, we develop an application to extract trajectories of individual droplets simulated via LB methods by adapting two different algorithms, one for object detection and another one for object tracking. The first task consists of training an artificial neural network to analyze digital images and locate the objects of interest, which in this case are the droplets. Later, the second task is to track the located objects across multiple sequential frames to infer their trajectories. Figure 3 shows the typical steps involved to accomplish the object recognition task [36]. In the following section, we briefly describe the algorithm employed for the object recognition.
x y Fig. 2 a, b Initial configurations of a three and a four-core monodisperse emulsion. The diameter of the cores is D i = 30 lattice sites while that of the shell is D O = 100 lattice sites. c, d Examples of instantaneous configurations of a the multi-core emulsions subject to a symmetric shear flow. The shear rate isγ 10 −4 in simulation units. In both cases internal cores rotate periodically clockwise, following an almost elliptical path triggered by the fluid vortex confined within the shell. For more details see Ref. [33,34]. Here Ca 0.2 and Re 1.2

Fig. 3
Typical steps in developing the deep learning-based application for object detection. In the training phase, difference between prediction and truth is used to improve the accuracy of the network by updating the parameters of the network. At the inference stage, a real world data, in our case from LB simulation, is fed to the network to obtain the bounding boxes around the droplets Eur. Phys. J. Plus (2021) 136:864 You Only Look Once (YOLO) is a state-of-the-art algorithm that employs a single multilayer network for object identification and classification. The YOLO algorithm has shown remarkable accuracy with high processing speed in identifying and classifying objects from the COCO dataset, which consists of 80 types (classes) of everyday objects [23]. The building structure of the YOLO network consists of two main components, a feature extractor and a detector. When an image is given as an input to the YOLO network, the extractor (also called backbone) produces feature representations at different scales (see Fig. 4). Such representations are then passed to the detector that outputs the bounding boxes along with the confidence score and the classes of the detected objects. In this work, we use the YOLO algorithm to identify droplets in the digital images produced by LB simulations of two physical systems. We employ the YOLO-v3 network, which uses the Darknet-53 network as a backbone. The Darknet-53 is a deep network consisting of 53 layers, which has shown significant improvement in object detection over its predecessors YOLO-v2 and YOLO-v1, which employ backbone with fewer layers [24].
The YOLO algorithm achieves high speed of image analysis due to its operating principle, namely by treating object detection as a regression instead of a classification task in determining class probabilities to the bounding boxes. The operating procedure of the YOLOv3 algorithm is sketched in Fig. 5. All objects in the given images are located and classified simultaneously. To achieve that, the YOLO algorithm divides the input image into S × S grid cells. Each grid cell is responsible for detecting an object if the object's center is within the cell domain. Each grid cell predicts B bounding boxes with their confidence score for the object and C conditional class probabilities for the given object belonging to a specific class. This information is then combined to produce the final output as a single bounding box around the detected object along with the class of that object. The final output is then passed to the DeepSORT algorithm for tracking the droplets.

DeepSORT
DeepSORT is an algorithm that tracks detected objects between two successive frames [22]. By analyzing sequential frames, DeepSORT can construct trajectories of individual objects. It uses the Hungarian algorithm to distinguish the objects detected in two consecutive frames and assigns individual objects their unique identity. Kalman filtering is then used to predict the future position of the objects based on their current positions.
The YOLO and the DeepSORT algorithms together accomplish droplet recognition and tracking from the simulated systems. In the next section, we outline the training data acquisition process employed in this work and later describe the training process for the YOLO network for droplet detection. Operating principle of the YOLO algorithm. Initially, an input image is split into S × S grid. In the next stage, multiple bounding boxes are predicted along with a class probability map (blue color highlights cells hosting an object belonging to the "droplet" class) to give final bounding boxes and classes of objects

Training data
Before training a network, acquiring training data is a crucial part of an object detection network. Training data consists of several images of the objects to be detected along with information about their location and dimensions. Practically, acquiring the training data is a labor-intensive task involving gathering the images and manually marking the positions and dimensions of the objects. For example, gathering thousands of images of street view cameras and marking the location of the cars to train a network to identify cars in a video feed. In addition, the prospect of manual marking of the objects makes the training data susceptible to human errors.
We avoided the labor-intensive process by preparing a synthetic training data set. In this dataset, each image is a collage with few solid white circles, mimicking the droplets placed randomly on a dark uniform background. The solid white circles of different sizes and shapes are prepared with commonly used computer graphics software. A separate text file is generated to note the positions and dimensions of the randomly placed droplets. A Python script generates several thousand images and associated text files in a matter of few seconds. The script is provided in the supplementary material [35]. We prepared several types of training datasets, and the detailed description is given in Appendix C.
We wish to emphasize that the training data, prepared synthetically, was intended to mimic the snapshots of the LB simulation, and it is not generated by a physical process associated with LB simulations. However, it is noteworthy that the synthetic data preparation method described above bypasses the labor-intensive data acquisition part and minimizes human errors, thus making the development process of the droplet tracking application much quicker and more precise. This method can potentially be used in several other object detection applications. In the next section, we describe the YOLO network's training process to identify the droplets with the synthetic data.

Training
The training of the YOLO network is carried out through an iterative process. A subset of training data, called batch size, is passed through the network in each iteration, and then the network predicts the bounding boxes around the detected objects. A loss value is computed based on the difference between the output and the ground truth. The supplied label information is taken as the ground truth. The network parameters (weights and biases of the nodes) are updated to minimize the total loss and improve the accuracy of the network. For the YOLOv3 algorithm, the loss value, also called total loss, is the weighted sum of three separate contributions, regression loss, confidence loss, and classification loss [23]. In the training process, the weights of these contributions to the total loss are the same as proposed in Ref. [23] to optimize the mean average precision and maintain the model stability.
At regular intervals, i.e., after a few iterations, the accuracy of the network is assessed by computing mean average precision (mAP) [37,38]. The mean average precision is a good measure to know how well the trained network performs on a dataset that is different from the training dataset. Typically, a separate dataset, called validation dataset, is compiled to compute the mean average precision.
We trained two YOLO networks to identify the droplets from the LB simulations. We adapted the code for training the network from Ref. [39]. The first network, called YOLOtiny, is a lighter version of the full YOLO network consisting of fewer convolutional layers than the full YOLO network to trade training and inference speed for accuracy. For many practical purposes, the lighter version (YOLO-tiny) does perform reasonably well.
For training the YOLO-tiny network, we consolidated a total of 10000 images, consisting of an equal number of two different types of synthetic images as shown in Fig. 14a,b. The parameters set for the training are mentioned here [40]. The parameters of the network are updated at the end of every batch. A separate dataset of 1000 images, called validation data, was prepared from the same type of images to calculate the mean average precision. The total loss and mean average precision as the training progresses are shown in Fig. 6a. As the training progresses, the total loss decreases, and the mean average precision of the network increases, indicating that the network is gaining performance with the training, reaching a total loss value very close to zero at the end of the training. The mean average precision value is saturated to 0.985, indicating that the network is 98.5% accurate in detecting the droplets in broad terms. Page 9 of 22 864 The second network we trained is the full YOLO network, with Darknet-53 as a backbone. Ten thousand images similar to the image shown in Fig. 14c were used as the training data. Another 1000 images of the same type were reserved as the validation data, and the training parameters are listed in [41]. In this case as well, as the training progresses, the total loss decreases and mean average precision augments (as seen in Fig. 6b) saturating to 0.99, i.e. the network is 99% accurate in detecting the droplets. The trained model in the form of a binary file containing the model's parameters (called weights) is saved. The weight file will be used later to analyze the data from LB simulations, and it is provided for both networks, the YOLO-tiny and the YOLO, as supplementary material [35].
By training the YOLO network, we accomplish the first task of droplet recognition from the digital images, while the DeepSORT algorithm handles the second task of droplet tracking. In the next section, we analyze two LB simulations as case studies to test and use the developed application.

Results
After the model is trained with the desired accuracy in terms of mean average precision, it is ready to analyze the real-world data. From simulations or experiments, the real word images can be given as input to the trained network, and individual droplets' trajectories can be obtained. This process is also called inference.
We run inference on the data generated with two LB simulations to test our trained network as described in Sect. 2. For inference, only the droplets (white color mass) were plotted on a dark background. The output video was then fed to the YOLO + DeepSORT network to infer the trajectories of individual droplets. We adapted code with Tensorflow implementation to run inference from Ref.
[42]. Figure 7 shows a visual depiction of the object recognition by the YOLO network and tracking of the located objects by the DeepSORT network in four sequential images from the LB simulations. The bounding boxes predicted by the YOLO network are shown in blue color, and the unique ID assigned by the DeepSORT algorithm is written above each bounding box (see video3.avi).

Multi-core emulsions
It is then straightforward to extract trajectories of the individual droplets across sequential frames since we have dimensions of all the bounding boxes in all the frames with their unique ID. The center of mass of the individual droplets is approximated as the geometric center of its bounding box. Figure 8 shows the paths traced by the droplets.

Soft granular media
In the second case, we run inference on LB simulation of soft granular media. This time, we employed the YOLO network to identify the droplets since the trained YOLO-tiny network makes many mistakes in droplet recognition. This is likely due to the increased complexity of the system, which presents a densely packed configuration of droplets moving through a narrow channel significantly deviating from its initial shape.
The YOLO network, however, can locate the droplets with almost perfect accuracy. Here, we note again that the training of the YOLO network was carried out with the training data consisting of randomly placed dense clusters of solid white circles. As before, Fig. 9 shows In Fig. 10, we plot the extracted trajectories of the center of mass of the individual droplets. The center of the bounding box is approximated as the center of mass. In this case, we obtain 49 trajectories for the 49 droplets. Each separate trajectory is shown with a unique color.  To measure the accuracy of the trained model, we compute mean square errors (MSE) by a Euclidean metric: Here, N is the total number of frames, y cm is the ground truth y coordinate of the center of mass,ŷ cm is the inferred y coordinate of the center of mass by the deep learning tool.
We measure the accuracy of the YOLO-tiny + DeepSORT model by comparing the inferred trajectories of the droplets with the known ground truth. In this case, the ground truth is the center of mass computed with an analytical expression (see Ref. [34]). In Fig. 11, we compare the ground truth y-coordinates of the center of mass (y cm ) of individual droplets with the inferred y-coordinate values (ŷ cm ) by the deep learning tool. We observe that the deep learning model agrees with the analytical expression after a few initial frames. However, in the initial frames, the deep learning model slightly deviates from the true values, even though both the curves are in qualitative agreement. The mean square value M SE is of the order of 10 −2 for all four tracked droplets, indicating a good agreement between the ground truth and the inferred values.
For measuring the accuracy of the full YOLO + DeepSORT model, we prepared a synthetic test case where the center of mass of the droplets move along some predefined paths (see Appendix D). The predefined paths serve as the ground truth. We employed the YOLO + DeepSORT model to infer trajectories of the droplets from the video and the comparison is We emphasize that we measure the accuracy of the developed application, including the object recognition and the object tracking process. The error values are indicative of the total expected error in extracted trajectories from the real-world data. Apart from the model's accuracy, the model's analysis speed is an important factor to consider while developing an application for practical use. In the next section, we report the inference speed, i.e., the rate of image analysis by the developed application on different computer hardware configurations.

Inference speed
We test the inference speed of the object detection and tracking model on a typical notebook computer and a commonly used GPU machine. For inference, we used Tensorflow implementation [42]. The inference speed, analyzed frame per second (FPS), for the YOLO-tiny and the YOLO network are tabulated in Table 1.
These data clearly show that there is room for improvement and optimization of the inference speed for both YOLO and DeepSORT, for instance, by coding them in other programming languages, such as C++. This is an interesting topic for future work.
Having accurate trajectories of moving objects is the first step to study the system. In the next section, we present an analysis of the inferred trajectories to test a hypothesis.  At first sight, the droplets seem to move like the agents in active matter systems, such as birds in a flock. Several statistical physics models have been proposed to explain the flocking behavior in animal groups [43][44][45], and experimental studies involving observation of moving animal groups inferred the interaction rules between the agents [46,47]. One prominent feature common in most of these models is that the agents (birds) move along the average direction of their neighbors. Given the superficial similarity between a flock and the soft granular media system studied here, we perform a quick analysis to test a hypothesis that the droplets behave like agents in an active matter system. To this end, we calculate the average moving direction θ S of the neighbors of each individual droplet. The neighbors of droplet i are defined as droplets within a circle of radius R, fixed on the center of the droplet i. We set the size of the neighborhood in such a way that few neighboring droplets are included (see Fig. 13a). Figure 13b shows a scatter plot between individual droplet's moving direction θ i and the average direction of their neighbors θ S for the entire duration of the simulation. The moving direction is measured as angle θ made by the velocity vector with the x-axis in a fixed frame of reference. Thus, the range of θ is from −π to +π. Each different color in this plot shows a separate identity of the individual droplets. The red line is a guideline for the eyes to show the situation when the individual droplet's moving direction is identical to that of its neighbors.
In Fig. 13, around 47% of all the total points are in the band when |θ i − θ S | < 0.12. i.e., when the angle between the droplet's motion and its neighbors is less than 7 degrees. This suggests that the moving direction of individual droplet deviates (by more than 7 degrees) from the average direction of their neighbors 53% of the time.
We conclude that individual droplets frequently deviate from the average direction of their neighbors than the individual birds do in a polarized flock. This is plausible since, besides some similarities, there are also several points of departure between the two systems. First, in soft granular media systems, the fluid flow carries the droplets along the flow direction, like a systematic wind and second, the wall boundaries impose a stringent constraint on the motion of the droplets. None of the two aspects above are generally considered in the modeling of active matter systems.
In this analysis, we compared only the polarization between the system of droplets and the ordered flock. Numerous other observations characterize a flock [48], and a detailed comparison is required to quantify the similarities between these two systems. Nevertheless, this raises an interesting question on how to generalize the flock equations to come up with a set of effective equations of the droplets in complex many-body flows such as the one explored here. We plan to address this question in future work.

Conclusions
We adapted and combined deep learning-based object recognition algorithm YOLO and object tracking algorithm DeepSORT to analyze data generated by fluid simulations of two soft flowing systems, multicore emulsions, and soft granular media. In particular, the trajectories of the individual droplets moving within a microchannel flow were extracted from the digital images of the system.
We determine the accuracy of the complete application by comparing the inferred trajectories with the ground truth trajectories computed by independent methods. Although we used LB simulation data as a case study; the developed application can also be applied data generated by two-dimensional experimental setups.
With the use of commonly available GPUs, the YOLO + DeepSORT based application can analyze the images at a remarkable rate, above the image capture rate of typical cameras, thus offering a practical, low-cost application capable of analyzing real-time data as it is acquired.
From the technical point of view, we showed that the synthetically prepared datasets could be used to train the object detection network with almost perfect accuracy, thus avoiding laborintensive, time-consuming training data acquisition for object recognition network training.
We combined two state-of-the-art algorithms to identify and track the droplets and tested the model on two physical systems. It would be interesting to see how the application works for other dense emulsion systems. Nevertheless, YOLO and DeepSORT have shown remarkable performance in object recognition and tracking in more complex environments. We hope and expect that further developments of the work presented here may pave the way to the automatic detection and tracking of moving agents in complex flows of scientific, engineering, and biological interest.
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

A Color-gradient approach with near contact interactions
Here we describe the LB approach used to simulate the translocation through a constriction of a high internal phase double emulsion. We employ a color-gradient [49][50][51][52][53][54] regularized LB method [55][56][57], which is built starting from two sets of probability distribution functions capturing the dynamics of the immiscible fluid components. Each set evolves via a sequence of streaming and collision steps [30,[58][59][60] where f k i (x, t) is the i th discrete probability distribution function for the k th component, giving the probability of finding a fluid particle at position x, time t and with discrete velocity c i . The index k is such that k = 1, 2, while i belongs to the range 0 ≤ i ≤ N set , where N set is the dimension of the set of discrete probability distribution functions and is equal to 8 for the two-dimensional nine speeds lattice (D2Q9) employed in this paper. The time step Δt is expressed in lattice units [30] and set to 1, a usual choice in the LB methods [58]. Finally, F rep i is a force aimed at upscaling the repulsive near-contact forces acting on scales much smaller than the resolved ones [61,62].
As stated above, the multicomponent approach employed in this work is based on a variant of the color-gradient LB model. Following the standard formalism, the collision operator can be split into three parts The first term (Ω k i ) (1) is the usual single relaxation time Bhatnagar-Gross-Krook collisional operator [58] while the second part (Ω k i ) (2) is the perturbation step, which accounts for the interfacial tension, and reads as where A k (k = 1, 2) and B i (i = 0, N set ) are suitable constants defined in [61]. In Eq. (4), Θ is a scalar phase field, defined as assuming the value 1 in the fluid component with density ρ 1 and the value −1 in the fluid component with density ρ 2 . It is worth observing that the constants A 1 , A 2 are related to the surface tension σ by The third term (Ω k i ) (3) is the recolouring operator, as described in [63], which aims at minimizing the mutual diffusion between the fluid components, thus favouring their separation.
Finally, the last term at the right hand side of Eq. (2) codes for the short-range, repulsive force at the interface between the two fluid components, aimed at frustrating the coalescence between interacting, neighboring droplets.

B Free energy LB models for interacting multicomponent fluids
In this subsection we shortly outline the multicomponent field model for immiscible fluid mixtures employed to simulate the physics of three and four-core emulsions. Further details can be found in [33,34,[64][65][66].
In this approach, a set of scalar phase-fields ψ i (r, t), i = 1, ...., M (where M is the number of cores) is used to model the droplet density (positive within each drop and zero outside), while the average fluid velocity of both drops and solvent is described by a vector  field v(r, t).
The dynamics of the fields ψ i (r, t) is governed by a set of convection-diffusion equations where D is the mobility, μ i = δF δψ i is the chemical potential and F = V f dV is the total free energy describing the equilibrium properties of the fluid suspension. The free energy density f is given by [67,68] where the first term is the double-well potential ensuring the existence of two coexisting minima, ψ i = ψ 0 inside the ith droplet and ψ i = 0 outside, and the second term stabilizes the droplet interface. The parameters a and k are two positive constants related to the surface tension σ = √ 8ak/9 and the width of interface ξ = 2 √ 2k/a [30,69]. Finally, the last term in Eq. (8) is a soft-core repulsion contribution which penalizes the overlap of droplets, and whose strength is tuned by the positive constant .
The fluid velocity v obeys the Navier-Stokes equation, which in the incompressible limit reads Here ρ is the fluid density, p is the isotropic pressure and η is the dynamic viscosity. A typical set of values of thermodynamic parameters used in simulations is the following: a = 0.07, k = 0.1, D = 0.1, η = 1.67, = 0.05, Δx = 1 (lattice step), Δt = 1 (time step). Equations (7) are solved using a finite difference scheme while Eq. (9) is integrated by means of a standard lattice Boltzmann method. The latter shares many features with the LB model described in the previous section. Indeed, it is built from a set of distribution functions whose evolution is governed by a discrete Boltzmann equation akin to Eq. (2) and defined on a D2Q9 lattice. In both methods it can be shown that, once conservation of mass and momentum are fulfilled, the Navier-Stokes equation can be obtained by performing a Chapman-Enskog expansion of the distribution functions [58,70]. However, unlike the previous algorithm, in this approach the thermodynamics is encoded in a free-energy employed to calculate the forces (chemical potential and pressure tensor) controlling the relaxation dynamics of the mixture. In addition, this model allows for an immediate tracking of position and speed of the droplets by computing the coordinates of the center of mass.

C Training data
Training data is used to train the YOLO network to locate the droplets in given input images. Now we describe the training data acquisition employed in this work. Training data for the YOLO network must consist of several images of the objects of interest and their positions in the images. Training data is passed through the network, and the network parameters are updated with every pass to increase the detection accuracy. In a typical exercise to train a network, training data is compiled by acquiring images taken by digital cameras in the real world. Objects in those images are manually identified and marked with bounding boxes encapsulating each object. A separate label file containing the class of the object, positions (x,y coordinates), and dimensions (width and height) of the bounding boxes is prepared.
We avoided the labor-intensive part of image acquisition and manual labeling by preparing a synthetic training dataset. The training dataset must contain several images that bear some visual features of the expected outcome of the LB simulations, as shown in Figs. 1 and 2. In our case, the outcome of the LB simulations are the images with the area occupied by the droplets shown by white color on a high contrast dark background. We prepared our training dataset by randomly placing solid white circles on a dark uniform background. The solid white circles were intended to resemble the droplets, and the high contrast background is intended for easy identification of the droplets from the background.
In the case of LB simulation of multi-core emulsions, the outcome consists of few elliptical shape droplets enclosed in a dark bag on a uniform white background. We incorporate these features in the training data set by preparing 10000 images each of two types of images. The first type of images (see a representative Fig. 14a) are prepared by placing the white solid circles and ellipses of randomly chosen semi-minor and semi-major axis on the background of size 1200 × 600 to teach the network to identify droplets of various sizes. In the second type of images (see Fig. 14b), two solid circles as droplets are placed randomly within a solid black circle. The position of the solid black circle is chosen randomly in the uniform white background. Fig. 14 a Randomly placed droplets on a black uniform background of size 1200 × 600. b Two droplets are randomly placed in a black envelope. The placement of the envelope is random within the white uniform background of size 1200 × 600 pixels. c Few random size clusters of droplets were placed randomly in a uniform background of size 1200 × 600. d Label information associated with each of the bounding boxes In the case of LB simulations of soft granular media, the most apparent feature is the dense droplets cluster. In this case, few clusters with the random number of solid white circles are placed in a uniform solid black background to compile training data. The training data consisted of 10000 such images (see Fig. 14c as an example).
The label information for every image is shown in Fig. 14d. Each bounding box encloses a single white droplet. In associated text files, the dimensions of the bounding boxes (x and y coordinates, width and height, and the class of object it encapsulates) are noted down. Images and the training data together form a training dataset for the YOLO network.

D Synthetic test case
To measure the accuracy of the developed YOLO + DeepSORT application, we prepared a test video. In this video, the center of mass of each droplet move along predefined trajectories (see video5.avi). The following equation gives the trajectory of individual droplets; x i (t + 1) = x i (t) + A, y i (t + 1) = y i (t) + B sin(kx i (t)).

(10)
Here, x i (t) and y i (t) are the coordinates of the i th droplet at time t. A, B and k are the suitable constants set as A = 5, B = 5 and k = 5.
In practice, we prepared several images and in each image we pasted 12 droplets at the location given by Eq. (10). Each droplet was placed with a random initial location, i.e. at pixel location given as (y i (t = 0),x i (t = 0)). These images were then sequentially stacked to prepare a video of continuously moving droplets on predefined paths. In Fig. 15 we show one instance of the droplet identification and tracking process. See video6.avi for the output of the developed application.