TTS-GAN: A Transformer-Based Time-Series Generative Adversarial Network

Abstract

Signal measurements appearing in the form of time series are one of the most common types of data used in medical machine learning applications. However, such datasets are often small, making the training of deep neural network architectures ineffective. For time-series, the suite of data augmentation tricks we can use to expand the size of the dataset is limited by the need to maintain the basic properties of the signal. Data generated by a Generative Adversarial Network (GAN) can be utilized as another data augmentation tool. RNN-based GANs suffer from the fact that they cannot effectively model long sequences of data points with irregular temporal relations. To tackle these problems, we introduce TTS-GAN, a transformer-based GAN which can successfully generate realistic synthetic time-series data sequences of arbitrary length, similar to the real ones. Both the generator and discriminator networks of the GAN model are built using a pure transformer encoder architecture. We use visualizations and dimensionality reduction techniques to demonstrate the similarity of real and generated time-series data. We also compare the quality of our generated data with the best existing alternative, which is an RNN-based time-series GAN.

TTS-GAN source code: github.com/imics-lab/tts-gan

Appendices

A Appendix 1: Training Details

We conduct all experiments on an Intel server with a 3.40 GHz CPU, 377 GB RAM memory and 2 Nvidia 1080 GPUs. For all datasets, the synthetic data are generated by a generator that takes random vectors of size (100, 1) as inputs. The transformer blocks in the generator and discriminator are both repeated three times. We adopt a learning rate of $1e-4$ for the generator and $3e-4$ for the discriminator. We follow the setting of LSGAN [23] and use loss function described in Sect. 3.3 to update model parameters. An Adam optimizer with $\beta _1 = 0.9$ and $\beta _2 = 0.999$, and a batch size of 32 for both generator and discriminator, are used for all experiments.

B Appendix 2: Similarity Scores

Feature Extraction. We extract several meaningful features from each input data sequence. They are the median, mean, standard deviation, variance, root mean square, maximum, and minimum values of each input sequence. Suppose we compute m features from all channels of each sequence and get a feature vector with the format $f = {<}feature_1, feature_2, ..., feature_m{>}$.

Average Cosine Similarity. For each pair of real signal feature vector $f_a$ and synthetic signal feature vector $f_b$, the vector has the size m, we can compute its cosine similarity as:

$$\begin{aligned} cos\_sim_{ab} = \frac{f_a\cdot f_b}{\left\| f_a \right\| \left\| f_b \right\| } = \frac{\sum _{i=1}^{m}f_{ai} f_{bi}}{\sqrt{\sum _{i=1}^{m}f_{ai}^{2}}{\sqrt{\sum _{i=1}^{m}f_{bi}^{2}}}} \end{aligned}$$

The average cosine similarity score is the average of each cosine similarity between pairs of feature vectors corresponding to real and synthetic signals of the same class. The average cosine similarity is computed as follows, where n the total number of signals:

$$\begin{aligned} avg\_cos\_sim = \frac{1}{n} \sum _{i=1}^{n}cos\_sim_i \end{aligned}$$

Average Jensen-Shannon Distance. The average jensen-shannon distance is the average of jensen-shannon distance between each feature from real signals and synthetic signals. For each pair of real signal feature $f_{i\_real}$ and synthetic signal feature $f_{i\_syn}$, we can compute its jensen-shannon distance as:

$$\begin{aligned} jen\_sim_i = \sqrt{\frac{D(f_{i\_real} || m)+D(f_{i\_syn} || m)}{2}} \end{aligned}$$

where m is the pointwise mean of $f_{i\_real}$ and $f_{i\_syn}$ and D is the Kullback-Leibler divergence. The average jensens-shannon distance is computed as:

$$\begin{aligned} avg\_jen\_dis = \sum _{i=1}^{m}jen\_sim_i \end{aligned}$$