Genetic-algorithm-optimized neural networks for gravitational wave classification

Abstract

Gravitational-wave detection strategies are based on a signal analysis technique known as matched filtering. Despite the success of matched filtering, due to its computational cost, there has been recent interest in developing deep convolutional neural networks (CNNs) for signal detection. Designing these networks remains a challenge as most procedures adopt a trial and error strategy to set the hyperparameter values. We propose a new method for hyperparameter optimization based on genetic algorithms (GAs). We compare six different GA variants and explore different choices for the GA-optimized fitness score. We show that the GA can discover high-quality architectures when the initial hyperparameter seed values are far from a good solution as well as refining already good networks. For example, when starting from the architecture proposed by George and Huerta, the network optimized over the 20-dimensional hyperparameter space has 78% fewer trainable parameters while obtaining an 11% increase in accuracy for our test problem. Using genetic algorithm optimization to refine an existing network should be especially useful if the problem context (e.g., statistical properties of the noise, signal model, etc) changes and one needs to rebuild a network. In all of our experiments, we find the GA discovers significantly less complicated networks as compared to the seed network, suggesting it can be used to prune wasteful network structures. While we have restricted our attention to CNN classifiers, our GA hyperparameter optimization strategy can be applied within other machine learning settings.

Appendices

Appendix 1: Fourier transform and inner product conventions

We summarize our conventions, which vary somewhat in the literature. Given a time domain vector, ${\mathbf {a}}$, the discrete version of the Fourier transform of ${\mathbf {a}}$ evaluated at frequency $f_p = p/T$ is given by

$$\begin{aligned} {{\tilde{a}}}(f_p) = {\tilde{a}}[p]&= \Delta t \sum _{n=0}^{N-1} a(t_n) e^{-2 \pi i f_p n \Delta t} \\&= \Delta t \sum _{n=0}^{N-1} a(t_n) e^{-2 \pi i n \frac{p}{N}} , \end{aligned}$$

(21)

where $0 \le p \le N-1$. Notice that the zero frequency ($f_p=0$) corresponds to $p = 0$, positive frequencies ($0< f_p < f_s / 2$) to values in the range $0 < p \le N/2$, and negative frequencies ($- f_s / 2 \le f < 0$) correspond to values in the range $N/2< p < N$. This follows from the usual assumptions that the signal is both periodic in the observation duration, ${a}(t) = {a}(t \pm T)$, and compactly supported, ${\tilde{a}}(f) = 0$ for $|f| \ge f_s / 2$, where $f_s = 1 / \Delta t$ is the sampling rate and $f_s / 2$ is the Nyquist frequency. Consequently, the Fourier transformed signal is periodic in k with a period of N, ${\tilde{a}}(f_k) = {\tilde{a}}(f_k \pm N \Delta f)$. The value $p = N/2$ corresponds to the Fourier transform at the maximum resolvable frequencies, $-f_s/2$ and $f_s/2$, for a given choice of $\Delta t$.

Given the Fourier transformed data, ${\tilde{a}}$ and ${\tilde{b}}$, the noise-weighted inner product $\langle \cdot , \cdot \rangle$ between ${\tilde{a}}$ and ${\tilde{b}}$ is defined as

$$\begin{aligned} \langle a , b\rangle = 2 \Delta f \sum _{i=0}^{N-1} \frac{a(f_i) b^*(f_i)}{S_n(f_i)} \approx 2 \int _{-f_s /2}^{f_s /2} \frac{a(f) b^*(f)}{S_n(f)} \hbox {d}f . \end{aligned}$$

(22)

Notice that by convention the inner product is defined with an overall factor of 2, but unlike Eq. 6 the full set of positive and negative frequencies are used. The continuum limit ($\Delta f \rightarrow 0$) of the summation makes clear that this is a (discretized) inner product between a(f) and b(f) over the domain $|f| \le f_s /2$. Note that because the time-domain signal is real the Fourier transformed signal satisfies ${\tilde{a}}^*(f) = {\tilde{a}}(-f)$. As a result, the inner product expression can be “folded-over”

$$\begin{aligned} \langle a , b\rangle = 4 {\mathfrak {R}}\sum _{i=0}^{N/2-1} \frac{a(f_i) b^*(f_i)}{S_n(f_i)} \approx 4 {\mathfrak {R}}\int _{0}^{f_s /2} \frac{a(f) b^*(f)}{S_n(f)} \hbox {d}f , \end{aligned}$$

(23)

which now features an integral over the positive frequencies and shows the inner product to be manifestly real. We then arrive at Eq. 6. This motivates the use of the term “inner product” when discussing Eq. 6 despite the fact that when taken at face value it does not satisfy the usual properties of an inner product while Eq. (22) does. Finally, some authors set the noise at the Nyquist frequency to 0 (see, for example, Ref. [59] discussion after Eq. 7.1.) frequency.

Appendix 2: Derivation of conditional probabilities used in likelihood-ratio test

A derivation of the standard inner product used in gravitational-wave analyses can be found in Ref. [81], which makes use of methods laid out in Ref. [58]. Here, we provide a brief derivation to highlight some of the assumptions that go into the classical filter.

In the absence of a signal, we assume that the detector is a stochastic process that outputs Gaussian noise with zero mean. The likelihood that some observed output ${\mathbf {s}}$ is purely noise is therefore given by a N-dimensional multivariate normal distribution

$$\begin{aligned} p({\mathbf {s}}| n) = \frac{\exp \left[ -\frac{1}{2}{\mathbf {s}}^{\mathsf {T}}\varvec{\Sigma }^{-1} {\mathbf {s}}\right] }{\sqrt{(2\pi )^{N} \det \varvec{\Sigma }}}, \end{aligned}$$

(24)

where $\varvec{\Sigma }$ is the covariance matrix of the noise, and $\det \varvec{\Sigma }$ is its determinant.

It is also common to assume that the noise is wide-sense stationary and ergodic. This is generally true on the time scales that a gravitational-wave from a compact binary merger passes through the sensitive band of the detector ($\sim \max {\mathcal {O}}(100\,\mathrm {s})$). In that case, $\varvec{\Sigma }$ is a real symmetric Toeplitz matrix with elements

$$\begin{aligned} \varSigma [j, k] = \frac{1}{2} R_{ss}[k-j] \end{aligned}$$

where

$$\begin{aligned} R_{ss}[k] \equiv \lim _{n\rightarrow \infty } \frac{1}{n} \sum _{l=-n}^{n-1} s[l]s[l+k] \end{aligned}$$

(25)

is the autocorrelation function of the data.

There is no general, analytic solution for $\varvec{\Sigma }^{-1}$. However, if $R_{ss}\rightarrow 0$ in finite time $\tau _{\max }$ and the observation time $T > 2\tau _{\max }$ (i.e., $\lceil N/2 \rceil > \lceil \tau _{\max }/\Delta t \rceil$), then $\varvec{\Sigma }$ is nearly a circulant matrix; it only differs in the upper-right and lower-left corners. All circulant matrices, regardless of the values of their elements, have the same eigenvectors [82]

$$\begin{aligned} u_p[k] = \frac{1}{\sqrt{N}} e^{-2\pi i k p/N}. \end{aligned}$$

(26)

We make the approximation that $\varvec{\Sigma }$ is circulant and use these eigenvectors to solve the eigenvalue equation, yielding

$$\begin{aligned} \lambda _p = \frac{1}{2} {\mathfrak {R}}\left\{ \sum _{l=-N/2}^{N/2-1} R_{ss}[l] e^{-2\pi i p l /N} \right\} . \end{aligned}$$

(27)

(The ${\mathfrak {R}}$ arises because the covariance is real and symmetric.) The error in this approximation decreases with increasing observation time; indeed, the eigenvalues of $\varvec{\Sigma }$ asymptote to Eq. 27 as $N \rightarrow \infty$ [82]. The autocorrelation function of ground-based gravitational-wave detectors $\approx 0$ for $\tau > {\mathcal {O}}(10\,\mathrm {ms})$. Since the observation time for a gravitational wave is $>{\mathcal {O}}(\mathrm {s})$, this approximation is valid in practice.

We recognize Eq. 27 as $1/\Delta t$ times the real part of the discrete Fourier transform of $R_{ss}[p]$.² Therefore, via the Wiener–Khinchin theorem,

$$\begin{aligned} \lambda _p = \frac{S_n[p]}{2\Delta t} \end{aligned}$$

(28)

where $S_n[p]$ is the discrete approximation of the power spectral density (PSD) of the noise at frequency $p/T \equiv p \Delta f$. Since the matrix of eigenvectors ${\mathbf {U}}$ are unitary, we have

$$\begin{aligned} \varSigma ^{-1}[j, k]&\approx \left[ {\mathbf {U}}\varvec{\Lambda }^{-1} {\mathbf {U}}^\dagger \right] [j, k] \\&\approx \frac{2 \Delta t}{N} \sum _{p=0}^{N-1} \frac{e^{-2\pi i j p/N} e^{2\pi i k p/N}}{S_n[p]} \\&= c_{jk} + 4 \Delta f (\Delta t)^2 \sum _{p=1}^{N/2-1} \frac{\cos \left( 2\pi (j-k)p/N\right) }{S_n[p]}, \end{aligned}$$

(29)

To go from the second to the third line, we have substituted $1/N = \Delta f \Delta t$ and have made use of the fact that $S_n[p]$ is symmetric about N/2; $c_{jk}$ depends only on the $p=0$ and $p=N/2$ terms, which correspond to the DC and Nyquist frequencies, respectively.

Gravitational-wave detectors have peak sensitivity within a particular frequency band $[f_0, f_{\max }]$ (for current generation detectors, this is $f \sim [20, 2000]\,$Hz). Outside of this range we can effectively treat the PSD as being infinite, making all terms in Eq. (29) with $p < \lfloor f_0 / \Delta f \rfloor \equiv p_0$ zero. Likewise, if we choose a sample rate $1/\Delta t > 2 f_{\max }$, then the Nyquist term is also effectively zero. The exponential term in the likelihood is therefore

$$\begin{aligned} \left[ {\mathbf {s}}^\mathsf {T}\varvec{\Sigma }^{-1} {\mathbf {s}}\right]&\approx 4 \Delta f \sum _{p=p_0}^{N/2-1} (\Delta t)^2 \sum _{j,k=0}^{N-1} s[j]s[k]\frac{\cos \left( 2\pi (j-k)p/N\right) }{S_n[p]} \\&\approx 4 \Delta f \sum _{p=p_0}^{N/2-1} \frac{\left| {\tilde{s}}\right| ^2[p]}{S_n[p]}. \end{aligned}$$

In going from the first to the second line, we have again recognized the sums over j, k as the discrete Fourier transforms over the real time-series data. We can further simplify this by defining the inner product Eq. (6), yielding Eq. (5) for the likelihood.

Appendix 3: How to generate Gaussian noise

Somewhat surprisingly, we are unaware of a resource that describes how to implement Eq. (4) to generate time-domain noise realizations. When implementing this expression one encounters sufficiently many subtleties that we will summarize our recipe here.

Eq. (4) specifies the statistical properties satisfied by the Fourier coefficients of the noise. Note that in the literature similar expressions for the discrete Fourier transform coefficients are sometimes given, which differs from ours.

Since the frequency-domain noise, ${\tilde{n}}(f_i)$, is complex, we need to be careful when sampling the real and imaginary parts. For example, if the desired property is $\langle {\tilde{n}}^*(f_i) {\tilde{n}}(f_j) \rangle =\delta _{ij}$, then

$$\begin{aligned} {\mathfrak {R}}({\tilde{n}}(f_i)) \sim {\mathcal N}\left( 0,\frac{1}{2}\right) , \qquad {\mathfrak {I}}({\tilde{n}}(f_i)) \sim {\mathcal N}\left( 0,\frac{1}{2}\right) , \end{aligned}$$

(30)

which gives

$$\begin{aligned} \langle {\tilde{n}}^*(f_i) {\tilde{n}}(f_j) \rangle = \langle {\mathfrak {R}}({\tilde{n}}(f_i))^2 + {\mathfrak {I}}({\tilde{n}}(f_i))^2 \rangle = \frac{1}{2} + \frac{1}{2} = 1 . \end{aligned}$$

(31)

Furthermore, for real time-domain functions we have ${\tilde{n}}^*(f) = n(-f)$ and so only the non-negative frequencies are independently sampled. When $f=0$, this condition implies that n(0) is real, whence ${\tilde{n}}(0) \sim {\mathcal N}(0,1)$. A similar property holds at the Nyquist frequency.

The neural networks considered in this paper use time-domain data. Synthetic time-domain noise realizations are constructed by taking an inverse Fourier transform of our frequency domain noise. In the time-domain, Eq. (4) becomes,

$$\begin{aligned} \langle n(t_i) \rangle = 0 , \qquad \langle n^2(t_i) \rangle = \frac{\Delta f}{2} \sum _{i=0}^{N-1} S_n(f_i), \end{aligned}$$

(32)

which follows directly from Eq. (4) and properties of the Fourier transform. We found Eq. (32) to be an indispensable sanity test of our time-domain noise realizations.