# Learning Quantitative Sequence–Function Relationships from Massively Parallel Experiments

## Abstract

A fundamental aspect of biological information processing is the ubiquity of sequence–function relationships—functions that map the sequence of DNA, RNA, or protein to a biochemically relevant activity. Most sequence–function relationships in biology are quantitative, but only recently have experimental techniques for effectively measuring these relationships been developed. The advent of such “massively parallel” experiments presents an exciting opportunity for the concepts and methods of statistical physics to inform the study of biological systems. After reviewing these recent experimental advances, we focus on the problem of how to infer parametric models of sequence–function relationships from the data produced by these experiments. Specifically, we retrace and extend recent theoretical work showing that inference based on mutual information, not the standard likelihood-based approach, is often necessary for accurately learning the parameters of these models. Closely connected with this result is the emergence of “diffeomorphic modes”—directions in parameter space that are far less constrained by data than likelihood-based inference would suggest. Analogous to Goldstone modes in physics, diffeomorphic modes arise from an arbitrarily broken symmetry of the inference problem. An analytically tractable model of a massively parallel experiment is then described, providing an explicit demonstration of these fundamental aspects of statistical inference. This paper concludes with an outlook on the theoretical and computational challenges currently facing studies of quantitative sequence–function relationships.

### Keywords

Sequence–function relationships Mutual information Likelihood Diffeomorphic modes Sort-Seq## 1 Introduction

A major long-term goal in biology is to understand how biological function is encoded within the sequences of DNA, RNA, and protein. The canonical success story in this effort is the genetic code: given an arbitrary sequence of messenger RNA, the genetic code allows us to predict with near certainty what peptide sequence will result. There are many other biological codes we would like to learn as well. How does the DNA sequence of a promoter or enhancer encode transcriptional regulatory programs? How does the sequence of pre-mRNA govern which exons are kept and which are removed from the final spliced mRNA? How does the peptide sequence of an antibody govern how strongly it binds to target antigens?

A major difference between the genetic code and these other codes is that while the former is qualitative in nature, the latter are governed by sequence–function relationships that are inherently quantitative. Quantitative sequence–function relationships^{1} describe any function that maps the sequence of a biological heteropolymer to a biologically relevant activity (Fig. 1a). Perhaps the simplest example of such a relationship is how the affinity of a transcription factor protein for its DNA binding site depends on the DNA sequence of that site (Fig. 1b). Such relationships are a key component of the more complicated relationship between the DNA sequence of a promoter or enhancer (which typically binds multiple proteins) and the resulting rate of mRNA transcription (Fig. 1c). In both of these cases, the activities of interest (affinity or transcription rate) can vary over orders of magnitude and yet still be finely tuned by adjusting the corresponding sequence (binding site or promoter/enhancer). Similarly, other sequence–function relationships, like the inclusion of exons during mRNA splicing or the affinity of a protein for its ligand, are fundamentally quantitative.

Experimental methods for measuring sequence–function relationships have improved dramatically in recent years. In the mid 2000s, multiple “high-throughput” methods for measuring the DNA sequence specificity of transcription factors were developed; these methods include protein binding microarrays (PBMs) [2, 3], *Escherichia coli* one-hybrid technology (E1H) [4], and microfluidic platforms [5]. The subsequent development and dissemination of ultra-high-throughput DNA sequencing technologies then led, starting in 2009, to the creation of a number of “massively parallel” experimental techniques for probing a wide range of sequence–function relationships (Table 1). These massively parallel assays can readily measure the functional activity of \(10^3\) to \(10^8\) sequences in a single experiment by coupling standard bench-top techniques to ultra-high-throughput DNA sequencing.

Massively parallel experiments used for studying various sequence–function relationships

Sequence | Activity | System | Name | Publication |
---|---|---|---|---|

DNA binding sites | Protein–DNA binding affinity | Purified protein | Bind-n-Seq | Zykovich et al. [6] |

HT-SELEX | Zhao et al. [7] | |||

Jolma et al. [8] | ||||

EMSA-Seq | Wong et al. [9] | |||

SELEX-Seq | Slattery et al. [10] | |||

Promoter/enhancer DNA | Transcription rate | Purified protein | Patwardhan et al. [11] | |

Bacteria | Sort-Seq | Kinney et al. [12] | ||

Cell culture | MPRA | Melnikov et al. [1] | ||

Mouse liver | Patwardhan et al. [13] | |||

Yeast | Sharon et al. [14] | |||

Mouse retina | CRE-Seq | Kwasniesk et al. [15] | ||

Protein | Ligand binding | Phage display | DMS | Fowler et al. [16] |

Cellular growth rate | Yeast | EMPIRIC | Hietpas et al. [17] | |

Toxin activity | Bacteria | Adkar et al. [18] | ||

H1N1 binding | Yeast display | Whitehead et al. [19] | ||

GPCR expression | Bacteria | Schlinkmann et al. [20] | ||

RNA | mRNA translation | Bacteria | Holmqvist et al. [21] | |

sRNA targeting | Bacteria | qSortSeq | Peterman et al. [22] | |

mRNA translation | Cell culture | Oikonomou et al. [23] | ||

mRNA translation | Cell culture | FACS-Seq | Noderer et al. [24] | |

Replication origins | DNA replication | Yeast | ARS-Seq | Liachko et al. [25] |

Endonuclease sites | DNA cutting | Purified protein | Thyme et al. [26] |

The ability to fit parametric models to these data reflects subtle but important distinctions between two objective functions used for statistical inference: (i) likelihood, which requires *a priori* knowledge of the experimental noise function and (ii) mutual information [29], a quantity based on the concept of entropy, which does not require a noise function. In contrast to the conventional wisdom that more experimental measurements will improve the model inference task, the standard maximum likelihood approach will *typically* never learn the right model, even in the infinite data limit, if one uses an imperfect model of experimental noise. Model inference based on mutual information does not suffer from this ailment.

Mutual-information-based inference is unable to pin down the values of model parameters along certain directions in parameter space known as “diffeomorphic modes” [28]. This inability is not a shortcoming of mutual information, but rather reflects a fundamental distinction between how diffeomorphic and nondiffeomorphic directions in parameter space are constrained by data. Analogous to the emergence of Goldstone modes in particle physics due to a specific yet arbitrary choice of phase, diffeomorphic modes arise from a somewhat arbitrary choice of the sequence-dependent activity that one wishes to model. Likelihood, in contrast to mutual information, is oblivious to the distinction between diffeomorphic and nondiffeomorphic modes.

We begin this paper by briefly reviewing a variety of massively parallel assays for probing quantitative sequence–function relationships. We then turn to the problem of learning parametric models of these relationships from the data that these experiments generate. After reviewing recent work on this problem [28], we extend this work in three ways. First, we show that “diffeomorphic modes” of the parametric activity model that one wishes to learn are “dual” to certain transformations of the corresponding model of experimental noise (the “noise function”). This duality reveals a symmetry of the inference problem, thereby establishing a close analogy with Goldstone modes. Next we compute and compare the Hessians of likelihood and mutual information. This comparison suggests an additional analogy between this inference problem and concepts in fluid mechanics. Finally, we work through an analytically tractable model of a massively parallel experiment of protein–DNA binding. This example explicitly illustrates the differences between likelihood- and mutual-information-based approaches to inference, as well as the emergence of diffeomorphic modes.

## 2 Massively Parallel Experiments Probing Sequence–Function Relationships

Some of the earliest massively parallel experiments were designed to measure the specificity of purified transcription factors for their DNA binding sites [6, 7, 8, 9, 10] (Fig. 2c). The library used in such studies consists of a fixed-length region of random DNA flanked by constant sequences used for PCR amplification. This library is mixed with the transcription factor of interest, after which protein–bound DNA is separated from unbound DNA, e.g., by running the protein–DNA mixture on a gel. Protein–bound DNA is then sequenced, along with the input library.

Using a library of random DNA to assay protein–DNA binding has the advantage that the same library can be used to study each protein. This is particularly useful when performing assays on many different proteins at once (e.g., [8, 35]). On the other hand, only a very small fraction of library sequences will be specifically bound by the protein of interest. Moreover, because proteins typically bind DNA in a non-specific manner, such experiments are often performed serially in order to achieve substantial enrichment.^{2}

The first massively parallel experiment to probe how multi-protein–DNA complexes regulate transcription in living cells was Sort-Seq [12] (Fig. 2d). The sequence library used in this experiment was generated by introducing randomly scattered mutations into a “wild type” sequence of interest, specifically, the 75 bp region of the promoter of the *lac* gene in *E. coli* depicted in Fig. 3a. A few million of these mutant promoters were cloned upstream of the green fluorescent protein (GFP) gene. Cells carrying these expression constructs were grown under conditions favorable to promoter activity and were then sorted into a small number of bins according to each cell’s measured fluorescence. This partitioning of cells was accomplished using fluorescence-activated cell sorting (FACS) [41], a method that can readily sort \({\sim }10^4\) cells per second. The mutant promoters within each sorted bin as well as within the input library were then sequenced, yielding measurements for \({\sim }2 \times 10^5\) variant promoter sequences. We note that advances in DNA sequencing have since made it possible to accumulate much more data, and it is no longer difficult to measure the activities of \({\sim }10^7\) different sequences in this manner.

Massively parallel experiments using mutagenized sequences provide data about sequence–function relationships within a localized region of sequence space centered on the wild type sequence of interest. Measuring these local relationships can provide a wealth of information about the functional mechanisms of the wild type sequence. For instance, the Sort-Seq data of [12] allowed the inference of an explicit biophysical model for how CRP and RNAP work together to regulate transcription at the *lac* promoter (Fig. 3b). In particular, the authors used their data to learn quantitative models for the in vivo sequence specificity of both CRP and RNAP. Model fitting also enabled measurement of the protein–protein interaction by which CRP is able to recruit RNAP and up-regulate transcription.

Mutagenized sequences have also been used extensively for “deep mutational scanning” experiments on proteins. In this context, selection experiments on mutagenized proteins allow one to identify protein domains critical for folding and function. A variety of deep mutational scanning experiments are described in [42].

## 3 Inference Using likelihood

*S*having a corresponding measurement

*M*. Due to experimental noise, repeated measurements of the same sequence

*S*can yield different values for

*M*. Our experiment therefore has the following probabilistic form:If we assume that the measurements for each sequence are independent, and if we have an explicit parametric form for

*p*(

*M*|

*S*), then we can learn the values of the parameters by maximizing the per-datum log likelihood,

*L*simply as the “likelihood.”

*M*of each sequence

*S*is a noisy readout of some underlying activity

*R*that is a deterministic function of that sequence. We call the function relating

*R*to

*S*the “activity model” and denote it using \(\theta (S)\). This activity model is ultimately what we want to understand. The specific way the activity

*R*is read out by measurements

*M*is then specified by a conditional probability distribution, \(\pi (M|R)\), which we call the “noise function.”

^{3}Our experiment is thus represented by the Markov chainThe corresponding likelihood is

Standard statistical regression requires that the noise function \(\pi \) be specified up-front. \(\pi \) can be learned either by performing separate calibration experiments, or by assuming a functional form based on an educated guess. This can be problematic, however. Consider inference in the large data limit, \(N \rightarrow \infty \), which is illustrated in Fig. 4. Likelihood is determined by both the model \(\theta \) and the noise function \(\pi \) (Fig. 4a). If we know the correct noise function \(\pi ^{*}\) exactly, then maximizing \(L(\theta ,\pi ^{*})\) over \(\theta \) is guaranteed to recover the correct model \(\theta ^{*}\). However, if we assume an incorrect noise function \(\pi '\), maximizing likelihood will typically recover an incorrect model \(\theta '\) (Fig. 4b).

## 4 Inference Using Mutual Information

*R*. Denote the true model \(\theta ^{*}\) and the corresponding true activity \(R^{*}\). The dependence between

*S*,

*M*, \(R^{*}\), and

*R*will then form a Markov chain,From the simple fact that

*M*depends on

*R*only through the value of \(R^{*}\), any dependence measure \(\mathcal {D}\) that satisfies the data processing inequality (DPI) [29] must satisfy

^{4}For the massively parallel experiments such as those in Fig. 2,

*R*is continuous and

*M*is discrete. In these cases, mutual information is given by

*p*(

*M*,

*R*) is the joint distribution of activity predictions and measurements resulting from the model \(\theta \). If one is able to estimate

*p*(

*M*,

*R*) from a finite sample of data, mutual information can be used as an objective function for determining \(\theta \) without assuming any noise function \(\pi \).

It should be noted that there are multiple dependence measures \(\mathcal {D}\) that satisfy DPI. One might wonder whether maximizing multiple different dependence measures would improve on the optimization of mutual information alone. The answer is not so simple. In [28] it was shown that if the correct model \(\theta ^{*}\) is within the space of models under consideration, then, in the large data limit, maximizing mutual information is equivalent to simultaneously maximizing every dependence measure that satisfies DPI. On the other hand, one rarely has any assurance that the correct model \(\theta ^{*}\) is within the space of parameterized models one is considering. In this case, considering different DPI-satisfying measures might provide a test for whether \(\theta ^{*}\) is noticeably outside the space of parameterized models. To our knowledge, this potential approach to the model selection problem has yet to be demonstrated.

## 5 Relationship Between Likelihood and Mutual Information

A third inference approach is to admit that we do not know the noise function \(\pi \) a priori, and to fit *both*\(\theta \) and \(\pi \) simultaneously by maximizing \(L(\theta , \pi )\) over this pair. It is easy to see why this makes sense: the division of the inference problem into first measuring \(\pi \), then learning \(\theta \) using that inferred \(\pi \), is somewhat artificial. The process that maps *S* to *M* is determined by both \(\theta \) and \(\pi \) and thus, from a probabilistic point of view, it makes sense to maximize likelihood over both of these quantities simultaneously.

*N*limit, maximizing likelihood over both \(\theta \) and \(\pi \) is equivalent to maximizing the mutual information between model predictions and measurements. Here we follow the argument given in [28]. In the large

*N*limit, likelihood can be written

*p*(

*M*|

*R*), and \(H[M] = - \sum _M p(M) \log p(M)\) is the entropy of the measurements, which does not depend on \(\theta \). To maximize \(L(\theta , \pi )\) it therefore suffices to maximize \(I(\theta )\) over \(\theta \) alone, then to set the noise function \(\pi (M|R)\) equal to the empirical noise function

*p*(

*M*|

*R*), which causes \(D(\theta ,\pi )\) to vanish.

Thus, when we are uncertain about the noise function \(\pi \), we need not despair. We can, if we like, simply learn \(\pi \) at the same time that we learn \(\theta \). We need not explicitly model \(\pi \) in order to do this; it suffices instead to maximize the mutual information \(I(\theta )\) over \(\theta \) alone.

## 6 Diffeomorphic Modes

### 6.1 Criterion for Diffeomorphic Modes

^{5}Consider an infinitesimal change in model parameters \(\theta \rightarrow \theta + d \theta \), where the components of \(d\theta \) are specified by

*S*. This transformation will preserve the rank order of

*R*-values only if

*dR*is the same for all sequences having the same value of

*R*. The change

*dR*must therefore be a function of

*R*and have no other dependence on

*S*. A diffeomorphic mode is a vector field \(v^\mathrm{dif}(\theta )\) that has this property at all points in parameter space. Specifically, a vector field \(v^\mathrm{dif}(\theta )\) is a diffeomorphic mode if and only if there is a function \(h(R,\theta )\) such that

### 6.2 Diffeomorphic Modes of Linear Models

*S*is a

*D*-dimensional vector and

*R*is an affine function of

*S*, i.e.

*S*and

*R*is linear in

*S*, the function \(h(R,\theta )\) must be linear in

*R*. Thus,

*h*must have the form

*a*component of \(v^\mathrm{dif}\) corresponds to adding a constant to

*R*while the

*b*component corresponds to multiplying

*R*by a constant.

Note that if we had instead chosen \(R = \sum _{i=1}^D \theta _i S_i\), i.e. left out the constant component \(\theta _0\), then there would be only one diffeomorphic mode, corresponding to multiplication of *R* by a constant. This fact will be used when we analyze the Gaussian selection model in Sect. 8.

### 6.3 Diffeomorphic Modes of a Biophysical Model of Transcriptional Regulation

*E. coli*

*lac*promoter (Fig. 3). This model was fit to Sort-Seq data in [12]. The form of this model is as follows. Let

*S*denote a \(4 \times D\) matrix representing a DNA sequence of length

*D*and having elements

*Q*of CRP to DNA was modeled in [12] as an “energy matrix”: each position in the DNA sequence was assumed to contribute additively to the overall energy. Specifically,

*P*of RNAP to DNA was modeled as

*R*resulting from these binding energies was assumed to be proportional to the occupancy of RNAP at its binding site. This transcription rate is given by

Because the binding sites for CRP and RNAP do not overlap, one can learn the parameters \(\theta _Q\) and \(\theta _P\) from data separately by independently maximizing *I*[*Q*; *M*] and *I*[*P*; *M*]. Doing this, however, leaves undetermined the overall scale of each energy matrix as well as the chemical potentials \(\theta _P^0\) and \(\theta _Q^0\). The reason is that the energy scale and chemical potential are diffeomorphic modes of energy matrix models and therefore cannot be inferred by maximizing mutual information.

However, if *Q* and *P* are inferred together by maximizing *I*[*R*; *M*] instead, one is now able to learn both energy matrices with a physically meaningful energy scale. The chemical potential of CRP, \(\theta _Q^0\), is also determined. The only parameters left unspecified are the chemical potential of RNA polymerase, \(\theta _P^0\), and the maximal transcription rate \(R_{\max }\). The reason for this is that in the formula for *R* in Eq. (24) the energies *P* and *Q* combine in a nonlinear way. This nonlinearity eliminates three of the four diffeomorphic modes of *P* and *Q*.^{6} See [28] for the derivation of this result.

### 6.4 Dual Modes of the Noise Function

^{7}For any sequence

*S*, this transformation induces a transformation of the model prediction

*S*.

For general choice of vector *v*, no function \(\tilde{v}\) will exist that satisfies Eq. (32). The reason is that \(\partial R/\partial \theta _i\) will typically depend on the sequence *S* independently of the value of *R*. In other words, for a fixed value of *M* and *R*, the left hand side of Eq. (32) will retain a dependence on *S*. The right hand side, however, cannot have such a dependence. The converse is also true: for general choice of the function \(\tilde{v}\), no vector *v* will exist such that Eq. (32) is satisfied for all sequences. This is evident from the simple fact that *v* is a finite dimensional vector while \(\tilde{v}\) is a function of the continuous quantity *R* and therefore has an infinite number of degrees of freedom.

*h*. Here we have added the superscript “dif” because this is precisely the definition of a diffeomorphic mode given in Eq. (16). In this case, the function \(\tilde{v}^\mathrm{dif}\) dual to this diffeomorphic mode \(v^\mathrm{dif}\) is seen to be

## 7 Error Bars from Likelihood, Mutual Information, and Noise-Averaged Likelihood

*N*. Specifically, we discuss the optimal parameters and corresponding error bars that are found by sampling \(\theta \) from posterior distributions of the form

- (a)
\(F(\theta ) = L(\theta ,\pi ^{*})\) is likelihood computed using the correct noise function \(\pi ^{*}\).

- (b)
\(F(\theta ) = L(\theta , \pi ')\) where \(\pi '\) differs from \(\pi ^{*}\) by a small but arbitrary error.

- (c)
\(F(\theta ) = L(\theta , \pi '')\) where \(\pi ''\) differs from \(\pi ^{*}\) by a small amount along a dual mode.

- (d)
\(F(\theta ) = I(\theta )\) is the mutual information between measurements and model predictions.

- (e)
\(F(\theta ) = L_\mathrm{na}(\theta )\) is the noise-averaged likelihood.

*p*(

*S*),

*p*(

*S*,

*M*),

*p*(

*S*|

*R*), and

*p*(

*S*|

*R*,

*M*), the empirical distributions obtained in the infinite data limit. \(\left\langle \cdot \right\rangle _\theta \) will indicate an average computed over parameter values \(\theta \) sampled from the posterior distribution \(p(\theta |\mathrm{data})\). Subscripts on \(\mathrm{cov}(\cdot )\) or \(\mathrm{var}(\cdot )\) should be interpreted analogously.

### 7.1 Likelihood

*N*. The posterior distribution \(p(\theta | \mathrm{data})\) will, in general, be maximized at some choice of parameters \(\theta ^o\) that deviates randomly from the correct parameters \(\theta ^{*}\). At large

*N*, \(p(\theta |\mathrm{data})\) will become sharply peaked about \(\theta ^o\) with a peak width governed by the Hessian of likelihood; specifically

*R*.

^{8}We thus see that, as long as the set of vectors \(\partial R/\partial \theta _i\) spans all directions in parameter space, the Hessian matrix \(H_{ij}\) will be nonsingular. Using \(F(\theta ) = L(\theta ,\pi ^{*})\) will therefore put constraints on all directions in parameter space, and these constraints will shrink with increasing data as \(N^{-1/2}\). This situation is illustrated in Fig. 7a.

*f*(

*M*,

*R*) and small parameter \(\epsilon \). It is readily shown (see Appendix 1) that the maximum likelihood parameters \(\theta '\) will deviate from \(\theta ^{*}\) by an amount

*N*and will therefore not shrink to zero in the large

*N*limit. Indeed, for any choice of \(\epsilon > 0\), there will always be an

*N*large enough such that this bias in \(\theta '\) dominates over the uncertainty due to finite sampling.

*H*is nonsingular, one can always find a vector

*w*such that the deviation of \(\theta '\) from \(\theta ^{*}\) in Eq. (42) points in any chosen direction of \(\theta \)-space. As long as the functions

*i*, a function

*f*can always be found that generates the vector

*w*in Eq. (42).

We therefore see that arbitrary errors in the noise function will bias the inference of model parameters in arbitrary directions. This fact presents a major concern for standard likelihood-based inference: if you assume an incorrect noise function \(\pi \), the parameters \(\theta \) that you then infer will, in general, be biased in an unpredictable way. Moreover, the magnitude of this bias will be directly proportional to the magnitude of the error in the log of your assumed noise function. This problem is illustrated in Fig. 7b.

### 7.2 Mutual Information

*v*in parameter space,

*v*, then

*v*must be a diffeomorphic mode. This is because

*J*(

*R*) is positive almost everywhere, the right hand side of Eq. (50) can vanish only if \(\sum _i v_i \frac{\partial R}{\partial \theta _i}\) does not differ between any two sequences that have the same

*R*value. There must therefore exist a function

*h*(

*R*) such that \(h(R) = \sum _i v_i \frac{\partial R}{\partial \theta _i}\) for all sequences

*S*. This is precisely the requirement in Eq. (16) that

*v*be a diffeomorphic mode.

However, except along diffeomorphic modes, we can generally expect that the constraints provided by likelihood and by mutual information will be of the same magnitude. This situation is illustrated in Fig. 7d. Indeed, in the next section we will see an explicit example where all nondiffeomorphic constraints imposed by mutual information are comparable to those imposed by likelihood.

Before proceeding, we note that the relationship between the Hessians of likelihood and mutual information suggests an analogy to fluid mechanics. Consider a trajectory in parameter space given by \(\theta _i(t) = t v_i\), where *t* is time and *v* is a velocity vector pointing in the direction of motion. This motion in parameter space will induce a motion in the prediction *R*(*t*) that the model provides for every sequence *S*. The set of sequences \(\left\{ S_n \right\} \) thus presents us with a dynamic cloud of “particles” moving about in *R*-space. At \(t = 0\), the quantity \({\langle {\dot{R}}^2 \rangle }_{S|R}\) will be proportional to the average kinetic energy of particles at location *R*. The quantity \({\langle {\dot{R}}\rangle }^2 _{S|R}\) will be proportional to the (per particle) kinetic energy of the bulk fluid element at *R*, a quantity that does not count energy due to thermal motion. In this way we see that \(-\sum _{i,j} H_{ij} v_i v_j\) is a weighted tally of total kinetic energy, whereas \(-\sum _{i,j} K_{ij} v_i v_j\) corresponds to a tally of internal thermal energy only, the kinetic energy of bulk motion having been subtracted out.

### 7.3 Noise-Averaged Likelihood

Noise-averaged likelihood provides constraints in between those of likelihood, computed using the correct noise function, and those of mutual information. This is illustrated in Fig. 7e. Whereas mutual information provides no constraints whatsoever on the diffeomorphic modes of \(\theta \), noise-averaged likelihood provides weak constraints in these directions. These soft constraints reflect the Hessian of \(\Delta (\theta )\) in Eq. (12). The constraints along diffeomorphic modes, however, have an upper bound on how tight they can become in the \(N \rightarrow \infty \) limit. This is because such constraints only reflect our prior \(p(\pi )\) on the noise function, not the information we glean from data.

## 8 Worked Example: Gaussian Selection

*S*, each of which is actually a

*D*-dimensional vector drawn from a Gaussian probability distribution

^{9}

*D*-dimensional vector defining the average sequence in the library. From this library we extract sequences into two bins, labeled \(M=0\) and \(M=1\). We fill the \(M=0\) bin with sequences sampled indiscriminately from the library. The \(M=1\) bin is filled with sequences sampled from this library with relative probability

*S*with a

*D*-dimensional vector \(\theta ^{*}\), i.e.,

*M*, along with \(N = N_0 + N_1\).

*R*:

*D*-dimensional vector we wish to infer. From the arguments above and in [28], it is readily seen that the magnitude of \(\theta \), i.e. \(|\theta |\), is the only diffeomorphic mode of the model: changing this parameter rescales

*R*, preserving rank order.

### 8.1 Bin-Specific Distributions

*p*(

*S*|

*M*) for each bin

*M*, as well as the conditional distribution

*p*(

*R*|

*M*) of model predictions. Because the sequences sampled for bin 0 are indiscriminately drawn from \(p_{\mathrm{lib}}\), we have

*M*, this distribution is defined as

### 8.2 Noise Function

*a*and

*b*are scalar parameters that we might or might not know

*a priori*. This, combined with the normalization requirement, \(\sum _M \pi (M|R) = 1\), gives

*b*is dual to the diffeomorphic mode \(|\theta |\), whereas the parameter

*a*is not dual to any diffeomorphic mode.

In the experimental setup used to motivate the Gaussian selection model, the parameter *a* is affected by many aspects of the experiment, including the concentration of the protein used in the binding assay, the efficiency of DNA extraction from the gel, and the relative amount of PCR amplification used for the bin 0 and bin 1 sequences. In practice, these aspects of the experiment are very hard to control, much less predict. From the results in the previous section, we can expect that if we assume a specific value for *a* and perform likelihood-based inference, inaccuracies in this value for *a* will distort our inferred model \(\theta \) in an unpredictable (i.e., nondiffeomorphic) way. We will, in fact, see that this is the case. The solution to this problem, of course, is to infer \(\theta \) alone by maximizing the mutual information \(I(\theta )\); in this case the values for *a* and *b* become irrelevant. Alternatively, one can place a prior on *a* and *b*, then maximize noise-averaged likelihood \(L_\mathrm{na}(\theta )\). We now analytically explore the consequences of these three approaches.

### 8.3 Likelihood

*L*becomes a function of \(\theta \),

*a*, and

*b*. Computing

*L*in the \(N \rightarrow \infty \) and \(\epsilon \rightarrow 0\) limits, we find that

*a*and for

*b*, then optimize \(L(\theta , a, b)\) over \(\theta \) alone by setting

*i*. By this criteria we find that the optimal model \(\theta ^o\) is given by a linear combination of \(\theta ^{*}\) and \(\mu \):

*c*is a scalar that solves the transcendental equation

*c*is determined only by the value of

*a*and not by the value of

*b*. Moreover, \(c = 1\) if and only if \(a = a^{*}\).

*b*. This finding follows the behavior described in Fig. 7c.

If \(a \ne a^{*}\), however, \(c \ne 1\). As a result, \(\theta ^o\) is a nontrivial linear combination of \(\theta ^{*}\) and \(\mu \), and will thus point in a different direction than \(\theta ^{*}\). This is true regardless of the value of *b*. This behavior is illustrated in Fig. 7b: errors in non-dual parameters of the noise function will typically lead to errors in nondiffeomorphic parameters of the activity model.

*N*is sufficiently large, the finite bias introduced into \(\theta ^o\) will cause \(\theta ^{*}\) to fall outside the inferred error bars.

### 8.4 Mutual Information

*N*. However, we find no constraint whatsoever on the component of \(\delta \theta \) parallel to \(\theta ^{*}\). These results are illustrated by Fig. 7d.

### 8.5 Noise-Averaged Likelihood

*a*and

*b*, i.e. \(p(\pi ) = p(a,b) = \mathcal {C}\) where \(\mathcal {C}\) is an infinitesimal constant. We find that

*N*limit that \(\delta \theta _\perp \) is constrained in the same way as if we had used mutual information. The noise function prior we have assumed further results in weak constraints on \(|\theta |\) that do not tighten as

*N*increases.

^{10}This is represented in Fig. 7e.

## 9 Discussion

The systematic study of quantitative sequence–function relationships in biology is just now becoming possible thanks to the development of a variety of massively parallel experiments. Concepts and methods from statistical physics are likely to prove valuable for understanding this basic class of biological phenomena as well as for learning sequence–function relationships from data.

In this paper we have discussed the problem of learning parametric models of sequence–function relationships from experiments having poorly characterized experimental noise. We have seen that standard likelihood-based inference, which requires an explicit model of experimental noise, will generally lead to incorrect model parameters due to errors in the assumed noise function. By contrast, mutual-information-based inference allows one to learn parametric models without having to assume any noise function at all. Mutual-information-based inference is unable to pin down the values of model parameters along diffeomorphic modes. This behavior reflects a fundamental difference between how diffeomorphic and nondiffeomorphic modes are constrained by data. Diffeomorphic modes arise from arbitrariness in the distinction between the activity model and the noise function. These findings were illustrated using an analytically tractable model for a massively parallel experiment.

The study of quantitative sequence–function relationships still presents many challenges, both theoretical and computational. One major practical difficulty with the mutual-information-based approach described here is accurately estimating mutual information from data. Although methods are available for doing this [44], it remains unclear whether any are accurate enough to enable computational sampling of the posterior distribution \(p(\theta |\mathrm{data}) \sim e^{NI(\theta )}\), as suggested here. Moreover, none of these estimation methods is regarded as definitive. We believe this lack of clarity regarding how to estimate mutual information reflects the fact that the density estimation problem itself has never been fully solved, even in one or two dimensions. We are hopeful, however, that field-theoretic methods for estimating probability densities [45, 46, 47] might help resolve the problem of mutual information.

The problem of model selection poses a major theoretical challenge. Ideally, one would like to explore a hierarchy of possible model classes when fitting parametric models to data. However, when considering effective models it is unclear how to move far beyond independent site models (e.g., energy matrices) due to the number of parameters growing exponentially with the length of the sequence. Moreover, when learning mechanistic models such as the model of the *lac* promoter featured in Fig. 3, it is unclear how to go about systematically testing different arrangements of binding sites, different protein–protein interactions, and so on. We emphasize that this model prioritization problem is fundamentally theoretical, not computational, and as of now there is little clarity on how to address this matter.

Finally, the geometric structure of sequence–function relationships presents an array of intriguing questions. For instance, very little is known (in any system) about how convex or glassy such landscapes in sequence space are, what their density of states looks like, etc.. Most of the biological and evolutionary implications of these aspects of sequence–function relationships also have yet to be worked out. We believe that the methods and ideas of statistical physics may lead to important insights into these questions in the near future.

## Footnotes

- 1.
These have also been called quantitative sequence-activity maps, or QSAMs [1].

- 2.
- 3.
We use the term “noise function” in order to be consistent with the terminology of [28] and to avoid deviating too much from the more standard terms “noise model” and “error model” used in the statistics and machine learning literature. We emphasize, however, that \(\pi \) defines much more than just the characteristics of experimental noise; \(\pi \) entirely specifies the relationship between measurements

*M*and the underlying activity*R*. Were it not for prior terminology, the term “measurement function” might be preferable to “noise function.” - 4.
See [43] for an extended discussion of mutual information as a measure of statistical association.

- 5.
Here, as throughout this paper, we restrict our attention to situations in which

*R*is a scalar. The case of vector-valued model predictions*R*is worked out in [28]. - 6.
The one additional diffeomorphic mode is created by the introduction of the parameter \(R_{\max }\).

- 7.
For the sake of clarity we suppress the \(\theta \)-dependence of \(v^\mathrm{dif}\), \(\tilde{v}^\mathrm{dif}\), and

*h*(*R*) in what follows. - 8.
In what follows we assume that \(J(R) > 0\) almost everywhere. This just reflects the assumption that our experiment actually does convey information about

*R*through the measurements*M*it provides. - 9.
For the sake of simplicity we set the covariance matrix of this distribution equal to the identity matrix. The more general case of a non-identity covariance matrix yields the same basic results. Also, we note that, while approximating discrete DNA sequences by continuous vectors might seem crude, it is only the marginal distributions

*p*(*R*|*M*) that matter for the inference problem. Most of the quantities*R*that one encounters in practice are computed by summing up contributions from a large number of different nucleotide positions. In such cases, the marginal distributions*p*(*R*|*M*) will often be nearly continuous and virtually indistinguishable from the marginal distributions one might obtain from a Gaussian sequence library. - 10.
In the case at hand, \(|\theta ^o|\) is pushed all the way to zero. This is an artifact of the simple flat prior

*p*(*a*,*b*). If we instead adopt a weak Gaussian prior on*b*, we can still carry out the computation of \(L_\mathrm{na}\) analytically, and in this case we find that \(|\theta ^o|\) is finite.

## Notes

### Acknowledgments

We would like to thank L. Peliti, O. Revoire, and T. Mora for organizing this special issue. This work was supported by the Simons Center for Quantitative Biology at Cold Spring Harbor Laboratory and the Starr Cancer Consortium (I7-A723).

### References

- 1.Melnikov, A., Murugan, A., Zhang, X., Tesileanu, T., Wang, L., Rogov, P., Feizi, S., Gnirke, A., Callan, C.G., Kinney, J.B., Kellis, M., Lander, E.S., Mikkelsen, T.S.: Systematic dissection and optimization of inducible enhancers in human cells using a massively parallel reporter assay. Nat. Biotechnol.
**30**(3), 271–277 (2012)CrossRefGoogle Scholar - 2.Mukherjee, S., Berger, M., Jona, G., Wang, X., Muzzey, D., Snyder, M., Young, R., Bulyk, M.: Rapid analysis of the DNA-binding specificities of transcription factors with DNA microarrays. Nat. Genet.
**36**(12), 1331–1339 (2004)CrossRefGoogle Scholar - 3.Berger, M., Philippakis, A., Qureshi, A., He, F., Estep, P., Bulyk, M.: Compact, universal DNA microarrays to comprehensively determine transcription-factor binding site specificities. Nat. Biotechnol.
**24**(11), 1429–1435 (2006)CrossRefGoogle Scholar - 4.Meng, X., Brodsky, M.H., Wolfe, S.A.: A bacterial one-hybrid system for determining the DNA-binding specificity of transcription factors. Nat. Biotechnol.
**23**(8), 988–994 (2005)CrossRefGoogle Scholar - 5.Maerkl, S., Quake, S.: A systems approach to measuring the binding energy landscapes of transcription factors. Science
**315**(5809), 233–237 (2007)CrossRefADSGoogle Scholar - 6.Zykovich, A., Korf, I., Segal, D.J.: Bind-n-Seq: high-throughput analysis of in vitro protein-DNA interactions using massively parallel sequencing. Nucleic Acids Res.
**37**(22), e151 (2009)CrossRefGoogle Scholar - 7.Zhao, Y., Granas, D., Stormo, G.D.: Inferring binding energies from selected binding sites. PLoS Comput. Biol.
**5**(12), e1000590 (2009)CrossRefMathSciNetADSGoogle Scholar - 8.Jolma, A., Kivioja, T., Toivonen, J., Cheng, L., Wei, G., Enge, M., Taipale, M., Vaquerizas, J.M., Yan, J., Sillanpää, M.J., Bonke, M., Palin, K., Talukder, S., Hughes, T.R., Luscombe, N.M., Ukkonen, E., Taipale, J.: Multiplexed massively parallel SELEX for characterization of human transcription factor binding specificities. Genome Res.
**20**(6), 861–873 (2010)CrossRefGoogle Scholar - 9.Wong, D., Teixeira, A., Oikonomopoulos, S., Humburg, P., Lone, I.N., Saliba, D., Siggers, T., Bulyk, M., Angelov, D., Dimitrov, S., Udalova, I.A., Ragoussis, J.: Extensive characterization of NF-\(\kappa \)B binding uncovers non-canonical motifs and advances the interpretation of genetic functional traits. Genome Biol.
**12**(7), R70 (2011)CrossRefGoogle Scholar - 10.Slattery, M., Riley, T., Liu, P., Abe, N., Gomez-Alcala, P., Dror, I., Zhou, T., Rohs, R., Honig, B., Bussemaker, H.J., Mann, R.S.: Cofactor binding evokes latent differences in DNA binding specificity between Hox proteins. Cell
**147**(6), 1270–1282 (2011)CrossRefGoogle Scholar - 11.Patwardhan, R.P., Lee, C., Litvin, O., Young, D.L., Pe’er, D., Shendure, J.: High-resolution analysis of DNA regulatory elements by synthetic saturation mutagenesis. Nat. Biotechnol.
**27**(12), 1173–1175 (2009)CrossRefGoogle Scholar - 12.Kinney, J.B., Murugan, A., Callan, C.G., Cox, E.C.: Using deep sequencing to characterize the biophysical mechanism of a transcriptional regulatory sequence. Proc. Natl. Acad. Sci. USA
**107**(20), 9158–9163 (2010)CrossRefADSGoogle Scholar - 13.Patwardhan, R.P., Hiatt, J.B., Witten, D.M., Kim, M.J., Smith, R.P., May, D., Lee, C., Andrie, J.M., Lee, S.I., Cooper, G.M., Ahituv, N., Pennacchio, L.A., Shendure, J.: Massively parallel functional dissection of mammalian enhancers in vivo. Nat. Biotechnol.
**30**(3), 265–270 (2012)CrossRefGoogle Scholar - 14.Sharon, E., Kalma, Y., Sharp, A., Raveh-Sadka, T., Levo, M., Zeevi, D., Keren, L., Yakhini, Z., Weinberger, A., Segal, E.: Inferring gene regulatory logic from high-throughput measurements of thousands of systematically designed promoters. Nat. Biotechnol.
**30**(6), 521–530 (2012)CrossRefGoogle Scholar - 15.Kwasnieski, J.C., Mogno, I., Myers, C.A., Corbo, J.C., Cohen, B.A.: Complex effects of nucleotide variants in a mammalian cis-regulatory element. Proc. Natl. Acad. Sci. USA
**109**(47), 19498–19503 (2012)CrossRefADSGoogle Scholar - 16.Fowler, D.M., Araya, C.L., Fleishman, S.J., Kellogg, E.H., Stephany, J.J., Baker, D., Fields, S.: High-resolution mapping of protein sequence-function relationships. Nat. Methods
**7**(9), 741–746 (2010)CrossRefGoogle Scholar - 17.Hietpas, R.T., Jensen, J.D., Bolon, D.N.A.: Experimental illumination of a fitness landscape. Proc. Natl. Acad. Sci. USA
**108**(19), 7896–7901 (2011)CrossRefADSGoogle Scholar - 18.Adkar, B.V., Tripathi, A., Sahoo, A., Bajaj, K., Goswami, D., Chakrabarti, P., Swarnkar, M.K., Gokhale, R.S., Varadarajan, R.: Protein model discrimination using mutational sensitivity derived from deep sequencing. Structure
**20**(2), 371–381 (2012)CrossRefGoogle Scholar - 19.Whitehead, T.A., Chevalier, A., Song, Y., Dreyfus, C., Fleishman, S.J., De Mattos, C., Myers, C.A., Kamisetty, H., Blair, P., Wilson, I.A., Baker, D.: Optimization of affinity, specificity and function of designed influenza inhibitors using deep sequencing. Nat. Biotechnol.
**30**(6), 543–548 (2012)CrossRefGoogle Scholar - 20.Schlinkmann, K.M., Honegger, A., Türeci, E., Robison, K.E., Lipovšek, D., Plückthun, A.: Critical features for biosynthesis, stability, and functionality of a G protein-coupled receptor uncovered by all-versus-all mutations. Proc. Natl. Acad. Sci. USA
**109**(25), 9810–9815 (2012)CrossRefADSGoogle Scholar - 21.Holmqvist, E., Reimegård, J., Wagner, E.G.H.: Massive functional mapping of a 5’-UTR by saturation mutagenesis, phenotypic sorting and deep sequencing. Nucleic Acids Res.
**41**(12), e122 (2013)CrossRefGoogle Scholar - 22.Peterman, N., Lavi-Itzkovitz, A., Levine, E.: Large-scale mapping of sequence-function relations in small regulatory RNAs reveals plasticity and modularity. Nucleic Acids Res.
**42**(19), 12177–12188 (2014)CrossRefGoogle Scholar - 23.Oikonomou, P., Goodarzi, H., Tavazoie, S.: Systematic identification of regulatory elements in conserved 3’ UTRs of human transcripts. Cell Rep.
**7**(1), 281–292 (2014)CrossRefGoogle Scholar - 24.Noderer, W.L., Flockhart, R.J., Bhaduri, A., Diaz de Arce, A.J., Zhang, J., Khavari, P.A., Wang, C.L.: Quantitative analysis of mammalian translation initiation sites by FACS-seq. Mol. Syst. Biol.
**10**(8), 748 (2014)CrossRefGoogle Scholar - 25.Liachko, I., Youngblood, R.A., Keich, U., Dunham, M.J.: High-resolution mapping, characterization, and optimization of autonomously replicating sequences in yeast. Genome Res.
**23**(4), 698–704 (2013)CrossRefGoogle Scholar - 26.Thyme, S.B., Song, Y., Brunette, T.J., Szeto, M.D., Kusak, L., Bradley, P., Baker, D.: Massively parallel determination and modeling of endonuclease substrate specificity. Nucleic Acids Res.
**42**(22), 13839–13852 (2014)CrossRefGoogle Scholar - 27.Kinney, J.B., Tkacik, G., Callan, C.G.: Precise physical models of protein-DNA interaction from high-throughput data. Proc. Natl. Acad. Sci. USA
**104**(2), 501–506 (2007)CrossRefADSGoogle Scholar - 28.Kinney, J.B., Atwal, G.S.: Parametric inference in the large data limit using maximally informative models. Neural Comput.
**26**(4), 637–653 (2014)CrossRefMathSciNetGoogle Scholar - 29.Cover, T., Thomas, J.: Elements of Information Theory, 1st edn. Wiley, New York (1991)CrossRefMATHGoogle Scholar
- 30.Paninski, L.: Convergence properties of three spike-triggered analysis techniques. Network-Comput. Neural
**14**(3), 437–464 (2003)CrossRefGoogle Scholar - 31.Sharpee, T., Rust, N., Bialek, W.: Analyzing neural responses to natural signals: maximally informative dimensions. Neural Comput.
**16**(2), 223–250 (2004)CrossRefMATHGoogle Scholar - 32.Sharpee, T., Sugihara, H., Kurgansky, A., Rebrik, S., Stryker, M., Miller, K.: Adaptive filtering enhances information transmission in visual cortex. Nature
**439**(7079), 936–942 (2006)CrossRefADSGoogle Scholar - 33.Kouh, M., Sharpee, T.O.: Estimating linear-nonlinear models using Rényi divergences. Network-Comput. Neural
**20**(2), 49–68 (2009)CrossRefMathSciNetGoogle Scholar - 34.Rajan, K., Marre, O., Tkacik, G.: Learning quadratic receptive fields from neural responses to natural signals: information theoretic and likelihood methods. Neural Comput.
**25**(7), 1661–1692 (2013)CrossRefMathSciNetGoogle Scholar - 35.Jolma, A., Yan, J., Whitington, T., Toivonen, J., Nitta, K.R., Rastas, P., Morgunova, E., Enge, M., Taipale, M., Wei, G., Palin, K., Vaquerizas, J.M., Vincentelli, R., Luscombe, N.M., Hughes, T.R., Lemaire, P., Ukkonen, E., Kivioja, T., Taipale, J.: DNA-binding specificities of human transcription factors. Cell
**152**(1), 327–339 (2013)CrossRefGoogle Scholar - 36.Oliphant, A., Brandl, C., Struhl, K.: Defining the sequence specificity of DNA-binding proteins by selecting binding sites from random-sequence oligonucleotides: analysis of yeast GCN4 protein. Mol. Cell. Biol.
**9**(7), 2944–2949 (1989)CrossRefGoogle Scholar - 37.Tuerk, C., Gold, L.: Systematic evolution of ligands by exponential enrichment: RNA ligands to bacteriophage T4 DNA polymerase. Science
**249**(4968), 505–510 (1990)CrossRefADSGoogle Scholar - 38.Ellington, A.D., Szostak, J.W.: In vitro selection of RNA molecules that bind specific ligands. Nature
**346**(6287), 818–822 (1990)CrossRefADSGoogle Scholar - 39.Blackwell, T.K., Weintraub, H.: Differences and similarities in DNA-binding preferences of MyoD and E2A protein complexes revealed by binding site selection. Science
**250**(4984), 1104–1110 (1990)CrossRefADSGoogle Scholar - 40.Wright, W., Binder, M., Funk, W.: Cyclic amplification and selection of targets (CASTing) for the myogenin consensus binding site. Mol. Cell. Biol.
**11**(8), 4104–4110 (1991)CrossRefGoogle Scholar - 41.Herzenberg, L., Sweet, R., Herzenberg, L.: Fluorescence-activated cell sorting. Sci. Am.
**234**(3), 108–117 (1976)CrossRefGoogle Scholar - 42.Fowler, D.M., Fields, S.: Deep mutational scanning: a new style of protein science. Nat. Methods
**11**(8), 801–807 (2014)CrossRefGoogle Scholar - 43.Kinney, J.B., Atwal, G.S.: Equitability, mutual information, and the maximal information coefficient. Proc. Natl. Acad. Sci. USA
**111**(9), 3354–3359 (2014)CrossRefMathSciNetADSGoogle Scholar - 44.Khan, S., Bandyopadhyay, S., Ganguly, A., Saigal, S., Erickson III, D., Protopopescu, V., Ostrouchov, G.: Relative performance of mutual information estimation methods for quantifying the dependence among short and noisy data. Phys. Rev. E
**76**(2), 026209 (2007)CrossRefMathSciNetADSGoogle Scholar - 45.Bialek, W., Callan, C., Strong, S.: Field theories for learning probability distributions. Phys. Rev. Lett.
**77**(23), 4693–4697 (1996)CrossRefMathSciNetMATHADSGoogle Scholar - 46.Kinney, J.B.: Estimation of probability densities using scale-free field theories. Phys. Rev. E
**90**(1), 011301(R) (2014)CrossRefADSGoogle Scholar - 47.Kinney, J.B.: Unification of field theory and maximum entropy methods for learning probability densities. Phys. Rev. E
**92**(3), 032107 (2015)Google Scholar

## Copyright information

**Open Access**This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.