An ensemble approach to the evolution of complex systems

Abstract

Adaptive systems frequently incorporate complex structures which can arise spontaneously and which may be non-adaptive in the evolutionary sense. We give examples from phase transition and fractal growth to develop the themes of cooperative phenomena and pattern formation. We discuss RNA interference and transcriptional gene regulation networks, where a major part of the topological properties can be accounted for by mere combinatorics. A discussion of ensemble approaches to biological systems and measures of complexity is presented, and a connection is established between complexity and fitness.

Appendices

Appendix 1 Ensembles, entropy, information

The terminology relating to probabilities and probability distributions shows a great deal of variance (!) between neighbouring disciplines, and this is why we found it necessary to supply this appendix. As physicists, we may be using a slightly different convention than the intended readership of this article.

1.1 Appendix 1.1 Ensembles and probabilities

In order to talk about the probability of occurrence of a certain configuration, event or value, one should first specify what class of systems one has in mind, where the frequency of a configuration, an event or the particular value of a stochastic variable, are to be sought. This class of systems is called the statistical ensemble, or ensemble for short. Theoretically, this imaginary ensemble consists of a very large number of copies of the system under consideration, prepared in the same way and subject to the same external constraints.

The ‘copy of a given system’ may be as simple as the casting of a die, or as complicated as a particular realization of a model network for the gene regulatory system of E. coli. The number of occurrences (N _x ) of a given value or event, in this particular ensemble (say the number X on the up face of the die, or the number of genes having X interactions with other genes) divided by the size of the ensemble (the number of throws of the die, the total number of genes in the network) tends to the probability p(X) of (sometimes called the marginal probability, or the marginal) of this value, event, etc., as the size (N) of the ensemble is increased without bound. This relation we can express symbolically as

$$ { \lim}_{N\to \infty}\left({N}_X/N\right)=p(X). $$

The reason for taking this (ideal) limit is that in any experimental situation, either on the field, in the laboratory, or when doing a computer simulation, for any finite number of samples N, the ratio (N _X /N ) itself is a fluctuating random variable. It can be rigorously shown, however, that in most cases, the variance of this ratio will tend to zero in this ideal limit, and this is how we would like to define our ‘probability distribution’, i.e. the function p(X) for all different allowed values of x.

In real-life experiments one has to do with finite populations, and therefore one has to make do with approximations to the probability distribution, in the form of p(X) ≈ (N _X / N ). Often one has to average over many different runs, populations, or realizations of numerical models, to reduce the variance.

1.2 Appendix 1.2 Some statistical concepts and notation

The average, or expectation value, of a stochastic variable x taking the values x _i with probabilities p _i can be expressed as 〈x〉 = ∑  _i p _i x _i . (In physics, one usually does not make a distinction between the name of the variable and the different values that it may take; instead of X in the above paragraph, here we have used x _i , and summed over i.) In case the variable takes continuous values, the sum goes over to an integral; the probability distribution {p _i } is then replaced by p(x)dx where p(x) is now a probability density function over allowed values of x. The statistical ensemble to which the variable x belongs is defined by the set of probabilities {p _i } or the probability density function p(x).

Correlations between stochastic variables in a given ensemble are measured via the correlation function. The correlation function between two variables is a measure of the degree to which they depart from being independent of each other, usually as a function of the distance between them. Note that in case the variable x and y are independent of each other the expectation value 〈xy〉 = 〈x〉〈y〉. Thus, 〈xy〉 − 〈x〉〈y〉 ≠ 0 indicates that the variables x and y are not independent, but ‘correlated.’ The two-point correlation function

$$ C(r)=\left\langle {x}_0{x}_r\right\rangle -\left\langle {x}_0\right\rangle \left\langle {x}_r\right\rangle $$

measures the correlation between two variables x ₀ and x _r as a function of r, where r may indicate their distance from each other in some metric space; we expect the correlation to decrease as a function of this distance. For example, density fluctuations ρ in a fluid at neighbouring points set off by a distance r from each other would obey a correlation function which decreases exponentially with r, unless the fluid is at its critical temperature and pressure, in which case it would decrease only as a power of r.

The values that the variables x and y might take could be letters from an alphabet. Given an ensemble of letter-sequences, we could compute the two-point correlation function between characters appearing at positions along the sequence separated by r letters.

1.3 Appendix 1.3 Information content and entropy

The definition of information given by Shannon (1948) is that it is equal to the entropy for a given probability distribution p _x (X) for the variable x to take on different values X,

$$ S=I=-{\displaystyle {\sum}_X{p}_x\left(\mathrm{X}\right) \log {p}_x\left(\mathrm{X}\right).} $$

Note that this definition of information is the same as the entropy up to a dimensional constant which we have set equal to unity. We will use the term information and entropy interchangeably throughout the article. We have used X to denote not necessarily one but a whole list of attributes with which a particular state may be specified, and the sum is over all such states. The probability distribution p _x (X) characterizes the statistical ensemble to which the system under consideration belongs. Note that for an equiprobable distribution, I is simply the logarithm of the number of possible states. On the other hand, if any of the probabilities p _x (X)

The mutual information between two probability distributions for the variables x and y, p _x (X) and p _y (Y) is defined as

$$ {\mathrm{I}}_M={\displaystyle \sum_{X,Y}{p}_{xy}\left(X,Y\right)}\left[ ln{p}_{xy}\left(X,Y\right)- \ln {p}_x(X)- \ln {p}_y(Y)\right] $$

where p _xy (X,Y) is the joint probability distribution.

The conditional information I _cond is defined by

$$ {I}_{\mathrm{cond}}={\displaystyle \sum_Y{p}_y(Y){\displaystyle \sum_X{p}_{xy}\left(X\Big|Y\right)}} \ln {p}_{xy}\left(X\Big|Y\right) $$

where p _xy (X|Y) is the conditional probability distribution of x conditional upon y. Note p _xy (X|Y) = p _xy (X,Y)/p _y (Y), which gives, I _M  = I _x  − I _cond. We have indicated the information content of the probability distribution of x by I _x , for clarity.

1.4 Appendix 1.4 Relative information and the effective length of sequences in terms of their relative information content

The probability matrix of a ‘binding site’ is found in the following way: One defines each entry of this matrix, p _ij , to be the probability with which the jth site along the binding sequence for that protein is occupied by A (i = 1), T(i = 2) ,C(i = 3) or G (i = 4). Note that i runs from 1 to 4 and j runs from 1 to n, where n is the length of the consensus sequence.

The information content of a binding site is equal to the entropy of the probability matrix of the binding site,

$$ S=I=-{\displaystyle {\sum}_{i,j}{p}_{ij} \log {p}_{ij}} $$

where i is summed from 1 to 4, and j from 1 to n. The information, I, is equal to zero if no ‘mistakes’ are allowed on any of the sites, i.e. there is one unique binding sequence. On the other hand, if the binding occurs in a totally indiscriminate way, so that any letter can appear with equal probability at any of the positions along the sequence, the information (entropy) takes the largest value it can take, namely I ₀ = n log 4.

Let us define the relative information to be I _R = I ₀ − I. This quantity measures how different the binding sequence is from a perfectly random one. The effective number n _e of strict constraints that have to be satisfied for the binding to occur is then given by the relative entropy divided by log4; i.e. n _e = I _R /log 4. Note that n _e will be close to zero if the binding sequence is very indiscriminate; it will be close to n if the binding is very strict.

The distance between two probability distributions P and Q may be defined as

$$ D={\displaystyle {\sum}_X{\left[P(X)-Q(X)\right]}^2.} $$

If Q is the uniform distribution, D is known as the ‘disequilibrium’ (Nicolis and Prigogine 1977), i.e. the distance of the probability distribution P from the uniform (or equiprobable) distribution.

The statistical complexity is defined as

$$ C\left[P\right]=J\left[P,Q\right]\times S(P)/{S}_{\max }{J}_{\max }, $$

where J[P,Q] is the Jensen–Shannon divergence is related to the relative information between the probability distributions P and Q. It is defined as

$$ J\left[P,Q\right]=S\left[\left(P+Q\right)/2\right]-\left[S(P)+S(Q)\right]/2. $$

The expression S [(P + Q) / 2] means, explicitly, S (U) = − Σ _X U(X) log U(X), where we have set [P(X) + Q(X)] / 2 = U(X).

Given that the number of allowed states of the system is N _s , the distribution Q may be chosen as the equiprobable distribution, Q(X) =1/N _s , for which the entropy of the system would be maximum. Alternatively, Q can be chosen as any reference state which has a greater degree of randomness than the system at hand, for example, the probability distribution of a null model (Martin et al. 2003; Lamberti et al. 2004; Rosso et al. 2009). Note that a similar statistical complexity can be defined in terms of the ‘disequilibrium’ rather than the Jensen–Shannon divergence.

Appendix 2 Network properties

2.1 Appendix 2.1 The adjacency matrix

Any network having N nodes can be represented by an N × N matrix called the adjacency matrix. The elements of this matrix, A _ij , can be assigned weights, which characterize the strength of interaction between the nodes indexed by i and j, running from 1 to N. If the network is not weighted, so that it simply has edges which are either there or absent, then the elements of the adjacency matrix are either zero or unity. If the network is directed, or if there are different types of nodes, the adjacency matrix will be asymmetric, and symmetric otherwise. In general, summing over the ith row of the matrix will give the in-degree of the ith node; similarly summing over the jth column will give the out-degree of the jth node.

$$ \begin{array}{cc}\hfill {d}_i^{\mathrm{in}}={\displaystyle \sum_j{A}_{ij}}\hfill & {d}_j^{\mathrm{out}}={\displaystyle \sum_i{A}_{ij}}\hfill \end{array} $$

For a symmetric adjacency matrix these two quantities are obviously equal to each other, and then one simply talks about the degree of a node.

2.2 Appendix 2.2 Some topological properties of networks

The ‘degree distribution’ of a network is defined as the relative frequency (in the limit of an infinite graph, the probability) of those nodes which have d other nodes connected to them by edges.

On a ‘classical’ random graph, or Erdös-Renyi (ER) network (Erdös and Renyi 1959), each pair of nodes is connected to each other with the same probability p, independently of all the other pairs. The expected degree, (average number of edges) for each node is pN, where N is the number of nodes (if self-connection is not allowed this is p (N − 1) ≈ pN for N> > 1 ). The degree distribution is P(d) = C(N;d) p ^d (1 − p)^{N − d} where C(N;d) is the number of different ways one can choose d objects out of N, regardless of order. The binomial distribution reduces, for small p and large N, to a Poissonian, with the mean and variance equal to pN.

The K-core decomposition of a graph is obtained in the following way: Throw out the disconnected nodes. Find the set of all nodes that have only one edge S1. Now remove all those single edges. Add to S1 all those nodes which now end up being singly connected and iterate until no singly connected nodes remain. Put in S2 all those nodes which now have two edges. Cut those edges and add to S2 all those nodes which have only two edges remaining as a result; iterate until you do not encounter any nodes with two edges any more. Repeat for 3, 4, …K edges, until you exhaust all the nodes. In this way, the graph is partitioned into K sets (‘shells’) that are successively more highly connected the larger K is. The number of such shells is an indicator of the complexity of the network.

The neighbourhood of a node (say node i) is defined as all those nodes to which i is directly connected, i.e. which are only one edge away from i. The clustering coefficient of a node i, namely c _i , is defined as the probability that the nodes in the neighbourhood of i are connected between themselves pairwise, and can be computed as the number of edges connecting the neighbours of i to each other, normalized by the number of distinct pairs of nodes in the neighbourhood of i. The clustering coefficient of a graph is found by averaging the clustering coefficients of the individual nodes. Clearly one can also define clustering coefficients averaged over different subsets of nodes, e.g. c(d) is the clustering coefficient averaged over all nodes having degree d.

The rich-club coefficient r(d) is defined as the probability that nodes having degree larger than k are connected pairwise between each other. It can be computed as the number of edges running between nodes with degree > d, normalized by the total number of distinct pairs of nodes with degree d.

The degree-degree correlation function is the correlation between the degrees of a node and the degrees of its neighbours. It is defined as d _nn = ∑ d' p(d | d') where the sum is over d' and p(d | d') is the conditional probability that a neighbour of a d-degree node has degree d'.

2.3 Appendix 2.3 Weight distribution of the miRNA-miRNA co-regulation network

Let A be the M×N adjacency matrix of a bi-partite graph, where there are two types of nodes, let us say a-type nodes and b-type nodes (as in the case of miRNA and mRNA), and a-type nodes can only be directly connected to the b-type nodes.

Then W = AA^T (where the superscript T denotes ‘transpose’) can be thought of as a ‘weighted’ adjacency matrix of the network of a-nodes, with the interactions being mediated by the b-nodes. The weight of an interaction is equal to the number of different b-nodes nodes via which two a-nodes are connected to each other.

The elements of W (the weighted M × M adjacency matrix of the miRNA-miRNA network, in our case) are given by ${w}_{ij}={\left(\mathbf{A}\kern0.5em {\mathbf{A}}^T\right)}_{ij}={\displaystyle \sum_{k=1}^N{A}_{ik}{A}_{jk}}$.

The off-diagonal elements are nonzero if the nodes i and j (both of type a) are connected to each other by at least one path consisting of two successive edges, leading from the ith type a-node to some type b-node and then back to the jth a-node (recall that nodes of the same type are not connected to each other in the miRNA-mRNA network). The weight w _ij is in fact equal to the number of such parallel two-step paths, between the ith and jth miRNAs. The diagonal elements are equal to twice the degree of the a-nodes in the original a-b network (the two-step paths in this case return to the ith a-type node from which they issued). The frequency f(w) of any given weight w among the off-diagonal elements of the matrix W, normalized by (1/2) times the number of off diagonal elements, gives the probability distribution function for the weights, P(w), viz.,

$$ P(w)=\frac{2f(w)}{M\left(M-1\right)}. $$

Here the factor of (1/2) comes from the fact that the number of off-diagonal elements is equal to twice the number of a-type pairs of nodes; M is the number of a-type nodes.

Appendix 3

3.1 Appendix 3.1 Sequence matching probabilities between random sequences

Let us consider sequences made up of characters drawn with equal probabilities (1/r) from an r-letter alphabet. The probability of one-to-one sequence matching (or Watson–Crick base-pairing) of a random sequence of length l' at least once within a longer random sequence of length l ≥ l' is

$$ p\left(l,l^{\prime}\right)=1-{{\left(1-{r}^{-l}\right)}^{l\prime}}^{-l+1}. $$

to a very good approximation (Mungan et al. 2005 and references therein). It should be noted that for l' > > l, p(l,l') ~ l' /r ^l .

If we make the simplifying assumption of equal probabilities of occurrence for all the nucleotides in these sequences, this relation can be used in two problems we have dealt with in this article: (i) for the probability of encountering the seven-nucleotide-long ‘seed’ sequence of the miRNA within the 3′ untranslated region of an mRNA, and (ii) the probability of encountering the consensus sequence for a transcription factor in the promoter region of a gene.

These considerations can easily be generalized to the case where the letters occur with different probabilities, and where the sequences also display nearest neighbour correlations between their elements. If the probabilities of occurrence of the different ‘letters’ (nucleotides) – in either the short (e.g. seed) or long (e.g. in the UTR) sequences – are appreciably different from each other, one has to replace r ^−l in p(l,l' ) by the appropriate product of the probabilities for the individual letters making up the seed region, within the zeroth order Markov approximation, or the product of the conditional probabilities within the first order Markov approximation (appendix 4).

3.2 Appendix 3.2 Computing the in-degree distribution for the miRNA-mRNA target network

Let us consider the example of the miRNA-mRNA target network. To compute the in-degree distribution (figure 2), we need a ‘binding probability’, i.e. probability that the seed sequences of a miRNA occurs at least once in the UTR of any given mRNA. The seed sequence on the miRNA is taken to have a fixed length (l = 7).

For sequences consisting of symbols drawn independently and with equal probabilities from a set of cardinality r, the probability to encounter any specific sequence of length l is simply given by (1/r)^l. In the case of the nucleotide alphabet A, T, C, G, clearly r = 4.

The distribution of the number of edges terminating on any mRNA with a UTR of length l' is given by the binomial distribution,

$$ {P}_{l\prime }(d)=C\left(M;d\right)p{\left(7,l\prime \right)}^d\left[1-p{\left(7,l\prime \right)}^{M-d}\right] $$

for a set of M miRNAs. The total in-degree distribution is then simply given by

$$ P(d)={\displaystyle \sum {P}_{l\prime }}(d)g\left(l^{\prime}\right) $$

where g(l') is the length distribution of the UTRs and the sum is over all allowed l'. The empirical, point for point length distribution of the UTRs was obtained from the databases (Grillo et al. 2010). We found that we could fit this distribution with a smooth function

$$ g(k)=\mathrm{N}{k}^{\psi }{e}^{-k/\kappa }, $$

where the normalization factor is given by N = Γ(ψ + 1)κ ^{( ψ+1 )}, Γ is the gamma function, and the values for the parameters ψ and κ for different species can be found in the following table.

Fitting parameters and first moments of UTR length distributions of the mRNAs of four organisms considered in section 3.1

Parameters	H. sapiens	B. taurus	X. tropicalis	C. elegans
κ	515	485	490	202
ψ	0.14127	0.261554	0.254772	0.253727
<k>	874	739	731	338

The degree distribution which is computed here for the miRNA-mRNa network is therefore a weighted superposition of different Poisson distributions (classical random networks – see appendix 2.2), with different connection probabilities p(7,l′), depending on the different UTR lengths (l′) present.

Appendix 4 Generating sequences via zeroth and first order Markov processes

Here we provide a working definition of a Markov process which generates a Markov sequence or Markov chain. More formal and correct definitions can be found in standard textbooks on probability theory.

A sequence (when we say a sequence we mean an ordered sequence) x ₁, x ₂ … x _N is called Markovian if the ith element in the sequence, x _i depends at most on the values of a finite number n of immediately preceding elements, i.e. {x _i − n , x _{i − n + 1} … x _i − 1}. The Markov sequence does not depend on the whole ‘history’ of the process.

In a Markov sequence of order 0, each element is chosen independently, according to a fixed probability distribution {p _I }, from the set of allowed elements (or ‘values’) X _I with I = 1,2, … M. Thus, the sequence has no memory of the preceding part of the sequence at all. Thus, the probability of encountering any particular subsequence of length l = 3, e.g. (X ₁₅ X ₃ X ₇), starting from any given position in the sequence is simply given by the probability of the product of the elements which appear, namely p ₁₅ p ₃ p ₇. In order to model a naturally occurring sequence of length N, one may estimate the probabilities p _I of the different characters from their frequencies N _I of occurrence, so that

$$ {p}_I\approx {N}_I/N. $$

For a Markov sequence of order 1, each subsequent element x _i+1 does depend on the preceding one, i.e. x _i . One then needs the set of M × M conditional probabilities, p(X _I |X _J ) = p(X _I ,X _J )/p(X _I ). On the left-hand side we have the conditional probability for X _J to appear, given that the preceding element is X _I . The equality is known as the Bayesian identity. On the right-hand side we have the joint probability for the elements X _I and X _J , divided by the probability of the element X _J . Then, the subsequence used in the example above has the probability p(X ₁₅|X ₃)p(X ₃|X ₇) to be encountered after the element X ₁₅.

In order to model the first order Markov process from which a sequence may have originated, one may once again estimate the conditional probabilities p(X _I |X _J ) from the conditional frequencies N _IJ via p(X _I ,X _J ) ≈ N _IJ /(N − 1), where N _IJ counts all 2-element subsequences X _I X _J in the entire sequence, and divides by the total number of 2-element subsequences, which is N−1 ≈ N.

Tables of probabilities used for generating zeroth and first order Markov series for miRNA seed and mRNA 3′ UTR sequences have been provided as supplementary material.