The importance of thermodynamics for molecular systems, and the importance of molecular systems for thermodynamics

Improved understanding of molecular systems has only emphasised the sophistication of networks within the cell. Simultaneously, the advance of nucleic acid nanotechnology, a platform within which reactions can be exquisitely controlled, has made the development of artificial architectures and devices possible. Vital to this progress has been a solid foundation in the thermodynamics of molecular systems. In this pedagogical review and perspective, we discuss how thermodynamics determines both the overall potential of molecular networks, and the minute details of design. We then argue that, in turn, the need to understand molecular systems is helping to drive the development of theories of thermodynamics at the microscopic scale.


Introduction
Thermodynamics was originally developed in the 19th Century, driven by the dawn of the industrial revolution (Carnot et al. 1960), and a desire to understand and optimise the extraction of useful work from engines. This work could be harnessed to pump water out of mines, or drive locomotives, for example. Although these machines were mechanical devices powered by the flow of heat, the fundamental source of the work was the chemical fuel -typically coal, initially. Chemical processes were incorporated into the framework of thermodynamics by Gibbs (1906), leading to an understanding of the spontaneity of, and energy exchanged during, chemical reactions and phase changes.
Thermodynamics as originally introduced was a theory based entirely on the interrelation of macroscopic observables, such as temperature, pressure, volume and energy. In the late 19th and early 20th centuries, the development of statistical mechanics provided a microscopic basis for the theory, explaining how these bulk properties emerge from microscopic system properties (Boltzmann 1964). In the process, the concept of entropy-the mysterious quantity whose increase is responsible for the thermodynamic arrow of time-was made far more concrete, as a statistical measure of the uncertainty of the microscopic state of a system. The exploration of statistical mechanics led to the development of theories of critical phenomena; explaining the exotic yet often universal behaviour of systems as they approach certain kinds of phase transitions (Yeomans 1992;Cardy 1996); statistical mechanics is also the fundamental tool underlying the field of molecular simulation (Frenkel and Smit 2001;Tuckerman 2010).
Statistical mechanics and thermodynamics are often introduced as the study of equilibrium states, in which there is no net tendency for the system to evolve over time unless driven from the outside. Nonetheless, from the earliest days, Boltzmann and others proposed theoretical descriptions for the evolution of non-equilibrium systems and their subsequent relaxation to equilibrium (Boltzmann 1964). The developing field of stochastic thermodynamics, in which the probabilistic description underlying statistical mechanics is extended to describe trajectories of nonequilibrium systems through state space, has recently provided remarkable understanding of systems arbitrarily far from equilibrium (Crooks 1999;Seifert 2005Seifert , 2012Esposito and Van den Broeck 2011;Jarzynski 2011).
Over the same period, our understanding of biomolecular systems has been transformed from complete ignorance to the ability to rationally design synthetic circuits and selfassembling architectures in vitro and in vivo. Indeed, although proteins were identified as far back as the early 19th Century (Mulder 1839;Teich and Needham 1992), their role as enzymes in living organisms was not demonstrated until 1926 (Sumner 1926), and protein structures were first solved in Kendrew et al. (1958) and Muirhead and Perutz (1963). Similarly, the genetic information-carrying role of biological DNA was first demonstrated in 1944 (Avery et al. 1944), the double-helical structure was solved in 1953 (Watson and Crick 1953) and the central dogma of molecular biology (that genes encoded in DNA are transcribed into RNA, and then translated into proteins) was first stated in 1958 (Crick 1958(Crick , 1970. Since then, through advances in crystallography, microscopy and other technologies, the molecular mechanisms of an enormous biochemical processes have been identified. Additionally, systems biology has shown how individual component reactions can combine to provide the complex behaviour exhibited by cells (Alon 2007)-although we remain far from a full understanding of such sophisticated systems.
In the process of understanding some of the molecular complexity of the cell, we have shown that it contains microscopic analogues of the mechanical engines of the 19th Century. Molecular motors such as myosin consume chemical fuel to generate locomotive forces (Howard 2001), and enzymatic pumps consume the same fuel to drive ions across membranes (Nelson 2004). These membranes then act as capacitors that provide an alternative supply of power, like batteries for electric motors.
A deep appreciation of natural biomolecular systems is worthwhile in and of itself, and it provides an important contribution to the advancement of medicine. But equally, this hard-won understanding has laid the groundwork for the engineering of artificial systems and devices. In synthetic biology, novel molecular circuitry is often built by connecting naturally-occuring or slightly mutated proteins via artificial transcriptional regulation pathways (Baldwin et al. 2012). At the same time, the molecular nanotechnology community has constructed systems based on artificial components, including non-biological DNA and RNA sequences (Chen et al. 2015) and even artificial proteins (Hsia et al. 2016;Bale et al. 2016). At the interface of these communities are those who combine the functionality of synthetic and natural components, both in vivo and in vitro (Fujii and Rondelez 2012; Green et al. 2014). In aggregate, this work has produced remarkable results, including nanoscale self-assembly (Rothemund 2006;Douglas et al. 2009;Ke et al. 2012), implementation of molecular computation and control architectures (Winfree et al. 1998;Seelig et al. 2006;Zechner et al. 2016) and repurposing of microbes for industry and healthcare (Baldwin et al. 2012;Kitney and Freemont 2012;Chubukov et al. 2016).
In this pedagogical perspective, we will first discuss the basics of traditional chemical thermodynamics as it applies to biomolecular systems (Sects. 2, 3). We subsequently show how these ideas shape our understanding and design of functional molecular systems, both at a fundamental and a practical level, in Sects. 4-7. A particular focus will be common misconceptions or pitfalls that result from careless treatment of the underlying thermodynamics. Finally, we briefly discuss the emerging field of stochastic thermodynamics in Sect. 8. This extension of traditional thermodynamics to fluctucating, far-from equilibrium contexts finds its most natural application in the analysis of molecular systems. Indeed, we will then argue that the very process of exploring abstract thermodynamic ideas in concrete biomolecular systems is in turn providing a deeper understanding of the fundamental thermodynamics.
2 Fundamentals of classical statistical mechanics 2.1 The partition function and thermodynamic quantities We will begin by considering the properties of an arbitrary, closed, equilibrium system, and then develop those ideas to arrive at the statistical mechanics of biochemical systems in particular. A closed system has fixed amounts of energy and matter. Nonetheless, a large closed system can access an enormous number of microstates specified by the positions and momenta ðx; pÞ of all the constituent degrees of freedom. In classical (as opposed to quantum) physics, these microstates are assumed to be present with a constant density q throughout the available phase space defined by ðx; pÞ Frenkel and Smit (2001). An experimenter that could perform the (impossible) task of measuring the precise microstate ðx; pÞ of our system would not get a predictable value. This uncertainty can be quantified by Pðx; pÞ, the probability per unit phase space volume of observing a microstate ðx; pÞ upon measurement. Initially, we are interested in characterising equilibrium systems, which exhibit no net flows between any pair of microstates and thus have Pðx; pÞ constant over time. The fundamental assumption of classical statistical mechanics is that in thermodynamic equilibrium, there is the maximal possible uncertainty in the microstate ðx; pÞ (Jaynes 1957). In other words, all accessible (equal-energy) microstates are equally probable in equilibrium (Huang 1987). In the rest of this Section, we further expand on the the properties of the equilibrium distribution that follow from this principle of equal a priori probability.
In the molecular context, we are usually interested in a relatively small system r thermally connected to a much larger environment R, rather than a system in total isolation. This larger environment might be the lab as a whole, or perhaps a water bath for elevated temperatures. In any case, we are typically not concerned with the details of this environment R, other than in its role as a source and sink of energy in the form of heat. Generally, we assume that the coupling is weak so that it is reasonable to separately consider the energies of R and r (Frenkel and Smit 2001;Huang 1987).
The combined system of r þ R remains a closed system, which can in principle reach equilibrium. However, due to energy exchange between r and R, the energy of each component fluctuates and microstates of r with different energies E r ðx; pÞ can be accessed. From applying the principle of equal a priori probability to the combined system of r þ R, it is possible to show that energy is shared such that microstates of r are occupied with a probability density (Frenkel and Smit 2001;Tuckerman 2010;Huang 1987) P eq r ðx; pÞ ¼ q where the environmental heat bath sets the temperature T, and the partition function Z r normalizes the distribution: Here, q is the constant density of microstates that cancels out during calculations, but ensures the correct dimensionality. Eq. 2 is the famous Boltzmann distribution (Frenkel and Smit 2001;Tuckerman 2010;Huang 1987) its form arises from sharing energy between r and R in such a way as to maximise overall uncertainty in the microstate. Atomistic models of molecular systems, such as AMBER and CHARMM (Orozco et al. 2003;Cino et al. 2012), are essentially semi-empirical energy functions E r ðx; pÞ. Coarse-grained models, such as Martini and oxDNA (Marrink et al. 2007;Doye et al. 2013), are attempts to capture the behaviour of a reduced set of key degrees of freedom with a similar energy model. In either case, the models are typically too complex to be analysed directly. Instead, simulation is used to sample microstates of r, allowing the equilibrium properties of the system to be inferred.
The key thermodynamic quantities follow from the partition function. Firstly, the internal energy of r is a straightforward average over microsate energies (Frenkel and Smit 2001;Tuckerman 2010;Huang 1987) The thermodynamic entropy is less obvious-it is interpreted as a measure of the statistical uncertainty of the microstate distribution P eq r ðx; pÞ Jaynes (1957). This interpretation is made plausible because the second law states that the entropy of a closed system cannot decrease with time as it converges towards equilibrium; and the principle of equal a priori probability that implies that the equilibrium distribution maximises uncertainty for a closed system. Specifically, for an arbitrary discrete distribution P(y) over the variable y, the uncertainty in y is described by the statistical entropy (Jaynes 1957;Shannon and Weaver 1949) H½P ¼ À X y PðyÞ ln PðyÞ: Note that H½P ! 0 is minimised when y takes a single value with probability 1, and maximised if p(y) is uniform (Shannon and Weaver 1949).
Allowing for continuous variables and introducing the constant k B to connect to physical quantities, the equilibrium thermodynamic entropy of r is given by a similar expression: dx; dp P eq r ðx; pÞ lnðP eq r ðx; pÞ=qÞ This definition of entropy in Eq. 5 is essentially the fundamental link between equilibrium statistical mechanics and macroscopic equilibrium thermodynamics. Finally, the free energy of r follows as (Frenkel and Smit 2001, Tuckerman 2010, Huang 1987) The thermodynamic quantities listed above are the natural quantities of interest when we consider a system being manipulated from the outside, and thereby transitioning between two distinct equilibria. However, in molecular systems, we typically set up the system in a non-equilibrium state, and allow it to evolve without further perturbation. We are then generally interested in questions such as: what are the molecular abundances in the eventual equilibrium state, and how fast does the system get there (if at all)? To answer these questions, it is helpful to define biochemical macrostates.
The importance of thermodynamics for molecular systems, and the importance of molecular... 5

Macrostates
The Boltzmann distribution specifies the relative abundances of microstates in equilibrium, and detailed models can be simulated to sample from this distribution. However, microstates are inaccessible in experiment, and inconvenient for theory. Instead, we typically consider biochemical macrostates. Rather than keep track of all of the atoms in a DNA molecule or protein, we might simply predict theoretically, measure experimentally or infer from simulation the behaviour of the position of the centre of mass, or gross conformational features of a molecular system. In other words, we group sets of microstates together into a relatively small number of macrostates, as illustrated schematically in Fig. 1. In this Subsection, we outline the macrostate-level description of thermodynamics that will be used throughout the review. In equilibrium, the occupancy of macrostate i is obtained by integrating over all microstates within it P eq r ðiÞ ¼ where we have defined the partial partition function Z r ðiÞ. This probability P eq r ðiÞ is related to the macrostate free energy F r ðiÞ via Note that macrostates with high free energy are improbable, and macrostates with low free energy are probable. In principle, any division into macrostates is valid, although only well-chosen macrostates are helpful. Typically, well-chosen macrostates are either directly identifiable in experiment, amenable to theoretical modelling, or both. Examples might include macrostates labelled by the number of proteins in dimeric complexes; the end-to-end extension of a biomolecule under stress; or the number of base pairs in a DNA hairpin.
There are two conceptually distinct contributions to the free energy F r ðiÞ (Eq 6): the average energy of macrostate i, dx; dp E r ðx; pÞP eq r ðx; pjiÞ; ð9Þ and the entropy of macrostate i, dx; dp P eq r ðx; pjiÞ lnðP eq r ðx; pjiÞ=qÞ: Here P eq r ðx; pjiÞ ¼ P eq r ðx; pÞ=P eq r ðiÞ is the equilibrium probability density of occupying microstate ðx; pÞ within i, given that the system is in one of the microstates within macrostate i. A low average energy implies that a macrostate i more probable, since individual microstates with lower energies are more probable. A high entropy S r ðiÞ implies that many microstates ðx; pÞ contribute to i; for a given average energy, a macrostate with more accessible microstates is more probable.

Free energies of biochemical reactions
We now discuss the standard statistical mechanical approach to biochemical reactions. Our discussion will justify the form of chemical potentials in dilute solution, and illustrate the meaning of free energies and standard free energies of reaction. In the subsequent sections, this basic framework will be applied to a range of contexts of relevance to natural and engineered molecular systems. Before proceeding, it is worth noting that, as in Sect. 2.1, we consider a system r that can exchange heat with its environment R, but which we have implicitly assumed to occupy a fixed volume V r . In chemical contexts, it is often more natural to consider a system maintained at constant pressure p. In this case, V r has to shrink or grow in response to reactions that tend to decrease or increase the internal pressure, respectively. In this case, the Gibbs free energy G eq r ¼ F eq r þ PV eq r , which plays a similar role to the Helmholtz free energy F in a constant pressure setting, is the key quantity.
Biomolecular processes, however, occur in aqueous solution, and the enormous numbers of water molecules present dominate the pressure exerted by the system (Nelson 2004). Reactions between the relatively small number of solute molecules therefore have almost no effect on the pressure and both theoretical work and experimental analyses usually assume constant volume (which is much easier to work with). Nonetheless, free energies of solute states are typically quoted in terms of the Gibbs  internal energy U. For internal consistency, we will continue to use F and U, but readers familiar with G and H should treat them as essentially equivalent. It is also worth noting that in biochemistry, it is more common to use the molar gas constant R rather than Boltzmann's constant k B . This is simply a question of measurement units; if R is used, all entropies and energies must be given per mole of substance, rather than per particle. The starting point for our analysis is to treat the solvent implicitly. Formally, this corresponds to integrating over the solvent degrees of freedom in the partition function (Eq. 2), leaving only effective interactions between solute degrees of freedom. In practice, we often just assume that this can be done, and take the effective solute interactions as an input. We then assume that all solutes, or complexes of solutes, can be assigned to a discrete set of molecular species. These species might include ATP, ADP and inorganic phosphate, or a set of individual DNA strands and their complexes: for example, DNA strand A, DNA strand B and duplex AB. It is helpful to define macrostates fNg of the entire solution in terms of the abundances of each of these species, fNg ¼ ðN A ; N B :::Þ. A typical macrostate of a small system, at this level of description, is schematically illustrated in Fig. 2.
For each species j, we can define the single-molecule partition function in the volume V r , z j r . This quantity is analogous to the partition function in Eq. 2, but the integral is performed only over the degrees of freedom of a single solute molecule of species j in a volume V r , with the solvent again treated implicitly. It should be noted that z j r is generally strongly temperature-dependent. In a dilute solution, the overall partition function of a macrostate fNg is essentially given by a product of the individual partition functions, since interactions between molecules that are not in a complex are weak. Hence the degrees of freedom for separate complexes are essentially independent and the partition function factorises. Thus where the product j runs over all species types, including complexes. The extra factorial term corrects for overcounting of states that should actually be viewed as indistinguishable, because they are related by the exchange of identical molecules (Frenkel and Smit 2001;Tuckerman 2010;Huang 1987). From this partition function, the free energies of chemical macrostates follow Here, we have highlighted the fact that the free energy decomposes into a sum over the contributions from each species, F j r ðN j Þ, which is a result of the diluteness approximation. The final line uses Stirling's approximation of ln N! % N ln N À N, which is highly accurate for large N. We can thus easily calculate the chemical potential l j r , which is the increase in system free energy due to the addition of another molecule of species j: in the limit that N j is large. l j r is simply the difference in free energy arising from adding one molecule of species j to the system.
Since each z j r is a partition function for a single molecule of species j in volume V r , it will grow proportionally to V r ; doubling the volume doubles the number of accessible positions for the molecule, but changes nothing else. It is thus convenient to normalise using a standard volume V 0 , typically taken as the volume in which a single molecule would constitute a concentration C 0 ¼ 1=V 0 of 1 mole per litre. Thus with a z j 0 ¼ z j r V 0 =V r dependent on the choice of standard volume V 0 , rather than system volume V r . This decomposition separates the chemical potential l j (or free energy per molecule of species j) into a term that depends only on the details of the effective interactions within the species, and a concentration-dependent term (Nelson 2004;Huang 1987).
The chemical potential appearing in Eq. 14 is of enormous use in analysing the thermodynamics of molecular A B C ABC BC AB Fig. 2 A self-assembling system, in which monomers A, B and C combine to form a trimer ABC. The macrostate in this snapshot is The importance of thermodynamics for molecular systems, and the importance of molecular... 7 systems. In particular, it enables us to calculate the difference in free energy between initial and final macrostates after any given molecular reaction, telling us how much more likely the final state is than the initial state in equilibrium. To see why, note that every possible reaction k is associated with a stoichiometric vector m kj , the number of molecules of species j produced by the reaction k (this number is negative if species j is consumed by the reaction). For example, if reaction k is A þ B ! AB, m kA ¼ À1, m kB ¼ À1 and m kAB ¼ 1. Thus the difference in macrostate free energy due to reaction k is simply (Nelson 2004) where the approximation is highly accurate if N j is large. Note that D k F r , like z j r and z j 0 , is generally strongly temperature-dependent. We will continue to specify the reaction in question by a subscript on D, so that D k indicates the change due to reaction k.
Equally, D k F r , through l j r , is dependent on the concentrations of the reactants and products. It is common to consider the value of D k F r at the reference concentration C j ¼ C 0 ¼1 M, D k F 0 , as a ''standard'' free energy of the reaction. D k F 0 can also be further subdivided into standard energetic and entropic contributions, D k U 0 and D k S 0 . This is convenient for book-keeping purposes, but it should be noted that this ''standard'' value is dependent on the (arbitrary) choice of C 0 , unless the number of reactants and products is the same. For example, for the simple bimolecular binding reaction A þ B ! AB, and the term inside the logarithm scales as V 0 . In such cases, it is inadvisable to over-interpret the sign and magnitude of the standard free energy; whether it is positive or negative depends upon the arbitrary standard concentration, which is 1 M in general. This dependence on V 0 feeds through to the standard entropy D k S 0 , so it is also unwise to read too much into the sign and magnitude of this quantity when the numbers of reactants and products differ. Indeed, for macromolecules such as DNA, RNA and proteins, 1 M corresponds to an incredibly concentrated solution where the dilute approximations above break down, and behaviour search as the formation of liquid crystals is observed (Nakata et al. 2007;De Michele et al. 2012). For processes involving unequal numbers of reactants and products, therefore, the standard free energy and entropy exist purely as book-keeping devices, and never describe the actual properties of a reaction. In typical dilute molecular systems, the concentration of the relevant components is orders of magnitude lower. Thus actual values of D k F r and D k S r are significantly more positive than the standard values for assembly reactions in which the number of reactants exceed the number of products.
4 The application of equilibrium thermodynamics to the design of self-assembling systems 4.1 Self-assembly Self-assembly occurs when molecules are mixed and autonomously bind to produce non-trivial structures. It should be distinguished from step-by-step directed synthesis in which each stage is separately coordinated by an experimenter through careful manipulation of solution conditions (Chen and Seeman 1991), although temperature ramps are often used to optimise results, as analysed in some detail in Sobczak et al. (2012). In biology, functional protein complexes (Ahnert et al. 2015) and virus capsids (Fraenkel-Conrat and Williams 1955;Johnston et al. 2010) must assemble accurately from their components; the rise of nucleic acid nanotechnology has facillitated the design of artificial systems that can mimic this behaviour, allowing precisely-controlled finite-size nanostructures (Rothemund 2006; Ke et al. 2012;Goodman et al. 2005;Tikhomirov et al. 2017). In this Section, we outline how the basic thermodynamics introduced in Sects. 2 and 3 both shape the fundamental behaviour of self-assembling systems, and guide the details of system design through tools such as Nupack (Dirks et al. 2007). To provide context, we will apply the general results to the challenge of assembling three molecular building blocks A, B and C into the complex ABC, as illustrated in Fig. 2. Molecules A, B and C could each be DNA strands or proteins, for example. The simplest design strategy for self-assembly is to ensure that microstates with many well-assembled structures are common in equilibrium. The most likely macrostate fNg is the one that maximises Z r ðfNgÞ, or equivalently minimises F r ðfNgÞ, subject to the constraints of stoichiometry. If a large number of molecules are present, as in typical experiments, fluctuations about the most likely macrostate are relatively small in equilibrium (Frenkel and Smit 2001;Tuckerman 2010;Huang 1987;Jaynes 1957), and hence we can infer equilibrium properties purely by analysing this most likely macrostate.

Identifying the typical behaviour of a selfassembling system in equilibrium
In this Subsection, we introduce the basic mass-action equilibria underlying the equilibrium yield of self-assembling systems, and demonstrate how they arise from identifying the microstates with low F r ðfNgÞ in dilute solutions. We then discuss the key properties of the resultant equilibria. To minimise F r ðfNgÞ, we must find fNg such that no possible change of fNg due to a reaction could reduce F r ðfNgÞ. In the limit of many molecules, this task is equivalent to identifying the fNg for which F r ðfNgÞ À F r ðfN 0 gÞ ¼ 0 for every possible reaction fNg ! fN 0 g. From Eq. 15 we therefore require for all reactions k. Thus the typical equilibrium state is the one in which chemical potentials are balanced for all reactions-a widely exploited result (Nelson 2004;Huang 1987). Substituting our expression for l j r (Eq. 14) into Eq. 17, we immediately see that at equilibrium for all reactions k. In the case of A þ B ! AB, in which we have introduced the equilibrium constant K eq AþB . The quantity K eq AþB is known as a constant because it depends only on the details of the interactions within each species as represented through z j 0 ; it is independent of system volume of the number of molecules present, although it will depend on quantities such as the temperature. It is also independent of the arbitrary reference volume V 0 , since each z j 0 scales with this volume. The result is ubiquitous in physical chemistry, and immediately generalises for other reactions (Nelson 2004).
If K eq is known from earlier experiments for each reaction k, or can be predicted from underlying theory, then variants of Eq. 19 can be constructed for each possible reaction and the typical concentrations in equilibrium can be found by solving the resultant simultaneous equations (when augmented with any conservation laws). Note that a relation such as Eq. 19 exists for every reaction at equilibrium, regardless of whether the reactants are involved in other reactions. Additionally, the same procedure can be followed for not just a single reaction k, but a series of reactions. For example, the combined reactions A þ B ! AB and AB þ C ! ABC imply that the relationship is meaningful even if there is no direct A þ B þ C ! ABC reaction (which would be essentially impossible without the formation of intermediate complexes).
To design a system that can self-assemble efficiently into ABC, therefore, we should choose molecules that provide a large K eq AþBþC , and also large K eq AþBC , K eq ABþC and K eq ACþB , since ABC needs to out-compete other potential species. Similarly, low values of K eq for reactions that produce off-target species such as AABB are important in preventing mis-assembly.
An often under-appreciated fact is that reactant concentrations are important in determining yields; it is not uncommon to hear a ''melting temperature''-approximately the point at which 50% of the maximum possible yield has been reached-quoted without reference to component concentrations. However, the fractional yield in equilibrium is sensitive to the initial concentrations of reactants. Lower initial concentrations of A, B and C imply a larger reduction in the concentration of complexes in equilibrium, since the numerator in relationships such as Eq. 20 must change enough to compensate for the reduction of all of the concentrations in the denominator.
Depending on the context, it may also be advantageous to use non-stoichiometric mixtures of components to increase yields-for example, an excess of A and B relative to C when assembling ABC. Doing so significantly enhances the fraction of C molecules incorporated into ABC structures in equilibrium, at the expense of leaving a pool of A and B which cannot possibly contribute to a target. To illustrate this effect, we plot the fractional yield C AB =C total B in the reaction A þ B AB as a function of K eq AþB in Fig. 3. We consider two cases, one with a stoichiometric mixture C total B ¼ C total  Non-stoichiometric conditions can increase the fractional of certain molecules incorporated into target complexes. We plot the equilibrium fraction of B molecules incorporated into AB (f B ) by the reaction A þ B AB, as a function of K eq AþB . Yield curves are plotted for C total The importance of thermodynamics for molecular systems, and the importance of molecular... 9 A. This approach is taken in the construction of scaffolded DNA origami (Rothemund 2006); an excess of staples is added to enhance the fraction of scaffold strands incorporated into well-formed structures. Saturating the system with an excess of one type of strand is a particularly useful approach if free strands of a complementary sequence are the main potential cause of leak reactions, especially given the possibility of pipetting errors. Murugan et al. have further proposed that concentrations of reactants could be judiciously chosen to avoid the formation of undesired offpathway structures (Murugan et al. 2015). Finally, it is worth noting that the equilibrium constants K eq are determined exclusively by the properties of the reactant and product species, through the ratio of the appropriate partition functions. Thus relationships such as Eqs. 19 and 20 cannot be altered by other molecules that are not produced or consumed by the process, or by the properties of intermediate complexes. A change in the equilibrium constant, as measured through the relative concentrations in equilibrium, can only arise from a change in the biochemistry of the initial and final species. This fact will be important when we discuss the role of non-equilibrium catalysts in Sect. 7.

The meaning of the equilibrium constant, and estimating its value
Relationships between concentrations such as Eqs. 19 and 20 can also be derived by assuming that reactions obey mass-action kinetics: i.e., that rates are proportional to concentrations of reactants involved. However, the statistical mechanical approach has two important advantages that are important in the context of self-assembly. We now proceed to outline these advantages. Firstly, the fact that relationships such as Eqs. 19 and 20 hold even without a direct one-step reaction linking reactants and products, and regardless of whether the species are involved in other reactions, is made clear from this thermodynamic perspective. But perhaps even more importantly, the derivation presented highlights the physical meaning of the equilibrium constant K eq ; it is determined by the ratios of partition functions of the molecular species involved in the reaction. The quantities z j 0 are partition functions for complex j in volume V 0 ; species with favourable (low) internal energies, or many accessible states, are favoured.
Crucially, these quantities z j 0 (or at least the relevant ratios) can often be predicted by simple theoretical models. For complex models (Orozco et al. 2003;Cino et al. 2012;Marrink et al. 2007;Doye et al. 2013), ratios of partition functions are the natural quantities to extract from simulation (Ouldridge et al. 2010;Ouldridge 2012), enabling direct prediction of equilibrium constants and hence comparison to experiment. Simpler approaches such as the nearest-neighbour models of DNA and RNA thermodynamics use basic postulates about z j 0 to make analytic predictions of the ratios of partition functions, and hence equilibrium constants, of an enormous number of selfassembling systems using a small set of parameters (San-taLucia and Hicks 2004; Turner and Matthews 2010). These widely-used tools have been a fundamental component of the growth of nucleic acid nanotechnology and have facilitated the analysis of natural RNA circuitry in cells, for example in Borujeni and Salis (2016).
It is hard to overestimate the usefulness of such a predictive tool, even given its finite accuracy. Without it, the systematic design of complex nucleic acid circuits from scratch would be far more challenging-particularly in terms of eliminating unintended interactions-and would require the measurement of many equilibrium constants K eq for each design.
It is instructive to consider how the nearest-neighbour model can be used by utilities such as Nupack (Dirks et al. 2007) to predict the concentration of complexes formed in equilibrium after mixing nucleic acids. Firstly, given a set of strands, all possible complexes below a certain size can be identified (Fig. 4). For each of these complexes j, it is Fig. 4 A schematic illustration of the calculation of equilibrium constants by algorithms such as Nupack (Dirks et al. 2007), based on nearest-neighbour models of nucleic acid thermodynamics (SantaLucia and Hicks 2004; Turner and Matthews 2010). Firstly, all possible complexes up to a maximum size (in this case 2) are enumerated. For each of these complexes, and the individual strands, the possible-base pairing macrostates are identified and assigned a free energy relative to the unstructured state. Summing over all contributions for one complex gives the partition function of the complex relative to the unstructured constituent strands; combining this with similar quantities estimated for the single strands themselves gives K eq AþB then necessary to estimate the equilibrium constant K eq j for complex formation from the constituent single strands at the appropriate temperature-with these equilibrium constants, the task reduces to solving a set of simultaneous equations involving expressions such as Eqs. 19 and 20 and conservation laws.
Equilibrium constants are estimated by working at the level of macrostates defined by the pattern of base pairing-a finer resolution than simply identifying the complexes present, but still far from a true microscopic enumeration of microstates. To estimate K eq AþB , for example, all base-pairing macrostates of both the complex AB and the individual strands A and B are enumerated (Fig. 4). For each macrostate i, the nearest-neighbour model predicts the standard free energy relative to a completely unstructured (base-pair free) macrostate using a small set of universal parameters that depend on the sequence of base-pair steps, and the context at the end of the continuous base-paired stations. From these free energies, the relative partition functions of macrostates can be calculated, and by summing over the partition function contributions from all macrostates for both the complex and the individual strands, the ratio z AB 0 =ðz A 0 z B 0 Þ can be estimated. Hence K eq AþB can be predicted through Eq. 19, and used to infer complex concentrations given the total concentrations of all strands.
5 Thermodynamics as a basis for the design of kinetic models

The importance of kinetics
Designing an equilibrium state to be consistent with a high yield of a self-assembling structure is the typical approach taken when engineering a self-assembling structure. Indeed, this is the main strategy employed hitherto in the field of nucleic acid nanotechnology when designing structures. However, the existence of a high yield in equilibrium doesn't guarantee successful assembly in finite time-the system might become trapped in metastable states, and fail to approach the equilibrium yield over a reasonable timescale. To understand why, it is important to develop kinetic models that can explore dynamical trajectories taken by systems. Additionally, static self-assembled structures are also not the only possible type of molecular system. There has been recent interest in non-equilibrium or dissipative self-assembly (Timonen et al. 2013), in which assembled structures are maintained in a non-equilibrium rather than an equilibrium steady state by a continuous input of energy. Indeed, this is almost a minimal description of a living organism. On a more detailed level, natural molecular circuits generate motion (Nelson 2004;Alberts et al. 2002), act as oscillatory clocks (Lubensky et al. 2017), and dynamically sense their environment (Mehta and Schwab 2012;Mehta et al. 2016; Govern and ten Wolde 2014a, b); researchers are now designing artificial systems with similar functionality (Zechner et al. 2016;Stricker et al. 2008;Zhang and Seelig 2011). In these dynamical systems, reaction kinetics is inherently important.
We now give a detailed discussion of the influence of thermodynamics on the kinetics of molecular systems. We start with a discussion of the biochemical master equation as a fundamental description of biochemical kinetics in Sect. 5.2, including pitfalls associated with poorly-chosen macrostates, before moving on to the constraints on the dynamics imposed by thermodynamic considerations in Sect. 5.3. Finally, we discuss how to build a dynamical model taking these constraints into account in Sect. 5.4.

Molecular systems as stochastic processes
Statistical mechanics is inherently probabilistic; the Boltzmann distribution (Eq. 1) is, after all, a probability distribution for finding the system r in a given microstate. It is therefore natural to describe system dynamics using a stochastic (or random) process (Van Kampen 2007) over these microstates. At a given time t, the system occupies a microstate ðx; pÞ with a probability density P r ðx; p; tÞ; system dynamics lead to an evolution of this distribution over time. After a long time, t ! 1, and in the absence of external driving, the system should relax towards a stationary (time-independent) distribution given by the Boltzmann distribution P eq r ðx; pÞ of Eq. 1 (assuming the state space is ergodic, as is typical).
For an ideal memoryless environment, the stochastic process should be Markovian (Van Kampen 2007) i.e., the future evolution of P r ðx; p; tÞ only depends on the past via the current value of P r ðx; p; tÞ. Equivalently, the outcome of leaving a microstate ðx; pÞ is independent of the route by which the system reached ðx; pÞ. Of course, when analysing molecular systems, we typically work at the level of chemical macrostates. We describe the system through the abundances of species, or perhaps the hydrogen-bonding patterns, rather than individual positions and momenta. In this case, our state space is discrete and the stochastic process involves a series of transitions by which the system undergoes discrete hops between macrostates.
Generally, it is assumed that the stochastic process is also Markovian at the macrostate level (Seifert 2011(Seifert , 2012Hill 1989). In other words, transitions from macrostate i to macrostate j occur at a fixed rate k ji (frequently zero), meaning that the time spent in state i prior to a transition is exponentially distributed; the average lifetime is given by The importance of thermodynamics for molecular systems, and the importance of molecular... 11 s i ¼ 1= P j k ji ; and the destination state is independent of the state from which the system entered i and the length of time spent within i. Formally, given this assumption, the probability distribution evolves according to the familiar Master equation (Van Kampen 2007). This type of model is so universal that it is often not realised just how big an assumption it is to treat coarse-grained dynamics as Markovian.
Consider Fig. 5, in which we have divided a two-dimensional microstate space into macrostates in an essentially arbitrary fashion, and plotted a sample trajectory of a simple process that is Markovian at the microstate level. Fundamentally, even if the dynamics is Markovian at the microstate level, coarse-graining introduces memory into the process. Trajectories that enter macrostate i from macrostate j are close to the border between the two, and hence are likely to quickly cross back into j. As a result, both the transition time and transition destination should, in general, depend in a complex manner the previous macrostates visited by the system, and the length of time for which the current macrostate has been occupied.
For a more concrete example, consider the strand exchange reaction shown in Fig. 6. Strand exchange is a basic process underlying much of DNA computation Zhang and Winfree 2009;Chen et al. 2013). It might be tempting to describe the system using three macrostates: A bound to B only; an ABC complex; and A bound to C only. Indeed, such an approach is sensible if one is only interested in the relative abundances of complexes in equilibrium. However, the system dynamics cannot be well-described by a Markov process at this level. The need to initiate and complete branch migration to exchange base pairs between AB and AC duplexes means that an ABC complex formed by AB þ C ! ABC is in reality much more likely to dissociate into AB þ C than a complex formed by AC þ B ! ABC, violating the assumptions of a Markov process. Even splitting the ABC macrostate into two separate macrostates, depending on whether AB or AC contains the most base pairs (Zhang and Winfree 2009), does not provide a satisfactory treatment of the system; it is necessary to resolve macrostates on at least the base-pair level to provide a predictive Markov model of system dynamics . A similar example in cells is the translation of RNA (Reuveni et al. 2011); if ''ribosome bound to RNA'' is treated as a single macrostate, rather than modelling codon incorporation as individual steps, a highly unrealistic exponential distribution of times for translation will be obtained.
So when is a Markov assumption reasonable? This is a subtle problem, but the basic idea is that macrostates must be carefully chosen so that a transition involves passing through an unfavourable (high energy or low entropy) set of microstates around the boundary. In this case, the system typically spends a long time fully exploring each macrostate before a sudden hop to a neighbouring one. Transitions, which are complete when the system has fully crossed the unfavourable microstates on the boundary, are rare-the time taken waiting to see a transition is long compared to the duration of the transition. When these assumptions hold, any memory of the previous macrostate is lost whilst exploring the new one, allowing the Markov assumption (Seifert 2011(Seifert , 2012Hill 1989). A more detailed discussion of when coarse-graining is possible, and how to do it systematically, is presented in Bo and Celani (2017).   Fig. 6 A practically-relevant example of a coarse-graining into macrostates that is inappropriate for treatment with Markovian dynamics. Consider a DNA-based toehold-exchange reaction, in which B and C compete for binding to A via strand displacement. It is perfectly possible to define three macrostates as shown above, and indeed doing so would enable the calculation of molecular abundances in equilibrium. However, it is inappropriate to model the dynamics as Markovian, because the microstates of the ABC complex are not typically fully-explored before it dissociates A simple example is given by comparing the trajectories of particles in two potential energy wells. In the first case, the well is quadratic, with a single minimum at x ¼ 0; in the second, it is quartic, with two minima at x ¼ À1 and x ¼ þ1 (see Fig. 7a). In both cases it is formally possible to define macrostates according to whether the particle occupies x\0 or x [ 0. However, only in the first case, in which the transition from x\0 to x [ 0 is associated with climbing over an unfavourable energy barrier, is it reasonable to describe this process using a Markov model at the macrostate level (compare the two trajectories in Fig. 7b). Returning to the toehold exchange reaction in Fig. 6, the three-state Markov model fails because a system that enters the ABC macrostate isn't likely to fully explore that macrostate prior to leaving it. Since branch migration is slow , the strand that has just bound will often detach before all branch migration intermediates have been explored, invalidating the requirements for a Markov model at the level of these macrostates.
From this point onwards we will assume a biomolecular system with discrete macrostates that have been well-chosen. Thus the dynamics is Markovian and can be well described by a master equation with rate parameters k ji (Van Kampen 2007). What we will say will also be applicable to a full description at the level of microstates ðx; pÞ.

Detailed balance
The transition rates k ji between all pairs of macrostates fully define system behaviour, given a particular initial condition. Knowledge of system thermodynamics (i.e., the free energy of macrostates F r ðiÞ) doesn't specify kinetics, but it does place strong and important restrictions. Firstly as noted in Sect. 5.2, the system should eventually relax to the equilibrium (Boltzmann) distribution over macrostates. Thus knowledge of the equilibrium distribution constrains the set of rate parameters fk ji g-they must result in the appropriate steady state.
Thermodynamic systems, however, are constrained much more tightly than this. A feature of equilibrium is that there should be no tendency of reactions to flow in one direction (Van Kampen 2007). This fact was alluded to when the concept of equilibrium was introduced in Sect. 2; in equilibrium, there should be no net flux of systems between any pair of microstates. For example, if a single molecule can occupy three different conformational macrostates, X, Y and Z, one could imagine a steady state with a systematic flow X ! Y ! Z ! X (Fig. 8a). Such a steady state is impossible in equilibrium. If it existed, it would be possible to use the systematic flow to power molecular machines (Sect. 7), which would violate the second law of thermodynamics. This feat would be analogous to powering a water mill using a completely flat and undisturbed pond.
Instead, in equilibrium, each individual transition must be balanced by its microscopic reverse (Fig. 8b); the total rate at which X ! Y transitions are observed should balance the total rate of Y ! X. This feature is known as the principle of detailed balance, and it is a central plank of the thermodynamics of molecular systems. In terms of the rate parameters, equating the total number of transitions per unit time, a, in both directions in equilibrium gives which clearly holds in Fig. 8b, but not Fig. 8a. Thus k ji k ij ¼ P eq r ðjÞ P eq r ðiÞ for a simple chemical system obeying detailed balance. The ratio of rate parameters k ji =k ij is then determined  Fig. 7 Macrostates that are separated by well-defined (free-) energy barriers lead to rare event kinetics and allow for a Markovian treatment. a Two potentials: a quadratic well with a single minimum, and a quartic well with two minima separated by a barrier. b Example trajectories of (overdamped) Brownian dynamics in these wells. For the quartic well (trajectory labelled ''quart'') transitions are rare events and each well is sampled representatively prior to transitions. It is therefore reasonable to assume Markovian dynamics between macrostates defined by x [ 0 and x\0. By contrast, for the quadratic single well (''quad'') this is not possible The importance of thermodynamics for molecular systems, and the importance of molecular... 13 exclusively by the difference in free energies between macrostates i and j. It should be immediately evident that rate parameters constrained in this way will necessarily lead to the appropriate equilibrium distribution P eq r ðiÞ / expðÀF r ðiÞ=k B TÞ, since by definition all pairs of transitions will cancel out if P r ðiÞ ¼ P eq r ðiÞ. Importantly, since the rate parameters are constant, the relationship between rate constants in Eq. 22 is a feature of the dynamics and holds for any P r ðiÞ, even when the system is out of equilibrium and P r ðiÞ 6 ¼ P eq r ðiÞ. An important consequence of detailed balance is that we can now see which transitions will tend to occur spontaneously in representative trajectories. If F r ðiÞ [ F r ðjÞ, we expect to see systems move from i to j more quickly than they would from j to i. In equilibrium, this tendency is compensated for by the relative population size, yielding detailed balance. This is true even if it takes several steps to reach j from i, since Eq. 22 holds for each of those steps. If ðF r ðiÞ À F r ðjÞÞ=k B T ) 1, then we expect to see i ! j occur spontaneously during trajectories, but we will essentially never observe a system starting in j and transitioning to i (unless we force it from the outside). In the context of a chemical reaction, for example A þ B AB, we expect to see systematically more transitions from left to right whilst D AþB F r ¼ l AB r À ðl A r þ l B r Þ\0. On a large scale with many molecules, we will effectively see the reaction spontaneously proceed in one direction determined by the sign of D AþB F r , until equilibrium (D AþB F r ¼ 0) is reached.

Parameterising a kinetic model for molecular systems
The relative rate of forwards and backwards transitions is thus fixed by the associated change in macrostate free energy. In this subsection, we discuss how this simple selfconsistency relation places strong constraints on the rate parameters that can be used to describe a biochemical system, if it is to be thermodynamically well-defined. Doing so is particularly important in developing physically reasonable models of self-assembly, polymerization or depolymerization Nguyen and Vaikuntanathan 2016;Andrieux and Gaspard 2008;Sartori and Pigolotti 2015;Gaspard 2016), and also nanotechnologically important reactions such as strand exchange (Zhang and Winfree 2009). Unless this physical constraint is applied when parameterising such models, unphysical cyclic flows of reactions will be observed in steady state. Preserving the relationship between free energy change and relative reaction rates is also essential if the costs of fuelconsuming systems (Mehta and Schwab 2012;Lan et al. 2012; Barato and Seifert 2015;), which will be discussed in Sect. 7, are to be understood. Note that this requirement of thermodynamic self-consistency is not unique to a particular approach to modelling a biochemical reaction network. We have been considered a fully-stochastic description at the level of the chemical master equation, Van Kampen (2007), Gillespie (2009), but similar reasoning also applies to modelling performed in the deterministic limit (Gillespie 2009) or using a chemical Langevin approximation (Van Kampen 2007;Gillespie 2009). All of these approaches are potentially thermodynamically well-defined, but care must be taken when parameterising them.
It is sufficient to consider how transition rates defining the chemical master equation should be chosen, since these directly determine the deterministic and Langevin approximations to the system. It is typical to assume that in dilute solutions, rates per unit volume are proportional to 2s -1 2s -1 X 1s -1 Z Y  Fig. 8 Detailed balance in a discrete state Markov process. a and b represent Markov processes defined on the discrete state space X, Y, Z. These processes both have the same stationary distribution P stat r ðiÞ, shown above. However, system a exhibits a tendency to flow in a clockwise direction X ! Y ! Z even if P r ðiÞ ¼ P stat r ðiÞ, whereas system b exhibits detailed balance; the net number of transitions between each pair of states cancels at P r ðiÞ ¼ P stat r ðiÞ ¼ P eq r ðiÞ, as can be explicitly verified concentrations of reactants. For example, in the reaction A þ B ! AB, the macrostate-dependent rate for the binding transition k N AB þ1;N AB N A ; N B ð Þis given by where / bind is a bimolecular rate constant, and the reverse rate is given by where / unbind is a unimolecular rate constant for unbinding. Such a choice is potentially thermodynamically consistent, but Eq. 22 combined with Eqs. 14 and 16 implies a specific relationship between the two rate constants. The familiar result is: We have thus seen how a constraint on the transition rate parameters in the chemical master equation translates into constraints on the familiar first and second order rate constants of association and dissociation. Similar results can be obtained for other reactions.
When constructing a kinetic model, it is advisable to start from a model for the free energies F r ðiÞ, and then impose Eq. 22 or Eq. 25 for each pair of reactions. Indeed, it is extremely hard to write down directly a set of transition rates fkg or rate constants f/g that respect a sensible free energy model for a complex system. Typically, the free energy difference implied by two distinct pathways between microstates will be inconsistent, resulting in unwanted steady-state reaction fluxes etc.
As an example, we might consider a small DNA nanostructure, whose assembly involves binding and unbinding of duplex sections. A simple assumption might be that all binding transitions have the same bimolecular rate constant (perhaps $ 10 6 M s À1 ), which sets k ij for all binding transitions. The inverse k ji transition rates then follow from the free energy of binding, F r ðiÞ À F r ðjÞ, which might be estimated via the nearest-neighbour model.
In general, both the problem of estimating F r ðiÞ and the absolute rates of one of each pair of transitions can be subtle Flamm et al. 2000;Schaeffer et al. 2015;Dannenberg et al. 2015), and the consequences for the dynamics can be profound. In particular, it is not always obvious how a change in D k F r might be manifest in the rates. For example, consider the association of two DNA strands A and B that can form hairpins in the singlestranded state. These hairpins serve to make the standard free energy of formation of a duplex D AþB F 0 less favourable (less negative), and hence reduce the ratio / bind =/ unbind through Eq. 25. It might be natural to assume that hairpins essentially reduce the rate of binding / bind , and indeed hairpins are deliberately used for this purpose to create metastable systems (Turberfield et al. 2003;Yin et al. 2008;Choi et al. 2014;Meng et al. 2016). However, experimental evidence suggests that in some cases, / bind is reduced by much less than expðÀD AþB F 0 =k B TÞ, implying that in fact most of the reduction in free energy is manifest as an increase in the off-rate / unbind Gao et al. (2006). This observation is supported by detailed simulation (Schreck et al. 2015), in which it is observed that hairpins form prior to full dissociation and stabilise the partially-melted state, accelerating unbinding. Similarly, experimental work supported by theory shows that the overall forwards rate of strand displacement and exchange reactions can be adjusted by orders of magnitude at a fixed overall standard free energy of reaction (Zhang and Winfree 2009;Machinek et al. 2014).
Despite these subtleties, simple kinetic models of complex molecular systems can provide deep insight into function. For example, the exquisite data provided by Zhang and Winfree (2009) on the rates of DNA strand displacement and toehold exchange reactions allow thermodynamically consistent modelling of those key processes, underlying the systematic design of complex molecular circuits using tools such as DSD (Lakin et al. 2011). In turn, the physical basis of these parameters have been probed by more detailed modelling ) at the base-pair level of description. Similarly, it has long been known that kinetics, rooted in a free-energy landscape, is fundamental to understanding how proteins fold (Dill and MacCallum 2012). Recent work has explored and manipulated the kinetics of assembly DNA origami and DNA brick assembly in a similar fashion, highlighting the importance of subtle cooperative effects Dunn et al. 2015;Reinhardt and Frenkel 2014;Song et al. 2012).

The consequences of reversibility in biochemical reactions, and the relation to thermodynamic reversibility
Reversibility is an important term in both thermodynamics (reversible processes generate no entropy) and the literature on chemical reaction networks (reversible transitions can occur backwards or forwards). In this section, we first explain why the microscopic reverse of all observed reactions must be possible, and how reaction networks modelled with one-directional transitions should be understood. We then demonstrate the consequences of microscopic reversibility for a particular molecular computation algorithm, highlighting potential issues with neglecting microscopic reversibility in system design. Finally, we contrast the meaning of thermodynamic reversibility with that of microscopically reversible transitions in an attempt to clarify a frequently misunderstood dichotomy.

Including reverse transitions in modelling
One immediate consequence of Eq. 22 is that if a forwards reaction is possible, so too is its reverse, with relative rates determined by the initial and final free energies. This concept is known as the principle of microscopic reversibility, and has far-reaching consequences. This principle suggests that both forwards and backwards transitions should always be explicitly included in any model. However, successful modelling is often done by treating certain reactions as totally irreversible. For example, nobody models transcription of RNA in cells by considering the reverse process by which RNA returns to the DNA and is destroyed base by base whilst in contact the gene that encoded it (Bennett 1982). Similarly, strand displacement kinetics is often fitted by assuming that the reaction proceeds to 100% completion (Machinek et al. 2014).
In general, whether it is reasonable to neglect reverse transitions depends upon the context, and the purpose of the modelling. Processes such as RNA transcription involve constant input of chemical fuel (this situation will be discussed in Sect. 7), and are thus far from equilibrium; in such cases, backwards transitions can be rendered irrelevant by the presence of alternative pathways. For example, RNA is digested by exo-and endonucleases in the cell, instead of needing to be destroyed by the reverse of transcription. Unless the modeller is interested in the actual thermodynamic work being done by the fuel in such cases, or in the case of relatively weak driving, explicit modelling of the reverse reactions is unimportant. For strand displacement, with a sufficiently long toehold, the equilibrium state is so biased towards the product of displacement that reverse reactions, leading to a residual concentration of the input, can be neglected. In many other contexts, however, it is important to include microscopically reversible transitions, because the free energy of transitions is relatively weak (as is the case of selfassembly near the melting temperature); because the overall thermodynamics is of interest to the modeller; or because reverse reactions, despite being typically slow, have a profound impact on system behaviour.
For example, in the field of molecular computation and algorithms, it is common to assume that reactions can be made totally irreversible (Zechner et al. 2016;Chen et al. 2012Chen et al. , 2013Cummings et al. 2015;Briat et al. 2016). An example is an algorithm for computing the parity (even/odd nature) of an initial number of molecules of type A, as discussed by Cummings et al. (2015) and illustrated in Fig. 9a. The algorithm introduces two other species, B and C, and the (assumed microscopically irreversible) reactions If the system is initially prepared with N 0 A molecules of type A, and 1 molecule of type B, it can be seen that the reactions will interconvert B and C (retaining N B þ N C ¼ 1) whilst reducing the number of A molecules. The final state will be a single molecule of type B if N 0 A is even, and a single molecule of type C if N 0 A is odd. The output of the network is thus the state of the B / C molecule in the limit of long time, which reports on the parity of N 0 A . Note that these reactions can in principle be implemented with DNA using a large supply of implicit ancillary molecules (Chen et al. 2013;. What happens if we consider reverse reactions, so that C ! A þ B and B ! A þ C are also possible, as in Fig. 9b? Then there is a finite chance that a system observed in the long-time limit will contain a B / C molecule that does not reflect the parity of N 0 A . Specifically, let us assume that for a single A and a single B being converted into a single C in the volume V r , in the absence of all other molecules of type A, B and C, the free-energy difference between macrostates is (Fig. 9b). For simplicity, let us also assume that the same free-energy difference applies to These free energies include the contributions from any implicit ancillary molecules. Provided the ancillary molecules are in excess (their concentrations are essentially unaffected by the reactions involving A, B and C), then the probability of observing N A molecules of type A in equilibrium is given by Fig. 9 Schematic illustration of the parity-computing algorithm of Cummings et al. (2015). a Reactions consume species A, and switch B into C and vice versa. Thus a system initated with an odd number N 0 A of A and a single B will result in an isolated C (as shown). Alternatively, a system initiated with an even number N 0 A of A molecules will result in a single isolated B. b The consequence of microscopic reversibility is a finite backwards rate for all transitions; we take the overall difference in free energy between N A ¼ 1 and N A ¼ 0 to be DF parity P eq r ðN A ¼ 1Þ ¼ P eq r ðN A ¼ 0Þ expðÀDF parity =k B TÞ; Each term includes an additional factor, e ÀDF parity =k B T , to account for the additional reaction that must take place to reach N A ¼ 0. The N A ! factor accounts for the fact that the free energy difference between states with N A and N A À 1 depends on the number of A present, as we previously saw in Sect. 3. One way to confirm the exact dependence is to note that the rate for C ! A þ B and B ! A þ C should be independent of N A , but A þ B ! C and A þ C ! B should occur with an overall rate proportional to N A . Incorporating this into the free energy gives the factors in Eq. 27 (a logarithmic growth in the free energy difference with N A ). When N A is odd, the readout from the reporter molecule B / C gives an incorrect readout for the parity of N 0 A . It can be seen that the even and odd terms of Eq. 27 correspond to terms in the expansion of hyperbolic functions of e ðÀDF parity =k B TÞ . Thus For DF parity large and negative-when the interconversion of an isolated A and B into an isolated C is favourable-the algorithm is accurate. For lower values of DF parity , the accuracy is reduced. Of course, in an abstract design it is possible to imagine DF parity is as large as possible, but the need to do this should be noted. It is also worth highlighting the fact that DF parity is not the standard free energy of the reaction; it is the free energy difference between a macrostate with a single A and a single B and a macrostate with a single C, in volume V r . In fact, it can be shown that (Ouldridge et al. 2010), where D AþB F 0 is the standard free energy of reaction, calculated in the reference volume V 0 ¼ 1=C 0 . Thus the larger the system volume V r , the larger the standard free energy must be to give the same accuracy. If N 0 A is large, V r will also have to be large to ensure that the system is dilute and functions as intended. Therefore the standard free energy required for accurate computation will be very large and negative, and will become more negative logarithmically with system size at fixed initial concentration C A . The robustness of other molecular algorithms to finite reverse reaction rates, and possible ways to mitigate these effects, remain important open questions -although it seems likely that the least robust algorithms will be those that require a specific macrostate as an output.

The meaning of thermodynamic reversibility
The microscopic reversibility of individual transitions should not be conflated with the idea of thermodynamic reversibility, despite the unfortunate similarity of nomenclature. Thermodynamic reversibility is not the property of an individual transition or even a set of rate parameters describing a system. Rather, it is a property of an entire process in which an experimentalist or machine in the environment manipulates the system from the outside, applying some protocol (changing the conditions with time, as shown in Fig. 10a). If the system and environment are both be restored to their initial conditions by a time-reversed protocol (Fig. 10b), then the process and its timereversed counterpart are thermodynamically reversible (Adkins 1987).
Reversible processes are important in thermodynamics because they do not increase the entropy of the universe, given by the sum of the entropy of r and its environment R (Adkins 1987). The entropy of the universe (an isolated system) cannot decrease, and excess entropy generation corresponds to wasted effort (or work). Hence reversible processes are the most efficient way of manipulating a system between a given start and end point. Historically, there has been significant interest in the minimal entropic cost of certain computational procedures, particularly the f f f f f f Fig. 10 The meaning of a reversible process. A schematic illustration of a process in which a time-dependent protocol (perhaps a compressing force) is applied to r, leading to change in r and exchange of heat with the environment R. The process is reversible if applying a time-reversed protocol as in b leads to both r and R being returned to their initial conditions The importance of thermodynamics for molecular systems, and the importance of molecular... manipulation of a single bit, and the possibility of reversible computing (Bennett 1982;Landauer 1961;Sagawa and Ueda 2009).
It is often proposed that molecular systems might be candidates for the construction of reversible computers (Mehta and Schwab 2012;Thachuck and Condon 2012;Thachuk 2013), and computational architectures with microscopically reversible dynamics have been analysed (Thachuck and Condon 2012;Thachuk 2013). Importantly, however, microscopic reversibility does not imply thermodynamic reversibility. In truth, all systems possess microscopic reversibility-ignoring it is simply a modelling assumption, as discussed in Sect. 6.1. Thermodynamic reversibility, however, depends on the initial conditions of the system, and the way in which a protocol is applied-not the intrinsic properties of a transition.
Consider, for example, a DNA-based toehold-exchange reaction (Fig. 6). We might be interested in switching the substrate strand A from a B-bound state to a C-bound state. If we set up the system to ensure that we start with a single AB duplex, and a separate C strand, and let it evolve naturally (i.e., implement a trivial protocol of ''do nothing'') the system will undergo repeated strand exchange reactions, and the final state will alternate between AB and AC. This isn't a full switch from AB to AC, but it is at least a change in the probability distribution over macrostates P r ðiÞ.
During this process, both forwards and backwards reactions occur, and continue to occur indefinitely. So is this a thermodynamically reversible process? It is not. If we were to reverse our protocol of ''doing nothing'', the system would not return to a state in which it was guaranteed to have an AB duplex; it would stay in an uncertain AB/AC state. Indeed, any overall change of a system (i.e., a change in the probability distribution P r ðiÞ) that occurs during a trivial ''do nothing'' protocol is necessarily thermodynamically irreversible, since the opposite change in P r ðiÞ will not occur under a time-reversed ''do nothing'' protocol. The actual entropy increase in the process can be calculated using the methods discussed in Sect. 8.
Note that it is not really meaningful to describe the individual reactions as reversible or irreversible in the thermodynamic sense. Once the system has relaxed to a completely uncertain equilibrium AB/AC state, strand exchange reactions will still occur from the perspective of individual trajectories. However, P r ðiÞ will not change, and the subsequent (trivial) evolution of the system is reversible. It is only the initial period of transitioning from a guaranteed AB state to an uncertain AB / AC state that is irreversible-despite the fact that it involves the same microscopic strand exchange reactions. Fundamentally, thermodynamic reversibility is a property of the overall change of the system r (through P r ðiÞ) and the environment R, rather than the individual molecular processes involved.
Does this observation prohibit reversible operations with molecular systems? In fact, it is possible to drive a change in the state of molecular systems reversibly. However, this must be done by externally changing the solution conditions in a quasistatic (slow) manner. For the above example, a reversible switch could be achieved by initially coupling the A strand to a buffer containing only B strands, and then replacing this buffer with a series of alternatives with gradually increasing ratios of C C =C B , as shown in Fig. 11. Eventually, in the limit C C =C B ! 1, the switch to AC will be complete. By reversing the protocol, we return to the initial AB configuration, and restore the buffers to their original condition-hence the process is reversible. Similar procedures for reversible copying of a bit and reversible construction of a polymer copy from a template are discussed in more detail in  and Ouldridge and ten Wolde (2017), respectively. Although perhaps impractical, they highlight the difference between a thermodynamically reversible operation, and an irreversible process in which microscopic reversibility is relevant.

Catalysis and the consumption of molecular fuel
Unlike the systems considered hitherto, living organisms do not rapidly tend towards equilibrium. Instead, they are kept out of equilibrium by a continuous supply of chemical fuel. Conceptually, we might imagine that this fuel is supplied from an enormous buffer that is not depleted on the time scale of interest, or perhaps that the fuel levels are or AB A AC C Fig. 11 A reversible switch of toehold-exchange system, achieved by coupling the volume containing strand A to buffers containing increasing amounts of strand C, and decreasing amounts of strand B. If this process is performed gradually enough, it is reversible continuously topped up by some process. Staying out of equilibrium is of course essential for living systems, since equilibrium systems are passive. In this Section we will explore some basic functionalities of systems that are continuously supplied with fuel molecules. We will see that continuous fuel consumption allows systems to establish non-equilibrium steady states. These steady-states can, for example, store the electrochemical free energy needed to propagate nerve signals, and permit molecular signalling without consumption of the upstream signalling molecule.
The prototypical example of a chemical fuel molecule is ATP, and so we will base our discussion around it. ATP consists of a sugar-base group attached to a chain of three phosphate groups (Alberts et al. 2002). All of these bonds are covalent, but phosphate groups can be removed by hydrolysis. In particular, a key reaction is the removal of phosphate to give ADP (Fig. 12a), This reaction has an associated equilibrium constant K eq ADPþP i and a standard free energy of ATP formation D ADPþP i F 0 . The cell uses the breakdown of food molecules to maintain a concentration imbalance of ATP, ADP and P i ; i.e., or equivalently Left in isolation, a solution of ATP, ADP and P i prepared with these concentrations would relax to equilibrium by converting ATP into ADP and P i . This process, which requires the disruption of covalent bonds, has extremely slow kinetics. Consequently the cell is able to build up and maintain a large concentration imbalance.
The kinetics of processes such as that in Eq. 30 can be accelerated by catalysts (Nelson 2004;Alberts et al. 2002). Catalysts provide an alternative reaction pathway between the endpoints, which might (for example) lower the barriers associated with the formation and disruption of covalent bonds, as in ATP hydrolysis. A schematic illustration of catalyst operation is given in Fig. 12b, and one might record the effective reaction as with E being the catalyst molecule. Importantly, the catalyst is released at the end of the process unchanged, and does not contribute to the free energy of the overall reaction (its stoichiometric coefficient is zero). Therefore the catalyst does not affect the equilibrium balance between its substrate molecules, it only accelerates the rate at which this equilibrium is reached (Nelson 2004). An equivalent way to view the action of a catalyst is that it accelerates both forwards and backwards rates equally. In nature, some catalysts are simply present to accelerate reaction kinetics. For example, amylase is present to accelerate the breakdown of starch into sugars. In principle, it can also accelerate the conversion of sugars into starch, but this is irrelevant in the high starch conditions in which it operates. Many catalysts act on the ATP hydrolysis reaction in Eq. 30, but simply accelerating the equilibration of this process serves no function. Doing so would merely reduce the concentration imbalance built up by the cell. The purpose of these catalysts is to couple ATP hydrolysis to other reactions through the details of their internal biochemistry, and thereby use the free energy of ATP hydrolysis to drive those reactions in one way or another.
A clear example is given by ion pumps, molecular machines that reside in membranes, and transport ions across these membranes (Fig. 13). Without coupling to ATP hydrolysis, these ion pumps could only facilitate the equilibration of ions on either side of the membrane (i.e., allow current to flow until the chemical potential of the ions is equal on both sides; l ion in ¼ l ion out ). If, however, outwards ion transport is tightly coupled to ATP breakdown, then the overall free energy of pumping an ion outward is which can be negative even if l ion in À l ion out [ 0, due to the imbalance of chemical fuel molecules maintained by the cell. If the non-equilibrium chemical potentials of the ATP, ADP and P i are maintained indefinitely, the ions will eventually reach a non-equilibrium steady state in which the outwards pumping is balanced by leaks back through other channels.
The non-equilibrium initial state of the fuel molecules is effectively a store of useful work. This store can be Fig. 12 a The breakdown of ATP into ADP and P i . b A catalyst E can enhance the reaction rate by providing an alternative pathway; the catalyst is not consumed by the overall reaction The importance of thermodynamics for molecular systems, and the importance of molecular... 19 consumed to drive another process-in this case, the transfer ions against their chemical potential bias. The maximum chemical potential bias l ion in À l ion out against which progress can be made is simply the excess free energy change associated with breakdown of a single ATP, l ATP r À ðl ADP r þ l P i r Þ. Transferring ions across membranes is very much like charging a capacitor, and cells use these capacitors to drive other processes, including the firing of nerve cells. There are numerous other natural processes in which molecular fuel consumption is used to drive a coupled reaction. Molecular motors such as myosin, kinesin and dynein catalyse ATP breakdown to bias the direction in which they walk along a track (Nelson 2004;Howard 2001;Alberts et al. 2002). Without coupling the forwards step to ATP hydrolysis, forwards and backwards steps would be equally likely, since all walker binding sites on the track have equal free energy by symmetry.
By coupling continuous fuel consumption to other molecular reactions via catalysts, it is therefore possible to drive those other reactions away from their natural equilibrium (Qian 2007;Beard and Qian 2008). A particularly important example are push-pull motifs, which are ubiquitous in the signalling mechanisms which pass information around the cell and illustrated in Fig. 14. In these small networks, the presence of active upstream catalysts leads to activation of downstream substrates, which can in turn propagate or respond to the signal.
Each push-pull motif consists of a protein that can be switched between its active (X Ã ) and inactive (X) states by binding of P i to one (or more) amino acid residues. If this ''phosphorylation'' could only occur through binding to and unbinding from P i in solution, then the activity level would swiftly tend towards that determined by the equilibrium constant of binding and the abundance of P i , Importantly, any upstream catalyst could only enhance this convergence to equilibrium, it could not change it. As discussed in Sect. 4.1, relative equilibrium abundances are determined exclusively by the free energies of reactants and products-the concentration of a transiently involved catalyst is irrelevant.
However, if ATP is present, phosphorylation can also occur via transfer of phosphate from ATP: If the ATP, ADP and P i concentrations are maintained such that D ADPþP i F r [ 0, it is impossible for the X=X Ã subsystem to reach equilibrium. The series of reactions shown in Fig. 14a will necessarily occur much more frequently on sample trajectories than the counterpart The reactions in Eq. 37 will occur more frequently since the net result of the reactions in Eq. 37 is the breakdown of ATP, whereas the net result of the reaction in Eq. 38 is the + μ in ion μ in out ATP ADP P i ion pump membrane Fig. 13 An illustration of the principle of an ion pump. The pump systematically drives ions from inside the membrane to outside. By coupling the ion transition to ATP ADP þ P i , ions can be systematically pumped against a chemical potential bias l ion out À l ion in ! 0. This pumping is possible due the large and negative contribution of the breakdown of ATP to the free energy difference associated with the transition Fig. 14 A push-pull motif. The downstream molecule X can be converted between its phosphorylation states by two pathways: by exchanging phosphate with ADP/ATP or solution. Catalysts can accelerate one or both of these reactions; in (a), a catalyst Y is shown accelerating interactions with ATP/ADP. The overall negative free energy change of ATP breakdown favours the direction of reactions shown, rather than their reversed counterparts, and so X=X Ã effectively undergoes a cycle. As a result, a signal related to the concentration of Y can be passed onto X Ã , which then propagates the signal further. This function of the push-pull motif is illustrated in the simplified diagram in (b) synthesis of ATP from ADP and P i , and the sign of The protein X systematically tends to be converted into X Ã by one pathway, and converted back into X by a completely different pathway (Fig. 14a). From the perspective of X=X Ã , detailed balance appears to be violated, and therefore the coupling to chemical fuel drives the X=X Ã subsystem out of equilibrium. As a result, when the hydrolysis-driven cycle of Eq. 37 becomes dominant relative to Eq. 38, the steady state ratio C X Ã =C X is determined purely by the relative rates of the two reactions within that cycle, Since the dominant reactions that interconvert X and X Ã are not a microscopic reverse pair, thermodynamics places no restrictions on their relative rates. In particular, it is perfectly possible for a catalyst to accelerate the kinetics of one reaction and not the other. The output ratio C X Ã =C X can then be sensitive to the concentration of an upstream catalyst (Y in Fig. 14a) that accelerates the exchange of phosphate between X and ATP. In this manner, signalling cascades can pass on information on the concentration of active upstream catalysts, illustrated schematically in Fig. 14b, as required. We emphasize that it would be impossible for Y to influence the X=X Ã ratio in this way without consuming fuel, or binding to the downstream molecule and remaining bound (i.e., with Y itself being consumed by the reaction).
We emphasize that the violation of detailed balance for X/X Ã only occurs because the fuel molecules are continuously being consumed, which prevents the X=X Ã system from reaching equilibrium. If the root source of the chemical fuel was also explicitly modelled as part of an extended description, we would see the system as a whole relaxing towards equilibrium through the reactions that turn over ATP; the apparent non-equilibrium steady state of X=X Ã is just a psuedo-steady state that arises because the dynamics of X=X Ã are much faster than that of the extended system as a whole. Furthermore, the fundamental relationship between transition rates derived from detailed balance, Eq. 22, still holds: the relative rate of each microscopic reverse pair of forward and backward transitions is still determined by the underlying DF in the usual way, with the contributions from the fuel molecules to DF taken into account.
The above catalytic activation motif is extremely powerful. Since the upstream molecule acts catalytically, it is not consumed in the act of passing on the signal. It is thus able to interact with other downstream proteins to either amplify or branch the signal (Mehta et al. 2016). Moreover, the downstream readout's persistent modification allows it to keep a long-lived record of the state of the upstream protein, permitting time-integration of signals (Govern and ten Wolde 2014a, b). These features are only possible in a motif that incorporates catalytic signal propagation, which in turn relies on coupling to a non-equilibrium fuel source.
Driving of reactions by coupling them to a non-equilibrium fuel supply is a central motif in natural systems (Nelson 2004;Qian 2007;Beard and Qian 2008); other examples include enhancement of substrate selectivity through ''kinetic proofreading'' (Hopfield 1974;Ninio 1975), important in replication, transcription and translation of nucleic acids; and maintenance of oscillations related to cell cycle (Tostevin and Howard 2006) and circadian clocks (Lubensky et al. 2017). Needless to say, these fuel-consuming processes are widely exploited in synthetic biology, and enzyme-driven processes in cell-free environments (Fujii and Rondelez 2012;Green et al. 2014;Stricker et al. 2008). There has also been some exploration of these ideas in nucleic acid nanotechnology, most notably in the design of autonomous DNA walkers (Bath et al. 2005(Bath et al. , 2009Wickham et al. 2011;Tian et al. 2005;Lund et al. 2010;Cha et al. 2014;Muscat et al. 2011), some of which are powered by base pairing alone (Muscat et al. 2011;Green et al. 2008;Omabegho et al. 2009). In particular, hairpins can be used as metastable non-equilibrium fuel (Turberfield et al. 2003;Green et al. 2008;Omabegho et al. 2009); the motion of the walker is coupled to catalysing reactions that are otherwise frustrated by the hairpin structure.
More generally, recent work has shown how to implement arbitrary chemical reaction networks (CRNs) as nucleic acid systems by realising each reaction as a multistage process involving ancillary molecules (Chen et al. 2013;. These ancillary molecules can function as fuel, in principle allowing catalytic driving out of equilibrium as discussed above (Chen et al. 2013). Most impressively, the Khammash group have used a nucleicacid-based architecture to implement a CRN-based noise filter powered by fuel consumption (Zechner et al. 2016). When considering the capabilities and design possibilities of CRNs, it is always worth noting whether non-equilibrium fuel is required to provide a large thermodynamic bias for the system to function as intended. Furthermore, when these implicit fuel molecules are systematically consumed by the system, detailed balance will in general be violated for the other species considered explicitly.
Notwithstanding the above examples, the common natural motif of a network powered by fuel consumption is relatively rare in non-enzymatic artificial nanotechnology. Given the uses of fuel-powered systems in nature, producing artificial analogues is an obvious goal. One major advantage would be that, given a constant supply (or sufficiently large buffer) of fuel, such systems could potentially operate indefinitely, performing repeated operations, rather than functioning as single-shot devices. Continuous operation would be essential for implementing circuits that perform functions like feedback-control (Briat et al. 2016). 8 Stochastic thermodynamics and its relevance to biochemical systems Most interesting molecular systems are out equilibrium. However, thus far we have only discussed thermodynamic quantities U eq r , S eq r and F eq r defined with respect to the equilibrium distribution. Macrostate free energies, energies and entropies, whilst useful, do not quantify the thermodynamic properties of a general non-equilibrium distribution. In this Section, we discuss how these thermodynamic ideas can be extended to individual stochastic trajectories and evolving non-equilibrium distributions over macrostates. In doing so, we will draw heavily upon the pioneering work of Crooks (1999), Esposito andVan den Broeck (2011), andSeifert (2005).
Section 6.2 suggests that entropy generation is related to the relative probability of forward and reverse trajectories. To proceed with this line of reasoning, we let z(t) be a sample trajectory generated by the underlying Markovian dynamics in the space of macrostates of the system, with initial value z(0) an final value zðsÞ. We then define P r ð½zðtÞjzð0ÞÞ as the probability of observing that trajectory given the initial condition, and P r ð½zðtÞjzðsÞÞ as the probability of observing exactly the time-reversed trajectory given a starting point of zðsÞ. Note that if a time-dependent external protocol is applied as in Sect. 6.2, that protocol must be time-reversed to calculate P r ð½zðtÞjzðsÞÞ. We ignore such a possibility for our simple biochemical systems.
The forthcoming arguments follow from a single assumption quantifying the relationship between the relative likelihoods of z(t) and its microscopic reversezðtÞ, and entropy changes (Crooks 1999;Seifert 2005;Jarzynski 2011;Seifert 2012): Here, DS R ½zðtÞ is the change in the entropy of the environment due to the trajectory z(t), given by the heat deposited therein. DS R ½zðtÞ ¼ ÀDU r ½zðtÞ=T in the absence of a time-dependent protocol, as we assume (Seifert 2011) (for these purposes, it is simplest to treat any fuel as being supplied by a large buffer that is explicitly modelled part of the system r). Eq. 39 holds very generally (Crooks 1999;Seifert 2005;Jarzynski 2011;Seifert 2012), and in particular is necessarily true for the simple molecular systems that we have considered. In fact, Eq. 39 follows fairly straight-forwardly from applying the fundamental relation derived from detailed balance, Eq. 22, at each sub-step of a trajectory.

Generalising thermodynamic quantities to nonequilibrium distributions and fluctuating trajectories
We are now ready to generalise the equilibrium quantities U eq r , S eq r and F eq r to non-equilibrium distributions over macrostates, and use Eq. 39 to derive constraints that show the power of these generalised quantities.
The obvious generalisation of the internal energy to an arbitrary distribution P r ðx; pÞ is the average of the microstate energy over P r ðx; pÞ. If the Markovian approximation for macrostate dynamics holds, then by definition the system is well-equilibrated within macrostates, even if different macrostates have non-equilibrium probabilities P r ðiÞ 6 ¼ P eq r ðiÞ. In this case, We also generalise the entropy in the same way, since the statistical quantity is well-defined for any distribution.
S r ½P r ðx; pÞ ¼ Àk B Z x;p dx; dp P r ðx; pÞ lnðP r ðx; pÞ=qÞ: When considering well-defined macrostates, this expression becomes S r ½P r ðiÞ ¼ Àk B X i P r ðiÞ ln P r ðiÞ þ X i P r ðiÞS r ðiÞ: The generalisation is slightly more complex than for the internal energy; we obtain a term corresponding to the average entropy of the macrostates and a term corresponding to the uncertainty in the macrostate. Similarly, we can generalise the free energy (Esposito and Van den Broeck 2011;Parrondo et al. 2015): Finally, it is useful to define a ''trajectory-dependent'' entropy production (Seifert 2005;Jarzynski 2011;Seifert 2012): Ds½zðtÞ ¼ DS R ½zðtÞ þ ðS r ðzðsÞÞ À S r ðzð0ÞÞ À k B ln P r ðzðsÞÞ P r ðzð0ÞÞ : This exotic quantity, when averaged over all possible trajectories z(t), will give the generalised entropy change of the entire process-hence its name. To see that P zðtÞ P r ½zðtÞDs½zðtÞ ¼ DS rþR , compare Eq. 45 to 43. Note that Eq. 45 contains a term for the entropy of the environment R, a term for the change in macrostate entropy S r , and a term related to the distribution of r over its macrostates, as required.
These generalised quantities are particularly useful due to the constraints placed on their evolution by Eq. 39. In particular, arguably the deepest result of stochastic thermodynamics is ''deriving'' the second law of thermodynamics for generalised entropies using Eq. 39. Moreover, the second law inequality follows from a fluctuation relation equality for the trajectory-dependent entropy, highlighting the underlying physics. We will now briefly outline this derivation for the relevant case of a simple chemical system that can only exchange heat with its environment. The general case is more complicated, but the ideas are similar (Crooks 1999;Seifert 2005;Jarzynski 2011;Seifert 2012).
Combining Eq. 39 and 45, we see that P r ½zðtÞ P r ½zðtÞ expðÀDs½zðtÞ=k B Þ ¼ 1: Here, P r ½zðtÞ is the probability of observing the trajectory z(t), including the initial probability of being at z(0), and P r ½zðtÞ is the probability of observing the reverse trajectory, including a distribution of initial states of the reverse trajectories given by P r ðzðsÞÞ. We can multiply by P r ½zðtÞ and sum over all possible trajectories, yielding X zðtÞ P r ½zðtÞ expðÀDs½zðtÞ=k B Þ ¼ X zðtÞ P r ½zðtÞ ¼ 1; in which we have used the fact that a sum over all trajectories z(t) is equivalent to a sum over all reverse trajectories zðtÞ. We have arrived at the celebrated fluctuation relation for entropy (Seifert 2005;Jarzynski 2011), Seifert (2012), where the average is defined over all possible trajectories z(t). The conventional second law then follows using Jensen's inequality lnhexpðf ðvÞÞi hf ðvÞi, which implies hDs½zðtÞ=k B i ¼ DS R ½zðtÞ þ S r ½P r ðsÞ À S r ½Pð0Þ or equivalently that the total entropy of system (calculated using the generalised entropy) and environment cannot decrease (Seifert 2005). The fluctuation relation is a remarkable result, implying that at the level of individual trajectories, ''negative entropy'' paths are possible but statistically unfavoured (Seifert 2005). The result effectively shows that the familiar second law follows from a more general statement about the statistical properties of trajectories, which in turn follows directly from the relatively simple assumption related to microscopic reversibility (Eq. 39). As a result, this fluctuation theorem emphasises the statistical interpretation of entropy and the second law, and their natural emergence from familiar stochastic dynamics. A host of alternative fluctuations can also be derived, depending on the context (Crooks 1999;Jarzynski 2011;Seifert 2012).
Given the definition in Eq. 44, the generalised free energy of a simple chemical system plays the same central role as the chemical free energy in equilibrium systems (Esposito and Van den Broeck 2011;Parrondo et al. 2015). Again using the fact that a simple chemical system can only exchange energy in the form of heat with its environment, DS R ¼ ÀðU½P 0 ðiÞ À U½PðiÞÞ=T (Seifert 2011). Thus TDS rþR ¼ TDS R þ TS r ½P 0 r ðiÞ À TS r ½P r ðiÞ ¼ ÀðU r ½P 0 r ðiÞ À U r ½P r ðiÞÞ þ TS r ½P 0 r ðiÞ À TS r ½P r ðiÞ ¼ ÀðF r ½P 0 r ðiÞ À F r ½P r ðiÞÞ: In more complex environments, alternative results are obtained which also included the external work done on the system (Esposito and Van den Broeck 2011;Parrondo et al. 2015;Seifert 2011). Since the total generalised entropy of Eq. 50 is guaranteed to be an non-decreasing function of time, we can rephrase the second law for s non-equilibrium molecular processes (in the absence of external work) in its most useful form: Eq. 51 is highly significant. Firstly, it emphasizes the importance of the (generalised) free energy as a thermodynamic resource in simple molecular systems. Since total F r ½P r ðiÞ can only decrease, if a process acts to increase F r ½P r ðiÞ for a subset of species, a compensatory decrease in F r ½P r ðiÞ must occur elsewhere. This observation generalises the discussion in Sect. 7 on the use of the high free energy of chemical fuel molecules to drive other components of a system out equilibrium.

Applications of stochastic thermodynamics
Many of the results of stochastic thermodynamics, including the fluctuation relation itself, are arguably more philosophically deep than immediately useful. However, the fluctuating, far-from-equilibrium nature of many biochemical systems -particularly at the single molecule level-often lends itself to analysis using these tools. In this Subsection we discuss some characteristic examples. Firstly, the generalised free energy and entropy allow us to meaningfully analyse the entropy generation (and hence irreversibility) of processes such as that considered in Sect. 6.1, when a duplex initially prepared in the AB state relaxes to an equilibrium of equal probability AB / AC, via repeated rounds of strand exchange. If the equilibrium distribution has AB and AC duplexes with equal probability (we assume that the strands are dilute enough that threestranded complex is rarely observed), then both macrostates have equal free energy. Thus P i P r ðiÞF r ðiÞ is equal in the initial and final states. Any change in generalised free energy can only arise from the second term in Eq. 44, the difference in generalised entropy due to the distribution over macrostates. Indeed, for a system initially guaranteed to be in macrostate AB, P i P r ðiÞ ln P r ðiÞ ¼ 0; whereas the uncertainty in the final macrostate gives À P i P 0 r ðiÞ ln P 0 r ðiÞ ¼ ln 2. Consequently for the irreversible strand exchange process discussed in Sect. 6.1. Note that the full definition of the generalised free energy, including the term arising from the entropy of the distribution over macrostates, is necessary to obtain this result. Perhaps even more interesting is the case when neither the initial nor the final distribution correspond to equilibrium. For example, a self-assembling structure need not reach the equilibrium distribution after a finite time, implying F r ½P 0 r ðiÞ [ F r ½P eq r ðiÞ, since the equilibrium state minimises the generalised free energy by definition. But since F r ½P r ðiÞ can only decrease with time (Eq. 51), producing a non-equilibrium distribution necessarily requires a higher initial generalised free energy than producing an equilibrium distribution. In other words, producing the non-equilibrium distribution has a greater minimal resource cost.
Recent work has analysed how higher initial free energies allow self-assembly of non-equilibrium structures (Sartori and Pigolotti 2015;Nguyen and Vaikuntanathan 2016). Intriguingly, these non-equilibrium assemblies can have a lower density of structural defects than in equilibrium (Sartori and Pigolotti 2015), suggesting a possible alternative to optimising assembly by encouraging the approach to equilibrium. Indeed, it has recently been shown that a related process, the production of polymer copies that persist after separation from their templates (as occurs in replication, transcription and translation in cells) is inherently an exercise in producing out-of-equilibrium structures. In this context, an equilibrium output is unrelated to its template, and so a high free energy initial state is required to give any accuracy at all (Ouldridge and ten Wolde 2017).
The ability to interpret the properties of fluctuating trajectories on a thermodynamic level has also proven useful in understanding biochemical systems. Molecular systems undergoing non-equilibrium stress-induced transitions in experiment can be understood using extensions to the above theory incorporating the application of external work (Collin et al. 2005;Engel et al. 2014). On a more theoretical level, an extremely common approach is to use imbalances in probability flows to infer the entropy generation (or free-energy consumption) of functional molecular networks (Mehta and Schwab 2012;Lan et al. 2012;Barato and Seifert 2015;. The entropy cost is paid by the consumption of a (typically implicit) molecular fuel molecule. Thus the resource cost of various molecular motifs, performing sensing, adaptive information-processing, timekeeping and force-generations, can be estimated and any tradeoffs between performance and cost explored. An additional tool that has emerged within this field is the ''thermodynamic uncertainty relation'' (Barato and Seifert 2015;, which imposes free-energy consumption bounds on the variability of processes that operate cyclically, as many molecular systems do. Fundamentally, this uncertainty relation is based on physical constraints imposed on system dynamics that are not captured by the fluctuation relation (Eq. 48) .
One important feature of stochastic thermodynamics is the ability to relate the relative probabilities of entire trajectories and their microscopic reverses to entropy generation (Eq. 39). This allows the simultaneous analysis and comparison of the entropy cost of distinct trajectories that move between macrostates via different pathways of different lengths. England has used this formalism to argue for a minimal bound on the entropy generation of replicators related to the overall birth and death rates (England 2013). Similarly, by considering the entropy generated by simple and more complex activation pathways, it was recently demonstrated that single-step activation is optimally efficient for signal-propagating pushpull networks of the type shown in Fig. 14b  ).

Conclusions
In this pedagogical perspective, we have attempted to outline the manner in which molecular thermodynamics impacts the behaviour of both natural and artificial biochemical systems. Starting from the basic equilibrium statistical mechanics of dilute molecular systems, we introduced the concept of biochemical macrostates and the free energy of a macrostate. We then discussed the stochastic kinetics of molecular systems, and the constraints on kinetics implied by the macrostate free energies and detailed balance-which are strong, but insufficient to fully specify dynamics. Finally, we discussed how stochastic thermodynamics allows for a consistent thermodynamic interpretation of the stochastic evolution of a far-from-equilibrium system, both at the level of individual trajectories and the probability distribution as a whole.
Simultaneously, we have tried to highlight the relevance of these concepts to molecular systems that are typically studied and engineered. Thermodynamic ideas dictate both fundamental principles and general limits of what is possible, as well as contributing to detailed mechanistic understaning and design. In particular, defining biochemical macrostates allows for modelling of biochemical equilibria, and the state of a self-assembling system that would be expected in the limit of infinitely long time. In finite time, however, a system may not even come close to equilibrium-it is therefore important to understand kinetics. Kinetics are also relevant in systems that do not reach equilibrium because they are coupled to a supply of molecular fuel. The operation of such systems, which are widespread in nature but currently less common in artificial contexts, can be understood from the overriding tendency of the fuel molecules to relax towards their equilibrium. Finally, by defining entropies and free energies of trajectories and non-equilibrium distributions, stochastic thermodynamics allows for an analysis of the resource cost of various far-from-equilibrium processes that are important to both natural and artificial systems.
At the same time, we have emphasised underlying assumptions that are often given insufficient consideration, and common misconceptions. For example, species concentration is an important component in the stability of self-assembling structures; a ''melting temperature'' should never be defined without reference to concentration, and standard free energies are not directly meaningful at typical molecular concentrations. Markov models at the level of biochemical macrostates are only appropriate if the macrostates are well-defined so that transitions between them are rare events. Kinetic models of systems that are not subject to driving should exhibit detailed balance in equilibrium. This fact is sometimes ignored without careful consideration of the consequences; the assumption of perfect irreversibility can have major effects, depending on the context. Finally, catalysts can only influence the yield of downstream substrates if catalysis is coupled to a supply of non-equilibrium fuel.
The flow of understanding isn't solely in one direction, however. Biomolecular systems can also contribute to the understanding of fundamental thermodynamics. Most obviously, although stochastic thermodynamics wasn't developed solely for molecular systems, it finds perhaps it most natural application there. Important processes in both natural and artificial molecular systems involve single molecules undergoing large fluctuations that are essential to function. Living systems are, almost by definition, kept in a far from equilibrium state. Pursuing important questions in such systems will thus drive understanding of the underlying thermodynamic principles, and the techniques used to study them. For example, many authors are currently probing the deep connections between information theory and (stochastic) thermodynamics in a quest to understand the limits and capabilities of cellular sensing, signalling and adaption networks (Govern and ten Wolde 2014b; Barato et al. 2014;Ito and Sagawa 2015).
Much of the difficulty in understanding fundamental thermodynamics stems from interpreting the results, rather than from mathematical complexity. This is partly because many of the concepts, beginning with entropy itself, are inherently abstract. As a result, debate persists about the proper interpretation of relatively fundamental systems and concepts (Maroney 2005;Ladyman et al. 2007;Norton 2011;Dunkel and Hilbert 2014)-debates that we have ignored in this perspective. Biomolecular systems, however, are inherently concrete-even when modelled in a simplistic manner. They can thus serve to demystify the debate. As an example, there has been much recent interest in the possibility of designing an engine that exploits 1s and 0s on a tape to do useful work (such as lifting a weight) (Mandal and Jarzynski 2012;Mandal et al. 2013). When the state of the tape is written as abstract 1s and 0s, the operation of such a device seems almost magical. However, for the device to run, the 1s and 0s must interact with a motor in a specific way: they must be physical. And a physical instantiation of the system can be constructed from biomolecules-at which point it becomes clear that the 1s and 0s correspond to two states of a molecular fuel molecule, and the motor is driven by an imbalance of fuel input just like any other motor (McGrath et al. 2017). The operation of the device is consistent with well-established laws of thermodynamics, rather than being particularly remarkable.
By constraining ourselves to physically plausible models of molecular systems, it is also much clearer what is possible. It is consequently harder to accidentally invent an unaccounted-for ''Maxwell Demon'' that performs an impossible task, or miss the requirements of a key step. Moreover, there is at least some sense in which the practical constraints of thought experiments become evident. A thermodynamic system can be designed on paper with arbitrary coupling between degrees of freedom, but when trying to instantiate it as a molecular system, even theoretically, the inherent trade-offs required become more obvious (McGrath et al. 2017).
In particular, the majority of thermodynamic studies focus on systems in which changes are driven by intervention from an external experimenter, who does work on the system (this idea is briefly introduced in Sects. 2, 6.2). From the perspective of stochastic thermodynamics, doing work corresponds to adjusting the system to change the energy of the microstates in a time-dependent manner. Whilst analysis in terms of external work is not wrong, it is again hard to understand what is fundamentally possible at zero cost, particularly for small systems. Generally, the actual physical mechanism by which work is applied is not considered; it is therefore unclear which work protocols are feasible, and whether work can actually be applied, recovered and stored efficiently (Gopalkrishnan, personal communication), as is assumed. For example, recent experiments on the manipulation of single colloids actually implement ''work'' protocols through a highly dissipative mechanism, the cost of which far exceeds any entropy generated by the motion of the colloid itself (which is typically the topic of interest) (Bérut et al. 2012;Jun et al. 2014). Similarly, in questions related to the minimal cost of operations on small systems, such as computing with a single bit, the cost of an experimenter deciding to implement each stage of the protocol is not obviously accounted for (Gopalkrishnan, personal communication).
By contrast, molecular systems are usually thought of as running autonomously, without external work (although this is not always the case ; Ouldridge and ten Wolde 2017). Due to the ability of molecules to diffuse and interact selectively with multiple partners, complex behaviour can arise simply from a system initiated in some non-equilibrium state and left to evolve (perhaps with a constant supply of non-equilibrium fuel). In these autonomous systems, all costs are explicit, and all behaviours based on plausible chemical reactions. A major challenge for the future of thermodynamics lies in understanding the fundamental differences between the capabilities of such autonomous systems, and those systems in which outside manipulation is permitted. In turn, understanding these autonomous systems will potentially allow us to build microscopic devices of extremely high efficiency that genuinely approach the fundamental lower bounds on cost .