Evolution of Ideas on Entropy

Abstract

Our stated goal is to develop a general theory of thermodynamics that we may apply to stochastic processes, but what is thermodynamics? At the mathematical level, thermodynamics is the calculus of entropy. The inequality of the second law forms the starting point for a large number of mathematical relationships that are written among the set of primary variables, entropy, energy, volume and number of particles, and a number of defined functions based on the primary set. The mathematical framework of classical thermodynamics is established as soon as the second law is formulated in mathematical form. The subsequent development of statistical mechanics left this framework intact, while making the connection to the microscopic structure of matter. Even as the basic mathematical framework did not change, the revolution lead by Gibbs introduced a statistical view of entropy that opened an entirely new viewpoint. Starting with Shannon’s work on information theory and Jaynes’s formulation of maximum entropy has now escaped from the realm of physics to invade other fields. In this chapter we review the evolution of ideas on entropy, from the classical view, to Gibbs, Shannon, and Jaynes, and cover the background that gives rise to our development in the chapters that follow.

Appendix: The Mathematical Calculus of Entropy

The mathematical relationships we recognize as thermodynamics are based on three key concepts: curvature (concavity/convexity), homogeneity, and Legendre transformations. These are briefly reviewed here, then are applied to obtain certain key results in thermodynamics.

1.1.1 Curvature

A function f(x ₁, x ₂⋯ ) is concave with respect to x ₁ if

$$\displaystyle \begin{aligned} f(\lambda_1 x_1+\lambda^{\prime}_1x^{\prime}_1,x_2\cdots) \geq \lambda_1f(x_1,x_2\cdots) + \lambda^{\prime}_1 f(x^{\prime}_1,x_2\cdots) \end{aligned}$$

for all positive λ ₁, $\lambda ^{\prime }_1=1-\lambda _1$ at fixed x ₂, ⋯. The second derivative of a concave function is negative:

$$\displaystyle \begin{aligned} \frac{\partial^2 f}{\partial x_1^2} \leq 0 . \end{aligned} $$

(1.32)

For a convex function these inequalities are inverted. If f is concave with respect to several independent variables, the concave inequality applies to each variable at a time, keeping all other variables constant. A multivariate function may have different curvatures with respect to different variables. For example, if f(x ₁, x ₂) is concave with respect to x ₁ and convex with respect to x ₂, then

$$\displaystyle \begin{aligned} f(\lambda x_1+(1-\lambda) x_1^{\prime},x_2) \geq a f(x_1,x_2)+(1-\lambda) f(x_1^{\prime},x_2) . \end{aligned} $$

(1.33)

and

$$\displaystyle \begin{aligned} f(x_1,\lambda x_2+(1-\lambda) x^{\prime}_2) \leq a f(x_1,x^{\prime}_2)+(1-\lambda) f(x_1,x^{\prime}_2) . \end{aligned} $$

(1.34)

It is an elementary property that f and (−f) have opposite curvatures: if f is concave, then (−f) is convex, and vice versa.

1.1.2 Homogeneity

A multivariate¹⁵ function f(x ₁, x ₂) is homogeneous in x ₁ and x ₂ with degree ν if

$$\displaystyle \begin{aligned} f(\lambda x_1,\lambda x_2) = \lambda^\nu f(x_1,y_1) \end{aligned} $$

(1.35)

for all λ. We differentiate with respect to λ to obtain

$$\displaystyle \begin{aligned} x_1 \frac{\partial f}{\partial (\lambda x_1)} + x_2 \frac{\partial f}{\partial (\lambda x_2)} = \nu\lambda^{\nu-1} f, \end{aligned} $$

(1.36)

then setting λ = 1,

$$\displaystyle \begin{aligned} x_1 f_1 + x_2 f_2 = \nu f , \end{aligned} $$

(1.37)

where f _i is a shortcut notation for the partial derivative with respect to x _i. This is Euler’s theorem for homogeneous functions of degree ν. For degree ν = 1 we obtain

(1.38)

The derivatives f ₁ and f ₂ are homogeneous in x ₁ and x ₂ with degree 0, i.e.,¹⁶

$$\displaystyle \begin{aligned} f_i(\lambda x_1,\lambda x_2) = f_i(x_1,x_2). \end{aligned} $$

(1.39)

Taking the differential of f in Eq. (1.38) with respect to all x _i and f _i we have

$$\displaystyle \begin{aligned} d f = f_1 d x_1+ f_2 d x_2 + f_1 d x_1 + x_2 d f_2, \end{aligned} $$

(1.40)

and since df = f ₁ dx ₁ + x ₂ df ₂,

(1.41)

This is the Gibbs-Duhem equation associated with the Euler form in Eq. (1.38).¹⁷ It expresses the fact that homogeneity, by virtue of being a special condition on f, imposes the constraint that the variations of the partial derivatives are not all independent.

If f(x ₁, ⋯x _m) is homogeneous with degree 1 only with respect to x ₁, ⋯x _k, then the Euler and the Gibbs-Duhem equation apply to these variables,

$$\displaystyle \begin{aligned} f = \sum_{i=1}^k x_i f_i,\end{aligned} $$

(1.42)

$$\displaystyle \begin{aligned} \sum_{i=1}^k f_i dx_i = 0,\end{aligned} $$

(1.43)

with the understanding that the remaining variables x _k+1, ⋯x _m are held constant.

1.1.3 Legendre Transformations

Given a monotonic function f(x ₁, x ₂, ⋯ ) with partial derivatives f _i, i = 1, 2⋯, its Legendre transformation with respect to variable x ₁ is¹⁸

$$\displaystyle \begin{aligned} F^{(1)}(f_1,x_2) = f - x_1 f_1 , \end{aligned} $$

(1.44)

The Legendre transformation takes a function of (x ₁, x ₂) and turns it into a function of (f ₁, x ₂), where f ₁ is the derivative of the original function with respect to the transformed variable. The usefulness of the Legendre may be appreciated better if we write the differentials of the original and of the transformed functions (the derivations are straightforward and can be found in the standard literature):

$$\displaystyle \begin{aligned} d f &= +f_1 d x_1 + f_2 d x_2 \\ d f F^{(1)} &= -x_1 d f_1 + f_2 d x_2 . \end{aligned} $$

The Legendre transformation with respect to x ₁ changes the independent variable from x ₁ to f ₁, and the corresponding partial derivative from f ₁ to − x ₁, i.e.,

$$\displaystyle \begin{aligned} \left(\displaystyle\frac{\partial F^{(1)}}{\partial f_1} \right)_{x_2,\cdots} = - \left(\displaystyle\frac{\partial f}{\partial x_1} \right)_{x_2\cdots} .\end{aligned} $$

(1.45)

These derivatives will usually be notated more simply as ∂F ⁽¹⁾∕∂f ₁ and ∂f∕∂x ₁ with the understanding that they are taken with respect to the proper set of independent variables of each function. If we Legendre-transform F ⁽¹⁾ with respect to f ₁ we obtain the original function f. This makes the Legendre transformation an involution—its inverters transformation is itself. The Legendre transformation can be applied to any subset of variables. For example, transforming f with respect to x ₁ and x ₂ we obtain

$$\displaystyle \begin{aligned} F^{(1,2)} = f - x_1 f_1 - x_2 f_2,\end{aligned} $$

(1.46)

whose differential is

$$\displaystyle \begin{aligned} d F^{(1,2)} = -x_1 d f_1 - x_2 d f_2 + f_3 d x_3 \cdots . \end{aligned} $$

(1.47)

The Legendre transformation inverts curvature with respect to the transformed variable and preserves it for all untransformed variables. If f(x ₁, x ₂⋯ ) is concave in all x _i, then F ⁽¹⁾(f ₁, x ₂, ⋯ ) is convex in f ₁ and concave in x ₂, x ₃⋯.

1.1.4 Thermodynamic Relationships

We assert that function S = S(E, V, N _k), defined in the positive quadrant of the (E, V, N _k) plane, has the following properties: it is positive, concave, and homogeneous with degree 1 with respect to all of its independent variables. These conditions reproduce all relationships of classical thermodynamics, as we demonstrate below.

1.1.4.1 Second Law

We apply the concave inequality to entropy. With λ + λ′ = 1 (λ, λ′≥ 0), we have

$$\displaystyle \begin{aligned} \begin{array}{rcl} &\displaystyle &\displaystyle S(\lambda E+\lambda' E',V,N_k) \geq \\ &\displaystyle &\displaystyle \qquad \qquad \qquad \qquad \lambda S(E,V,N_k) + \lambda' S(E',V,N_k) = \\ &\displaystyle &\displaystyle \qquad \qquad \qquad \qquad \qquad \qquad \qquad S(\lambda E,\lambda V,\lambda N_k) + S(\lambda' E',\lambda' V,\lambda' N_k) . \end{array} \end{aligned} $$

Setting

$$\displaystyle \begin{aligned} \hspace{60pt} &E_1 =\lambda E, &&V_1 =\lambda V, &&N_{k,1} =\lambda N_k,\\ &E_2 =\lambda' E', &&V_2 =\lambda' V, &&N_{k,2} =\lambda' N_k, \hspace{60pt}\end{aligned} $$

the concave inequality becomes

$$\displaystyle \begin{aligned} S(E_1+E_2, V_1+V_2, N_{k,1}+N_{k,2}) \geq S(E_1, V_1, N_{k,1}) + S(E_2, V_2, N_{k,2}) ,\end{aligned} $$

(1.48)

which is the inequality of the second law. It is always possible to obtain an infinite set of positive V ₁, V ₂, N _k,1, N _k,2, that satisfy these equations for any positive V , N _k, and any λ, λ′ = 1 − λ, such that 1 ≥ λ ≥ 0. The inequality in Eq. (1.48) therefore applies for any positive U ₁, U ₂, V ₁, V ₂, N _k,1, and N _k,2.

1.1.4.2 Entropy Equation

By Euler’s theorem,¹⁹

$$\displaystyle \begin{aligned} \frac{S(E,V,N_k)}{k_B} &= E \left(\displaystyle\frac{\partial S}{\partial E} \right)_{V,N_k} + V \left(\displaystyle\frac{\partial S}{\partial V} \right)_{E,N_k} + \sum_k \gamma_k \left(\displaystyle\frac{\partial S}{\partial N_k} \right)_{E,V,N_{j\neq k}} \\ &= \beta E + \alpha V + \sum_k \gamma_k N_k , \end{aligned} $$

(1.49)

where β, α, and γ _k are the derivatives of entropy:

$$\displaystyle \begin{aligned} \beta = \frac{1}{k_B T},\quad \alpha = \frac{p}{k_B T},\quad \gamma_k = -\frac{\mu_k}{k_B T} . \end{aligned} $$

(1.50)

These derivatives are homogeneous with degree 0, i.e., they are intensive functions of the state.

1.1.4.3 Thermodynamic Potentials

A family of thermodynamic potentials is obtained by Legendre transforming entropy with respect to various combinations of its independent variables. We give one example: the Gibbs function G, which is commonly defined as

$$\displaystyle \begin{aligned} G = E - TS - pV = E-\frac{S/k_B}{\beta} - \frac{\alpha V}{\beta}. \end{aligned} $$

(1.51)

We express this in the equivalent form

$$\displaystyle \begin{aligned} -\beta G = \frac{S^{(1,2)}(E,V,N_k)}{k_B} \end{aligned} $$

(1.52)

which identifies the product − βG as the Legendre transformation of S(E, V, N _k)∕k _B with respect to E and V . The product − βG is a function of β, α, and N _k (as is G), and its differential is written immediately by application of Eq. (1.47)

$$\displaystyle \begin{aligned} -d\left(\beta G\right) = -E d\beta - V d\alpha + \sum_k \gamma_k d N_k. \end{aligned} $$

(1.53)

Substituting Eq. (1.50) for β, α, and γ _k we obtain the more familiar result

$$\displaystyle \begin{aligned} d G = -S d T + V d p + \sum_k \mu_k d N_k . \end{aligned} $$

(1.54)

At fixed T and p, the Gibbs energy is homogeneous in N _k with degree 1. From Euler’s theorem for a mixed set of extensive and intensive variables, given in Eq. (1.42), we obtain

$$\displaystyle \begin{aligned} G = \sum_k \mu_k N_k,\quad \text{(const. }p, T).{} \end{aligned} $$

(1.55)

Applying the Gibbs-Duhem equation with respect to all extensive variables while keeping the intensive variables constant we have

$$\displaystyle \begin{aligned} 0 = \sum_k N_k d\mu_k \quad \text{(const. }p, T). \end{aligned} $$

(1.56)

In thermodynamics the Gibbs-Duhem equation is specifically associated with this result. We use Gibbs-Duhem in a more general sense to refer to a condition on the simultaneous variation of intensive properties as a companion of the Euler equation, which governs the variation of the extensive variables.

1.1.4.4 Stability Criteria

The concave condition on S implies that the second derivatives of entropy are negative:

$$\displaystyle \begin{aligned} \left(\frac{\partial^2 S}{\partial E^2}\right)_{V,N_k} \leq 0,\quad \left(\frac{\partial^2 S}{\partial V^2}\right)_{E,N_k} \leq 0,\quad \left(\frac{\partial^2 S}{\partial N_k^2}\right)_{E,V,N_{j\neq k}} \leq 0 . \end{aligned}$$

All stability criteria are obtained by manipulation of these derivatives. To develop stability criteria for the Gibbs energy, we first note that the product − βG (see Eq. (1.52)) is concave with respect to the untransformed variables N _k, as is entropy. Accordingly, at fixed β and α the Gibbs energy is convex in N _k (the negative sign flips the curvature). Thus the equilibrium Gibbs energy is at a minimum with respect to partitioning of components at fixed β and α, or equivalently, at fixed T and p:

$$\displaystyle \begin{aligned} G(T,p,N_k) \leq G(T,p,\lambda_k N_k) + G(T,p,(1-\lambda_k)N_k). \end{aligned} $$

(1.57)

Similar expressions can be written for the other thermodynamic potentials (energy, free energy, and enthalpy). We will not write them down as our intention is not to reproduce the complete set of thermodynamic relationships but rather to demonstrate how to obtain these results by combining curvature, homogeneity, and Legendre transformations.