Gibbs Distribution From Sequentially Predictive Form of the Second Law

We propose a prequential or sequentially predictive formulation of the work extraction where an external agent repeats the extraction of work from a heat engine by cyclic operations based on his predictive strategy. We show that if we impose the second law of thermodynamics in this situation, the empirical distribution of the initial microscopic states of the engine must converge to the Gibbs distribution of the initial Hamiltonian under some strategy, even though no probability distribution are assumed. We also propose a protocol where the agent can change only a small number of control parameters linearly coupled to the conjugate variables. We find that in the restricted situation the prequential form of the second law of thermodynamics implies the strong law of large numbers of the conjugate variables with respect to the control parameters. Finally, we provide a game-theoretic interpretation of our formulation and find that the prequential work extraction can be interpreted as a testing procedure for random number generator of the Gibbs distribution.


Introduction
Equilibrium statistical mechanics gives a tool for calculating macroscopic thermodynamic quantities from the Hamiltonian characterizing microscopic properties of the system [1]. A fundamental assumption of equilibrium statistical mechanics is that microscopic states are randomly sampled according to the Gibbs distribution for the Hamiltonian. While we can obtain statistical properties of observables, such as means and variances, from this probabilistic assumption, the assumption is also consistent with the second law of thermodynamics. In fact, we cannot extract a strictly positive amount of work through any cyclic process on average if the initial probability distribution is Gibbssian [2][3][4][5][6][7]. This result can be regarded as a derivation of the second law of thermodynamics from statistical mechanics. It is natural Communicated by Hal Tasaki.
B Ken Hiura hiura.ken.88n@st.kyoto-u.ac.jp 1 Department of Physics, Kyoto University, Kyoto 606-8502, Japan to ask whether the second law conversely characterizes the Gibbs distribution or not. This question has been traditionally studied in terms of passivity [2][3][4][5]. These studies showed that the initial probability distribution is Gibbssian if and only if any number of copies of the identical state satisfy the second law of thermodynamics. In this approach, we crucially assume that the system is described by a probability distribution on the phase space. In contrast, our question we consider in this paper is how the probabilistic description based on the Gibbs distribution emerges from the second law of thermodynamics, particularly the absence of the perpetual motion machine of the second kind.
A similar question was posed in the context of probability theory. In measure-theoretic probability theory, we assign a real value in [0, 1] to each event under certain compatibility conditions such as the additivity. Although measure-theoretic probability theory is a useful tool to analyze the random behavior of phenomena in nature, it does not provide a characterization of randomness itself. As an alternative approach, Shafer and Vovk proposed game-theoretic probability theory [8,9]. In a gambling, they say that the gambling is fair if the gambler never become infinitely rich in the limit as the gamble continues and that the sequence of events obtained from such gambling is random. Based on these ideas, they proved, say, the law of large numbers in terms of the gambling, without using measuretheoretic concepts. If we think of the work extraction as a certain gambling between an agent and the nature, the second law of thermodynamics corresponds to the fairness condition in the gambling. If this reasoning is true, we can expect that randomness of the microscopic states in an equilibrium system is characterized through the second law of thermodynamics without using the probabilistic assumption in statistical mechanics. The purpose of the paper is to validate this idea and to answer in the affirmative.

Elementary Example: Single-Particle Ideal-Gas Engine
We consider a single-particle ideal-gas engine as an elementary example to clarify our problem. A single particle is confined in a box of volume V = L 3 and in contact with a heat bath having a temperature T . An external agent attempts to extract work from the system. The agent inserts a barrier at the center of the box, x = L/2, and moves the barrier quasistatically to x = (1 − μ)L, where μ ∈ (0, 1) specifies the final position of the barrier. When the particle is on the left side (resp. right side), we set ω = 0 (resp. ω = 1). Since the volume of the region in which the particle is confined after the operation is μ ω (1 − μ) 1−ω V , the work extracted in this process is given by where k B is the Boltzmann constant and we have used the equation of state P(T , V , N ) = N k B T /V for the ideal gas. Finally, the agent removes the barrier and the system returns to the initial state so that the overall process becomes cyclic. According to the second law of thermodynamics, the mechanical work extracted by any cyclic operations is always nonpositive. Eq. (1.1), however, becomes positive for some μ and ω, e.g., μ = 1/4 and ω = 0. In statistical mechanics and stochastic thermodynamics, this apparent inconsistency is considered to arise from the fluctuations in small systems. To resolve the inconsistency, we assume that ω is a random variable obeying the equilibrium distribution P 1/2 ({ω = 0}) = P 1/2 ({ω = 1}) = 1/2. Then, although the second law is violated with positive probability for μ = 1/2, P 1/2 (W μ > 0) = 1/2 > 0, due to the fluctuation, the expectation value of the extracted work W μ is non-positive for any operations μ, E P 1/2 [W μ ] = k B T ln 4μ(1−μ) ≤ 0, and the second law of thermodynamics remains true on average. Moreover, the second law conversely characterizes the equilibrium distribution. To prove this statement, we assume that the initial distribution of ω is given by P ρ ({ω = 0}) = 1 − ρ and P ρ ({ω = 1}) = ρ, where ρ ∈ (0, 1) quantifies the inhomogeneity of the particle distribution. The expectation value of the extracted work is given by E P ρ [W μ ] = k B T (D(P ρ P 1/2 ) − D(P ρ P μ )). Here is the Kullback-Leibler divergence between the Bernoulli distributions P ρ and P μ . If the initial distribution is not uniform, i.e., ρ = 1/2, we have that E P ρ [W ρ ] = k B T D(P ρ P 1/2 ) > 0 by choosing μ = ρ. Therefore, the condition that E P μ [W μ ] ≤ 0 for any μ ∈ (0, 1) implies ρ = 1/2. In summary, the non-positivity of the averaged work is equivalent to the equilibrium condition for the initial distribution in this example. We stress again that the extracted work W μ (ω) is positive for some initial state ω and operation μ. Therefore, when we speak of the validity of the second law of thermodynamics in small systems, we have to consider a situation that we prepare microscopic states and extract work many times. Moreover, if the empirical frequency of the initial microscopic states is biased from the equilibrium distribution, the second law may be violated suggested by the argument in the previous paragraph. In this sense, the second law in small systems may requires the stochastic behavior of the initial microscopic states as well as the equilibrium condition for the probability distribution characterizing its stochasticity. In this paper, we ask how can we formulate mathematically the emergence of stochasticity characterized by the equilibrium distribution from the second law of thermodynamics. To answer the question, we investigate two issues in this subsection.
First, we provide a mathematical definition of "stochastic behavior of the initial microscopic states". In this paper, we consider the situation that the agent repeats the cyclic operations infinitely many times and ask whether an infinite sequence ω 1 ω 2 · · · ∈ {0, 1} N + of initial microscopic states in the experiments is a random sequence or not with respect to the equilibrium distribution P 1/2 . Here N + = {1, 2, . . . } denotes the set of positive integers and ω n ∈ {0, 1} denotes the initial position of the particle in the n-th cycle. An example of non-random sequence is 00000 . . . , which corresponds to the situation that the particle is always in the left side of the box. Although the theory of algorithmic randomness [10,11] provides a reasonable and rigorous definition of randomness for individual sequences, we pay attention to only the convergence of the empirical distribution and empirical mean in this paper. For the single-particle ideal-gas engine, we regard an infinite sequence ω 1 ω 2 . . . representing the positions of the particle as a random sequence if the sequence satisfies the strong law of large numbers (SLLN), We discuss the relation to the algorithmic randomness in Sect. 5. Second, we formulate the second law of thermodynamics without probability measures. Since our purpose is to clarify the emergence of the probabilistic description, we cannot suppose the underlying probability distribution at the starting point of the discussion and use the standard definition of the second law E[W μ ] ≤ 0. To remove the probability distribution and expectation value from the description, we consider again the situation that the agent repeats the cyclic operations infinitely many times (Fig. 1). In contrast to averaging the extracted work for a single cyclic operation, in repeated cyclic operations, we allow the agent to choose a different operation in each cycle. A critical problem here is to specify the information that the agent can use when he decides the cyclic operation in each cycle.
In this paper, we apply the prequential scheme [12] or predictive scheme, where the agent determines the volume fraction μ n ∈ (0, 1) in the n-th cycle depending on only the past history ω 1 , . . . , ω n−1 of the initial positions up to the (n − 1)-th cycle 1 . In other words, the agent predicts the position of the particle in the n-th cycle from the past results ω 1 . . . ω n−1 and performs a cyclic operation based on the prediction. An assignment ω 1 . . . ω n−1 → μ n for each n = 1, 2, . . . represents a prediction scheme called a strategy for the agent and denoted byμ. For a given strategyμ for the agent, the accumulation of the extracted work Wμ is given by We define the violation of the second law in terms of the asymptotic behavior of Wμ. We say that an infinite sequence ω 1 ω 2 · · · ∈ {0, 1} N + of initial positions of the particle violates the second law of thermodynamics under the strategyμ if the total amount of the extracted work from the heat engine diverges to infinity, i.e., This means that the agent can extract work from such a sequence as much as he wants by repeating the cyclic operations sufficiently many times. For example, if the agent chooses μ t = 1/4 for any t ∈ N + , the infinite sequence 0000 . . . violates the second law because Wμ(ω 1 . . . ω n ) = nk B T ln(3/2) diverges to infinity as n → ∞. We adopt this definition as the second law in our study because it refers to no probability measure. We note that this definition is consistent with equilibrium statistical mechanics. In fact, if we assume that ω 1 , ω 2 , . . . are independent and identically distributed random variables obeying the product distribution P where we have defined a random variable Wμ n (ξ ):=Wμ(ω 1 . . . ω n ) for ξ = ω 1 ω 2 · · · ∈ {0, 1} N + . We present a proof of (1.5) in Appendix 1. Based on the above two arguments, our question is formulated as follows. Instead of assuming that ω n is a random variable and introducing a probability measure from the beginning, we ask which sequences can be realized under the second law of thermodynamics and what statistical properties such sequences have commonly. As we noted before, in this paper, we consider the strong law of large numbers (1.2) as a relevant statistical property. The SLLN is formulated in probability theory as This statement means that the relative frequencies of positions of particle becomes one half almost surely. However, we give a different formulation of the SLLN in our setting because  (1) no probability measure enters the setting. According to the excellent book "Probability and Finance: It's Only a Game!" by Shafer and Vovk [8], there exists a strategyμ SLLN for the agent such that if an infinite sequence ω 1 ω 2 · · · ∈ {0, 1} N + of positions of the particle does not satisfy the SLLN (1.2), the second law of thermodynamics is then violated, i.e., (1.4) holds for the sequence. Equivalently, if the sequence retains the second law under the strategŷ μ SLLN , the positions of the particle necessarily obey the SLLN (1.2). This implies that even though no probability measure is assumed, the empirical distribution for positions of the particle must be consistent with the equilibrium distribution due to the constraint by the second law. From the viewpoint of statistical mechanics, the second law of thermodynamics is a consequence of the equilibrium distribution and equilibrium statistical mechanics gives a microscopic foundation for the second law. However, according to the theorem by Shafer and Vovk, the second law of thermodynamics (1.4) under some strategy requires that the sequence must be random in the sense that it satisfies the SLLN (1.2), and leads to the equilibrium distribution in this sense. Our purpose of the paper is to investigate this novel aspect of the relationship between equilibrium statistical mechanics and the second law of thermodynamics for generic small systems.

Main Results
The argument in subsect. 1.1 suggests that statistical properties of equilibrium states are characterized by the second law of thermodynamics. In this paper, we extend the above example to a system with a generic Hamiltonian on a finite state space. First, we consider the situation that the external agent has an ability to prepare arbitrary Hamiltonians during cyclic operations. We show that there exists a strategy for the agent such that the empirical distribution for a sequence satisfying the second law of thermodynamics necessarily converges to the Gibbs distribution of the initial Hamiltonian (Theorem 2.3). As in the case of the single-particle ideal-gas engine, this result can be interpreted as a statement that the empirical statistics must be consistent with the assumption of statistical mechanics, i.e., the Gibbs distribution, due to the second law. Second, we study the empirical statistics in the same manner when we restrict the ability of the agent. The assumption that the agent can prepare arbitrary Hamiltonians is too demanding for his ability because we control only a small number of parameters in the Hamiltonian, such as the magnetic field, in many physical situations. The main aim of the present paper is to propose a protocol corresponding to such a restricted situation and to determine what statistical properties are observed in that situation. This restriction weakens the ability of the agent, and we thus expect that the statistical property the second law imposes also weakens. We find that the Gibbs distributions for Hamiltonians having a small number of parameters linearly coupled to conjugate variables form an exponential family and propose a new protocol where the agent has to construct his strategy by changing only these parameters. Our main contribution is that in the protocol there exists a strategy such that the empirical mean of the conjugate variable with respect to the control parameter for sequence satisfying the second law converges to the equilibrium value (Theorem 3.3). This result suggests that there is a hierarchy of statistical properties observed under the second law according to his ability.

Related Studies
Several studies shares the same mathematical structures and techniques with our work, although our motivation and formalism in this paper is quite different from the usual one in statistical mechanics and stochastic thermodynamics. We review related studies in this subsection.
First, an important property of the exponential of the accumulation of the extracted work is martingality, which is a fundamental concept in the theory of stochastic processes [14,15]. The martingale property is useful to investigate the statistics at stopping times and extreme value statistics. In the context of nonequilibrium thermodynamics, the novel statistical properties of stochastic entropy production were recently studied [16][17][18][19] based on this property of martingales and the fact that the exponentiated negative entropy production or its modification is a martingale. Although our results are also based on the fact that the exponential of the accumulation of the extracted work is a martingale 2 , we use another property that was first found by Jean Ville [20], the characterization of almost sure properties in terms of martingales. See subsect. 2.4. As other interesting study concerning the martingale property in physical systems, see [21] for example.
Second, we use universal coding theory [22] to construct a strategy for the agent. The prediction strategy in universal coding is useful for proving our main result because our problem is similar to the coding or prediction of the outcomes ω n in the n-th cycle from the past sample data ω 1 . . . ω n−1 where the performance of the prediction is measured by the log-loss function. The analogy between gambling and source coding problems was first discussed by Kelly [23]. See also Chapter 6 of [24]. In addition, the analogy with information thermodynamics was pointed out in [25][26][27]. Specifically, Ref. [27] applies universal coding theory to information thermodynamics to construct an optimal work extraction protocol. A crucial difference of our work from this study is that we analyze the asymptotic behavior of extracted work for individual sequences [25,28] and attempt to find statistical properties shared by sequences satisfying the second law.
Finally, we stress that our studies are based on an analogy between the work extraction in thermodynamics and betting in game-theoretic probability theory [8,9]. Game-theoretic probability theory is a mathematical formulation of probability theory alternative to the conventional measure-theoretic one. In game-theoretic probability theory, a gambler named Skeptic bets money on head or tail of a coin and a dealer named Reality choose the outcome. By repeating this gamble infinitely many times and imposing the duty that Reality must chooses a sequence of outcomes such that Skeptic cannot make infinitely much money, we study what statistical behavior is observed. For instance, as mentioned in subsect. 1.1, it is possible to construct an explicit strategy such that Skeptic's capital grows infinitely as long as the sequence violates the law of large numbers. The work extraction we proposed in this paper can be regarded as a game played between two players, Skeptic and Reality. The external agent who attempts to extract work from the heat engine as much as possible corresponds to the gambler Skeptic and the world who prepares the particle to retain the second law of thermodynamics corresponds to the dealer Reality. Although the first main result of this paper in Sect. 2 is a straightforward extension of the theorem proved by Shafer and Vovk, the possibility of analogous analysis of thermodynamics and the novel aspect of the relationship between statistical mechanics and the second law are new findings of this paper as long as the author knows.

Outline of the Paper
The remainder of the paper is organized as follows. In Sect. 2, we formulate the work extraction in a similar manner to the single-particle ideal-gas engine in subsect. 1.1 and give the first main result. We also discuss the mathematical backgrounds, Ville's theorem, behind our result. In Sect. 3, we propose another protocol where the ability of the agent is restricted, and prove the second main result. In Sect. 4, we discuss a game-theoretic interpretation of our protocols. We end our paper with concluding remarks in Sect. 5.

Notations
This subsection summarize notations we use throughout this paper. Since the author explain notations when they are first used, the readers can skip this subsection.
Let N + = {1, 2, . . . } be the set of positive integers. We use Ω to denote a finite set representing a microscopic state space. Ω * :={ }∪(∪ ∞ n=1 Ω n ) denotes the set of finite strings over Ω, where is the empty string. A string over Ω with length n is written as ω n = ω 1 . . . ω n ∈ Ω * , ω i ∈ Ω. We use ξ = ω 1 ω 2 · · · ∈ Ω N + to denote an infinite sequence on Ω. A real-valued function H : Ω → R defines a Hamiltonian on the state space Ω. For a positive real number β > 0, the Gibbs distribution for the Hamiltonian H at the inverse temperature β is defined as a probability distribution on Ω with density is the free energy.

Setup
Let us consider a physical system whose state space is given by a finite set Ω. The thermodynamic property of the system in contact with the heat bath is described by a Hamiltonian H : Ω → R and an inverse temperature β of the bath. According to Kelvin's principle, which is one of the representation of the second law of thermodynamics, the positive amount of work cannot be extracted by any cyclic operations. Here an operation is said to be cyclic if the initial and final Hamiltonians coincide. This principle leads to the absence of the perpetual motion machine of the second kind. However, the second law of thermodynamics may be violated for some individual initial state and cyclic operation in the finite system. Therefore, the second law in small systems is usually formulated as a statement on the non-positivity of the expectation value of the extracted work assuming the underlying probability distribution. In this subsection, we review the usual formulation of the second law of thermodynamics in terms of the expectation value.
Let us consider the following type of cyclic process [29] to avoid taking the dynamical evolution of the system into consideration: (P1) The agent quenches the Hamiltonian adiabatically from the initial Hamiltonian H to another one H . (P2) The agent equilibrates the system with the inverse temperature β. (P3) The agent resets the system quasi-statically and isothermally Let ω be an initial state of the system. We suppose that the microscopic state ω does not change during the adiabatic quenching process (P1). Under this assumption, the extracted work in the process (P1) is given by the decrease in internal energy H (ω) − H (ω). In the process (P2), the agent touches the system with the heat bath having the inverse temperature β and the system relaxes to the new equilibrium state for the quenched Hamiltonian H . This equilibration process obviously requires no mechanical work. In the process (P3), the agent changes the Hamiltonian from the quenched one H to the initial one H quasi-statically to make the whole process cyclic. Moreover, we crucially assume that the work extracted in the quasi-static isothermal process is equal to the decrease in free energy 3 . This assumption implies that the extracted work in the process (P3) is given by Here the free energy for the Hamiltonian H at the inverse temperature β is defined as F β (H ):= − β −1 ln ω∈Ω e −β H (ω) . Therefore, the total amount of work extracted in this cyclic process is given by Now we suppose that the initial state is sampled according to an initial density ρ. The expectation value of the work extracted during the above cyclic process is given by is the Gibbs density function for the Hamiltonian with the inverse temperature β and D( p q):= ω∈Ω p(ω) ln( p(ω)/q(ω)) is the Kullback-Leibler divergence between two densities p and q. If the initial distribution is the Gibbs distribution for the initial Hamiltonian, the expectation value of the extracted work is non-positive for Therefore, the second law of thermodynamics expressed in the form of the expectation value during the cyclic process (P1)-(P3) is equivalent to that the initial distribution is the Gibbs distribution for the initial Hamiltonian.
The single-particle ideal-gas engine in subsect. 1.1 is formally considered to be an example of the above setup. Example 2.1 (Single-particle ideal-gas engine) Let Ω = {0, 1} be a state space. Each state ω ∈ {0, 1} codes the position of the particle in the box. The effective Hamiltonian for the single-particle ideal gas is defined as where V is the total volume of the box and the parameter μ ∈ (0, 1) indicates the position of the barrier. The free energy for the Hamiltonian is given by F β (H μ ) = −β −1 ln V . If the initial and quenched parameters are 1/2 and μ, respectively, the extracted work is written as which is identical to Eq. (1.1).

Prequential Formulation
The purpose of the present paper is to investigate the emergence of equilibrium statistical mechanics from the second law of thermodynamics without referring to any probability measure. Hence we have to remove the probability measure from the definition of the second law. In this subsection, as such a formulation, we give a prequential definition of the second law.
To remove the probability distribution, we consider the situation that the agent repeats cyclic processes (P1)-(P3) infinitely many times. First, the agent performs a cyclic process according to the protocol (P1)-(P3). Let ω 1 be an initial state and H 1 a quenched Hamiltonian in this process. After the first cycle, the agent determines a quenched Hamiltonian H 2 in the second cycle depending on the initial state ω 1 in the first cycle and performs the cyclic process (P1)-(P3) again. In general, we suppose that the agent chooses a quenched Hamiltonian in the n-th cycle depending on the past history ω 1 . . . ω n−1 ∈ Ω n−1 up to the (n − 1)-th cycle. The assignment of a quenched Hamiltonian in the n-th cycle to each past history ω 1 . . . ω n−1 specifies a strategy for the agent to extract work. Therefore, we call a functionĤ : Ω * → R Ω strategy in this paper. Here Ω * denotes the set of finite strings over Ω including the empty string . For a strategyĤ ,Ĥ (·| ) : Ω → R represents a quenched Hamiltonian in the first cycle andĤ (·|ω 1 . . . ω n−1 ) : Ω → R represents a quenched Hamiltonian in the n-th cycle when the initial states up to the (n − 1)-th cycle are ω 1 . . . ω n−1 . This scheme in which the agent decides his action based on the past history of outcomes is called prequential in statistics [12] and causal or nonanticipating in information theory of gambling and portfolio theory [24].
The accumulation of the extracted work up to the n-th cycle is given by the sum of the extracted work in each cycle. For a strategyĤ , we define the function WĤ : Ω * → R as WĤ ( ) = 0 and for ω n :=ω 1 . . . ω n ∈ Ω n . From the definition, WĤ (ω n ) gives the accumulation of the work up to the n-th cycle under the strategyĤ when the initial states up to the n-th cycle are ω 1 . . . ω n . Now we provide a prequential definition of the second law of thermodynamics.

Definition 2.2
Let ξ = ω 1 ω 2 · · · ∈ Ω N + be an infinite sequence over Ω andĤ be a strategy for the agent. We say that Definition 2.2 is regarded as a definition of the perpetual motion machine of the second kind for individual sequences of states. Let ω 1 be an initial microscopic state of the engine in the first cycle. By performing the cyclic process (P1)-(P3) for the initial state, the agent extract work from the engine by WĤ (ω 1 ). The engine cannot be regarded as a perpetual motion machine of the second kind only because the value WĤ (ω 1 ) is positive. To say that the engine violates the second law, we require that for any given positive value W 0 > 0, the agent should extract an amount of work larger than W 0 by repeating cyclic processes as many times as he needs. Therefore, we define the second law for infinite sequences of initial microscopic states of the engine as indicated in Definition 2.2. We note that while in the above definition the violation of the second law depends on both a strategy the agent applies and an infinite sequences of microscopic states, the definition needs no underlying probability measure.

Convergence of Empirical Distribution to Gibbs Distribution
Instead of introducing probability distributions, we consider which sequences satisfies the second law of thermodynamics and what statistical properties are shared among these sequences. In general, a statistical property is described by a subset of infinite sequences E ⊆ Ω N + . In this paper, as a relevant statistical property, we focus on only the strong law of large numbers, i.e., Here the quantity n i=1 1 {ω} (ω i )/n is the empirical density for the string ω n quantifying the relative frequency of ω in ω n = ω 1 . . . ω n . If the sequence ξ = ω 1 ω 2 . . . are sampled identically and independently according to the Gibbs distribution, the SLLN (2.6) happens almost surely. In this sense, the SLLN (2.6) is a statistical property observed in equilibrium. Our purpose is to find a strategyĤ such that any sequences ξ ∈ Ω N + satisfying the second law of thermodynamics under the strategyĤ in the sense of Definition 2.2 necessarily obey the SLLN, i.e., ξ ∈ SLLN. The existence of such strategy implies that the empirical statistics of the sequences of initial states observed under the constraint by the second law must be consistent with equilibrium statistical mechanics. Based on the above reasoning, we provide the following theorem: This is the first main result of this paper. We make several remarks on Theorem 2.3. First, we stress again that no probability measure for initial microscopic states are assumed. One may find that the Gibbs distribution is implicitly inserted in the definition of the extracted work (2.3) through the assumption that the work extracted in the quasi-static isothermal process is equal to the decrease in free energy. Although this reasoning is actually true as we will see in subsect. 2.4, the assumption on the form of the extracted work does not immediately leads to the stochastic behavior of initial states and the content of Theorem 2.3 remains highly non-trivial. Second, we stress the difference from the argument in the end of subsect. 2.1. There, if the initial distribution deviates from the Gibbs distribution for the initial Hamiltonian, we have to choose the quenched Hamiltonian depending on the deviation in order to extract a positive amount of work. In contrast, Theorem 2.3 claims the existence of a single universal strategy under which the second law is automatically violated for the sequence whose empirical statistics deviate from the Gibbs distribution even if we do not know the deviation.
We explain in subsect. 2.4 the reason why the Gibbs distribution appears in Theorem 2.3 although no probability measure is assumed in our setting. We remark that Theorem 2.3 is just a straightforward extension of the case of single-particle ideal-gas engine in subsect. 1.1 and it is nothing new mathematically. In addition to the proof in textbooks of Shafer and Vovk [8,9], there are several proofs of Theorem 2.3 such as Ref. [30] based on the maximum likelihood strategy and Ref. [31] based on the Bayesian strategy. Nevertheless we prove Theorem 2.3 as a special case of Theorem 3.3.
Hereafter, we say that a strategy forces (resp. weakly forces) an event E ⊆ Ω N + if infinite sequences over Ω that do not satisfy E violate (resp. weakly violate) the second of thermodynamics under strategy. According to this terminology, Theorem 2.3 claims the existence of a strategy that forces the strong law of large numbers for the empirical distribution.

Martingale and Ville's Theorem
We clarify a general structure behind Theorem 2.3. Although we construct explicitly a strategy that make the empirical distribution converge the Gibbs distribution in Appendix 1, Theorem 2.3 follows from a more general theorem proved by Ville [20]. The essence of Ville's theorem is some kind of equivalence between the asymptotic behavior of martingales and almost sure properties.
To see this, we first clarify the martingale property of the exponentiated work. The accumulation of the extracted work (2.3) can be written as the logarithmic likelihood ratio function for Gibbs distributions: Since g βĤ (·|ω i−1 ) specifies the conditional probability density conditioned on the past history ω i−1 , the function (2.8) gives a probability density on Ω n . Conversely, a stochastic process on Ω N + with strictly positive probability densitiesq specifies a strategy for the agent in our setting through the relation g βĤ (·|ω n−1 ) (ω n ) =q(ω n |ω n−1 ). Therefore, a strategyĤ is identified with a stochastic process having strictly positive densities. Let us consider a discrete time stochastic process M : Ω * → R. We say that M is a martingale with respect to a probability measure P on Ω N + if E P [M n |ω n−1 ] = M(ω n−1 ) for any ω n−1 ∈ Ω * [14]. Here M n (ω 1 ω 2 . . . ):=M(ω n ) and E P [ · |ω n ] denotes the conditional expectation conditioned on the past history ω n . For a fixed strategyĤ , it is easy to see that the exponential of the accumulation of the extracted work e βWĤ is a positive martingale with respect to the infinite product of Gibbs distributions g Ville's theorem [8,9,20] in our setting claims that a measurable set E ⊆ Ω N + has probability one with respect to g N + β H if and only if there exists a positive martingale M : Ω * → R + with respect to the same probability measure such that lim n→∞ M n (ξ ) = ∞ for any ξ / ∈ E. Using the equivalence between positive martingales and strategies mentioned above, we rephrase Ville's theorem as follows.
Theorem 2.4 ( [8,9,20]) Let E ⊆ Ω N + be a measurable set. The property E happens almost surely with respect to g N + β H if and only if there exists a strategyĤ : Ω * → R Ω that forces E. Ville's theorem clarifies the reason why the convergence to the Gibbs distribution occurs under the second law in spite that no probability measure comes into our setting. Let us consider the experiment that we sample initial microscopic states infinitely many times independently and identically according to the Gibbs distribution g β H . We take a statistical property E ⊆ Ω N + with probability one in the experiment such as the SLLN (2.6). According to Ville's theorem, there exists a strategyĤ such that any sequences violate the second law under the strategy if they do not have the property E. In other words, whether a statistical property is consistent with equilibrium statistical mechanics or not is characterized in terms of the violation of the second law of thermodynamics. In addition, our definition of the second law (Definition 2.2) needs no probability measure. This is an essential structure that makes it possible to argue the emergence of equilibrium statistical mechanics from the second law without referring to any probability measures.

Analysis of Parametric Hamiltonian
In the setting in Sect. 2, the agent is assumed to have an ability to prepare any complicated Hamiltonian containing non-local and many-body interactions. As a more physical situation, it is natural to restrict the possible operations into changing a small number of parameters in a certain class of Hamiltonians. In this section, we investigate such situation.

Preliminaries: Exponential Family
We focus on the Hamiltonians on Ω with a certain number of externally controllable parameters. We assume that the Hamiltonians have the form β H θ (ω):= − θ · φ(ω) − h(ω), where the parameter θ corresponds to the control parameters taking values in the parameter space Θ ⊆ R k , φ : Ω → R k is the conjugate variable with respect to θ , the dot " · " denotes the usual inner product in R k and h : Ω → R is the remaining part of the Hamiltonian. Θ is assumed to be an open and convex subset of R k . Moreover, we suppose that the representation of the Hamiltonians is minimal, i.e., both {θ i : i = 1, . . . , k} and {φ i : i = 1, . . . , k} are affinely independent.
where h ex is the external field and J is the coupling constant. If the agent changes only the magnetic field uniformly, the parameter is chosen as θ = βh ex ∈ Θ:=R, the conjugate variable as the total magnetization φ(ω) = N i=1 σ i , and the remaining part as h(ω) = β J N −1 i=1 σ i σ i+1 . In this case, the expectation of φ is the average magnetization μ(θ) = n i=1 P θ (σ i ) and the Fisher information is I (θ ) = β −1 χ, where χ is the magnetic susceptibility. If the agent changes the coupling constant J in addition to the magnetic field, the parameter becomes twodimensional vector θ = (βh ex , β J ), the conjugate variable φ = ( n i=1 σ i , n−1 i=1 σ i σ i+1 ) and the remaining part h = 0. The parametrization for a given physical model is thus not unique in general.
for every i ∈ Ω. The exponential family is therefore the family of strictly positive distributions on Ω. The empirical mean of φ is identified with the empirical density and the expectation μ i (θ ) is given by p i for i = 1, . . . , k.

Setup and Result
The protocol we study in this subsection is almost the same as that in Sect. 2. A crucial difference is that the agent cannot prepare an arbitrary Hamiltonian in general and he has to construct a strategy by tuning only a small number of control parameters. We assume that the initial Hamiltonian has the form β H θ = −θ · φ − h for some θ ∈ Θ. A strategy the agent applies is characterized by a functionθ : Ω * → Θ. The operation in the n-th cycle is performed according to the following protocol: (P1) The agent quenches the Hamiltonian adiabatically from the initial Hamiltonian H θ to another one Hθ (ω n−1 ) when the initial states up to the (n − 1)-th cycle are ω n−1 = ω 1 . . . ω n−1 . (P2) The agent equilibrates the system with the inverse temperature β. (P3) The agent resets the system quasi-statically and isothermally.
The accumulation of extracted work Wθ :=W Hθ in this protocol under the strategyθ is given by Let E ⊆ Ω N + be an almost sure property under the infinite product of the Gibbs distributions p Although there exists a strategyĤ that forces E according to Ville's theorem (Theorem 2.4), the strategyĤ may not be realizable within the above protocol. In general, the decrease in the number of possible strategies the agent can apply leads to the decrease in the variety of almost sure properties forced by the second law. The question we study in this section is what statistical properties are forced when we restrict the ability of the agent.
We expect from the expression (3.3) that such properties are restricted to the statistics of the conjugate variable φ with respect to the control parameter θ . Indeed, we find that there exists a strategy in the restricted protocol that forces the strong law of large numbers for the conjugate variable.

Theorem 3.3 Let φ n be the empirical mean of φ defined by
There exists a strategyθ : Ω * → Θ that forces This is the second main result of the present paper. Theorem 2.3 is a special case of Theorem 3.3 because the empirical mean of φ in Example 3.2 can be identified with the empirical density.
Since μ(θ) is the equilibrium value of the conjugate variable, Theorem 3.3 implies that the infinite sequences satisfying the second law are indistinguishable from random sequences sampled from the equilibrium distribution as long as we observe the conjugate variable. In contrast to Theorem 2.3, the relative frequencies of initial microscopic states may not converge to the Gibbs distribution under the strategy. For instance, when the agent controls the homogeneous magnetic field on the free spin system, the average magnetization should converge to the equilibrium value but the relative frequencies of spin configurations having the same magnetization may not be controlled. Thus the statistical properties the agent can force under the second law depend on what kind of physical operations are allowed for the agent.

Sketch of Proof of Theorem 3.3
We sketch a proof of Theorem 3.3. We present the details of the proof in Appendix 1.
By the following lemma, it is sufficient to prove the existence of a strategy that weakly forces (3.5) and takes values in a compact set containing the initial parameter θ as an interior point. The proof is presented in Appendix 1.

Lemma 3.4 Let E
⊆ Ω N be an event and Θ 0 ⊂ Θ a compact convex subset with θ ∈ intΘ 0 . If there exists a strategy that weakly forces E and takes values in Θ 0 , then there is also a strategy that forces E and takes values in the same set Θ 0 .
Our method of the proof is based on the maximum likelihood strategy. If the agent applies a constant strategy, i.e.,θ = θ , the accumulation of extracted work up to the n-th cycle is given by The maximum likelihood strategy is defined as the maximizer of (3.6) for each ω n . Because the maximizer of (3.6) over the open set Θ may not exist, we restrict the range of the strategy to a compact set Θ 0 containing the initial parameter θ as an interior point. The maximum likelihood estimatorθ ML : Ω * → Θ 0 with respect to Θ 0 is defined aŝ We define the corresponding conjugate parameter space 0 :=μ(Θ 0 ), which is also a compact convex subset of . We setμ ML (ω n ) = μ(θ ML (ω n )). Because ∇ θ ln p θ (ω) = φ(ω) − μ(θ), the maximum likelihood estimatorθ ML andμ ML are given as the solution of the equation if and only if φ n ∈ 0 , which does not always hold. The maximum likelihood strategy was originally studied in the context of universal coding and sequential prediction. Kotłowski and Grünwald obtained the following lower bound in [28]: where B:= max ω∈Ω |φ(ω)|, C 0 := max μ∈ 0 μ and I 0 := max μ∈ 0 I (μ) . We note that these constants are all finite due to the compactness of Ω and 0 . By using this result, we have a lower bound for the accumulation of the extracted work under the maximum likelihood strategy: The details of the calculation is presented in Appendix 1. We remark that the lower bound (3.10) is valid for any individual sequences ω n . The inequality (3.10) implies that the maximum likelihood strategy forces weakly the convergence of pθ ML (ω) to p θ (ω) for any ω ∈ Ω, and therefore it also forces weakly φ n → μ(θ). This completes the proof.
We have several remarks before ending the subsection.
(1) The lower bound (3.10) has much information on the asymptotic behavior. If the second law is weakly valid, i.e., sup n Wθ ML < ∞, then Pθ ML converges to P θ with respect to the Kullback-Leibler distance with the convergence rate O( √ ln n/n) and the convergence factor I 0 (B + C 0 ) 2 /2. We also note that if the convergence (3.5) does not hold, the accumulation of the extracted work grows at least linearly. Specifically, the extracted work per cycle becomes positive infinitely many times, lim sup n→∞ Wθ n n > 0, (3.11) for sequences violating the strong law of large numbers (3.5). For instance, ifθ ML (ω n ) → θ ( = θ) as n → ∞, the rate of the extracted work is given by the Kullback-Leibler distance between P θ and P θ . This implies that our results (Theorems 2.3 and 3.3) are valid even if we apply (3.11) as a definition of the violation of the second law. (2) The maximum likelihood estimatorθ ML (ω n ) defined by (3.7) depends on ω n through only φ n because the conjugate variable φ is the sufficient statistic for the exponential family P (Θ, φ, h). Therefore, even if only the information on the past history of φ is given to the agent, he can perform the maximum likelihood strategy and can forces the strong law of large numbers for the conjugate variable.

Game-Theoretic Interpretation
We discuss a game-theoretic interpretation of the protocol of this paper. The interpretation is based on the analogy with game-theoretic probability theory [8,9]. We interpret the work extraction as a game played between two players, Skeptic and Reality. The player Skeptic doubts that the statistical property of the heat engine is described by equilibrium statistical mechanics, particularly the Gibbs distribution. He attempts to test the hypothesis of equilibrium statistical mechanics by actually extracting work from the engine many times and observing whether the second law of thermodynamics holds or not. If the accumulation of the extracted work diverges to infinity, i.e., the second law is violated, then Skeptic rejects the hypothesis. On the other hand, the player Reality decides initial Fig. 2 The accumulations of the extracted work for sequences generated by the Gibbs distributions with various parameters versus the number of cycles. The top (resp. bottom) dotted line indicates the Kullback-Leibler distance between P θ=±0.2 (resp. P θ=±0.1 ) and P θ=0 microscopic states of the heat engine. The duty of Reality is to prevent Skeptic from extracting the infinite amount of work and to make the second law of thermodynamics valid in our world.
The algorithm of the game corresponding to the protocol we discussed in Sect. 2 is as follows.

Protocol 4.1 (Generic Hamiltonian)
Parameter: an initial Hamiltonian H : Ω → R, an inverse temperature β > 0. Players: Skeptic, Reality. Protocol: W ( ) = 0. FOR n = 1, 2, . . . : Skeptic announces H n Reality announces ω n ∈ Ω W n : In Protocol 4.1, H n :=Ĥ (·|ω n−1 ) denotes the quenched Hamiltonian in the n-th cycle and W n = WĤ (ω n ) the accumulation of extracted work up to the n-th cycle. We remark that Reality can decides an initial microscopic state ω n after the announcement by Skeptic. Therefore, we allow Reality to move strategically in Protocol 4.1. Nevertheless, Theorem 2.3 is still valid even if Reality decides microscopic states strategically because whether a strategy forces a property or not is independent of the strategy Reality applies. Theorem 2.3 implies that there exists a strategy for Skeptic such that Reality has to converge the empirical distribution to the Gibbs distribution. In other words, Reality is forced to act probabilistically in a manner consistent with equilibrium statistical mechanics due to the second law.
As an application of Theorem 2.3, we consider a testing procedure of random number generator. Let us suppose that Reality claims that she finds an algorithm to generate random numbers with respect to the Gibbs distribution. Skeptic doubts her claim and attempts to test it. Although there are several criteria that the random numbers should satisfy, we consider the strong law of large numbers (2.6) for the empirical distribution as a criterion here. If the random number obtained from Reality's generator does not satisfy the SLLN, Skeptic rejects her claim and concludes that the random number generator does not work well. According to Theorem 3.3, Skeptic can confirm the validity of the SLLN by performing cyclic operations for microscopic states prepared by the generator. In other words, the second law of thermodynamics can be used as a test for the random number generators.
Similarly, the algorithm of the protocol in Sect. 3 is as follows.

Concluding Remarks
In this paper, we provided a novel formulation of the second law of thermodynamics and showed that there exist strategies for the agent that force statistical properties consistent with equilibrium statistical mechanics, i.e., the strong law of large numbers for the empirical distribution and empirical mean. In the protocol where the agent is able to prepare arbitrary Hamiltonians, the empirical distribution must converge to the Gibbs distribution for the initial Hamiltonian under some strategy. In the protocol where the agent can change a small number of parameters in the Hamiltonian, the maximum likelihood strategy forces the strong law of large numbers for the conjugate variable. Before ending the paper, we discuss future directions of the study.
First, we considered the simple settings where initial microscopic states are prepared independently and identically. It is important to study in the same spirit more complicated settings such as stochastic thermodynamics [33,34], information thermodynamics [35], and quantum systems [2,3]. In particular, it is a future subject to treat the stochastic evolution of microscopic state in our framework and to connect it with thermodynamic quantities. The extension to the quantum settings might be more difficult because the quantum theory has a probabilistic structure different from the classical one. See the chapter 10.6 in [9] for game-theoretic formulations for Born's rule and quantum computation.
Secondly, we assume in our settings that the work extracted in the quasi-static isothermal process is equal to the decrease in free energy defined by F β (H ) = −β −1 ln ω∈Ω e −β H (ω) . This assumption allows us to connect the work extracted in the quasi-static isothermal process with the microscopic Hamiltonian, without explicitly referring to the time evolution of the system. However, we should remove it to clarify the emergence of the probabilistic structure from thermodynamics. Although one way to do so is to replace the difference in free energy, F β (Ĥ ) − F β (H ), in Protocol 4.1 by the work in the quasi-static process defined only in terms of mechanics, it is a challenging task because we have to take the dynamical properties of the system into account explicitly.
Thirdly, we investigated the asymptotic behavior of the work extracted from a finite system as the number of cycles n goes to infinity. While our results are valid for small systems as with stochastic thermodynamics, the second law of thermodynamics is believed to hold almost surely for macroscopic systems without repeating the operations. Developing our formulation for macroscopic thermodynamics is a future direction of the study.
Fourthly, we assumed in our protocol that the agent can use the information about microscopic states in the past. However, this assumption may be too demanding when the size of the system is large. In a more realistic situation, the agent can measure only the value of extracted work. It is important to study statistical properties forced by strategies under such situation.
Fifthly, the approach based on the prequential form allows us to classify the statistical property of the equilibrium state according to the ability of the agent. In the usual formulation of statistical mechanics, the law of large numbers for the empirical distribution and for the empirical mean of the conjugate variable are both formulated as the events with probability one. In this sense, there is no distinction between these probabilistic laws. However, in the prequential approach, what kind of probabilistic laws can be observed depends on our ability. It is interesting to provide a detailed classification of the probabilistic laws of equilibrium states according to the ability of the agent to operate the system.
Finally, we mention the probabilistic feature of infinite sequences. Although the empirical distribution for the simple binary sequence 01010101 . . . converges to the uniform distribution on {0, 1}, it is not sufficiently random according to our intuition. However, Theorems 2.3 and 3.3 refer to only the empirical distribution and mean. To say that an infinite sequence is random with respect to a probability distribution, we have to require stronger conditions on the empirical statistics. The algorithmic theory of randomness provides an idealized notion of randomness on the basis of computability theory and martingale theory [10,11]. A class of randomness is defined as the intersection of sets that are weakly forced by strategies with some computability condition. For instance, computable strategies specify the class of randomness called computable random. Although there exists no universal computable strategy, i.e., the computable randomness cannot be forced by a single strategy, to investigate such class in the context of thermodynamics may be interesting. We hope that the present paper provides an insight into studying thermodynamics from the viewpoints of the martingale structure and computability of strategies. As a study in the same spirit, see [36] showing the relevance of algorithmic randomness to thermodynamic irreversibility.

K. Hiura
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.

A Proof of Eq. (1.5)
We prove a stronger statement P{sup n Wμ n = ∞} = 0. First, we note that P{sup n Wμ n = ∞} = P{sup n e βWμ n = ∞}. e βWμ n is a positive martingale and bounded in L 1 because E P (e βWμ n ) = 1 for any n. We have from the martingale convergence theorem that e βWμ ∞ := lim n→∞ e βWμ n exists and is a finite non-negative value almost surely. Hence, P{sup n Wμ n = ∞} = 0.

B Exponentiated Extracted Work = Martingale
For a given strategyĤ : Ω * → R, the exponential of the accumulation of extracted work e βWĤ is a positive martingale with respect to g N + β H . In fact, By defining the strategyĤ through the relation g βĤ (·|ω n−1 ) (ω n ) =q(ω n |ω n−1 ), we have that e βWĤ (ω n ) = M(ω n ) for any ω n ∈ Ω * including the case ω n = .

C Proof of Theorem 3.3
We present a complete proof of Theorem 3.3. First, we prove Lemma 3.4. It is proved in the same way as Lemma 3.1 in [8].