# Classical Simulated Annealing Using Quantum Analogues

- 750 Downloads

## Abstract

In this paper we consider the use of certain classical analogues to quantum tunneling behavior to improve the performance of simulated annealing on a discrete spin system of the general Ising form. Specifically, we consider the use of multiple simultaneous spin flips at each annealing step as an analogue to quantum spin coherence as well as modifications of the Boltzmann acceptance probability to mimic quantum tunneling. We find that the use of multiple spin flips can indeed be advantageous under certain annealing schedules, but only for long anneal times.

## Keywords

Simulated annealing Quantum annealing Optimization methods Coherent tunneling## 1 Introduction

Simulated annealing is a classical numerical optimization technique based on the physical process of annealing used to slowly cool a material to its lowest energy state [1]. The method was first described by Metropolis and colleagues in 1953 [2] and later generalized by Hastings in 1970 [3] into what is now known as the Metropolis–Hastings algorithm, a subclass of more general Markov chain Monte Carlo methods [4].

A related optimization procedure, quantum annealing, has also been proposed for solving hard optimization problems [5, 6]. In quantum annealing, the temperature is held fixed while the cost function slowly evolves from one with a simple structure to one representing the problem cost function. By the adiabatic theorem, if this process is performed slowly enough the system will eventually settle into a global minimum energy state [7, 8]. Several generations of devices that implement quantum annealing for the Ising model have been built by D-Wave Systems, Inc. and used to solve a variety of optimization problems [9, 10, 11, 12, 13]. Whether such a device provides a quantum speedup over classical methods, and the mechanisms by which such a speedup would be possible, remain as yet unclear [14, 15, 16].

It has been suggested that the mechanism of tunneling may provide a speedup for quantum annealing over classical simulated annealing for coupled spin systems, as the coherent tunneling of multiple spin states would allow the system to escape deep local minima and explore the state space more widely [17, 18, 19]. Colloquially, tunneling allows one to pass *through* a barrier rather than having to jump *over* it. For a typical simulated annealing algorithm, each candidate state would be selected by choosing one spin at random to flip. Such modest steps would seem prudent in traversing a well-structured cost function but may prove insufficient when dealing with a virtually structureless landscape, as is typical of most hard optimization problems. This suggests that flipping several spins at once, in analogy with coherent quantum tunneling, may prove advantageous. On the other hand, such wild excursions could be suboptimal, or perhaps even disastrous.

This idea is not new. Based on insights from percolation theory, Swendsen and Wang suggested that lattice spin simulations may be performed more efficiently when large clusters are changed in a single move [20, 21]. This approach was shown to work well when clusters are chosen based on physically motivated problems, such as ferromagnetic spin lattices, but proved ineffective for more general spin glass problems. Houdayer has developed a cluster algorithm for two-dimensional spin glasses that moves between isoenergetic clusters in order to explore the state space more widely while avoiding costly random rejection steps in the Metropolis–Hastings algorithm [22]. Based on this work, Zhu, Ochoa, and Katzgraber have developed a more general isoenergetic cluster algorithm that is efficient for any spatial dimension, including the Chimera graph structure of the D-Wave system [23]. All of these approaches leverage either the specific graph structure or additional computation to select clusters.

The use of multiple spin flips is similar to the *k*-opt strategy used in local optimization, wherein a neighborhood of size *k* is examined to select the lowest energy state among *k* neighbors of the current state [24]. When *k* is determined adaptively, this heuristic often provides high-quality solutions in polynomial time, but some problem instances may require exponentially many iterations, as large neighborhoods must be searched exhaustively [25]. In Ref. [19] the *k*-opt procedure for fixed *k* is referred to as “algorithmic tunneling”, due to its similarity with quantum tunneling, and found to give improved performance over the popular differencing method of Karmarkar and Karp when applied to the number partitioning problem [26]. This suggests that large-neighborhood searches may indeed be advantageous if the cost of searching each neighborhood can be restricted.

In this paper, we consider a variation of the *k*-opt strategy in which the neighborhood is selected randomly, and may be as large as the state space, but is used only to select a single candidate state. We then examine the benefits, and tradeoffs, of such an approach as compared to a baseline simulated annealing procedure using only single-spin flips and a fast annealing schedule. Isakov and colleagues have considered a similar approach, wherein multiple spins flips are performed simultaneously and probabilistically based on the energy differences of individual spins [27]. Here we consider a variation of this approach wherein the candidate spin configuration is selected at random, independent of the energy cost. We also consider a modification of the standard Boltzmann acceptance probability to one resembling quantum tunneling probabilities and examine the relative impact of this modification.

The outline of the paper is as follows. Section 2 describes the baseline and modified simulated annealing algorithms considered. Their relative performance over a number of different problems and for large anneal times is considered in Sect. 3. In Sect. 4 we consider the relative impact on performance of reducing the anneal time, and we summarize our conclusions in Sect. 5.

## 2 Algorithm Description

The basic Metropolis algorithm may be described as follows. Given an energy cost function that we wish to minimize and an initial point in the corresponding state space, we select (by some means) a new candidate state. If the energy decreases, then the candidate state is accepted and the iterative process continues. Otherwise, the candidate state is accepted with a probability that decreases with a notional temperature parameter. In the original Metropolis algorithm, this probability is given by the Boltzmann factor. The random acceptance step allows the algorithm to escape local minima, but with a probability that decreases with temperature. Thus, if the system cools slowly enough, it will eventually settle into a global minimum [28].

*n*spin-\(\frac{1}{2}\) particles with \(N = 2^n\) possible spin configurations. The cost function is based on the general Ising model, which, for \(\varvec{s} \in \mathcal {S}\), takes the form

*i*and \(J_{ij} \in \mathbb {R}\) represents the coupling between spins

*i*and

*j*. Thus, when \(h_i\) and \(s_i\) are of the same sign, the energy is lowered. Similarly, \(J_{ij} > 0\) represents ferromagnetic coupling, which energetically favors similarly aligned spins, while \(J_{ij} < 0\) represents antiferromagnetic coupling, which favors oppositely aligned spins. For the problems we shall consider, the coupling is always symmetric (i.e., \(J_{ij} = J_{ji}\)), though no special graph connectivity constraints are assumed. The optimization problem consists of finding at least one point \(\varvec{s}_* \in \mathcal {S}\) such that \(E(\varvec{s}_*) \le E(\varvec{s})\) for all \(\varvec{s} \in \mathcal {S}\). The class of general Ising optimization problems, with no special coupling constraints, is known to be a nondeterministic polynomial (NP) hard [29].

By transforming \(\varvec{s}\) to \(\varvec{x} \in \{0, 1\}^n\), where \(x_i = (s_i + 1)/2\), the Ising problem may be converted to an equivalent Quadratic Unconstrained Binary Optimization (QUBO) problem. We may then take \(\varvec{x}\) to represent the binary digits of the integer \(x \in \{0, \ldots , N-1\}\), where \(x = x_1 2^{n-1} + \cdots + x_n 2^0\). In terms of \(\varvec{x}\), the cost function takes a similar form to that of the Ising model.

*n*, while for \(h_i = 0\) it scales quadratically. Specifically, we use the upper bound

*k*, we select a new, candidate state \(\varvec{s}'(k) \in \mathcal {S}\) such that one of the

*n*spins, chosen at random, is flipped in sign. If \(E(\varvec{s}'(k)) < E(\varvec{s}(k))\), this candidate state is accepted and \(\varvec{s}(k+1) = \varvec{s}'(k)\); otherwise, it is accepted with probability \(p(\varvec{s}'(k), \varvec{s}(k), T(k))\), where

The modified algorithm, which we call simulated annealing with multiple simultaneous spin flips (SAM), is identical to the baseline SA algorithm described above with the exception that, when selecting a candidate state, a random number of spins (drawn uniformly from 1 to *n*) are chosen to be flipped. This allows for wider excursions in the state space, similar to what is done in fast annealing methods for continuous state spaces [31, 32].

## 3 Performance Analysis

In what follows, we shall consider the performance of the baseline SA and proposed SAM algorithms relative to a number of different representative problems. First, a random realization of the problem is generated. For each of the two algorithms, the annealing procedure is then repeated a certain number of times, with a different, randomly selected initial state each time. The algorithms are applied to the same problem realization, and the resulting probability of success (i.e., of settling on one of the global minima) is computed. Upper and lower 95 % confidence bounds are computed using the (exact) Clopper-Pearson method [33].

The number of annealing steps is taken to be \(K = N\), which is the minimum number required to traverse the entire state space at least once using only single-spin flips. This guarantees that a solution can always be found, even if the algorithm fails to find it.

### 3.1 False Minimum Problem

This problem, also known as the “weak-strong cluster networks” problem, arises from the work by Boixo, Denchev, and colleagues, who use it to study the effects of collective tunneling on quantum annealing [17, 19]. The energy function has a global minimum at \(x = 0\) (all spin down) and a false minimum at \(x = N-1\) (all spin up). The local magnetic fields are defined such that \(h_i = 1-\varepsilon > 0\) for \(i \le n/2\) and \(h_i = -1\) for \(i > n/2\). The couplings are defined as in Ref. [17] and correspond to a Chimera graph structure, thereby restricting *n* to be a multiple of 4. The resulting energy gap between the false and true minima is \(\Delta E = n \varepsilon \).

*R*times: for \(n =4\), \(R = 10\,000\); for \(n \in \{8,12\}\), \(R = 1000\); for \(n = 16\), \(R = 100\). The results are shown in Fig. 1. As can be seen, the SA algorithm performs much more poorly than the SAM algorithm due to its propensity to get trapped in the false minimum. Interestingly, the SA performance becomes consecutively worse with increasing

*n*, while the SAM algorithm’s performance tends to increase with increasing problem size. This behavior is believed to be due to the fact that the fast annealing schedule used tends to limit the rate of acceptance and, hence, “freeze” the system early in the annealing process.

### 3.2 Zero Coupling Problem

It may be suspected that an overly aggressive move strategy may perform more poorly for “easy” problems in which a single spin flip can bring the state closer to the global minimum. We therefore compare the SA and SAM algorithms for the case in which there is no coupling between the spins (i.e., \(J_{ij} = 0\) for all *i*, *j*), and the local magnetic fields \(h_i\) are taken to \(\pm 1\) with equal probability.

*n*, a single problem realization is drawn and used for both algorithms. The results are shown in Fig. 2. As expected, the SA algorithm performs better than the SAM algorithm, particularly for low values of

*n*, but the two quickly converge in performance for

*n*greater than about 12.

### 3.3 Uniform Spin-Glass Problem

The spin-glass problem takes its inspiration from a physical spin glass consisting of a collection of disordered spins, itself analogous to the disordered positions of a glassy material [34, 35]. The uniform spin-glass problem is characterized by having \(h_i = 0\) for all *i* and \(J_{ij} = \pm 1\) with equal probability. By convention, \(J_{ii} = 0\) and \(J_{ji} = J_{ij}\). This problem is an example of a frustrated spin system and has been studied extensively in the context of both simulated and quantum annealing [12]. We consider spin-glass problems on a fully connected graph (i.e., one in which every spin is connected to every other spin), as more restricted graph structures may be solved more easily [36, 37].

*n*is shown in Fig. 3. There we see that the performance of SA is somewhat better than that of SAM for low values of

*n*(around 4), but for larger values the opposite is true. In particular, the probability of success for SA tends to decrease with increasing

*n*, while that of SAM tends to increase, saturating to near unity for

*n*greater than about 12. Although we show a single problem instance here for illustrative purposes, this behavior was found to be typical of random problem instances, as described later in Sect. 3.5.

### 3.4 Gaussian Spin-Glass Problem

The final type of problem considered was a Gaussian version of the spin-glass problem. The Gaussian spin-glass problem is similar to the uniform spin-glass problem, except that the coupling \(J_{ij}\) is now taken to be Gaussian distributed with zero mean and unit standard deviation. Because of the continuous distribution of coupling strengths, the gap between neighboring energy levels can be arbitrarily small, and, as *n* increases, this gap will tend to decrease. Consequently, we expect that the global minimum will be more difficult to find than in the case of the uniform spin-glass problem.

*n*(about 4), while SAM tends to outperform SA for larger problems. However, the performance of both methods tends to decrease with increasing

*n*for values larger than about 8.

### 3.5 Randomized Problems

The previous subsections considered a single problem instance for each value of *n* and compared the probability of success for the two methods SA and SAM as a function of problem size for multiple repetitions of the same problem instance. There we found that, typically, SAM performs better than SA, at least for the more difficult problems (i.e., \(n > 4\)). One may wonder how typical these results are over an ensemble of problem instances.

To address this question we considered 100 random realizations of the two spin-glass problems. For each problem realization, the two methods SA and SAM were run over multiple repetitions, as before. To compare them, we plot the time to solution for a 99 % probability of success for SAM versus that of SA, with a diagonal line indicating equivalency of the two methods. Given a probability of success \(p_{s}\) and total annealing time \(t_a\), the time to solution is given by \(t_{a}\log (1-0.99)/\log (1-p_{s})\), which represents the average number of repeated anneals, and their cumulative time, to achieve an overall \(99\,\%\) probability of success.

### 3.6 Quantum-like Tunneling

## 4 Reduced Annealing Times

*K*. Figure 9 plots the probabilities of success for SA, SAM, SAQ, and BF versus the relative annealing time

*K*/

*N*for two values of

*n*using random instances of the Gaussian spin-glass problem. From these plots, several observations can be made. For

*K*/

*N*near unity, the brute-force approach is, of course, the best of the three, although a more optimal annealing schedule would give nearly the equal performance. As

*n*increases, though, the SAM and SAQ methods quickly approach ideal performance, while the SA method experiences decreasing performance, much as was seen in the previous examples. There is, however, a crossover point below which the brute-force approach is no longer advantageous. This point increases with

*n*for SAM and SAQ but decreases with

*n*for SA. Thus, for near-maximal annealing times (\(K \lesssim N\)), BF is slightly better than SAM and SAQ, which is much better than SA.

For intermediate annealing times, SAM and SAQ are superior to SA, and both are superior to BF. There is, however, another crossover point such that, for any smaller number of annealing steps, SA outperforms SAM and SAQ. In this regime, SA benefits from locally optimal searches that, over many repetitions, start at different points in the configuration state space. SAM and SAQ, by contrast, search too widely for the limited annealing time allotted and, therefore, have trouble finding an optimal solution, even if they start near one. We note that the crossover point at which SA, SAM, and SAQ are equal corresponds to a relative annealing time *K* / *N* that decreases with increasing problem size, even though the total number of annealing steps, *K*, continues to increase. For the Gausian spin-glass problem, this crossover point is about \(K/N \approx 8\,\%\) for \(n = 12\) and \(K/N \approx 1\,\%\) for \(n = 16\).

The above examples considered success on a single instance of running the given algorithm for a certain annealing time *K*. One may also consider running the algorithm multiple times using a shorter annealing time \(K' \le K\) such that \(K/K' \in \mathbb {N}\) and taking the best result (i.e., the configuration giving the lowest energy). For the brute-force approach, it can be shown that this strategy is never advantageous; in other words, for all \(K'\) we have \(1-(1-p_{K'})^{K/K'} \le p_K\), since \(p_K\) drops sharply with decreasing *K*. The probabilities of success for the SA, SAM, and SAQ methods initially drop much more slowly with decreasing *K* and, so, for relatively large values of \(K'\) it may be quite advantageous to perform multiple shorter anneals. For very small values of \(K'\), however, the probability of success decreases more rapidly with decreasing \(K'\), eventually devolving to the brute-force approach, and, so, this becomes no longer an advantageous strategy. Thus, performing multiple repetitions of the simulated annealing process, for either method, can prove to be of great benefit, but care must be taken that the probability of success on any one instance is not too low (i.e., close to \(p_K\)).

## 5 Conclusions

In this paper we have considered and compared three different approaches (SA, SAM, SAQ) to performing classical simulated annealing on discrete spin systems. The baseline SA approach considers only one spin flip per annealing step, while the proposed SAM and SAQ methods flip a random and unbounded number of spins at each step. Both SAM and SAQ use an analogue of quantum spin coherence, with the former using a Boltzmann acceptance probability and the latter using an analogue to the quantum tunneling probability. Performance was considered using a variety of problem types and instances based on the general Ising model with arbitrary spin coupling, which were then solved numerically.

Based on this analysis, we find that the SAM and SAQ methods generally outperform the SA method when the problem is difficult and the number of annealing steps (i.e., the annealing time) is large. For easier problems or smaller annealing times, SA typically performs better than both SAM and SAQ. This may be interpreted to mean that when there is little time to search it is usually best to search locally. Of course, SA and SAM are two extremes in a spectrum of approaches, and an optimal strategy would likely use a number of simultaneous spin flips that increases with the anneal time, together with an optimized annealing schedule. The SAM and SAQ methods themselves perform nearly identically, suggesting that multiple simultaneous spin flips are the dominant mechanism for their performance.

We also found that it can be advantageous to perform multiple, shorter anneals rather than a single anneal with the same total time. For very short anneal times, however, the probability of success approaches that of a simple brute-force search and, hence, the advantage of multiple short anneals becomes lost. Thus, care must be taken in using an annealing procedure that individually gives a very low probability of success.

Finally, we note that, to the extent that these analogues do mimic behavior in a true quantum annealing system, a quantum speedup, relative to the baseline SA method, will only be realized for large anneal times. Of course, if the coherence times are much shorter than the anneal times, this advantage may never be realized. Nevertheless, using multiple spin flips it may be possible to mimic such a speedup by means of a suitable classical system.

## Notes

### Acknowledgments

This work was supported by the Office of the Secretary of Defense under Contract No. N00024-07-D-6200-0743 and by Applied Research Laboratories, The University of Texas at Austin, under an Independent Research and Development Grant.

## References

- 1.Kirkpatrick, S., Gelatt Jr., C.D., Vecchi, M.P.: Optimization by simulated annealing. Science
**220**, 671 (1983)ADSMathSciNetCrossRefMATHGoogle Scholar - 2.Metropolis, N., Rosenbluth, A.W., Rosenbluth, M.N., Teller, A.H., Teller, E.: Equation of state calculations by fast computing machines. J. Chem. Phys.
**21**, 1087 (1953)ADSCrossRefGoogle Scholar - 3.Hastings, W.K.: Monte Carlo sampling methods using Markov chains and their applications. Biometrika
**57**, 97 (1970)MathSciNetCrossRefMATHGoogle Scholar - 4.Gilks, W.R., Richardson, S., Spiegelhalter, D.J.: Markov Chain Monte Carlo in Practice. Chapman and Hall/CRC, London (1996)MATHGoogle Scholar
- 5.Farhi, E., Goldstone, J., Gutmann, S., Lapan, J., Lundgren, A., Preda, D.: A quantum adiabatic evolution algorithm applied to random instances of an np-complete problem. Science
**292**, 472 (2001)ADSMathSciNetCrossRefMATHGoogle Scholar - 6.Das, A., Chakrabarti, B.K.: Quantum annealing and analog quantum computation. Rev. Mod. Phys.
**80**, 1061 (2008)ADSMathSciNetCrossRefMATHGoogle Scholar - 7.Finnila, A.B., Gomez, M.A., Sebenik, C., Stenson, C., Doll, J.D.: Quantum annealing: a new method for minimizing multidimensional functions. Chem. Phys. Lett.
**219**, 343 (1994)ADSCrossRefGoogle Scholar - 8.Kadowaki, T., Nishimori, H.: Quantum annealing in the transverse Ising model. Phys. Rev. E
**58**, 5355 (1998)ADSCrossRefGoogle Scholar - 9.McGeoch C.C., Wang C.: Experimental evaluation of an adiabatic quantum system for combinatoral optimization. In: Proceedings of the ACM International Conference on Computing Frontiers, p. 23, ACM, New York (2013)Google Scholar
- 10.Perdomo-Ortiz, A., Fleugemann, J., Narasimhan, S., Biswas, R., Smelyanskiy, V.N.: A quantum annealing approach for fault detection and diagnosis of graph-based systems. Eur. Phys. J. Spec. Top.
**224**, 131 (2015)CrossRefGoogle Scholar - 11.O’Gorman, B., Perdomo-Ortiz, A., Babbush, R., Aspuru-Guzik, A., Smelyanskiy, V.: Bayesian network structure learning using quantum annealing. Eur. Phys. J. Spec. Top.
**225**, 163 (2015)CrossRefGoogle Scholar - 12.Venturelli, D., Mandrà, S., Knysh, S., O’Gorman, B., Biswas, R., Smelyanskiy, V.: Quantum optimization of fully connected spin glasses. Phys. Rev. X
**5**, 031040 (2015)Google Scholar - 13.King J., Yarkoni S., Nevisi M.M., Hilton J.P., McGeoch C.C.: Benchmarking a quantum annealing processor with the time-to-target metric, arXiv:1508.05087 (2015)
- 14.Rønnow, T.F., Wang, Z., Job, J., Boixo, S., Isakov, S.V., Wecker, D., Martinis, J.M., Lidar, D.A., Troyer, M.: Defining and detecting quantum speedup. Science
**345**, 420 (2014)ADSCrossRefGoogle Scholar - 15.Hen, I., Job, J., Albash, T., Rønnow, T.F., Troyer, M., Lidar, D.A.: Probing for quantum speedup in spin-glass problems with planted solutions. Phys. Rev. A
**92**, 042325 (2015)ADSCrossRefGoogle Scholar - 16.Mandrà S., Zhu Z., Wang W., Perdomo-Ortiz A., Katzgraber H.G.: Strengths and weaknesses of weak-strong cluster problems: A detailed overview of state-of-the-art classical heuristics vs quantum approaches, arXiv:1604.01746 (2016)
- 17.Boixo, S., Rønnow, T.F., Isakov, S.V., Wang, Z., Wecker, D., Lidar, D.A., Martinis, J.M., Troyer, M.: Evidence for quantum annealing with more than one hundred qubits. Nat. Phys.
**10**, 218 (2014)CrossRefGoogle Scholar - 18.Isakov S.V., Mazzola G., Smelyanskiy V.N., Jiang Z., Boixo S., Neven H., Troyer M.: Understanding quantum tunneling through quantum Monte Carlo simulations, arXiv:1510.08057 (2015)
- 19.Denchev V.S., Boixo S., Isakov S.V., Ding N., Babbush R., Smelyanskiy V., Martinis J., Neven H., What is the computational value of finite range tunneling? arXiv:1512.02206 (2015)
- 20.Swendsen, R.H., Wang, J.-S.: Nonuniversal critical dynamics in Monte Carlo simulations. Phys. Rev. Lett.
**52**, 86 (1987)ADSCrossRefGoogle Scholar - 21.Wolff, U.: Collective Monte Carlo updating for spin systems. Phys. Rev. Lett.
**62**, 361 (1989)ADSCrossRefGoogle Scholar - 22.Houdayer, J.: A cluster Monte Carlo algorithm for 2-dimensional spin glasses. Eur. Phys. J. B
**22**, 479 (2001)ADSCrossRefGoogle Scholar - 23.Zhu, Z., Ochoa, A.J., Katzgraber, H.G.: Efficient cluster algorithm for spin glasses in any space dimension. Phys. Rev. Lett.
**115**, 077201 (2015)ADSCrossRefGoogle Scholar - 24.Lin, S., Kernighan, B.W.: An effective heuristic algorithm for the Traveling-Salesman problem. Oper. Res.
**21**(2), 798 (1973)MathSciNetCrossRefMATHGoogle Scholar - 25.Chandra B., Karloff H., Tovey C., New results on the old \(k\)-opt algorithm for the TSP,” in Proceedings of the 5th Annual ACM-SIAM Symposium on Discrete Algorithms, p. 150 (1994)Google Scholar
- 26.Karmarkar N., Karp R.M.: The differencing method of set partitioning, type Tech. Rep. number UCB/CSD 82/113 ( institution Computer Science Division, University of California, Berkeley, CA, 1982)Google Scholar
- 27.Isakov, S.V., Zintchenko, I.N., Rønnow, T.F., Troyer, M.: Optimised simulated annealing for ising spin glasses. Comp. Phys. Comm.
**192**, 265 (2015)ADSMathSciNetCrossRefGoogle Scholar - 28.Salamon P., Sibani P., Frost R.: Facts, conjectures, and improvements for simulated annealing, in SIAM Monographs on Mathematical Modeling and Computation, Society for Industrial and Applied Mathematics, Philadelphia (2002)Google Scholar
- 29.Barahona, F.: On the complexity of Ising spin glass models. J. Phys. A
**15**, 3241 (1982)ADSMathSciNetCrossRefGoogle Scholar - 30.White S.R.: Concepts of scale in simulated annealing, in AIP Conf. Proc., vol. 122 (1984)Google Scholar
- 31.Szu, H., Hartley, R.: Fast simulated annealing. Phys. Lett. A
**122**, 157 (1987)ADSCrossRefGoogle Scholar - 32.Ingber, L.: Very fast simulated re-annealing. Math. Comput. Modeling
**12**, 967 (1989)MathSciNetCrossRefMATHGoogle Scholar - 33.Clopper, C., Pearson, E.S.: The use of confidence or fiducial limits illustrated in the case of the binomial. Biometrika
**26**, 404 (1934)CrossRefMATHGoogle Scholar - 34.Edwards, S.F., Anderson, P.W.: Theory of spin glasses. J. Phys. F
**5**, 965 (1975)ADSCrossRefGoogle Scholar - 35.Sherrington, D., Kirkpatrick, S.: Solvable model of a spin-glass. Phys. Rev. Lett.
**35**, 1792 (1975)ADSCrossRefGoogle Scholar - 36.Katzgraber, H.G., Hamze, F., Andrist, R.S.: Glassy Chimeras could be blind to quantum speedup: designing better benchmarks for quantum annealing machines. Phys. Rev. X
**4**, 019901 (2015)Google Scholar - 37.Zintchenko, I., Hastings, M.B., Troyer, M.: From local to global ground states in Ising spin glasses. Phys. Rev. B
**91**, 024201 (2015)ADSCrossRefGoogle Scholar - 38.Hamming, R.W.: Error detecting and error correcting codes. Bell Syst. Tech. J.
**29**, 147 (1950)MathSciNetCrossRefGoogle Scholar

## Copyright information

**Open Access**This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.