A Method to Compute an Appropriate Sample Size of a Two-Level Test for the NIST Test Suite

  • Hiroshi HaramotoEmail author
  • Makoto Matsumoto
Conference paper
Part of the Springer Proceedings in Mathematics & Statistics book series (PROMS, volume 241)


Statistical testing of pseudorandom number generators (PRNGs) is indispensable for their evaluation. A common difficulty among statistical tests is how we consider the resulting probability values (p-values). When a suspicious p-value, such as \(10^{-3}\), is observed, it is unclear whether it is due to a defect of the PRNG or merely by chance. In order to avoid such a difficulty, testing the uniformity of p-values provided by a certain statistical test is widely used. This procedure is called a two-level test. The sample size at the second level requires a careful choice because too large sample leads to the erroneous rejection, but this choice is usually done through experiments. In this paper, we propose a criterion of an appropriate sample size when we use the Frequency test, the Binary Matrix Rank test and the Runs test at the first level in the NIST test suite. This criterion is based on \(\chi ^2\)-discrepancy, which measures the differences between the expected distribution of p-values and the exact distribution of those. For example, when we use the Frequency test with the sample size \(10^6\) as the first level test, an upper bound on the sample size at the second level derived by our criterion is 125, 000.


Pseudorandom number generators Statistical testing NIST SP800-22 Two-level tests Chi-square discrepancy 



We are thankful to Editor Professor Art Owen and the referees, who informed of numerous improvements on the manuscript. This research has been supported in part by JSPS Grant-In-Aid #26310211, #15K13460, #16K13750, #17K14234, #18K03213, and JST CREST “Theory of hyper uniformity and its development in randomness appeared in sciences.”


  1. 1.
    Bassham III, L.E., Rukhin, A.L., Soto, J., Nechvatal, J.R., Smid, M.E., Barker, E.B., Leigh, S.D., Levenson, M., Vangel, M., Banks, D.L., Heckert, N.A., Dray, J.F., Vo, S.: Sp 800-22 rev. 1a. a statistical test suite for random and pseudorandom number generators for cryptographic applications. Technical report, National Institute of Standards & Technology, Gaithersburg, MD, United States (2010)Google Scholar
  2. 2.
    Gibbons, J.D., Chakraborti, S.: Nonparametric Statistical Inference, 5th edn. Chapman and Hall/CRC, Boca Raton (2010)zbMATHGoogle Scholar
  3. 3.
    L’Ecuyer, P.: Software for uniform random number generation: distinguishing the good and the bad. In: Proceedings of the 2001 Winter Simulation Conference, vol. 95–105. IEEE Press (2001).
  4. 4.
    L’Ecuyer, P., Simard, R.: TestU01: a C library for empirical testing of random number generators. ACM Trans. Math. Softw. 33(4), Art. 22, 40 (2007)CrossRefGoogle Scholar
  5. 5.
    L’Ecuyer, P., Cordeau, J.F., Simard, R.: Close-point spatial tests and their application to random number generators. Op. Res. 48(2), 308–317 (2000). Scholar
  6. 6.
    L’Ecuyer, P., Simard, R., Wegenkittl, S.: Sparse serial tests of uniformity for random number generators. SIAM J. Sci. Comput. 24(2), 652–668 (2002)MathSciNetCrossRefGoogle Scholar
  7. 7.
    Marsaglia, G., Tsay, L.H.: Matrices and the structure of random number sequences. Linear Algebra Appl. 67, 147–156 (1985). Scholar
  8. 8.
    Matsumoto, M., Nishimura, T.: Mersenne twister: a 623-dimensionally equidistributed uniform pseudo-random number generator. ACM Trans. Model. Comput. Simul. 8(1), 3–30 (1998)CrossRefGoogle Scholar
  9. 9.
    Matsumoto, M., Nishimura, T.: A nonempirical test on the weight of pseudorandom number generators. In: Monte Carlo and quasi-Monte Carlo methods 2000, pp. 381–395 (2000). Springer, Berlin (2002)CrossRefGoogle Scholar
  10. 10.
    Pareschi, F., Rovatti, R., Setti, G.: Second-level NIST randomness tests for improving test reliability. In: 2007 IEEE International Symposium on Circuits and Systems, pp. 1437–1440 (2007).
  11. 11.
    Pareschi, F., Rovatti, R., Setti, G.: On statistical tests for randomness included in the NIST SP800-22 test suite and based on the binomial distribution. IEEE Trans. Inf. Forensics Secur. 7(2), 491–505 (2012). Scholar
  12. 12.
    Simard, R., L’Ecuyer, P.: Computing the two-sided Kolmogorov-Smirnov distribution. J. Stat. Softw. 39(11), 1–18 (2011).

Copyright information

© Springer International Publishing AG, part of Springer Nature 2018

Authors and Affiliations

  1. 1.Ehime UniversityMatsuyama, EhimeJapan
  2. 2.Hiroshima UniversityHigashi-Hiroshima, HiroshimaJapan

Personalised recommendations