Skip to main content

Direct sampling with a step function

Statistics and Computing volume 33, Article number: 22 (2023) Cite this article

Abstract

The direct sampling method proposed by Walker et al. (JCGS 2011) can generate draws from weighted distributions possibly having intractable normalizing constants. The method may be of interest as a tool in situations which require drawing from an unfamiliar distribution. However, the original algorithm can have difficulty producing draws in some situations. The present work restricts attention to a univariate setting where the weight function and base distribution of the weighted target density meet certain criteria. Here, a variant of the direct sampler is proposed which uses a step function to approximate the density of a particular augmented random variable on which the method is based. Knots for the step function can be placed strategically to ensure the approximation is close to the underlying density. Variates may then be generated reliably while largely avoiding the need for manual tuning or rejections. A rejection sampler based on the step function allows exact draws to be generated from the target with lower rejection probability in exchange for increased computation. Several applications of the proposed sampler illustrate the method: generating draws from the Conway-Maxwell Poisson distribution, a Gibbs sampler which draws the dependence parameter in a random effects model with conditional autoregression structure, and a Gibbs sampler which draws the degrees-of-freedom parameter in a regression with t-distributed errors.

This is a preview of subscription content, access via your institution.

Access options

Buy single article

Instant access to the full article PDF.

USD 39.95

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

References

  • Achddou, J., Lam-Weil, J., Carpentier, A., Blanchard, G.: A minimax near-optimal algorithm for adaptive rejection sampling. In Aurélien Garivier and Satyen Kale, editors, Proceedings of the 30th International Conference on Algorithmic Learning Theory, volume 98 of Proceedings of Machine Learning Research, pages 94–126. PMLR, 22–24 Mar 2019

  • Ahrens, J.H.: Sampling from general distributions by suboptimal division of domains. Grazer Math. Berichte 319, 20 (1993)

    MATH  Google Scholar 

  • Ahrens, J.H.: A one-table method for sampling from continuous and discrete distributions. Computing 54(20), 127–146 (1995)

    Article  MathSciNet  MATH  Google Scholar 

  • Alzer, H.: On some inequalities for the gamma and psi functions. Math. Computat. 66(217), 373–389 (1997)

    Article  MathSciNet  MATH  Google Scholar 

  • Banerjee, S., Roy, A.: Linear Algebra and Matrix Analysis for Statistics. CRC, Chapman and Hall (2014)

  • Benson, A., Friel, N.: Bayesian inference, model selection and likelihood estimation using fast rejection sampling: the Conway-Maxwell-Poisson distribution. Bayes. Anal. 16(3), 905–931 (2021)

    MathSciNet  Google Scholar 

  • Braun, M. and Damien, P.: (2011) Generalized direct sampling for hierarchical Bayesian models. https://arxiv.org/abs/1108.2245

  • Braun, M., Damien, P.: Scalable rejection sampling for Bayesian hierarchical models. Market. Sci. 35(3), 427–444 (2016)

    Article  Google Scholar 

  • Carpenter, B., Gelman, A., Hoffman, M.D., Lee, D., Goodrich, B., Betancourt, M., Brubaker, M., Guo, J., Li, P., Riddell, A.: Stan: a probabilistic programming language. J. Statist. Soft. 76(1), 1–32 (2017)

    Article  Google Scholar 

  • Chanialidis, C., Evers, L., Neocleous, T., Nobile, A.: Efficient Bayesian inference for COM-Poisson regression models. Statist. Comput. 23, 595–608 (2018)

    Article  MathSciNet  MATH  Google Scholar 

  • Cormen, T.H., Leiserson, C.E., Rivest, R.L., Stein, C.: Introduction to Algorithms. The MIT Press: 3rd edition. (2009)

  • Cressie, N.: Statistics for Spatial Data. Wiley, New Jersey (1991)

    MATH  Google Scholar 

  • Devroye, L.: Non-Uniform Random Variate Generation. Springer, London (1986)

    Book  MATH  Google Scholar 

  • Duane, S., Kennedy, A.D., Pendleton, B.J., Roweth, D.: Hybrid Monte Carlo. Phys. Lett. B 195(2), 216–222 (1987)

    Article  MathSciNet  Google Scholar 

  • Erraqabi, A., Valko, M., Carpentier, A., Maillard, O.: Pliable rejection sampling. In Maria Florina Balcan and Kilian Q. Weinberger, editors, Proceedings of The 33rd International Conference on Machine Learning, volume 48 of Proceedings of Machine Learning Research, pages 2121–2129, New York, USA, 20–22 Jun 2016. PMLR

  • Eddelbuettel, D.: Seamless R and C++ Integration with Rcpp. Springer, New York (2013)

    Book  MATH  Google Scholar 

  • Evans, M., Swartz, T.: Random variable generation using concavity properties of transformed densities. J. Computat. Graph. Statist. 7(4), 514–528 (1998)

    Google Scholar 

  • Gaunt, R.E., Iyengar, S., Olde Daalhuis, A.B., Simsek, B.: An asymptotic expansion for the normalizing constant of the Conway-Maxwell-Poisson distribution. Ann. Instit. Statist. Math. 71, 163–180 (2019)

    Article  MathSciNet  MATH  Google Scholar 

  • Gelman, A., Carlin, J.B., Stern, H.S., Dunson, D.B., Vehtari, A., Rubin, D.B.: Bayesian Data Analysis, 3rd edn. CRC Press, Chapman and Hall (2013)

    Book  MATH  Google Scholar 

  • Geman, S., Geman, D.: Stochastic relaxation, Gibbs distributions, and the Bayesian restoration of images. IEEE Trans. Patt. Analy. Mach. Intell. 6(6), 721–741 (1984)

    Article  MATH  Google Scholar 

  • Geweke, J.: Priors for macroeconomic time series and their application. Econom. Theory 10(3–4), 609–632 (1994)

    Article  MathSciNet  Google Scholar 

  • Gilks, W.R., Wild, P.: Adaptive rejection sampling for Gibbs sampling. J. Royal Statist. Soc. Ser. C (Appl. Statist.) 41(2), 337–348 (1992)

  • Gilks, W.R., Best, N.G., Tan, K.K.C.: Adaptive rejection Metropolis sampling within Gibbs sampling. J. Royal Statist. Soc. Ser. C (Appl. Statist.) 44(4), 455–472 (1995)

  • Görür, D., Teh, Y.-W.: Concave-convex adaptive rejection sampling. J. Computat. Graph. Statist. 20(3), 670–691 (2011)

    Article  MathSciNet  Google Scholar 

  • Guikema, S.D., Goffelt, J.P.: A flexible count data regression model for risk analysis. Risk Analy. 28(1), 213–223 (2008)

    Article  Google Scholar 

  • Hastings, W.K.: Monte Carlo sampling methods using Markov chains and their applications. Biometrika 57(1), 97–109 (1970)

    Article  MathSciNet  MATH  Google Scholar 

  • Hoffman, M.D., Gelman, A.: The No-U-Turn sampler: Adaptively setting path lengths in Hamiltonian Monte Carlo. J. Mach. Learn. Res. 15(47), 1593–1623 (2014)

    MathSciNet  MATH  Google Scholar 

  • Holden, L., Hauge, R., Holden, M.: Adaptive independent Metropolis-Hastings. Ann. Appl. Probab. 19(1), 395–413 (2009)

    Article  MathSciNet  MATH  Google Scholar 

  • Hosszejni, D.: Bayesian estimation of the degrees of freedom parameter of the Student-\(t\) distribution—a beneficial re-parameterization, (2021). https://arxiv.org/abs/2109.01726

  • Joseph, M. mbjoseph/carstan: First release, December (2016). https://doi.org/10.5281/zenodo.210407

  • Lange, K.L., Little, R.J.A., Taylor, J.M.G.: Robust statistical modeling using the t distribution. J. Am. Statist. Assoc. 84(408), 881–896 (1989)

    MathSciNet  Google Scholar 

  • Lange, K.: Numerical Analysis for Statisticians, 2nd edn. Springer, London (2010)

    Book  MATH  Google Scholar 

  • Lee, D.: CARBayes: an R package for Bayesian spatial modeling with conditional autoregressive priors. J. Statist. Soft. 55(13), 1–24 (2013)

    Article  Google Scholar 

  • Levin, D.A., Peres, Y.: Markov Chains and Mixing Times. American Mathematical Society, 2nd edn, (2017)

  • Lee, D.: CARBayesdata: Data Used in the Vignettes Accompanying the CARBayes and CARBayesST Packages, (2020). URL https://CRAN.R-project.org/package=CARBayesdata. R package version 2.2

  • Martino, L., Míguez, J.: A generalization of the adaptive rejection sampling algorithm. Statist. Comput. 21(4), 633–647 (2011)

  • Murray, I., Ghahramani, Z., MacKay, D.J.C.: MCMC for doubly-intractable distributions. In Proceedings of the Twenty-Second Conference on Uncertainty in Artificial Intelligence, UAI’06, pages 359–366, Arlington, Virginia, USA, 2006. AUAI Press. ISBN 0974903922

  • Martino, L., Read, J., Luengo, D.: Independent doubly adaptive rejection Metropolis sampling within Gibbs sampling. IEEE Trans. Sign. Process. 63(12), 3123–3138 (2015)

    Article  MathSciNet  MATH  Google Scholar 

  • Martino, L., Luengo, D., Míguez, J.: Independent Random Sampling Methods. Springer International Publishing, Cham (2018)

    Book  MATH  Google Scholar 

  • Møller, J., Pettitt, A.N., Reeves, R., Berthelsen, K.K.: An efficient Markov chain Monte Carlo method for distributions with intractable normalising constants. Biometrika 93(2), 451–458 (2006)

    Article  MathSciNet  MATH  Google Scholar 

  • Metropolis, N., Rosenbluth, A.W., Rosenbluth, M.N., Teller, A.H., Teller, E.: Equation of state calculations by fast computing machines. J. Chem. Phys. 21(6), 1087–1092 (1953)

    Article  MATH  Google Scholar 

  • Neal, R.M.: Slice sampling. Ann. Statist. 31(3), 705–767 (2003)

    Article  MathSciNet  MATH  Google Scholar 

  • Nishimura, A., Dunson, D.B., Lu, J.: Discontinuous Hamiltonian Monte Carlo for discrete parameters and discontinuous likelihoods. Biometrika. 107(2), 365–380 (2020)

    Article  MathSciNet  MATH  Google Scholar 

  • R Core Team: R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing, Vienna, Austria, (2022). https://www.R-project.org/

  • Rivlin, T.J.: An Introduction to the Approximation of Functions. Dover, New York (1981)

  • Patil, G.P., Rao, C.R.: Weighted distributions and size-biased sampling with applications to wildlife populations and human families. Biometrics 34(2), 179–189 (1978)

    Article  MathSciNet  MATH  Google Scholar 

  • Salvatier, J., Wiecki, T.V., Fonnesbeck, C.: Probabilistic programming in Python using PyMC3. Peer J. Comput. Sci. 2, e55 (2016)

  • Shmueli, G, Minka, TP, Kadane JB, Borle, S, Boatwright, P: A useful distribution for fitting discrete data: revival of the Conway-Maxwell-Poisson distribution. J. Royal Statist. Soc. Ser. C (Appl. Statist) 54(1), 127–142 (2005)

  • Tanner, M.A., Wong, W.H.: The calculation of posterior distributions by data augmentation. J. Amer. Statist. Associat. 82(398), 528–540 (1987)

    Article  MathSciNet  MATH  Google Scholar 

  • Walker, S.G., Laud, P.W., Zantedeschi, D., Damien, P.: Direct sampling. J. Computat. Graph. Statist. 20(3), 692–713 (2011)

    Article  Google Scholar 

  • von Neumann, J.: Various techniques used in connection with random digits. In: Householder, A.S., Forsythe, G.E., Germond, H.H. (eds.) Monte Carlo Method, volume 12 of National Bureau of Standards Applied Mathematics Series, chapter 13, pp. 36–38. US Government Printing Office, Washington, DC. (1951)

    Google Scholar 

  • Zhou, G.: Mixed Hamiltonian Monte Carlo for mixed discrete and continuous variables. In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS’20, Red Hook, NY, USA, 2020. Curran Associates Inc. ISBN 9781713829546

Download references

Acknowledgements

The author is grateful to Drs. Scott Holan, Kyle Irimata, Ryan Janicki, and James Livsey at the U.S. Census Bureau for discussions which motivated this work, and to the associate editor and two anonymous referees for their attention to the manuscript.

Author information

Authors and Affiliations

  1. Center for Statistical Research and Methodology, U.S. Census Bureau, Washington, DC, 20233, U.S.A.

    Andrew M. Raim

Authors
  1. Andrew M. Raim
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Andrew M. Raim.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Disclaimer: This article is released to inform interested parties of ongoing research and to encourage discussion of work in progress. Any views expressed are those of the author and not those of the U.S. Census Bureau.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Raim, A.M. Direct sampling with a step function. Stat Comput 33, 22 (2023). https://doi.org/10.1007/s11222-022-10188-x

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s11222-022-10188-x

Keywords

  • Weighted distribution
  • Intractable normalizing constant
  • Inverse CDF sampling
  • Rejection sampling
  • Gibbs sampling