# Parallel probability density approximation

## Abstract

Probability density approximation (PDA) is a nonparametric method of calculating probability densities. When integrated into Bayesian estimation, it allows researchers to fit psychological processes for which analytic probability functions are unavailable, significantly expanding the scope of theories that can be quantitatively tested. PDA is, however, computationally intensive, requiring large numbers of Monte Carlo simulations in order to attain good precision. We introduce *Parallel PDA* (pPDA), a highly efficient implementation of this method utilizing the Armadillo C++ and CUDA C libraries to conduct millions of model simulations simultaneously in graphics processing units (GPUs). This approach provides a practical solution for rapidly approximating probability densities with high precision. In addition to demonstrating this method, we fit a piecewise linear ballistic accumulator model (Holmes, Trueblood, & Heathcote, 2016) to empirical data. Finally, we conducted simulation studies to investigate various issues associated with PDA and provide guidelines for pPDA applications to other complex cognitive models.

## Keywords

R C++ CUDA GPU Kernel density estimate Markov chain Monte Carlo Bayesian modeling Probability density approximation## Notes

### Author note

W.R.H. was supported by National Science Foundation (USA) Grant SES-1530760. A.H. is supported by Australian Research Council Discovery Project DP160101891

## References

- Ball, K., & Sekuler, R. (1982). A specific and enduring improvement in visual motion discrimination.
*Science*,*218*, 697–698. https://doi.org/10.1126/science.7134968 CrossRefPubMedGoogle Scholar - Beaumont, M. A. (2010). Approximate Bayesian computation in evolution and ecology.
*Annual Review of Ecology, Evolution, and Systematics*,*41*, 379–406.CrossRefGoogle Scholar - Brooks, S., Gelman, A., Jones, G., & Meng, X. L. (2011). Handbook of Markov chain Monte Carlo. New York, NY: CRC Press.CrossRefGoogle Scholar
- Brown, S. D., & Heathcote, A. (2008). The simplest complete model of choice response time: Linear ballistic accumulation.
*Cognitive Psychology*,*57*, 153–178. https://doi.org/10.1016/j.cogpsych.2007.12.002 CrossRefPubMedGoogle Scholar - Cheng, J., Grossman, M., & McKercher, T. (2014). Professional CUDA C programming. Indianapolis, IN: Wiley.Google Scholar
- Chiu, S.-T. (1991). Bandwidth selection for kernel density estimation.
*Annals of Statistics*,*19*, 1883–1905. Retrieved from http://www.jstor.org/stable/2241909 CrossRefGoogle Scholar - Cisek, P., Puskas, G. A., & El-Murr, S. (2009). Decisions in changing conditions: the urgency-gating model.
*Journal of Neuroscience*,*29*, 11560–11571CrossRefGoogle Scholar - Dawson, M. R. (1988). Fitting the ex-Gaussian equation to reaction time distributions.
*Behavior Research Methods, Instruments, & Computers*,*20*, 54–57.CrossRefGoogle Scholar - Dutilh, G., Annis, J., Brown, S. D., Cassey, P., Evans, N. J., Grasman, R. P. P. P., . . . Donkin, C. (2018). The quality of response time data inference: A blinded, collaborative assessment of the validity of cognitive models.
*Psychonomic Bulletin & Review*. Advance online publication. https://doi.org/10.3758/s13423-017-1417-2 CrossRefGoogle Scholar - Gelman, A. (2014). Bayesian data analysis. Boca Raton, FL: CRC Press.Google Scholar
- Goldenshluger, A., & Lepski, O. (2011). Bandwidth selection in kernel density estimation: Oracle inequalities and adaptive minimax optimality.
*Annals of Statistics*,*39*, 1608–1632. https://doi.org/10.1214/11-AOS883 CrossRefGoogle Scholar - Gureckis, T. M., & Love, B. C. (2009). Learning in noise: Dynamic decision-making in a variable environment.
*Journal of Mathematical Psychology*,*53*, 180–193.CrossRefGoogle Scholar - Harris, M. (2007). Optimizing parallel reduction in CUDA. Retrieved from http://docs.nvidia.com/cuda/samples/6_Advanced/reduction/doc/reduction.pdf
- Heathcote, A. (2004). Fitting Wald and ex-Wald distributions to response time data: An example using functions for the S-PLUS package.
*Behavior Research Methods, Instruments, & Computers*,*36*, 678–694. https://doi.org/10.3758/BF03206550 CrossRefGoogle Scholar - Heathcote, A., Brown, S. D., & Wagenmakers, E.-J. (2015). An introduction to good practices in cognitive modeling. In B. U. Forstmann & E.-J. Wagenmakers (Eds.), An introduction to model-based cognitive neuroscience (pp. 25–48). New York, NY, US: Springer Science + Business Media.Google Scholar
- Heathcote, A., Lin, Y.-S., Reynolds, A., Strickland, L., Gretton, M., & Matzke, D. (2018). Dynamic models of choice.
*Behavior Research Methods*. Advance online publication. https://doi.org/10.3758/s13428-018-1067-y CrossRefGoogle Scholar - Hoffman, M. D., & Gelman, A. (2014). The no-U-turn sampler: Adaptively setting path lengths in Hamiltonian Monte Carlo.
*Journal of Machine Learning Research*,*15*, 1593–1623.Google Scholar - Hohle, R. H. (1965). Inferred components of reaction times as functions of foreperiod duration.
*Journal of Experimental Psychology*,*69*, 382–386. https://doi.org/10.1037/h0021740 CrossRefPubMedGoogle Scholar - Holmes, W. R. (2015). A practical guide to the Probability Density Approximation (PDA) with improved implementation and error characterization.
*Journal of Mathematical Psychology*,*68–69*, 13–24. https://doi.org/10.1016/j.jmp.2015.08.006 CrossRefGoogle Scholar - Holmes, W. R., & Trueblood J. S. (2018). Bayesian analysis of the piecewise diffusion decision model.
*Behavior Research Methods*,*50*, 730–743. https://doi.org/10.3758/s13428-017-0901-y CrossRefPubMedGoogle Scholar - Holmes, W. R., Trueblood, J. S., & Heathcote, A. (2016). A new framework for modeling decisions about changing information: The piecewise linear ballistic accumulator model.
*Cognitive Psychology*,*85*, 1–29.CrossRefGoogle Scholar - Hu, B., & Tsui, K.-W. (2005). Distributed evolutionary Monte Carlo with applications to Bayesian analysis (Working paper). Madison, WI: University of Wisconsin, Department of Statistics.Google Scholar
- Hu, B., & Tsui, K.-W. (2010). Distributed evolutionary Monte Carlo for Bayesian computing.
*Computional Statistics and Data Analysis*,*54*, 688–697. https://doi.org/10.1016/j.csda.2008.10.025 CrossRefGoogle Scholar - Luce, R. D (1986). Response times. New York, NY: Oxford University Press.Google Scholar
- Matzke, D., & Wagenmakers, E.-J. (2009). Psychological interpretation of the ex-Gaussian and shifted Wald parameters: A diffusion model analysis.
*Psychonomic Bulletin & Review*,*16*, 798–817. https://doi.org/10.3758/PBR.16.5.798 CrossRefGoogle Scholar - McClelland, J. L. (1979). On the time relations of mental processes: An examination of systems of processes in cascade.
*Psychological Review*,*86*, 287–330. https://doi.org/10.1037/0033-295X.86.4.287 CrossRefGoogle Scholar - Miletić, S., Turner, B. M., Forstmann, B. U., & van Maanen, L. (2017). Parameter recovery for the Leaky Competing Accumulator model.
*Journal of Mathematical Psychology*,*76*, 25–50. https://doi.org/10.1016/j.jmp.2016.12.001 CrossRefGoogle Scholar - Neal, R. M. (1994). An improved acceptance procedure for the hybrid Monte Carlo algorithm.
*Journal of Computational Physics*,*111*, 194–203.CrossRefGoogle Scholar - Nvidia. (2018). CUDA C programming guide PG-02829-001_v9.1 | March 2018. Retrieved 20 Apr 2018 from https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html
- Palestro, J. J., Sederberg, P. B., Osth, A. F., van Zandt, T., & Turner, B. M. (2018). Likelihood-free methods for cognitive science. Cham, Switzerland: Springer.CrossRefGoogle Scholar
- Parzen, E. (1962). On estimation of a probability density function and mode.
*Annals of Mathematical Statistics*,*33*, 1065–1076.CrossRefGoogle Scholar - R Core Team. (2017). R: A language and environment for statistical computing. Vienna, Austria: R Foundation for Statistical Computing. Retrieved from https://www.R-project.org/ Google Scholar
- Ratcliff, R. (1978). A theory of memory retrieval.
*Psychological Review*,*85*, 59–108. https://doi.org/10.1037/0033-295X.85.2.59 CrossRefGoogle Scholar - Ratcliff, R., & McKoon, G. (2008). The diffusion decision model: Theory and data for two-choice decision tasks.
*Neural Computation*,*20*, 873–922. https://doi.org/10.1162/neco.2008.12-06-420 CrossRefPubMedPubMedCentralGoogle Scholar - Robert, C. P. (1995). Simulation of truncated normal variables.
*Statistics and Computing*,*5*, 121–125. https://doi.org/10.1007/BF00143942 CrossRefGoogle Scholar - Roberts, G. O., & Rosenthal, J. S. (2001). Optimal scaling for various Metropolis–Hastings algorithms.
*Statistical Science*,*16*, 351–367.CrossRefGoogle Scholar - Sanderson, C., & Curtin, R. (2016). Armadillo: A template-based C++ library for linear algebra.
*Journal of Open Source Software*,*1*(2), 26. https://doi.org/10.21105/joss.00026 CrossRefGoogle Scholar - Schwarz, W. (2002). On the convolution of inverse Gaussian and exponential random variables.
*Communications in Statistics Theory and Methods*,*31*, 2113–2121.CrossRefGoogle Scholar - Silverman, B. W. (1982). Algorithm AS 176: Kernel density estimation using the fast Fourier transform.
*Journal of the Royal Statistical Society: Series C*,*31*, 93–99. https://doi.org/10.2307/2347084 CrossRefGoogle Scholar - Silverman, B. W. (1986). Density estimation for statistics and data analysis. London, UK: Chapman & Hall.CrossRefGoogle Scholar
- Sisson, S. A., & Fan, Y. (2010).
*Likelihood-free Markov chain Monte Carlo*. arXiv preprint. arXiv:1001.2058.Google Scholar - Smith, P. L. (2016). Diffusion theory of decision making in continuous report.
*Psychological Review*,*123*, 425–451. https://doi.org/10.1037/rev0000023 CrossRefPubMedGoogle Scholar - Spiegelhalter, D. J., Best, N. G., Carlin, B. P., & van der Linde, A. (2002). Bayesian measures of model complexity and fit.
*Journal of the Royal Statistical Society: Series B*,*64*, 583–639.CrossRefGoogle Scholar - Ter Braak, C. J. F. (2006). A Markov Chain Monte Carlo version of the genetic algorithm Differential Evolution: Easy Bayesian computing for real parameter spaces.
*Statistics and Computing*,*16*, 239–249.CrossRefGoogle Scholar - Thura, D., Beauregard-Racine, J., Fradet, C.-W., & Cisek, P. (2012). Decision-making by urgency gating: Theory and experimental support.
*Journal of Neurophysiology*,*108*, 2912–2930.CrossRefGoogle Scholar - Trueblood, J. S., Holmes, W. R., Seegmiller, A. C., Douds, J., Compton, M., Szentirmai, E., . . . Eichbaum, Q. (2018). The impact of speed and bias on the cognitive processes of experts and novices in medical image decision-making.
*Cognitive Research: Principles and Implications*,*3*, 28.Google Scholar - Tsetsos, K., Usher, M., & McClelland, J. L. (2011). Testing multi-alternative decision models with non-stationary evidence.
*Frontiers in Neuroscience*,*5*, 63. https://doi.org/10.3389/fnins.2011.00063 CrossRefPubMedPubMedCentralGoogle Scholar - Turner, B. M., & Sederberg, P. B. (2012). Approximate Bayesian computation with differential evolution.
*Journal of Mathematical Psychology*,*56*, 375–385.CrossRefGoogle Scholar - Turner, B. M., & Sederberg, P. B. (2014). A generalized, likelihood-free method for posterior estimation.
*Psychonomic Bulletin & Review*,*21*, 227–250. https://doi.org/10.3758/s13423-013-0530-0 CrossRefGoogle Scholar - Turner, B. M., Sederberg, P. B., Brown, S. D., & Steyvers, M. (2013). A method for efficiently sampling from distributions with correlated dimensions.
*Psychological Methods*,*18*, 368–384. https://doi.org/10.1037/a0032222 CrossRefPubMedPubMedCentralGoogle Scholar - Usher, M., & McClelland, J. L. (2001). The time course of perceptual choice: The leaky, competing accumulator model.
*Psychological Review*,*108*, 550–592. https://doi.org/10.1037/0033-295X.111.3.757 CrossRefGoogle Scholar - Van Zandt, T. (2000). How to fit a response time distribution.
*Psychonomic Bulletin & Review*,*7*, 424–465. https://doi.org/10.3758/BF03214357 CrossRefGoogle Scholar - Verdonck, S., Meers, K., & Tuerlinckx, F. (2016). Efficient simulation of diffusion-based choice RT models on CPU and GPU.
*Behavior Research Methods*,*48*, 13–27. https://doi.org/10.3758/s13428-015-0569-0 CrossRefPubMedGoogle Scholar