Computational Economics

, Volume 30, Issue 1, pp 19–40 | Cite as

Reproducible research in computational economics: guidelines, integrated approaches, and open source software

  • Giovanni BaiocchiEmail author


Traditionally, computer and software applications have been used by economists to off-load otherwise complex or tedious tasks onto technology, freeing up time and intellect to address other, intellectually more rewarding, aspects of research. On the negative side, this increasing dependence on computers has resulted in research that has become increasingly difficult to replicate. In this paper, we propose some basic standards to improve the production and reporting of computational results in economics for the purpose of accuracy and reproducibility. In particular, we make recommendations on four aspects of the process: computational practice, published reporting, supporting documentation, and visualization. Also, we reflect on current developments in the practice of computing and visualization, such as integrated dynamic electronic documents, distributed computing systems, open source software, and their potential usefulness in making computational and empirical research in economics more easily reproducible.


Economic methodology Econometric software Other computer software 

JEL classification

B4 C87 C88 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. Amman H., Kendrick D., Rust J. (eds). (1996). Handbook of computational economics (Vol 1). Amsterdam The Netherlands, Elsevier North-HollandGoogle Scholar
  2. Anderson R.G., Greene W.H., McCullough B., & Vinod H.D. (2005). The role of data and program code archives in the future of economic research. Working Paper No. 2005-014B, FRB of St. Louis.Google Scholar
  3. Baiocchi G. (2003). Managing econometric projects using Perl. Journal of Applied Econometrics 18(3): 371–378CrossRefGoogle Scholar
  4. Baiocchi G. (2004). Using Perl for statistics: Data processing and statistical computing. Journal of Statistical Software 11(1): 1–81Google Scholar
  5. Baiocchi G., Distaso W. (2003). GRETL: Econometric software for the GNU generation. Journal of Applied Econometrics 18(1): 105–110CrossRefGoogle Scholar
  6. Belsey D.A., Kuh E., Welsch R.E. (1980). Regression diagnostics. New York, WileyGoogle Scholar
  7. Buckheit J.B., Donoho D.L. (1995). Wavelets and statistics, chapter Wavelab and Reproducible Research. Berlin, New York, Springer, pp. 55–81Google Scholar
  8. Cleveland W.S. (1994). The elements of graphing data. Summit, New Jersey, Hobart PressGoogle Scholar
  9. Cleveland W.S. (1993). Visualizing data. Summit, New Jersey, Hobart PressGoogle Scholar
  10. de Leeuw J. (2005). On abandoning XLISP-STAT. Journal of Statistical Software 13(7): 1–81Google Scholar
  11. Dewald W.G., Thursby J.G., Anderson R.G. (1986). Replication in empirical economics: The journal of money, credit and banking project. American Economic Review 76(4): 587–603Google Scholar
  12. Eddelbuettel D. (2000). Econometrics with Octave. Journal of Applied Econometrics 15(5): 531–542CrossRefGoogle Scholar
  13. Eddelbuettel D. (2003). Quantian: A scientific computing environment. In Proceedings of the 3rd international workshop on distributed statistical computing (DSC 2003) March 20–22, Vienna, Austria, Vienna, Austria. Technische Universitt Wien.Google Scholar
  14. Gentle J.E. (2003). Random number generation and Monte Carlo methods (2nd ed). New York, SpringerGoogle Scholar
  15. Gentleman R. (2004). Some perspectives on statistical computing. Technical report, Department of Biostatistics, Harvard School of Public Health, Boston, Massachusetts, USA.Google Scholar
  16. Gentleman R. (2005). Reproducible research: A bioinformatics case study. Statistical Applications in Genetics and Molecular Biology, 4(1), Article 2. Available at: Scholar
  17. Gentleman R., & Lang D.T. (2004).Statistical analyses and reproducible research. Bioconductor Project Working Papers. Working Paper 2Google Scholar
  18. Greene W. (2000). Econometric analysis (4th ed). New York, Prentice HallGoogle Scholar
  19. Hoaglin D.C., Andrews D.F. (1975). The reporting of computation-based results in statistics. The American Statistician 29(3): 122–126CrossRefGoogle Scholar
  20. Ihaka R., Gentleman R. (1996). R: A language for data analysis and graphics. Journal of Computational and Graphical Statistics 5, 299–314CrossRefGoogle Scholar
  21. Judd K., Tesfatsion L. (eds). (2006). Handbook of computational economics: Agent-based computational economics (Vol 2). Amsterdam, The Netherlands, Elsevier North-HollandGoogle Scholar
  22. Kernighan B.W., Pike R. (1999). The practice of programming. Reading, MA, Addison-WesleyGoogle Scholar
  23. Kernighan B.W., Plauger P.J. (1978). The elements of programming style (2nd ed). New York, NY, McGraw HillGoogle Scholar
  24. Knopper K. (2003). Knoppix. Available at Scholar
  25. Knüsel L. (1995). On the accuracy of statistical distributions in GAUSS. Computational Statistics and Data Analysis 20, 699–702CrossRefGoogle Scholar
  26. Knuth D.E. (1983). Literate programming. Technical report STAN-CS-83-981, Stanford University, Department of Computer Science.Google Scholar
  27. Knuth D.E. (1984). Literate programming. The Computer Journal 27(2): 97–111CrossRefGoogle Scholar
  28. Knuth D.E. (1992). Literate programming. CSLI Lecture Notes Number 27 Stanford, CA, USA: Stanford University Center for the Study of Language and Information.Google Scholar
  29. Koenker R. (1996). Reproducible econometric research. Technical report, Department of Econometrics, University of Illinois, Urbana-Champaign, IL.Google Scholar
  30. Koenker R. (2006). Reproducibility in econometrics research. Technical report, Department of Econometrics, University of Illinois, Urbana-Champaign, IL.∼ roger/repro.html.Google Scholar
  31. Leisch F. (2002). Dynamic generation of statistical reports using literate data analysis. In: Härdle W. (eds), Proceedings in computational statistics. Heidelberg Germany, Physika Verlag, pp. 575–580Google Scholar
  32. Leontief W.W. (1966). Input-output economics. In: Leontief W.W. (eds), Input-output economics chapter 2. New York, Oxford University Press, pp. 13–29Google Scholar
  33. Lerner J., Triole J. (2002). The simple economics of open source. Journal of Industrial Economics 52, 197–234Google Scholar
  34. MacKinnon J.G. (1999). The Linux operating system: Debian GNU/Linux. Journal of Applied Econometrics 14(4): 443–452CrossRefGoogle Scholar
  35. McCullough B. (1998). Assessing the reliability of statistical software: Part I. The American Statistician 52, 358–366CrossRefGoogle Scholar
  36. McCullough B. (1999). Assessing the reliability of statistical software: Part II. The American Statistician 53(1): 149–159CrossRefGoogle Scholar
  37. McCullough B., Vinod H. (1999a). The numerical reliability of econometric software. Journal of Economic Literature 37(2): 633–665Google Scholar
  38. McCullough B., Vinod H. (1999b). The numerical reliability of econometric software. Journal of Economic Literature XXXVII: 633–665Google Scholar
  39. McCullough B.D., McGeary K.A., Harrison T.D. (2006). Lessons from the JMCB Archive. Journal of Money, Credit, and Banking 38(4): 1093–1107CrossRefGoogle Scholar
  40. McCullough B.D., Renfro C.G. (1999). Benchmarks and software standards: A case study of GARCH procedures. Journal of Economic and Social Measurement 25(2): 59–71Google Scholar
  41. Racine J. (2000). The cygwin tools: a GNU toolkit for windows. Journal of Applied Econometrics 15(3): 331–341CrossRefGoogle Scholar
  42. Racine J., Hyndman R. (2002). Using R to teach econometrics. Journal of Applied Econometrics 17(2): 175–189CrossRefGoogle Scholar
  43. Ramanathan R. (2002). Introductory econometrics with applications (5th ed). Orlando Florida, Harcourt College PublishersGoogle Scholar
  44. Rossini A.J., Heiberger R.M., Sparapani R., Mächler M., Hornik K.(2004). Emacs speaks statistics: A multiplatform, multi-package development environment for statistical analysis. Journal of Computational and Graphical Statistics 13(1): 247–261CrossRefGoogle Scholar
  45. Sawitzki G. (1999). Software components and document integration for statistical computing. In Proceedings ISI Helsinki 1999 (52nd session) Bulletin of the International Statistical Institute Tome LVIII, Book 2, pp. 117–120Google Scholar
  46. Sawitzki G. (2005). Keeping statistics alive in documents. Computational Statistics 17(1): 65–88CrossRefGoogle Scholar
  47. Schwab M., Karrenbach M., Claerbout J. (2000). Making scientific computations reproducible. Computing in Science and Engineering 2(6): 61–67CrossRefGoogle Scholar
  48. St. Laurent A.M. (2004). Understanding open source and free software licensing: A straightforward guide to the complex world of licensing. Sebastopol, CA, USA: O’Reilly & Associates.Google Scholar
  49. Stallman R. (1985). The GNU manifesto. Dr. Dobb’s Journal of Software Tools 10(3): 30–35Google Scholar
  50. Stokes H. (2004). On the advantage of using two or more econometric software systems to solve the same problem. Journal of Economic and Social Measurement 29, 307–320Google Scholar
  51. Tufte E.R. (2001). The visual display of quantitative information (2nd ed). Cheshire Connecticut, Graphics PressGoogle Scholar
  52. Ueberhuber C.W. (1997). Numerical computation: Methods, software, and analysis (Vol 1). Berlin Heidelberg, Germany, SpringerGoogle Scholar
  53. Varian H.R. (eds). (1992). Economic and financial modeling with mathematica. New York, TELOS/SpringerGoogle Scholar
  54. Varian H.R. (eds). (1996). Computational economics: Economic and financial analysis with mathematica. New York, TELOS/SpringerGoogle Scholar
  55. Vinod H.D. (2000). Review of GAUSS for windows, including its numerical accuracy. Journal of Applied Econometrics 14(2): 211–220CrossRefGoogle Scholar
  56. Vinod H.D. (2001). Care and feeding of reproducible econometrics. Journal of Econometrics 100(1): 87–88CrossRefGoogle Scholar
  57. Wooldridge J. (2002). Introductory econometrics: A modern approach (2nd ed). Mason OH, Thomson South-WesternGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC 2007

Authors and Affiliations

  1. 1.Department of Economics and FinanceUniversity of DurhamDurhamUK

Personalised recommendations