Skip to main content

Assessing the effect of high performance computing capabilities on academic research output

Abstract

This paper uses nonparametric methods and some new results on hypothesis testing with nonparametric efficiency estimators and applies these to analyze the effect of locally available high performance computing (HPC) resources on universities’ efficiency in producing research and other outputs. We find that locally available HPC resources enhance the technical efficiency of research output in Chemistry, Civil Engineering, Physics, and History, but not in Computer Science, Economics, nor English; we find mixed results for Biology. Our research results provide a critical first step in a quantitative economic model for investments in HPC.

This is a preview of subscription content, access via your institution.

Fig. 1
Fig. 2

Notes

  1. 1.

    HPC instrumentation is commonly referred to as “supercomputers,” but is heterogeneous in nature. By “HPC instrumentation,” or simply “HPC,” we mean specialized computer systems designed to solve challenging computational problems that cannot be solved using desktop or similar, commonly available machines. As such, what constitutes HPC instrumentation necessarily evolves over time; what was regarded as HPC in 1980, 1990, or even 2000 would be unremarkable today aside from historical interest. Today, HPC instruments typically consist of large clusters, where hundreds or thousands of central processing units (CPUs) in close proximity to each other are linked by a fast, tightly integrated network, but may also include specialized machines capable of producing extraordinary numbers of computations per second. Our use of “HPC” does not include grid computing systems such as Condor systems, which may offer high-throughput computing for certain types of problems, but which are not well-suited for applications that require a large amount of communication between CPUs.

  2. 2.

    Examples of such programs funded by NSF are the Blue Waters project and the Extreme Science and Engineering Discovery Environment (XSEDE). One can also purchase time on HPC systems from Amazon and other providers.

  3. 3.

    As is well known, the nonparametric efficiency estimators described below in Sect. 2 do not permit measurement error. Then again, the normally distributed error term typically included on the right-hand side of fully parametric models allows only for measurement error in the left-hand side variable; any measurement error in the right-hand side variables that is correlated with the error terms leads to endogeneity problems. We do not claim that our choice of estimators is perfect; however, our choice addresses two obvious problems. See the recent survey by Simar and Wilson (2013) for a discussion of other estimation methods that might be used and the various tradeoffs that are involved.

  4. 4.

    Throughout, inequalities involving vectors are assumed to hold element by element; e.g., \({\varvec{a}}\le {\varvec{b}}\) denotes \(a_j\le b_j\) for each \(j=1,\;\ldots ,\;k\), where \(k\) is the length of \({\varvec{a}}\) and \({\varvec{b}}\).

  5. 5.

    Kneip et al. (1998, 2008, 2014) work in the input direction, but the results of each extend trivially to the output direction after appropriate changes in notation. The characterization of smoothness in Assumption 2.6 is stronger than required for the consistency of the nonparametric estimators. Kneip et al. (1998) require only Lipschitz continuity of the efficiency scores, which is implied by the simpler, but stronger requirement presented here. However, derivation of limiting distributions of the nonparametric estimators has been obtained by Kneip et al. (2008) only with the stronger assumption made here.

  6. 6.

    The proof given by Park et al. (2000) is for the input-oriented estimator, but the result extends almost trivially to the output orientation. See Daouia et al. (2013) for a much simpler proof relying on results from extreme value theory.

  7. 7.

    The frontier \({\mathcal {P}}^\partial \) exhibits globally CRS if and only if for any \(({\varvec{x}},{\varvec{y}})\in {\mathcal {P}}\), \((t{\varvec{x}},t{\varvec{y}})\in {\mathcal {P}}\;\forall \;t\in [0,\infty )\).

  8. 8.

    Alternatively, one might want to test CRS versus non-increasing returns to scale (NIRS). The DEA, NIRS estimator \(\widehat{\mathcal {P}}_\text {NIRS}({\mathcal {S}}_n)\) of \({\mathcal {P}}\) is obtained by changing the constraint \({\varvec{i}}_n'{\varvec{q}}=1\) in (2.5) to \({\varvec{i}}_n'{\varvec{q}}\le 1\). By construction, \(\widehat{\mathcal {P}}_\text {VRS}({\mathcal {S}}_n)\subseteq \widehat{\mathcal {P}}_\text {NIRS}({\mathcal {S}}_n)\subseteq \widehat{\mathcal {P}}_\text {CRS}({\mathcal {S}}_n)\). Kneip et al. (2013) establish that the VRS estimator of \(\lambda ({\varvec{x}},{\varvec{y}})\) remains consistent under CRS (which is no surprise), but with the faster CRS convergence rate \(n^{2/(p+q)}\) (which is surprising). Properties of the NIRS estimator of \(\lambda ({\varvec{x}},{\varvec{y}})\) remain unknown, and are likely complicated in view of the result of Kneip et al. (2013) for the VRS estimator under CRS. Since \(\widehat{\mathcal {P}}_\text {VRS}({\mathcal {S}}_n)= \widehat{\mathcal {P}}_\text {NIRS}({\mathcal {S}}_n)\) in the limit if \({\mathcal {P}}^\partial \) is NIRS, it is easy to imagine that the VRS estimator of \(\lambda ({\varvec{x}},{\varvec{y}})\) remains consistent if \({\mathcal {P}}^\partial \) is NIRS, though its convergence rate might depend on where the point \(({\varvec{x}},{\varvec{y}})\) lies in \({\mathcal {P}}\). Since the properties of the NIRS estimator have not been worked out, we test CRS versus VRS instead of testing CRS versus NIRS.

  9. 9.

    Throughout, the subscript “\(\bullet \)” will be used when either “FDH,” “VRS,” or “CRS” is applicable.

  10. 10.

    Note that the result in (3.19) is valid only for \((p+q)\le 5\). See Kneip et al. (2013) for an alternative test statistic for cases where \((p+q)>5\).

  11. 11.

    The NRC data count publications by departments according to authors’ affiliations, with no adjustments for coauthor relationships. Hence a publication coauthored by two authors in the same department is counted only once, while a publication coauthored by two authors in different departments is counted twice, once for each department. While this is not perfect, it is far from clear what would be an ideal weighting scheme. Deans, department chairs, and authors of articles on department rankings have used a variety of schemes, and there seems to be little agreement on what might be the “best” weighting scheme. Given that we examine a set of eight disparate disciplines, we have little choice but to use the data that are available; the alternative would be to search through all publications in our eight disciplines appearing in our time frame, identify the affiliation of each publication’s authors, and then apply what would still be an arbitrary weighting scheme. This would present a formidable computational burden, perhaps requiring machine learning and other methods. In addition, one would likely want to use different weighting schemes across different disciplines; e.g., in Economics, it is somewhat uncommon for graduate assistants to be listed as coauthors, whereas this is more common in Computer Science and perhaps other disciplines.

  12. 12.

    We suspect that in most cases, universities have substantial incentives to submit benchmark times. Appearing on the Top 500 list is a marketing tool for faculty and graduate student recruitment, and may provide prestige for administrators, while the opportunity cost is a few days of down-time for a system, and perhaps a few disgruntled faculty users. On the other hand, benchmark times for systems owned by secretive government agencies or private companies may be less likely to be submitted; for private companies, the system down-time required to run the benchmark translates into foregone profits. So, while the Top 500 list may be unreliable for gauging HPC capacity of governments and private companies, it likely gives a much better idea of HPC capacity for universities.

  13. 13.

    These data are available upon request.

  14. 14.

    Nor do we have information on queuing times for jobs that run on Top 500 systems. In principle, we could look at the number of processors, nodes, or theoretical performance (in terms of floating point operations per second) per faculty member, but this would be misleading for several reasons. For example, on some systems (e.g., the Palmetto cluster at Clemson University) faculty who contribute funds are given ownership rights, and have higher priority than other uses. In addition, some systems contribute resources to the XSEDE program or sell time to commercial users. Many systems exhibit peak-load characteristics; e.g., usage may be higher near the ends of academic semesters and in the weeks before NSF submission deadlines than at other times.

  15. 15.

    In each case, the sample means are computed using all observations in a given group. As made clear by Kneip et al. (2014), the “full” sample mean remains a consistent estimator, although it is not useful for making inference if \((p+q)\) is too large.

  16. 16.

    For English, History, and Physics, Theorem 4.4 of Kneip et al. (2014) as well as the test of equivalent mean efficiency across haves and have-nots require a sample mean of a subset of estimated efficiencies within each group since the FDH estimator is used, and its convergence rate of \(1/(p+q)=1/4\) is less than \(1/3\). The sample mean based on all of the available efficiency estimates remains, however, a consistent estimator of the corresponding population mean. See Kneip et al. (2014) for additional discussion.

  17. 17.

    Recall from the discussion in Section 2 that by construction, the efficiency estimates are bounded below at 0 and above at 1. Ordinary kernel density estimators are biased and inconsistent near support boundaries. This problem is addressed by using the reflection method described by Silverman (1986) and Simar and Wilson (1998) to produce the density estimates shown in Fig. 2.

References

  1. Afriat S (1972) Efficiency estimation of production functions. Int Econ Rev 13:568–598

    Article  Google Scholar 

  2. Apon A, Ahalt S, Dantuluri V, Gurdgiev C, Limayem M, Ngo L, Stealey M (2010) High performance computing instrumentation and research productivity in U.S. universities. J Inf Technol Impact 10:87–98

  3. Banker RD, Charnes A, Cooper WW (1984) Some models for estimating technical and scale inefficiencies in data envelopment analysis. Manag Sci 30:1078–1092

    Article  Google Scholar 

  4. Bonaccorsi A, Daraio C (2003) A robust nonparametric approach to the analysis of scientific productivity. Res Eval 12:47–69

    Article  Google Scholar 

  5. Brooks H (1986) The positive sum strategy. In: Landau R, Rosenberg N (eds) National science policy and technological innovation, vol 1. National Academy Press, Washington, DC, pp 119–165

    Google Scholar 

  6. Charnes A, Cooper WW, Rhodes E (1978) Measuring the efficiency of decision making units. Eur J Oper Res 2:429–444

    Article  Google Scholar 

  7. Daouia A, Simar L, Wilson PW (2013) Measuring firm performance using nonparametric quantile-type distances. Discussion paper, Institut de Statistique Biostatistique et Sciences Actuarielles, Université Catholique de Louvain, Louvain-la-Neuve, Belgium

  8. Deprins D, Simar L, Tulkens H (1984) Measuring labor inefficiency in post offices. In: Pestieau MMP, Tulkens H (eds) The performance of public enterprises: concepts and measurements. North-Holland, Amsterdam, pp 243–267

    Google Scholar 

  9. Dongarra JJ, Bunch J, Moler C, Stewart G (1976) LINPACK users’ guide. SIAM Publications, Philadelphia

    Google Scholar 

  10. Färe R (1988) Fundamentals of production theory. Springer-Verlag, Berlin

    Book  Google Scholar 

  11. Farrell MJ (1957) The measurement of productive efficiency. J R Stat Soc A 120:253–281

    Article  Google Scholar 

  12. Furman JL, Porter ME, Stern S (2002) The determinants of national innovative capacity. Res Policy 31: 899–933

  13. Jeong SO, Park BU, Simar L (2010) Nonparametric conditional efficiency measures: asymptotic properties. Ann Oper Res 173:105–122

    Article  Google Scholar 

  14. Kepner J (2004) High performance computing productivity model synthesis. Int J High Perform Comput Appl 18:505–516

    Article  Google Scholar 

  15. Kneip A, Park B, Simar L (1998) A note on the convergence of nonparametric DEA efficiency measures. Econom Theory 14:783–793

    Article  Google Scholar 

  16. Kneip A, Simar L, Wilson PW (2008) Asymptotics and consistent bootstraps for DEA estimators in non-parametric frontier models. Econom Theory 24:1663–1697

    Article  Google Scholar 

  17. Kneip A, Simar L, Wilson PW (2011) A computationally efficient, consistent bootstrap for inference with non-parametric DEA estimators. Comput Econ 38:483–515

    Article  Google Scholar 

  18. Kneip A, Simar L, Wilson P W (2013) Testing hypotheses in nonparametric models of production. Discussion paper #2013/48, Institut de Statistique Biostatistique et Sciences Actuarielles, Université Catholique de Louvain, Louvain-la-Neuve, Belgium

  19. Kneip A, Simar L, Wilson PW (2014) When bias kills the variance: Central limit theorems for DEA and FDH efficiency scores, Econ Theory. Forthcoming

  20. Leyden DP, Link AN (1992) Government’s role in innovation. Kluwer Academic Publishers, Boston

    Book  Google Scholar 

  21. NAS (2008) The potential impact of high-end capability computing on four illustrative fields of science and engineering, The National Academies Press, Washington. DC, National Research Council of the National Academies

  22. O’Donnell CJ, Rao DSP, Battese GE (2008) Metafrontier frameworks for the study of firm-level efficiencies and technology ratios. Empir Econ 34:231–255

    Article  Google Scholar 

  23. Ostriker JP, Holland PW, Kuh CV, Voytuk JA (2011) A data-based assessment of research-doctorate programs in the united states (2011). http://www.nap.edu/rdp/

  24. Park BU, Jeong S-O, Simar L (2010) Asymptotic distribution of conical-hull estimators of directional edges. Ann Stat 38:1320–1340

    Article  Google Scholar 

  25. Park BU, Simar L, Weiner C (2000) FDH efficiency scores from a stochastic point of view. Econ Theory 16:855–877

    Article  Google Scholar 

  26. PCAST: 2010, Report to the president and congress–designing a digital future: federally funded research and development in networking and information technology. Executive Office of the President, President’s Council of Advisors on Science and Technology

  27. PITAC: 1999, Information technology research: Investing in our future. President’s Information Technology Advisory Committee Report to the President

  28. Robinson PM (1988) Root-\(n\)-consistent semiparametric regression. Econometrica 56:931–954

    Article  Google Scholar 

  29. Shephard RW (1970) Theory of cost and production functions. Princeton University Press, Princeton

    Google Scholar 

  30. Silverman BW (1986) Density estimation for statistics and data analysis. Chapman and Hall, London

    Book  Google Scholar 

  31. Simar L, Wilson PW (1998) Sensitivity analysis of efficiency scores: how to bootstrap in nonparametric frontier models. Manag Sci 44:49–61

    Article  Google Scholar 

  32. Simar L, Wilson PW (1999a) Some problems with the Ferrier/Hirschberg bootstrap idea. J Prod Anal 11:67–80

    Article  Google Scholar 

  33. Simar L, Wilson PW (1999b) Of course we can bootstrap DEA scores! But does it mean anything? Logic trumps wishful thinking. J Prod Anal 11:93–97

    Article  Google Scholar 

  34. Simar L, Wilson PW (2000) Statistical inference in nonparametric frontier models: the state of the art. J Prod Anal 13:49–78

    Article  Google Scholar 

  35. Simar L, Wilson PW (2007) Estimation and inference in two-stage, semi-parametric models of productive efficiency. J Econom 136:31–64

    Article  Google Scholar 

  36. Simar L, Wilson PW (2008) Statistical inference in nonparametric frontier models: recent developments and perspectives, chap. 4. In: Fried H, Lovell CAK, Schmidt S (eds) The measurement of productive efficiency, 2nd edn. Oxford University Press, Oxford, pp 421–521

    Google Scholar 

  37. Simar L, Wilson PW (2011a) Inference by the \(m\) out of \(n\) bootstrap in nonparametric frontier models. J Prod Anal 36:33–53

    Article  Google Scholar 

  38. Simar L, Wilson PW (2011b) Two-stage DEA: caveat emptor. J Prod Anal 36:205–218

    Article  Google Scholar 

  39. Simar L, Wilson PW (2013) Estimation and inference in nonparametric frontier models: Recent developments and perspectives. Found Trends Econom 5:183–337

    Article  Google Scholar 

  40. Sterling T (2004) Productivity metrics and models for high performance computing. Int J High Perform Comput Appl 18:433–440

    Article  Google Scholar 

  41. Tichenor S, Reuther A (2006) Making the business case for high performance computing: a benefit-cost analysis methodology. CTWatch Quart 2:2–8

    Google Scholar 

  42. Top 500: 2013, Top 500 Supercomputer Sites. http://www.top500.org/

  43. Toutkoushian RK, Webber K (2011) Measuring the research performance of postsecondary institutions. In: Shin JC, Toutkoushian RK, Teichler U (eds) University rankings. Springer, Dordrecht, pp 123–144

    Chapter  Google Scholar 

Download references

Acknowledgments

Research support from U.S. National Science Foundation grant no. SMA-1243436 is gratefully acknowledged. Any remaining errors are solely our responsibility.

Author information

Affiliations

Authors

Corresponding author

Correspondence to Paul W. Wilson.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Apon, A.W., Ngo, L.B., Payne, M.E. et al. Assessing the effect of high performance computing capabilities on academic research output. Empir Econ 48, 283–312 (2015). https://doi.org/10.1007/s00181-014-0833-7

Download citation

Keywords

  • Efficiency
  • Frontier
  • Nonparametric
  • Inference

JEL Classification

  • C12
  • C14
  • C44
  • H52