Skip to main content

Advertisement

Log in

The impact of computer-based tutorials on high school math proficiency

  • Published:
Empirical Economics Aims and scope Submit manuscript

Abstract

The benefits of mathematical-related skills are well documented in the economics and education literature. Even in spite of such evidence, proficiency levels among US high school students remain persistently low. This is especially true for the State of Nevada. As a result, the Clark County School District (CCSD) made available to students a computer-aided math tutorial prior to taking the High School Proficiency Exam (HSPE) in mathematics. As such, we utilize a novel dataset and explore the impact of computer-aided learning on mathematics proficiency rates for 10th and 11th graders in the CCSD. Our results provide some evidence of increased proficiency in mathematics related to tutorial participation. This is especially true for minority students. However, causal claims are limited due to the inability to rule out a zero lower bound on the estimated average treatment effects.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Similar content being viewed by others

Notes

  1. Using district data for the 2009/2010 school year, O’Brien et al. (2011) find that a 49.6% math proficiency rate for 10th graders in Clark County lags behind three comparable peer districts: Broward County (73.0%), Houston ISD (68.0%), and Miami-Dade County (73.0%).

  2. See http://www.succeedinmath.org/.

  3. Unfortunately, we do not have scores for either the tutorial pre- or posttest. Note, these pre- and posttest scores from the tutorial are not the HSPE scores (our outcome variable of interest).

  4. The institutional settings of NCLB in Nevada have evolved over the years. Nevada submitted an initial plan called “All Children Can Succeed” in June 2002 (Nevada Department of Education 2002). At the onset, Nevada relied on the Terra Nova exam scores to define proficiency across various subjects, which was already in use for the statewide testing program. However, Nevada later implemented revised methods relying more on criterion-referenced exams to determine proficiency levels, hence the use of HSPE scores in mathematics (and other subjects) as a means of satisfying the upper-grade testing requirement in NCLB (Nevada Legislative Council Bureau 2005).

  5. We focus on the first attempt to keep the identification as clean as possible. In particular, we are concerned that with some students having taken the exam multiple times over a short period, outcomes can be impacted if learning has occurred with respect to material on the HSPE, the testing environment, etc. (see Matton et al. 2011). We further condition the exam scores on the first attempt not being a “retest” exam score. We chose not to include these “retest” students because it was unclear as to when they actually retested and what the environment was like in which they retested. However, we again conduct the analysis including these students, and the results are qualitatively similar.

  6. In applying the two methods, we use the Stata command -tebounds- for the nonparametric bounds approach of Kreider et al. (2012) and the Stata command -bmte- for the minimum biased estimator of Millimet and Tchernis (2013). See McCarthy et al. (2014) and McCarthy et al. (2015) for further details on the Stata commands.

  7. Like Kreider et al. (2012), we drop \(X\in \varOmega \) going forward for notational simplicity.

  8. Given the nature of the computerized tutorial, we precisely know who participated in the tutorial program since actual time spent in the program is digitally recorded. Thus, unlike Kreider et al. (2012) and Millimet and Roy (2015) who rely on self-reported participation, we do not allow for misclassification errors.

  9. See Proposition 1 in Manski and Pepper (2000).

  10. Steenbergen-Hu and Cooper (2013) review 26 different reports on intelligent tutoring systems spanning 1997 to 2010 and find that tutoring had at worse a nonnegative impact on achievement outcomes.

  11. As noted in McCarthy et al. (2015), combining the MTR assumption with the MTS–MIV assumptions is slightly more complicated since \(P[{\textit{MP}}(j)=1 ] ,\) \(j=0,1\) are bounded separately within each of the k cells. Since the MTR assumption requires that tutorial participation have no negative impact on the probability of being proficient in math, then it must be the case that \(B_{k}^{L}\) of \(P[{\textit{MP}}(1)=1] \) must be strictly less than \(B_{k}^{U}\) of \(P[{\textit{MP}}(0)=1] \) \(\forall \) \(k=1,\ldots ,K.\)

  12. Likelihood ratio tests for heteroskedasticity were conducted, and results are available upon request.

  13. All Imbens and Manski (2004) confidence intervals reported are at the 95% level.

  14. The covariates utilized in each model are indicators for minority status, gender, English proficient, refugee status, gifted and talented, and for being in the tenth grade. Note, the exogenous selection results presented in the previous section correspond to OLS when regressing math proficiency on only a treatment indicator.

  15. The cost of $5.00 per license comes from conversations with individuals at CCSD who recall the program during the 2006/2007 school year. The numbers provided to us were in the range of $4.00 to $5.00 per license. We opt to use the more aggressive $5.00 per license to construct an upper bound on the cost per additional proficient student.

References

  • Altonji JG, Elder TE, Taber CR (2005) Selection on observed and unobserved variables: assessing the effectiveness of catholic schools. J Polit Econ 113:151–184

    Article  Google Scholar 

  • Annie E. Casey Foundation (2016) KIDS COUNT data book, 2016: state trends in child well-being. The Annie E. Casey Foundation, Baltimore

  • Barrow L, Markman L, Rouse CE (2009) Technology’s edge: the educational benefits of computer-aided instruction. Am Econ J Econ Policy 1:52–74

    Article  Google Scholar 

  • Beal CR, Walles R, Arroyo I, Woolf BP (2007) On-line tutoring for math achievement testing: a controlled evaluation. J Interact Online Learn 6:43–55

    Google Scholar 

  • Campuzano L, Dynarski M, Agodini R, Rall K (2009) Effectiveness of reading and mathematics software products: findings from two student cohorts (NCEE 2009-4042). Washington, DC: National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education

  • Carnevale AJ, Rose SJ, Cheah B (2011) The college payoff: education, occupations, lifetime earnings. Technical report, Georgetown University Center on Education and the Workforce

  • Cheung ACK, Slavin RE (2013) The effectiveness of educational technology applications for enhancing mathematics achievement in K-12 classrooms: a meta-analysis. Educ Res Rev 9:88–113

    Article  Google Scholar 

  • Claessens A, Engel M (2013) How important is where you start? early mathematics knowledge and later school success. Teach Coll Rec 115:1–29

    Google Scholar 

  • Clements DH, Sarama J (2011) Early childhood mathematics intervention. Science 333:968–970

    Article  Google Scholar 

  • Duncan GJ, Magnuson K (2013) Investing in preschool programs. J Econ Perspect 27:109–132

    Article  Google Scholar 

  • Fryer RG, Levitt SD (2004) Understanding the Black–White test score gap in the first two years of school. Rev Econ Stat 86(2):447–464

    Article  Google Scholar 

  • Fryer RG, Levitt SD (2006) The Black–White test score gap through third grade. Am Law Econ Rev 8(2):249–281

    Article  Google Scholar 

  • Fryer RG, Levitt SD (2010) An empirical analysis of the gender gap in mathematics. Am Econ J Appl Econ 2(2):210–240

    Article  Google Scholar 

  • Grogger J (1996) Does school quality explain the recent Black/White wage trend? J Labor Econ 14:231–253

    Article  Google Scholar 

  • Gurun A, Millimet DL (2008) Does private tutoring payoff? IZA working paper no. 3637

  • Heckman JJ (2006) Skill formation and the economics of investing in disadvantaged children. Science 312:1900–1902

    Article  Google Scholar 

  • Hedges LV, Nowell A (1995) Sex differences in mental test scores, variability, and numbers of high-scoring individuals. Science 269:41–45

    Article  Google Scholar 

  • Hemphill FC, Vanneman A (2011) Achievement gaps: how Hispanic and White students in public schools perform in mathematics and reading on the national assessment of educational progress (NCEE 2011-459). National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education, Washington, DC

  • Hirano K, Imbens G (2001) Estimation of causal effects using propensity score weighting: an application to data on right heart catheterization. Health Serv Outcomes Res Methodol 2:259–278

  • Imbens G, Manski C (2004) Confidence intervals for partially identified parameters. Econometrica 72:1845–1857

    Article  Google Scholar 

  • Kena G, Musu-Gillette L, Robinson J, Wang X, Rathbun A, Zhang J, Wilkinson-Flicker S, Barmer A, Dunlop Velez E (2015) The condition of education 2015. U.S. Department of Education Report (NCES 2015-144), National Center for Education Statistics, Washington, DC

  • Kling JR, Liebman JB, Katz LF (2007) Experimental analysis of neighborhood effects. Econometrica 75:83–119

    Article  Google Scholar 

  • Kreider B, Pepper JV, Gundersen C, Jolliffe D (2012) Identifying the effects of SNAP (food stamps) on child health outcomes when participation is endogenous and misreported. J Am Stat Assoc 107:958–975

    Article  Google Scholar 

  • Kulik JA, Fletcher JD (2016) Effectiveness of intelligent tutoring systems: a meta-analytic review. Rev Educ Res 86:42–78

    Article  Google Scholar 

  • Lai F, Luo R, Zhang S, Huang X, Rozelle S (2015) Does computer-assisted learning improve learning outcomes? evidence from a randomized experiment in migrant schools in Beijing. Econ Educ Rev 47:34–48

    Article  Google Scholar 

  • Leahey E, Guo G (2001) Gender differences in mathematical trajectories. Soc Forces 80(2):713–732

    Article  Google Scholar 

  • Lee LF (1984) Tests for the bivariate normal distribution in econometric models with selectivity. Econometrica 52:843–863

    Article  Google Scholar 

  • Manski C (1995) Identification problems in the social sciences. Harvard University Press, Cambridge

    Google Scholar 

  • Manski C (1997) Monotone treatment response. Econometrica 65:1311–1334

    Article  Google Scholar 

  • Manski C, Pepper J (2000) Monotone instrumental variables: with an application to the returns to schooling. Econometrica 68:997–1010

    Article  Google Scholar 

  • Matton N, Vautier S, Raufaste E (2011) Test-specificity of the advantage of retaking cognitive ability tests. Int J Sel Assess 19:11–17

    Article  Google Scholar 

  • McCarthy I, Millimet DL, Tchernis R (2014) The bmte command: methods for the estimation of treatment effects when exclusion restrictions are unavailable. Stata J 14:670–683

    Google Scholar 

  • McCarthy I, Millimet DL, Roy M (2015) Bounding treatment effects: stata command for the partial identification of the average treatment effect with endogenous and misreported treatment assignment. Stata J 15:411–436

    Google Scholar 

  • Millimet DL, Roy M (2015) Partial identification of the long-run causal effect of food security on child health. Empir Econ 48:83–141

    Article  Google Scholar 

  • Millimet DL, Tchernis R (2013) Estimation of treatment effects without an exclusion restriction: with an application to the analysis of the school breakfast program. J Appl Econ 28:982–1017

    Google Scholar 

  • Murnane RJ, Willett JB, Levy F (1995) The growing importance of cognitive skills in wage determination. Rev Econ Stat 77:251–266

    Article  Google Scholar 

  • Nevada Department of Education (2002) The Nevada plan: ‘all children can succeed’: consolidated plan for the implementation of the no child left behind act. State of Nevada Department of Education, Carson City

  • Nevada Legislative Counsel Bureau (2005) Report to the 73rd Session of the Nevada Legislature by the Legislative Committee on Education. Legislative Counsel Bulletin No. 05-15. Legislative Counsel Bureau, Carson City

  • O’Brien B, Brown S, Eno J, Pontoni K (2011a) Educational and operational efficiency study: analysis of student performance for the Clark County school district. Technical report, American Institute for Research

  • O’Brien B, Tucker N, Zuber T (2011b) Educational and operational efficiency study: comparative analysis of academic performance for the Clark County school district. Technical report, American Institute for Research

  • OECD (2014) PISA 2012 results in focus: what 15-year-olds know and what they can do with what they know. OECD Publishing, Paris

    Google Scholar 

  • Pepper J (2000) The intergenerational transmission of welfare receipt: a nonparametric bounds analysis. Rev Econ Stat 82:472–488

    Article  Google Scholar 

  • Sohn K (2012) A new insight into the gender gap in math. Bull Econ Res 64(1):135–155

    Article  Google Scholar 

  • Steenbergen-Hu S, Cooper H (2013) A meta-analysis of the effectiveness of intelligent tutoring systems on K-12 students’ mathematical learning. J Educ Psychol 105:970–987

    Article  Google Scholar 

  • Todd PE, Wolpin KI (2003) On the specification and estimation of the production function for cognitive achievement. Econ J 113:3–33

    Article  Google Scholar 

  • Todd PE, Wolpin KI (2007) The production of cognitive achievement in children: home, school, and racial test score gaps. J Hum Cap 1:91–136

    Article  Google Scholar 

Download references

Acknowledgements

The authors have benefited from helpful suggestions from the co-editors, Wenhua Di and Daniel Millimet, and two anonymous referees, as well as participants at the Intent versus Impact: Evaluating Individual- and Community-Based Programs conference hosted by the Federal Reserve Bank of Dallas.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Ian K. McDonough.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

McDonough, I.K., Tra, C.I. The impact of computer-based tutorials on high school math proficiency. Empir Econ 52, 1041–1063 (2017). https://doi.org/10.1007/s00181-016-1189-y

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00181-016-1189-y

Keywords

JEL Classification

Navigation