Skip to main content

Advertisement

SpringerLink
Log in

Untangling complex effects of disabilities and accommodations within a multilevel IRT framework

  • Published:
Quality & Quantity Aims and scope Submit manuscript

Abstract

A small number of previous studies on setting accommodation suggest that this accommodation is beneficial for students with special needs; a majority of the research has been survey type studies concentrating on participants’ subjective judgements. To test previous assumptions that the use of this accommodation is valid, and disentangle the complex person- and item-level effects of the data, this study extended the conventional multilevel item response theory model by explicitly modeling multiple student-level predictors and the interactions between the predictors to examine the differential difficulties of math and reading items for students participating in provincial assessments, specifically in relation to whether they received setting-only accommodation and had a learning disability. Both group differences in difficulty across all items and for individual items [differential item functioning (DIF)] were examined. The results show that math and reading items were relatively more difficult for accommodated than non-accommodated students in learning disabilities and non-disabled groups; several items were found to exhibit DIF. These empirical findings demonstrate that there is good potential to apply the methodology to structured data at and beyond accommodation studies.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

References

  • American Educational Research Association, American Psychological Association, & National Council on Measurement in Education: Standards for Educational and Psychological Testing. Author, Washington, DC (2014)

  • American Educational Research Association, American Psychological Association, & National Council on Measurement in Education: Standards for Educational and Psychological Testing. Author, Washington, DC (1999)

  • Beretvas, S.N., Cawthon, S.W., Lockhart, L.L., Kaye, A.D.: Assessing impact, DIF, and DFF in accommodated item scores: a comparison of multilevel measurement model parameterizations. Educ. Psychol. Measur. 72(5), 754–773 (2012)

    Article  Google Scholar 

  • Beretvas, S.N., Kamata, A.: Introduction to the special section on the multilevel measurement model for dichotomous items. In: Smith, E., Smith, R. (eds.) Rasch Measurement: Advanced and Specialized Applications, pp. 291–302. JAM Press, Maple Grove, MN (2007)

    Google Scholar 

  • Beretvas, S.N., Walker, C.M.: Distinguishing differential testlet functioning from differential bundle functioning using the multilevel measurement model. Educ. Psychol. Measur. 7, 200–223 (2012)

    Article  Google Scholar 

  • Bielinski, J., Sheinker, A., Ysseldyke, J: Varied opinions on how to report accommodated test scores: Findings based on CTB/McGraw-Hill’s framework for classifying accommodations (NCEO Synthesis Rep. 49). University of Minnesota, National Center on Educational Outcomes, Minneapolis (2003)

  • Bolt, S.E., Thurlow, M.L.: Item-level effects of the read-aloud accommodation for students with reading disabilities. Assess. Eff. Interv. 33, 15–28 (2007)

    Article  Google Scholar 

  • Bolt, S.E., Ysseldyke, J.E.: Comparing DIF across math and Reading/Language arts tests for students receiving a read-aloud accommodation. Appl. Measur. Educ. 19, 329–355 (2006)

    Article  Google Scholar 

  • Bouck, E.C.: Calculating the value of graphing calculators for seventh-grade students with and without disabilities: a pilot study. Remedial Spec. Educ. 30, 207–215 (2009)

    Article  Google Scholar 

  • Brown, W: Virginia Teachers’ Perceptions and Knowledge of Test Accommodations for Students with Disabilities. Ph.D. dissertation, The College of William and Mary, Virginia (2007)

  • Byrd, T: East Tennessee State University Faculty Attitudes and Student Perceptions in Providing Accommodations to Students with Disabilities. Ed.D. dissertation, East Tennessee State University, Tennessee (2010)

  • Calhoon, M.B., Fuchs, L.S., Hamlett, C.L.: Effects of computer-based test accommodations on mathematics performance assessments for secondary students with learning disabilities. Learn. Disabil. Q. 23, 271–282 (2000)

    Article  Google Scholar 

  • Camilli, G., Shepard, L.A.: Methods for Identifying Biased Test Items. Sage Publications, Thousand Oaks, CA (1994)

    Google Scholar 

  • Cawthon, S.W.: Science and evidence of success: two emerging issues in assessment accommodations for students who are deaf or hard of hearing. J. Deaf Stud. Deaf Educ. 15, 185–203 (2010)

    Article  Google Scholar 

  • Cawthon, S.W., Kaye, A.D., Lockhart, L.L., Beretvas, S.N.: Effects of linguistic complexity and accommodations on estimates of ability for students with learning disabilities. J. Sch. Psychol. 50(3), 293–316 (2012)

    Article  Google Scholar 

  • Cheong, Y.F.: Analysis of school context effects on differential item functioning using hierarchical generalized linear models. Int. J. Test. 6, 57–79 (2006)

    Article  Google Scholar 

  • Cheong, Y.F., Kamata, A.: Centering, scale indeterminacy, and differential item functioning detection in hierarchical generalized linear and generalized linear mixed models. Appl. Measur. Educ. 26(4), 233–252 (2013)

    Article  Google Scholar 

  • Cheong, Y.F., Raudenbush, S.W.: Measurement and structural models for children’s problem behaviors. Psychol. Methods 5, 477–495 (2000)

    Article  Google Scholar 

  • Cho, H., Lee, J., Kingston, N.: Examining the effectiveness of test accommodation using DIF and a mixture IRT model. Appl. Measur. Educ. 25, 281–304 (2012)

    Article  Google Scholar 

  • Christensen, L.L., Braam, M., Scullin, S., Thurlow, M.L.: 2009 State Policies on Assessment Participation and Accommodations for Students with Disabilities (Synthesis Report 83). University of Minnesota, National Center on Educational Outcomes, Minneapolis, MN (2011)

    Google Scholar 

  • Christensen, L.L., Lazarus, S.S., Crone, M., Thurlow, M.L.: 2007 State Policies on Assessment Participation and Accommodations for Students with Disabilities (Synthesis Report 69). University of Minnesota, National Center on Educational Outcomes, Minneapolis, MN (2008)

    Google Scholar 

  • Cohen, A.S., Gregg, N., Deng, M.: The role of extended time and item content on a high-stakes mathematics test. Learn. Disabil. Res. Pract. 20, 225–233 (2005)

    Article  Google Scholar 

  • Cook, L., Eignor, D., Sawaki, Y., Steinberg, J., Cline, F.: Using factor analysis to investigate accommodations used by students with disabilities on an English-language arts assessment. Appl. Meas. Educ. Spec. Issue Test. Stud. Disabil. 23, 187–208 (2010)

    Article  Google Scholar 

  • DeMars, C.: Item Response Theory. Oxford University Press, New York, NY (2010)

    Book  Google Scholar 

  • Dorans, N.J. Holland, P.W: DIF Detection and Description: Mantel-Haenszel and Standardization. Paper presented at the Educational Testing Services/AFHRL conference. Princeton, NJ (1992)

  • Education Quality and Accountability Office (EQAO): Guide for Administering the Grade 3 and Grade 6 Assessments of Reading, Writing, and Mathematics (Spring 2006). Education Quality and Accountability Office, Ontario (2006)

    Google Scholar 

  • Education Quality and Accountability Office (EQAO): Curriculum connections in language: Reading and writing. Retrieve from http://www.eqao.com/ (2011)

  • Elliott, S.N., Marquart, A.M.: Extended time as a testing accommodation: its effects and perceived consequences. Except. Child. 70, 349–367 (2004)

    Article  Google Scholar 

  • Fletcher, J., Francis, D.J., Boudousquie, A., Copeland, K., Young, V., Kalinowski, S., et al.: Effects of accommodations on high-stakes testing for students with reading disabilities. Counc. Except. Child. 72, 136–150 (2006)

    Article  Google Scholar 

  • Finch, W.H., Finch, M.E.H.: Investigation of specific learning disability and testing accommodations based differential item functioning using a multilevel multidimensional mixture item response theory model. Educ. Psychol. Measur. 73, 973–993 (2013)

    Article  Google Scholar 

  • Fuchs, L.S., Fuchs, D.: Fair and unfair testing accommodations. Sch. Adm. 56(10), 24 (1999)

    Google Scholar 

  • Fuchs, L.S., Fuchs, D., Capizzi, A.M.: Identifying appropriate test accommodations for students with learning disabilities. Focus Except. Child. 37(6), 1–8 (2005)

    Google Scholar 

  • George-Ezzelle, C., Skaggs, G: Examining the Validity of GED Tests Scores with Scheduling and Setting Accommodations. (GED Testing Service Research Studies 2004-1). GED Testing Service, Washington, DC (2004)

  • Gregg, N.: Increasing access to learning for the adult basic education learner with learning disabilities: evidence-based accommodation research. J. Learn. Disabil. 45(1), 47–63 (2012)

    Article  Google Scholar 

  • Haladyna, T.M: Developing and Validating Multiple-Choice Test Items. Hillsdale: Lawrence Erlbaum Associates, New Jersey (1994)

  • Hambleton, R.K., Rogers, H.J.: Detecting potentially biased test items: comparison of IRT area and Mantel-Haenszel methods. Appl. Measur. Educ. 2, 313–334 (1989)

    Article  Google Scholar 

  • Hambleton, R.K., Swaminathan, H., Rogers, H.J.: Fundamentals of Item Response Theory. Sage Publications, Newbury Park, CA (1991)

    Google Scholar 

  • Helwig, R., Tindal, G.: An experimental analysis of accommodation decisions on large-scale mathematics tests. Except. Child. 69, 211–225 (2003)

    Article  Google Scholar 

  • Holland, P.W., Thayer, D.T: An alternate definition of the ETS delta scale of item difficulty (ETS Program Statistics Research Technical Report No. 85–64). ETS, Princeton, NJ (1985)

  • Johnson, E., Monroe, B.: Simplified language as an accommodation on math tests. Assess. Eff. Interv. 29, 35–45 (2004)

    Article  Google Scholar 

  • Kamata, A: Some generalizations of the Rasch model: An application of the hierarchical generalized linear model. ProQuest Information & Learning). Diss. Abstr. Int. Sect. A Hum. Soc. Sci. 60(3-A), 0715–0715 (1999)

  • Kamata, A.: Item analysis by the hierarchical generalized linear model. J. Educ. Meas. 38, 79–93 (2001)

    Article  Google Scholar 

  • Laitusis, C.C.: Examining the impact of audio presentation on tests of reading comprehension. Appl. Measur. Educ. 23, 153–167 (2010)

    Article  Google Scholar 

  • Lazarus, S.S., Thurlow, M.L., Lail, K.E., Christensen, L.: A longitudinal analysis of state accommodations policies: twelve years of change, 1993–2005. J. Spec. Educ. 43(2), 67–80 (2009)

    Article  Google Scholar 

  • Lin, P.Y., Lin, Y.C.: Examining student factors in sources of setting accommodation DIF. Educ. Psychol. Measur. 74(5), 759–794 (2014)

    Article  Google Scholar 

  • Linacre, J.M., Wright, B.D.: Mantel-Haenszel DIF and PROX are equivalent! Rasch Meas. Trans. 3, 52–53 (1989)

    Google Scholar 

  • Lindstrom, J.H., Gregg, N.: The role of extended time on the SAT® for students with learning disabilities and/or attention-deficit/hyperactivity disorder. Learn. Disabil. Res. Pract. 22, 85–95 (2007)

    Article  Google Scholar 

  • McKevitt, B.C., Elliott, S.N.: Effects of perceived consequences of using read-aloud and teacher-recommended testing accommodations on a reading achievement test. Sch. Psychol. Rev. 32(4), 583–600 (2003)

    Google Scholar 

  • Middleton, K., Laitusis, C.C: Examining test items for differential distractor functioning among students with learning disabilities. Research Report, 07-43. Educational Testing Service, Princeton, NJ (2007)

  • Mislevy, R.J.: Foundations of a new test theory. In: Frederiksen, N., Mislevy, R.J., Bejar, I.I. (eds.) Test Theory for a New Generation of Tests, pp. 19–39. Lawrence Erlbaum Associates, Hillsdale, NJ (1993)

    Google Scholar 

  • Ontario Ministry of Education: Special Education: A Guide for Educators. Retrieved from http://www.edu.gov.on.ca/ (2001)

  • Ontario Ministry of Education: The Ontario Curriculum, Grades 1–8: Language. Retrieved from http://www.edu.gov.on.ca/ (2005)

  • Ontario Ministry of Education: The Ontario Curriculum, Grades 1–8: Mathematics. Retrieved from http://www.edu.gov.on.ca/ (2006)

  • Ontario Ministry of Education: Policy/Program Memorandum no. 127. Retrieved from http://www.ontla.on.ca/ (2009)

  • Pastor, D.A.: The use of multilevel item response theory modeling in applied research: an illustration. Appl. Measur. Educ. 16(3), 223–243 (2003)

    Article  Google Scholar 

  • Phillips, S.E.: High-stakes testing accommodations: validity versus disabled rights. Appl. Measur. Educ. 7, 93–120 (1994)

    Article  Google Scholar 

  • Pitoniak, M.J., Royer, J.M.: Testing accommodations for examinees with disabilities: a review of psychometric, legal, and social policy issues. Rev. Educ. Res. 71, 53–104 (2001)

    Article  Google Scholar 

  • Raju, N.: The Area Between Two Item Characteristic Curves. Springer, New York (1988)

    Google Scholar 

  • Raju, N.: Determining the significance of estimated signed and unsigned areas between two item response functions. Appl. Psychol. Meas. 14, 197–207 (1990)

    Article  Google Scholar 

  • Raudenbush, S.W., Bryk, A.S.: Hierarchical Linear Models: Applications and Data Analysis Methods, 2nd edn. Sage, Newbury Park, CA (2002)

    Google Scholar 

  • Rogers, H.J., Swaminathan, H.: A comparison of logistic regression and Mantel–Haenszel procedures for detecting differential item functioning. Appl. Psychol. Meas. 17, 105–116 (1993)

    Article  Google Scholar 

  • SAS: SAS 9.2 User’s Guide, 2nd edm. SAS Institute, Cary, North Carolina (2010)

  • Siegel, L.: Issues in the definition and diagnosis of learning disabilities. J. Learn. Disabil. 32, 304–319 (1999)

    Article  Google Scholar 

  • Sireci, S., Li, S., Scarpati, S: The effects of test accommodation on test performance: A review of the literature (Center for Educational Assessment Research Report no. 485). University of Massachusetts, School of Education, Amherst (2003)

  • Sireci, S.G., Scarpati, S.E., Li, S.: Test accommodations for students with disabilities: an analysis of the interaction hypothesis. Rev. Educ. Res. 75, 457–490 (2005)

    Article  Google Scholar 

  • Stoneman, Z., Gibson, S.: Situational influences on assessment performance. Except. Child. 46, 166–169 (1978)

    Google Scholar 

  • Thurlow, M.L., Lazarus, S.S., Thompson, S.J., Morse, A.B.: State policies on assessment participation and accommodations for students with disabilities. J. Spec. Educ. 38, 232–240 (2005)

    Article  Google Scholar 

  • Thurlow, M., Rogers, C., Christensen, L.: Science Assessments for Students with Disabilities in School Year 2006–2007: What we Know About Participation, Performance, and Accommodations (Synthesis Report 77). University of Minnesota, National Center on Educational Outcomes, Minneapolis, MN (2010)

    Google Scholar 

  • Trammell, J.K.: The impact of academic accommodations on final grades in a postsecondary setting. J. Coll. Read. Learn. 34(1), 76–90 (2003)

    Article  Google Scholar 

  • Vaughn, S., Linan-Thompson, S., Hickman, P.: Response to instruction as a means of identifying students with reading/learning disabilities. Except. Child. 69, 391 (2003)

    Google Scholar 

  • Vellutino, F., Scanlon, D., & Lyon, G. R.: Differentiating between difficult-to-remediate and readily remediated poor readers. J. Learn. Disabil. 33, 223–238 (2000)

    Article  Google Scholar 

  • Williams, N.J., Beretvas, S.N.: DIF identification using HGLM for polytomous items. Appl. Psychol. Meas. 30(1), 22–42 (2006)

    Article  Google Scholar 

  • Wilson, M.: Constructing Measures: An Item Response Modeling Approach. Erlbaum, Mahwah, NJ (2004)

    Google Scholar 

  • Wyse, A.E., Mapuranga, R.: Differential item functioning analysis using Rasch item information functions. Int. J. Test. 9(4), 333 (2009)

    Article  Google Scholar 

  • Zuriff, G.E.: Extra examination time for students with learning disabilities: an examination of the maximum potential thesis. Appl. Measur. Educ. 13, 99–117 (2000)

    Article  Google Scholar 

  • Zwick, R., Thayer, D.T., Lewis, C.: An empirical Bayes approach to Mantel-Haenszel DIF analysis. J. Educ. Meas. 36, 1–28 (1999)

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. University of Saskatchewan, Saskatoon, SK, Canada

    Pei-Ying Lin

  2. University of Toronto, Toronto, ON, Canada

    Ruth A. Childs

  3. University of Texas - Rio Grande Valley, Edinburg, TX, USA

    Yu-Cheng Lin

Authors
  1. Pei-Ying Lin
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ruth A. Childs
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yu-Cheng Lin
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Pei-Ying Lin.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Lin, PY., Childs, R.A. & Lin, YC. Untangling complex effects of disabilities and accommodations within a multilevel IRT framework. Qual Quant 50, 2767–2788 (2016). https://doi.org/10.1007/s11135-015-0288-8

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11135-015-0288-8

Keywords

Search

Navigation