Skip to main content
SpringerLink
Log in

Research Design Considerations in Human Science Research: Reconciling Conceptions of Science, Theories of Measurement and Research Methods

  • Conference paper
  • First Online:
Pacific Rim Objective Measurement Symposium (PROMS) 2016 Conference Proceedings

  • 234 Accesses

Abstract

How might psychometrics go about improving the meaningfulness and productivity of its routinely employed procedures? A long history of critical and educational efforts has not stemmed widespread misconceptions and misuses of methods and models. A framework for contextualizing the respective principles and procedures of different measurement theories sets the stage for finding an alternative path toward general improvements in psychometric practice. Positivist, anti-positivist, and post-positivist philosophies of science inform paradigmatically distinct measurement principles and procedures. Connecting measurement and the assumptions of these paradigms enables a mapping of measurement activities within the separate philosophical approaches, grounding research design. The philosophical distinctions provide, then, an analytic tool for comparing and contrasting measurement theories. Some aspects of positivism and anti-positivism incompatible with historical and contemporary measurement theory suggest that an amodern, post-positivist approach to measurement offers untried potentials for new and creative research approaches.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 129.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 169.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 169.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  • Akkerman, S., van den Bossche, P., Admiraal, W., Gijselaers, W., Segers, M., Simons, R.-J., et al. (2007). Reconsidering group cognition: From conceptual confusion to a boundary area between cognitive and socio-cultural perspectives? Educational Research Review, 2, 39–63.

    Article  Google Scholar 

  • Barbanti, C. (2016). School as a myriad of sociomaterial assemblages: Renewed aims, processes and enactments in educational research through actor-network theory. Scuola Democratica, 7(1), 183–198.

    Google Scholar 

  • Bateson, G. (1972). Steps to an ecology of mind: Collected essays in anthropology, psychiatry, evolution, and epistemology. Chicago: University of Chicago Press.

    Google Scholar 

  • Berkson, J. (1938). Some difficulties of interpretation encountered in the application of the chi-square test. American Statistical Association Journal, 33(201–204), 526–536.

    Article  Google Scholar 

  • Beretvas, N., & Kamata, A. (Eds.). (2007). Part II. Multi-level measurement Rasch models. In E. V. Smith, Jr. & R. M. Smith (Eds.), Rasch measurement: Advanced and specialized applications (pp. 291–470). Maple Grove, MN: JAM Press.

    Google Scholar 

  • Birnbaum, A. (1968). Some latent trait models and their use in inferring an examinee’s ability, Part 5. In F. M. Lloyd & M. R. Novick (Eds.), Statistical theories of mental test scores (pp. 395–479). Reading, MA: Addison-Wesley.

    Google Scholar 

  • Brennan, R. L. (1997). A perspective on the history of generalizability theory. Educational Measurement: Issues and Practice, 16(4), 14–20.

    Article  Google Scholar 

  • Bryk, A. S., & Raudenbush, S. W. (1992). Hierarchical linear models: Applications and data analysis methods. Newbury Park, CA: Sage.

    Google Scholar 

  • Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). New York: Academic Press.

    Google Scholar 

  • Cronbach, L. J. (1951). Coefficient alpha and the internal structure of tests. Psychometrika, 16, 297–334.

    Article  Google Scholar 

  • Cronbach, L. J., Rajaratnam, N., & Gleser, G. C. (1963). Theory of generalizability: A liberalization of reliability theory. British Journal of Statistical Psychology, 16, 137–163.

    Article  Google Scholar 

  • Davis, J. (2009). Complementary research methods in humanistic and transpersonal psychology: A case for methodological pluralism. The Humanistic Psychologist, 37(1), 4–23.

    Article  Google Scholar 

  • DeBoeck, P., & Wilson, M. (2014). Multidimensional explanatory item response models. In S. P. Reise & D. Revicki (Eds.), Handbook of item response theory modeling: Applications to typical performance assessment (pp. 252–271). New York: Routledge.

    Google Scholar 

  • Decuypere, M., & Simons, M. (2016). Relational thinking in education: topology, sociomaterial studies, and figures. Pedagogy, Culture & Society, 24(3), 371–386.

    Article  Google Scholar 

  • Derrida, J. (2003). Interview on writing. In G. A. Olson & L. Worsham (Eds.), Critical intellectuals on writing (pp. 61–69). Albany, NY: State University of New York Press.

    Google Scholar 

  • Dewey, J. (1944/1916). Democracy and education: An introduction to the philosophy of education. New York: The Free Press.

    Google Scholar 

  • Dewey, J. (2012). Unmodern philosophy and modern philosophy. In P. Deen, (Ed.) Carbondale, IL: Southern Illinois University Press.

    Google Scholar 

  • Duncan, O. D. (1992). What if? Contemporary Sociology, 21(5), 667–668.

    Article  Google Scholar 

  • Duncan, O. D., & Stenbeck, M. (1988). Panels and cohorts: Design and model in the study of voting turnout. In C. C. Clogg (Ed.), Sociological methodology 1988 (pp. 1–35). Washington, DC: American Sociological Association.

    Google Scholar 

  • Engelhard, G., Jr. (1991). Research traditions and history of measurement. Rasch Measurement Transactions, 4(4), 126.

    Google Scholar 

  • Engelhard, G., Jr. (2013). Invariant measurement: Using Rasch models in the social, behavioral, and health sciences. New York: Routledge Academic.

    Google Scholar 

  • Falk, R. (1986). Misconceptions of statistical significance. Journal of Structural Learning, 9, 83–96.

    Google Scholar 

  • Fenwick, T., & Edwards, R. (2010). Actor-network theory in education. New York: Routledge.

    Google Scholar 

  • Fenwick, T., & Edwards, R. (2013). Performative ontologies: Sociomaterial approaches to researching adult education and lifelong learning. European Journal for Research on the Education and Learning of Adults, 4(1), 49–63.

    Article  Google Scholar 

  • Fisher, W. P., Jr. (1997). What scale-free measurement means to health outcomes research. Physical Medicine & Rehabilitation State of the Art Reviews, 11(2), 357–373.

    Google Scholar 

  • Fisher, W. P., Jr. (1999). Foundations for health status metrology: The stability of MOS SF-36 PF-10 calibrations across samples. Journal of the Louisiana State Medical Society, 151(11), 566–578.

    Google Scholar 

  • Fisher, W. P., Jr. (2000). Objectivity in psychosocial measurement: What, why, how. Journal of Outcome Measurement, 4(2), 527–563.

    Google Scholar 

  • Fisher, W. P., Jr. (2003). Mathematics, measurement, metaphor, metaphysics: Parts I & II. Theory & Psychology, 13(6), 753–828.

    Article  Google Scholar 

  • Fisher, W. P., Jr. (2004). Meaning and method in the social sciences. Human Studies: A Journal for Philosophy and the Social Sciences, 27(4), 429–454.

    Article  Google Scholar 

  • Fisher, W. P., Jr., & Cavanagh, R. (2016). Measurement as a medium for communication and social action, I & II. In Q. Zhang & H. H. Yang (Eds.), Pacific Rim Objective Measurement Symposium (PROMS) 2015 Conference Proceedings (pp. 153–182). Berlin: Springer.

    Google Scholar 

  • Fisher, W. P., Jr., Oon, E. P. T., & Benson, S. (2017). Applying Design Thinking to systemic problems in educational assessment information management. Journal of Physics Conference Series (in review).

    Google Scholar 

  • Fisher, W. P., Jr., & Stenner, A. J. (2015). The role of metrology in mobilizing and mediating the language and culture of scientific facts. Journal of Physics Conference Series, 588(012043).

    Google Scholar 

  • Fisher, W. P., Jr., & Stenner, A. J. (2016). Theory-based metrological traceability in education: A reading measurement network. Measurement, 92, 489–496.

    Article  Google Scholar 

  • Fisher, W. P., Jr., & Stenner, A. J. (2017). Ecologizing vs modernizing in measurement and metrology. Journal of Physics Conference Series (in review).

    Google Scholar 

  • Fisher, W. P., Jr., & Wilson, M. (2015). Building a productive trading zone in educational assessment research and practice. Pensamiento Educativo: Revista de Investigacion Educacional Latinoamericana, 52(2), 55–78.

    Google Scholar 

  • Fox, S. (2005). An actor-network critique of community in higher education: Implications for networked learning. Studies in Higher Education, 30(1), 95–110.

    Article  Google Scholar 

  • Frodeman, R. (2014). Hermeneutics in the field: The philosophy of geology. In B. Babich & D. Ginev (Eds.), The multidimensionality of hermeneutic phenomenology (pp. 69–80). Heidelberg: Springer International Publishing.

    Chapter  Google Scholar 

  • Gadamer, H.-G. (1989). Truth and method (J. Weinsheimer & D. G. Marshall, Trans.) (Rev. ed.). New York: Crossroad.

    Google Scholar 

  • Gadamer, H.-G. (1991). Gadamer on Gadamer. In H. J. Silverman (Ed.), Continental philosophy (Vol. IV, pp. 13–19)., Gadamer and hermeneutics New York: Routledge.

    Google Scholar 

  • Galison, P., & Stump, D. J. (1996). The disunity of science: Boundaries, contexts, and power. Palo Alto, CA: Stanford University Press.

    Google Scholar 

  • Galison, P. (1997). Image and logic: A material culture of microphysics. Chicago: University of Chicago Press.

    Google Scholar 

  • Gorin, J. S., & Mislevy, R. J. (2013). Inherent measurement challenges in the next generation science standards for both formative and summative assessment (K-12 Center at Educational Testing Service Invitational Research Symposium on Science Assessment). Princeton, NJ: ETS.

    Google Scholar 

  • Green, S. B., Lissitz, R. W., & Mulaik, S. A. (1977). Limitations of coefficient alpha as an index of test unidimensionality. Educational and Psychological Measurement, 37(4), 827–833.

    Article  Google Scholar 

  • Guttman, l. (1944). A basis for scaling qualitative data. American Sociological Review, 9(2), 139–150.

    Article  Google Scholar 

  • Guttman, L. (1950). The basis for scalogram analysis. In S. A. Stoufer, L. Guttman, E. A. Suchman, P. L. Lazarsfeld, S. A. Star, & J. A. Clasuen (Eds.), Measurement and prediction (Vol. IV, pp. 60–90). Princeton, NJ: Princeton University Press.

    Google Scholar 

  • Hambleton, R. K., Swaminathan, H., & Rogers, L. (1991). Fundamentals of item response theory. Newbury Park, CA: Sage.

    Google Scholar 

  • Heelan, P. A. (1982). Hermeneutical realism and scientific observation. In PSA: Proceedings of the Biennial Meeting of the Philosophy of Science Association (Vol. 1, pp. 77–87).

    Google Scholar 

  • Heidegger, M. (1977). The question concerning technology and other essays. New York: Harper & Row.

    Google Scholar 

  • Heidegger, M. (1991). The principle of reason (R. Lilly, Trans.). Bloomington, IN: Indiana University Press (Original work published 1957).

    Google Scholar 

  • Howe, K. R. (2009). Straw makeovers, dogmatic holisim, and interesting conversation. Educational Rsearcher, 38(6), 463–466.

    Article  Google Scholar 

  • Johnson, R. B., & Onwuegbuzie, A. J. (2004). Mixed methods research: A research paradigm whose time has come. Educational Researcher, 33(7), 14–26.

    Article  Google Scholar 

  • Joreskog, K. G. (1971). Statistical analysis of sets of congeneric tests. Psychometrika, 36(2), 109–133.

    Article  Google Scholar 

  • Kuder, G. F., & Richardson, M. W. (1937). The theory of estimation of test reliability. Psychometrika, 2, 151–160.

    Article  Google Scholar 

  • Kuhn, T. S. (1961). The function of measurement in modern physical science. Isis, 52, 161–190.

    Article  Google Scholar 

  • Latour, B. (1987). Science in action: How to follow scientists and engineers through society. New York: Harvard University Press.

    Google Scholar 

  • Latour, B. (1993). We have never been modern. Cambridge, MA: Harvard University Press.

    Google Scholar 

  • Latour, B. (1998). To modernise or ecologise? That is the question. In B. Braun & N. Castree (Eds.), Remaking reality: Nature at the millennium (pp. 221–242). London: Routledge.

    Google Scholar 

  • Latour, B. (2005). Reassembling the social: An introduction to actor-network-theory. Oxford, England: Oxford University Press.

    Google Scholar 

  • Latour, B. (2013). An inquiry into modes of existence (C. Porter, Trans.). Cambridge, MA: Harvard University Press.

    Google Scholar 

  • Lazersfeld, P. (1966). Concept formation and measurement in the behavioral sciences: Some historical observations. In G. J. Direnzo (Ed.), Concepts, theory, and explanation in the behavioral sciences (pp. 144–202). New York: Random House.

    Google Scholar 

  • Lecoutre, M.-P., Poitevineau, J., & Lecoutre, B. (2003). Even statisticians are not immune to misinterpretations of Null Hypothesis Significance Tests. International Journal of Psychology, 38(1), 37–45.

    Article  Google Scholar 

  • Lewis, D., & Burke, C. J. (1949). The use and misuse of the chi-square test. Psychological Bulletin, 46(6), 433–487.

    Article  Google Scholar 

  • Linacre, J. M. (1989). Many-facet Rasch measurement. Chicago, IL: MESA Press. (http://www.winsteps.com/a/facets-manual.pdf).

  • Mari, L., & Wilson, M. (2014). An introduction to the Rasch measurement approach for metrologists. Measurement, 51, 315–327.

    Article  Google Scholar 

  • Mead, R. J. (2009). The ISR: Intelligent student reports. Journal of Applied Measurement, 10(2), 208–224.

    Google Scholar 

  • Meehl, P. E. (1967). Theory-testing in psychology and physics: A methodological paradox. Philosophy of Science, 34(2), 103–115.

    Article  Google Scholar 

  • Mokken, R. J. (1971). A theory and procedure of scale analysis. The Hague: Mouton/Berlin: De Gruyter.

    Google Scholar 

  • Nersessian, N. J. (2002). Maxwell and “the method of physical analogy”: Model-based reasoning, generic abstraction, and conceptual change. In D. Malament (Ed.), Reading natural philosophy: Essays in the history and philosophy of science and mathematics (pp. 129–166). Lasalle, IL: Open Court.

    Google Scholar 

  • Nersessian, N. J. (2006, December). Model-based reasoning in distributed cognitive systems. Philosophy of Science, 73, 699–709.

    Article  Google Scholar 

  • Nersessian, N. J. (2008). Creating scientific concepts. Cambridge, MA: MIT Press.

    Google Scholar 

  • Nersessian, N. J. (2012). Engineering concepts: The interplay between concept formation and modeling practices in bioengineering sciences. Mind, Culture, and Activity, 19, 222–239.

    Article  Google Scholar 

  • O’Connell, J. (1993). Metrology: The creation of universality by the circulation of particulars. Social Studies of Science, 23, 129–173.

    Article  Google Scholar 

  • Overton, W. F. (2015). Processes, relations and relational-developmental-systems. In W. F. Overton & P. C. M. Molenaar (Eds.), Theory and method. Volume 1 of the handbook of child psychology and developmental science (7th ed., Vol. 1, pp. 9–62). Hoboken, NJ: Wiley.

    Google Scholar 

  • Pendrill, L. (2014). Man as a measurement instrument [Special Feature]. NCSLi Measure: The Journal of Measurement Science, 9(4), 22–33.

    Article  Google Scholar 

  • Pendrill, L., & Fisher, W. P., Jr. (2015). Counting and quantification: Comparing psychometric and metrological perspectives on visual perceptions of number. Measurement, 71, 46–55.

    Article  Google Scholar 

  • Peters, M. A., & Burbules, N. C. (2004). Poststructuralism and educational research. Lanham, MD: Rowman & Littlefield Publishers Inc.

    Google Scholar 

  • Piety, P. (2011). Educational data use: A sociotechnical process. Measurement: Interdisciplinary Research & Perspectives, 9(4), 217–221.

    Google Scholar 

  • Popper, K. (1959). The Logic of Scientific Discovery. New York: Basic Books.

    Google Scholar 

  • Price, D. J. D. (1986). Of sealing wax and string. In D. J. D. Price (Ed.), Little science, big science–and beyond (pp. 237–253). New York: Columbia University Press.

    Google Scholar 

  • Quine, W. V. O. (1951). Two dogmas of empiricism. The Philosophical Review, 60, 20–43.

    Article  Google Scholar 

  • Rasch, G. (1960/1980). Probabilistic models for some intelligence and attainment tests (Reprint, with Foreword and Afterword by B. D. Wright, Chicago: University of Chicago Press, 1980). Copenhagen, Denmark: Danmarks Paedogogiske Institut.

    Google Scholar 

  • Ricoeur, P. (1967). Husserl: An analysis of his phenomenology (J. Wild, Ed.) (E. G. Ballard & L. E. Embree, Trans.). (Northwestern University Studies in Phenomenology & Existential Philosophy.) Evanston, IL: Northwestern University Press.

    Google Scholar 

  • Romer, T. A. (2013). Nature, education and things. Studies in the Philosophy of Education, 32, 641–652.

    Article  Google Scholar 

  • Rozeboom, W. W. (1960). The fallacy of the null-hypothesis significance test. Psychological Bulletin, 57(5), 416–428.

    Article  Google Scholar 

  • Scribner, R. A., Cohen, D. A., & Fisher, W. P. (2000). Evidence of a structural effect for alcohol outlet density: A multilevel analysis. Alcoholism, Clinical and Experimental Research, 24(2), 188–195.

    Article  Google Scholar 

  • Shrader-Frechette, K. (2014). Tainted: How philosophy of science can expose bad science. New York: Oxford University Press.

    Book  Google Scholar 

  • Sijtsma, K. (2009). On the use, the misuse, and the very limited usefulness of Cronbach’s alpha. Psychometrika, 74(1), 107–120.

    Article  Google Scholar 

  • Sijtsma, K. (2016). Playing with data–or how to discourage questionable research practices and stimulate researchers to do things right. Psychometrika, 81(1), 1–15.

    Article  Google Scholar 

  • Spearman, C. (1904). “General Intelligence”, objectively determined and measured. American Journal of Psychology, 13, 201–293.

    Article  Google Scholar 

  • Star, S. L., & Ruhleder, K. (1996). Steps toward an ecology of infrastructure: Design and access for large information spaces. Information Systems Research, 7(1), 111–134.

    Article  Google Scholar 

  • Stenner, A. J., Fisher, W. P., Jr., Stone, M. H., & Burdick, D. S. (2013). Causal Rasch models. Frontiers in Psychology: Quantitative Psychology and Measurement, 4(536), 1–14.

    Google Scholar 

  • Stone, M. H., Wright, B., & Stenner, A. J. (1999). Mapping variables. Journal of Outcome Measurement, 3(4), 308–322.

    Google Scholar 

  • Sutton, J., Harris, C. B., Keil, P. G., & Barnier, A. J. (2010). The psychology of memory, extended cognition, and socially distributed remembering. Phenomenology and the Cognitive Sciences, 9(4), 521–560.

    Article  Google Scholar 

  • Thorndike, E. L. (1904). An introduction to the theory of mental and social measurements. New York: Teachers’ College, Columbia University.

    Book  Google Scholar 

  • Thurstone, L. L. (1935). The vectors of mind. Chicago: University of Chicago Press.

    Google Scholar 

  • Thurstone, L. L., & Chave, E. J. (1929). The measurement of attitude: A psychophysical method and some experiments for measuring attitude toward the church. Chicago: The University of Chicago Press.

    Book  Google Scholar 

  • Treagust, D. F., Won, M., & Duit, R. (2014). Paradigms in science education research. In N. G. Lederman & S. K. Abell (Eds.), Handbook of Research on Science Education (Vol. II, pp. 3–17). New York: Routledge.

    Google Scholar 

  • Whitehead, A. N. (1911). An introduction to mathematics. New York: Henry Holt and Co.

    Google Scholar 

  • Wilson, M. (Ed.). (2004). National society for the study of education yearbooks. Vol. 103, Part II: Towards coherence between classroom assessment and accountability. Chicago, IL: University of Chicago Press.

    Google Scholar 

  • Wilson, M. (2005). Constructing measures: An item response modeling approach. Mahwah, NJ: Lawrence Erlbaum Associates.

    Google Scholar 

  • Wilson, M. R. (2013). Using the concept of a measurement system to characterize measurement models used in psychometrics. Measurement, 46, 3766–3774. Retrieved from http://www.sciencedirect.com/science/article/pii/S0263224113001061.

    Article  Google Scholar 

  • Wilson, M., Mari, L., Maul, A., & Torres Irribara, D. (2015). A comparison of measurement concepts across physical science and social science domains: Instrument design, calibration, and measurement. Journal of Physics Conference Series, 588(012034).

    Google Scholar 

  • Wise, M. N. (Ed.). (1995). The values of precision. Princeton, NJ: Princeton University Press.

    Google Scholar 

  • Wright, B. D., Mead, R. J., & Ludlow, L. H. (1980). KIDMAP: person-by-item interaction mapping (Technical Report No. MESA Memorandum #29). Chicago: MESA Press.

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. School of Education, Curtin University, GPO Box U1987, Perth, WA, 6845, Australia

    Robert F. Cavanagh

  2. University of California, Berkeley, CA, USA

    William P. Fisher Jr.

Authors
  1. Robert F. Cavanagh
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. William P. Fisher Jr.
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Robert F. Cavanagh .

Editor information

Editors and Affiliations

  1. College of Foreign Studies, Jiaxing University, Jiaxing, China

    Quan Zhang

Rights and permissions

Reprints and permissions

Copyright information

© 2018 Springer Nature Singapore Pte Ltd.

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Cavanagh, R.F., Fisher, W.P. (2018). Research Design Considerations in Human Science Research: Reconciling Conceptions of Science, Theories of Measurement and Research Methods. In: Zhang, Q. (eds) Pacific Rim Objective Measurement Symposium (PROMS) 2016 Conference Proceedings. Springer, Singapore. https://doi.org/10.1007/978-981-10-8138-5_5

Download citation

  • DOI: https://doi.org/10.1007/978-981-10-8138-5_5

  • Published:

  • Publisher Name: Springer, Singapore

  • Print ISBN: 978-981-10-8137-8

  • Online ISBN: 978-981-10-8138-5

  • eBook Packages: EducationEducation (R0)

Publish with us

Policies and ethics

Search

Navigation