Skip to main content
SpringerLink
Log in

Replicating Text: The Cumulation of Knowledge in Social Science

  • Published:
Quality and Quantity Aims and scope Submit manuscript

Abstract

Obtaining a statistically significant result does not necessarily tell us whether we would obtain significant results in other, similar studies, particularly if the original sample sizes were small. This is why we are supposed to replicate experiments. The present study concerns social science events that cannot be repeated by virtue of their being historically situated. Among social science events, many textual data are datable and, by definition, unrepeatable. One solution to this quandary lies in bootstrap replications, which are based on the original data. A case in point is that of founding political speeches such as those that buoy the European construction. We analyze and compare 82 speeches made by President Delors over the period 1988–1994, and 28 by President Santer over the period 1995–1997. We have all these speeches (N = 110) concorded as to which words are used, how often, where, and when, with the help of a computer-aided content analysis package. We then test various hypotheses using replication bootstrap estimates, that is, by replicating the original sample a large number of times and recreating several thousand samples from the population so created.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  • Bakan, D. (1966). The test of significance in psychological research, Psychological Bulletin 66: 423–437.

    Google Scholar 

  • Bedell, G. (1998, February). Can Microsoft stay ahead?, Prospect 27: 58–61.

    Google Scholar 

  • Benjafield, J. & Muckenheim, R. (1989). An historicodevelopmental analysis of the Regressive Imagery Dictionary, Empirical Studies of the Arts 7: 79–88.

    Google Scholar 

  • Berlin, I. (1974). Four Essays on Liberty. London: Oxford University Press.

    Google Scholar 

  • Bestgen, Y. (1994). Can emotional valence in stories be determined from words?, Cognition and Emotion 8: 21–36.

    Google Scholar 

  • Brooks, D. (1995). The House of Balthus. Sydney, Australia: Allen & Unwin.

    Google Scholar 

  • Cirincione, C. & Gurrieri, G.A. (1997). Computer-intensive methods in the social sciences, Social Science Computer Review 15: 83–97.

    Google Scholar 

  • Cohen, J. (1994). The earth is round (p <:05), American Psychologist 49: 997–1003.

    Article  Google Scholar 

  • Crosbie, J. (1993). Interrupted time-series analysis with brief single-subject data, Journal of Consulting and Clinical Psychology 61: 966–974.

    Google Scholar 

  • Dalgleish, L.I. (1994). Discriminant analysis: Statistical inference using the jackknife and bootstrap procedures, Psychological Bulletin 116: 498–508.

    Google Scholar 

  • de Man, P. (1993). Romanticism and Contemporary Criticism: The Gauss Seminar and Other Papers. Baltimore: The John Hopkins University Press.

    Google Scholar 

  • Desrochers, A. & Bergeron, M. (1992). Valeurs de fréquence subjective et d'imagerie pour un échantillon de 1,916 substantifs de la langue française [Subjective frequency and imagery values for a sample of 1,916 nouns of the French language] (Manuscript RR36): Laboratoire de Psychologie Cognitive, Ecole de Psychologie, Université d'Ottawa, Ottawa, Ontario K1N 6N5, Canada.

    Google Scholar 

  • Diaconis, P. & Efron, B. (1983). Computer intensive methods in statistics, Scientific American 248: 96–108.

    Google Scholar 

  • Eagly, A.H. & Chaiken, S. (1993). The Psychology of Attitudes. Forth Worth, TX: Harcourt Brace Jovanovich.

    Google Scholar 

  • Efron, B. (1981). Nonparametric estimates of standard error: The jackknife, the bootstrap, and other resampling methods, Biometrika 68: 589–599.

    Google Scholar 

  • Efron, B. (1982). The Jackknife, the Bootstrap and Other Resampling Plans (1986 ed.) (Vol. 38). Philadelphia, PA: Society for Industrial and Applied Mathematics.

    Google Scholar 

  • Efron, B. & Tibshirani, R. (1991, 26 July). Statistical data in the computer age, Science 253: 390–395.

    Google Scholar 

  • Efron, B. & Tibshirani, R. (1993). Introduction to the Bootstrap. London: Chapman & Hall.

    Google Scholar 

  • Evans, W. (1996). Computer-supported content analysis, Social Science Computer Review 14: 269–279.

    Google Scholar 

  • Ferguson, N. (ed.) (1997). Virtual History.Alternatives and Counterfactuals. London: Picador.

    Google Scholar 

  • Fiske, D.W. (1961). The inherent variability of behavior. In: D.W. Fiske & S.R. Maddi (eds), Functions of Varied Experience. Homewood, IL: The Dorsey Press, Inc., pp. 326–354.

    Google Scholar 

  • Flesch, R. (1948). A new readability yardstick, Journal of Applied Psychology 32: 221–233.

    Google Scholar 

  • Fogel, R.W. (1964). Railroads and American Economic Growth.Essays in Econometric History. Baltimore: Hopkins.

    Google Scholar 

  • Fogel, R.W. (1969). The new history: Its findings and methods. In: D.K. Rowney & J.Q. Graham (eds), Quantitative History: Selected Readings in the Quantitative Analysis of Historical Data. Homewood, IL: The Dorsey Press, Inc., pp. 320–325.

    Google Scholar 

  • Fowles, J. (1969). The French Lieutenant's Woman. London: Cape.

    Google Scholar 

  • Gottman, J.M. (1981). Time-series Analysis: A Comprehensive Introduction for Social Scientists. Cambridge, England: Cambridge University Press.

    Google Scholar 

  • Gunning, R. (1968). The Technique of Clear Writing (Revised ed.). New York: McGraw-Hill Book Company.

    Google Scholar 

  • Handelman, S.A. (1982). The Slayers ofMoses: The Emergence of Rabbinic Interpretation inModern Literary Theory. Albany: State University of New York Press.

    Google Scholar 

  • Handelman, S.A. (1985). Fragments of the rock: Contemporary literary theory and the study of Rabbinic texts-a response to David Stern, Prooftexts 5: 75–103.

    Google Scholar 

  • Hendrickx, A. & Peeters, G. (submitted). Ambiguity from a social-perceptual and a linguistic perspective: An integrated approach, Journal of Language and Social Psychology.

  • Hewstone, M. (1983). The role of language in attribution processes. In: J. Jaspers, F.D. Fincham & M. Hewstone (eds), Attribution Theory and Research: Conceptual Developments and Social Dimensions. London: Academic Press, pp. 241–260.

    Google Scholar 

  • Hjorth, J.S.U. (1994). Computer-Intensive Statistical Methods: Validation Model Selection and Bootstrap. London: Chapman & Hall.

    Google Scholar 

  • Hogenraad, R., Daubies, C. & Bestgen, Y. (1995). Une théorie et une méthode générale d'analyse textuelle assistée par ordinateur. Le système PROTAN (PROTocol ANalyzer) (Version du 2 mars 1995) [A general theory and method of computer-aided text analysis: The PROTAN system (PROTocol Analyzer), Version of March 2, 1995] [Computer program]. Louvain-la-Neuve, Belgium: Psychology Department, Catholic University of Louvain.

    Google Scholar 

  • Hogenraad, R., McKenzie, D.P. & Martindale, C. (1997). The enemy within: Autocorrelation bias in content analysis of narratives, Computers and the Humanities 30: 433–439.

    Google Scholar 

  • Ide, N.M. (1989). A statistical measure of theme and structure, Computers and the Humanities 23: 277–283.

    Google Scholar 

  • Iker, H.P. (1974a). An historical note on the use of word-frequency contiguities in content analysis, Computers and the Humanities 8: 93–98.

    Google Scholar 

  • Iker, H. P. (1974b). SELECT: A computer program to identify associationally rich words for content analysis: I. Statistical results, Computers and the Humanities 8: 313–319.

    Google Scholar 

  • Iker, H.P. & Harway, N.I. (1969). A computer systems approach toward the recognition and analysis of content. In: G. Gerbner, O.R. Holsti, K. Krippendorff, W.J. Paisley, & P.J. Stone (eds), The Analysis of Communication Content.Developments in Scientific Theories and Computer Techniques. New York: John Wiley & Sons, Inc., pp. 381–405.

    Google Scholar 

  • Iker, H.P. & Klein, R.H. (1974). WORDS: A computer system for the analysis of content, Behavior Research Methods & Instrumentation 6: 430–438.

    Google Scholar 

  • Johnson-Laird, P.N. & Oatley, K. (1989). The language of emotions: An analysis of a semantic field, Cognition and Emotion 3: 81–123.

    Google Scholar 

  • Kermode, F. (1973). The use of the codes. In: S. Chatman (ed.), Approaches to Poetics. New York: Columbia University Press, pp. 51–79.

    Google Scholar 

  • Klein, R.H. (1976). A computer analysis of the Schreber memoirs, The Journal of Nervous and Mental Disease 162: 373–384.

    Google Scholar 

  • Leleu, S. (1987). Un atlas sémantique de concepts d'émotion: Normes et validation [A semantic lexicon of emotion: Norms and validation]. Unpublished MA Thesis, Psychology, Catholic University of Louvain.

  • Mailloux, S.L., Johnson, M.E., Fisher, D.G. & Pettibone, T. J. (1995). How reliable is computerized assessment of readability?, Computers in Nursing 13: 221–225.

    Google Scholar 

  • Mandel, D.R., & Lehman, D.R. (1996). Counterfactual thinking and ascriptions of cause and preventability, Journal of Personality and Social Psychology 7: 450–463.

    Google Scholar 

  • Martindale, C. (1975). Romantic Progression: The Psychology of Literary History.Washington, DC: Hemisphere.

    Google Scholar 

  • Martindale, C. (1976). Primitive mentality and the relationship between art and society, Scientific Aesthetics 1: 5–18.

    Google Scholar 

  • Martindale, C. (1979). The night journey: Trends in the content of narratives symbolizing alteration of consciousness, Journal of Altered States of Consciousness 4: 321–343.

    Google Scholar 

  • Martindale, C. (1990). The Clockwork Muse: The Predictability of Artistic Change. NewYork: Basic Books.

    Google Scholar 

  • McKenzie, D.P., Mackinnon, A.J., Péladeau, N., Bruce, P.C., Onghena, P., Clark, D.M., Harrigan, S. & McGorry, P.D. (1996). Comparing correlated kappas by resampling: Is one level of agreement significantly different from another, Journal of Psychiatric Research 30: 483–492.

    Google Scholar 

  • McTavish, D. G.& Pirro, E. B. (1990). Contextual content analysis, Quality & Quantity 24: 245–265.

    Google Scholar 

  • Meehl, P.E. (1990). Why summaries of research on psychological theories are often uninterpretable, Psychological Reports 66 (Monograph Supplement 1-V66), pp. 195–244.

  • Miall, D.S. (1988). Affect and narrative: A model of response to stories, Poetics 17: 259–272.

    Google Scholar 

  • Miall, D.S. (1992). Estimating changes in collocations of key words across a large text: A case study of Coleridge's notebooks, Computers and the Humanities 26: 1–12.

    Google Scholar 

  • Micceri, T. (1989). The unicorn, the normal curve, and other improbable creatures, Psychological Bulletin 105: 156–166.

    Google Scholar 

  • Musil, R. (1995). The Man Without Qualities [Mann ohne Eigenschaften] (Sophie Wilkins and Burton Pike, Trans.). London: Picador (Original work published 1942).

    Google Scholar 

  • Ohmae, K. (1993). The rise of the region state, Foreign Affairs 72: 78–87.

    Google Scholar 

  • Osgood, C.E., May, W.H. & Miron, M.S. (1975). Cross-cultural Universals of Affective Meaning. Urbana: University of Illinois Press.

    Google Scholar 

  • Paletz, D.L. (1996). Advanced information technology and political communication, Social Science Computer Review 14: 75–77.

    Google Scholar 

  • Péladeau, N. (1996). Simstat for Windows. User's guide (Version 1.2, May 1997) [Computer program]. Montréal, Canada: Provalis Research.

    Google Scholar 

  • Péladeau, N. & Lacouture, Y. (1993). SIMSTAT: Bootstrap computer simulation and statistical program for IBM personal computers. Behavior Research Methods, Instruments, & Computers 25: 410–413.

    Google Scholar 

  • Pocock, J. G. A. (1993, 7 January). What do we mean by it? (Review of J. H. Burns and M. Goldie, The Cambridge History of Political Thought: 1450–1700, Cambridge: Cambridge University Press). London Review of Books 15: 11–12.

    Google Scholar 

  • Roese, N.J. (1997). Counterfactual thinking, Psychological Bulletin 121: 133–148.

    PubMed  Google Scholar 

  • SAS Institute, Inc. (1985). SAS © User's Guide: Statistics, Version 5 Edition. Cary, NC: SAS Institute Inc.

    Google Scholar 

  • SAS Institute, Inc. (1993). SAS/ETS © User's Guide, Version 6 Edition. Cary, NC: SAS Institute Inc.

    Google Scholar 

  • Schmidt, F.L. (1992). What do data really mean? Research findings, meta-analysis, and cumulative knowledge in psychology. American Psychologist 47: 1173–1181.

    Google Scholar 

  • Schmidt, F.L. (1996). Statistical significance testing and cumulative knowledge in psychology: Implications for training of researchers, Psychological Methods 1: 115–129.

    Google Scholar 

  • Scott, R.L., Thompson, B. & Sexton, D. (1989). Structure of a short form of the Questionnaire on Resources and Stress: A bootstrap factor analysis, Educational and Psychological Measurement 49: 409–419.

    Google Scholar 

  • Shapin, S. (1996). The Scientific Revolution. Chicago: The University of Chicago Press.

    Google Scholar 

  • Shklar, J.N. (1984). Putting cruelty first. In: J.N. Shklar (ed.), Ordinary Vices. Cambridge, MA: Belknap Press of Harvard University Press, pp. 7–44.

    Google Scholar 

  • Spence, D.P. & Owens, K.C. (1990). Lexical co-occurrence and association strength, Journal of Psycholinguistic Research 19: 317–330.

    Google Scholar 

  • Stern, J.P. (1992). The education of the master race. In: J.P. Stern (ed.), The Heart of Europe: Essays on Literature and Ideology. Oxford: Blackwell, pp. 78–93.

    Google Scholar 

  • Stine, R. (1989). An introduction to bootstrap methods: Examples and ideas, Sociological Methods and Research 18: 243–291.

    Google Scholar 

  • Stoppard, T. (1993). Arcadia. London: Faber and Faber.

    Google Scholar 

  • Tetlock, P.E. & Belkin, A. (eds). (1996). Counterfactual Thought Experiments in World Politics.Logical, Methodological, and Psychological Perspectives. Princeton: Princeton University Press.

    Google Scholar 

  • Thompson, B. (1988). Program FACSTRAP: A program that computes bootstrap estimates of factor structures. Educational and Psychological Measurement 48: 681–686.

    Google Scholar 

  • Thompson, B. (1995). Exploring the replicability of a study's results: Bootstrap statistics for the multivariate case. Educational and Psychological Measurement 55: 84–94.

    Google Scholar 

  • Tonn, B.E. (1996). Global society and information technology: Social science challenges in the 21st century. Social Science Computer Review 14: 78–80.

    Google Scholar 

  • Urwin, D.W. (1995). The Community of Europe: A History of European Integration since 1945 (2nd ed.). London: Longman.

    Google Scholar 

  • Wallace, D. (1997). SIMSTAT for Windows. Social Science Computer Review 15: 310–312.

    Google Scholar 

  • Weber, R.P. (1983). Measurement models for content analysis. Quality & Quantity 17: 127–149.

    Google Scholar 

  • Wilks, Y. (1997). Senses and texts. Computers and the Humanities 31: 77–90.

    Google Scholar 

  • Young, G.A. (1994). Bootstrap: More than a stab in the dark. Statistical Science 9: 382–415.

    Google Scholar 

  • Zoski, K.W. & Jurs, S. (1996). An objective counterpart to the visual scree test for factor analysis: The standard error scree. Educational and Psychological Measurement 56: 443–451.

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Psychology, Catholic University of Louvain, Louvain-, la-Neuve, Belgium

    Robert Hogenraad

  2. Victorian Transcultural Psychiatry Unit, St. Vincent's Hospital and Department of Psychiatry, University of Melbourne, Melbourne, Australia

    Dean P. McKENZIE

Authors
  1. Robert Hogenraad
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Dean P. McKENZIE
    View author publications

    You can also search for this author in PubMed Google Scholar

Rights and permissions

Reprints and permissions

About this article

Cite this article

Hogenraad, R., McKENZIE, D.P. Replicating Text: The Cumulation of Knowledge in Social Science. Quality & Quantity 33, 97–116 (1999). https://doi.org/10.1023/A:1026421730175

Download citation

  • Issue Date:

  • DOI: https://doi.org/10.1023/A:1026421730175

Search

Navigation