Advertisement

Evidence and Argumentation in Educational Research

  • Nicholas C. BurbulesEmail author
Chapter
  • 760 Downloads
Part of the Educational Research book series (EDRE, volume 4)

Abstract

Researchers have generally stood in a ‘trust me’ relation, in between their readers and the information they analyze. (‘Information’ here can mean quantitative or qualitative data, textual information, multimedia resources, and so on – the bases of analysis or interpretation that underlie the knowledge claims made by the research itself.) While researchers might be quite explicit about their methods of data collection and analysis, the typical situation is that the only ‘raw’ data the reader sees are those samples or summaries the author chooses to use to illustrate particular claims – and this process is by definition selective. This selectivity even pertains to results or quotations cited from other research; we assume that they represent the other work accurately, but unless we are familiar with it ourselves we can never be sure. Such selectivity is not the only area in which the reader must implicitly trust the honesty and integrity of the researcher, but it is one of the most important because we must assume that the samples are representative and not simply chosen tendentiously to buttress specific conclusions.

Keywords

Knowledge Claim Public Money Digital Publishing Multimedia Resource Personal Disclosure 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

10.1 I

Researchers have generally stood in a ‘trust me’ relation, in between their readers and the information they analyze. (‘Information’ here can mean quantitative or qualitative data, textual information, multimedia resources, and so on – the bases of analysis or interpretation that underlie the knowledge claims made by the research itself.) While researchers might be quite explicit about their methods of data collection and analysis, the typical situation is that the only ‘raw’ data the reader sees are those samples or summaries the author chooses to use to illustrate particular claims – and this process is by definition selective. This selectivity even pertains to results or quotations cited from other research; we assume that they represent the other work accurately, but unless we are familiar with it ourselves we can never be sure. Such selectivity is not the only area in which the reader must implicitly trust the honesty and integrity of the researcher, but it is one of the most important because we must assume that the samples are representative and not simply chosen tendentiously to buttress specific conclusions.

Now, sometimes it is possible for the reader to check some of the researcher’s arguments and evidence: Sometimes enough statistical data are provided to allow for reanalyses using other tools; readers can interpret quotations or interview transcripts in divergent ways; where an article or book is cited, readers can check the original source and so on. Even within the confines of the publication itself, readers can question the logic of arguments, or raise doubts about the rhetorical moves of persuasion that carry the audience toward certain conclusions. Thus it is not necessary for the reader to have all the original data in order to critically assess research reasoning or results.

A number of years ago I wrote an essay analyzing the text of a children’s story book called ‘Tootle’ (Burbules, 1986). When it was published, I insisted that the journal, at some considerable inconvenience, include the full text of the story. (It would have been even better if they could have included the illustrations too.) It was a point of principle for me that the reader should be able to see the passages I quoted from, in context; that they should be able to see the passages I did not quote; that they should be able to read the story for themselves and decide whether my interpretation was worthy or not. The choice to republish the full text raised a number of problems: copyright approval, for one, but also the question of whether the original author and publisher would provide permission to republish their text for the purposes of an essay that was highly critical of the ideological subtexts of the story (Burbules, 2002).

In most other research contexts, the ‘raw data’ comprise much more than just a few hundred words copied from a children’s book: hundreds of surveys; hours and hours of audio or video recordings; boxes full of raw data that preceded statistical summary and analysis. The data disclosed in any study can only be the tip of the iceberg of all the evidence the author drew upon in making his or her conclusions, and so it is understandable to wonder about what is not being shown or discussed – not necessarily as an accusation of fraud or deception, but precisely because no study ever makes all of its evidence known.

Until recently it was impossible to imagine that every journal article, every book, could be published along with the data that support it. Today, the virtually unlimited capacity of digital publishing makes this feasible. Unlike print publishing, electronic publishing is not governed by an economics of scarcity. Since much data today is collected (or transcribed) in digital forms, it would be relatively straightforward to upload it as an appendix or addendum to a work published online (or on a publisher’s web site conjoined with a text published in print). This could include a variety of multimedia resources for research that uses them. Now that we can publish this information, the question before us is, Should we? That is the question I explore in this essay.

10.2 II

The ‘open source’, ‘open access’ ethos in today’s technology culture promotes the general idea that ‘knowledge wants to be free.’ In the place of a proprietary, often commercialized economy, more and more participants see new technologies as a medium for a more commons-based mode of interaction. Free access replaces fee-based access. Collaborative involvement replaces individual credit. Openness and transparency replace control. Much of the so-called ‘Web 2.0’ revolution is about public access to data bases that allow novel recombinations of data across different integrative purposes (‘mash-ups’). New data-mining techniques allow for reinterpretations of massive data bases. This open-access ethos is not just a free-floating normative article of faith. Yochai Benkler (2002) argues empirically that such commons-based peer production is often more productive than systems based more on individual ownership, motivation, and credit.

Researchers have traditionally spoken of ‘their’ data, the gathering of which often does entail hours upon hours of painstaking planning and collection. Think of the historian, filling hundreds of note-cards with information culled from dusty archives; or the interviewer, carefully building relations of trust and creatively formulating questions to get informants to share their stories. The collection of data is never mindless and routine – it is itself an exercise of talent, energy, and insight, and as such is a manifestation of this researcher with all of his or her strengths and shortcomings.

Moreover, even at this level an element of selectivity enters in: No one collects ‘all’ the data (whatever that would mean). Theoretically guided judgments about what is important and relevant enter in at even the earliest stage of research (What questions to ask; what indices to measure; why pick this topic of inquiry in the first place?). And so it must be said at the outset that a researcher’s data are in one sense distinctively ‘theirs’, every bit as much as anything else they produce.

But there is a separate, more troublesome sense of this phrase, ‘their’ data: the researcher who happens upon the collected letters of a major literary figure and hoards them so that others cannot review them, or the head of a research team composed of research aides or laboratory assistants whose collective efforts (often funded by public moneys) produced the data under consideration, but who expects to be listed as a coauthor on all studies coming out of that data. In such cases the personally proprietary attitude toward data seems less justified.

Researchers want to protect ‘their’ data, understandably, because they hope to get a number of publications out of it, or because they are jealous of preserving their unique access to it. They fear that if the information were made publicly available someone else might beat them to further publications, or steal some insight or discovery they want for themselves. But here one needs to weigh the understandable self-interest of the researcher with the public interest in open access to information.

When one is publishing their research, are they publishing only the results, or the entire process of data collection, analysis, and interpretation? Who deserves credit for sharing a body of data, which others may use for their own purposes? Might scholars become as famous and widely credited for producing a rich and fertile set of data, as they might become for their own specific analyses of it? What new forms of attribution, acknowledgment, and coauthorship might be appropriate for this more distributed model of inquiry? These are new questions we need to be asking.

10.3 III

Making the data available along with the research analyses of it manifests a different relation between the researcher and the audience, and a different conception of public knowledge. In place of a ‘trust me’ relation, the researcher invites the readers to check the analysis against the data that support it. They may discover errors or omissions that critique the original study or extend its analyses further. This provides an independent check on credibility and validity. Such access also allows that other analyses and other uses of the data are possible. The wider community of inquiry can benefit by collaborating, in a sense, with the original author through these other projects. It is a relation based more on transparency – and, one might even add, generosity – than the more typical possessive, proprietary mode. Readers are invited into examining the evidence for argumentation, the rhetorics of persuasion, and the choices of selective disclosure and nondisclosure that underlie any research study. Indeed, contrasting the original data with the published uses made of them invites questions about the inevitability of selectivity and judgment in making these determinations. There is not a single, direct line of inference from any set of data to particular conclusions, and readers should be made aware of this.

Many years ago, I was part of a research project in which one of the research assistants committed an innocent but devastating error in judgment that invalidated nearly all of the data he collected. None of those data made it into the final analysis, of course – but the fact that it happened, and the choice the authors made in not mentioning that it happened, could not have been known to anyone but those directly involved in the project. Should the error have been disclosed, even though it did not affect anything that was used for the analysis in the published article? This was a judgment made conscientiously, but others might have made that judgment differently. My own sense, over the intervening years, is that many – perhaps most – empirical studies experience similar difficulties along the way, which are very rarely discussed in the final report. Research is an imperfect human practice, like any other, conducted by serious but fallible human beings. The kind of transparency argued for here would allow readers to make independent judgments about the importance of such research errors or snafus and their relevance to assessing the research results.

In such a new relation the authority and credibility of research knowledge comes to be based in a wider and more distributed set of relations. The author’s individual voice, style, or point of view should be respected in relation to their own inquiries, but they need not bound or limit the possibilities of other independent inquiries. Such a system might make us more aware of the rhetorics and standards of evidence that make us find research arguments compelling, while illuminating the possibility of alternative treatments of the same material. Just as different biographies might make us think about the diverse characterizations that can be made of a ‘single’ lifetime, we might become better attuned to the different ways in which the ‘same’ classroom, or the ‘same’ teacher at work, can be viewed and assessed. Such a diversity of analyses and interpretations need not yield relativistic conclusions, but it should make us question the simpleminded assumption that all studies necessarily converge on a single ‘truth.’ On the other hand, it may also turn out that multiple analyses of the same data establish stronger intersubjective agreement on at least some aspects of a study, yielding greater confidence in them.

There are wider benefits as well. No data set is ever exhausted by a single analysis, or set of analyses; there are always other theoretical frames, other questions, that might be brought to bear. Multiple data sets, developed by different researchers in different contexts for different purposes, may yield broader conclusions when compiled together or studied in a comparative way. The open data approach proposed here illuminates the importance of these theoretical frameworks and assumptions.

Some of the researchers who access these data and use them in new ways may be talented ‘amateurs’ who do not have the institutional funding or support to allow them to carry out large data collection themselves. As in many other cases of the new digital knowledge network, a plurality of perspectives can give voice to scholars who have been closed out of traditional publication avenues; as with ‘blogs’ and other new media, it is sometimes the ‘amateurs’ and ‘outsiders’ who contribute fresh and original insights to a knowledge enterprise. Publishing open data in the way described here can be vastly empowering to these new voices and perspectives: It becomes an occasion to create an eclectic community of inquiry around a body of shared information and a related set of questions growing out of it.

Obviously, some very large data sets, generated through public funding, are already made widely accessible in this way, across a range of different disciplines. But data in educational research may have a special status in this regard, since education is a public concern and since so much of this research is funded through public moneys. It is an area in which research carried out in another context – a different school, city, or country – might yield important insights into one’s own. And so some wider public good is also served by making such data freely and widely available.

10.4 IV

There are certainly serious issues in making data freely and widely available: protections of legitimate intellectual property claims; issues about the confidentiality of research subjects or informants who have developed a relation of trust with the researcher but who have not agreed to make their personal disclosures widely available; the special vulnerability of children and other research subjects who need the protection of anonymity; the threatened status of ‘whistleblowers’ who cannot afford to be identified and who would not participate if they knew their comments would be publicized in a manner that might allow them to be identified.

The reward systems developed for academic researchers also work generally against this open-source ethos: when one is rewarded for numbers of publications, for ‘original’ or ‘unique’ discoveries, for single-authored or first-authored articles, and so on, where is the incentive for releasing data that others might use for their own publication and professional benefit? In my college, some of the hallways are lined with boxes with faculty names scrawled on the side, containing primary data materials, some covered with the dust of many years, just waiting for the day when the original author might decide they can be reused again. Imagine an alternative scholarly economy in which the original researcher may get fewer studies out of the data, while the research community as a whole might get more. This is probably a beneficial trade-off from the larger perspective of the public good but not necessarily from the standpoint of the individual self-interest of the researcher. In whose favour do we adjudicate this trade-off of benefits?

As noted, such a revised relation between the researcher and audience also might yield a different set of attitudes about the finality of research results; about the bases of research authority and credibility; about the necessity of selectivity in representing the data; about the possibilities and benefits of multiple, sometimes conflicting, analyses of the same data; about changing rules for intellectual property and plagiarism; and about a wider shift of emphasis from the individual researcher to the research community as the locus of epistemological control and accountability.

This shift also reflects a different sense of research ethics and responsibilities: If I know that I will be expected to share the underlying data on which I draw my conclusions, I might be especially scrupulous in staying within the bounds of what I can confidently assert. I might be more open to acknowledging the selectivity – and the criteria for selectivity – in what I choose to disclose, or not to disclose, in the published form of my study. I might be more generous in acknowledging the possibility, and the value, of alternative interpretations of my data. And I might be more attuned to my own situatedness within broader communities of inquiry to which I have obligations going beyond the protection of my own individual research interests and autonomy.

In all of these respects, the possibility of publishing – that is, making public – the data that accompany published research opens up new ways of thinking about the point and purpose of doing research at all. Some important issues, along the lines of problems described here, will have to be worked out. In various academic fields professional societies may have a key role to play in setting such rules and standards. But overall it seems a policy that journal editors and publishers should seriously consider. Agencies that fund research with public money may mandate the release of data for the public good (and some are beginning to do that). There are reasons why many individual researchers will be reluctant to do so unless they are required to; but I expect that more and more people will choose to do this, as the avenues for making data public become safe and convenient, because they recognize the wider good served when knowledge is made free. Researchers may find that the benefit of having their data used (with attribution or acknowledgement) in other researchers’ studies constitutes a new kind of role and service to the research community – one, presumably, with reciprocal benefits. In research as in other contexts, a rising tide lifts all boats.

References

  1. Benkler, Y. (2002). Coase’s penguin, or, linux and the nature of the firm. The Yale Law Journal, 112(3), 369–446.CrossRefGoogle Scholar
  2. Burbules, N. (1986). Tootle: A parable of schooling and destiny. Harvard Educational Review, 56(3), 239–256.Google Scholar
  3. Burbules, N. (2002). Tootle revisited: Fifteen years down the track. In S. Merriam & D. Vreeland (Eds.), Types of qualitative inquiry: Exemplars for study and discussion (pp. 348–351). San Francisco: Jossey-Bass.Google Scholar

Copyright information

© Springer Science+Business Media B.V. 2009

Authors and Affiliations

  1. 1.Department of Educational Policy StudiesUniversity of IllinoisUrbana-ChampaignUSA

Personalised recommendations