Distorted transmission

Conclusion

So far I sought to describe and explain the process wherein university circuit Seattle-Denver experiment research became press reports. I conclude by seeking some remedies for distorted diffusion of quantitative social research findings. First I contrast an earlier study's explanations of distorted diffusion with my own. Then, drawing on my description and explanations of Seattle-Denver distortion, I suggest how university circuit researchers could have achieved less distorted transmission. Finally, I consider some emerging trends in university circuit research that seem to augur well for the reduction of distortion in future transmission of quantitative social research findings.

An earlier study

The civil rights movement of the early 1960s and urban revolts of the mid-1960s led to legal system responses during the early 1970s. Federal courts ordered public school districts to draw up and implement desegregation plans. In 1975 James Coleman argued that courtordered public school desegregation was self-defeating because it led to massive withdrawal of whites from public schools. He based his argument on his own quantitative social research findings, and presented it in six papers, media interviews and articles, court affidavits, and Congressional testimony. A 1976 university circuit journal article by Thomas F. Pettigrew and Robert L. Green criticized Coleman's 1975 research and transmission.

Pettigrew and Green argued that press reports of Coleman's work contained distortions for at least four reasons. First, Coleman began his media campaign four months before offering other university circuit researchers any technical details of research findings. Second, he constantly changed those technical details during the course of his eightmonth media campaign. Third, there was “consistent confusion between Coleman's personal opinions and his research findings.”¹ Finally, quantitative social researchers had little experience in dealing with the media and “within the news media, social science has yet to be elevated to the status of a regular, specialized ‘beat’.”²

Some of the reasons Pettigrew and Green provide for the distorted diffusion of Coleman's work resemble my explanations for the distorted transmission of Seattle-Denver findings. For example, Seattle-Denver researchers changed marital events findings and later models just as Coleman changed technical details. But Seattle-Denver merely updated findings and altered a model assumption that the updates suggested was false while Coleman radically redesigned his research for no apparent reason. Also, newspapers not having a specialized “beat” resembles my explanation that university and popular circuit life-worlds differ. But whereas newspapers could add a “beat,” how to alter life-world differences between circuits is less clear.

The reasons Pettigrew and Green offer for distortion center on Coleman's breaches of standard quantitative social research practice. He brought findings to the media without prior peer review, radically redesigned his research for no apparent reason, and explicitly interjected his political opinions into his reported findings. Seattle-Denver researchers committed no clear breaches of standard practice, but distorted transmission of their findings occurred nonetheless. Thus while breaches of standard practice may explain some distortion in Coleman's case, my account suggests that such distortion generally has deeper origins.

Reduced distortion in Seattle-Denver

Pettigrew and Green conclude by raising Foucault's question of the responsibility of those who write: “we firmly believe that social science can and should influence public-policy issues on which it can responsibly bring research and theory to bear.”³ How could the Seattle-Denver university circuit researchers have brought their research to bear more responsibly on the public-policy issue of whether or not there should be an NIT? My account suggests the researchers' lapses of responsibility were of two sorts. First, they failed to consider adequately the limited capacity of research techniques available now to answer their questions. Second, they paid insufficient attention to the political origins of those questions.

The Seattle-Denver researchers should have conducted a complete sensitivity analysis of each of their findings to simultaneous attrition, self-selection, and subject-reporting biases. For the marital events findings, the complete sensitivity analysis also should have covered simultaneous reconciliation and transiency biases. Each journal article and working paper that presented findings should have contained the full results of such a sensitivity analysis. When a paper updated earlier findings it should have subjected the updated, not earlier, findings to sensitivity analysis. Had the researchers taken these simple steps most of the initial state distortion in the transmission of their findings would have disappeared.

In the late 1960s liberals such as Moynihan and conservatives such as Long understood an NIT as income redistribution toward the poor in response to their revolt's pressure. Moynihan sincerely hoped an NIT would strengthten poor family stability, but also sought to use that hope as a pitch with which to sell an NIT's income redistribution toward the poor and thus restore social peace. Long sincerely feared an NIT would lead masses of the poor to stop working, but also sought to use that fear to block an NIT's income redistribution toward the poor. By 1978, however, tax revolt pressure not to redistribute income toward the poor was replacing the subsided pressure of the poor's revolt.⁴

Addressing Moynihan, the Seattle-Denver researchers should have said only that eight years of careful inquiry with the best available statistical methods had provided no reason to believe that an NIT would strenghten poor family stability. Addressing Long, the researchers should have said only that eight years of careful inquiry with the best available econometric methods had provided no reason to suppose that an NIT would lead masses of the poor to stop working. Such testimony would have answered the questions the political debate had raised without going further than the weaknesses in the researchers' methods responsibly allow. And such cautious and conservative testimony would have left little room for distortion in transmission.

Cautious and conservative testimony also might have forced honesty and responsibility into the political debate. Such testimony would have shorn liberals of their family stability sales pitch for, and conservatives of their work reduction fear-mongering against, an NIT. The debate could then have focused on how much if any income redistribution toward the poor the body politic wanted to undertake and which if any social classes would be forced to provide the income. Instead, the researchers' testimony gave Congress “scientific” justification for scrapping an NIT and joining the nascent executive branch retreat, in the face of rising tax revolt pressure, from income redistribution toward the poor.

Emerging self-reflection

My life-world explanation noted that one source of distorted transmission of Seattle-Denver findings was the researchers' shared background assumption that they practiced a science akin to physics or biology. In developing his life-world notion Habermas argues: The life-world is that remarkable thing that dissolves and disappears before our eyes as soon as we try to take it up piece by piece. The life-world functions in relation to processes of communication as a resource for what goes into explicit expression. But the moment this background knowledge enters communicative expression, where it becomes explicit knowledge and thereby subject to criticism, it loses precisely those characteristics by virtue of which it belonged to the life-world structures: certainty, background character, impossibility of being gone behind.⁵

Suppose Habermas's argument is correct and my life-world explanation of distortion generalizes to other instances of quantitative social research diffusion. Then by critically examining the background assumption that it practices science, the quantitative social research community could reduce distortion in transmission of its work. Such a literature of self-reflection does indeed appear to be emerging.

A 1968 article by Bill Alonso offers early critical self-reflection: “avoid as far as possible models which proceed by chains” because errors “will compound through the operations of the model, as the dependent variables of one step in the chain become the ‘exogeneous’ inputs into the next step.”⁶ For example, in Seattle-Denver “normal income for each family was estimated judgmentally because of the absence of any reliable models to make the assignment.”⁷ Then normal income became an input in estimating change in hours-worked and change in hours-worked became an input in estimating the cost of a national NIT. Hence the cost estimate contained error compounded from the original errors of judgmentally estimated normal income. Recent articles by Ed Leamer and other econometricians question standard practices of quantitative social research. ... disorganized studies of fragility are inefficient, haphazard, and confusing. ... What we need instead are organized sensitivity analyses. We must insist that all empirical studies offer convincing evidence of inferential sturdiness. We need to be shown that minor changes in the list of variables do not alter fundamentally the conclusions, nor does a slight reweighting of observations, nor correction for dependence among observations, etcetera, etcetera.... Normally, this experimentation is limited to a small subset of the possible models that could have been estimated. Suppose instead that we consider the whole continuum of models....⁸

So Leamer calls for future empirical studies to include systematic analysis of the sensitivity of findings to inclusion of variables, weighting of observations, choice of models, and other researcher data analytic decisions. Seattle-Denver studies included no such systematic analysis of findings to researcher data analytic decisions. For example, the marital-events research included no systematic analysis of findings to inclusion of variables, weighting of observations, rate definition, reconciliation noncounting, and choice of a model with no transiency effect.

Recent articles by statisticians explore the connection between random assignment and cause and effect conclusions. In one such article, “regarding large sample significance tests, the usual statistical procedures are shown to be conservative.”⁹ That is, p-values in random assignment studies may understate cause and effect relations. I argued that random assignment studies of human social behavior may overstate cause and effect relations because of attrition, self-selection, and subject-reporting biases. Hence the connection between random assignment and social cause and effect relations is weaker than many believed when the income maintenance experiments began. Statisticians have even asked philosophers to help explore the connection further.¹⁰

Critical reflection on their techniques' limitations by econometricians, sociometricians, and statisticians should reduce future initial state transmission distortion of their empirical findings. One hopes those researchers' critical reflections will become a part of the academic curriculum. For example, a seminar on the limitations of statistical techniques taught jointly by quantitative social researchers and philosophers might become required of all social “science,” journalism, and law students. Such curriculum reform should eventually lessen differences in background assumptions in the different circuits' lifeworlds and so further reduce future distortion in transmission of quantitative social research findings. Consider finally the following passage: In order to design a bridge an engineer must have the assurance that, if he follows certain formulas dealing with gravitational stresses and the strength of materials, his bridge will hold. He must be able to predict physical events. That he has such an assurance and ability the entire science of building engineering will attest. Now one circumstance alone has made this prediction possible, — the fact that natural events, in the realm of bridge-construction, operate in a fairly orderly and predictable manner. We have laws of physics. Similarly, an industrial chemist works out formulas upon the basis of which managers plan large scale operations and investments. Such planning depends upon the possibility of reliable prediction; and this prediction, in turn, rests upon the fact that there are such things as laws of chemistry. A medical director lays plans for public sanitation which are of the greatest social value. This, again, is possible only because he can predict, according to certain laws, the course of the propagation of micro-organisms. In every field of natural science we find that the ability to plan depends upon our ability to predict, and prediction, in turn, is possible only because events happen, in that field, according to fairly definite and universal laws. But how about the field of social science? What broad and precise societal laws have been discovered upon which intelligent social prediction and planning can be based? To this question the labors of social scientists have, in general, returned a baffling answer: There are none. Events in society, depending as they do on so many, continually shifting circumstances, simply do not fit themselves into any orderly and predictable array. A few rough generalizations there are, to be sure; but most of these are either too vague to apply, or they foretell only what will happen under conditions of a highly specific sort, so that, in our ignorance of when or where these conditions will obtain, our attempt at prediction is practically worthless. Such social or economic “laws” as we possess will never enable us to plan a structure which, like the bridge of the engineer, will do its work unaltered through change and storm.¹¹

So, for example, Seattle-Denver researchers gathered data during a period of rising divorce rates and widespread revolt against work. Once divorce rates leveled off and social tastes and preferences for laboring returned to a hard work ethic, Seattle-Denver's findings on maritalstability and hours-worked reductions probably became overestimates of what to expect from an NIT's introduction. So the passage, written by a social theorist in 1933, contains lessons on the limitations of our techniques that many of us quantitative social researchers have yet to learn.