Juries and Other Reliable Predictors

Abstract

Holmes saw juries as a conduit into the courtroom of popular feelings and prejudices, ensuring that the law is kept in accord with the will of the community. Thus juries are analogous to the training dataset: they provide the givens, the ground truth. A judge who is faced with an intractable question about which she would rather not make a decision will look for a way to send that question to the jury. Likewise, with the successes of machine learning it can be tempting to hand off decisions from humans such as sentencing judges or bank loan officers, to machines; and as the machine is nothing more than pattern-matching to its training data, this is really handing off to the dataset. Because juries have this authoritative role, Holmes treated their decisions with special caution; he placed the corrective elsewhere, namely in the hands of the court of appeal. His caution, and his approach, have relevance for the machine learning age.

In the formalist’s account, a jury settles factual questions; only the judge settles questions of law.¹ Holmes thought that juries exercise a wider influence than that.² “I don’t like to be told that I am usurping the functions of the jury if I venture to settle the standard of conduct myself in a plain case,” Holmes wrote in dryly humorous vein to his long-time friend Frederick Pollock. “Of course, I admit that any really difficult question of law is for the jury, but I also don’t like to hear it called a question of fact…”³ Whatever the proper way to describe the division of responsibilities between judge and jury, Holmes saw the latter to be a sort of conduit into the courtroom of the general understandings—and feelings—prevalent in the community. In its best aspect, the jury brought the collective experience of the community to questions that best are settled through the application of common sense. The jury, as an embodiment of community experience, is called upon when what matters is “the nature of the act, and the kind and degree of harm done, considered in the light of expediency and usage.”⁴ However, the jury is involved in several vexing problems. Analogous problems arise with machine learning.

7.1 Problems with Juries, Problems with Machines

Juries, wrote Holmes , “will introduce into their verdict a certain amount—a very large amount, so far as I have observed—of popular prejudice.”⁵ True, juries “thus keep the administration of law in accord with the wishes and feelings of the community.”⁶ Reading this “accord with” clause on its own, one might think it flattery. Reading in context, one sees that Holmes had no intention to flatter juries . Addressing why we have juries at all, Holmes added,

[S]uch a justification is a little like that which an eminent English barrister gave me many years ago for the distinction between barristers and solicitors. It was in substance that if law was to be practised somebody had to be damned, and he preferred that it should be somebody else.⁷

Holmes, in stating that juries produce results “in accord with the wishes and feelings of the community,” was stating a fact about the behavior of juries as he had observed many times juries to behave. He was not conferring a blessing upon them for behaving that way.

Three particular problems with juries suggested to Holmes a need for caution. In each a similar problem may be discerned with machine learning.

First, there is a problem of accountability. Holmes, in referring to the barrister who preferred that “somebody else” bear the blame, was referring to the practice of English advocates not to make a point about advocacy but to make a point about decision-making. If a judge is faced with an intractable question about which he would really prefer not to make a decision but must for purposes of deciding the case, he looks for a way to send that question to the jury. Some decisions are unlikely to please all parties concerned. Some decisions are likely, instead, to provoke criticism and resistance. If an authority, such as a judge , has the discretion to devolve such decisions upon another actor, then the temptation exists to do so, because if the other actor makes the decision then the authority removes himself from blame.

A temptation to devolve decisions exists with machine learning. The role of machine learning systems in actual decision practice grows apace. The growth is visible, or foreseeable, in banks,⁸ on highways,⁹ on the battlefield,¹⁰ in the courtroom.¹¹ Instead of a human being making the decision—such as a bank loan officer or a sentencing judge—the decision is given to a machine . A certain distance now separates the human being from the decision and its consequences. If “somebody had to be damned,” e.g. for denying home mortgages or giving long jail sentences on invidious criteria, then authorities and their institutions might well like to see that distance increase. Given enough distance, the machine makes the decision and is, perhaps, to be damned; but the human being holds up his hands and declares, Don’t look at me.

So far, decision making by machine has not relieved human beings of the duty, where the law imposes it, to give account for actions that they set in train. It has however started to raise questions of causation.¹² Those questions are likely to multiply if the separation continues to increase (as it probably will) between a human being setting a machine process in train and the practical impact of the machine decision . Proposals, such as that in the European Parliament in 2017 to confer legal personality on machines,¹³ would pave the way to even further separation (and for that reason, among others, are a bad idea).¹⁴ We find a timely message in Holmes’s caution toward the devolution of decisions to juries .

Second, there is propagation of “popular prejudice.” Just as the jury is faithful to the experience it brings to the court room and thus delivers a verdict that reflects the patterns in that experience, so will the machine be faithful to its training data. If the training data embody a pattern of community prejudice that we do not wish to follow, then we will not be pleased with the decision that the machine delivers.¹⁵ Holmes viewed juries as a mechanism to transmit community experience into legal decision. Affirming that they are reliable in that function is to concede a certain admiration, but it is just as much to sound a warning. No less reliable is the machine learning system in producing an output using the data that it is given. Both are mechanisms that rely on their givens—their experience and the data. Their outputs necessarily reflect the patterns they find therein.¹⁶

Finally, there is the problem of how to scrutinize black-box decision making. When policy-makers call for machine decision making to be scrutinized, they encounter a difficulty resembling that which arises when we try to figure out how a jury reached its decision: it is difficult to see inside the decision making mechanism. In Chapter 2,¹⁷ we addressed the challenges involved in explaining the outputs of black-box systems. Holmes noted the challenges in testing juries and similar bodies. In considering a decision by a state taxation tribunal, which because that tribunal was constituted of laypersons was like a jury in the relevant way, Holmes wrote, “how uncertain are the elements of the evidence, and in what unusual paths it moves”¹⁸; an appellate court must “mak[e] allowance for a certain vagueness of ideas to be expected in the lay mind.”¹⁹ Even the judge , thought Holmes, tended, “[w]here there is doubt…” to produce decisions derived from reasons that are “disguised and unconscious.”²⁰ Even the judge, schooled in the formalities of logic, produces outputs that are inscrutable. The jury is all the less likely to supply a clear trace of what led it to decide as it did. Here again is the phenomenon of occluded reasoning, the hidden layers of legal process. Explaining a machine learning output presents in a new setting a problem familiar since at least Holmes’s day in law.

7.2 What to Do About the Predictors?

Holmes, after he came to be cast in the role of progenitor of legal realism, was widely described as complacent about social problems, callous, even, in the face of injustice. Present-day observers have gone so far as to call Holmes “corrosive” because he seemed to accept the state of affairs as it was.²¹ In one judgment, which his critics have often cited, Holmes upheld a statute of Virginia under which the Commonwealth sterilized certain persons deemed “feebleminded.”²² Holmes supplied personal material, too, that later observers would use to characterize him as cold or resigned. Writing in 1927 in a letter to his friend Harold Laski, Holmes said “I do accept ‘a rough equation’ between isness and oughtness.”²³ To Morris Cohen, he wrote in 1921, “I do in a sense worship the inevitable.”²⁴ Writing around the same time to another correspondent, Holmes doubted that rational improvements would be made in the world at least in the immediate future:

We all try to make the kind of a world that we should like. What we like lies too deep for argument and can be changed only gradually, often through the experience of many generations.²⁵

One might read Holmes to counsel acceptance of whatever prediction the patterns in past experience suggest. And, yet, Holmes took account of the desire for change—the effort by all to “make the kind of a world that we should like.” He also showed personal interest in members of the younger generation who sought change; the correspondent to whom he wrote about the kind of a world that we should like was John C. H. Wu, a 22 year old law student at the time and much concerned with lifting China, his native land, out of its then century-long malaise.²⁶ The standard account—of Holmes as fatalist—is not supported by his record of encouraging people such as Wu in their efforts to escape the limits of experience.²⁷

Moreover, concerning experience and its influence on decision, Holmes did not think that all mechanisms for discerning patterns in experience merit equal deference. The legislature was the mechanism, in Holmes’s understanding, that a court was least to question; deference to the legislature was a precept that Holmes faithfully applied. It was that deference that is visible in his judgment in the Virginia sterilization case. One sees it as well in his dissents in Lochner and similar cases, where he decried the Court for second-guessing laws that had been enacted under proper legislative procedures and that aimed at various purposes that today would be called socially progressive.

In a number of appeals from jury verdicts, Holmes had a very different response than he had to challenges against statutes. He overturned jury verdicts or dissented against majorities that didn’t. Moore v. Dempsey is the most prominent of the cases in which Holmes considered how a jury, perfectly “in accord with the wishes and feelings of the community,” produced a result that demanded correction. Five defendants, all African American, had been arrested. The grand jury that returned indictments against them had been comprised of whites only, including the members of an attempted lynch mob. The evidence against them was scarce; their defense lawyers, appointed by the court, had done nothing in their defense; and the “Court and the neighborhood were thronged with an adverse crowd that threatened the most dangerous consequences to anyone interfering with the desired result.”²⁸ The jury at trial found the defendants guilty; the sentence was death. In Holmes’s words, “counsel, jury and judge were swept to the fatal end by an irresistible wave of public passion.”²⁹ True, the situation in Moore v. Dempsey had been that the facts “if true as alleged… [made] the trial absolutely void.”³⁰ The body assembled to function as a jury had not functioned as a jury at all. At the same time, the transactions in the courthouse exemplified, in extremis, the jury as decision-maker. Extreme example though it was, the case shone a light on how the jury works, and juries work that way even when all persons involved have acted in good faith: juries find their patterns in the experience around them. A corrective is called for, when that experience discords with our better understanding of the world we wish to have.

As the data that trains the machine learning system is a given, both in Latin grammar and in the process of machine learning, so too is the experience that Holmes understood to be the main influence on law. Holmes’s personal outlook was congenial to accepting givens. However, Holmes was alert to the danger that givens present for certain kinds of decision-making that rely upon them as their inputs . He didn’t suggest that the jury could do any differently. He placed the corrective someplace else, namely in the hands of the court of appeal. We will turn in the next chapter to consider more closely how Holmes understood the corrective to work in a particular situation—that where public authorities had garnered evidence in breach of the constitutional right of a defendant—and how a practice we will call inferential restraint has been necessary in legal procedures and is likely to be in machine learning as well.

Notes

• 1. For a classic account, see Leonard (ed.), The New Wigmore (2010) § 1.2 pp. 3–5.

• 2. Holmes had experience with juries . He judged many appeals from trials in which they were empaneled. Moreover, for most of the time when Holmes served on the Massachusetts Supreme Judicial Court, its members also sat as trial judges (in divorces, murders, certain contractual disputes, contests over wills and trusts, suits in equity). Budiansky 183–84. Further to Holmes’s jury trials, see Zobel, 8 Massachusetts Legal History 35–47 (2002); 36 Boston Bar J. 25–28 (1992).

• 3. Letter of Holmes to Pollock (May 13, 1898): reprinted De Wolfe Howe (ed.) (1942) 85–86 (ellipses original). Twenty-first century writers also have doubted the classic distinction: see, e.g., Allen & Pardo, The Myth of the Law-Fact Distinction, 97 Nw. U. L. Rev. 1769–1807 (2003). See also Zuckerman, Law, Fact or Justice? 66 Boston U. L. Rev. 487 (1986).

• 4. Middlesex v. McCue, 149 Mass. 103; 21 N.E. 230 (1889). Cf. Commonwealth v. Perry, 139 Mass. 198, 29 N.E. 356 (1885).

• 5. Law in Science and Science in Law , 12 Harv. L. Rev. at 460.

• 6. Id.

• 7. Id. In England and Wales and in other countries of the Commonwealth, the practice of law is divided between solicitors and barristers. Broadly speaking, the division is between lawyers who don’t argue in court (solicitors) and those who do (barristers). The exclusivity of the barristers’ rights of appearance in court that prevailed in Holmes’s day has been qualified since, but the basic division remains the same. For an account of the relationship by a UK law firm in 2016, see https://www.slatergordon.co.uk/media-centre/blog/2016/09/difference-between-a-lawyer-a-solicitor-and-a-barrister-explained/.

• 8. Bartlett, Morse, Stanton & Wallace, Consumer-Lending Discrimination in the Era of FinTech (Oct. 2018). http://faculty.haas.berkeley.edu/morse/research/papers/discrim.pdf. Cf., considering a different problem of accountability, Ji, Are Robots Good Fiduciaries: Regulating Robo-Advisors Under the Investment Advisers Act of 1940, 117 Col. L. Rev. 1543 (2017).

• 9. As to a possible tort law solution, see Geistfeld, A Roadmap for Autonomous Vehicles: State Tort Liability, Automobile Insurance, and Federal Safety Regulation, 105 Cal. L. Rev. 1611 (2017). Cf. Kowert, Note: The Foreseeability of Human-Artificial Intelligence Interactions, 96 Tex. L. Rev. 181 (2017).

• 10. See Ford, AI, Human-Machine Interaction, and Autonomous Weapons: Thinking Carefully About Taking ‘Killer Robots’ Seriously, Arms Control and International Security Papers, 1(2) (April 20, 2020). Cf. Feickert et al., U.S. Ground Forces Robotics and Autonomous Systems (RAS) and Artificial Intelligence (AI) (Congressional Research Service Reports, Nov. 1, 2018).

• 11. See for example Roth, Trial by Machine, 104 Geo. L. J. 1243 (2016). Cf. Rolnick Borchetta, Curbing Collateral Punishment in the Big Data Age: How Lawyers and Advocates Can Use Criminal Record Sealing Statutes to Protect Privacy and the Presumption of Innocence, 98 B.U. L. Rev. 915 (2018).

• 12. As to causation and challenges in dealing with legal liability for machine outputs , see Chapter 8, p. 97 n. 12.

• 13. Civil Law Rules on Robotics, European Parliament Resolution (Feb. 16, 2017).

• 14. For objections to proposals to confer legal personality on machine learning systems and the like, see Bryson, Diamantis & Grant (2017). See also Brożek & Jakubiec (2017).

• 15. A point made by Kroll et al., supra n. 76 at 687.

• 16. Considering this similarity, one might be cautious about using a machine learning system to “debias” a jury. We are not talking here about using machines such as polygraphs as supplements to decision-making by a jury; we express no view here about such machines. As to “debiasing juries” with machines, see Roth op. cit. at 1292–94.

• 17. See Chapter 2, pp. 24–27.

• 18. Coulter et al. v. Louisville & Nashville Railroad Company, 196 U.S. 599, 609 25 S.Ct. 342, 344 (Holmes, J.) (1905).

• 19. 196 U.S. at 610, 25 S.Ct. at 345.

• 20. 12 Harv. L. Rev. at 461.

• 21. Jackson, op. cit. 130 Harv. L. Rev. at 2368 (2017).

• 22. Buck v. Bell, 274 U.S. 200 (Holmes, J., 1927). As to the circumstances of the case and the drafting of Holmes’s judgment see Budiansky 428–430.

• 23. Letter from Holmes to Laski (June 1, 1927), reprinted in 2 Holmes-Laski Letters 948 (Mark De Wolfe Howe ed., 1953).

• 24. Letter from Holmes to Morris Cohen (Jan. 30, 1921), in Felix S. Cohen, The Holmes-Cohen Correspondence (1948) 9 J. Hist. Ideas 3, 27.

• 25. Letter from Holmes to Wu, December 12, 1921, reprinted in Justice Holmes to Doctor Wu: An Intimate Correspondence, 1921–1932 (1947) 2–3.

• 26. Wu, who was studying at the University of Michigan and approached the 80-year-old Associate Justice evidently on self-introduction, maintained a long correspondence with Holmes. Wu went on to have a distinguished career as academic, lawyer, and diplomat.

• 27. There were others noted for their activism and whom Holmes nevertheless admired and had long friendships with. Felix Frankfurter and Louis Brandeis (who both in time served as Associate Justices of the U.S. Supreme Court, Brandeis overlapping with Holmes’s tenure, Frankfurter taking a seat later) were prominent examples. See Budiansky 328–29, 387–90; 356–59, 362–63.

• 28. Moore v. Dempsey, 261 U.S. 86, 89 (Holmes, J., 1923).

• 29. 261 U.S. at 91.

• 30. 261 U.S. at 92.