Skip to main content
SpringerLink
Log in

Statistical Issues in Assessing the Reliability of Eyewitness Identification

  • Chapter
  • First Online:
Statistics in the Public Interest

Part of the book series: Springer Series in the Data Sciences ((SSDS))

  • 791 Accesses

Abstract

Among the 375+ wrongful convictions identified by the Innocence Project that were later overturned by DNA evidence resurrected from the crime scene, nearly ∼70% involved eyewitness testimony (www.innocenceproject.org). Courtroom identifications from an eyewitness can be tremendously powerful evidence in a trial. Yet memory is not a perfect video recording of events, and one’s recollection of the events surrounding an incident is even less reliable. In October 2014, the National Academy of Sciences (NAS) issued a landmark report evaluating the scientific research on memory and eyewitness identification. The Committee of researchers (psychologists, statisticians, sociologists) and judicial representatives (judges, attorneys) reviewed the published research on the factors that influence the accuracy and consistency of eyewitnesses’ identifications, conducted via laboratory and field studies. Steve Fienberg chaired the Committee that reviewed and ultimately authorized the release of that report. Unbeknownst to many, however, Steve himself played an important role behind the scenes by providing advice to the Committee that greatly strengthened the report. In this chapter, I will describe some of the research on memory and recollection relevant to eyewitness identification, the shortcomings in the statistical methods that had been used in evaluating laboratory studies, and Committee’s recommendations for standardizing procedures and further research using statistically designed experiments. The astounding insights that Steve provided to the Committee illustrate another one of many instances where Steve left his mark and made the world a better place.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 109.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 139.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 139.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    innocenceproject.org, accessed 1 September 2021.

  2. 2.

    For the heartbreaking details of this case, see www.thestory.org/stories/2013-06/jennifer-thompson.

  3. 3.

    See http://theconversation.com/police-photo-lineups-how-background-colours-can-skew-eye-witness-identification-116329 for an example of a simultaneous lineup. The article notes the challenges in constructing a fair lineup, which include the potential bias from different background colors for all photographs and the instructions given to the eyewitness.

  4. 4.

    The actual counts in Table 2 in Brewer and Wells [3] are: 299 (high similarity, target present); 300 (low similarity, target present); 301 (high similarity, target absent); 300 (low similarity, target absent). In each of these four conditions, ∼150 participants viewed a “biased” lineup and the other half viewed an “unbiased” (sometimes called “fair”) lineup.

  5. 5.

    Notice that Wang and Gatsonis use AUC, versus pAUC = partial area under the curve; see discussions about AUC versus pAUC in the articles by Pepe [33, p.311] and Walter [40].

References

  1. Benjamini, Yoav; Hochberg, Yusef (1995): Controlling the false discovery rate: A practical and powerful approach to multiple testing, Journal of the Royal Statistical Society, Series B 57: 289–300.

    MathSciNet  MATH  Google Scholar 

  2. Brown, Christopher D.; Davis, Herbert T. (2006), Receiver operating characterics curves and related decision measures: A Tutorial, Chemometrics and Intelligent Laboratory Systems 80: 24–38.

    Article  Google Scholar 

  3. Brewer, Neil; Wells, Gary L. (2006). The Confidence-Accuracy Relationship in Eyewitness Identification: Effects of Lineup Instructions, Foil Similarity, and Target-Absent Base Rates,” Journal of Experimental Psychology: Applied 12(1): 11–30.

    Google Scholar 

  4. Carlson C.A.; Gronlund S.D.; Clark S.E. (2008), Lineup composition, suspect position, and the sequential lineup advantage, Journal of Experimental Psychology-Applied 14(2): 118–128.

    Article  Google Scholar 

  5. Carlson, C.A.; (2011). Influence of a perpetrator’s distinctive facial feature on eyewitness identification from simultaneous versus sequential lineups, Applied Psychology in Criminal Justice 7:77–92.

    Google Scholar 

  6. Carlson, Curt A.; Carlson, Maria A. (2014). An evaluation of lineup presentation, weapon presence, and a distinctive feature using ROC analysis, Journal of Applied Research in Memory and Cognition 3(2): 45–53.

    Article  Google Scholar 

  7. Chu, Haitao; Cole, Stephen R. (2006), Bivariate meta-analysis of sensitivity and specificity with sparse data: a generalized linear mixed model approach, Journal of Clinical Epidemiology 59:1331–1333.

    Article  Google Scholar 

  8. Clark, Steven E. (2005), A re-examination of the effects of biased lineup instructions in eyewitness identification, Law and Human Behavior 29(4): 395–424.

    Article  Google Scholar 

  9. Clark, Steven E.; Howell, Ryan T.; Davey, Sherrie L. (2008), Regularities in eyewitness identification. Law and Human Behavior, 32(3), 187–218. https://doi.org/10.1007/s10979-006-9082-4

    Article  Google Scholar 

  10. Deffenbacher K.A.; Bornstein, B.H.; McGorty, E.K.; Penrod, S.D. (2008), Forgetting the once-seen face: estimating the strength of an eyewitness’s memory representation, Journal of Experimental Psychology: Applied 14(2): 139–150.

    Google Scholar 

  11. Dodson, Chad S.; Dobolyi, David G. (2016), Confidence and eyewitness identifications: the cross-race effect, decision time, and accuracy. Applied Cognitive Psychology, 30:113–125.

    Article  Google Scholar 

  12. Dodson, Chad S.; Garrett, Brandon L.; Kafadar, Karen; Yaffe, Joanne (2020), Eyewitness Identification Speed: Slow identifications from highly confident eyewitnesses hurt perceptions of their testimony, Journal of Applied Research in Memory and Cognition 10:259–267.

    Article  Google Scholar 

  13. DuMouchel, William (1994), Hierarchical Bayes linear models for meta-analysis. Technical Report 27, National Institute of Statistical Sciences, 19 T.W. Alexander Drive, PO Box 14006, Research Triangle Park, NC 27709, September 1994.

    Google Scholar 

  14. Fawcett, J.M.; Russell, E.J.; Peace, K.A.; Christie, J. (2013), Of guns and geese: A meta-analytic review of the ‘weapon focus’ literature, Psychology, Crime & Law 19:35–66. https://doi.org/10.1080/1068316X.2011.599325

    Article  Google Scholar 

  15. Garrett, Brandon L.; Liu, Alice J.; Kafadar, Karen; Yaffe, Joanne; Dodson, Chad S. (2020): Factoring the Role of Eyewitness Evidence in the Courtroom. Journal of Empirical Legal Studies 17(3): 556–579.

    Article  Google Scholar 

  16. GEP Box, WG Hunter, JS Hunter (2005), Statistics for Experimenters: Design, Innovation, and Discovery, 2nd ed., Wiley, New York.

    MATH  Google Scholar 

  17. Hastie, Trevor; Friedman, Jerome H.; Tibshirani, Robert (2013), The Elements of Statistical Machine Learning, 2nd edition, Springer: New York.

    Google Scholar 

  18. Junaidi, Darfiana Nur; Stojanovski, Elizabeth (2012), Bayesian estimation of a meta-analysis model using Gibbs sampler. In Proceedings of the Fifth Annual ASEARC Conference - Looking to the future - Programme and Proceedings.

    Google Scholar 

  19. Juslin, Peter; Olsson, Nils; Winman, Anders (1996), Calibration and diagnosticity of confidence in eyewitness identification: comments on what can be inferred from low confidence-accuracy correlation. Journal of Experimental Psychology, 22(5):1304–1316.

    Google Scholar 

  20. Kensinger, E.A.; Schacter, D.L. (2008), Neural processes supporting young and older adults’ emotional memories, Journal of Cognitive Neuroscience 20(7):1161–1173.

    Article  Google Scholar 

  21. Lasko, Thomas A.; Bhagwat, Jui G.; Zou, Kelly H; Ohno-Machado, Lucila (2005), The use of receiver operating characteristic curves in biomedical informatics, Journal of Biomedical Informatics 38(5), 404–415.

    Article  Google Scholar 

  22. Lee, Peter (2012), Bayesian Statistics: An Introduction, 4th ed., Wiley, New York.

    Google Scholar 

  23. Lindsay, R.C.L., Wells, Gary L. (1985). Improving eyewitness identifications from lineups: Simultaneous versus sequential lineup presentation. Journal of Applied Psychology, 70(3), 556–564. https://doi.org/10.1037/0021-9010.70.3.556

    Article  Google Scholar 

  24. Lindsay, R.C.L. (1999), Applying applied research: Selling the sequential lineup, Applied Cognitive Psychology, 13, 219–225.

    Article  Google Scholar 

  25. Liu, Alice; Kafadar, Karen; Garrett, Brandon; Yaffe, Joanne (2020), Bringing new statistical approaches to eyewitness evidence. Chapter 21 in Handbook of Statistics in Forensic Science (Eds. David L. Banks, Karen Kafadar, David H. Kaye, Maria Tackett), CRC Press.

    Google Scholar 

  26. Liu, Alice Jia (2020). Errors-In-Variables and Random Forests: Theory and Application to Eyewitness Identification Data. Ph.D. Disseration, University of Virginia, Charlottesville, Virginia

    Google Scholar 

  27. Luby, Amanda S. (2017), Strengthening analyses of line-up procedures: a log-linear model framework, Law, Probability and Risk 16:241–257.

    Article  Google Scholar 

  28. Meissner, C.A.; Brigham, J.C. (2001), Thirty years of investigating the own-race bias in memory for faces: A meta-analytic review, Psychology, Public Policy, and Law 7(1): 3–35, https://doi.org/10.1037/1076-8971.7.1.3

    Article  Google Scholar 

  29. Meissner, C.A.; Tredoux, C.G.; Parker, J.F.; MacLin, O.H. (2005), Eyewitness decisions in simultaneous and sequential lineups: A dual process signal detection theory analysis, Memory & Cognition 33, 783–792.

    Article  Google Scholar 

  30. Mickes, Laura; Flowe, Heather D.; Wixted, John T. (2012), Receiver operating characteristic analysis of eyewitness memory: Comparing the diagnostic accuracy of simultaneous and sequential lineups, Journal of Experimental Psychology: Applied, 18(4): 361–376.

    Google Scholar 

  31. National Research Council (NRC 2014): Identifying the Culprit: Assessing Eyewitness Evidence, National Academies Press (https://doi.org/10.17226/18891).

  32. Palmer, M.A., Brewer, N. (2012), Sequential lineup presentation promotes less-biased criterion setting but does not improve discriminability. Law and Human Behavior, 36, 247–255.

    Article  Google Scholar 

  33. Pepe, Margaret Sullivan (2000), Receiver Operator Characteristic Methodology, Journal of the American Statistical Association 95(449), 308–311.

    Article  Google Scholar 

  34. Reitsma, Johannes B.; Glas, Afina S.; Rutjes, Anne W.S.; Scholten, Rob J.P.M.; Bossuyt, Patrick M.; Zwinderman, Aeilko H. (2005), Bivariate analysis of sensitivity and specificity produces informative summary measures in diagnostic reviews. Journal of Clinical Epidemiology, 58:982–990, 2005.

    Google Scholar 

  35. Rhodes, M.G.; Anastasi, J.S. (2012), The Own-Age Bias in Face Recognition: A Meta-Analytic and Theoretical Review,” Psychological Bulletin 138(1): 146–174.

    Article  Google Scholar 

  36. Roediger, H.L.; Wixted; J.H.; DeSoto, K.A. (2012), The curious complexity between confidence and accuracy in reports from memory, in: L. Nadel & W. Sinnot-Armstrong (Eds.), Memory and Law, Oxford, Oxford University Press, 84–118.

    Chapter  Google Scholar 

  37. Shiu, Shang-Ying; Gatsonis, Constantine A. (2008), The predictive receiver operating characteristic curve for the joint assessment of the positive and negative predictive values, Philosophical Transactions of the Royal Society A 366:2313–2333.

    Article  MathSciNet  Google Scholar 

  38. Steblay, Nancy K.; Dysart, Jennifer; Fulero, Solomon; Lindsay, R.C.L. (2001). Eyewitness accuracy rates in sequential and simultaneous lineup presentations: A meta-analytic comparison, Law and Human Behavior 25(5): 459–473

    Article  Google Scholar 

  39. Steblay, Nancy K.; Dysart, Jennifer; Wells, Gary L. (2011). Seventy-two Tests of the Sequential Lineup Superiority Effect: A Meta-Analysis and Policy Discussion, Psychology Public Policy and Law 17(1): 99–139,

    Article  Google Scholar 

  40. Walter, S.D. (2005), The partial area under the summary ROC curve, Statistics in Medicine 24(13), 2025–2040.

    Article  MathSciNet  Google Scholar 

  41. Wells, Gary L.; Rydell, S.M.; Seelan, E.P. (1993), The selection of distractors for eyewitness lineups, Journal of Applied Psychology, 78, 835–844.

    Article  Google Scholar 

  42. Wang, Pei; Gatsonis, Constantine (2008), Hierarchical models for ROC curve summary measures: Design and analysis of multi-reader, multi-modality studies of medical tests, Statistics in Medicine 27:243–256 (https://doi.org/10.1002/sim.2828).

  43. Wells, Gary L.; Steblay, Nancy K.; Dysart, Jennifer E. (2015), Double-Blind Photo-Lineups Using Actual Eyewitnesses: An Experimental Test of a Sequential versus Simultaneous Lineup Procedure, Law and Human Behavior 39(1), 1–14. https://doi.org/10.1037/lhb0000096.

    Article  Google Scholar 

  44. Wells, Gary L. (2014) Eyewitness identification: Probative value, criterion shifts, and policy regarding the sequential lineup, Current Directions in Psychological Science, 23:11–16.

    Article  Google Scholar 

  45. Wells, Gary L.; Kovera, Margaret B.; Douglass, Amy B.; Brewer, Neil, Meissner, Christian A.; Wixted, John T. (2020), Policy and procedure recommendations for the collection and preservation of eyewitness identification evidence, Law and Human Behavior, 44(1), 3–36. https://doi.org/10.1037/lhb0000359

    Article  Google Scholar 

Download references

Acknowledgements

An early version of this paper was presented at a workshop during the Probability and Statistics in Forensic Science program at the Isaac Newton Institute for Mathematical Sciences supported by EPSRC Grant Number EP/K032208/1. The author thanks the Institute for its hospitality and support during the program. The author also gratefully acknowledges her co-investigators on a grant from Arnold Ventures (Chad Dodson, Brandon Garrett, Joanne Yaffe). This chapter was prepared in part with support from this grant. The views expressed herein are solely those of the author and do not necessarily represent the views of Arnold Ventures or the Isaac Newton Institute.

Author information

Authors and Affiliations

  1. Department of Statistics, University of Virginia, Charlottesville, VA, USA

    Karen Kafadar

Authors
  1. Karen Kafadar
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Karen Kafadar .

Editor information

Editors and Affiliations

  1. Department of Statistics, Iowa State University, Ames, IA, USA

    Alicia L. Carriquiry

  2. Department of Sociology, Stony Brook University, Stony Brook, NY, USA

    Judith M. Tanur

  3. Statistics Department, Carnegie Mellon University, Pittsburgh, PA, USA

    William F. Eddy

Rights and permissions

Reprints and permissions

Copyright information

© 2022 Springer Nature Switzerland AG

About this chapter

Check for updates. Verify currency and authenticity via CrossMark

Cite this chapter

Kafadar, K. (2022). Statistical Issues in Assessing the Reliability of Eyewitness Identification. In: Carriquiry, A.L., Tanur, J.M., Eddy, W.F. (eds) Statistics in the Public Interest. Springer Series in the Data Sciences. Springer, Cham. https://doi.org/10.1007/978-3-030-75460-0_11

Download citation

Publish with us

Policies and ethics

Search

Navigation