Ethics and Information Technology

, Volume 12, Issue 1, pp 29–42 | Cite as

Engaging rational discrimination: exploring reasons for placing regulatory constraints on decision support systems

  • Oscar H. GandyJr.Email author
Original Paper


In the future systems of ambient intelligence will include decision support systems that will automate the process of discrimination among people that seek entry into environments and to engage in search of the opportunities that are available there. This article argues that these systems must be subject to active and continuous assessment and regulation because of the ways in which they are likely to contribute to economic and social inequality. This regulatory constraint must involve limitations on the collection and use of information about individuals and groups. The article explores a variety of rationales or justifications for establishing these limits. It emphasizes the unintended consequences that flow from the use of these systems as the most compelling rationale.


Race Discrimination Surveillance Ambient intelligence Privacy Insurance Technology assessment 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. ACORN Fair Housing. (2007). Foreclosure exposure: A study of racial and income disparities in home mortgage lending in 172 American cities. Chicago: ACORN Housing Corporation.Google Scholar
  2. Altman, E., & Saunders, A. (1998). Credit risk measurement: Developments over the last 20 years. Journal of Banking and Finance, 21:1721–1742. See also, Mester, L. (1997). What’s the point of credit scoring? Business Review, October/November, pp. 3–16.Google Scholar
  3. Amegashie, J. A. (2008). Socially-tolerable discrimination. MPRA paper no. 8543, Munich Personal RePEc Archive.
  4. Armour, J. (1997). Negrophobia and reasonable racism (pp. 35–67). New York: New York University Press.Google Scholar
  5. Ayres, I. (2007). Market power and inequality: A competitive conduct standard for assessing when disparate impacts are unjustified. California Law Review, 95, 669–719.Google Scholar
  6. Ayres, I., Vars, F., & Zakariya, N. (2005). To insure prejudice: Racial disparities in taxicab tipping. Yale Law Journal, 114, 1640.Google Scholar
  7. Baker, C. E. (2002). Media, markets, and democracy. New York: Cambridge University Press.Google Scholar
  8. Balkin, J. M. (1997). The constitution of status. Yale Law Journal, 106, 2366.CrossRefGoogle Scholar
  9. Barnes, P. (2006). Capitalism 3.0: A guide to reclaiming the commons. San Francisco: Berrett-Koehler.Google Scholar
  10. Baumgartner, F., & Jones, B. (1993). Agendas and instability in American politics. Chicago: University of Chicago Press.Google Scholar
  11. Becker, G. (1971). The economics of discrimination (2nd ed.). Chicago: University of Chicago Press.Google Scholar
  12. Belavkin, R. (2001). The role of emotion in problem solving. In C.␣Johnson (ed), Proceedings of the AISB ‹01 Symposium on emotion, cognition and affective computing (pp. 49–57). York: Heslington. See also, Canamero, D. (2001). Modeling motivations and emotions as a basis for intelligent behavior. Agents ‹97, ACM, 1997, and Aaron Sloman. Beyond shallow models of emotion. Cognitive Processing, 2:177–198.Google Scholar
  13. Birnbaum, B. (2003). Insurers’ use of credit scoring for homeowners insurance in Ohio. Columbus: Ohio Civil Rights Commission.Google Scholar
  14. Bowker, G. C., & Star, S. L. (1999). Sorting things out: Classification and its consequences. Cambridge: The MIT Press.Google Scholar
  15. Brey, P. (2005). Freedom and privacy in ambient intelligence. Ethics and Information Technology, 7, 157–166.CrossRefGoogle Scholar
  16. Brockett, P. L., & Golden, L. L. (2007). Biological and psychobehavioral correlates of credit scores and automobile insurance losses: Toward an explication of why credit scoring works. The Journal of Risk and Insurance, 74, 23–63.CrossRefGoogle Scholar
  17. Camerer, C., & Fehr, E. (2006). When does “Economic Man” dominate social behavior? Science, 311, 47–52.CrossRefGoogle Scholar
  18. Carusi, A. (2008). Data as representation: Beyond anonymity in e-research ethics. International Journal of Internet Research Ethics, 1, 37–65.Google Scholar
  19. Case, M. A. (2002–2003). Developing a taste for not being discriminated against. Stanford Law Review, 55:2273–2291.Google Scholar
  20. Chandler, S. (2002). Visualizing adverse selection: An economic approach to the law of insurance underwriting. Connecticut Insurance Law Journal, 8, 435–503.Google Scholar
  21. Charles, C. Z. (2003). The dynamics of racial residential segregation. Annual Review of Sociology, 29, 167–207.CrossRefGoogle Scholar
  22. Chin, G. J. (2002). Race, the war on drugs, and the collateral consequences of criminal conviction. The Journal of Gender, Race & Justice, 6(25), 3–275.MathSciNetGoogle Scholar
  23. Committee on Networked Systems of Embedded Computers. (2001). Embedded, everywhere: A research agenda for networked systems of embedded computers. Washington: National Research Council, National Academy Press.Google Scholar
  24. Dahrendorf, R. (1979). Life chances: Approaches to social and political theory. Chicago: University of Chicago Press.Google Scholar
  25. DiPrete, T., & Eirich, G. (2006). Cumulative advantage as a mechanism for inequality: A review of theoretical and empirical developments. Annual Review of Sociology, 32, 271–297.CrossRefGoogle Scholar
  26. Elster, J. (1990). When rationality fails. In K. S. Cook & M. Levi (Eds.), The limits of rationality (pp. 19–51). Chicago: University of Chicago Press.Google Scholar
  27. Florescu, D., Koller, D., & Levy, A. (1997). Using probabilistic information in data integration. Proceedings of the 23rd VLDB Conference. Athens: Greece, pp. 216–225.Google Scholar
  28. Friedewald, M., Lindner, R., & Wright, D. (eds.), (2006). Policy options to counteract threats and vulnerabilities in ambient intelligence, SWAMI. Deliverable D3: A report of the SWAMI consortium to the European Commission under contract 006507.
  29. Gandy, O. H., Jr. (1982). Beyond agenda setting: Information subsidies and public policy. Norwood: Ablex. See also, Gandy, O. H., Jr. (2003). Public opinion surveys and the formation of privacy policy. Journal of Social Issues, 59:283–299.Google Scholar
  30. Gandy, O. (1993). The panoptic sort: A political economy of personal information. Boulder: Westview Press.Google Scholar
  31. Gandy, O. (1995). It’s discrimination stupid. In J. Brook & I. Boal (Eds.), Resisting the virtual life: The culture and politics of information (pp. 35–47). San Francisco: City Lights Books.Google Scholar
  32. Gandy, O. (1996). Legitimate business interest: No end in sight? An inquiry into the status of privacy in cyberspace (pp. 77–137). Chicago: University of Chicago Legal Forum.Google Scholar
  33. Gandy, O. (2006a). Quixotics unite! Engaging the pragmatists on rational discrimination. In D. Lyon (Ed.), Theorizing surveillance: The Panopticon and beyond (pp. 318–336). Portland: Willan Publishing.Google Scholar
  34. Gandy, O. H., Jr. (2006b). Data mining, surveillance, and discrimination in the post-9/11 environment. In K. Haggerty & R. Ericson (Eds.), The new politics of surveillance and visibility (pp. 363–384). Toronto: University of Toronto Press.Google Scholar
  35. Gandy, O., & Baruh, L. (2006). Racial profiling: They said it was against the law!. University of Ottawa Law & Technology Journal, 3, 297–327.Google Scholar
  36. Giddens, A. (1984). The constitution of society (pp. 5–14). Cambridge: Polity Press.Google Scholar
  37. Graham, S. (2005). Software-sorted geographies. Progress in Human Geography, 29, 1–19.CrossRefGoogle Scholar
  38. Haddawy, P. (1999). An overview of some recent developments in␣Bayesian problem solving techniques. The AI Magazine, 20, 11–19.Google Scholar
  39. Hahn, R., & Tetlock, P. C. (2007). Has economic analysis improved regulatory decisions? Working paper 07–08. Washington: AEI-Brookings Joint Center for Regulatory Studies.Google Scholar
  40. Harcourt, B. (2007). Against prediction: Profiling, policing, and punishment in an actuarial age. Chicago: University of Chicago Press.Google Scholar
  41. Harmon, A. (2008). Congress passes bill to bar bias based on genes. In New York Times Online,
  42. Hausman, D., & McPherson, M. (1996). Economic analysis and moral philosophy. New York: Cambridge University Press.Google Scholar
  43. Hildebrandt, M. (2006). Profiling: From data to knowledge. Datenschutz und Datensicherheit, 30, 548–552.CrossRefGoogle Scholar
  44. Hildebrandt, M., & Meints, M. (eds). (2006). D7.7: RFID, Profiling, and AmI. Future of identity in the information society (FDIS Consortium).Google Scholar
  45. Hildebrandt, M., Gutwirth, S., & De Hert, P. (2005). Implications of profiling practices on democracy and the rule of law. Deliverable D7.4, FIDIS Consortium.Google Scholar
  46. Holland, P. W. (2008). Causation and race. In T. Zuberi & E. Bonita-Silva (Eds.), White logic, White methods (pp. 93–109). Lanham: Rowman & Littlefield.Google Scholar
  47. Jost, J., Nosek, B., & Gosling, S. (2008). “Ideology: Its resurgence in social, personality, and political psychology. Perspectives on Psychological Science, 3(2), 126–136.CrossRefGoogle Scholar
  48. Kahn, S., & Lambert, A. (2001). Perceptions of rational discrimination: When do people attempt to justify race-based prejudice? Basic and Applied Social Psychology, 23, 43–53.Google Scholar
  49. Kang, J., & Banaji, M. (2006). Fair measures: A behavioral realist revision of affirmative action. California Law Review, 94, 1063–1118.CrossRefGoogle Scholar
  50. Klasen, S. (2006). The efficiency of equity. Discussion paper no. 145, Goettingen: Ibero-America Institute for Economic Research.Google Scholar
  51. Kleindorfer, P., Kunreuther, H., & Shoemaker, P. (1993). Decision sciences: An integrative perspective (pp. 86–104). New York: Cambridge University Press.Google Scholar
  52. Knoch, D., Pascual-Leone, A., Meyer, K., Treyer, V., & Fehr, E. (2006). Diminishing reciprocal fairness by disrupting the right prefrontal cortex. Science, 314, 829–832.CrossRefGoogle Scholar
  53. Koomey, J., et al. (2002). Sorry wrong number: The use and misuse of numerical facts in analysis and media reporting of energy issues. Annual Review of Energy and the Environment, 27:119–158. See also, Best, J. (2001). Damned lies and statistics: Untangling numbers from the media, politicians, and activists. Berkeley: University of California Press.Google Scholar
  54. Kurzweil, R. (2005). The singularity is near: When humans transcend biology. New York: Viking.Google Scholar
  55. Lafuente-Rojo, A., Abascal-Gonzalez, J., & Cai, Y. (2007). Ambient intelligence: Chronicle of an announced technological revolution. UPGRADE, 8, 8–12.Google Scholar
  56. Lakoff, G. (2002). Moral politics: How liberals and conservatives think. Chicago: University of Chicago Press. See also, Wilson, M. S. (2004). Values and political ideology: Rokeach’s two-value model in a proportional representation environment. New Zealand Journal of Psychology 23.Google Scholar
  57. Landes, W. M., & Posner, R. A. (2004). The political economy of intellectual property law. Washington: The AEI Press.Google Scholar
  58. Le Grand, J. (1990). Equity versus efficiency: The elusive trade-off. Ethics, 100, 554–568.CrossRefGoogle Scholar
  59. Lejano, R. P. (2008). Technology and institutions: A critical appraisal of GIS in the planning domain. Science, Technology and Human Values, 33, 653–678.CrossRefGoogle Scholar
  60. Lemmens, T., & Thiery, Y. (2007). Insurance and human rights: What can Europe learn from Canadian anti-discrimination law? In H. Cousy & C. V. Schoubroeck (Eds.), Discrimination in insurance. Maklu: Academia-Bruylant.Google Scholar
  61. Link, B., & Phelan, J. C. (2001). Conceptualizing stigma. Annual Review of Sociology, 27, 363–385.CrossRefGoogle Scholar
  62. Medina, H. (2004). Consumer protection: Federal and state agencies face challenges in combating predatory lending. Washington: United States General Accounting Agency. See also, Engel, K., & McCoy, P. (2002). A tale of three markets: The law and economics of predatory lending. Texas Law Review, 80:1255–1367.Google Scholar
  63. Moll, J., Oliveira-Souza, R., & Zahn, R. (2008). The neural basis of moral cognition: Sentiments, concepts, and values. Annals of the New York Academy of Sciences, 1124, 161–180.CrossRefGoogle Scholar
  64. Norman, P. (2003). Statistical discrimination and efficiency. The Review of Economic Studies, 70, 615–627.zbMATHCrossRefMathSciNetGoogle Scholar
  65. Nosek, B., Greenwald, A., & Banaji, M. (2006). The implicit association test at age 7: A methodological and conceptual review. In J. A. Bargh (Ed.), Social psychology and the unconscious: The automaticity of higher mental processes (pp.␣265–292). London: Psychology Press.Google Scholar
  66. Okun, A. (1975). Equality and efficiency: The big tradeoff. Washington: The Brookings Institution.Google Scholar
  67. Patrick, G. (2007). The locus opus: Playing with privacy in a world of ambient intelligence. Manuscript online at
  68. Pirttila, J., & Uusitalo, R. (2007). Leaky bucket in the real world: Estimating inequality aversion using survey data. Paper presented to the Conference on Public Sector Economics. Munich.Google Scholar
  69. Quillian, L. (2006). New approaches to understanding racial prejudice and discrimination. Annual Review of Sociology, 32, 299–328.CrossRefGoogle Scholar
  70. Ramirez, S. A. (2004). Bearing the costs of racial inequality: Brown and the myth of the equality/efficiency trade-off. Washburn Law Journal, 44, 87–104.Google Scholar
  71. Rawls, A. J. (1971). Theory of justice. Cambridge: Harvard University Press.Google Scholar
  72. Regan, M. C., Jr. (2007). Moral intuitions and organizational culture. Saint Louis University Law Journal, 51, 941–988.Google Scholar
  73. Rokeach, M. (1973). The nature of human values. New York: Free Press.Google Scholar
  74. Rouvroy, A. (2008). Privacy, data protection, and the unprecedented challenges of ambient intelligence. Studies in Ethics, Law, and Technology, 2, 1–51.CrossRefGoogle Scholar
  75. Saguaro Seminar on Civic Engagement in America. (2000). Better together. Cambridge: John F. Kennedy School of Government, Harvard University.Google Scholar
  76. Sampson, R., Morenoff, J., & Gannon-Rowley, T. (2002). Assessing ‹neighborhood effects’: Social processes and new directions in research. Annual Review of Sociology, 28, 443–478.CrossRefGoogle Scholar
  77. Sandefur, R. (2008). Access to civil justice and race, class and gender inequality. Annual Review of Sociology, 34(16), 1–20.Google Scholar
  78. Sanfey, A., Rilling, J., Aronson, J., Nystrom, L., & Cohen, J. (2003). The neural basis of economic decision-making in the Ultimatum game. Science, 300, 1755–1758.CrossRefGoogle Scholar
  79. Schauer, F. F. (2003). Profiles, probabilities and stereotypes. Cambridge: Harvard University Press.Google Scholar
  80. Schneider, J. D. (2004). The psychology of stereotyping (pp. 197–206). New York: The Guilford Press.Google Scholar
  81. Schneier, B. (2003). Beyond Fear: Thinking sensibly about security in an uncertain world. New York: Copernicus Books. See also, Wall, D. S. (2007). Cybercrime: The transformation of crime in the information age. Cambridge: Polity.Google Scholar
  82. Sen, A. (1992). Inequality reexamined. Cambridge: Harvard University Press.Google Scholar
  83. Tenner, E. (1996). Why things bite back: Technology and the revenge of unintended consequences. New York: Vintage Books.Google Scholar
  84. The Genetic Nondiscrimination Act (2008). Public Law No: 110–233, 21 May.Google Scholar
  85. Thiery, Y., & Van Shoubroeck, C. (2006). Fairness and equality in insurance classification. The Geneva Papers, 31:190–211. See also, Marmor, T., & Mashaw, J. (2006). Understanding social insurance: Fairness, affordability, and the ‹modernization’ of Social Security and Medicare. Health Affairs, 25:114–134.Google Scholar
  86. Van Hoyweghen, I., Horstman, K., & Schepers, R. (2005). ‹Genetics is not the issue’: Insurers on genetics and life insurance. New Genetics and Society, 24, 79–98.CrossRefGoogle Scholar
  87. Waldo, J., Lin, H., & Millett, L. (Eds.). (2007). Engaging privacy and information technology in a digital age. Washington: National Academies Press.Google Scholar
  88. Weiss, M. J. (1988). The clustering of America. New York: Harper & Row.Google Scholar
  89. Weiss, J. M. (2000). The clustered world: How we live, what we buy, and what it all means about who we are. Boston: Little, Brown & Co.Google Scholar
  90. Whitby, A., Josang, A., & Indulska, J. (2004). Filtering out unfair ratings in Bayesian reputation systems, Proceedings of the autonomous agents and multi agent systems conference. New York.Google Scholar
  91. Wilson, M. S. (2005). A social-value analysis of postmaterialism. The Journal of Social Psychology, 145(2), 209–224.CrossRefGoogle Scholar
  92. Wortham, L. (1986). Insurance classification: Too important to be left to the actuaries. Journal of Law Reform, 19, 349–423.Google Scholar
  93. Yamamiya, Y., Cash, T., Melnyk, S., Posavac, H., & Posavac, S. (2005). Women’s exposure to thin-and beautiful media images: Body image effects of media-ideal internalization and impact-reduction interventions. Body Image, 2:74–80. See also, Johnson, P., McCreary, D., & Mills, J. (2007). Effects of exposure to objectified male and female media images on men’s psychological well-being. Psychology of Men and Masculinity, 8:95–102.Google Scholar
  94. Zuberi, T. (2001). Thicker than blood: How racial statistics lie. Minneapolis: University of Minnesota Press, pp. 126–130. See also, Freedman, D. A. (1997). From association to causation via regression. In V. McKim & S. Turner (eds), Causality in crisis? Statistical methods and the search for causal knowledge in the social sciences (pp. 113–162). Notre Dame: University of Notre Dame Press.Google Scholar

Copyright information

© Springer Science+Business Media B.V. 2009

Authors and Affiliations

  1. 1.Annenberg School for CommunicationUniversity of PennsylvaniaPhiladelphiaUSA

Personalised recommendations