Skip to main content
SpringerLink
Log in

Accounting for Taste: Using Propensity Score Methods to Evaluate the Documentary Film, Waiting for “Superman”

  • Chapter
  • First Online:
Arts Evaluation and Assessment

Abstract

Evaluating the impact of the arts can be difficult for a variety of reasons, including the cost of survey administration and the issue of selection bias. However, recent media impact studies offer models for economically evaluating programming that appeals to particular personal tastes, supplying methodologies that can be applied to measure the impact of the arts. This chapter explores a study of the documentary film Waiting for “Superman,” in which propensity score matching (PSM) was used to identify covariates for propensity to be exposed to the film. Survey respondents are matched based on scores attached to degree of propensity to be exposed; matched propensity scores that differ according to exposure allow for evaluation of impact while controlling for selection bias.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
eBook
USD 19.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 29.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 129.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    “To a Bayesian, estimates of this probability are posterior predictive probability of assignment to treatment 1 for a unit with vector x of covariates” (Rosenbaum & Rubin, 1983).

  2. 2.

    The binary dependent variable is 1 for exposed and 0 for not exposed.

  3. 3.

    Sensitivity analyses can be conducted to test for the possibility that unobserved characteristics of the subjects independently affect the likelihood of treatment. For example, in the Landrum 2001 study, they hypothesized the existence of an unobserved binary variable related to propensity score and updated estimates of the effect of treatment after adjusting for the hypothetical confounder. They used sensitivity analysis to show that their actual conclusions were mildly sensitive to unobserved effects that are extreme but within range of the observed effects.

  4. 4.

    An alternative to the nearest neighbor method is to use a “caliper” or radius of score standard deviations to find a match (Rubin, 2001, p. 177).

  5. 5.

    “The propensity score was created by regressing high exposure to Tsha Tsha (versus none and low) on the identified variables, and then using the resulting probability of watching predicted by those 15 variables for purposes of matching. The probability (propensity score) in the present case varied between 0.03 and 0.97, with an average value of 0.50, which is the same as the percentage of those above the median with a high level of recall of the drama. The lower a respondent scored on the 15 variables, the lower the propensity score, and conversely, the higher a respondent scored on the 15 variables, the higher their propensity score. The sample of respondents was divided into six groups across the range of propensity scores. The first group consisted of all respondents with propensity scores below 0.20. The groups ranged from 0.20 to 0.40; 0.40 to 0.60; 0.60 to 0.80; 0.80 to 0.90, and those with scores above 0.90. Within each of these strata or blocks there was no statistically significant difference between the average propensity score of respondents who watched Tsha Tsha and those who did not watch. There was also no statistically significant difference for each of the 15 variables used to construct the score. Thus, within each of these six groups , viewers and non-viewers were statistically equivalent in the same way they would be if they had been randomly assigned to each group” (CADRE, 2005, pg. 20).

References

  • Bandura, A. (1977). Social learning theory. Englewood Cliffs, NJ: Prentice Hall.

    Google Scholar 

  • Bandura, A., Ross, D., & Ross, S. A. (1963). Imitation of film-mediated aggressive models. Journal of Abnormal and Social Psychology, 66, 3–11.

    Article  Google Scholar 

  • Belfiore, E., & Bennett, O. (2010). The social impact of the arts: An intellectual history. New York: Palgrave Macmillan.

    Google Scholar 

  • Bertrand, J. T., O’Reilly, K., Denison, J., Anhang, R., & Sweat, M. (2006). Systematic review of the effectiveness of mass communication programs to change HIV/AIDS-related behaviors in developing countries. Health Education Research, 21(4), 567–597.

    Article  Google Scholar 

  • Brown, A. S., & Novak, J. L. (2007). Assessing the intrinsic impacts of a live performance. San Francisco, CA: WolfBrown.

    Google Scholar 

  • Centre for AIDS Development, Research and Evaluation. (2005). Tsha Tsha: Key findings of the evaluation of episodes 1–26. Retrieved January 18, 2011, from http://www.m-mc.org/spotlight/southafrica_tshatsha/CADREeval.pdf

  • Cochran, W. G. (1968). The effectiveness of adjustment by subclassification in removing bias in observational studies. Biometrics, 24, 295–313.

    Article  Google Scholar 

  • Conquergood, D. (1988). Health theater in a Hmong refugee camp: Performance, communication and culture. TDR: Journal of Performance Studies, 32, 174–208.

    Article  Google Scholar 

  • Danielsson, S. (2004). The propensity score and estimation in nonrandom surveys: An overview. Retrieved January 18, 2011, from http://www.statistics.su.se/modernsurveys/publ/11.pdf

  • Dehejia, R. H., & Wahba, S. (2002). Propensity score-matching methods for nonexperimental causal studies. The Review of Economics and Statistics, 84(1), 151–161.

    Article  Google Scholar 

  • Do, M. P., & Kincaid, D. L. (2006). Impact of an entertainment-education television drama on health knowledge and behavior in Bangladesh: An application of propensity score matching. Journal of Health Communication, 11, 301–325.

    Article  Google Scholar 

  • Galloway, S. (2009). Theory-based evaluation and the social impact of the arts. Cultural Trends, 18(2), 125–148.

    Article  Google Scholar 

  • Landrum, M. B., & Ayanian, J. Z. (2001). Causal effect on ambulatory specialty care on mortality following myocardial infarction: A comparison of propensity score and instrumental variable analyses. Health Services & Outcomes Research Methodology, 2, 221–245.

    Article  Google Scholar 

  • Marcus, P. M., Huang, G. C., Beck, V., & Miller, M. J. (2010). The impact of a primetime cancer storyline: From individual knowledge and behavioral intentions to policy-level changes. Journal of Cancer Education, 25(4), 484–489.

    Article  Google Scholar 

  • Morgan, S. E., Movius, L., & Cody, M. J. (2009). The power of narratives: The effect of entertainment television organ donation storylines on the attitudes, knowledge, and behaviors of donors and nondonors. Journal of Communication, 59(1), 135–151.

    Article  Google Scholar 

  • Murphy, S. T., Hether, H. J., Felt, L. J., & Buffington, S. C. (2012). Public diplomacy in prime time: Exploring the potential of entertainment education in international diplomacy. American Journal of Media Psychology, 5, 5–32.

    Google Scholar 

  • Nahm, S., Le, K., Buffington, S., Schiman, N., Raider, S., & Resko, S. (2010). Engaging youth through entertainment education through partnership and collaboration. Cases in Public Health Communication & Marketing, 4(2), 57–78.

    Google Scholar 

  • National Endowment for the Arts and Arts & Humanities Research Council. (2014). Measuring cultural engagement: A quest for new terms, tools, and techniques.

    Google Scholar 

  • Posner, M. A., Ash, A. S., Freund, K. M., Moskowitz, M. A., & Shwartz, M. (2001). Comparing standard regression, propensity score matching, and instrumental variables methods for determining the influence of mammography on stage of diagnosis. Health Services & Outcomes Research Methodology, 2, 279–290.

    Article  Google Scholar 

  • Rosenbaum, P. R., & Rubin, D. B. (1983). The central role of the propensity score in observational studies for causal effects. Biometrika, 70(1), 41–55.

    Article  Google Scholar 

  • Rubin, D. B. (2001). Using propensity scores to help design observational studies: Application to the tobacco litigation. Health Services & Outcomes Research Methodology, 2, 169–188.

    Article  Google Scholar 

  • Singhal, A. (2004). Empowering the oppressed through participatory theater. Investigacion y Desarrollo, 12(1), 138–163.

    Google Scholar 

  • Singhal, A., & Rogers, E. M. (1989). Prosocial television for development in India. In R. E. Rice & C. K. Atkin (Eds.), Public communication campaigns (2nd ed., p. 331). Newbury Park, CA: Sage.

    Google Scholar 

  • Singhal, A., & Rogers, E. M. (1999). Entertainment-education: A communication strategy for social change. Mahwah, NJ: Lawrence Erlbaum Associates.

    Google Scholar 

  • TakePart. (2010). Waiting for Superman impact. Retrieved September 12, 2016, from http://www.takepart.com/waiting-for-superman/impact

  • Winkelmayer, W. C., & Kurth, T. (2004). Propensity scores: Help or hype? Nephrology, Dialysis, Transplantation, 19, 1671–1673.

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Annenberg School for Communication and Journalism, University of Southern California, Los Angeles, CA, USA

    Johanna Blakley

  2. Anthropology & Sociology, The New School for Public Engagement, New York, NY, USA

    Sheena Nahm

Authors
  1. Johanna Blakley
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Sheena Nahm
    View author publications

    You can also search for this author in PubMed Google Scholar

Editor information

Editors and Affiliations

  1. PANCH Research, LLC, Chicago, IL, USA

    Rekha S. Rajan

  2. MUSE Research, LLC, Kensington, MD, USA

    Ivonne Chand O'Neal

Additional information

The authors would like to thank Heesung Shin for her considerable contributions to this research project, as well as Grace Huang for her assistance in its earliest phase. The authors would also like to thank Veronica Jauriqui and Adam Amel Rogers for graphic design support and editorial and formatting assistance during the final phases of this work.

Rights and permissions

Reprints and permissions

Copyright information

© 2018 The Author(s)

About this chapter

Cite this chapter

Blakley, J., Nahm, S. (2018). Accounting for Taste: Using Propensity Score Methods to Evaluate the Documentary Film, Waiting for “Superman” . In: Rajan, R., O'Neal, I. (eds) Arts Evaluation and Assessment. Palgrave Macmillan, Cham. https://doi.org/10.1007/978-3-319-64116-4_12

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-64116-4_12

  • Published:

  • Publisher Name: Palgrave Macmillan, Cham

  • Print ISBN: 978-3-319-64115-7

  • Online ISBN: 978-3-319-64116-4

  • eBook Packages: EducationEducation (R0)

Publish with us

Policies and ethics

Search

Navigation