Journal of Urban Health

, Volume 95, Issue 3, pp 431–439 | Cite as

Reliability and One-Year Stability of the PIN3 Neighborhood Environmental Audit in Urban and Rural Neighborhoods

  • Anna K. Porter
  • Fang Wen
  • Amy H. Herring
  • Daniel A. Rodríguez
  • Lynne C. Messer
  • Barbara A. Laraia
  • Kelly R. Evenson


Reliable and stable environmental audit instruments are needed to successfully identify the physical and social attributes that may influence physical activity. This study described the reliability and stability of the PIN3 environmental audit instrument in both urban and rural neighborhoods. Four randomly sampled road segments in and around a one-quarter mile buffer of participants’ residences from the Pregnancy, Infection, and Nutrition (PIN3) study were rated twice, approximately 2 weeks apart. One year later, 253 of the year 1 sampled roads were re-audited. The instrument included 43 measures that resulted in 73 item scores for calculation of percent overall agreement, kappa statistics, and log-linear models. For same-day reliability, 81% of items had moderate to outstanding kappa statistics (kappas ≥ 0.4). Two-week reliability was slightly lower, with 77% of items having moderate to outstanding agreement using kappa statistics. One-year stability had 68% of items showing moderate to outstanding agreement using kappa statistics. The reliability of the audit measures was largely consistent when comparing urban to rural locations, with only 8% of items exhibiting significant differences (α < 0.05) by urbanicity. The PIN3 instrument is a reliable and stable audit tool for studies assessing neighborhood attributes in urban and rural environments.


Built environment Physical activity Active transport Measurement Walkability 



The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.


Funding for this study was provided by the National Institutes of Health (NIH) /National Cancer Institute (#CA109804-01). AKP was supported by a National Research Service Award T32 post-doctoral research fellowship, funded by the NIH, National Heart, Lung, and Blood Institute (#T32HL007055).

Compliance with ethical standards

This study was reviewed and approved by the Institutional Review Board at UNC.

Supplementary material

11524_2018_243_MOESM1_ESM.docx (66 kb)
Supplemental Table 1 (DOCX 65 kb)
11524_2018_243_MOESM2_ESM.docx (42 kb)
Supplemental Table 2 (DOCX 42 kb)


  1. 1.
    Sallis JF, Owen N, Fisher EB. Ecological models of health behavior. In: Glanz K, Rimer BK, Viswanath K, editors. Health behavior and health education: theory, research, and practice. 4th ed. San Francisco, CA: Jossey-Bass; 2008. p. 465–86.Google Scholar
  2. 2.
    Sallis JF, Cervero RB, Ascher W, Henderson KA, Kraft MK, Kerr J. An ecological approach to creating active living communities. Annu Rev Public Health. 2006;27:297–322.CrossRefPubMedGoogle Scholar
  3. 3.
    Sallis JF. Measuring physical activity environments: a brief history. Am J Prev Med. 2009;36(4 Suppl):S86–92.CrossRefPubMedPubMedCentralGoogle Scholar
  4. 4.
    Brownson RC, Hoehner CM, Day K, Forsyth A, Sallis JF. Measuring the built environment for physical activity: state of the science. Am J Prev Med. 2009;36(4 Suppl):S99–123.e112.CrossRefPubMedPubMedCentralGoogle Scholar
  5. 5.
    Bethlehem JR, Mackenbach JD, Ben-Rebah M, Compernolle S, Glonti K, Bárdos H, et al. The SPOTLIGHT virtual audit tool: a valid and reliable tool to assess obesogenic characteristics of the built environment. Int J Health Geogr. 2014;13:52.Google Scholar
  6. 6.
    Clarke P, Ailshire J, Melendez R, Bader M, Morenoff J. Using Google Earth to conduct a neighborhood audit: reliability of a virtual audit instrument. Health Place. 2010;16(6):1224–9.CrossRefPubMedPubMedCentralGoogle Scholar
  7. 7.
    Bader MDM, Mooney SJ, Lee YJ, Sheehan D, Neckerman KM, Rundle AG, et al. Development and deployment of the Computer Assisted Neighborhood Visual Assessment System (CANVAS) to measure health-related neighborhood conditions. Health Place. 2015;31:163–72.Google Scholar
  8. 8.
    Mooney SJ, Bader MDM, Lovasi GS, Neckerman KM, Teitler JO, Rundle AG. Validity of an ecometric neighborhood physical disorder measure constructed by virtual street audit. Am J Epidemiol. 2014;180(6):626–35.CrossRefPubMedPubMedCentralGoogle Scholar
  9. 9.
    Rundle AG, Bader MD, Richards CA, Neckerman KM, Teitler JO. Using Google Street View to audit neighborhood environments. Am J Prev Med. 2010;40(1):94–100.CrossRefGoogle Scholar
  10. 10.
    Brownson RC, Hoehner CM, Brennan LK, Cook RA, Elliott MB, McMullen KM. Reliability of 2 instruments for auditing the environment for physical activity. J Phys Act Health. 2004;1(3):191–208.CrossRefGoogle Scholar
  11. 11.
    Nickelson J, Wang AR, Mitchell QP, Hendricks K, Paschal A. Inventory of the physical environment domains and subdomains measured by neighborhood audit tools: a systematic literature review. J Environ Psychol. 2013;36:179–89.CrossRefGoogle Scholar
  12. 12.
    Millstein RA, Cain KL, Sallis JF, Conway TL, Geremia C, Frank LD, et al. Development, scoring, and reliability of the Microscale Audit of Pedestrian Streetscapes (MAPS). BMC Public Health. 2013;13:403.Google Scholar
  13. 13.
    Malecki KC, Engelman CD, Peppard PE, Nieto FJ, Grabow ML, Bernardinello M, et al. The Wisconsin Assessment of the Social and Built Environment (WASABE): a multi-dimensional objective audit instrument for examining neighborhood effects on health. BMC Public Health. 2014;14Google Scholar
  14. 14.
    Su M, Du YK, Liu QM, et al. Objective assessment of urban built environment related to physical activity—development, reliability and validity of the China Urban Built Environment Scan Tool (CUBEST). BMC Public Health. 2014;14:109.CrossRefPubMedPubMedCentralGoogle Scholar
  15. 15.
    Scanlin K, Haardoefer R, Kegler MC, Glanz K. Development of a pedestrian audit tool to assess rural neighborhood walkability. J Phys Act Health. 2014;11(6):1085–96.CrossRefPubMedGoogle Scholar
  16. 16.
    Leung CW, Gregorich SE, Laraia BA, Kushi LH, Yen IH. Measuring the neighborhood environment: associations with young girls’ energy intake and expenditure in a cross-sectional study. Int J Behav Nutr Phys Act. 2010;7:52.Google Scholar
  17. 17.
    Parsons JA, Singh G, Scott AN, Nisenbaum R, Balasubramaniam P, Jabbar A, et al. Standardized observation of neighbourhood disorder: does it work in Canada? Int J Health Geogr. 2010;9:6.Google Scholar
  18. 18.
    Schaefer-McDaniel N. Neighborhood stressors, perceived neighborhood quality, and child mental health in New York City. Health Place. 2009;15(1):148–55.CrossRefPubMedGoogle Scholar
  19. 19.
    Michael YL, Keast EM, Chaudhury H, Day K, Mahmood A, Sarte AF. Revising the senior walking environmental assessment tool. Prev Med. 2009;48(3):247–9.CrossRefPubMedGoogle Scholar
  20. 20.
    Franzini L, Elliott MN, Cuccaro P, Schuster M, Gilliland MJ, Grunbaum JA, et al. Influences of physical and social neighborhood environments on children's physical activity and obesity. Am J Public Health. 2009;99(2):271–8.Google Scholar
  21. 21.
    Furr-Holden CD, Smart MJ, Pokorni JL, et al. The NIfETy method for environmental assessment of neighborhood-level indicators of violence, alcohol, and other drug exposure. Prev Sci. 2008;9(4):245–55.CrossRefPubMedPubMedCentralGoogle Scholar
  22. 22.
    Zhu X, Lee C. Walkability and safety around elementary schools economic and ethnic disparities. Am J Prev Med. 2008;34(4):282–90.CrossRefPubMedGoogle Scholar
  23. 23.
    King D. Neighborhood and individual factors in activity in older adults: results from the neighborhood and senior health study. J Aging Phys Act. 2008;16(2):144–70.CrossRefPubMedGoogle Scholar
  24. 24.
    Kamphuis CB, Giskes K, Kavanagh AM, et al. Area variation in recreational cycling in Melbourne: a compositional or contextual effect? J Epidemiol Community Health. 2008;62(10):890–8.CrossRefPubMedGoogle Scholar
  25. 25.
    Foltete J-C, Piombini A. Urban layout, landscape features and pedestrian usage. Landsc Urban Plan. 2007;81:225–34.CrossRefGoogle Scholar
  26. 26.
    Zenk SN, Schulz AJ, Mentz G, House JS, Gravlee CC, Miranda PY, et al. Inter-rater and test-retest reliability: methods and results for the neighborhood observational checklist. Health Place. 2007;13(2):452–65.Google Scholar
  27. 27.
    Suminski RR, Heinrich KM, Poston WS, Hyder M, Pyle S. Characteristics of urban sidewalks/streets and objectively measured physical activity. J Urban Health. 2008;85(2):178–90.CrossRefPubMedGoogle Scholar
  28. 28.
    Clifton KJ, Livi Smith AD, Rodriguez D. The development and testing of an audit for the pedestrian environment. Landsc Urban Plan. 2007;80(1–2):95–110.CrossRefGoogle Scholar
  29. 29.
    Hoehner CM, Ivy A, Ramirez LK, Handy S, Brownson RC. Active neighborhood checklist: a user-friendly and reliable tool for assessing activity friendliness. Am J Health Promot. 2007;21(6):534–7.CrossRefPubMedGoogle Scholar
  30. 30.
    Wright PA, Kloos B. Housing environment and mental health outcomes: A levels of analysis perspective. J Environ Psychol. 2007;27(1):79–89.CrossRefPubMedPubMedCentralGoogle Scholar
  31. 31.
    McMillan TE. The relative influence of urban form on a child’s travel mode to school. Transp Res A Policy Pract. 2007;41(1):69–79.CrossRefGoogle Scholar
  32. 32.
    Boarnet MG, Day K, Alfonzo M, Forsyth A, Oakes M. The Irvine-Minnesota inventory to measure built environments: reliability tests. Am J Prev Med. 2006;30(2):153–9.CrossRefPubMedGoogle Scholar
  33. 33.
    Andresen EM, Malmstrom TK, Miller DK, Wolinsky FD. Reliability and validity of observer ratings of neighborhoods. J Aging Health. 2006;18(1):28–36.CrossRefPubMedGoogle Scholar
  34. 34.
    Laraia BA, Messer L, Kaufman JS, Dole N, Caughy M, O’Campo P, et al. Direct observation of neighborhood attributes in an urban area of the US south: characterizing the social context of pregnancy. Int J Health Geogr. 2006;5:11.Google Scholar
  35. 35.
    Craig CL, Brownson RC, Cragg SE, Dunn AL. Exploring the effect of the environment on physical activity: a study examining walking to work. Am J Prev Med. 2002;23(2 Suppl):36–43.CrossRefPubMedGoogle Scholar
  36. 36.
    Pikora TJ, Bull FC, Jamrozik K, Knuiman M, Giles-Corti B, Donovan RJ. Developing a reliable audit instrument to measure the physical environment for physical activity. Am J Prev Med. 2002;23(3):187–94.CrossRefPubMedGoogle Scholar
  37. 37.
    Caughy MO, O'Campo PJ, Patterson J. A brief observational measure for urban neighborhoods. Health Place. 2001;7(3):225–36.CrossRefPubMedGoogle Scholar
  38. 38.
    Loukaitou-Sideris A, Liggett R, Iseki H, Thurlow W. Measuring the effects of built environment on bus stop crime. Environ Plan B: Plan Design. 2001;28(2):255–80.CrossRefGoogle Scholar
  39. 39.
    Evenson KR, Sotres-Alvarez D, Herring AH, Messer L, Laraia BA, Rodríguez DA. Assessing urban and rural neighborhood characteristics using audit and GIS data: derivation and reliability of constructs. Int J Behav Nutr Phys Act. 2009;6:44.CrossRefPubMedPubMedCentralGoogle Scholar
  40. 40.
    Knapp EA, Nau C, Brandau S, DeWalle J, Hirsch AG, Bailey-Davis L, et al. Community Audit of Social, Civil, and Activity Domains in Diverse Environments (CASCADDE). Am J Prev Med. 2017;52(4):530–40.Google Scholar
  41. 41.
    Brown AF, Morris DM, Kahn KL, et al. The healthy community neighborhood initiative: rationale and design. Ethn Dis. 2016;26(1):123–32.CrossRefPubMedPubMedCentralGoogle Scholar
  42. 42.
    Frizzelle BG, Evenson KR, Rodriguez DA, Laraia BA. The importance of accurate road data for spatial applications in public health: customizing a road network. Int J Health Geogr. 2009;8:24.CrossRefPubMedPubMedCentralGoogle Scholar
  43. 43.
    Census 2000 urban and rural classifications. Accessed Sept 26, 2017.
  44. 44.
    Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics. 1977;33(1):159–74.CrossRefPubMedGoogle Scholar
  45. 45.
    Tanner MA, Young MA. Modeling agreement among raters. J Am Stat Assoc. 1985;80(389):175–80.CrossRefGoogle Scholar
  46. 46.
    Tanner MA, Young MA. Modeling ordinal scale disagreement. Psychol Bull. 1985;98(2):408–15.CrossRefPubMedGoogle Scholar
  47. 47.
    Agresti A. An introduction to categorical data analysis. New York, NY: Wiley; 1996.Google Scholar

Copyright information

© The New York Academy of Medicine 2018

Authors and Affiliations

  1. 1.Department of Epidemiology, Gillings School of Global Public HealthUniversity of North Carolina – Chapel HillChapel HillUSA
  2. 2.Department of Statistical ScienceDuke UniversityDurhamUSA
  3. 3.Department of City and Regional Planning, Institute for Transportation StudiesUniversity of California, BerkeleyBerkeleyUSA
  4. 4.Oregon Health & Science University - Portland State University School of Public HealthPortlandUSA
  5. 5.Division of Community Health Sciences, School of Public HealthUniversity of California, BerkeleyBerkeleyUSA
  6. 6.Department of Epidemiology, Gillings School of Global Public Health, Center for Health Promotion and Disease PreventionUniversity of North Carolina – Chapel HillChapel HillUSA

Personalised recommendations