Abstract
Reliable and stable environmental audit instruments are needed to successfully identify the physical and social attributes that may influence physical activity. This study described the reliability and stability of the PIN3 environmental audit instrument in both urban and rural neighborhoods. Four randomly sampled road segments in and around a one-quarter mile buffer of participants’ residences from the Pregnancy, Infection, and Nutrition (PIN3) study were rated twice, approximately 2 weeks apart. One year later, 253 of the year 1 sampled roads were re-audited. The instrument included 43 measures that resulted in 73 item scores for calculation of percent overall agreement, kappa statistics, and log-linear models. For same-day reliability, 81% of items had moderate to outstanding kappa statistics (kappas ≥ 0.4). Two-week reliability was slightly lower, with 77% of items having moderate to outstanding agreement using kappa statistics. One-year stability had 68% of items showing moderate to outstanding agreement using kappa statistics. The reliability of the audit measures was largely consistent when comparing urban to rural locations, with only 8% of items exhibiting significant differences (α < 0.05) by urbanicity. The PIN3 instrument is a reliable and stable audit tool for studies assessing neighborhood attributes in urban and rural environments.
Similar content being viewed by others
References
Sallis JF, Owen N, Fisher EB. Ecological models of health behavior. In: Glanz K, Rimer BK, Viswanath K, editors. Health behavior and health education: theory, research, and practice. 4th ed. San Francisco, CA: Jossey-Bass; 2008. p. 465–86.
Sallis JF, Cervero RB, Ascher W, Henderson KA, Kraft MK, Kerr J. An ecological approach to creating active living communities. Annu Rev Public Health. 2006;27:297–322.
Sallis JF. Measuring physical activity environments: a brief history. Am J Prev Med. 2009;36(4 Suppl):S86–92.
Brownson RC, Hoehner CM, Day K, Forsyth A, Sallis JF. Measuring the built environment for physical activity: state of the science. Am J Prev Med. 2009;36(4 Suppl):S99–123.e112.
Bethlehem JR, Mackenbach JD, Ben-Rebah M, Compernolle S, Glonti K, Bárdos H, et al. The SPOTLIGHT virtual audit tool: a valid and reliable tool to assess obesogenic characteristics of the built environment. Int J Health Geogr. 2014;13:52.
Clarke P, Ailshire J, Melendez R, Bader M, Morenoff J. Using Google Earth to conduct a neighborhood audit: reliability of a virtual audit instrument. Health Place. 2010;16(6):1224–9.
Bader MDM, Mooney SJ, Lee YJ, Sheehan D, Neckerman KM, Rundle AG, et al. Development and deployment of the Computer Assisted Neighborhood Visual Assessment System (CANVAS) to measure health-related neighborhood conditions. Health Place. 2015;31:163–72.
Mooney SJ, Bader MDM, Lovasi GS, Neckerman KM, Teitler JO, Rundle AG. Validity of an ecometric neighborhood physical disorder measure constructed by virtual street audit. Am J Epidemiol. 2014;180(6):626–35.
Rundle AG, Bader MD, Richards CA, Neckerman KM, Teitler JO. Using Google Street View to audit neighborhood environments. Am J Prev Med. 2010;40(1):94–100.
Brownson RC, Hoehner CM, Brennan LK, Cook RA, Elliott MB, McMullen KM. Reliability of 2 instruments for auditing the environment for physical activity. J Phys Act Health. 2004;1(3):191–208.
Nickelson J, Wang AR, Mitchell QP, Hendricks K, Paschal A. Inventory of the physical environment domains and subdomains measured by neighborhood audit tools: a systematic literature review. J Environ Psychol. 2013;36:179–89.
Millstein RA, Cain KL, Sallis JF, Conway TL, Geremia C, Frank LD, et al. Development, scoring, and reliability of the Microscale Audit of Pedestrian Streetscapes (MAPS). BMC Public Health. 2013;13:403.
Malecki KC, Engelman CD, Peppard PE, Nieto FJ, Grabow ML, Bernardinello M, et al. The Wisconsin Assessment of the Social and Built Environment (WASABE): a multi-dimensional objective audit instrument for examining neighborhood effects on health. BMC Public Health. 2014;14
Su M, Du YK, Liu QM, et al. Objective assessment of urban built environment related to physical activity—development, reliability and validity of the China Urban Built Environment Scan Tool (CUBEST). BMC Public Health. 2014;14:109.
Scanlin K, Haardoefer R, Kegler MC, Glanz K. Development of a pedestrian audit tool to assess rural neighborhood walkability. J Phys Act Health. 2014;11(6):1085–96.
Leung CW, Gregorich SE, Laraia BA, Kushi LH, Yen IH. Measuring the neighborhood environment: associations with young girls’ energy intake and expenditure in a cross-sectional study. Int J Behav Nutr Phys Act. 2010;7:52.
Parsons JA, Singh G, Scott AN, Nisenbaum R, Balasubramaniam P, Jabbar A, et al. Standardized observation of neighbourhood disorder: does it work in Canada? Int J Health Geogr. 2010;9:6.
Schaefer-McDaniel N. Neighborhood stressors, perceived neighborhood quality, and child mental health in New York City. Health Place. 2009;15(1):148–55.
Michael YL, Keast EM, Chaudhury H, Day K, Mahmood A, Sarte AF. Revising the senior walking environmental assessment tool. Prev Med. 2009;48(3):247–9.
Franzini L, Elliott MN, Cuccaro P, Schuster M, Gilliland MJ, Grunbaum JA, et al. Influences of physical and social neighborhood environments on children's physical activity and obesity. Am J Public Health. 2009;99(2):271–8.
Furr-Holden CD, Smart MJ, Pokorni JL, et al. The NIfETy method for environmental assessment of neighborhood-level indicators of violence, alcohol, and other drug exposure. Prev Sci. 2008;9(4):245–55.
Zhu X, Lee C. Walkability and safety around elementary schools economic and ethnic disparities. Am J Prev Med. 2008;34(4):282–90.
King D. Neighborhood and individual factors in activity in older adults: results from the neighborhood and senior health study. J Aging Phys Act. 2008;16(2):144–70.
Kamphuis CB, Giskes K, Kavanagh AM, et al. Area variation in recreational cycling in Melbourne: a compositional or contextual effect? J Epidemiol Community Health. 2008;62(10):890–8.
Foltete J-C, Piombini A. Urban layout, landscape features and pedestrian usage. Landsc Urban Plan. 2007;81:225–34.
Zenk SN, Schulz AJ, Mentz G, House JS, Gravlee CC, Miranda PY, et al. Inter-rater and test-retest reliability: methods and results for the neighborhood observational checklist. Health Place. 2007;13(2):452–65.
Suminski RR, Heinrich KM, Poston WS, Hyder M, Pyle S. Characteristics of urban sidewalks/streets and objectively measured physical activity. J Urban Health. 2008;85(2):178–90.
Clifton KJ, Livi Smith AD, Rodriguez D. The development and testing of an audit for the pedestrian environment. Landsc Urban Plan. 2007;80(1–2):95–110.
Hoehner CM, Ivy A, Ramirez LK, Handy S, Brownson RC. Active neighborhood checklist: a user-friendly and reliable tool for assessing activity friendliness. Am J Health Promot. 2007;21(6):534–7.
Wright PA, Kloos B. Housing environment and mental health outcomes: A levels of analysis perspective. J Environ Psychol. 2007;27(1):79–89.
McMillan TE. The relative influence of urban form on a child’s travel mode to school. Transp Res A Policy Pract. 2007;41(1):69–79.
Boarnet MG, Day K, Alfonzo M, Forsyth A, Oakes M. The Irvine-Minnesota inventory to measure built environments: reliability tests. Am J Prev Med. 2006;30(2):153–9.
Andresen EM, Malmstrom TK, Miller DK, Wolinsky FD. Reliability and validity of observer ratings of neighborhoods. J Aging Health. 2006;18(1):28–36.
Laraia BA, Messer L, Kaufman JS, Dole N, Caughy M, O’Campo P, et al. Direct observation of neighborhood attributes in an urban area of the US south: characterizing the social context of pregnancy. Int J Health Geogr. 2006;5:11.
Craig CL, Brownson RC, Cragg SE, Dunn AL. Exploring the effect of the environment on physical activity: a study examining walking to work. Am J Prev Med. 2002;23(2 Suppl):36–43.
Pikora TJ, Bull FC, Jamrozik K, Knuiman M, Giles-Corti B, Donovan RJ. Developing a reliable audit instrument to measure the physical environment for physical activity. Am J Prev Med. 2002;23(3):187–94.
Caughy MO, O'Campo PJ, Patterson J. A brief observational measure for urban neighborhoods. Health Place. 2001;7(3):225–36.
Loukaitou-Sideris A, Liggett R, Iseki H, Thurlow W. Measuring the effects of built environment on bus stop crime. Environ Plan B: Plan Design. 2001;28(2):255–80.
Evenson KR, Sotres-Alvarez D, Herring AH, Messer L, Laraia BA, Rodríguez DA. Assessing urban and rural neighborhood characteristics using audit and GIS data: derivation and reliability of constructs. Int J Behav Nutr Phys Act. 2009;6:44.
Knapp EA, Nau C, Brandau S, DeWalle J, Hirsch AG, Bailey-Davis L, et al. Community Audit of Social, Civil, and Activity Domains in Diverse Environments (CASCADDE). Am J Prev Med. 2017;52(4):530–40.
Brown AF, Morris DM, Kahn KL, et al. The healthy community neighborhood initiative: rationale and design. Ethn Dis. 2016;26(1):123–32.
Frizzelle BG, Evenson KR, Rodriguez DA, Laraia BA. The importance of accurate road data for spatial applications in public health: customizing a road network. Int J Health Geogr. 2009;8:24.
Census 2000 urban and rural classifications. https://www.census.gov/geo/reference/ua/urban-rural-2000.html. Accessed Sept 26, 2017.
Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics. 1977;33(1):159–74.
Tanner MA, Young MA. Modeling agreement among raters. J Am Stat Assoc. 1985;80(389):175–80.
Tanner MA, Young MA. Modeling ordinal scale disagreement. Psychol Bull. 1985;98(2):408–15.
Agresti A. An introduction to categorical data analysis. New York, NY: Wiley; 1996.
Disclaimer
The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.
Funding
Funding for this study was provided by the National Institutes of Health (NIH) /National Cancer Institute (#CA109804-01). AKP was supported by a National Research Service Award T32 post-doctoral research fellowship, funded by the NIH, National Heart, Lung, and Blood Institute (#T32HL007055).
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
This study was reviewed and approved by the Institutional Review Board at UNC.
Electronic supplementary material
Supplemental Table 1
(DOCX 65 kb)
Supplemental Table 2
(DOCX 42 kb)
Rights and permissions
About this article
Cite this article
Porter, A.K., Wen, F., Herring, A.H. et al. Reliability and One-Year Stability of the PIN3 Neighborhood Environmental Audit in Urban and Rural Neighborhoods. J Urban Health 95, 431–439 (2018). https://doi.org/10.1007/s11524-018-0243-9
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11524-018-0243-9