UpStart Parent Survey: A New Psychometrically Valid Tool for the Evaluation of Prevention-focused Parenting Programs
- First Online:
- 1.3k Downloads
Parents are the most significant influence on the growth and development of young children. All parents can increase their knowledge of developmental milestones and parenting practices by participating in effective programs that offer information and support. However, there is limited outcome evaluation of programs offering these services. Prevention-focused parenting programs (P-FPPs) are key frontline services designed to educate parents and improve the overall well-being of children. Evaluation of these programs is currently weak; this is not to say they are ineffective, rather that their effectiveness has been poorly evaluated. Rigorous evaluation of P-FPPs would support informed funding and evidence-based policy decisions. The purpose of this study was to conduct a preliminary psychometric analysis of the UpStart Parent Survey (USPS)—a tool developed specifically for evaluating this type of program. Preliminary analysis revealed uni-dimensionality of each scale, strong internal consistency and temporal stability, as well as strong concurrent validity on 9 of the 11 items examined with an urban Canadian population. In its first round of psychometric evaluation, the USPS demonstrated promise as a brief, easy to administer, scientifically rigorous tool for the evaluation of prevention-focused parenting programs.
KeywordsMeasurement Child development Intervention Parenting
Over the past decade, there has been a dramatic and essential demand for accountability with evidence-based programming. Currently, most parenting programs have limited evidence of their effectiveness [1, 2], with the majority of research literature limited to intervention programs targeted at high risk children and families (e.g., [3, 4]). Prevention-focused parenting programs (P-FPPs) are key front-line services that help parents learn about child development and healthy parent/child relationships, introduce valuable support networks, and promote parental connection to their community. In order to thrive, P-FPPs must demonstrate they achieve their intended outcomes and improve parental functioning.
The foundational capacities for lifelong learning and health are established early in life . Parents have the potential to significantly contribute to the healthy development of young children  and most parents will benefit from additional information, support, or guidance. Research indicates that there is a substantial gap in what parents believe they know and what they need to know to support their child’s development . P-FPPs are designed to serve as primary and secondary prevention supports, aimed at educating, increasing resiliency, improving relationships between parents and their children, and promoting parental and family competence, in populations who are not in crisis. The goal of most P-FPPs is to build parenting capacity to prevent problems before they occur, by increasing protective factors such as knowledge of child development, healthy parenting skills, and parental competence and satisfaction . While community-based P-FPPs are well-positioned to address these needs and support growing families, are they actually realizing these objectives?
Rigorous evaluation requires the use of reliable and valid measurement tools. In a community setting, program providers and program funders often have conflicting goals and opposing ideology when defining effective programs and useful measurement tools. While both are committed to collecting meaningful data, program providers prefer evaluations that are unobtrusive, quickly completed, and easy to interpret. Conversely, program evaluators favor measurement tools that have strong psychometric properties and present an accurate assessment of program outcomes. Typically, one group’s goal is met at the expense of the other, as scientifically rigorous measures tend to be long and burdensome . Furthermore, instruments generally examine a single outcome variable, such as self-efficacy  or parenting stress  and avoid the evaluation of broad concepts. The ideal tool for community based programs would be a brief measurement instrument that evaluated multiple outcome variables, while maintaining low respondent burden and strong psychometric properties.
Decades of research have shown that specific factors enhance positive parent/child relationships and increase the likelihood of positive child outcomes . In response to this research, a consensus conference was held in Canada and a set of outcomes common to effective parenting programs were identified. The consensus group used prior literature relevant to their regional context to recommend specific outcomes important for effective parenting education programs . The common outcomes generated were: quality of life, self-efficacy, family functioning, social support, parenting knowledge and skills, parental competence, emotional health, parenting stress, and formal and informal support systems. These specific indicators were used in a Western Canadian urban setting as the foundation for the UpStart Parent Survey (USPS).
The purpose of this study was to conduct a preliminary psychometric analysis of the USPS. The USPS contained three separate but inter-related subscales: Parenting Knowledge, Parenting Experiences, and Program Satisfaction. Constructs measured by the USPS include; parenting knowledge and skills, self-efficacy, mental health, social support, parenting stress, and family functioning, These concepts are strongly linked to successful parenting and child well-being, and are expected to be improved in effective P-FPPs. The psychometric evaluation investigated (a) internal consistency reliability of the USPS, (b) temporal stability (test–retest) reliability, and (c) concurrent validity of individual items on the USPS Parenting Experiences subscale with established measurements previously deemed psychometrically valid and reliable.
Participants (345 parents/caregivers of young children) were recruited at P-FPPs between April 2010 and March 2011. Program leaders were oriented to the project by the research team prior to April 2010. Parent participants who completed only the USPS filled out the survey at the end of the last P-FPPs session and returned the USPS to the program leader. The program leader then returned all of the surveys to the research office in the stamped, addressed envelope provided.
Once the process was well-established, program facilitators randomly recruited a subset of the participants to complete the second and third component of the study—a test/retest sample (n = 22), and a ‘gold-standard’ concurrent validity sample (n = 53). The ‘gold-standard’ component of the study included all of the validation measures; participants completed the USPS and the validation measures for psychometric analysis. For each of the components, consenting parents were given instructions for each measure along with an addressed, stamped envelope to return their packages to the research office. Parent participants who completed component two or three were mailed a $20CDN gift card to acknowledge their time. The study was approved by the Mount Royal University, Human Research Ethics Board and the University of Calgary, Conjoint Health Research Ethics Board.
Characteristics of individuals participating in the prevention-focused education programs who completed an USPS
50 years or over
Level of education
Less than high school
High school diploma
Certificate/diploma after high school
Less than $20,000/year
More than $80,000/year
English as a first language
South East Asian
Birth parent of child
Number of children/household
Three or more
Prevention-focused Parenting Programs
Six different agencies offering 10 separate programs for parents of young children participated in this study. The parenting programs contributing to this investigation were diverse, and most had previously demonstrated evidence of effectiveness on outcomes, such as parenting morale, social support, and parenting roles and responsibilities. All programs underwent program evaluation processes that included reliably measurable outcomes. Programs targeted parents of children 6 years of age and under, and were located in geographically diverse areas of the city. The programs were offered for a defined length of time that varied from 4 to 11 weeks; weekly classes lasted between 2 and 3 h.
The primary focus of all programs was education and support, and each program had well-established curricula tailored to their parenting population. Typical child growth and development, as well as parenting strategies for young children, were essential elements to each program. Programs varied their dissemination style; some programs included only a classroom/parent discussion and learning component, others include both classroom and parent/child learning components.
UpStart Parent Survey
The UpStart Parent Survey (USPS) was designed as a brief, paper and pencil, self-report measure of common outcomes expected of P-FPPs. The USPS is comprised of three subscales: (1) Parenting Knowledge/Skills (PK), (2) Parenting Experience (PE), and (3) Program Satisfaction (PS). The survey takes approximately 15 min to complete.
The Parenting Knowledge/Skills subscale (PK) is a 10 item scale that captures concepts such as growth and development, discipline strategies, child health and safety, and parental responses to everyday challenges. These items were uniquely developed by the research team from the P-FPP curricula. The PK scale uses a 7-point Likert scale ranging from 1 (Strongly Disagree) to 7 (Strongly Agree). Scores on individual items were summed to create a total scale score. The theoretical range of scores is 10–70 with higher scores indicating greater parenting knowledge and skills. Each item on the PK scale offers a Not Covered response option for concepts that were not addressed in the specific program curriculum. Not Covered responses were analyzed as missing values.
Using a similar 7-point Likert scale, [1 (Strongly Disagree) to 7 (Strongly Agree)], the Parenting Experiences subscale (PE) includes 11 items that capture additional common outcomes of parenting programs including parenting self-efficacy, emotional health, social support, parenting stress, and family functioning. The PE items were designed by the researchers from the common outcomes identified at the consensus conference, and were subsequently validated in this study with standardized measures. Scores on individual items were summed to create a PE total scale score. The theoretical range of scores is 11–77 with higher scores indicating a more positive parenting experience.
The PK and PE subscales were designed as a post-test/retrospective pre-test. The parent reports a “today” score and a “before this program” score for each item that assesses both post-test and retrospective pre-test particulars. To clarify, a retrospective pre-test is administered at the same time as the post-test but asks respondents to reflect upon their level of understanding or skill prior to the intervention/training. A post-test/retrospective pre-test  survey design is recommended when the goal of program evaluation is (a) the assessment of individual perceptions of change , (b) establishment of trust , conservation of limited program time, and (c) provider guided reflection on personal growth related to the program . Post-test/retrospective pre-tests provide parents with an opportunity to reflect on how much they have learned over the course of the program.
The Program Satisfaction (PS) subscale contains 7 items that capture engagement in, and satisfaction with the program. The PS utilizes a 5-point Likert scale ranging from, strongly disagree to strongly agree. Scores on individual items were summed to create a PS total score. The theoretical range of scores is 7–35 with higher scores indicating greater satisfaction with the parenting program.
Validation measures were selected by the consensus committee to align with aspects of the PE scale, as well as exhibit (a) strong psychometric properties, (b) suitability to a wide ranging demographic, and (c) ability to capture constructs critical to social support, parenting efficacy, and family functioning .
Brief Family Assessment Measure: General Scale
The brief FAM-General Scale  is a short version (14-item) of the FAM scale that provides an overall rating of family functioning. The scale can be administered and scored in under 10 min; responses range from 0 (strongly agree) to 3 (strongly disagree). Items are summed and scores translated to T-scores. Higher scores indicate disrupted family functioning. Test–retest reliability is .56–.66 over 12 days, and internal consistency was reported at .86–.94 . Cronbach’s alpha for this study was found to be .73.
Parenting Morale Index
The Parenting Morale Index (PMI)  is a 10-item, self-report, paper and pencil measure designed to capture psychological energy, positive parenting spirits, and enthusiasm for parenting. Six items are reverse scored, but all items range from 1 (not at all) to 5 (very often). Items are summed to create a total score. Higher scores indicate higher parenting morale. Cronbach’s alpha has been reported at .86 , and was .82 for this study.
Family Support Scale
The Family Support Scale (FSS) is a self-report measure that assesses the helpfulness of formal and informal social support sources for parents raising young children . The scale includes 20 items designed to measure social support through examination of informal kinship, social organization items, formal kinship, immediate family, specialized professional services, and generic professional services. Respondents rate social support on a 5-point Likert scale ranging from 1 (not helpful) to 5 (extremely helpful). The FSS was scored by summing items: higher scores indicate greater social support . Cronbach’s alpha has been previously reported as .85 . For this study, Cronbach’s alpha was .76.
SF-8 Health Survey
The SF-8 health survey is a shortened version of the SF-36, and is designed to assess 8 domains of health, each with a single item: physical functioning, role limitations due to physical health, bodily pain, general health perceptions, vitality, social functioning, role limitations due to emotional problems, and mental health . Each survey produces a psychometrically-based physical component summary and a mental component summary. The SF-8 is scored and normalized so 50 is the average score or norm. Scores well below average are problematic and indicate poor health . Cronbach’s alphas for this study were .66 (physical component summary), .80 (mental component summary), and .83 (total survey).
Tool to Measure Parenting Self-Efficacy (TOPSE Parenting Evaluation). The TOPSE is a self-report, paper and pencil measure designed to assess parents’ perceived ability to manage their children . The tool was originally created to evaluate education programs for parents. The TOPSE has 8 separate subsections comprised of 48 items, of which 6 are reverse scored. Respondents rate items using a 10-point Likert scale ranging from 0 (completely disagree) to 10 (completely agree). Items are summed to create a total score; higher scores indicate greater parenting self-efficacy. Cronbach’s alphas ranged from .69 to .85 for each of the subscales . Cronbach’s alphas for this study ranged from .62 to .91.
Data were examined for errors, outliers, and patterns of missing values. For the USPS missing values (up to 2 missing value per subscale) were replaced with the mean score for the individual participant on the specific subscale. This, however, was rarely employed, as there were few missing responses (0.6–5.7 %) on any of the subscales. Data were not markedly skewed for any measure. Significance was set at p < .05 for all statistical tests. All analyses were conducted using SPSS 19.0 software for Windows. Descriptive statistics for all measures were calculated. Using principal components method without rotation, we conducted factor analysis using Bartlett’s Test of Sphericity and scree plots on the items for each subscale of the USPS. Internal consistency reliability was assessed using Cronbach’s alpha and an alpha greater than .70 is deemed acceptable for a new scale . Pearson’s correlations were used to assess convergent validity between the target and validation measures . Cohen’s  guidelines were used to interpret the strength of the correlations (i.e., small = .10, medium = .30, and large = .50).
Descriptive statistics, reliabilities, and percentiles, for the Upstart Survey subscales
Upstart Survey Subscales
Number of items
Correlations between caregiver scores on parenting experience target items and baseline validation measures
Parenting experience items
Pearson’s r correlation
I have confidence in my parenting skills
TOPSE: Self acceptance
I feel positive in my role as a parent
Parenting Morale Index
I know who to call and where I can go in the community when I need help
TOPSE: Learning and knowledge
I know where I can get answers to my parenting questions
TOPSE: Learning and knowledge
I have someone to talk to when I need support
Parenting Morale Index
I am able to manage stress
SF-8: Mental Component Score
I know ways to meet my family’s needs with the money and resources I have
TOPSE: Control Subscale
My emotional health is good (that is, I do not feel anxious, depressed or irritated)
SF-8: Mental Component Score
I know how to “speak up” for what my family and children need
I feel supported by my partner in my parenting (if you parent alone, please check “not covered”)
Family Support Scale
In our family, we take the time to listen to each other
This study contributes to community-based research by presenting and evaluating a simple, efficient, program evaluation tool. The Upstart Parent Survey is a new measure of common outcomes expected to be achieved following participation in an effective parenting education program. In its first round of psychometric evaluation, the USPS demonstrated promise as a brief, easy to administer measure for prevention-focused parenting programs (P-FPP). The uni-dimensionality of the survey’s PK, PE, and PS subscales reinforced our hypothesis that each scale measured a single concept. The internal consistency reliabilities for each scale were strong for a new measure. Temporal stability over 2 weeks for the post-test scale scores was satisfactory.
Concurrent validity for most items was moderate to strong. Two variables, PK: I make time to read with my child everyday and PE: I know who to call and where to go in the community when I need help exhibited low shared variance. Further investigation of these two items illustrated that the PK item was worded inappropriately; following additional investigation the authors believe that “everyday” induced defensive responding by the participants. The PE item was double barreled; the item asks 2 separate questions in one statement. In order to address these weaknesses, these constructs will be corrected when an updated version of the USPS is drafted. There were very few missing values on the PK, PE, or PS subscales of the USPS. This suggested that parents found the USPS acceptable for providing information about themselves and their families. This also indicated that language and reading level of the survey were accessible to a culturally diverse population.
The current study is limited by parental perceptions of change related to the program. Parental perceptions of change may be biased. Future studies should include observational measures of parent–child interaction, or health and other social indictors such as validated cases of child maltreatment. The study was also limited by sample size of some of the components, and a sample that was primarily mothers. Future studies should increase the number of fathers sampled because parental responses between fathers and mothers may vary based on differences in paternal expectations.
Future versions of the USPS may wish to reverse score some of the items to prevent convenience response bias based upon high scores for all items being in the same column. This allows researchers to distinguish whether or not participants are appropriately reading each question. Although the majority of the sample population was Caucasian, there was enough ethnic diversity to permit initial generalization to multi-cultural regions. This, however, will require further investigation. Subsequent evaluation of the USPS will also include a traditional pre-test/post-test group that will be appropriately compared to a retrospective pre-test group. In addition, cognitive interviews with a small number of respondents will be carried out to ensure participants and evaluators are referring to comparable concepts.
Owing to time constraints and respondent burden, retrospective pre-test/post-test designs are a pragmatically desirable approach to capturing program evaluation of P-FPPs. Although proponents of the traditional pre-test challenge the ‘accuracy’ of the retrospective pre-test, evidence supports the validity of this type of assessment under certain circumstances . The results from this study indicate that the USPS may be a sufficient tool to evaluate effectiveness of the ability of P-FPPs to improve parenting knowledge and skills, and key outcomes associated with improved family functioning. As funding becomes increasingly dependent upon demonstration of effective programming, the need for rigorous but easy to adminster evaluation tools has become apparent. Despite its minor limitations, the UpStart Parent Survey offers prevention-focused parenting programs an efficient, psychometrically valid measurement tool that can be implemented in its current form.
We would like to thank Dr. L.A. Keown, Adjunct Research Professor, Carleton University, Ottawa, for statistical consultation. We would also like to thank the participants and program leaders for taking time from their busy schedules to participate in the study. We gratefully acknowledge funding from UpStart: Champions for Children and Youth, The United Way of Calgary and Area; and the Centre for Criminology and Justice Research through Mount Royal University.
This article is distributed under the terms of the Creative Commons Attribution License which permits any use, distribution, and reproduction in any medium, provided the original author(s) and the source are credited.
- 3.Karoly, L. A., Kilburn, M. R., & Bigelow, J. H., et al. (2001). Assessing costs and benefits of early childhood intervention programs: Overview and application to the Starting Early Starting Smart Program. Executive Summary. Seattle, WA: Casey Family Programs.Google Scholar
- 5.National Scientific Council on the Developing Child. (2010). The foundations of lifelong health are built in early childhood. Boston, MA: Center on the Developing Child at Harvard University.Google Scholar
- 8.Benzies, K., Trute, B., & Worthington, C., et al. (2010) Assessing psychological well-being in mothers of children with disabilities: Evaluation of the Parenting Morale Index and Family Impact of Childhood Disability Scale. Journal of Pediatric Psychology. Advanced Access: September 15, 2010.Google Scholar
- 10.Abidin, R. R. (1997). Parenting Stress Index: A measure of the parent–child system. In C. P. Zalaquett & R. J. Wood (Eds.), Evaluating stress: A book of resources (pp. 277–291). Lanham, MD: Scarecrow Press.Google Scholar
- 12.Alberta Centre for Child Family & Community Research. (2007). Common outcomes initiative: Roundtable final report. Edmonton, AB: Author.Google Scholar
- 16.3 Cheers for the Early Years. (June 12, 2009). Parenting Needs Assessment: Final Report. Calgary, AB: Author.Google Scholar
- 18.Skinner, H. A., Steinhauer, P., & Santa-Barbara, J. (1995). FAM-III manual. Toronto, ON: Multi-health Systems.Google Scholar
- 20.Dunst, C. J., Jenkins, V., & Trivette, C. M. (1984). The family support scale: Reliability and validity. Journal of Individual, Family and Community Wellness, 1, 45–52.Google Scholar
- 23.Ware, J. E., Kosinski, M., Dewey, J. E., et al. (2001). SF-8 Health survey manual—how to score and interpret single-item health measures: A manual for users of the SF-8 health survey. Lincoln, RI: Quality Metric Incorporated.Google Scholar
- 24.Kazdin, A. E. (2003). Methodological issues and strategies in clinical research (3rd ed.). Washington, D.C.: American Psychological Association.Google Scholar
- 25.Cohen, J. (1977). Statistical power for the behavioral sciences. New York: Academic Press.Google Scholar
- 26.Lamb, T. (2005). The retrospective pretest: An imperfect but useful tool. The Evaluation Exchange, 11(2), 18–19.Google Scholar