Comparing Maternal Child Health Problems and Outcomes Across Public Health Nursing Agencies
- First Online:
- Cite this article as:
- Monsen, K.A., Fulkerson, J.A., Lytton, A.B. et al. Matern Child Health J (2010) 14: 412. doi:10.1007/s10995-009-0479-9
- 209 Views
To use aggregated data from health informatics systems to identify needs of maternal and child health (MCH) clients served by county public health agencies and to demonstrate outcomes of services provided. Participating agencies developed and implemented a formal standardized classification data comparison process using structured Omaha System data. An exploratory descriptive analysis of the data was performed. Summary reports of aggregated and analyzed data from records of clients served and discharged in 2005 were compared. Client problems and outcomes were found to be similar across agencies, with behavioral, psychosocial, environmental and physiological problems identified and addressed. Differential improvement was noted by problem, outcome measure, and agency; and areas for enhancing intervention strategies were prioritized. Problems with greatest improvement across agencies were Antepartum/postpartum and Family planning, and least improvement across agencies were Neglect and Substance use. Findings demonstrated that public health nurses address many serious health-related problems with low-income high-risk MCH clients. MCH client needs were found to be similar across agencies. Public health nurse home visiting services addressed important health issues with MCH clients, and statistically significant improvement in client health problems occurred consistently across agencies. The data comparison processes developed in this project were useful for MCH programs, and may be applicable to other program areas using structured client data for evaluation purposes. Using informatics tools and data facilitated needs assessment, program evaluation, and outcomes management processes for the agencies, and will continue to play an integral role in directing practice and improving client outcomes.
KeywordsNeeds assessmentOutcomesEvaluationOmaha SystemInformatics
Economic and technological trends are driving forces in current health care system dynamics. Mandates to demonstrate health improvement, show cost effectiveness, and deliver data and information electronically underlie efforts to evaluate and report client outcomes in public health agencies . The emerging field of health care informatics offers solutions such as analysis of computerized documentation systems to address these issues. Public health agencies are adopting computerized documentation systems with the goal of using information to inform and improve practice . Early adopters of computerized documentation systems are beginning to re-use data for needs assessments and program evaluation purposes. This paper describes such a project completed by four Minnesota county public health agencies that purchased and implemented computerized documentation systems in the late 1990s. The agencies jointly created practice standards, and developed and implemented a data quality support infrastructure. They developed tools to address gaps in practice and documentation quality: client assessment standards, care plans that articulated best practices, and outcome rating guides. These data quality efforts laid the groundwork to test the feasibility of determining maternal and child health (MCH) problems and outcomes, and comparing data across counties .
The scope and outcomes of MCH services are challenging to understand and measure. Limited data are available for individual public health agencies , and no previous literature or examples are available for comparing data across public health agencies. This project provides an example of how aggregated data from computerized documentation systems can be used to identify needs of MCH clients served by county public health agencies, and demonstrate outcomes of services provided.
The conceptual framework for this data comparison study was Donabedian’s health care quality framework, and its structure, process, and outcome concepts . For this study, structure was operationalized as the data and practice quality infrastructure for public health nursing practice; process was operationalized as the data reporting format developed to share data; and outcome was operationalized by comparing final ratings, and amount and significance of change scores across agencies to describe MCH client population needs and client outcomes.
Four Minnesota county public health departments participated in this project. The four counties are geographically contiguous, with Counties One, Three and Four being primarily suburban; and County Two being primarily urban. Each of the counties offered home visiting services to low-income, high risk MCH clients.
Some differences in home visiting client eligibility existed between the county programs. County One served first-time pregnant and parenting women through pregnancy and the first year of the infant’s life. County Two served any low income, high risk parenting client referred, and in addition had a special program for pregnant and parenting adolescents. The focus of the program was to keep adolescents in school to foster their wellbeing and future economic independence. County Three provided services to any low income high risk parent. County Four provided services to any low income high risk parent, with a separate service delivery model for clients receiving child protection services at the time of referral. Demographics of the clients served in each county were not reported, however all clients met program eligibility requirements, including low income (within 200% of poverty) and/or high risk (e.g. single parent, substance use, mental illness) criteria. County Two had substantially more racial and ethnic diversity (“white alone” = 79.5%) than the others (“white alone” range = 92.9–94.9%) .
This exploratory, descriptive study analyzed structured data that had been aggregated, analyzed, and reported in the public domain. In compliance with HIPAA and the Minnesota Data Practices Act, only de-identified and aggregated data were reported by the agencies. The study was conducted by the participating agency partners using only the published data; thus, individual level data were not available. This is a common agency practice for protecting privacy, but it provides challenges for outcome evaluation. The data analyzed by the agencies originated in individual client records. Each individual client (including children) in the sample had a unique client record. Public health nurses conducted comprehensive client assessments using a standardized, problem-oriented framework (the Omaha System ); and recorded assessments and service interventions in the agency’s computerized documentation system. All clients served in 2005 were included in the agency analyses. Personnel in each agency uniformly extracted, aggregated, and analyzed data using descriptive and inferential statistical methods. First, the agencies conducted a descriptive analysis of the frequencies of problems addressed. Then, paired samples t tests were conducted to evaluate the significance of client improvement by problem for each agency. The maximum number of clients with a given health problem was 1101 (County Two, Growth and development problem) and the minimum number of clients for any given health problem was 3 (County One, Residence problem). Agencies developed a standard tabular format for reporting Omaha System data (see Appendix Table 5) which facilitated the present study’s descriptive analysis.
Instrument: The Omaha System
Summary of Omaha System components and data points used for public health nurse documentation in the four local public health agencies
Component and purpose
Example for caretaking/parenting problem
Problem Classification Scheme assessment instrument to identify actual or potential client problems
42 problems in 4 domains (Environmental, Psychosocial, Physiological, and Health related behaviors), with associated mutually exclusive signs and symptoms. Problems may be identified at the individual, family, and community levels
Signs/Symptoms difficulty providing physical care/safety, difficulty providing emotional nurturance, difficulty providing cognitive learning experiences and activities, difficulty providing preventive and therapeutic health care, expectations incongruent with stage of growth and development, dissatisfaction/difficulty with responsibilities, difficulty interpreting or responding to verbal/nonverbal communication, neglectful, abusive, other
Intervention Scheme intervention descriptions to communicate and document problem-specific actions
Each intervention has three defined sections problem (focus of the intervention), category (action of the intervention), and target (additional information about the intervention). One undefined section allows for added specificity
Caretaking/parenting—case management—Caretaking/parenting skills-parenting classes;
Caretaking/parenting—teaching, guidance, and counseling—medical/dental care—routine preventive care (examples of typical intervention pathways for Caretaking/parenting and other MCH problems are available on-line at http://omahasystemmn.org/KBS_care_plans.htm)
This supplemental rating guide for the Caretaking/parenting problem is used in the four local public health agencies to support data reliability when documenting knowledge, behavior, and status ratings
No knowledge about appropriate parenting practices for age; believes inaccurate information
Minimal knowledge of appropriate parenting practices for age, but not aware of need to prioritize child’s welfare over personal gratification
Able to identify some appropriate parenting practices for age, some understanding of prioritizing child’s welfare
Knows appropriate parenting practices for child’s age
Knows appropriate parenting practices for age, and ways of mediating the environment for the child’s benefit
Does not provide physical care, relates to infant or child in indifferent or hostile manner. Inappropriate discipline for age
Provides minimal physical care. Parenting not contingent upon cues from infant or child. Rarely disciplines appropriately for age
Provides adequate physical care some of the time, sometimes shows nurturing behavior. Inconsistently disciplines appropriately for age
Provides adequate physical care some of the time, often shows nurturing behavior. Usually disciplines appropriately for age and situation
Provides adequate physical care and nurtures consistently, parenting contingent upon cues. Discipline appropriate for age and situation
Consistently anxious/negative about parenting. Parental rights in danger of termination due to abuse or neglect
Frequently anxious/negative about parenting responsibilities. Expectations rarely appropriate for age
Expresses some positive feelings about parenting responsibilities. Expectations sometimes appropriate for age
Usually positive about parenting responsibilities. Expectations usually appropriate for age
Enjoys parenting. Engaged in children’s lives. Expectations appropriate for age. Child thriving
Descriptive methods were used to analyze a data report created and published by the four agencies. The overall rank order of problems was achieved using all problem frequencies by agency. For example, the Growth and development problem ranked 1st or 2nd for all agencies, so it was ranked first over all, and the Neglect problem was ranked 8th, 9th, and 10th for all agencies, so it was ranked last over all. The Income problem and Family planning problem were closely ranked. Income was ranked 3rd, 4th, 6th, and 9th, while Family planning was ranked 3rd, 4th, and 6th. Because one agency ranked Income as the 9th problem, Income ranked lower than Family planning overall.
Due to the inability to disaggregate the data, outcomes evaluation was conducted in a similar fashion, using the original agency analysis. Significance of improvement for each problem was based on the original agency t test results. To compare problem-specific improvement across agencies, the problems were grouped into thirds (top, middle, bottom) for each agency by the amount of improvement for each outcome.
Descriptive analysis results of rank order and frequency ranges of Omaha System problems for all agencies
Growth and development
1, 2, 3, 4
2, 3, 5
3, 4, 6
3, 4, 6, 9
5, 6, 7, 8
5, 7, 10
6, 7, 9
6, 8, 10
8, 9, 10
Descriptive analysis results of differential improvement by knowledge, behavior, and status outcome, Omaha System problem for problems that consistently improved most and consistently improved least
Most improved problems
Growth and development
Least improved problems
Differences by Outcome Measure
Problem-specific outcomes for knowledge, behavior, and status varied (see Appendix Table 5). Knowledge ratings on admission were lowest for the Income problem and highest for the Caretaking/parenting problem, Knowledge ratings on discharge were lowest for the Emotional stability problem and highest for the Antepartum/postpartum problem. The range in knowledge improvement for all problems and agencies was 0 for the Neglect problem to 1.15 for the Substance use problem.
Behavior ratings on admission were lowest for the Substance use problem and highest for the Growth and development problem. Behavior ratings on discharge were lowest for the Substance use problem and highest for the Growth and development problem. The range in behavior change for all problems and agencies was −0.20 for the Neglect problem, to 1.1 for the Family planning problem.
In the Omaha System, there is a unique set of signs and symptoms for each problem. When a client exhibits at least one sign or symptom for a problem, the status rating is 4 or less, and it is called an actual problem. Clients who do not exhibit signs or symptoms but are at risk for problems receive a status rating of 5 for that problem, and it is called a potential problem. To understand the prevalence of the actual problems and the effectiveness of services provided for actual problems, the counties conducted their analysis of status outcomes using the subset of actual problems (problems with status ratings of 4 or less upon admission to services). The number of clients in the subset for each problem is noted in the status section of the outcomes table (see Appendix Table 5). The ratio of actual problems to the total was relatively higher for Income, Family planning, Emotional stability, and Substance use; and relatively lower for Growth and development, Abuse, and Neglect.
Status ratings on admission were lowest for the Family planning problem, and highest for the Neglect problem. Status ratings on discharge were lowest for the Income problem and highest for the Antepartum/postpartum problem. The range in status change for all problems and agencies was 0.24 for the Substance use problem to 1.88 for the Family planning problem.
Differences by Problem
Outcomes varied by problem. For example, there was consistently high improvement for the Antepartum/postpartum problem, and consistently low improvement for the Neglect and Substance use problems. Knowledge improvement for Caretaking/parenting was similar to knowledge improvement for Growth and development for all agencies, with differences between the two knowledge ratings ranging from 0 to 0.14. Income and Residence, are typically thought of as psychosocial and environmental problems, but are known to be determinants of health. For Income, client status improved for 32.3–100%. For Residence, client status improved for 33.3–66.7%. Mental health status improved for 39.9–52.6% of clients. Family planning and Substance use are both health-related behaviors. For all agencies, Family planning behavior and status showed more change than Substance use, both in percent of clients who improved, and in the amount of change. In most agencies, Abuse and Neglect problems tended to have disproportionately higher status ratings compared to knowledge and behavior ratings; possibly due to shelter or foster care placement for clients with actual problems.
Differences by Agency
With only four agencies, it was not possible to statistically evaluate differences between agencies. Counties Two and Four were most similar in rank order of problems, rating scores, and percent improvement. The ratings from County Three were higher than the other three: in County Three, 47% of discharge ratings were greater than 4, compared to 3% for County Two, and 20% for Counties One and Four. To compare problem-specific outcomes across agencies, the problems were grouped into thirds (top, middle, bottom). Each agency had 1–2 differences out of the 10 possible problems from the overall groups.
The objectives of this exploratory, descriptive study were to use aggregated data from computerized documentation systems to identify needs of MCH clients served by county public health agencies, and to demonstrate outcomes of services provided. The results provided valuable information regarding which needs are being met by public health nurses during home visits and which needs were not met by the interventions. This study demonstrates that structured data can be used to inform and improve MCH services and to demonstrate program effectiveness. The agencies collaborating in this study mutually created and implemented data quality standards in advance of the data collection , increasing confidence in the results .
Examining overall problem frequencies demonstrated that MCH clients in each county had similar health needs; and that agencies addressed these important needs. The data comparison confirmed that Antepartum/postpartum, Caretaking/parenting, and Growth and development were the most common problems addressed. In the Omaha System, services to pregnant and parenting clients and their children are described using these three problems, respectively. Therefore, these three Omaha System problems could be used to describe services and outcomes in any program serving the MCH population. In addition, results showed that agencies were also providing frequent services for clinically important problems such as Mental health, Substance use, Abuse, and Neglect, demonstrating a need for these services within the MCH client population for each county.
Outcomes evaluation showed significant improvement in 84% of problems addressed, indicating that public health nursing visits may positively affect client knowledge, behavior, and status related to these health issues. This is consistent with positive results demonstrated by programs of research studying the effectiveness of public health nurse home visiting programs with low income, high risk parents [13, 14]. In general, knowledge change scores for all problems in all agencies tended to be greater than behavior change scores. Differential improvement was noted by outcome, problem, and agency, and further study is necessary to determine the reasons for these differences. For example, program eligibility differences across counties, such as serving only first time pregnant women or serving any low income, high risk parent, may explain variation in client problems, baseline rating scores, and outcomes.
The Substance use and Neglect problems showed least improvement in outcomes across counties. The findings suggest that agencies should review their intervention protocols for these challenging issues, ensuring that best practices are being implemented in response, and implementing intervention changes if indicated. A repeat data analysis following intervention change would illuminate the potential improvement in intervention effectiveness.
The Omaha System problems included in this analysis were the most frequently addressed with clients. Other problems such as Nutrition,Sanitation, and Interpersonal relationship were also addressed with clients by public health nurses, but they occurred less frequently, and thus were not reflected in the limited data report used for this study. Public health nurse intervention effectiveness research to date has failed to capture the scope of practice of public health nurses with MCH clients, instead focusing on the intervention as “visit condition” . Public health nurse home visiting interventions are diverse, complex, and tailored to meet specific client needs , as indicated by the differences in problem frequencies between agencies. Structured data generated by computerized documentation systems have potential to illuminate the relationship between interventions and improvement in client outcomes.
Program evaluation findings based on observational data cannot show causation of outcomes . Rather, programs can report change for clients who received services, acknowledging that agency services are not the only influence on client outcomes. There are many possible explanations for the observed changes, especially in community settings. There is potential for variation in outcomes due to such factors as differences in program eligibility, program services, and client populations. Mutually applied data selection parameters would improve understanding of between agency differences. For example, selecting data for a particular client group such as all first time parents, and reporting associated demographic information and length of services would improve data interpretation. Some software vendors have begun the process of programming Omaha System data outcome reports so that data may be selected by multiple criteria in order to facilitate comparison of population-specific outcomes.
The increasing availability of computerized documentation data sets will foster opportunities to advance intervention effectiveness and outcomes research. This preliminary study shows that such data sets are useful to describe MCH problems and intervention effectiveness across agencies, and that results of data analysis can be used to direct public health practice.
Four Minnesota county public health departments successfully developed and implemented a formal standardized classification data comparison process using structured data. An exploratory descriptive analysis of the data was performed. Results showed that public health nurses address many serious health-related problems with low-income high-risk MCH clients. Based on the results, the agencies will evaluate intervention strategies for the problems with least improvement. The data comparison process described in this article would be applicable to any program or client group with modifications specific to particular demographic and program characteristics. Use of informatics technology and a standardized classification facilitated needs assessment, program evaluation, and outcomes management processes for the agencies, and will continue to play an integral role in directing practice and improving client outcomes.
The authors wish to acknowledge the administrators and public health nurses in each of the four counties for their dedicated services to families, and for their support and contribution to the project.