The Maternal and Child Health Bureau (MCHB) has
initiated a series of measures designed to increase accountability and improve
decision-making. One of these efforts is an assessment of the impact of discretionary
grants in the category of Special Projects of Regional and National Significance
(SPRANS), which comprise an array of demonstration, research, and training
grants.The evaluation of the Maternal and Child Health (MCH) Training Program
described in this document is part of this effort.
The National Center for Education in Maternal and Child Health (NCEMCH) at Georgetown
University was awarded a grant that included as one of its objectives the
development of a model for evaluating SPRANS projects. The first program to
be evaluated was the Healthy Tomorrows Partnership for Children Program.The
MCH Training Program is the second SPRANS program to be evaluated. The evaluation
focuses on the 13 long-term training priorities supported by MCHB; the continuing
education grants, which are quite different from the long-term priorities,
are not included in the study.
The training evaluation consisted of two phases. Because little had been written
describing the Training Program, the purpose of phase I was to chronicle the
program's history and development and to identify themes common to the 13
priorities. The product of phase I, Building the Future: The Maternal and
Child Health Training Programwas based on a review of
Training Program documents, interviews with current and former federal staff
associated with the program, and information obtained in focus groups from
grantees in seven of the training priorities. An important outcome was greater
clarity about the overall goals of the program. Phase II, the results of which
are presented here, is designed to broadly assess the program's accomplishments,
identify problems, and provide recommendations focused on program operations
MCHB and its predecessor
agencies have funded long-term training in maternal and child health since
the 1940s.The Children's Bureau
took a holistic approach to the care of children and families, viewing health,
social, and emotional needs as inseparable and equally important. This perspective
has permeated the MCH Training Program throughout its history. Another
hallmark of the program is its long-standing focus on vulnerable populations,
including children with special health care needs and underserved women, children,
Much more than just a mechanism to support the education of individuals, the
Training Program was designed to be a vehicle for national MCH infrastructure
building. Training Program grantees were to be key partners with the federal
government and the states in improving the health of women and children through
their work with professional associations, public agencies, and voluntary
organizations, and to constitute a ready and willing cadre of individuals
with expertise, dedication, and commitment to children.
THE TRAINING PROGRAM AND THE
The Training Program is a key resource for MCHB as it strives to address the following goals articulated in its strategic plan:
- Goal 1: Eliminate Barriers and Health Disparities: The Training Program promotes this goal through an educational focus on health disparities, development of outreach services for children and families who have poor access to health services,and policy work,such as service on advisory committees or task forces.
- Goal 2: Ensure Quality of Care: A major aspect of the Training Program is quality improvement in the provision of health services. Grantees develop practice guidelines, assist states and communities with evaluation, disseminate research findings to various communities, and provide quality training for a new generation of MCH leaders and practitioners.
- Goal 3: Improve the Health Infrastructure and System: Trainees are taught the value of comprehensive systems of care, cultural competence, and family-centered care. Many grantees function as local, state, and national advocates to improve the health care system.
Interrelationship of the Training and Block Grant Programs
The MCH Block Grant program and the Training Program represent
complementary approaches to addressing the health of women and children.
In addition, the Training Program both directly and indirectly supports
the Block Grant program. Examples of direct support include technical assistance
and continuing education provided by Training Program faculty for state
Title V programs, and examples of indirect support include activities such
as standards development, policy work, information dissemination, applied
research, and the education of a new generation of practitioners. The Training
Program thus enhances MCHB's ability to serve as a catalyst for change and
strengthens the context for the delivery of MCH services.
The MCH pyramid (Figure 1) is a graphical representation of the activities supported
by MCHB. It identifies the levels of services provided through Title V.
The Training Program is located in the base of the pyramid. Without this
foundation, the other MCH functions would be severely compromised. The base
consists of infrastructure-building services, including assessment and assurance
functions and training. These infrastructure-building services were noted
as critical areas of emphasis for public health programs in the landmark
study conducted by the Institute of Medicine, The Future of Public Health.
Interrelationship of Special MCH Initiatives and the Training Program
MCHB supports a number of special initiatives, and the alliances MCHB has established with universities through the Training Program are critical to the success of these initiatives. A few illustrative examples are provided below:
• Children with Special Health Care Needs: MCHB, in particular its
Division of Services for Children with Special Health Care Needs, works
to improve services for children with a variety of disabilities. States
receive MCH Block Grant funds to ensure that services are adequate and of
high quality. Several training grant priorities focus on children with special
health care needs. The interdisciplinary approach of priorities such as
LEND ensures that children with complex health and social needs receive
coordinated care from a variety of disciplines. Most LEND projects work
collaboratively with their state offices; in two states, the LEND program
actually administers the state program for children with special health
care needs. LEND grantees provide many of the experts (LEND program faculty)
who are equipped to treat and diagnose children with neurodevelopmental
disabilities, and LEND grantees deliver an array of clinical services within
most states, serving as a referral source for the state programs. LEND grantees
also provide community training and advocacy for special needs children,
supporting the work of MCHB and state offices.Other
Training Program priorities, such as pediatric pulmonary centers, behavioral
pediatrics, pediatric dentistry, communication disorders, and the occupational
and physical therapy projects, also focus on children with special health
- Office of Adolescent Health: Established in statute, MCHB's Office of
Adolescent Health strives to improve the health of the nation's adolescents
through special discretionary grants, policy work, support of Title V
programs in improving adolescent health, and interagency collaborationThe
LEAH grants are integral partners in this work. They provide technical
assistance to the state adolescent health coordinators, conduct research
that furthers the goals of the Office of Adolescent Health, and generate
policy documents that foster awareness of adolescent health issues. LEAH
grantees form the core of the Society for Adolescent Medicine, the key
professional association focused on adolescent health, and through this
association they advocate for new policies and treatment approaches to
benefit adolescentsThe Office of Adolescent Health's two
policy center grants are housed in the same university departments as
two of the LEAH grants; both the LEAH projects and the policy centers
are strengthened through the complementary activities of the two categories
- Crosscutting Initiatives: From time to time, MCHB supports initiatives
that cut across all its offices and divisions. The Training Program enables
MCHB to more effectively accomplish such initiatives, as shown by the
example of Bright Futures. Designed to improve quality of care for children
and their families, Bright Futures is a set of expert guidelines and a
practical developmental approach to providing health supervision for children
from birth through adolescence, and consists of a variety of tools for
health professionals, families, and communities. The Training Program
has been pivotal to the success of Bright Futures. For example, many of
the experts on the panels assembled to develop the Bright Futures guidelines
were either graduates of the Training Program or current faculty.In
addition, several training projects visited as a part of this evaluation
have fostered curriculum changes, both within their own universities and
nationwide, that build on the Bright Futures guidelines. The Training
Program has played a central role in Bright Futures, albeit one that has
gone largely unrecognized. A great strength of the MCH Training Program
over time has been its implicit recognition of the way in which social
change occurs through the synergy created by service providers, policymakers,
academics, and family members working in concert. The Training Program
is integral to this process.
GOALS OF THE MCH
The MCH Training Program addresses a diverse set of needs. And yet it has developed a cohesive set of goals that distinguishes it from other federal health training programs. The five goals of the MCH Training Program are to:
- train leaders;
- address the special health and social needs of women, infants, children, and adolescents;
- foster interdisciplinary care;
- change attitudes and practice (e.g., toward family-centered and culturally competent care); and
- emphasize the public health approach.
Prior to the publication of Building the Future: The Maternal and Child Health
Training Program,These goals had not been
clearly articulated in written documents.
The MCH Training Program aims to train a new generation of leaders
who can advocate for children and their families, provide quality clinical
services, teach, and conduct research. Leadership training is a strategy
chosen by MCHB to maximize the impact of a program with limited resources
relative to need. Although it could be argued that the MCH Training Program
has always trained leaders, this aspect of the program has recently become
explicit and more central to the MCH Training Program mission.
Addressing the Special Needs of Women, Children, and Adolescents
A key characteristic of the MCH Training Program is its focus on women, children, and adolescents. Historically, health professionals in a number of fields have not received adequate training in serving the special health and social needs of these populations, a situation that continues to the present day. The MCH Training Program is intended to address this gap.
Fostering Interdisciplinary Care
As children began to survive previously untreatable complications
of birth, and as once-fatal illnesses became treatable, some health care
providers turned their attention to the complex health and social needs
of children with chronic health problems. Single disciplines cannot address
the needs of many of the children who have special needs, and so, in the
1960s, an interdisciplinary model of care emerged from the experience of
the University Affiliated Facilities (later renamed as University Affiliated
Programs).This model fosters
collaboration among faculty and trainees from various disciplines as they
work together to address the multifaceted issues of children with special
health care needs. The MCH Training Program is currently one of the only
sources of support for this type of training.
Changing Attitudes and Practice
Quality health care services are community-based, family-centered,
and culturally competent.In addition,
health care should be coordinated, and health services should be integrated
with other systems that serve women, children, and families (including education,
justice, and social services). Noted in the Title V legislation that defined
the Children with Special Health Care Needs (CSHCN) program, these aspects
of service delivery have come to constitute core MCH values, and the Training
Program attempts to ensure that they are integrated in each training project
and that graduates of the Training Program reflect these values in their
Emphasizing the Public Health Approach
The MCH Training Program has attempted
to broaden the perspective of clinicians to an understanding of public health,
of preventing problems from occurring among population groups. The public
health approach recognizes that many health problems are rooted in the behavior
of individuals and in their social context and that the environment plays
a major role in health.In contrast
with the clinical medical approach, which explores the history and health
conditions that may have led to health problems in a single individual,
the public health approach focuses on identifying patterns among groups.
It has four basic steps: (1) clearly define the problem; (2) identify risk
and protective factors; (3) develop and test interventions; and (4) implement
interventions.The public health approac h is a rational and organized way
to marshal prevention efforts and ensure that they are effective.
Our faculty have learned about the public health perspective,
advocacy, cultural competency, and family-centered care. Faculty who come
from a clinical background were not trained with such a model.
This is at the forefront of exemplary practice.
Project director, Occupational Therapy
NEEDS ADDRESSED BY THE MCH TRAINING PROGRAM
Although the grantees of all 13 MCH Training
Program priorities incorporate the general goals of the Program in their
projects, the specific needs that individual priorities address vary considerably.
Although MCHB recently funded a graduate training and continuing education
needs assessment, it has not done so in the past rather, the priorities
have arisen over time in an ad hoc way. This evaluation found that training
needs, as reflected in the 13 priorities, are qualitatively different and
may be conceptualized in several ways, as follows:
Lack of racial and ethnic diversity: In the absence of racial and ethnic
diversity in a field, important issues may be overlooked in the provision
of services and quality of care may be compromised. To ensure access and
enhance quality, individuals from diverse backgrounds may need to be encouraged
to receive training and then supported financially, academically, and
emotionally. The Historically Black Colleges and Universities (HBCU) priority
is intended to increase the number of professionals from diverse backgrounds
providing primary care in community-based settings, with an emphasis on
the special needs of families of African-American and Hispanic descent.
Additionally, efforts must be made to increase cultural competency among nonminority MCH professionals.
• The scope and/or trajectory of a particular health problem: A health
problem may affect many people, be quite severe, and/or be dramatically
increasing in scope, and the resources to address it may be inadequate.
Asthma is an example: it is sometimes fatal, affects hundreds of thousands
of children, and is growing in prevalence.However,
the resources to address asthma are not commensurate with its scope. The
PPC training projects are working to understand and control asthma, along
with other significant pulmonary diseases. Dental disease in children, and
adolescent suicide, are other problems of great scope, ones that are preventable;
these are being addressed by the pediatric dentistry and adolescent health
• Lack of a doctoral-level professoriate: In some health fields, the master's degree is the terminal degree, and persons capable of effectively teaching trainees (i.e., persons with doctorates) are few. An example is the field of communication disorders. The master's degree is the certifying degree for practicing audiologists and speech/language pathologists. In a robust economy, there is little or no economic incentive for practitioners to pursue a doctoral degree.Alternatively, a field may experience a decrease in the number of doctoral-trained individuals, with universities then having difficulty recruiting qualified faculty for available positions. MCH programs in schools of public health, for example, report difficulties in finding and attracting appropriately trained faculty. Pediatric dentistry is another field with difficulty recruiting academics. Of those trained in pediatric dentistry, most tend to pursue private practice.
- Complexity of clinical problems: Some children are particularly difficult
to treat, especially those with multiple disabilities and/or illnesses.
An example is a child who is both autistic and blind. Such children typically
require the services of a variety of health care professionals who have
had special training, but these professionals may be in short supply.
A recent study found that pediatricians lack training in providing medical
care to children with special health care needs.Children
with special health care needs face not only complex clinical issues,
but often have social and educational needs that must be met as well.
Individuals trained in an interdisciplinary model that focuses on addressing
such complex needs in collaboration with other professionals (both health-
and non health-related) are well-suited to provide this type of care,
but may be even more difficult to find. The LEND priority addresses the
need for training specialists to work with children with neurodevelopmental
and related disorders.
- Special needs of subpopulation groups: Some population groups may be quite
large and have special needs that have gone unmet. This is the case with
adolescents. Adolescents have high rates of certain risk behaviors, such
as use of cigarettes, alcohol, and other drugs, and also high rates of
obesity and sexually transmitted dis-eases.And
yet, adolescent medicine is a relatively new subspecialty with few trained
practi-tioners.The LEAH projects train professionals
in several disciplines to serve adolescents and promote improvements in
- Perceived urgency of a problem: A health care problem may be viewed as
urgent, perhaps because new research has documented its prevalence or because practitioners in the field find that they confront it daily
and lack the resources or knowledge to address it. Behavioral problems
of children is an example: primary care practitioners are encountering
increasing numbers of children with mental health and/or behavioral problems,
such as attention-deficit hyperactivity disorder (ADHD) or depression,
but most have neither the knowledge nor the requisite skills to treat
children with these problems.The behavioral
pediatrics training projects aim to address this deficiency.
- Inadequacy of MCH content in basic training programs: Some professional
training programs are designed to educate generalists who can serve the
needs of a variety of patients or clients. However, these programs may
lack appropriate MCH content. Examples include the fields of social work,
occupational therapy, physical therapy, respiratory therapy, nursing,
and nutrition. The MCH Training Program priorities in these disciplines
address gaps in basic professional education.
The Training Program has evolved over time to address needs as they have emerged. As social and medical issues change, new needs may be identified.
FOCUSING THE EVALUATION
The MCH Training Program is both large and complex. To focus the evaluation and ensure that the most important questions were asked, an MCH Training Program logic model (Figure 2) was developed in collaboration with the project's advisory board. A logic model helps to clarify the theory of any program and elucidates presumed relationships among different levels of action. The Training Program logic model shows that the outputs of the Program include technical assistance, consultation, and continuing education; research; clinical services innovations; faculty development; curricular changes; and increased numbers of students receiving training in MCH. These outputs lead to a set of intermediate outcomes that include dissemination of knowledge to the field; increased knowledge of how to serve the health and social needs of the MCH populations; improved delivery of clinical care; and the training of leaders, all of which generate better-quality care, more-integrated systems, and more-informed policy decisions, with the ultimate outcome of improved health for families.
Using the logic model as a guide, a set of evaluation questions was developed
again in collaboration with the advisory board and a methodology appropriate
to each question was identified.
Issues selected for analysis included the ways in which resources are utilized
by training projects; the types of activities supported by the Training
Program; the experiences of beneficiaries of the Training Program, including
trainees and recipients of continuing education and technical assistance;
the perceived impact of training projects on trainees; the ways in which
projects are integrated into trainees universities; and policy and administrative
issues of potential interest to MCHB.
The methodologies selected for the evaluation included a review of the FY 1999 continuation applications for all 101 projects (record review); site visits to 31 training projects with interviews of multiple individuals at each site; focus groups with state Title V program staff and federal regional MCH consultants; and telephone interviews with 110 trainees who graduated from the training projects in either 1990 or 1995. Each of these methods is briefly described below. Technical documents, including questionnaires and other data collection instruments, will be posted on the NCEMCH Web site (http://www.ncemch.org/spr/default.html# mchbtraining).
A review of the FY 1999 continuation applications for all 101 long-term training grants was undertaken first. This provided evaluation staff with an in-depth understanding of the program and was used to collect data that could be aggregated across projects. In addition, the record review allowed the evaluation staff the opportunity to review information from all projects, not just those that were site-visited. Data collected from the record review guided the development of protocols for the site visits; an effort was made to solicit only that information not available from materials that grantees had already provided to MCHB. General topics on which information was collected included the following:
- Administrative and organizational structure
- The educational program
- Demographic and other information on both current and former trainees
- Continuing education activities
- Technical assistance services, including policy work
- Research and publications of faculty
A form was developed to record quantitative data abstracted from continuation applications so that the data could be aggregated across the projects. However, a number of problems were apparent in this aspect of the study: (1) The variability in the types of training provided in different projects means that the validity of cross-category aggregations are suspect at best; and (2) there is no consistency in definitions among projects, even within the same priority, and thus aggregating data, particularly on such variables as the number of individuals receiving technical assistance or continuing education, is problematic. Nevertheless, this analysis represented the first time that all projects had been systematically reviewed for this type of information, and it provided a snapshot of aspects of the entire program at one point in time.
All data collected were stored in a database in FileMaker Pro for Mac OS Version 4.1 (Claris Corporation, Santa Clara, CA). Quantitative data were analyzed using SPSS for Windows, Release
10.1 (SPSS Incorporated, Chicago, IL), whereas narrative responses were summarized and examined for patterns in FileMaker Pro.
Site visits to training projects were undertaken in order to collect information on the major themes that emerged in phase I of the evaluation, to probe for additional information, and to provide an opportunity to interview beneficiaries of the projects. In order to ensure that the full scope of the Training Program was adequately reflected in the site visits, a set of criteria was developed to guide the selection of projects. The criteria included:
- geographic diversity;
- projects at publicly as well as privately funded universities;
- projects located in universities with multiple MCHB training grants, as well
as those in universities with only a single grant; and
- projects that have been funded for a long period of time, as well as those that
were more recently funded. In addition, projects representing each of
the 13 priorities were included, and priorities with the greatest dollar investment
by MCHB were over-sampled.
The site visits provided rich and in-depth information about the projects. A potential weakness of the site visits was the necessity of relying on the project directors to identify interviewees. Thus, there may be an inherent bias towards a favorable view of the projects. Nevertheless, the fact that many individuals at each site were interviewed enhanced the validity of the findings. Site visits are one of the best methods for developing a clear picture of a project.
Thirty-one training project sites were visited over the course of 8 months. (See Appendix B for a list of site-visited projects and project directors.) During the visits, interviews were conducted with the project director, dean and/or department chair, faculty, current trainees and recent graduates, and recent recipients of continuing education and/or technical assistance. Interview protocols were developed for each category of interviewee.
Data gathered during the site visits were stored in a FileMaker Pro database,
and narrative site visit reports for each site were prepared describing
the team's findings.
Title V Focus Groups
Because Title V agencies should be key partners of training projects, the experiences of Title V directors in working with faculty and trainees of training grants were explored through a series of focus groups. An in-person focus group was conducted with five state Title V directors at the 2000 meeting of the Association of Maternal and Child Health Programs (AMCHP). Because the project budget did not permit additional in-person focus groups with other state Title V program staff, telephone focus groups were substituted. Title V directors (both state MCH and CSHCN directors) representing 6 of the 10 Health Resources and Services Administration (HRSA) regional offices (regions I, IV, V, VII, VIII, and IX) participated in these calls, along with federal regional office staff. Because most focus group participants in this study knew each other and were accustomed to meeting via monthly telephone conference calls, this approach may have been as fruitful as an in-person focus group.
Focus groups are an effective way to obtain the opinions of several individuals on a broad array of topics. The ability of participants to build on each other's ideas stimulates thinking and tends to result in comprehensive information. Thus, focus groups have become an important qualitative method for obtaining opinion-based information.
Narrative data from the focus groups were summarized and analyzed for patterns.
Interviews of Former Trainees
A significant outcome of the training projects is the trainees who complete the programs. Consequently, the evaluation included an appraisal of former trainees perceptions of the impact of the Training Program on their professional development. In particular, the study attempted to determine whether trainees who completed the program either 5 or 10 years ago believe that they have become leaders and whether they attribute their success as a leader to the training they received.
A sample of 423 former trainees across 12 training priority
areas was generated. Former trainees from the HBCU priority were excluded
from the sample, as this category of grants does not financially support
long-term trainees. Details of the sample selection process are included
in Appendix C. The former trainees were contacted to either participate
in a brief telephone interview or to provide written responses to the interview
questions, which were mailed to them. Nonrespondents were followed up on
with a postcard and multiple telephone calls. A total of 110 interviews
were completed, yielding a 26 percent overall response rate, and a 35 percent
response rate among trainees for whom addresses and/or telephone numbers
were presumed valid.
Both quantitative and qualitative data were obtained from the interviews and stored in a FileMaker Pro database. Quantitative data were analyzed in Stata statistical software release 7.0 (Stata Corporation, College Station, Texas), whereas the narrative data were summarized and examined for trends in the FileMaker Pro database.
The interviews provided the perceptions of the respondents regarding the extent
to which they currently exercise leadership and their assessment of the
impact of the Training Program on their careers. Budget constraints precluded
the use of additional methodologies to further verify the former trainees
beliefs about these issues. However, the perceptions of the individuals
most directly affected by the Training Program provide strong evidence of
This report presents the findings of an evaluation of a large, complex, and multifaceted program. Several qualitative methods were used to describe and analyze the program. Study methods selected were those most appropriate to the particular questions being addressed. The use of multiple methodologies helped validate the findings from each individual method. Findings are presented in the following chapters of this report, along with a set of recommendations designed to improve the MCH Training Program and help it accomplish its mission.