<< Return
to Outcome Measurement System
Which performance measure goals
are model-based?
For PY 2008, the following measures in the Center
and CTS Report Cards use model-based goals:
-
HSD/GED attainment rate (Centers)
-
Literacy gain rate (Centers)
-
Numeracy gain rate (Centers)
-
Graduate average wage rate (Centers and CTS Agencies)
-
Graduate average weekly earnings at 6 months after initial placement (Centers
and CTS Agencies)
Return to top
Why are goals model-based for
some measures?
The primary rationale for using models to set
individualized goals is to ensure equity in making comparisons of performance
across centers and agencies by statistically adjusting for circumstances that
are beyond the operator's control. A model helps to ensure
fairness by formally recognizing that actual performance on a given outcome
measure depends not only on management practices, but also in part on the
abilities of the students it receives and the characteristics of the
environment in which it operates. By setting individualized
goals that adjust for differences in key factors that are beyond the operator's
control, this helps to "level the playing field" in assessing performance.
For example, using a model-based approach, centers that serve relatively
more students that are placed in high-wage labor markets would be assigned a
higher graduate average initial wage goal than centers that serve students that
are predominantly placed in low-wage labor markets.
Return to top
What is a model?
To determine the individualized center or agency goals for
each measure, a multiple regression model is estimated that relates the outcome
measure with various factors that affect the outcome measure but that are
beyond the control of the center or agency. The regression model essentially
estimates the effects of each of the different factors on the outcome measure,
holding all other factors constant. Then, by summing up the effects of each of
the factors, the model determines the total adjustment to be made in setting
the center's or agency's individualized goal.
Models are estimated using the most recent data available for the outcome
measures and for each of the factors used in the model. The PY 2008 models for the
three in-program measures used data for students who terminated between March 1, 2007
and February 29, 2008; the model for the graduate average wage rate measure used data
that was received by the Job Corps Data Center between March 1, 2007 and February 29, 2008;
the model for the 6-month average weekly earnings measures used data for placed graduates
who completed the appropriate post-program follow up survey between March 1, 2007 and February 29, 2008.
Return to top
What data sources are used to develop measures of factors included in the
models?
The data sources used to develop measures of factors and
the types of factors that are measured from each source include:
-
SPAMIS
-
Student's age at entry (Center models)
-
Student's age at termination (Agency models)
-
Student's education at entry (Center models)
-
Student's education at termination (Agency models)
-
Student's initial TABE scores (Center models)
-
Student's termination TABE scores (Agency models)
-
Student's vocational program
-
BLS Covered Employment and Wages
-
Average annual wage in all industries in the county the student is placed
-
US Census Data
-
Percent of families living in poverty in the county the student is placed
-
Center GED/HSD Survey
-
Percent of students at centers with significant pre-test barriers to GED attainment, including test score requirements
- Percent of students with enrollment age 16 and ineligible to take the GED test
- Percent of students at centers with significant retest barriers to GED attainment
-
Other
-
Percent of students placed in a state with a minimum wage that is higher than
the federal minimum wage
Return to top
How are the factors
calculated?
The factors are calculated as an average of all students
in the specific pools for each measure. Thus, for the HSD/GED attainment rate
measure, the factors are calculated among all students who did not have a GED
or high school diploma at entry. For the graduate average wage model, the
factors are calculated among all graduates with job placements. Because the
measures are calculated over specific pools of students, the values will not
generally match the information readily available to centers in their MIS
systems. Moreover, because the pools change across measures, the average value
of the same factor for a given center is likely to be different across
measures.
Return to top
How are the specific factors in the
model chosen?
The specific factors that are included in each final model
are based on extensive analysis, as well as a review by a work group comprised
of Job Corps program staff at the national, regional and center levels. Several
criteria are considered in selecting the specific factors for each model,
including:
-
Data must be available and consistently measured for all students in the pools.
-
The factor must be viewed as primarily beyond the control of the center
operator.
-
The factor should exhibit a statistically significant relationship with the
outcome measure.
-
The relationship between the outcome measure and the factor should meet a
common sense test (i.e., older students are more likely to receive
higher-paying jobs than younger students).
-
Minimizing complexity or ensuring ease of understanding the model should be
considered when selecting factors.
Return to top
What is a worksheet?
The worksheets provide a convenient summary of the results
of the models in setting each center's or agency's individualized goal for each
measure. All of the factors included in the PY 2008 model are listed in the
column entitled "Local Adjustment Factors." Column (2) shows the national
averages for each of the factors for all students who are in the pools. Column
(4) displays the regression weights estimated from the model that show the
effect of a given change in a specific factor on the specific outcome measure.
The adjustments that are made for differences in local factors are calculated
using the steps below:
-
The center's or agency's average values for each factor are entered in Column
(1).
-
To determine how different the center or agency is from the national average on
each factor, the entries in Column (2) are subtracted from the entries in
Column (1) and the difference is entered in Column (3).
-
The values in Column (3) that indicate how different the center or agency is on
each factor are multiplied by the weight for that factor entered in Column (4)
that shows how much changes in the factor affect the outcome measure of
interest, and the product is recorded in Column (5).
-
Sum each of the values in Column (5) and record the sum in the "Subtotal" cell,
which indicates the total adjustments that should be made because of
differences in characteristics of the center or agency from the national
average.
-
Add the total adjustment in the "Subtotal" cell to the national goal to obtain
the center's or agency's model-based goal on the measure.
Return to top
What if I have any questions?
If you have additional questions concerning your
model-adjusted goals, please contact Chris Conboy at (202) 693-3093 or Terry
Johnson at (206) 528-3113.
Return to top
|