Quasicohort and Period Measures Are Weak Proxies for the New Federal Graduation Rate: Florida Data, 2003‐2009 Graduation Years Sherman Dorn February 23, 2012 This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 United States License. Citation: Dorn, S. (2012, February 23). Quasicohort and period measures are weak proxies for the new federal graduation rate: Florida DATA, 2003‐2009 graduation years. Available at SSRN: http://ssrn.com/abstract=2009965. Summary: Analysis of newly‐released data from the Florida Department of Education suggests that commonly‐used proxies for high school graduation are generally weak predictors of the new federal rate. In February 2012, the Florida Department of Education released data on high school graduation using the new, mandated federal definition of graduation rate (Florida Education Information & Accountability Services [FEIAS], 2012). This new definition requires states to calculate four‐year graduation measures using longitudinal data, following ninth‐grade students through graduation, continued attendance, attrition, or transfer. Because the report contains historical data from the 2003 graduating class through 2011, this report provides an opportunity to compare the longitudinal rate to proxy measures researchers have been using for more than a decade (see Warren & Halpern‐Manners, 2009, for a discussion of methodological issues). Some proxies have been “quasicohort” in using administrative records of enrollment and graduation across several years, with the assumption (or hope) of a strong relationship between reported cross‐sectional data and the data from following students longitudinally. One proxy is a period measure synthesizing a hypothetical cohort from cross‐sectional data in two successive years. This short paper is an initial comparison of the most commonly‐used proxy measures to the new federally‐defined rate. Dorn, “Weak Proxies for the New Federal Rate,” Feb. 2012, p. 1. Electronic copy available at: http://ssrn.com/abstract=2009965 Method Data As of early 2012, sufficient information from the Common Core of Data (CCD) is available to allow calculation of the proxy measures for the 2003‐2009 graduating years, and with 67 countywide school districts, this provides data for 469 county‐year combinations. Elements drawn from the CCD include enrollment in eighth through twelfth grades for the years before graduation and regular diplomas reported in the year of graduation. Data from FEIAS (2012) include the reported federal graduation rates for 2003‐2009. One small county’s 2008 data were excluded (Gilchrist) because of abnormally low diplomas reported, leaving 468 records used in this paper. Measures For each county‐year combination, the following quasicohort or period measures were calculated from CCD elements: Basic completion rate, 8th and 9th grade bases (hereafter BCS‐8 and BCS‐9; see Haney, Madaus, Abrams, Wheelock, Miao, & Gruia, 2004). This quasicohort measure divides the diplomas in the graduation year (t) by the eighth‐ or ninth‐grade enrollment five or four falls previously (t‐5 or t‐4). The recognized weakness of BCS‐9 is the prevalence of grade retention in 9th grade. As Warren and Halpern‐Manners (2009) note, neither BCS‐8 nor BCS‐9 adjust for student mobility. Averaged freshman graduation rate (AFGR; see Seastrom et al., 2006). This quasicohort measure also compares diplomas in graduation year t to an estimate of the starting cohort, and it attempts to adjust for grade retention by using an average of eighth‐, ninth‐, and tenth‐grade enrollments as the denominator, from the fall of years t‐5, t‐4, and t‐3, respectively. This measure does not adjust for mobility. Cumulative promotion index (CPI; see Swanson, 2004). This measure is a period measure, chaining one‐year quasicohort promotion ratios (e.g., tenth‐grade enrollment in year t divided by ninth‐grade enrollment in year t‐1) from ninth grade in year t‐1 to diplomas reported for year t. Dorn, “Weak Proxies for the New Federal Rate,” Feb. 2012, p. 2. Electronic copy available at: http://ssrn.com/abstract=2009965 Results Summary measures and bivariate relationships Tables 1 and 2 display summary statistics and a correlation matrix for these measures. The medians and means of BCR‐9 and CPI are closer to the federal rate median and mean than AFGR or BCR‐8, which are biased upwards against the federal measure. The standard deviation for the federal rate is smaller than the standard deviation for the proxy measures. The correlation matrix shows that AFGR and BCR‐9 have the highest correlations with the federal rate, but each of them still have only moderate associations with the federal definition, with an R2 of .54 and .53, respectively. Another way to explore the relationship between the proxy measures and the federal graduation rate for Florida 2003‐2009 is through Figure 1, where each panel shows a different comparison. Table 1. Summary statistics, graduation measures Fed Rate AFGR BCR-8 BCR-9 CPI Minimum 32.5% 31.5% 34.7% 23.9% 23.4% Maximum 85.2% 90.0% 98.3% 96.8% 98.3% Median 59.6% 67.1% 71.8% 58.5% 62.3% Mean 60.1% 66.6% 71.0% 58.5% 62.2% 8.2% 9.5% 10.9% 10.6% 11.7% SD Table 2. Correlation matrix, graduation measures Fed Rate AFGR BCR-8 AFGR .77 BCR-8 .70 .92 BCR-9 .75 .94 .78 CPI .61 .71 .66 BCR-9 .68 Dorn, “Weak Proxies for the New Federal Rate,” Feb. 2012, p. 3. 70% AFGR 50% 70% BCR‐9 90% 110% 90% Federal Graduation Rate Federal Graduation Rate 100% 90% 80% 70% 60% 50% 40% 30% 30% 50% 100% 90% 80% 70% 60% 50% 40% 30% 30% 50% 70% BCR‐8 90% Federal Graduation Rate Federal Graduation Rate 100% 90% 80% 70% 60% 50% 40% 30% 30% 100% 90% 80% 70% 60% 50% 40% 30% 30% 50% 70% CPI 90% Figure 1. Proxy measures vs. federal graduation rate. Each panel shows the bivariate distribution of measures for the specified relationship, with a OLS estimation line. See Table 2 for correlations. Residual analysis Because the proxies have been commonly used as substitutes for true cohort measures, it is also important to examine the residual errors for each proxy. Table 3 provides summary statistics of the residuals (no decimals displayed for percentages). BCR‐ 9 and CPI have the least bias (‐2% for BCR‐9 and 2% for CPI). The range for residual errors is greatest for CPI (78%) and lowest for AFGR (42%), and the same is true for standard deviation (9% for CPI as highest, 6% for AFGR as lowest). The skew is positive for each residual set, but the median is very close to the mean in each case. . Table 3. Summary statistics, residual error of graduation measures Minimum Maximum Median Mean SD Skew AFGR -17% 25% 6% 7% 6% 0.10 BCR-8 -10% 38% 11% 11% 8% 0.22 BCR-9 -31% 25% -2% -2% 7% 0.16 CPI -36% 42% 2% 2% 9% 0.32 Dorn, “Weak Proxies for the New Federal Rate,” Feb. 2012, p. 4. Discussion This paper analyzes the performance of commonly‐used proxy measures with the first public release of multiple years of graduation rates using the new federal definition that requires longitudinal tracking. In general, the proxy measures perform poorly, with two clearly‐biased measures (AFGR and BCR‐8), and with all measures having no better than moderate correlations with the new federal rate. At least using the data available for Florida from the state department of education and the Common Core of Data, all researchers’ attempts to improve on a basic quasicohort rate with administrative data (BCR‐9) have resulted in measures that are either more biased, or with lower correlations with the federal rates, or both. Several measures available using population data, to adjust for migration, have more promise, but these measures are not available at the school or district level (Warren & Halpern‐Manners, 2009). The conclusion is sobering but perhaps expected: you cannot measure your longitudinal graduation rate until you do so, and one should not accept any substitute, no matter how attractive. References Florida Education Information & Accountability Services. (2012, February). Florida’s federal high school graduation rates, 2010‐11. Tallahassee, FL: Florida Department of Education. Retrieved from http://www.fldoe.org/eias/eiaspubs/word/FedGradRate‐ 1011.doc Haney, W., Madaus, G., Abrams, L., Wheelock, A., Miao, J., & Gruia, I. (2004). The education pipeline in the United States, 1970–2000. Chestnut Hill, MA: National Board on Educational Testing and Public Policy, Boston College. Seastrom, M., et al. (2006). A review and analysis of alternative high school graduation rates. NCES 2006–604 and 605. Washington, DC: National Center for Education Statistics. Swanson, C. B. (2004). Who graduates? Who doesn't? A statistical portrait of public high school graduation, class of 2001. Washington, DC: Urban Institute. Retrieved from http://www.urban.org/publications/410934.html Warren, J. R., & Halpern‐Manners, A. (2009). Measuring high school graduation rates at the state level: What difference does methodology make? Sociological Methods & Research, 38(1), 3‐37. doi: 10.1177/0049124109339374. Dorn, “Weak Proxies for the New Federal Rate,” Feb. 2012, p. 5.