Arizona’s School Accountability System 2008 Technical Manual Tom Horne Superintendent of Public Instruction Arizona Department of Education Volume I: AZ LEARNS Achievement Profiles Research & Evaluation Arizona Department of Education Robert Franciosi, Deputy Associate Superintendent Achieve@azed.gov Published by the Arizona Department of Education, January 2009. The Arizona Department of Education of the State of Arizona does not discriminate on the basis of race, religion, color, national origin, sex, disability or age in its programs, activities or in its hiring and employment practices. If you have questions or grievances related to this guideline, please contact the Administrative Services DAS at (602)542-3186. Table of Contents 1. INTRODUCTION ...................................................................................................................................................3 WHAT’S NEW FOR THE 2008 AZ LEARNS ACHIEVEMENT PROFILES ......................................... 3 2. OVERVIEW OF THE AZ LEARNS EVALUATION SYSTEM ........................................................................4 GENERAL PROCESS TO CALCULATE AN ACHIEVEMENT PROFILE ................................................. 5 3. TIMELINE ............................................................................................................................................................... 6 TIMELINE ..................................................................................................................................... 6 4. CALCULATION OF AIMS STATUS AND GROWTH POINTS ......................................................................7 AWARDING STATUS POINTS ......................................................................................................... 7 AWARDING IMPROVEMENT POINTS .............................................................................................. 8 NEW COMPUTATION OF THE AVERAGE ANNUAL CHANGE IN DIFFERENCE OF ACHIEVEMENT ..... 9 CALCULATION OF SUBJECT/GRADE SCALE POINTS FROM AIMS ............................................... 10 SPECIAL CASES .......................................................................................................................... 11 5. THE MEASURE OF ACADEMIC PROGRESS ................................................................................................ 13 THE GROWTH INDEX .................................................................................................................. 13 CALCULATING EXPECTED GROWTH ........................................................................................... 13 AWARDING AZ LEARNS POINTS ............................................................................................. 14 SCHOOLS WITHOUT MAP ........................................................................................................... 15 6. GRADUATION AND DROPOUT RATES ......................................................................................................... 16 GRADUATION RATE ................................................................................................................... 16 DROPOUT RATE ......................................................................................................................... 17 7. EVALUATING PERFORMANCE ON THE ENGLISH LANGUAGE ASSESSMENT ............................... 18 8. EVALUATING THE TOTAL SCALE SCORE VALUE TO DETERMINE A SCHOOL CLASSIFICATION ................................................................................................................................................... 19 9. APPLICATION OF THRESHOLD CRITERIA FOR EXCELLING AND HIGHLY PERFORMING SCHOOLS .................................................................................................................................................................. 21 10. AZ LEARNS ACHIEVEMENT PROFILE APPEALS PROCESS ................................................................ 25 PROCEDURE AND TIMELINE ....................................................................................................... 25 SPECIAL APPEALS FOR 2008 ...................................................................................................... 28 11. AZ LEARNS ACHIEVEMENT PROFILES FOR ALTERNATIVE SCHOOLS ........................................ 29 DEFINITION OF AN ALTERNATIVE SCHOOL ................................................................................ 29 GENERAL PROCESS TO PRODUCE ACHIEVEMENT PROFILES FOR ALTERNATIVE SCHOOLS ......... 30 CALCULATION OF AN ACHIEVEMENT PROFILE FOR AN ALTERNATIVE SCHOOL ......................... 30 CLASSIFICATION SCALES FOR ALTERNATIVE SCHOOLS ............................................................. 32 12. AZ LEARNS ACHIEVEMENT PROFILES FOR K-2 SCHOOLS .............................................................. 33 APPENDIX I: SUBSTANTIVE APPEAL COMMITTEE EVALUATION RUBRIC FOR AZ LEARNS ..... 34 APPENDIX II: SUBSTANTIVE APPEAL RUBRIC FOR AZ LEARNS .............................................. 35 Arizona's Accountability System: Volume I Arizona Department of Education 1 List of Tables and Figures TABLE 4.1 STATUS POINTS AWARDED ............................................................................................. 8 TABLE 4.2 PERCENTAGE OF STUDENTS ........................................................................................... 9 TABLE 4.3 YEAR-TO-YEAR CHANGE IN PERCENTAGES .................................................................... 9 TABLE 4.4 IMPROVEMENT POINTS ................................................................................................. 10 TABLE 4.5 RULES FOR DETERMINING UB95 FOR SMALL N AND P ................................................. 12 TABLE 5.1 PARAMETERS FOR CALCULATING EXPECTED GROWTH FOR MATH .............................. 14 TABLE 5.2 PARAMETERS FOR CALCULATING EXPECTED GROWTH FOR READING .......................... 14 TABLE 8.1 ELEMENTARY SCHOOL CLASSIFICATION CUT POINTS ................................................... 19 TABLE 8.2 HIGH SCHOOL CLASSIFICATION CUT POINTS ................................................................ 20 TABLE 8.3 COMBINATION SCHOOL CLASSIFICATION CUT POINTS .................................................. 20 TABLE 9.1 PARAMETERS FOR CALCULATING Z-SCORES ............................................................... 23 TABLE 9.2 NUMBER OF STUDENTS EXCEEDING THE STANDARD—THIRD GRADE ......................... 23 TABLE 9.3 AVERAGE Z-SCORES .................................................................................................... 24 FIGURE 11.1 METHOD FOR EVALUATING ALTERNATIVE SCHOOLS (ELEMENTARY SCHOOLS) ...... 31 FIGURE 11.2 METHOD FOR EVALUATING ALTERNATIVE SCHOOLS (HIGH SCHOOLS) ................... 32 TABLE 11.1 ALTERNATIVE SCHOOL CLASSIFICATION CUT POINTS ............................................... 32 TABLE 12.1 AZ LEARNS SCALE FOR K-2 SCHOOLS .................................................................... 33 Arizona's Accountability System: Volume I Arizona Department of Education 2 1. Introduction In November of 2001, Arizona voters approved Proposition 301 which, among other things, provided funds to the Arizona Department of Education (ADE) to develop ―a system to measure school performance based on student achievement, including student performance on the AIMS test.‖ The legislative requirements for the accountability system are stated in section 15-241 (ARS § 15-241) of the Arizona Revised Statutes. The accountability system created to satisfy the statute is referred to as the Arizona LEARNS. The school evaluation given by ADE to each school is referred to as the school’s achievement profile. This manual describes the method and processes used to generate the 2008 AZ LEARNS achievement profiles. It provides formulas, parameters, and business rules that make up the profile calculation. Its intent is to document and explain the methods used and justify the policies adopted. What’s New for the 2008 AZ LEARNS Achievement Profiles Two major changes to the AZ LEARNS school evaluation formula were approved by the State Board of Education on March 24, 2008. The first was a revamping of the Growth/Improvement portion of AZ LEARNS. This was done to include all grades tested with the AIMS since there are now sufficient number of years scores for grades 4, 6, and 7, as well as to simplify the calculation and make it more valid. These changes are described in detail in chapter 4. The second is the implementation of a unified achievement profile for schools serving both elementary/middle and high school grades. Details of the changes for unified profiles are included in chapter 8. Arizona's Accountability System: Volume I Arizona Department of Education 3 2. Overview of the AZ LEARNS Evaluation System This section provides an overview of how AZ LEARNS achievement profiles are determined. More detailed discussions of the methodology used to determine the profiles, including descriptions of equations, algorithms, and data used are given in subsequent chapters. Arizona law (ARS § 15-241) mandates that the Arizona Department of Education shall compile an annual achievement profile for each public school. It specifies that the profiles of schools serving grades K-8 shall be based on: Percent of students who pass AIMS. Arizona Measure of Academic Progress (MAP). Performance on the English language proficiency test. The law specifies that the profiles of high schools shall be based on: Percent of students who pass AIMS. Dropout rate Graduation rate Performance on the English language proficiency test. Starting in 2008, a school that serves both grades K-8 and high school receives a unified achievement profile based on all of the above performance measures. The law also calls for the ADE to use a research based methodology that shall: Include performance of pupils at all achievement levels Account for pupil mobility Account for the distribution of pupil achievement Include longitudinal indicators of academic performance. A research based methodology is defined as ―the systematic and objective application of statistical and quantitative research principles to determine a standard measurement of acceptable academic progress for each school‖. Arizona's Accountability System: Volume I Arizona Department of Education 4 The law also calls for a system of parallel achievement profiles for accommodation schools/alternative schools as defined by the Board of Education. General Process to Calculate an Achievement Profile The achievement profile for a school serving grades 3-8 consists of the following performance measures: 1. A status measure based on the performance of students on all three sections of the AIMS (reading, writing, and mathematics) in the current year. 2. A measure of improvement in aggregate student performance on the AIMS compared to the baseline year. 3. A measure of growth in individual student performance. This is the Measure of Academic Progress (MAP). 4. A measure of student performance on the state’s English language proficiency assessment: AZELLA. 5. A measure of whether the school made Adequate Yearly Progress (AYP) as defined by the No Child Left Behind Act of 2001. In order to comply with the federal requirement that the state have an integrated accountability system, a school’s AYP determination is factored into the calculation of its achievement profile. 6. In order to create the incentive for schools to improve the achievement of average and above-average students, a school cannot earn the highly performing or excelling labels unless the percentage of its students exceeding the standard on AIMS met specific thresholds. 7. The achievement profile for a high school is made up of all of the above components except for measure 3 (MAP). In addition, the following performance measures are used for high schools: Dropout rate. Graduation rate. Schools are awarded scale score points based on their performance on measures one through five, and seven. Scale score points are then summed up for each school and compared to a scale that relates scale score points to the five profile labels: excelling, highly performing, performing plus, performing, and underperforming. Performance measure six is then examined to determine if the school has earned the highest labels of highly performing or excelling. Arizona's Accountability System: Volume I Arizona Department of Education 5 3. Timeline Districts and charter holders are solely responsible for submitting the data necessary for calculating achievement profiles for their schools and for ensuring its accuracy. From May 30, 2008 through September 12, 2008 schools and districts were given an opportunity to review and correct their testing data through on-line applications. The primary purpose of the process was to allow districts and charter holders to correct the information for individual students. From February 21, 2008 through June 30, 2008 schools and districts were given the opportunity to review and correct the data used for calculating the five-year graduation rate and dropout rates used in the AZ LEARNS profile. All program membership and demographic information relevant to AZ LEARNS profiles were taken by matching test records to the state’s SAIS database of student records. Consequently the only information that schools needed to correct in the ADE AIMS testing file were students’ SAIS IDs (needed for matching). If program membership or other information was incorrect, schools and districts were required to correct it in the SAIS database. Schools were not allowed to correct the indicators for alternate testing modifications. IMPORTANT NOTE: The criteria used to select AIMS scores for AZ LEARNS profiles differ from the criteria used to select scores for adequate yearly progress under NCLB. Indeed, the criteria differ among the separate components of the AZ LEARNS evaluation. The criteria also differ from the scores provided to schools by the testing contractor, and the scores publicly reported by ADE which are available here: http://azed.gov/researchpolicy/AIMSResults/. Timeline The timeline for AZ LEARNS Achievement Profile process was: February 21, 2008. Start of statistical review and appeals of graduation rate and dropout rate data. March 10, 2008. Opening of application process for alternative schools. May 30, 2008. Opening of data verification process. June 30, 2008. Closing of application process for alternative schools. June 30, 2008. Closing of statistical review and appeals of graduation rate data. September 3, 2008. Preliminary release of AZ LEARNS achievement profiles for all schools. Opening of window for appeals submissions. September 12, 2008. Closing of appeals window for all schools. October 8, 2008. Public release of AZ LEARNS achievement profiles for all schools. Arizona's Accountability System: Volume I Arizona Department of Education 6 4. Calculation of AIMS Status and Growth Points In the AZ LEARNS profile, schools are awarded scale score points based on student performance on the AIMS. Points are determined by performance in the current year, and improvement in student performance from the baseline year. Awarding Status Points The following method is used to calculate status points for every subject and grade offered by a school in which the AIMS test is administered. First, calculate the percent passing in the current year. The percentage is compared to the scale given in table 4.1 which in turn gives the status points achieved for the subject and grade. Percent Passing in the current year # Students Passing AIMSin the current year # Students tested in the current year The points are rounded to the nearest hundredth e.g. .675 = .68; .672 = .67. Data Used A student’s score is excluded from the calculation if any of the following criteria are met: 1. The student was not matched to SAIS with a valid ID. 2. Student received no score on the test. 3. The student received an alternate accommodation on the test. 4. Student was not English proficient. A student was considered not proficient if it was indicated that she was a participant in the English language learner program for three years or less. 5. The student was not enrolled in the school for the full academic year. A student was considered enrolled for the full academic year if she enrolled in a school during the first 10 school days of the school year and remained enrolled up through the testing date. The calculation for high schools includes all students in grades 10 through 12 who have taken the AIMS either in the fall or spring. If a high school student took the test twice in a school year in the same school, the highest of the two scores was used. Arizona's Accountability System: Volume I Arizona Department of Education 7 Table 4.1 Status Points Awarded Grade Subject 3 3 3 4 4 4 5 5 5 6 6 6 7 7 7 8 8 8 HS HS HS Math Read Writ Math Read Writ Math Read Writ Math Read Writ Math Read Writ Math Read Writ Math Read Writ Status Points 1 < 51% < 46 < 56 < 46 < 40 < 42 < 42 < 42 < 47 < 34 < 38 < 49 < 35 < 41 < 64 < 27 < 35 < 58 <5 < 16 < 18 Status Points 2 Status Points 3 Status Points 4 Status Points 5 51 to 64% 46 to 59 56 to 67 46 to 60 40 to 53 42 to 53 42 to 56 42 to 56 47 to 57 34 to 49 38 to 52 49 to 60 35 to 49 41 to 54 64 to 73 27 to 41 35 to 48 58 to 70 5 to 13 16 to 28 18 to 30 65 to 78% 60 to 73 68 to 78 61 to 75 54 to 69 54 to 65 57 to 72 57 to 73 58 to 69 50 to 66 53 to 69 61 to 73 50 to 66 55 to 69 74 to 82 42 to 59 49 to 65 71 to 82 14 to 29 29 to 46 31 to 48 79 to 88% 74 to 84 79 to 87 76 to 86 70 to 81 66 to 77 73 to 85 73 to 84 70 to 80 67 to 81 70 to 82 74 to 83 67 to 81 70 to 82 83 to 89 60 to 76 66 to 79 83 to 90 30 to 51 47 to 65 49 to 67 89 to 94% 85 to 92 88 to 93 87 to 93 82 to 90 78 to 85 86 to 92 85 to 92 81 to 87 82 to 90 83 to 91 84 to 90 82 to 90 83 to 90 90 to 94 77 to 87 80 to 88 91 to 95 52 to 70 66 to 79 68 to 80 Status Points 6 ≥ 95% ≥ 93 ≥ 94 ≥ 94 ≥ 91 ≥ 86 ≥ 93 ≥ 93 ≥ 88 ≥ 91 ≥ 92 ≥ 91 ≥ 91 ≥ 91 ≥ 95 ≥ 88 ≥ 89 ≥ 96 ≥ 71 ≥ 80 ≥ 81 A school is awarded status points for each subject/grade it offers in which the AIMS test is administered. Example. In the current year, 66 percent of the students in Gila Monster Elementary passed the math portion of the third grade AIMS. This value places the subject/grade in Status grouping three. Gila Monster Elementary has earned three status points for this particular subject/grade. Awarding Improvement Points Determining a school’s improvement points for each subject/grade combination is based on the average of the annual changes in the difference between the percentage of students passing and the percentage of students scoring ―falls far below‖ on the AIMS. The improvement points earned are determined by the average of the annual changes from 2005 or when a school opened, whichever is latest. Improvement points are awarded for all grades three through eight and high school. No minimum group size is applied. Arizona's Accountability System: Volume I Arizona Department of Education 8 New Computation of the Average Annual Change in Difference of Achievement Improvement points for a subject/grade are calculated in the following five steps: 1. For every year for which the school was open, or until 2005, whichever is latest, for each subject/grade combination the percentage of student passing and the percentage of students scoring ―falls far below‖ is computed. 2. The difference in these percentages is then taken. 3. The annual change between the differences is found. 4. The average of the annual changes is determined. 5. The average of the annual changes is compared to the new Improvement Points Table (table 4.4) to determine the AZ LEARN points for the subject/grade combination. All values are rounded to the nearest hundredth e.g. .675 = .68; .672 = .67. Data Used The rules for using student scores for the growth measure are the same as for the status measure. For high school students who took the test multiple times, the highest score in each year is used. Example. The following example demonstrates how growth points are calculated for a single grade and subject. Table 4.2 shows the percentage of students passing and scoring ―falls far below‖, and the difference in the two percentages from 2005 to 2008 of AIMS scores for a single subject and grade for a hypothetical school. Year 2005 2006 2007 2008 Table 4.2. Percentage of Students Percent Percent Falls Difference Passing Far Below 52 18 34 54 17 37 57 15 42 60 10 50 Table 4.3 shows the year-to-year changes in the differences from table 4.2. Table 4.3. Year-to-Year Change in Percentages Years Difference Year-to year change 2005 to 2006 37 - 34 3 2006 to 2007 42 - 37 5 2007 to 2008 50 - 42 8 Arizona's Accountability System: Volume I Arizona Department of Education 9 The growth measure is the average (3 + 5 + 8)/3 = 5.3. The improvement point thresholds are given in table 4.4. This table is used to convert the average annual improvement measure to AZ LEARNS improvement points. All grades three through eight and high school are now included where in years prior only grades three, five, eight and high school could earn these points. The first year for assessing grades four, six and seven was 2005. Grade Subject 3-8 Math 3-8 Read 3-8 Writ HS Math HS Read HS Writ Table 4.4. Improvement Points AZ AZ AZ AZ LEARNS LEARNS LEARNS LEARNS Points 1 Points 2 Points 3 Points 4 < -16% -16 to -9% -8 to 0% 1 to 9% < -14 -14 to -8 -7 to 0 1 to 7 < -16 -16 to -8 -7 to 0 1 to 11 < -16 -16 to -7 -6 to 4 5 to 15 < -12 -12 to -5 -4 to 4 5 to 13 < -10 -10 to -2 -1 to 7 8 to 16 AZ LEARNS Points 5 10 to 17% 8 to 14 12 to 20 16 to 25 14 to 21 17 to 25 AZ LEARNS Points 6 ≥ 18% ≥ 15 ≥ 21 ≥ 26 ≥ 22 ≥ 26 Calculation of Subject/Grade Scale Points from AIMS The total scale score points derived from AIMS performance are calculated by adding the status points awarded to the AZ LEARNS improvement points awarded. A 70 percent weight is given to the school’s strongest point value (status or improvement) and a 30 percent weight to the other point value. Example. In third grade mathematics, Gila Monster Elementary has earned three status points and four improvement points. Because it has earned more scale points for its improvement, the improvement points are given a 70 percent weight and the status points are given a 30 percent weight. Thus, the total scale points earned for third grade math are (3 X 0.3) + (4 X 0.7) = 3.7. Example. In mathematics, Desert Mountain Vista High School has earned five status group scale points and two improvement points. Because it has earned more scale points for its status, the status scale points are given a 70 percent weight and the improvement points are given a 30 percent weight. Thus, the total scale points earned for math are (5 X 0.7) + (2 X 0.3) = 4.1. If a subject/grade group earned six status points, 100 percent of the weight is given to the status points. As long as data is available for the past two years, for a subject/grade combination, improvement points are both computed and combined with the status points. Also, due to changes in the method for computing improvement points, no minimum group size was used to determine total scale points. Example. Gila Monster Elementary is missing data for fifth grade writing for both 2005 and 2006 but has scores for 2007 and 2008. Its status points for that subject/grade combination put it in status group five and it received three improvement points. Because it has earned more scale Arizona's Accountability System: Volume I Arizona Department of Education 10 points for its status, the status points are given a 70 percent weight and the improvement points are given a 30 percent weight. Thus, the total scale points earned for fifth grade writing are (5 X 0.7) + (3 X 0.3) = 4.4. Example. The newly opened High Mountain Elementary has no test scores for any year prior to 2008. Its status points for fourth grade reading put it in status group three but received no improvement points because there was no comparison year. 100 percent weight is given to the status group in this case. Thus it receives three scale points for fourth grade reading. The points derived from AIMS for all subject/grade combinations for a school are averaged across grades by subject and added to the scale score values for other performance measures. Special Cases Missing Data. If a school is missing current year AIMS test data for a subject/grade combination it offers, it receives zero scale points for that subject/grade for the achievement profile calculation for the year. Small Grades. Subject/grade combinations with less than ten students meeting the selection criteria in the current year were not evaluated. Second Look for Small Schools. Because of the high-stakes consequences of being labeled an underperforming school, and because of the uncertainty of measurement involved with small sample sizes, a school with a small number of students is given a ―second look‖ if it faces the possibility of receiving an underperforming profile. If the preliminary label of a school is underperforming, then the AIMS scale score points for that school are recalculated for each subject/grade group that has less than 16 students. For each subject/grade combination with less than 16 students, the upper bound of the 95-percent confidence interval is used to calculate to which baseline group the school belongs. If the recalculated points move the school into a classification higher than underperforming, the school receives a profile performing, regardless of the points earned. Let p = the percent of students in a group passing the AIMS and n=the number of students in the group. Then the equation for the upper bound of the 95 percent confidence interval (UB95) is: UB95 p 1.96 p(1 p) n . As can be seen from the equation, the confidence interval depends upon the percent of students who passed the test and the number of students tested. Thus, the confidence interval will differ among grades, subjects, and schools. The equation is an approximation of the confidence interval for a binomially distributed variable. It uses the standard normal distribution and is sufficiently accurate if the group size and percentage of students passing are large enough. For small values of n and small p, a more accurate estimate of the confidence interval is made using statistical tables that provide Arizona's Accountability System: Volume I Arizona Department of Education 11 confidence intervals for a binomially distributed variable.1 The tables are applied using the rules given in table 4.5. Table 4.5. Rules for Determining UB95 for Small n and p If n ≥ 0 and n < 8, and p ≥ 0.00 and p < 0.04, then UB95=0.42; p ≥ 0.04 and p < 0.10, then UB95=0.50; p ≥ 0.10 and p < 0.20, then UB95=0.60. If n ≥ 16 and n < 20, and p ≥ 0.00 and p < 0.04, then UB95=0.24; p ≥ 0.04 and p < 0.10, then UB95=0.32; p ≥ 0.10 and p < 0.20, then UB95=0.44. If n ≥ 8 and n < 10, and p ≥ 0.00 and p < 0.04, then UB95=0.37; p ≥ 0.04 and p < 0.10, then UB95=0.45; p ≥ 0.10 and p < 0.20, then UB95=0.55. If n ≥ 20 and n < 24, and p ≥ 0.00 and p < 0.04, then UB95=0.21; p ≥ 0.04 and p < 0.10, then UB95=0.29; p ≥ 0.10 and p < 0.20, then UB95=0.42. If n ≥ 10 and n < 12, and p ≥ 0.00 and p < 0.04, then UB95=0.33; p ≥ 0.04 and p < 0.10, then UB95=0.41; p ≥ 0.10 and p < 0.20, then UB95=0.52. If n ≥ 24 and n < 30, and p ≥ 0.00 and p < 0.04, then UB95=0.18; p ≥ 0.04 and p < 0.10, then UB95=0.27; p ≥ 0.10 and p < 0.20, then UB95=0.38. If n ≥ 12 and n < 16, and p ≥ 0.00 and p < 0.04, then UB95=0.27; p ≥ 0.04 and p < 0.10, then UB95=0.35; p ≥ 0.10 and p < 0.20, then UB95=0.47. If n ≥ 30 and n < 40, and p ≥ 0.00 and p < 0.04, then UB95=0.15; p ≥ 0.04 and p < 0.10, then UB95=0.23; p ≥ 0.10 and p < 0.20, then UB95=0.36. 1 Mansfield, Edwin. 1991. Statistics for Business and Economics, 4th Edition. New York: W.W. Norton and Company. 280-284. Arizona's Accountability System: Volume I Arizona Department of Education 12 5. The Measure of Academic Progress The measure of academic progress (MAP) is a measure of individual student academic growth measured by performance on the AIMS reading and mathematics tests. (Writing is not included in the MAP analysis.) It uses a conventional value-added approach to measuring student progress. Calculating the number of points earned from MAP has three steps: 1. Individual expectations of one year’s growth (OYG) are calculated for each student. 2. The expectation of OYG is subtracted from the actual growth achieved by the student to determine a growth index. 3. The average growth index for school is calculated by averaging growth indices for individual students across all grades and subjects. Schools are awarded AZ LEARNS points based on this average. Actual growth, expected growth, and the growth index are expressed in AIMS scale score points. The Growth Index In order to control for a ceiling effect and for student mobility, the MAP analysis is done using a standard value-added model. The value-added model is used to calculate an estimate of the expected for each student for each subject. The expected growth is then subtracted from actual growth to determine a student’s growth index. Example. Student A scored 478 on the 6th grade math test in 2007. Her expected growth for 7th grade in 2007 is 38. In 2008 she scores a 528 on the 7th grade math test. Her actual growth is 528 – 478 = 50. Her growth index is 50 – 38 = 12. Example. Student B scored 490 on the 4th grade reading test in 2007. His expected growth for 5th grade in 2008 is 12. In 2007 he scores a 500 on the 5th grade reading test. His actual growth is 500 – 490 = 10. His growth index is 10 – 12 = -2. The results of the MAP analysis for students and schools are reported in terms of the growth index. Growth indices for individual students can be averaged across schools, grades, and subjects to measure performance. Calculating Expected Growth The expected growth for an individual student is calculated using the following formula: Expected Growth = A + B X (2007 scale score) + C X (FAY); Arizona's Accountability System: Volume I Arizona Department of Education 13 where FAY = 1 if the student has been enrolled for the full 2008 academic year and FAY = 0 if not. A student is considered to have been enrolled the full 2008 academic year if she has enrolled within the first two weeks of the school year and remained enrolled up to the first day of the testing window. The parameters A, B, and C are given in the tables below: Table 5.1. Parameters for Calculating Expected Growth for Math Student’s Grade Current Year A B C 4 98.9308 -0.1514 6.027 5 107.7715 -0.174 5.7754 6 75.6373 -0.1268 6.999 7 121.1295 -0.1896 7.1203 8 54.1785 -0.0823 7.0308 Table 5.2. Parameters for Calculating Expected Growth for Reading Student’s Grade Current Year A B C 4 120.9638 -0.2269 3.7108 5 153.1619 -0.2879 4.106 6 105.8317 -0.2016 4.343 7 88.3119 -0.1492 5.1193 8 89.8856 -0.1688 7.0786 Example. Student A scored 478 on the 6th grade math test in 2007. She has been enrolled in her current school all year. Her expected growth for 7th grade in 2008 is 120.1295 + (-0.1896) X 478 + 7.1203 = 36.62 = 37. Example. Student B scored 490 on the 4th grade reading test in 2007. He has not been enrolled in his current school for the full year. His expected growth for 5th grade in 2008 is 153.1619 + (0.2879) X 490 = 12.09 = 12. Please see the 2006 MAP report available on the ADE web site for a description of how the parameters were developed. Awarding AZ LEARNS Points The Growth Index for a school is calculated by averaging the student-level growth indices across all grades for math and reading. The number of AZ LEARNS points earned by a school is calculated with the following formula: AZ LEARNS points = 5.2 + .22 X (School’s growth index.) Arizona's Accountability System: Volume I Arizona Department of Education 14 The scale is bounded at the bottom by 2. So if the result of the above formula is less than 2, a school would earn 2 points. No maximum is placed on the number of points a school may earn. This is to award schools that make outstanding growth with students. The scale was developed so that MAP would receive the same weight in the AZ LEARNS profile that it has in previous years. Example. School A’s growth index is -1.7. The number of AZ LEARNS points awarded to the school is 5.2 + .22 X (-1.7) = 4.8. Example. School B’s growth index is 3.2. The number of AZ LEARNS points awarded to the school is 5.2 + .22 X (3.2) = 5.9. Example. School C’s growth index is -20.0. Since 5.2 + .22 X (-20.0) = 0.8 is less than 2. The number of AZ LEARNS points awarded to the school is 2. Schools without MAP MAP is only evaluated for grades four through eight using student scores from grades three through eight.) Schools do not receive points from MAP if they do not have a grade evaluated for MAP or if they have less than 16 scores in the MAP analysis. Arizona's Accountability System: Volume I Arizona Department of Education 15 6. Graduation and Dropout Rates In their AZ LEARNS profile, high schools may earn one point for their graduation rate and one point for their dropout rate. Graduation Rate The Graduation Rate is a five-year, longitudinal measure of how many students graduate from high school. The formula used to calculate the graduation rate is: Graduation Rate Number in cohort that graduated in within five years Original cohort transfers in - transfers out The graduation rate is rounded to two digits, e.g.: .705=.71; .704=.70. A school can earn one AZ LEARNS scale point for its graduation rate in one of three ways: 1. If its three-year average graduation rate is 90 percent or greater. 2. If its three-year average graduation rate is less than 90 percent and its baseline rate is greater than or equal to 74 percent, then a school will earn one point if the average of its graduation rates over the most recent three years is 1 percentage point greater than its baseline rate. 3. If its three-year average graduation rate is less than 90 percent and its baseline rate is less than 74 percent, then a school will earn one point if the average of its graduation rates over the most recent three years is 2 percentage point greater than its baseline rate. The three-year average is calculated by taking the total number of combined five-year graduates for the most recent three graduating classes and dividing by the total number of students in the combined classes. Graduation Rate 2005 2006 2007 five - year graduates Combined number of students in three cohorts The baseline graduation rate is the graduation rate for the cohort class of 2000. If a school does not have a graduation rate for the cohort class of 2000 and does not have a three-year average graduation rate of at least 90 percent, the number of points earned through the dropout rate is doubled. Arizona's Accountability System: Volume I Arizona Department of Education 16 Dropout Rate The dropout rate is an annual measure of how many students drop out of a school during a twelve-month reporting period. Dropout Rate Number of students who dropped out Number of students who were enrolled during the school year Like the graduation rate, a school can earn one AZ LEARNS scale point for its dropout rate in one of three ways: 1. If the three-year average dropout rate is 6 percent or less, the target is automatically met. 2. If its three-year average dropout rate is greater than 6 percent and its baseline rate is greater than or equal to 9 percent, then a school will earn one point if the average of its dropout rates over the most recent three years is 2 percentage point less than its baseline rate. 3. If its three-year average graduation rate is less than 6 percent and its baseline rate is less than 9 percent, then a school will earn one point if the average of its dropout rates over the most recent three years is 1 percentage point less than its baseline rate. The three-year average is calculated by taking the total number of dropouts for the most recent three years and dividing by the total number of students enrolled. Three - year dropout rate Total number of dropouts for 2006, 2007 and 2008 Total number of students enrolled in 2006, 2007 and 2008 The baseline dropout rate is the dropout rate for the 2000 school year. Schools without a baseline were evaluated only on step one above. Arizona's Accountability System: Volume I Arizona Department of Education 17 7. Evaluating Performance on the English Language Assessment State law governing AZ LEARNS now requires that the ―results of English language learners tests‖ be included as part of the AZ LEARNS school evaluation. {ARS 15-241 (D) (3) and ARS 15-241 (E) (4)} In order to comply with this requirement, the following change was made to the AZ LEARNS profile. A school receives one scale score point if the percentage of English language learner students reclassified during the year across all grades served is greater or equal to 30 percent. Only students identified as continuing are included in the analysis. Only groups of 16 or more students are evaluated This modification maintains the emphasis of the AZ LEARNS evaluation on how well students learn the state academic standards. Consequently, most of the weight of the evaluation is still placed on a school’s AIMS results compared to performance on the English language learner test. Arizona's Accountability System: Volume I Arizona Department of Education 18 8. Evaluating the Total Scale Score Value to Determine a School Classification The tables below show the total number of scale score points that schools must earn in order to receive a given classification. A school may receive up to 18 scale points from status and growth; unlimited scale points from MAP if it is an elementary or middle school; a single point if it was able to reclassify 30 percent of its English language learners, and up to two graduation/drop out scale points if it is a high school. A school receives one scale point if it made adequate yearly progress (AYP). For each school, the applicable scale score cut points for classification labels are calculated by averaging the AIMS scale points received for each subject; adding the AYP and ELL points received; adding points from MAP if it is an elementary school; and adding points from the graduation/dropout rate if it is a high school. The classification cut points for elementary schools are given in the first two columns of table 8.1 and those for high schools are given in the first column of table 8.2. In order for schools to be classified as highly performing or excelling, there is an additional Z-score criteria that has to be met, which is explained in the next chapter. Until 2008, schools serving both the elementary/middle and high school grade spans received two separate AZ LEARNS achievement profiles: one for the elementary/middle grades, and one for the high school grades. This created some confusion and issues, especially if a school received an underperforming profile at one level and a performing or higher profile at another. In 2007, state law was amended to allow the calculation of a unified profile for combination (elementary or middle and high) schools. The combined profile comprises all of the components of the AZ LEARNS evaluation at both levels. The classification cut points for schools with combination grades are given in the first two columns of table 8.3. Table 8.1. Elementary School Classification Cut points Performance Thresholds Profile Underperforming Performing Performing Plus Highly Performing Excelling Scale with MAP Non-MAP Scale <13 13 to 15.9 16 or more 16 to 18.9 19 or more <8 8 to 12.9 13 or 20 13 to 14.9 15 to 20 Arizona's Accountability System: Volume I z-score NA NA 0.45 to 0.99 1.00 or more Arizona Department of Education 19 Table 8.2. High School Classification Cut points Performance Thresholds Profile Underperforming Performing Performing Plus Highly Performing Excelling Scale Points z-score <14 14 to 16.9 17 or more 17 to 19.9 20 or more NA NA 0.45 to 0.99 1.00 or more Table 8.3. Combination School Classification Cut points Performance Thresholds Profile Underperforming Performing Performing Plus Highly Performing Excelling Scale with MAP Non-MAP Scale <14 14 to 16.9 17 or more 17 to 19.9 20 or more <9 9 to 13.9 14 to 22 14 to 15.9 16 to 22 z-score NA NA 0.45 to 0.99 1.00 or more Special Rules: Schools with no subject/grade combination with at least ten students meeting the selection criteria for the status measure in the current year were automatically assigned a ―Performing‖ profile. Arizona's Accountability System: Volume I Arizona Department of Education 20 9. Application of Threshold Criteria for Excelling and Highly Performing Schools To ensure continued focus on improving the academic achievement of all students, including those students currently demonstrating proficiency, threshold criteria are applied to determine excelling and highly performing schools. To be deemed a highly performing or excelling school, a school must meet certain levels in the percentage of its students exceeding the standard on the AIMS. Schools must not only receive a total scale value that places them into either excelling or highly performing, but must also meet the requisite percentage of students in the exceeds-the-standard category on AIMS to be designated as either excelling or highly performing schools. The application of threshold criteria for excelling and highly performing schools results in the following scenarios. 1. A school that receives a total scale value that places it in the excelling classification and meets the requisite percentage of students in the exceeds level on AIMS necessary for an excelling classification will be designated an excelling school. 2. A school that receives a total scale value that places it in the excelling classification and did not meet the requisite percentage of students in the exceeds level on AIMS necessary for a excelling classification, but did met the requisite percentage of students in the exceeds category on AIMS necessary for a highly performing classification will be designated as a highly performing school. 3. A school that receives a total scale value that places it in the excelling classification and did not meet either the requisite percentage of students in the Exceeds category on AIMS necessary for the excelling classification or the highly performing classification will be designated as a performing school. Such schools are recognized with the non-statutory designation of performing plus. 4. A school that receives a total scale value that places it in the highly performing classification and meets the requisite percentage of students in the exceeds category on AIMS necessary for an excelling classification will be designated as a highly performing school. 5. A school that receives a total scale value that places it in the highly performing classification and meets the requisite percentage of students in the exceeds category on AIMS necessary for a highly performing classification will be designated as a highly performing school. 6. A school that receives a total scale value that places it in the highly performing classification and did not meet either the requisite percentage of students in the exceeds category on AIMS necessary for an excelling classification or highly performing Arizona's Accountability System: Volume I Arizona Department of Education 21 classification will be designated as a performing school. Such schools are recognized with the non-statutory designation of performing plus. 7. A school that receives a total scale value that places it in the performing classification will be designated as a performing school, regardless if the school meets the requisite percentage of students in the exceeds category on AIMS necessary for an excelling classification or a highly performing classification. 8. A school that receives a total scale value that places it in the underperforming classification will be designated as an underperforming school, regardless if the school meets the requisite percentage of students in the exceeds category on AIMS necessary for an excelling classification or a highly performing classification. The determination of whether a school met the goals for percent exceeding is based on a z-score calculated in the following manner: 1. For each school, the percent exceeding is calculated by grade across all three subjects for 2008 and for 2006 through 2008. The one-year estimation of percent exceeding gives credit to schools that have shown improvement in the percent of students exceeding the standard. 2. The percent-exceeding is converted into a z-score for each grade by subtracting the statewide average for that grade for percent exceeding and dividing by the statewide standard deviation for that grade. (These parameters are given in table 8.1) Note: The same statewide parameters are used to calculate both the one-year and threeyear z-score. This is because the purpose of the z-score is to create standard scores comparable across grades, not to create scores comparing a school’s performance to the state norm. The z-scores are fixed parameters so if a school shows improvement over time its z-score will increase regardless of the movement of the state average over the same period. 3. The one- and three-year z-scores are averaged across all grades for a school. Each school will have two z-scores: one based on its three-year average for percent exceeding, the other based on the percent exceeding for the current year. 4. The higher of the one- and three-year averages are taken and compared to the performance thresholds. To be a highly performing school, the school must first meet its appropriate highly performing classification scale cut point, as explained in chapter 8, plus the average z-score for a school must be greater than or equal to 0.45. To be an excelling school, the school must first meet its appropriate excelling classification scale cut point plus the average z-score for a school must be greater than or equal to 1.00. Arizona's Accountability System: Volume I Arizona Department of Education 22 Table 9.1. Parameters for Calculating ZScores Grade Average Standard Deviation 3 0.124576 0.093475 4 0.111652 0.087267 5 0.092426 0.080033 6 0.095892 0.085301 7 0.06592 0.0666 8 0.052172 0.056997 10 0.058823 0.082827 Example. The following table shows the third grade AIMS scores for Gila Monster Elementary over the past three years. Subject Year 2006 2007 2008 Total Table 9.2. Number of Students Exceeding the Standard—Third Grade Reading Mathematics Writing # Exceeding #Tested # Exceeding #Tested # Exceeding #Tested 25 100 24 100 23 100 24 105 23 105 22 105 26 99 25 99 24 99 75 304 72 304 69 304 The three-year average for percent of students exceeding the standard is: Percent 75 72 69 304 304 304 Exceeding 23.6%. The one-year average for percent of students exceeding the standard is: Percent Exceeding 25 25 24 99 99 99 25.3%. The three-year z-score for third grade is: z - score .236 - .125 1.19. .093 The one-year z-score for third grade is: z - score Arizona's Accountability System: Volume I .253 - .125 1.38. .093 Arizona Department of Education 23 Gila Monster Elementary serves grades K-6. The one- and three-year average z-scores for the entire school are: Grade K-2 3 4 5 6 Average Table 9.3. Average Z-scores One-year Three-year NA NA 1.38 1.19 1.02 .86 .89 .72 .95 .80 1.06 .89 Gila Monster’s highest z-score is 1.06 for the one-year average. This z-score is greater than 1.00, so Gila Monster Elementary would earn an excelling — if it has earned sufficient scale points. Arizona's Accountability System: Volume I Arizona Department of Education 24 10. AZ LEARNS Achievement Profile Appeals Process Procedure and Timeline In accordance with A.R.S. §15-241, school administrators are allowed the opportunity to appeal an achievement profile classification on behalf of the schools for which they are responsible. Step 1: Data Correction. The first step in completing the AZ LEARNS Appeals Process required districts and schools to review their data in order to confirm its accuracy. Data correction took place February 21, 2008 through September 12, 2008. It is important to note that districts and charter holders were solely responsible for verifying information for their districts and schools. If a district or charter holder did not verify the information for its district and schools through the correction process, the ADE assumed the schools on file and the data available were correct as listed. Step 2: Appeal Application. Administrators choosing to appeal an AZ LEARNS profile completed the AZ LEARNS Appeal Application, which was accessible via the common logon during the specified appeal window. Appeals were only accepted through the website application. Appeals sent to ADE via email, fax, or mail/delivery were not accepted. Schools were able to appeal AZ LEARNS profiles in two categories: data (statistical) and non-data (substantive) reasons – districts and schools were not limited to one category and were able to appeal in both if necessary. Statistical appeals are appeals of the accuracy of the data used in the AZ LEARNS profile. Given the extensive time allowed to view and correct the data, it is expected that any errors should be corrected by the time preliminary profiles are released. Statistical appeals were not granted unless the underlying data was corrected. Substantive appeals are arguments by districts and schools that that circumstances outside of the district’s/school’s control negatively affected school performance. Administrators that chose to appeal an AZ LEARNS profile must have clearly articulated the issues they believe merited an appeal. Administrators must have submitted evidence that the issues they believe merited an appeal directly resulted in a significant decrease in student academic achievement as demonstrated on the AIMS. The evidence must have been submitted to ADE at the time the appeal was submitted. Failure to provide this evidence resulted in the appeal not being granted. Evidence submitted after the appeal deadline closed was not considered. Once appeals were submitted through the Common Logon, the school/district/charter holder received an email verifying that the appeal was received. NOTE: In order to protect student privacy and the integrity of the appeals process, schools were asked to refer to a specific student only by that student’s SAIS ID. The SAIS ID was required so that ADE staff could verify the contentions in the appeal. Arizona's Accountability System: Volume I Arizona Department of Education 25 The ADE, if necessary, requested that a district or school administrator provide additional information/evidence to assist in the appeals process. Only those requests for additional information that were provided during the specified timeframe allotted were included in the appeals process. Requests submitted after the specified timeframe were excluded from the appeals process. Unsolicited additional information submitted after the appeal deadline was not accepted. Step 3: Appeal Resolution. After all appeals were submitted and the appeals window closed, the ADE began to process the appeals. Appeals were addressed categorically, not necessarily in the order received, so the fact that a district or school submitted its appeal during the first day of the appeal window did not mean it necessarily received a decision first during the resolution process. Statistical appeals were resolved only through recalculation of the AZ LEARNS profile by ADE staff using any corrected data submitted by the school. The purpose of a statistical appeal is principally to advise ADE staff that data was in error and has been corrected. Calculations submitted by schools via an appeal were not taken at face value nor used to alter a profile if the underlying data was not corrected. Schools that were labeled underperforming for a third consecutive year were entitled to a site visit to determine if the label was warranted. These site visits were important as the third year underperforming label merited an alternative designation of failing to meet the academic standards in accordance with statute. After the statistical review of appeals took place, schools labeled underperforming for a third consecutive time were scheduled to receive that visit. Teams from the Research and Evaluation and State Intervention sections of the ADE visited each of the third year underperforming schools to gather additional supporting data for the appeal as well as gather information related to the school’s Arizona school improvement plan (ASIP) as required by statute. All information gathered from the site visits was taken to the substantive appeal committee for use in the third stage of the appeal process. Only third year underperforming schools received site visits as part of the appeals process. Substantive appeals were resolved in a committee process. Committee members represented a diverse background of ADE staff and school administrators to ensure that appeals were considered from multiple perspectives. Appeals were evaluated using an appeals rubric approved by the State Board of Education that evaluated the argument presented and whether or not the evidence provided to support the argument was compelling. The appeals rubric consisted of a three-tiered system for appeal evaluation: 1) Initial review of the appeal to determine its merit. 2) Review of the evidence provided. 3) Committee recommendation. Initial Review. The substantive appeal rubric provided for three categories that apply during the initial review. Each appeal was classified into the categories based on the information provided in the appeals. 1. Mitigating Factors Outside the School’s Control. Appeals of this nature referenced when the school indicated significant issues that affected test scores outside of the school’s control. If a school provided information detailing a significant event that impacted test scores, which was clearly outside the school’s control, the appeal was deemed as passing the initial review. Appeals involving the adverse affect of school or Arizona's Accountability System: Volume I Arizona Department of Education 26 district policies; errors made by school or district personnel regarding test administration or data entry; or events whose impact could have been foreseen and mitigated by school or district action were not considered valid appeals. 2. Implementation of the School Improvement Plan. Appeals in this category discussed how the school was actively, consistently, and reliably implementing the school improvement plan; that the priorities of the solutions team had been addressed; the school improvement plan had been revised and updated to address assessed ongoing needs; and that professional development that supported the targeted goals had been planned and implemented. If a school provided information highlighting their school improvement plan, it may have accentuated the other two components on the rubric. However, per statute, a successful implementation of the school improvement plan alone cannot change a classification of a school. Additionally, only third-year underperforming schools may include information about their ASIP in their appeal. Review of Evidence. Once the appeals progressed through the first tier of the rubric, initial review, the evidence provided to support the appeal was evaluated. In this tier, three determinations were possible: 1. Compelling evidence: In this area, the school or district adequately provided information that led the committee to conclude that, had the circumstance been different, the achievement profile would have been different as well. If a school had a special circumstance that affected a certain grade and was able to demonstrate that the specific grades test scores suffered, the school was deemed to have provided compelling evidence. Again, if a school provided compelling evidence highlighting their school improvement plan, it may have accentuated the other two components on the rubric (data calculations/mitigating factors). However, per statute, a successful implementation of the school improvement plan alone could not change the classification of a school. 2. Not compelling evidence: Appeals were categorized in this area when they were able to provide information that a significant issue could have impacted the school’s performance but did not provide detailed, specific information as to specific outcomes that hindered the school’s performance. 3. Not applicable evidence: If an appeal was submitted, made it through the initial review, and presented evidence that was not linked in any way to the performance of the school, the evidence was deemed not applicable. If the evidence did not directly support the claim made in the appeal, it was deemed not applicable. Committee Recommendation. Once the appeal and evidence were reviewed, the committee arrived at a decision as to the outcome of the appeal. There were three possible outcomes: 1. Appeal granted and AZ LEARNS determination changed. In these cases, the appeal successfully made it through the initial review and evidentiary stages. It was determined that the points needed to change classifications would have been earned by the school had the special circumstance/data discrepancy not occurred. Therefore, the classification for the school was changed. Arizona's Accountability System: Volume I Arizona Department of Education 27 2. Appeal granted and AZ LEARNS determination remains the same. In these cases, the appeal successfully made it through the initial review and evidentiary stages. However, it was determined that the criteria needed to change classifications were not earned by the school had the special circumstance/data discrepancy not occurred. For example, a school provided information and evidence that their AYP point was not accurately included in the calculations and the committee determined the school provided information to prove they earned the one point set aside for AYP in the AZ LEARNS formula. However, the school needed 5 points to get from underperforming to performing. Therefore, the one point earned was not enough to change their designation, so their determination remained the same. 3. Appeal Denied. In these cases, the appeal did not successfully make it through the initial review and evidentiary stages. Therefore, the classification for the school remained the same. Step 4. Notification of Result Sent to Schools. Once all appeals were resolved, notifications were sent to the districts and/or schools that had filed appeals. The contact person of record for the district/school received an email from Achieve with directions as to how to access appeal information via the Common Logon when the appeal had been processed. Districts and schools were notified before the final public release of the AZ LEARNS profiles as to the outcome of the appeal process. All appeals were final. Special Appeals for 2008 Appeals for AYP determinations negatively impacted by the inclusion of ELL students were allowed during the 2008 school year. Appeal of AYP determination. A school may have appealed its AYP determination if it did not make AYP because of the inclusion of ELL students and not making AYP adversely affected its AZ LEARNS profile. If a school appealed the inclusion of ELL students in its AYP determination, the determination was recalculated with ELL students counting as tested but not included in the AMO calculation. Arizona's Accountability System: Volume I Arizona Department of Education 28 11. AZ LEARNS Achievement Profiles for Alternative Schools In 2004, the ADE published profiles for alternative schools for the first time. Alternative schools are defined as schools that meet the Board-approved definition as schools whose sole and clearly-stated mission is to serve specific populations of at-risk students. Alternative school status is granted by application to the ADE. A.R.S. §15-241 makes an allowance for a ―parallel‖ evaluation method for alternative schools. When AZ LEARNS achievement profiles were first issued in 2002 the Board determined that alternative schools would not receive an achievement profile using the conventional AZ LEARNS methodology, and that ADE should develop an alternate method for evaluating these schools. Definition of an Alternative School The following is the definition of an alternative school as approved by the Board of Education in 2002. There are currently 171 schools that have been granted alternative school status. 1. A school operated by a school district must have adopted a mission statement that clearly identifies its purpose and intent to serve a specific student population (please see criterion three) that will benefit from an alternative school setting. A charter school must be expressly chartered to serve a specific student population that will benefit from an alternative school setting. 2. The educational program and related student services of the school must match the mission or charter of the school. 3. The school must intend to serve students exclusively in one or more of the following categories: Students with behavioral issues (documented history of disruptive behavior) Students identified as dropouts Students in poor academic standing who are either severely behind on academic credits (more than one year) or have a demonstrated pattern of failing grades Pregnant and/or parenting students Adjudicated youth 4. Any school offering secondary instruction for academic credit used to fulfill Arizona State Board of Education graduation requirements (in part or in full) must offer a diploma of high school graduation. Arizona's Accountability System: Volume I Arizona Department of Education 29 General Process to Produce Achievement Profiles for Alternative Schools The small number and wide variety of alternative schools makes it difficult to develop reliable parameters for measures of school performance. Consequently, a single method of evaluating alternative schools was created that employed performance measures available to the majority of schools. In plainer terms, it would be too inaccurate to develop separate methods for large alternative high schools, small alternative high schools, alternative middle schools, and small alternative middle schools. Given the small number of schools in each category, benchmark performances and expectations would be very unreliable. Consequently, ADE developed a single rubric to evaluate all alternative schools. The method for calculating an achievement profile for alternative schools is as follows: 1. AIMS scale score points are calculated using Status only. Status points were calculated like the regular schools based on percentage of students passing the AIMS test. However, due to the small size of many of these schools, this was computed based on data from at least 32 students or from student scores from 2008 back to 2005, whichever came first. 2. Added evidence points are not included in the achievement profile for alternative schools. Because most alternative schools below the high school level are quite small and have highly mobile populations, there would be little or no students in the Measure of Academic Progress analysis. Consequently, the MAP analysis would be highly volatile and inaccurate. 3. Points based on an alternative school’s dropout rate are calculated as for all other schools. 4. As with other schools, alternative schools will receive one scale score point if they make AYP for the current year. 5. Alternative schools will only receive labels of performing and underperforming. Because of the uncertainty of measurement associated with small sample sizes and the high stakes of school labels, schools initially determined to be underperforming receive a ―second look.‖ Instead of determining baseline groups based on the mean percent of students passing AIMS, an alternate baseline group for these schools is determined based on the upper bound of a 95 percent confidence interval around the mean. If a school initially determined to be underperforming moves to a higher classification due to the ―second look,‖ that school will receive a performing label. Calculation of an Achievement Profile for an Alternative School In addition to AIMS scale score points, alternative schools also earn scale score points via their dropout rates and their current AYP status. The methods used to calculate scale score points earned by alternative schools for these performance measures are the same as the methods used for other schools. Figures 11.1 and 11.2 give a graphical summary of the method for evaluating alternative schools at the elementary and high school levels. Arizona's Accountability System: Volume I Arizona Department of Education 30 A school may receive up to six scale score points for each subject/grade combination it serves. This is represented by the far left column in each figure. A school may receive one point if it has made adequately yearly progress (AYP) under the methodology mandated by the No Child Left Behind Act. This is represented by the column second-from-left in each figure. If a school is a high school it may receive one point based on its dropout rate. This is the third-from-left column in Figure 11.2. The total points earned by a school are added up and compared to the school classification scale to determine a school’s preliminary classification—the final column in Figures 11.1 and 11.2. Figure 11.1. Method for Evaluating Alternative Schools (Elementary Schools) Each Subject/Grade Whole School Status Grouping School Classification Scale AYP 6 5 1 4 Made AYP 3 2 = + P U 1 0 0 Missing Data Missed AYP Arizona's Accountability System: Volume I Arizona Department of Education 31 Figure 11.2. Method for Evaluating Alternative Schools (High Schools) Each Subject/Grade Status Grouping Whole School Whole School AYP Dropout Rate 1 1 Made Dropout Rate Requirement School Classification Scale 6 5 4 3 + Made AYP = + P 2 1 0 Missing Data U 0 Missed AYP 0 Did not make requirements Classification Scales for Alternative Schools The following classification scales are used for alternative schools. Separate classification scales are used for high schools and elementary schools due to extra points being available to high schools from the dropout rate. Table 11.1. Alternative School Classification Cut Points Subject/Grade Subject/Grade Subject/Grade Subject/Grade Combination 1 Combination 3 Combination 6 Combination 9 Underperforming Elementary Schools Performing Underperforming High Schools Performing < 2.0 < 5.0 < 11.0 < 16.0 2.0 5.0 11.0 16.0 < 2.0 < 6.0 < 11.0 < 18.0 2.0 6.0 12.0 18.0 Arizona's Accountability System: Volume I Arizona Department of Education 32 12. AZ LEARNS Achievement Profiles for K-2 Schools When AZ LEARNS achievement profiles were first issued in 2002 the Board determined that K-2 schools would not receive an achievement profile using the conventional AZ LEARNS methodology, and that ADE should develop an alternate method for evaluating these schools. In 2004, the ADE published profiles for K-2 schools for the first time. K-2 schools are schools that serve only grades kindergarten through second grade. Since the AIMS is not administered at any of the grades served by these schools, the AZ LEARNS profiles are based solely on the performance of the schools’ second graders on the state’s norm-referenced test. The method of calculating the profile for these schools is straightforward: 1. The average normal curve equivalents (NCE) on the reading and math portions of the test are calculated for the most current year for a school’s second graders. 2. The average normal curve equivalents for the school are added together, and 3. Compared to a scale to determine the school’s label. Example. In 2008, the average NCE for second graders in Gila Monster Primary was 52 for math and 48 for reading. The two averages summed together yield 52 + 48 = 100 points. This is sufficient for Gila Monster Primary to be a highly performing school. Table 12.1 provides the performance thresholds for K-2 schools. Table 12.1. AZ LEARNS Scale for K-2 Schools Achievement Profile Points Underperforming <70 Performing 70 to 96.9 Highly Performing 97 to 105.9 Excelling 106 or more Arizona's Accountability System: Volume I Arizona Department of Education 33 Appendix I: Substantive Appeal Committee Evaluation Rubric for AZ LEARNS Evaluation Criteria Data Calculation Discrepancies i.e., school attempts to compare data details with their data sets and gets different numbers Special Circumstances Outside the Control of School/District Administration or Management i.e., school indicates significant teacher attrition; environmental issues/events; adverse testing conditions; school/community emergency/crisis Initial Review (Please check the applicable option) Not ADE data are Data does not match applicable accurate and that of ADE. School calculations are submits evidence of correct. discrepancies and provides additional data. Review of Evidence Provided Compelling evidence Comments Not compelling evidence Not applicable evidence Not applicable Policy/Methodology Issues i.e., school disagrees with use of baseline Arizona's Accountability System: Volume I Special circumstances that were outside of the school’s control, were not a substantial cause of the overall school performance. School had a situation that was unavoidable and outside of the school’s control and hindered the test administration or student performance. This situation resulted in adverse data for the year(s) in question. Compelling evidence Not compelling evidence Not applicable evidence The ADE will not accept/review appeals related to policy/methodology. Arizona Department of Education 34 Appendix II: Substantive Appeal Rubric for AZ LEARNS Team Decision AZ LEARNS Substantive Appeal (Results represent group consensus regarding appeal) Reason Reviewed Data Calculation Discrepancies Special Circumstances Policy/Methodological Issues Initial Review Review of Evidence N/A Correct Data N/A Compelling Not data/calculation does not evidence compelling match evidence N/A Did not cause Adverse N/A Compelling Not overall result evidence compelling performance based on evidence situation The ADE will not accept/review appeals related to policy/methodology. Please indicate appropriate response(s) by checking within the box(es) provided. Committee Recommendation: Granted Denied Final Appeal Decision: Granted Denied Comments: Appeal Result: Arizona's Accountability System: Volume I Arizona Department of Education 35 Appendix III: Substantive Appeal Committee Evaluation Rubric for AZ LEARNS Evaluation Criteria Data Calculation Discrepancies i.e., school attempts to compare data details with their data sets and gets different numbers Special Circumstances Outside the Control of School/District Administration or Management i.e., school indicates significant teacher attrition; environmental issues/events; adverse testing conditions; school/community emergency/crisis Initial Review (Please check the applicable option) Not ADE data are Data does not match applicable accurate and that of ADE. School calculations are submits evidence of correct. discrepancies and provides additional data. Review of Evidence Provided Compelling evidence Comments Not compelling evidence Not applicable evidence Not applicable Policy/Methodology Issues i.e., school disagrees with use of baseline Arizona's Accountability System: Volume I Special circumstances that were outside of the school’s control, were not a substantial cause of the overall school performance. School had a situation that was unavoidable and outside of the school’s control and hindered the test administration or student performance. This situation resulted in adverse data for the year(s) in question. Compelling evidence Not compelling evidence Not applicable evidence The ADE will not accept/review appeals related to policy/methodology. Arizona Department of Education 36 Appendix IV: Substantive Appeal Rubric for AZ LEARNS Team Decision AZ LEARNS Substantive Appeal (Results represent group consensus regarding appeal) Reason Reviewed Data Calculation Discrepancies Special Circumstances Policy/Methodological Issues Initial Review Review of Evidence N/A Correct Data N/A Compelling Not data/calculation does not evidence compelling match evidence N/A Did not cause Adverse N/A Compelling Not overall result evidence compelling performance based on evidence situation The ADE will not accept/review appeals related to policy/methodology. Please indicate appropriate response(s) by checking within the box(es) provided. Committee Recommendation: Granted Denied Final Appeal Decision: Granted Denied Comments: Appeal Result: Arizona's Accountability System: Volume I Arizona Department of Education 37