Arizona’s School Accountability System 2007 Technical Manual Tom Horne Superintendent of Public Instruction Arizona Department of Education Volume I: AZ LEARNS Achievement Profiles Research & Evaluation Arizona Department of Education Robert Franciosi, Deputy Associate Superintendent Achieve@azed.gov Published by the Arizona Department of Education, January 2008. The Arizona Department of Education of the State of Arizona does not discriminate on the basis of race, religion, color, national origin, sex, disability or age in its programs, activities or in its hiring and employment practices. If you have questions or grievances related to this guideline, please contact the Administrative Services DAS at (602)542-3186. Table of Contents 1. INTRODUCTION..................................................................................................................... 3 WHAT’S NEW FOR THE 2007 AZ LEARNS ACHIEVEMENT PROFILES ......................................... 3 2. OVERVIEW OF THE AZ LEARNS EVALUATION SYSTEM......................................... 4 GENERAL PROCESS TO CALCULATE AN ACHIEVEMENT PROFILE ................................................. 5 3. TIMELINE ................................................................................................................................ 6 TIMELINE ..................................................................................................................................... 6 4. CALCULATION OF AIMS STATUS AND GROWTH POINTS ...................................... 7 AWARDING STATUS POINTS ......................................................................................................... 7 AWARDING GROWTH POINTS ....................................................................................................... 8 CALCULATION OF SUBJECT/GRADE SCALE POINTS FROM AIMS ............................................... 11 SPECIAL CASES: NEW SCHOOLS, AND MISSING DATA ............................................................... 12 5. THE MEASURE OF ACADEMIC PROGRESS................................................................. 14 THE GROWTH INDEX .................................................................................................................. 14 CALCULATING EXPECTED GROWTH ........................................................................................... 14 AWARDING AZ LEARNS POINTS.............................................................................................. 16 SCHOOLS WITHOUT MAP........................................................................................................... 16 6. GRADUATION AND DROPOUT RATES.......................................................................... 17 GRADUATION RATE ................................................................................................................... 17 DROPOUT RATE ......................................................................................................................... 18 7. EVALUATING PERFORMANCE ON THE ENGLISH LANGUAGE ASSESSMENT ............................................................................................................................ 19 8. EVALUATING THE TOTAL SCALE SCORE VALUE TO DETERMINE A SCHOOL CLASSIFICATION .................................................................................................................... 20 9. APPLICATION OF THRESHOLD CRITERIA FOR EXCELLING AND HIGHLY PERFORMING SCHOOLS ...................................................................................................... 22 10. AZ LEARNS ACHIEVEMENT PROFILE APPEALS PROCESS................................. 26 PROCEDURE AND TIMELINE ....................................................................................................... 26 11. AZ LEARNS ACHIEVEMENT PROFILES FOR ALTERNATIVE SCHOOLS ......... 31 DEFINITION OF AN ALTERNATIVE SCHOOL ................................................................................ 31 GENERAL PROCESS TO PRODUCE ACHIEVEMENT PROFILES FOR ALTERNATIVE SCHOOLS ......... 32 CALCULATION OF AN ACHIEVEMENT PROFILE FOR AN ALTERNATIVE SCHOOL ......................... 32 CLASSIFICATION SCALES FOR ALTERNATIVE SCHOOLS ............................................................. 34 12. AZ LEARNS ACHIEVEMENT PROFILES FOR K-2 SCHOOLS................................ 35 APPENDIX I: SUBSTANTIVE APPEAL COMMITTEE EVALUATION RUBRIC FOR AZ LEARNS ..... 36 APPENDIX II: SUBSTANTIVE APPEAL RUBRIC FOR AZ LEARNS .............................................. 37 Arizona's Accountability System: Volume I Arizona Department of Education i List of Tables and Figures Table 4.1. Status Points Awarded............................................................................................10 Table 4.2. Number of Students................................................................................................12 Table 4.3. Growth Points.........................................................................................................13 Table 4.4. AIMS Point Distributions by Status and Growth Groups.......................................14 Table 4.5. Rules for Determining UB95 for Small n and p.....................................................15 Table 5.1. Parameters for Calculating Expected Growth for Math.........................................17 Table 5.2. Parameters for Calculating Expected Growth for Reading....................................17 Table 8.1. Elementary School Classification Cut Points .........................................................22 Table 8.2. High School Classification Cut Points....................................................................23 Table 9.1. Parameters for Calculating Z-Scores ......................................................................26 Table 9.2. Number of Students Exceeding the Standard - Third Grade..................................26 Table 9.3. Average Z-Scores...................................................................................................27 Figure 11.1 Method for Evaluating Alternative Schools (Elementary Schools) .....................35 Figure 11.2 Method for Evaluating Alternative Schools (High Schools) ...............................36 Table 11.1 Alternative School Classification Cut Points ........................................................36 Table 12.1 AZ LEARNS Scale for K-2 Schools .....................................................................37 Arizona's Accountability System: Volume I Arizona Department of Education ii 1. Introduction In November of 2001, Arizona voters approved Proposition 301 which, among other things, provided funds to the Arizona Department of Education (ADE) to develop “a system to measure school performance based on student achievement, including student performance on the AIMS test.” The legislative requirements for the accountability system are stated in section 15-241 (ARS § 15-241) of the Arizona Revised Statutes. The accountability system created to satisfy the statute is referred to as the Arizona LEARNS. The school evaluation given by ADE to each school is referred to as the school’s achievement profile. This manual describes the method used to generate the 2007 achievement profiles. It provides formulas, parameters, and business rules that make up the profile calculation. It also describes the AZ LEARNS process for 2007. Its intent is to document and explain the methods used and justify the policies adopted. What’s New for the 2007 AZ LEARNS Achievement Profiles In the 2006 session, the legislature amended the law governing the AZ LEARNS school evaluation process to require that the “results of English language learners tests” be included as part of the AZ LEARNS school evaluation. {ARS 15-241 (D) (3) and ARS 15-241 (E) (4)}. On June 25, 2007 the State Board of Education approved changes, detailed in chapter 7, to the AZ LEARNS formula to comply with this requirement. Arizona's Accountability System: Volume I Arizona Department of Education 3 2. Overview of the AZ LEARNS Evaluation System This section provides an overview of how AZ LEARNS achievement profiles are determined. More detailed discussions of the methodology used to determine the profiles, including descriptions of equations, algorithms, and data used are given in subsequent chapters. Arizona law (ARS § 15-241) mandates that the Arizona Department of Education shall compile an annual achievement profile for each public school. It specifies that the profiles of schools serving grades K-8 shall be based on: • Arizona Measure of Academic Progress (MAP). • Percent of students who pass AIMS. • Performance on the English language proficiency test. The law specifies that the profiles of high schools shall be based on: • Percent of students who pass AIMS. • Drop out rate • Graduation rate • Performance on the English language proficiency test. A school that serves both grades K-8 and high school receives two separate achievement profiles. The law also calls for the ADE to use a research based methodology that shall: • Include performance of pupils at all achievement levels • Account for pupil mobility • Account for the distribution of pupil achievement • Include longitudinal indicators of academic performance. A research based methodology is defined as “the systematic and objective application of statistical and quantitative research principles to determine a standard measurement of acceptable academic progress for each school”. Arizona's Accountability System: Volume I Arizona Department of Education 4 The law also calls for a system of parallel achievement profiles for accommodation schools/alternative schools as defined by the Board of Education. General Process to Calculate an Achievement Profile The achievement profile for a school serving grades 3-8 consists of the following performance measures: 1. A status measure based on the performance of students on all three sections of the AIMS (reading, writing, and mathematics) in the current year. 2. A measure of improvement in aggregate student performance on the AIMS compared to the baseline year. 3. A measure of growth in individual student performance. Academic Progress (MAP). This is the Measure of 4. A measure of student performance on the state’s English language proficiency assessment: AZELLA. 5. A measure of whether the school made Adequate Yearly Progress (AYP) as defined by the No Child Left Behind Act of 2001. In order to comply with the federal requirement that the state have an integrated accountability system, a school’s AYP determination is factored into the calculation of its achievement profile. 6. In order to create the incentive for schools to improve the achievement of average and above-average students, a school cannot earn the highly performing or excelling labels unless the percentage of its students exceeding the standard on AIMS met specific thresholds. 7. The achievement profile for a high school is made up of all of the above components except for measure 3 (MAP). In addition, the following performance measures are used for high schools: • Dropout rate. • Graduation rate. Schools are awarded scale score points based on their performance on measures one through five, and seven. Scale score points are then summed up for each school and compared to a scale that relates scale score points to the five profile labels: excelling, highly performing, performing plus, performing, and underperforming. Performance measure six is then examined to determine if the school has earned the highest labels of highly performing or excelling. Arizona's Accountability System: Volume I Arizona Department of Education 5 3. Timeline Districts and charter holders are solely responsible for submitting the data necessary for calculating achievement profiles for their schools and for ensuring its accuracy. From June 13, 2007 through September 21, 2007 schools and districts were given an opportunity to review and correct their testing data through on-line applications. The primary purpose of the process was to allow districts and charter holders to correct the information for individual students. From July 13, 2007 through August 29, 2007 schools and districts were given the opportunity to review and correct the data used for calculating the five-year graduation rate and dropout rates used in the AZ LEARNS profile. All program membership and demographic information relevant to AZ LEARNS profiles were taken by matching test records to the state’s SAIS database of student records. Consequently the only information that schools needed to correct in the ADE AIMS testing file were students’ SAIS IDs (needed for matching). If program membership or other information was incorrect, schools and districts were required to correct it in the SAIS database. Schools were not allowed to correct the indicators for alternate testing modifications. IMPORTANT NOTE: The criteria used to select AIMS scores for AZ LEARNS profiles differ from the criteria used to select scores for adequate yearly progress under NCLB. Indeed, the criteria differ among the separate components of the AZ LEARNS evaluation. The criteria also differ from the scores provided to schools by the testing contractor, the scores publicly reported by ADE, and the scores available through the ADE AIMS wizard located at www.ade.az.gov/profile/publicview. Timeline The timeline for AZ LEARNS Achievement Profile process was: • June 13, 2007. Opening of data verification process. • July 13, 2007. Start of statistical review and appeals of graduation rate and dropout rate data. • July 26, 2007. Opening of application process for alternative schools. • August 24, 2007. Closing of application process for alternative schools. • August 29, 2007. Closing of statistical review and appeals of graduation rate data. • September 11, 2007. Preliminary release of AZ LEARNS achievement profiles for all schools. Opening of window for appeals submissions. • September 21, 2007. Closing of appeals window for all schools. • October 15, 2007. Public release of AZ LEARNS achievement profiles for all schools. Arizona's Accountability System: Volume I Arizona Department of Education 6 4. Calculation of AIMS Status and Growth Points In the AZ LEARNS profile, schools are awarded scale score points based on student performance on the AIMS. Points are determined by performance in the current year, and improvement in student performance from the baseline year. Awarding Status Points The following method is used to calculate status points for every subject and grade offered by a school in which the AIMS test is administered. First, calculate the percent passing in the current year. The percentage is compared to a scale which in turn gives the status points achieved for the subject and grade. Percent Pa ssing in the current year = # Students Passing AIMS in the current year # Students tested in the current year The points are rounded to the nearest hundredth e.g. .675 = .68; .672 = .67. Data Used A student’s score is excluded from the calculation if any of the following criteria are met: 1. The student was not matched to SAIS with a valid ID. 2. Student received no score on the test. 3. Student was not English proficient. A student was considered not proficient if it was indicated that she was a participant in the English language learner program for three years or less. Starting in 2006, counting for the number of years in the ELL program started in kindergarten rather than first grade. To remain consistent, test scores for third graders in 2006 and 2007 were excluded if the students had been participants in the ELL program for four years or less. 4. The student received an alternate accommodation on the test. 5. The student was not enrolled in the school for the full academic year. A student was considered enrolled for the full academic year if she enrolled in a school during the first 10 school days of the school year and remained enrolled up through the testing date. The calculation for high schools includes all students in grades 10 through 12 who have taken the AIMS either in the fall or spring. If a high school student took the test twice in a school year in the same school, the highest of the two scores was used. Arizona's Accountability System: Volume I Arizona Department of Education 7 Grade 3 3 3 4 4 4 5 5 5 6 6 6 7 7 7 8 8 8 HS HS HS Subject Math Read Writ Math Read Writ Math Read Writ Math Read Writ Math Read Writ Math Read Writ Math Read Writ Status Points 1 <51% <46 <56 <46 <40 <42 <42 <42 <47 <34 <38 <49 <35 <41 <64 <27 <35 <58 <5 <16 <18 Table 4.1 Status Points Awarded Status Status Status Points 2 Points 3 Points 4 51-64% 65-78% 79-88% 46-59 60-73 74-84 56-67 68-78 79-87 46-60 61-75 76-86 40-53 54-69 70-81 42-53 54-65 66-77 42-56 57-72 73-85 42-56 57-73 73-84 47-57 58-69 70-80 34-49 50-66 67-81 38-52 53-69 70-82 49-60 61-73 74-83 35-49 50-66 67-81 41-54 55-69 70-82 64-73 74-82 83-89 27-41 42-59 60-76 35-48 49-65 66-79 58-70 71-82 83-90 5-13 14-29 30-51 16-28 29-46 47-65 18-30 31-48 49-67 Status Points 5 89-94% 85-92 88-93 87-93 82-90 78-85 86-92 85-92 81-87 82-90 83-91 84-90 82-90 83-90 90-94 77-87 80-88 91-95 52-70 66-79 68-80 Status Points 6 >=95% >=93 >=94 >=94 >=91 >=86 >=93 >=93 >=88 >=91 >=92 >=91 >=91 >=91 >=95 >=88 >=89 >=96 >=71 >=80 >=81 A school is awarded status points for each subject/grade it offers in which the AIMS test is administered. Example. In the current year, 66 percent of the students in Gila Monster Elementary passed the math portion of the third grade AIMS. This value places the subject/grade in Status grouping three. Gila Monster Elementary has earned three status points for this particular subject/grade. Awarding Growth Points Determining a school’s growth points for each subject/grade combination is based on student movement out of the falls far below (FFB) category and student movement into the meet/exceeds (M/E) category. The growth points earned are determined by the difference between average percentages in 2006 and 2007 and the baseline percentage (2005). Schools are expected to increase the percentage of students that meet the standards over time, and decrease the percentage that fall far below the standards over time. Growth points were only awarded for grades three, five, eight, and high school. Arizona's Accountability System: Volume I Arizona Department of Education 8 Equations Used to Calculate Growth Points Growth points for a subject/grade are calculated in the following seven steps. Equation 1 Baseline FFB = # Students FFB in baseline year # Tested in baseline year Baseline M/E = # Students M/E in baseline year # Tested in baseline year Equation 2 Equation 3 2 - Yr. Avg. FFB = # Students FFB in most recent two years # Tested in most recent two years 2 - Yr. Avg. M/E = # Students M/E in most recent two years # Tested in most recent two years Equation 4 Equation 5 Change FFB = 2 - Yr. Avg. FFB - Baseline FFB Equation 6 Change M/E = 2 - Yr. Avg. M/E - Baseline M/E Equation 7 Growth Points = Change M/E - Change FFB All values are rounded to the nearest hundredth e.g. .675 = .68; .672 = .67. Data Used The rules for using student scores for the growth measure are the same as for the status measure. For high school students who took the test multiple times, the highest score in the baseline year is used in the baseline percentage, and the high score in the most recent two years is used in the two-year average. Arizona's Accountability System: Volume I Arizona Department of Education 9 Example. The following example demonstrates how growth points are calculated. Table 4.2 shows three years of AIMS scores for a single subject and grade for a hypothetical school. Year 2004 2005 2006 Table 4.2. FFB 15 10 10 Number of Students A M E 35 30 30 35 30 35 25 35 40 Total 110 110 110 The following equations show the steps used to calculate the growth points given the test scores in the above table. Equation 1 Baseline FFB = 15 = 0.1364 110 Baseline M/E = 60 = 0.5455 110 Equation 2 Equation 3 2 - Yr. Avg. FFB = 10 + 10 = 0.0909 110 + 110 2 - Yr. Avg. M/E = 65 + 75 = 0.6364 110 + 110 Equation 4 Equation 5 Change FFB = 0.0909 - 0.1364 = - 0.0455 Equation 6 Change M/E = 0.6364 - 0.5455 = 0.0909 Equation 7 Growth Points = 0.0909 - (-0.0455) = 0.1364 = .14 Arizona's Accountability System: Volume I Arizona Department of Education 10 The growth point thresholds are given in table 4.3. Please refer to the technical manual for the 2003 profiles (release date February 17, 2004) for a description of how the cut points were established. Grade 3 3 3 5 5 5 8 8 8 HS HS HS Subject Math Read Writ Math Read Writ Math Read Writ Math Read Writ Growth Points 1 <-15% <-20 <-13 <-8 <-10 <0 <18 <-11 <17 <6 <-16 <-24 Table 4.3. Growth Points Growth Growth Growth Points 2 Points 3 Points 4 -15 - -7 -6 - 1% 2 - 9% -20 - -14 -13 - -7 -6 – 0 -13 - -5 -4 – 3 4 -12 -8 – 0 1–8 9 -17 -10 - -3 -2 – 5 6 – 13 0–9 10 – 20 21 – 31 18 – 26 27 – 35 36 – 44 -11 - -3 -2 – 7 8 – 16 17 – 27 28 – 38 39 – 49 6 – 13 14 – 21 22 – 29 -16 - -7 -6 – 3 4 – 12 -24 - -14 -13 - -4 -3 - 7 Growth Points 5 10 - 17% 1–8 13 – 21 18 – 25 14 – 22 32 – 41 45 – 53 17 – 25 50 – 60 30 – 37 13 – 22 8 - 18 Growth Points 6 >=18% >=9 >=22 >=26 >=23 >=42 >=54 >=26 >=61 >=38 >=23 >=19 Calculation of Subject/Grade Scale Points from AIMS The total scale score points derived from AIMS performance are calculated by adding the status points awarded to the growth points awarded. A 70 percent weight is given to the school’s strongest point value (baseline group or growth point group) and a 30 percent weight to the other point value. Table 5.3 shows the scale points earned per subject/grade for all combinations of baseline and growth group scale points. Example. In third grade mathematics, Gila Monster Elementary has earned three status points and four growth group scale points. Because it has earned more scale points for its growth group, the growth group points are given a 70 percent weight and the status group scale points are given a 30 percent weight. Thus, the total scale points earned for third grade math are (.7 X 4) + (.3 X 3) = 3.7. Example. In mathematics, Desert Mountain Vista High School has earned five status group scale points and two growth group scale points. Because it has earned more scale points for its status group, the status scale points are given a 70 percent weight and the baseline group scale points are given a 30 percent weight. Thus, the total scale points earned for third grade math are (.3 X 2) + (.7 X 5) = 4.1. If a subject/grade group earned six status points, 100 percent of the weight is given to the status points. If the subject/grade group has less than 16 test scores in the baseline year or any other year used in the calculations the group is only awarded status points which are weighted 100 percent. Arizona's Accountability System: Volume I Arizona Department of Education 11 Example. Gila Monster Elementary is missing data for fifth grade writing for both 2003 and 2004. Its status points for that subject/grade combination put it in status group five. 100 percent weight is given to the status group in this case. Thus it receives 5 AIMS scale points for fifth grade writing. The points derived from AIMS for all subject/grade combinations for a school are averaged across grades by subject and added to the scale score values for other performance measures. Table 4.4. AIMS Point Distributions by Status and Growth Groups Growth Point Status Point 1 2 3 4 5 1 1.0 1.7 2.4 3.1 3.8 2 1.7 2.0 2.7 3.4 4.1 3 2.4 2.7 3.0 3.7 4.4 4 3.1 3.4 3.7 4.0 4.7 5 3.8 4.1 4.4 4.7 5.0 6 6 6 6 6 6 6 4.5 4.8 5.1 5.4 5.7 6 Special Cases: New Schools, and Missing Data If a school is missing AIMS test data for a subject/grade combination it offers, it receives zero scale points for that subject/grade for the achievement profile calculation for the test year in which the data are missing. Because of the high-stakes consequences of being labeled an underperforming school, and because of the uncertainty of measurement involved with small sample sizes, it is prudent to give a school with a small number of students a “second look” if it faces the possibility of receiving an underperforming profile. If the preliminary label of a school is underperforming, then the AIMS scale score points for that school are recalculated for each subject/grade group that has less than 16 students. For each subject/grade combination with less than 16 students, the upper bound of the 95-percent confidence interval is used to calculate to which baseline group the school belongs. If the recalculated points move the school into a classification higher than underperforming, the school receives a profile performing, regardless of the points earned. Let p = the percent of students in a group passing the AIMS and n=the number of students in the group. Then the equation for the upper bound of the 95 percent confidence interval (UB95) is: UB95 = p + 1.96 p(1 − p) n . As can be seen from the equation, the confidence interval depends upon the percent of students who passed the test and the number of students tested. Thus, the confidence interval will differ among grades, subjects, and schools. Arizona's Accountability System: Volume I Arizona Department of Education 12 The equation is an approximation of the confidence interval for a binomially distributed variable. It uses the standard normal distribution and is sufficiently accurate if the group size and percentage of students passing are large enough. For small values of n and small p, a more accurate estimate of the confidence interval is made using statistical tables that provide confidence intervals for a binomially distributed variable. 1 The tables are applied using the rules given in table 4.4. Table 4.5. Rules for Determining UB95 for Small n and p. If n>=0 and n < 8, and p>= 0 and p <.04, then UB95=.42; p>=.04 and p <.10, then UB95=.50; p>=.10 and p <.20, then UB95=.60; If n>=8 and n < 10, and p>= 0 and p <.04, then UB95=.37; p>=.04 and p <.10, then UB95=.45; p>=.10 and p <.20, then UB95=.55; If n>=10 and n < 12, and p>= 0 and p <.04, then UB95=.33; p>=.04 and p <.10, then UB95=.41; p>=.10 and p <.20, then UB95=.52; If n>=12 and n < 16, and p>= 0 and p <.04, then UB95=.27; p>=.04 and p <.10, then UB95=.35; p>=.10 and p <.20, then UB95=.47; If n>=16 and n < 20, and p>= 0 and p <.04, then UB95=.24; p>=.04 and p <.10, then UB95=.32; p>=.10 and p <.20, then UB95=.44; If n>=20 and n < 24, and p>= 0 and p <.04, then UB95=.21; p>=.04 and p <.10, then UB95=.29; p>=.10 and p <.20, then UB95=.42; If n>=24 and n < 30, and p>= 0 and p <.04, then UB95=.18; p>=.04 and p <.10, then UB95=.27; p>=.10 and p <.20, then UB95=.38; If n>=30 and n < 40, and p>= 0 and p <.04, then UB95=.15; p>=.04 and p <.10, then UB95=.23; p>=.10 and p <.20, then UB95=.36; 1 Mansfield, Edwin. 1991. Statistics for Business and Economics, 4th Edition. New York: W.W. Norton and Company. 280-284. Arizona's Accountability System: Volume I Arizona Department of Education 13 5. The Measure of Academic Progress The measure of academic progress (MAP) is a measure of individual student academic growth measured by performance on the AIMS reading and mathematics tests. (Writing is not included in the MAP analysis.) It uses a conventional value-added approach to measuring student progress. Calculating the number of points earned from MAP has three steps: 1. Individual expectations of one year’s growth (OYG) are calculated for each student. 2. The expectation of OYG is subtracted from the actual growth achieved by the student to determine a growth index. 3. The average growth index for school is calculated by averaging growth indices for individual students across all grades and subjects. Schools are awarded AZ LEARNS points based on this average. Actual growth, expected growth, and the growth index are expressed in AIMS scale score points. The Growth Index In order to control for a ceiling effect and for student mobility, the MAP analysis is done using a standard value-added model. The value-added model is used to calculate an estimate of the expected for each student for each subject. The expected growth is then subtracted from actual growth to determine a student’s growth index. Example. Student A scored 478 on the 6th grade math test in 2006. Her expected growth for 7th grade in 2007 is 38. In 2007 she scores a 528 on the 7th grade math test. Her actual growth is 528 – 478 = 50. Her growth index is 50 – 38 = 12. Example. Student B scored 490 on the 4th grade reading test in 2006. His expected growth for 5th grade in 2007 is 12. In 2007 he scores a 500 on the 5th grade reading test. His actual growth is 500 – 490 = 10. His growth index is 10 – 12 = -2. The results of the MAP analysis for students and schools are reported in terms of the growth index. Growth indices for individual students can be averaged across schools, grades, and subjects to measure performance. Calculating Expected Growth The expected growth for an individual student is calculated using the following formula: Expected Growth = A + B X (2005 scale score) + C X (FAY); Arizona's Accountability System: Volume I Arizona Department of Education 14 where FAY = 1 if the student has been enrolled for the full 2007 academic year and FAY = 0 if not. A student is considered to have been enrolled the full 2007 academic year if she has enrolled within the first two weeks of the school year and remained enrolled up to the first day of the testing window. The parameters A, B, and C are given in the tables below: Table 5.1. Parameters for Calculating Expected Growth for Math Student’s Grade Current Year A B C 4 98.9308 -0.1514 6.027 5 107.7715 -0.174 5.7754 6 75.6373 -0.1268 6.999 7 121.1295 -0.1896 7.1203 8 54.1785 -0.0823 7.0308 Table 5.2. Parameters for Calculating Expected Growth for Reading Student’s Grade Current Year A B C 4 120.9638 -0.2269 3.7108 5 153.1619 -0.2879 4.106 6 105.8317 -0.2016 4.343 7 88.3119 -0.1492 5.1193 8 89.8856 -0.1688 7.0786 Example. Student A scored 478 on the 6th grade math test in 2006. She has been enrolled in her current school all year. Her expected growth for 7th grade in 2007 is 120.1295 + (-0.1896) X 478 + 7.1203 = 36.62 = 37. Example. Student B scored 490 on the 4th grade reading test in 2006. He has not been enrolled in his current school for the full year. His expected growth for 5th grade in 2007 is 153.1619 + (0.2879) X 490 = 12.09 = 12. Please see the 2006 MAP report available on the ADE web site for a description of how the parameters were developed. Arizona's Accountability System: Volume I Arizona Department of Education 15 Awarding AZ LEARNS Points The Growth Index for a school is calculated by averaging the student-level growth indices across all grades for math and reading. The number of AZ LEARNS points earned by a school is calculated with the following formula: AZ LEARNS points = 5.2 + .22 X (School’s growth index.) The scale is bounded at the bottom by 2. So if the result of the above formula is less than 2, a school would earn 2 points. No maximum is placed on the number of points a school may earn. This is to award schools that make outstanding growth with students. The scale was developed so that MAP would receive the same weight in the AZ LEARNS profile that it has in previous years. Example. School A’s growth index is -1.7. The number of AZ LEARNS points awarded to the school is 5.2 + .22 X (-1.7) = 4.8. Example. School B’s growth index is 3.2. The number of AZ LEARNS points awarded to the school is 5.2 + .22 X (3.2) = 5.9. Example. School C’s growth index is -20.0. Since 5.2 + .22 X (-20.0) = 0.8 is less than 2. The number of AZ LEARNS points awarded to the school is 2. Schools without MAP MAP is only evaluated for grades four through eight using student scores from grades three through eight.) Schools do not receive points from MAP if they do not have a grade evaluated for MAP or if they have less than 16 scores in the MAP analysis. Arizona's Accountability System: Volume I Arizona Department of Education 16 6. Graduation and Dropout Rates In their AZ LEARNS profile, high schools may earn one point for their graduation rate and one point for their dropout rate. Graduation Rate The Graduation Rate is a five-year, longitudinal measure of how many students graduate from high school. The formula used to calculate the graduation rate is: Graduation Rate = Number in cohort that graduated in within five years Original cohort + transfers in - transfers out The graduation rate is rounded to two digits, e.g.: .705=.71; .704=.70. A school can earn one AZ LEARNS scale point for its graduation rate in one of three ways: 1. If its three-year average graduation rate is 90 percent or greater. 2. If its three-year average graduation rate is less than 90 percent and its baseline rate is greater than or equal to 74 percent, then a school will earn one point if the average of its graduation rates over the most recent three years is 1 percentage point greater than its baseline rate. 3. If its three-year average graduation rate is less than 90 percent and its baseline rate is less than 74 percent, then a school will earn one point if the average of its graduation rates over the most recent three years is 2 percentage point greater than its baseline rate. The three-year average is calculated by taking the total number of combined five-year graduates for the most recent three graduating classes and dividing by the total number of students in the combined classes. Graduation Rate = 2004 + 2005 + 2006 five - year graduates Combined number of students in three cohorts The baseline graduation rate is the graduation rate for the cohort class of 2000. The graduation rate is not evaluated for schools that do not have a baseline. Instead, the number of points earned through the dropout rate is doubled. Arizona's Accountability System: Volume I Arizona Department of Education 17 Dropout Rate The dropout rate is an annual measure of how many students drop out of a school during a twelve-month reporting period. Dropout Rate = Number of students who dropped out Number of students who were enrolled during the school year Like the graduation rate, a school can earn one AZ LEARNS scale point for its dropout rate in one of three ways: 1. If the three-year average dropout rate is 6 percent or less, the target is automatically met. 2. If its three-year average dropout rate is greater than 6 percent and its baseline rate is greater than or equal to 9 percent, then a school will earn one point if the average of its dropout rates over the most recent three years is 2 percentage point less than its baseline rate. 3. If its three-year average graduation rate is less than 6 percent and its baseline rate is less than 9 percent, then a school will earn one point if the average of its dropout rates over the most recent three years is 1 percentage point less than its baseline rate. The three-year average is calculated by taking the total number of dropouts for the most recent three years and dividing by the total number of students enrolled. Three - year dropout rate = Total number of dropouts for 2005 - 2007 Total number of students enrolled 2005 - 2007 The baseline dropout rate is the dropout rate for the 2000 school year. Schools without a baseline were evaluated only on step one above. . Arizona's Accountability System: Volume I Arizona Department of Education 18 7. Evaluating Performance on the English Language Assessment State law governing AZ LEARNS now requires that the “results of English language learners tests” be included as part of the AZ LEARNS school evaluation. {ARS 15-241 (D) (3) and ARS 15-241 (E) (4)} In order to comply with this requirement, the following change was made to the AZ LEARNS profile. A school receives one scale score point if the percentage of English language learner students reclassified during the year across all grades served is greater or equal to 30 percent. Only students identified as continuing are included in the analysis. Only groups of 16 or more students are evaluated This modification maintains the emphasis of the AZ LEARNS evaluation on how well students learn the state academic standards. Consequently, most of the weight of the evaluation is still placed on a school’s AIMS results compared to performance on the English language learner test. . Arizona's Accountability System: Volume I Arizona Department of Education 19 8. Evaluating the Total Scale Score Value to Determine a School Classification The tables below show the total number of scale score points that schools must earn in order to receive a given classification. A school may receive up to 18 scale points from status and growth; unlimited scale points from MAP if it is an elementary or middle school; a single point if it was able to reclassify 30 percent of its English language learners, and up to two graduation/drop out scale points if it is a high school. A school receives one scale point if it made adequate yearly progress (AYP). For each school, the applicable scale score cut points for classification labels are calculated by averaging the AIMS scale points received for each subject; adding the AYP and ELL points received; adding points from MAP if it is an elementary school; and adding points from the graduation/dropout rate if it is a high school. The classification cut points for elementary schools are given in the first two columns of table 8.1 and those for high schools are given in the first column of table 8.2. In order for schools to be classified as highly performing or excelling, there is an additional Z-score criteria that has to be met, which is explained in the next chapter. Table 8.1. Elementary School Classification Cut points Performance Thresholds Profile Underperforming Performing Performing Plus Highly Performing Excelling Arizona's Accountability System: Volume I Scale with MAP Non-MAP Scale <13 13-15.9 16 or more <8 8-12.9 NA 13-19 NA 16-18.9 13-14.9 19 or more 15-19 z-score 0.450.99 1.00 or more Arizona Department of Education 20 Table 8.2 High School Classification Cut Points Performance Thresholds Profile Underperforming Performing Performing Plus Highly Performing Excelling Arizona's Accountability System: Volume I Scale Points z-score <9.6 9.6-14.9 15 or more NA NA 15-16.9 0.45-0.99 17 or more 1.00 or more Arizona Department of Education 21 9. Application of Threshold Criteria for Excelling and Highly Performing Schools To ensure continued focus on improving the academic achievement of all students, including those students currently demonstrating proficiency, threshold criteria are applied to determine excelling and highly performing schools. To be deemed a highly performing or excelling school, a school must meet certain levels in the percentage of its students exceeding the standard on the AIMS. Schools must not only receive a total scale value that places them into either excelling or highly performing, but must also meet the requisite percentage of students in the exceeds-the-standard category on AIMS to be designated as either excelling or highly performing schools. The application of threshold criteria for excelling and highly performing schools results in the following scenarios. 1. A school that receives a total scale value that places it in the excelling classification and meets the requisite percentage of students in the exceeds category on AIMS necessary for an excelling classification will be designated an excelling school. 2. A school that receives a total scale value that places it in the excelling classification and did not meet the requisite percentage of students in the exceeds category on AIMS necessary for a excelling classification, but did met the requisite percentage of students in the exceeds category on AIMS necessary for a highly performing classification will be designated as a highly performing school. 3. A school that receives a total scale value that places it in the excelling classification and did not meet either the requisite percentage of students in the Exceeds category on AIMS necessary for the excelling classification or the highly performing classification will be designated as a performing school. Such schools are recognized with the non-statutory designation of performing plus. 4. A school that receives a total scale value that places it in the highly performing classification and meets the requisite percentage of students in the exceeds category on AIMS necessary for an excelling classification will be designated as a highly performing school. 5. A school that receives a total scale value that places it in the highly performing classification and meets the requisite percentage of students in the exceeds category on AIMS necessary for a highly performing classification will be designated as a highly performing school. 6. A school that receives a total scale value that places it in the highly performing classification and did not meet either the requisite percentage of students in the exceeds category on AIMS necessary for an excelling classification or highly performing Arizona's Accountability System: Volume I Arizona Department of Education 22 classification will be designated as a performing school. Such schools are recognized with the non-statutory designation of performing plus. 7. A school that receives a total scale value that places it in the performing classification will be designated as a performing school, regardless if the school meets the requisite percentage of students in the exceeds category on AIMS necessary for an excelling classification or a highly performing classification. 8. A school that receives a total scale value that places it in the underperforming classification will be designated as an underperforming school, regardless if the school meets the requisite percentage of students in the exceeds category on AIMS necessary for an excelling classification or a highly performing classification. The determination of whether a school met the goals for percent exceeding is based on a z-score calculated in the following manner: 1. For each school, the percent exceeding is calculated by grade across all three subjects for both 2005-07 and 2007. The introduction of a one-year estimation of percent exceeding is to give credit to schools that have shown improvement in the percent of students exceeding the standard. 2. The percent-exceeding is converted into a z-score for each grade by subtracting the statewide average for that grade for percent exceeding and dividing by the statewide standard deviation for that grade. (These parameters are given in table 8.1) Note: The same statewide parameters are used to calculate both the one-year and threeyear z-score. This is because the purpose of the z-score is to create standard scores comparable across grades, not to create scores comparing a school’s performance to the state norm. The z-scores are fixed parameters so if a school shows improvement over time its z-score will increase regardless of the movement of the state average over the same period. 3. The one- and three-year z-scores across all grades for a school: each school will have two z-scores: one based on its three-year average for percent exceeding, another based on the percent exceeding for the current year. 4. The higher of the one- and three-year averages are taken and compared to the performance thresholds. To be a highly performing school, the average z-score for a school must be greater than or equal to 0.45. To be an excelling school, the average zscore for a school must be greater than or equal to 1.00. Arizona's Accountability System: Volume I Arizona Department of Education 23 Table 9.1. Parameters for Calculating Z-Scores Grade Average 3 4 5 6 7 8 10 0.124576 0.111652 0.092426 0.095892 0.06592 0.052172 0.058823 Standard Deviation 0.093475 0.087267 0.080033 0.085301 0.0666 0.056997 0.082827 Example. The following table shows the third grade AIMS scores for Gila Monster Elementary over the past three years. Table 9.2. Number of Students Exceeding the Standard—Third Grade Subject Year 2003 2004 2005 Total Reading # Exceeding #Tested 25 100 24 105 26 99 75 304 Mathematics # Exceeding #Tested 24 100 23 105 25 99 72 304 Writing # Exceeding #Tested 23 100 22 105 24 99 69 304 The three-year average for percent of students exceeding the standard is: Percent Exceeding = 75 + 72 + 69 = 23.6%. 304 + 304 + 304 The one-year average for percent of students exceeding the standard is: Percent Exceeding = 25 + 25 + 24 99 + 99 + 99 = 25.3%. The three-year z-score for third grade is: z - score = .236 - .125 .093 = 1.19. The one-year z-score for third grade is: z - score = .253 - .125 .093 = 1.38. Gila Monster serves grades K-6. The one- and three-year average z-scores for the entire school are: Arizona's Accountability System: Volume I Arizona Department of Education 24 Table 9.3. Average Z-scores Grade K-2 3 4 5 6 Average One-year NA 1.38 1.02 .89 .95 1.06 Three-year NA 1.19 .86 .72 .80 .89 The one-year average is greater than 1.00, so Gila Monster would earn an excelling—if it has earned sufficient scale points. Arizona's Accountability System: Volume I Arizona Department of Education 25 10. AZ LEARNS Achievement Profile Appeals Process Procedure and Timeline In accordance with A.R.S. §15-241, school administrators are allowed the opportunity to appeal an achievement profile classification on behalf of the schools for which they are responsible. Step 1: Data Correction. The first step in completing the AZ LEARNS Appeals Process required districts and schools to review their data in order to confirm its accuracy. Data correction took place June 13 through September 21, 2007. It is important to note that districts and charter holders were solely responsible for verifying information for their districts and schools. If a district or charter holder did not verify the information for its district and schools through the correction process, the ADE assumed the schools on file and the data available were correct as listed. Step 2: Appeal Application. Administrators choosing to appeal an AZ LEARNS profile completed the AZ LEARNS Appeal Application, which was accessible via the common logon during the specified appeal window. Appeals were only accepted through the website application. Appeals sent to ADE via email, fax, or mail/delivery were not accepted. Schools were able to appeal AZ LEARNS profiles in two categories: data (statistical) and non-data (substantive) reasons – districts and schools were not limited to one category and were able to appeal in both if necessary. Statistical appeals are appeals of the accuracy of the data used in the AZ LEARNS profile. Given the extensive time allowed to view and correct the data, it is expected that any errors should be corrected by the time preliminary profiles are released. Statistical appeals were not granted unless the underlying data was corrected. Substantive appeals are arguments by districts and schools that that circumstances outside of the district’s/school’s control negatively affected school performance. Administrators that chose to appeal an AZ LEARNS profile must have clearly articulated the issues they believe merited an appeal. Administrators must have submitted evidence that the issues they believe merited an appeal directly resulted in a significant decrease in student academic achievement as demonstrated on the AIMS. The evidence must have been submitted to ADE at the time the appeal was submitted. Failure to provide this evidence resulted in the appeal not being granted. Evidence submitted after the appeal deadline closed was not considered. Once appeals were submitted through the Common Logon, the school/district/charter holder received an email verifying that the appeal was received. NOTE: In order to protect student privacy and the integrity of the appeals process, schools were asked to refer to a specific student only by that student’s SAIS ID. The SAIS ID was required so that ADE staff could verify the contentions in the appeal. Arizona's Accountability System: Volume I Arizona Department of Education 26 The ADE, if necessary, requested that a district or school administrator provide additional information/evidence to assist in the appeals process. Only those requests for additional information that were provided during the specified timeframe allotted were included in the appeals process. Requests submitted after the specified timeframe were excluded from the appeals process. Unsolicited additional information submitted after the appeal deadline was not accepted. Step 3: Appeal Resolution. After all appeals were submitted and the appeals window closed, the ADE began to process the appeals. Appeals were addressed categorically, not necessarily in the order received, so the fact that a district or school submitted its appeal during the first day of the appeal window did not mean it necessarily received a decision first during the resolution process. Statistical appeals were resolved only through recalculation of the AZ LEARNS profile by ADE staff using any corrected data submitted by the school. The purpose of a statistical appeal is principally to advise ADE staff that data was in error and has been corrected. Calculations submitted by schools via an appeal were not taken at face value nor used to alter a profile if the underlying data was not corrected. Schools that were labeled underperforming for a third consecutive year were entitled to a site visit to determine if the label was warranted. These site visits were important as the third year underperforming label merited an alternative designation of failing to meet the academic standards in accordance with statute. After the statistical review of appeals took place, schools labeled underperforming for a third consecutive time were scheduled to receive that visit. Teams from the Research and Evaluation and State Intervention sections of the ADE visited each of the third year underperforming schools to gather additional supporting data for the appeal as well as gather information related to the school’s Arizona school improvement plan (ASIP) as required by statute. All information gathered from the site visits was taken to the substantive appeal committee for use in the third stage of the appeal process. Only third year underperforming schools received site visits as part of the appeals process. Substantive appeals were resolved in a committee process. Committee members represented a diverse background of ADE staff and school administrators to ensure that appeals were considered from multiple perspectives. Appeals were evaluated using an appeals rubric approved by the State Board of Education that evaluated the argument presented and whether or not the evidence provided to support the argument was compelling. The appeals rubric consisted of a three-tiered system for appeal evaluation: 1) Initial review of the appeal to determine its merit. 2) Review of the evidence provided. 3) Committee recommendation. Initial Review. The substantive appeal rubric provided for three categories that apply during the initial review. Each appeal was classified into the categories based on the information provided in the appeals. 1. Mitigating Factors Outside the School’s Control. Appeals of this nature referenced when the school indicated significant issues that affected test scores outside of the school’s control. If a school provided information detailing a significant event that impacted test scores, which was clearly outside the school’s control, the appeal was deemed as passing the initial review. Appeals involving the adverse affect of school or Arizona's Accountability System: Volume I Arizona Department of Education 27 district policies; errors made by school or district personnel regarding test administration or data entry; or events whose impact could have been foreseen and mitigated by school or district action were not considered valid appeals. 2. Implementation of the School Improvement Plan. Appeals in this category discussed how the school was actively, consistently, and reliably implemented the school improvement plan; that the priorities of the solutions team had been addressed; the school improvement plan had been revised and updated to address assessed ongoing needs; professional development that supported the targeted goals had been planned and implemented. If a school provided information highlighting their school improvement plan, it may have accentuated the other two components on the rubric. However, per statute, a successful implementation of the school improvement plan alone cannot change a classification of a school. Additionally, only third-year underperforming schools may include information about their ASIP in their appeal. Review of Evidence. Once the appeals progressed through the first tier of the rubric, initial review, the evidence provided to support the appeal was evaluated. In this tier, three determinations were possible: 1. Compelling evidence: in this area, the school or district adequately provided information that led the committee to conclude that, had the circumstance been different, the achievement profile would have been different as well. If a school had a special circumstance that affected a certain grade and was able to demonstrate that the specific grades test scores suffered, the school was deemed to have provided compelling evidence. Again, if a school provided compelling evidence highlighting their school improvement plan, it may have accentuated the other two components on the rubric (data calculations/mitigating factors). However, per statute, a successful implementation of the school improvement plan alone could not change the classification of a school. 2. Not compelling evidence: appeals were categorized in this area when they were able to provide information that a significant issue could have impacted the school’s performance but did not provide detailed, specific information as to specific outcomes that hindered the school’s performance. 3. Not applicable evidence: if an appeal was submitted, made it through the initial review, and presented evidence that was not linked in any way to the performance of the school, the evidence was deemed not applicable. If the evidence did not directly support the claim made in the appeal, it was deemed not applicable. Committee Recommendation. Once the appeal and evidence were reviewed, the committee arrived at a decision as to the outcome of the appeal. There were three possible outcomes: 1. Appeal granted and AZ LEARNS determination changed. In these cases, the appeal successfully made it through the initial review and evidentiary stages. It was determined that the points needed to change classifications would have been earned by the school had the special circumstance/data discrepancy not occurred. Therefore, the classification for the school was changed. Arizona's Accountability System: Volume I Arizona Department of Education 28 2. Appeal granted and AZ LEARNS determination remains the same. In these cases, the appeal successfully made it through the initial review and evidentiary stages. However, it was determined that the criteria needed to change classifications were not earned by the school had the special circumstance/data discrepancy not occurred. For example, a school provided information and evidence that their AYP point was not accurately included in the calculations and the committee determined the school provided information to prove they earned the one point set aside for AYP in the AZ LEARNS formula. However, the school needed 5 points to get from underperforming to performing. Therefore, the one point earned was not enough to change their designation, so their determination remained the same. 3. Appeal Denied. In these cases, the appeal did not successfully make it through the initial review and evidentiary stages. Therefore, the classification for the school remained the same. Step 4. Notification of Result Sent to Schools. Once all appeals were resolved, notifications were sent to the districts and/or schools that had filed appeals. The contact person of record for the district/school received an email from Achieve with directions as to how to access appeal information via the Common Logon when the appeal had been processed. Districts and schools were notified before the final public release of the AZ LEARNS profiles as to the outcome of the appeal process. All appeals were final. Special Appeals for 2007 The following appeals were allowed for special circumstances during the 2006 school year. Appeal of writing scores. Due to anomalies in scoring of the writing portion of the AIMS DPA, schools serving grades three through eight had the option to appeal the status and growth points earned in their profile. Schools could appeal to be held harmless from any adverse impact by requesting that their profile be recalculated using the total points earned in 2005 for status and growth in writing. Example. School A serves grades K-5. Below are the status/growth points earned by the school in writing for 2005 and 2007. Grade 3 4 5 Average 2005 3.3 3 4.7 3.7 2007 2.3 3 3.7 3 In 2005 the school earned 3.7 points from writing. This year the school is earned 3 points in 2006. Upon appeal, the school’s final profile for 2007 would be calculated using 3.7 points rather than 3. Arizona's Accountability System: Volume I Arizona Department of Education 29 • This appeal was only available for grades 3-8. • This appeal was only granted at the level of total points earned, not for individual grades. • This appeal was only for the status/growth portion of the formula. Any impact of scoring on the percent of students exceeding was already been accounted for. Appeal of AYP determination. A school may have appealed its AYP determination if it did not make AYP because of the inclusion of ELL students or the exclusion of students with alternate accommodations, and not making AYP adversely affected its AZ LEARNS profile. • If a school appealed the inclusion of ELL students in its AYP determination, the determination was recalculated with ELL students counting as tested but not included in the AMO calculation. • If a school appealed the exclusion of students tested with alternate accommodations, the determination was recalculated with such students counting as tested but as not proficient in the AMO calculation. Arizona's Accountability System: Volume I Arizona Department of Education 30 11. AZ LEARNS Achievement Profiles for Alternative Schools In 2004, the ADE published profiles for alternative schools for the first time. Alternative schools are defined as schools that meet the Board-approved definition as schools whose sole and clearly-stated mission is to serve specific populations of at-risk students. Alternative school status is granted by application to the ADE. A.R.S. §15-241 makes an allowance for a “parallel” evaluation method for alternative schools. When AZ LEARNS achievement profiles were first issued in 2002 the Board determined that alternative schools would not receive an achievement profile using the conventional AZ LEARNS methodology, and that ADE should develop an alternate method for evaluating these schools. Definition of an Alternative School The following is the definition of an alternative school as approved by the Board of Education in 2002. There are currently 171 schools that have been granted alternative school status. 1. A school operated by a school district must have adopted a mission statement that clearly identifies its purpose and intent to serve a specific student population (please see criterion three) that will benefit from an alternative school setting. A charter school must be expressly chartered to serve a specific student population that will benefit from an alternative school setting. 2. The educational program and related student services of the school must match the mission or charter of the school. 3. The school must intend to serve students exclusively in one or more of the following categories: • Students with behavioral issues (documented history of disruptive behavior) • Students identified as dropouts • Students in poor academic standing who are either severely behind on academic credits (more than one year) or have a demonstrated pattern of failing grades • Pregnant and/or parenting students • Adjudicated youth 4. Any school offering secondary instruction for academic credit used to fulfill Arizona State Board of Education graduation requirements (in part or in full) must offer a diploma of high school graduation. Arizona's Accountability System: Volume I Arizona Department of Education 31 General Process to Produce Achievement Profiles for Alternative Schools The small number and wide variety of alternative schools makes it difficult to develop reliable parameters for measures of school performance. Consequently, a single method of evaluating alternative schools was created that employed performance measures available to the majority of schools. In plainer terms, it would be too inaccurate to develop separate methods for large alternative high schools, small alternative high schools, alternative middle schools, and small alternative middle schools. Given the small number of schools in each category, benchmark performances and expectations would be very unreliable. Consequently, ADE developed a single rubric to evaluate all alternative schools. The method for calculating an achievement profile for alternative schools is as follows: 1. AIMS scale score points are calculated using Status only. Status points were calculated like the regular schools based on percentage of students passing the AIMS test. 2. Added evidence points are not included in the achievement profile for alternative schools. Because most alternative schools below the high school level are quite small and have highly mobile populations, there would be little or no students in the Measure of Academic Progress analysis. Consequently, the MAP analysis would be highly volatile and inaccurate. 3. Points based on an alternative school’s dropout rate are calculated as for all other schools. 4. As with other schools, alternative schools will receive one scale score point if they make AYP for the current year. 5. Alternative schools will only receive labels of performing and underperforming. Because of the uncertainty of measurement associated with small sample sizes and the high stakes of school labels, schools initially determined to be underperforming receive a “second look.” Instead of determining baseline groups based on the mean percent of students passing AIMS, an alternate baseline group for these schools is determined based on the upper bound of a 95 percent confidence interval around the mean. If a school initially determined to be underperforming moves to a higher classification due to the “second look,” that school will receive a performing label. Calculation of an Achievement Profile for an Alternative School In addition to AIMS scale score points, alternative schools also earn scale score points via their dropout rates and their current AYP status. The methods used to calculate scale score points earned by alternative schools for these performance measures are the same as the methods used for other schools. Figures 11.1 and 11.2 give a graphical summary of the method for evaluating alternative schools at the elementary and high school levels. Arizona's Accountability System: Volume I Arizona Department of Education 32 • A school may receive up to six scale score points for each subject/grade combination it serves. This is represented by the far left column in each figure. • A school may receive one point if it has made adequately yearly progress (AYP) under the methodology mandated by the No Child Left Behind Act. This is represented by the column second-from-left in each figure. • If a school is a high school it may receive one point based on its dropout rate. This is the third-from-left column in Figure 11.2. • The total points earned by a school are added up and compared to the school classification scale to determine a school’s preliminary classification—the final column in Figures 11.1 and 11.2. Figure 11.1. Method for Evaluating Alternative Schools (Elementary Schools) Each Subject/Grade Whole School Status Grouping AYP School Classification Sc ale 6 5 1 4 Made AYP 3 2 = + P U 1 0 0 Missing Data Missed AYP Arizona's Accountability System: Volume I Arizona Department of Education 33 Figure 11.2. Method for Evaluating Alternative Schools (High Schools) Each Subject/Grade Status Grouping Whole School Whole School AYP Dropout Rate 1 1 Made Dropout Rate Requirement School Classification Scale 6 5 4 3 + Made AYP + = P 2 1 0 Missing Data U 0 Missed AYP 0 Did not make requirem ents Classification Scales for Alternative Schools The following classification scales are used for alternative schools. Separate classification scales are used for high schools and elementary schools due to extra points being available to high schools from the dropout rate. Table 11.1. Alternative School Classification Cut Points Subject/Grade Subject/Grade Subject/Grade Subject/Grade Combination Combination Combination Combination 1 3 6 9 Elementary Underperforming < 2.0 < 5.0 < 11.0 < 16 Schools Performing 2.0 5.0 11.0 16 High Underperforming <2.0 <6.0 <1.0 <18.0 Schools Performing 2.0 6.0 12.0 18.0 Arizona's Accountability System: Volume I Arizona Department of Education 34 12. AZ LEARNS Achievement Profiles for K2 Schools When AZ LEARNS achievement profiles were first issued in 2002 the Board determined that K-2 schools would not receive an achievement profile using the conventional AZ LEARNS methodology, and that ADE should develop an alternate method for evaluating these schools. In 2004, the ADE published profiles for K-2 schools for the first time. K-2 schools are schools that serve only grades kindergarten through second grade. Since the AIMS is not administered at any of the grades served by these schools, the AZ LEARNS profiles are based solely on the performance of the schools’ second graders on the state’s norm-referenced test. The method of calculating the profile for these schools is straightforward: 1. The average normal curve equivalents (NCE) on the reading and math portions of the test are calculated for the most current year for a school’s second graders. 2. The average normal curve equivalents for the school are added together, and 3. Compared to a scale to determine the school’s label. Example. In 2007, the average NCE for second graders in Gila Monster Elementary was 52 for math and 48 for reading. The two averages summed together yield 52 + 48 = 100 points. This is sufficient for Gila Monster to be a highly performing school. Table 12.1 provides the performance thresholds for K-2 schools. Table 12.1. AZ LEARNS Scale for K-2 Schools Points <70 70 97 106 Arizona's Accountability System: Volume I Achievement Profile Underperforming Performing Highly Performing Excelling Arizona Department of Education 35 Appendix I: Substantive Appeal Committee Evaluation Rubric for AZ LEARNS Evaluation Criteria Data Calculation Discrepancies i.e., school attempts to compare data details with their data sets and gets different numbers Special Circumstances Outside the Control of School/District Administration or Management i.e., school indicates significant teacher attrition; environmental issues/events; adverse testing conditions; school/community emergency/crisis Policy/Methodology Issues i.e., school disagrees with use of baseline Initial Review (Please check the applicable option) Not ADE data are Data does not match applicable accurate and that of ADE. School calculations are submits evidence of correct. discrepancies and provides additional data. Review of Evidence Provided Compelling evidence Comments Not compelling evidence Not applicable evidence Not applicable Special circumstances that were outside of the school’s control, were not a substantial cause of the overall school performance. School had a situation that was unavoidable and outside of the school’s control and hindered the test administration or student performance. This situation resulted in adverse data for the year(s) in question. Compelling evidence Not compelling evidence Not applicable evidence The ADE will not accept/review appeals related to policy/methodology. Arizona's Accountability System: Volume I Arizona Department of Education 36 Appendix II: Substantive Appeal Rubric for AZ LEARNS Team Decision AZ LEARNS Substantive Appeal (Results represent group consensus regarding appeal) Reason Reviewed Data Calculation Discrepancies Special Circumstances Policy/Methodological Issues Initial Review Review of Evidence N/A Correct Data N/A Compelling Not data/calculation does not evidence compelling match evidence N/A Did not cause Adverse N/A Compelling Not overall result evidence compelling performance based on evidence situation The ADE will not accept/review appeals related to policy/methodology. Please indicate appropriate response(s) by checking within the box(es) provided. Committee Recommendation: Granted Denied Final Appeal Decision: Granted Denied Comments: Appeal Result: Arizona's Accountability System: Volume I Arizona Department of Education 37