Arizona’s School Accountability System 2006 Technical Manual Tom Horne Superintendent of Public Instruction Arizona Department of Education Volume I: AZ LEARNS Achievement Profiles Research & Evaluation Arizona Department of Education Robert Franciosi, Deputy Associate Superintendent Achieve@azed.gov Published by the Arizona Department of Education, January 2007. The Arizona Department of Education of the State of Arizona does not discriminate on the basis of race, religion, color, national origin, sex, disability or age in its programs, activities or in its hiring and employment practices. If you have questions or grievances related to this guideline, please contact the Administrative Services DAS at (602)542-3186. Table of Contents 1. INTRODUCTION......................................................................................................... 4 What’s New for the 2006 AZ LEARNS Achievement Profiles ......................................... 4 2. OVERVIEW OF THE AZ LEARNS EVALUATION SYSTEM............................. 5 General Process to Calculate an Achievement Profile ....................................................... 6 3. DATA CORRECTION AND TIMELINE.................................................................. 7 Timeline .............................................................................................................................. 7 4. CALCULATION OF AIMS STATUS AND GROWTH SCALE POINTS ............ 9 Awarding Status Points....................................................................................................... 9 Awarding Growth Points .................................................................................................. 10 Calculation of Subject/Grade Scale Points from AIMS ................................................... 13 Special Cases: New Schools, and Missing Data............................................................... 14 5. THE MEASURE OF ACADEMIC PROGRESS..................................................... 16 The Growth Index ............................................................................................................. 16 Calculating Expected Growth............................................................................................17 Awarding AZ LEARNS Points..........................................................................................18 Schools Without MAP.......................................................................................................18 6. GRADUATION AND DROPOUT RATES.............................................................. 20 Graduation Rates............................................................................................................... 20 Dropout Rates ................................................................................................................... 20 7. EVALUATING THE TOTAL SCALE SCORE VALUE TO DETERMINE A SCHOOL CLASSIFICATION ...................................................................................... 22 Arizona's Accountability System: Volume I Arizona Department of Education i 8. APPLICATION OF THRESHOLD CRITERIA FOR EXCELLING AND HIGHLY PERFORMING SCHOOLS ......................................................................... 23 9. AZ LEARNS ACHIEVEMENT PROFILE APPEALS PROCESS....................... 27 Procedure and Timeline .................................................................................................... 27 Special Appeals for 2006...................................................................................................30 10. AZ LEARNS ACHIEVEMENT PROFILES FOR ALTERNATIVE SCHOOLS ....................................................................................................................... 32 Definition of an Alternative School.................................................................................. 32 General Process to Produce Achievement Profiles for Alternative Schools .................... 32 Calculation of an Achievement Profile for an Alternative School ................................... 33 Classification Scales for Alternative Schools ................................................................... 35 11. AZ LEARNS ACHIEVEMENT PROFILES FOR K-2 SCHOOLS.................... 36 Appendix I: Substantive Appeal Committee Evaluation Rubric for AZ LEARNS......... 38 Appendix II: Substantive Appeal Rubric for AZ LEARNS ............................................ 38 Arizona's Accountability System: Volume I Arizona Department of Education ii List of Tables and Figures Table 4.1. Status Points Awarded.....................................................................................10 Table 4.2. Number of Students.........................................................................................12 Table 4.3. Growth Points..................................................................................................13 Table 4.4. AIMS Point Distributions by Status and Growth Groups................................14 Table 4.5. Rules for Determining UB95 for Small n and p..............................................15 Table 5.1. Parameters for Calculating Expected Growth for Math..................................17 Table 5.2. Parameters for Calculating Expected Growth for Reading.............................17 Table 5.3. AZ LEARNS Scale for Schools Not Receiving MAP Points ........................18 Table 7.1. Elementary School Classification Cut Points ..................................................22 Table 7.2. High School Classification Cut Points.............................................................22 Table 8.1. Parameters for Calculating Z-Scores ...............................................................25 Table 8.2. Number of Students Exceeding the Standard - Third Grade...........................25 Table 8.3. Average Z-Scores............................................................................................26 Figure 10.1 Method for Evaluating Alternative Schools (Elementary Schools) ..............34 Figure 10.2 Method for Evaluating Alternative Schools (High Schools) ........................35 Table 10.1 Alternative School Classification Cut Points .................................................35 Table 11.1 AZ LEARNS Scale for K-2 Schools ..............................................................36 Arizona's Accountability System: Volume I Arizona Department of Education iii 1. Introduction In November of 2001, Arizona voters approved Proposition 301 which, among other things, provided funds to the Arizona Department of Education (ADE) to develop “a system to measure school performance based on student achievement, including student performance on the AIMS test.” The legislative requirements for the accountability system are stated in section 15-241 (ARS § 15-241) of the Arizona Revised Statutes. The accountability system created to satisfy the statute is referred to as the Arizona LEARNS. The school evaluation given by ADE to each school is referred to as the school’s achievement profile. This manual describes the method used to generate the 2006 achievement profiles. It provides formulas, parameters, and business rules that make up the profile calculation. It also describes the AZ LEARNS process for 2006. Its intent is to document and explain the methods used and justify the guidelines adopted. This manual will give the ambitious user all the information required to calculate the achievement profile of his or her school. What’s New for the 2006 AZ LEARNS Achievement Profiles When developing modifications to the method of calculating school achievement profiles for 2006, the ADE, as in previous years, consulted with its Advisory Group—a diverse group ranging from measurement experts and curriculum coordinators to Principals. These experts volunteered their time to undertake the difficult task of advising the department on the complex issue of state-level school accountability. The Arizona State Board of Education reviewed, commented upon, and approved changes in the AZ LEARNS achievement profile methodology. The ADE provided the Board information packets that outlined the decisions regarding the formula that needed to be made. This documentation can be found online via the AZ LEARNS web site, http://www.ade.az.gov/azlearns. For 2006, the following changes were made to the AZ LEARNS system. • A new method for calculating the Measure of Academic Progress based on AIMS scores. This was approved by the Board on September 18, 2006. • New performance thresholds for K-2 schools. This was approved by the Board on September 18, 2006. Arizona's Accountability System: Volume I Arizona Department of Education 4 2. Overview of the AZ LEARNS Evaluation System This section provides an overview of how AZ LEARNS achievement profiles are determined. More detailed discussions of the methodology used to determine the profiles, including descriptions of equations, algorithms, and data used are given in subsequent chapters. Arizona law (ARS § 15-241) mandates that the Arizona Department of Education shall compile an annual achievement profile for each public school. It specifies that the profiles of schools serving grades K-8 shall be based on: • Arizona Measure of Academic Progress (MAP). • Percent of students who pass AIMS. The law specifies that the profiles of high schools shall be based on: • Drop out rate • Graduation rate • Percent of students who pass AIMS. A school that serves both grades K-8 and high school receives two separate achievement profiles. The law also calls for the ADE to use a research based methodology that shall: • Include performance of pupils at all achievement levels • Account for pupil mobility • Account for the distribution of pupil achievement • Include longitudinal indicators of academic performance. A research based methodology is defined as “the systematic and objective application of statistical and quantitative research principles to determine a standard measurement of acceptable academic progress for each school”. The law also calls for a system of parallel achievement profiles for accommodation schools/alternative schools as defined by the Board of Education. Arizona's Accountability System: Volume I Arizona Department of Education 5 General Process to Calculate an Achievement Profile The achievement profile for a school serving grades 3-8 consists of the following performance measures: 1. A status measure based on the performance of students on all three sections of the AIMS (reading, writing, and mathematics) in the current year. 2. A measure of improvement in aggregate student performance on the AIMS compared to the baseline year. 3. A measure of growth in individual student performance. Academic Progress (MAP). This is the Measure of 4. A measure of whether the school made Adequate Yearly Progress (AYP) as defined by the No Child Left Behind Act of 2001. In order to comply with the federal requirement that the state have an integrated accountability system, a school’s AYP determination is factored into the calculation of its achievement profile. 5. In order to create the incentive for schools to improve the achievement of average and above-average students, a school cannot earn the highly performing or excelling labels unless the percentage of its students exceeding the standard on AIMS met specific thresholds. 6. The achievement profile for a high school is made up of all of the above components except for measure 3. In addition, the following performance measures are used for high schools: • Dropout rate. • Graduation rate. Schools are awarded scale score points based on their performance on measures 1-4, 6, and 7. Scale score points are then summed up for each school and compared to a scale that relates scale score points to the five profile labels: excelling, highly performing, performing plus, performing, and underperforming. Performance measure five is then examined to determine if the school has earned the highest labels of highly performing or excelling. Arizona's Accountability System: Volume I Arizona Department of Education 6 3. Timeline Districts and charter holders are solely responsible for submitting the data necessary for calculating achievement profiles for their schools and for ensuring its accuracy. Because of the stakes involved and the volume and scope of the data used, the ADE considers it prudent to allow districts and charter holders to review their data before preliminary AZ LEARNS achievement profiles calculations are carried out. From June 14, 2006 through October 6, 2006 schools and districts were given an opportunity to review and correct their testing data through the common logon on the ADE web site. The primary purpose of the application was to allow districts and charter holders to correct the information for individual students. A link was provided through the common logon that allowed schools to download student-level testing data in order to make any necessary corrections. From July 21, 2006 through August 25, 2006 schools and districts were given the opportunity to review and correct the data used for calculating the five-year graduation rate used in the AZ LEARNS profile. The 2006 profiles were the first in which all program membership and demographic information relevant to AZ LEARNS profiles was taken by matching test records to the state’s SAIS database of student records. Consequently the only information that schools needed to correct in the ADE AIMS testing file were students’ SAIS ID’s (need for matching) and if the student received alternate accommodations (only collected on the testing document). If program membership or other information was incorrect, schools and districts were required to correct it in the SAIS database. Schools were not allowed to correct the testing modifications after the close of the AYP appeal process on August 15. IMPORTANT NOTE: The criteria used to select AIMS scores for AZ LEARNS profiles differ from the criteria used to select scores for adequate yearly progress under NCLB. Indeed, the criteria differ among the separate components of the AYP evaluation. The criteria also differ from the scores provided to schools by the testing contractor, the scores publicly reported by ADE, and the scores available through the ADE AIMS wizard located at www.ade.az.gov/profile/publicview. Timeline The timeline for AZ LEARNS Achievement Profile process was: • April 5, 2006. Opening of application process for alternative schools. • June 14, 2006. Opening of data verification process. • July 21, 2006. Start of statistical review and appeals of graduation rate data. • August 25, 2006. Closing of application process for alternative schools. Arizona's Accountability System: Volume I Arizona Department of Education 7 • August 25, 2006. • September 22, 2005. Preliminary release of AZ LEARNS achievement profiles for all schools. Opening of window for appeals submissions. • October 6, 2006. Closing of appeals window for all schools. • October 25, 2005. Public release of AZ LEARNS achievement profiles for all schools. Closing of statistical review and appeals of graduation rate data. Arizona's Accountability System: Volume I Arizona Department of Education 8 4. Calculation of AIMS Status and Growth Points In the AZ LEARNS profile, schools are awarded scale score points based on student performance on the AIMS. Points are determined by performance in the current year, and improvement in student performance from the baseline year. Awarding Status Points The following method is used to calculate Status points for every subject and grade offered by a school. First, calculate the percent passing in the current year. The percentage is compared to a scale which in turn gives the status points achieved for the subject and grade. Percent Pa ssing in the current year = # Students Passing AIMS in the current year # Students tested in the current year The points are rounded to the nearest hundredth e.g. .675 = .68; .672 = .67. Data Used A student’s score is excluded from the calculation if any of the following criteria are met: 1. Student received no score on the test. 2. Student was not English proficient. A student was considered not proficient if it was indicated that she was a participant in the English language learner program for three years or less. In a change from previous years, counting for the number of years in the ELL program started in kindergarten rather than first grade. However to remain consistent with previous years, in 2006 third graders were excluded if they had been participants in the ELL program for four years or less. 3. The student received an alternate accommodation on the test. 4. The student was not enrolled in the school for the full academic year. A student was considered enrolled for the full academic year if she enrolled in a school during the first 10 school days of the school year and remained enrolled up through the testing date. This information was obtained from SAIS. Students not matched to SAIS were assumed to be continuously enrolled. The calculation for high schools includes all students in grades 10 through 12 who have taken the AIMS for both the fall and spring tests. If a high school took the test twice in a school year, the highest of the two scores was used. Arizona's Accountability System: Volume I Arizona Department of Education 9 Grade 3 3 3 4 4 4 5 5 5 6 6 6 7 7 7 8 8 8 HS HS HS Subject Math Read Writ Math Read Writ Math Read Writ Math Read Writ Math Read Writ Math Read Writ Math Read Writ Status Points 1 <51% <46 <56 <46 <40 <42 <42 <42 <47 <34 <38 <49 <35 <41 <64 <27 <35 <58 <5 <16 <18 Table 4.1 Status Points Awarded Status Status Status Points 2 Points 3 Points 4 51-64% 65-78% 79-88% 46-59 60-73 74-84 56-67 68-78 79-87 46-60 61-75 76-86 40-53 54-69 70-81 42-53 54-65 66-77 42-56 57-72 73-85 42-56 57-73 73-84 47-57 58-69 70-80 34-49 50-66 67-81 38-52 53-69 70-82 49-60 61-73 74-83 35-49 50-66 67-81 41-54 55-69 70-82 64-73 74-82 83-89 27-41 42-59 60-76 35-48 49-65 66-79 58-70 71-82 83-90 5-13 14-29 30-51 16-28 29-46 47-65 18-30 31-48 49-67 Status Points 5 89-94% 85-92 88-93 87-93 82-90 78-85 86-92 85-92 81-87 82-90 83-91 84-90 82-90 83-90 90-94 77-87 80-88 91-95 52-70 66-79 68-80 Status Points 6 >=95% >=93 >=94 >=94 >=91 >=86 >=93 >=93 >=88 >=91 >=92 >=91 >=91 >=91 >=95 >=88 >=89 >=96 >=71 >=80 >=81 A school is awarded status points for each subject/grade it offers. Example. In the current year, 66 percent of the students in Gila Monster Elementary passed the math portion of the third grade AIMS. This value places the subject/grade in Status grouping three. Gila Monster Elementary has earned three status points for this particular subject/grade. Awarding Growth Points Determining a school’s growth points for each subject/grade combination is based on student movement out of the falls far below (FFB) category and student movement into the meet/exceeds (M/E) category. The growth points earned are determined by the difference between average percentages in 2005 and 2006 and the baseline percentage (2004). A school has made positive change if the two-year average percent of students that fall into the M/E category is higher than the baseline percentage or if the two-year average percent FFB is lower than the baseline. A school has made negative change if the two-year average percentage of students in the M/E category is lower than the baseline or if the percentage of students in the FFB category is higher than the baseline percentage. Schools are expected to increase the percentage of students that meet the standards over time, and decrease the percentage that fall far below the standards over time. Arizona's Accountability System: Volume I Arizona Department of Education 10 Equations Used to Calculate Growth Points Growth points for a subject/grade are calculated in the following seven steps. Equation 1 Baseline FFB = # Students FFB in baseline year # Tested in baseline year Baseline M/E = # Students M/E in baseline year # Tested in baseline year Equation 2 Equation 3 2 - Yr. Avg. FFB = # Students FFB in most recent two years # Tested in most recent two years 2 - Yr. Avg. M/E = # Students M/E in most recent two years # Tested in most recent two years Equation 4 Equation 5 Change FFB = 2 - Yr. Avg. FFB - Baseline FFB Equation 6 Change M/E = 2 - Yr. Avg. M/E - Baseline M/E Equation 7 Growth Points = Change M/E - Change FFB All values are rounded to the nearest hundredth e.g. .675 = .68; .672 = .67. Data Used The rules for using student scores for the growth measure are the same as for the status measure. If a high school student took the test multiple times over each of the three years, the highest score for each year was used. Arizona's Accountability System: Volume I Arizona Department of Education 11 Example. The following example demonstrates how growth points are calculated. Table 4.2 shows three years of AIMS scores for a single subject and grade for a hypothetical school. Year 2004 2005 2006 Table 4.2. Number of Students FFB A M 15 35 30 10 35 30 10 25 35 E 30 35 40 Total 110 110 110 The following equations show the steps used to calculate the growth points given the test scores in the above table. Equation 1 Baseline FFB = 15 = 0.1364 110 Baseline M/E = 60 = 0.5455 110 Equation 2 Equation 3 2 - Yr. Avg. FFB = 10 + 10 = 0.0909 110 + 110 2 - Yr. Avg. M/E = 65 + 75 = 0.6364 110 + 110 Equation 4 Equation 5 Change FFB = 0.0909 - 0.1364 = - 0.0455 Equation 6 Change M/E = 0.6364 - 0.5455 = 0.0909 Equation 7 Growth Points = 0.0909 - (-0.0455) = 0.1364 = .14 Arizona's Accountability System: Volume I Arizona Department of Education 12 The growth point thresholds are given in table 4.3. Please refer to the technical manual for the 2003 profiles (release date February 17, 2004) for a description of how the cut points were established. Grade 3 3 3 5 5 5 8 8 8 HS HS HS Subject Math Read Writ Math Read Writ Math Read Writ Math Read Writ Growth Points 1 <-15% <-20 <-13 <-8 <-10 <0 <18 <-11 <17 <6 <-16 <-24 Table 4.3. Growth Points Growth Growth Growth Points 2 Points 3 Points 4 -15 - -7 -6 - 1% 2 - 9% -20 - -14 -13 - -7 -6 – 0 -13 - -5 -4 – 3 4 -12 -8 – 0 1–8 9 -17 -10 - -3 -2 – 5 6 – 13 0–9 10 – 20 21 – 31 18 – 26 27 – 35 36 – 44 -11 - -3 -2 – 7 8 – 16 17 – 27 28 – 38 39 – 49 6 – 13 14 – 21 22 – 29 -16 - -7 -6 – 3 4 – 12 -24 - -14 -13 - -4 -3 - 7 Growth Points 5 10 - 17% 1–8 13 – 21 18 – 25 14 – 22 32 – 41 45 – 53 17 – 25 50 – 60 30 – 37 13 – 22 8 - 18 Growth Points 6 >=18% >=9 >=22 >=26 >=23 >=42 >=54 >=26 >=61 >=38 >=23 >=19 Calculation of Subject/Grade Scale Points from AIMS The total scale score points derived from AIMS performance are calculated by adding the baseline points awarded to the growth points awarded. A 70 percent weight is given to the school’s strongest point value (baseline group or growth point group) and a 30 percent weight to the other point value. Table 5.3 shows the scale points earned per subject/grade for all combinations of baseline and growth group scale points. Example. In third grade mathematics, Gila Monster Elementary has earned three status points and four growth group scale points. Because it has earned more scale points for its growth group, the growth group points are given a 70 percent weight and the status group scale points are given a 30 percent weight. Thus, the total scale points earned for third grade math are (.7 X 4) + (.3 X 3) = 3.7. Example. In mathematics, Desert Mountain Vista High School has earned five status group scale points and two growth group scale points. Because it has earned more scale points for its status group, the status scale points are given a 70 percent weight and the baseline group scale points are given a 30 percent weight. Thus, the total scale points earned for third grade math are (.3 X 2) + (.7 X 5) = 4.1. If a subject/grade group earned six status points, 100 percent of the weight is given to the status points. If the subject/grade group has less than 16 test scores in the baseline year or any other year used in the calculations the group is only awarded status points. Arizona's Accountability System: Volume I Arizona Department of Education 13 Example. Gila Monster Elementary is missing data for fifth grade writing for both 2003 and 2004. Its status points for that subject/grade combination put it in status group five. 100% weight is given to the Status group in this case. Thus it receives 5 AIMS scale points for fifth grade writing. The points derived from AIMS for all subject/grade combinations for a school are averaged across grades by subject and added to the scale score values for other performance measures. Table 4.4. AIMS Point Distributions by Status and Growth Groups Growth Points Status Points 1 2 3 4 5 1 1.0 1.7 2.4 3.1 3.8 2 1.7 2.0 2.7 3.4 4.1 3 2.4 2.7 3.0 3.7 4.4 4 3.1 3.4 3.7 4.0 4.7 5 3.8 4.1 4.4 4.7 5.0 6 6 6 6 6 6 6 4.5 4.8 5.1 5.4 5.7 6 Special Cases: New Schools, and Missing Data If a school is missing AIMS test data for a subject/grade combination it offers, it receives zero scale points for that subject/grade for the achievement profile calculation for the test year in which the data are missing. Because of the high-stakes consequences of being labeled an underperforming school, and because of the uncertainty of measurement involved with small sample sizes, it is prudent to give a school with a small number of students a “second look” if it faces the possibility of receiving an underperforming profile. If the preliminary label of a school is underperforming, then the AIMS scale score points for that school are recalculated for each subject/grade group that has less than 16 students. For each subject/grade combination with less than 16 students, the upper bound of the 95-percent confidence interval is used to calculate to which baseline group the school belongs. If the recalculated points move the school into a higher classification, the school receives a performing profile. Let p=the percent of students in a group passing the AIMS and n=the number of students in the group. Then the equation for the upper bound of the 95 percent confidence interval (UB95) is: UB95 = p + 1.96 p(1 − p) n . As can be seen from the equation, the confidence interval depends upon the percent of students who passed the test, and the number of students tested. Thus, the confidence interval will differ among grades, subjects, and schools. Arizona's Accountability System: Volume I Arizona Department of Education 14 The equation is an approximation of the confidence interval for a binomially distributed variable. It uses the standard normal distribution and is sufficiently accurate if the group size and percentage of students passing are large enough. For small values of n and small p, a more accurate estimate of the confidence interval is made using statistical tables that provide confidence intervals for a binomially distributed variable. 1 The tables are applied using the rules given in table 4.4. Table 4.5. Rules for Determining UB95 for Small n and p. If n>=0 and n < 8, and p>= 0 and p <.04, then UB95=.42; p>=.04 and p <.10, then UB95=.50; p>=.10 and p <.20, then UB95=.60; If n>=8 and n < 10, and p>= 0 and p <.04, then UB95=.37; p>=.04 and p <.10, then UB95=.45; p>=.10 and p <.20, then UB95=.55; If n>=10 and n < 12, and p>= 0 and p <.04, then UB95=.33; p>=.04 and p <.10, then UB95=.41; p>=.10 and p <.20, then UB95=.52; If n>=12 and n < 16, and p>= 0 and p <.04, then UB95=.27; p>=.04 and p <.10, then UB95=.35; p>=.10 and p <.20, then UB95=.47; If n>=16 and n < 20, and p>= 0 and p <.04, then UB95=.24; p>=.04 and p <.10, then UB95=.32; p>=.10 and p <.20, then UB95=.44; If n>=20 and n < 24, and p>= 0 and p <.04, then UB95=.21; p>=.04 and p <.10, then UB95=.29; p>=.10 and p <.20, then UB95=.42; If n>=24 and n < 30, and p>= 0 and p <.04, then UB95=.18; p>=.04 and p <.10, then UB95=.27; p>=.10 and p <.20, then UB95=.38; If n>=30 and n < 40, and p>= 0 and p <.04, then UB95=.15; p>=.04 and p <.10, then UB95=.23; p>=.10 and p <.20, then UB95=.36; 1 Mansfield, Edwin. 1991. Statistics for Business and Economics, 4th Edition. New York: W.W. Norton and Company. 280-284. Arizona's Accountability System: Volume I Arizona Department of Education 15 5. The Measure of Academic Progress The 2006 AZ LEARNS profiles were the first to incorporate a new Measure of Academic Progress (MAP) based on the AIMS test. Although the Arizona Department of Education has been presenting MAP analyses for seven years, previous years’ MAP scores have out of necessity been calculated using the state’s norm-referenced test. Starting with the 2006 profiles, MAP will directly measure student progress in learning state standards as measured by performance on the AIMS. The new MAP uses a conventional value-added approach to measuring student growth an accepted method to evaluate the impact of programs and schools. In addition to a new test and a new method of measuring growth, the new MAP is also the first time student records were matched across years using a unique student identifier (the SAIS ID). The proposed new MAP has three straightforward steps: 1. An individual expectation of one year’s growth (OYG) is calculated for each student. 2. The expectation of OYG is subtracted from the actual growth achieved by the student to determine a Growth Index. 3. The average growth index for school is calculated by averaging growth indices for individual students across all grades and subjects. Schools are awarded AZ LEARNS points based on this average. Actual growth, expected growth, and the growth index are expressed in AIMS scale score points. The Growth Index In order to control for a ceiling effect and for student mobility, the MAP analysis is done using a standard value-added model. The value-added model is used to calculate an estimate of expected for each student for each subject. The expected growth is then subtracted from actual growth to determine a student’s Growth Index. Example. Student A scored 478 on the 6th grade math test in 2005. Her expected growth for 7th grade in 2006 is 38. In 2006 she scores a 528 on the 7th grade math test. Her actual growth is 528 – 478 = 50. Her growth index is 50 – 38 = 12. Example. Student B scored 490 on the 4th grade reading test in 2005. His expected growth for 5th grade in 2006 is 12. In 2006 he scores a 500 on the 5th grade reading test. His actual growth is 500 – 490 = 10. His growth index is 10 – 12 = -2. The results of the MAP analysis for students and schools are reported in terms of the growth index. Growth indices for individual students can be averaged across schools, grades, and subjects to measure performance. Arizona's Accountability System: Volume I Arizona Department of Education 16 Calculating Expected Growth The expected growth for an individual student is calculated using the following formula: Expected Growth = A + B X (2005 scale score) + C X (FAY); where FAY = 1 if the student has been enrolled for the full 2006 academic year and FAY = 0 if not. A student is considered to have been enrolled the full 2006 academic year if she has enrolled within the first two weeks of the school year and remained enrolled up to the time of the test. The parameters A, B, and C are given in the tables below: Table 5.1. Parameters for Calculating Expected Growth for Math Student’s Grade Current A B C Year 4 98.9308 -0.1514 6.027 5 107.7715 -0.174 5.7754 6 75.6373 -0.1268 6.999 7 121.1295 -0.1896 7.1203 8 54.1785 -0.0823 7.0308 Table 5.2. Parameters for Calculating Expected Growth for Reading Student’s Grade Current Year A B C 4 120.9638 -0.2269 3.7108 5 153.1619 -0.2879 4.106 6 105.8317 -0.2016 4.343 7 88.3119 -0.1492 5.1193 8 89.8856 -0.1688 7.0786 Example. Student A scored 478 on the 6th grade math test in 2005. She has been enrolled in her current school all year. Her expected growth for 7th grade in 2006 is 120.1295 + (-0.1896) X 478 + 7.1203 = 36.62 = 37. Example. Student B scored 490 on the 4th grade reading test in 2005. He has not been enrolled in his current school for the full year. His expected growth for 5th grade in 2006 is 153.1619 + (0.2879) X 490 = 12.09 = 12. Please see the 2006 MAP report available on the ADE web site for a description of how the parameters were developed. Arizona's Accountability System: Volume I Arizona Department of Education 17 Awarding AZ LEARNS Points The Growth Index for a school is calculated by averaging the student-level growth indices across all grades and subjects. The number of AZ LEARNS points earned by a school is calculated with the following formula: AZ LEARNS points = 5.2 + .22 X (School’s growth index.) The scale is bounded at the bottom by 2. So if the result of the above formula is less than 2, a school would earn 2 points. No maximum is placed on the number of points a school may earn. This is to award schools that make outstanding growth with students. The scale was developed so that MAP would receive the same weight in the AZ LEARNS profile that it has in previous years. Example. School A’s growth index is -1.7. The number of AZ LEARNS points awarded to the school is 5.2 + .22 X (-1.7) = 4.8. Example. School B’s growth index is 3.2. The number of AZ LEARNS points awarded to the school is 5.2 + .22 X (3.2) = 5.9. Example. School C’s growth index is -20.0. Since 5.2 + .22 X (-20.0) = 0.8 is less than 2. The number of AZ LEARNS points awarded to the school is 2. Schools without MAP Schools did not receive points from MAP if they did not have a grade evaluated for MAP or if they had less than 16 scores in the MAP analysis. An alternate scale was developed for schools that did not receive MAP points. The scale was constructed so that the MAP and nonMAP profiles would be as congruent as possible. To do this, the points earned by schools in the 2005 AZ LEARNS profiles were compared with and without MAP. A scale was then constructed so that as many schools as possible would earn the same profile without MAP points as they would if MAP points were included. The table below shows the current scale that includes MAP, the non-MAP scale, the distribution of schools in 2005, and the distribution of those same schools without MAP points and using the proposed scale. Table 5.3. AZ LEARNS Scale for Schools Not Receiving MAP Points Distribution of Schools Scale with Non-MAP 2005 with MAP Using Non-MAP Profile MAP Scale Included Scale Underperforming <13 <8 6% 8% Performing 13-15.9 8-12.9 34% 55% Performing Plus 16-27 13-19 31% 12% Highly Performing 16-18.9 13-14.9 15% 14% Excelling 19-27 15-19 13% 12% Example. School A is a K-3 and hence does not have a MAP score. It must earn 8 points to be a performing school. Arizona's Accountability System: Volume I Arizona Department of Education 18 Example. School B is a K-6 and hence does have a MAP score. It must earn 13 points to be a performing school. Arizona's Accountability System: Volume I Arizona Department of Education 19 6. Graduation and Dropout Rates In their AZ LEARNS profile, high schools may earn one point for their graduation rate and one point for their dropout rate. Graduation Rate The Graduation Rate is a five-year, longitudinal measure of how many students graduate from high school. The formula used to calculate the graduation rate is: Graduation Rate = Number in cohort that graduated in within five years Original cohort + transfers in - transfers out The graduation rate is rounded to two digits, e.g.: .705=.71; .704=.70. A school can earn one AZ LEARNS scale point for its graduation rate in one of three ways: 1. If its three-year average graduation rate is 89.5 percent or greater. 2. If its three-year average graduation rate is less than 89.5 percent and its baseline rate is greater than or equal to 73.5 percent, then a school will earn one point if the average of its graduation rates over the most recent three years is 1 percentage point greater than its baseline rate. 3. If its three-year average graduation rate is less than 89.5 percent and its baseline rate is less than 73.5 percent, then a school will earn one point if the average of its graduation rates over the most recent three years is 1.5 percentage point greater than its baseline rate. The three-year average is calculated by taking the total number of combined five-year graduates for the most recent three graduating classes and dividing by the total number of students in the combined classes. Graduation Rate = 2003 + 2004 + 2005 five - year graduates Combined number of students in three cohorts The baseline graduation rate is the graduation rate for the cohort class of 2000 or its first graduation class, whichever is earlier. Dropout Rate The dropout rate is an annual measure of how many students drop out of a school during a twelve-month reporting period. Arizona's Accountability System: Volume I Arizona Department of Education 20 Dropout Rate = Number of students who dropped out Number of students who were enrolled during the school year Like the graduation rate, a school can earn one AZ LEARNS scale point for its dropout rate in one of three ways: 1. If the three-year average dropout rate is 6 percent or less, the target is automatically met. 2. If its three-year average dropout rate is greater than 6 percent and its baseline rate is greater than or equal to 9.4 percent, then a school will earn one point if the average of its dropout rates over the most recent three years is 1.5 percentage point less than its baseline rate. 3. If its three-year average graduation rate is less than 6 percent and its baseline rate is less than 9.4 percent, then a school will earn one point if the average of its dropout rates over the most recent three years is 1 percentage point less than its baseline rate. The three-year average is calculated by taking the total number of dropouts for the most recent three years and dividing by the total number of students enrolled. Three - year dropout rate = Total number of dropouts for 2004 - 2006 Total number of students enrolled 2004 - 2006 The baseline dropout rate is the dropout rate for the 2000 school year or its first year of operation, whichever is earlier. . Arizona's Accountability System: Volume I Arizona Department of Education 21 7. Evaluating the Total Scale Score Value to Determine a School Classification The tables below show the total number of scale score points that schools must earn in order to receive a given classification. A school may receive up to 18 scale points from status and growth; unlimited scale points from MAP if it is an elementary or middle school; and up to two graduation/drop out scale points if it is a high school. A school receives one scale point if it made adequate yearly progress (AYP). For each school, the applicable scale score cut points for classification labels are calculated by averaging the AIMS scale points received for each subjects; adding the AYP point received; adding points from MAP if it is an elementary school; and adding points from the graduation/dropout rate if it is a high school. The classification cut points for elementary schools are given in the first two columns of table 7.1 and those for high schools are given in the first column of table 7.2. In order for schools to be classified as highly performing or excelling, there is an additional Z-score criteria that has to be met, which is explained in the next chapter. Table 7.1. Elementary School Classification Cut points Performance Thresholds Profile Underperforming Performing Performing Plus Highly Performing Excelling Scale with MAP <13 Non-MAP Scale <8 z-score 13-15.9 16-27 16-18.9 19-27 8-12.9 13-19 13-14.9 15-19 NA NA 0.45-0.99 1.00 or more Table 7.2 High School Classification Cut Points Performance Thresholds Profile Underperforming Performing Performing Plus Highly Performing Excelling Scale Points <9.6 9.6-14.9 15 or more 15-16.9 17 or more Arizona's Accountability System: Volume I z-score NA NA < 0.45 0.45 or more Arizona Department of Education 22 8. Application of Threshold Criteria for Excelling and Highly Performing Schools To ensure continued focus on improving the academic achievement of all students, including those students currently demonstrating proficiency, threshold criteria are applied to determine excelling and highly performing schools. To be deemed a highly performing or excelling school, a school must meet certain levels in the percentage of its students exceeding the standard on the AIMS. Schools must not only receive a total scale value that places them into either excelling or highly performing, but must also meet the requisite percentage of students in the exceeds-the-standard category on AIMS to be designated as either excelling or highly performing schools. The application of threshold criteria for excelling and highly performing schools results in the following scenarios. 1. A school that receives a total scale value that places it in the excelling classification and meets the requisite percentage of students in the exceeds category on AIMS necessary for an excelling classification will be designated an excelling school. 2. A school that receives a total scale value that places it in the excelling classification and did not meet the requisite percentage of students in the exceeds category on AIMS necessary for a excelling classification, but did met the requisite percentage of students in the exceeds category on AIMS necessary for a highly performing classification will be designated as a highly performing school. 3. A school that receives a total scale value that places it in the excelling classification and did not meet either the requisite percentage of students in the Exceeds category on AIMS necessary for the excelling classification or the highly performing classification will be designated as a performing school. Such schools are recognized with the non-statutory designation of performing plus. 4. A school that receives a total scale value that places it in the highly performing classification and meets the requisite percentage of students in the exceeds category on AIMS necessary for an excelling classification will be designated as a highly performing school. 5. A school that receives a total scale value that places it in the highly performing classification and meets the requisite percentage of students in the exceeds category on AIMS necessary for a highly performing classification will be designated as a highly performing school. 6. A school that receives a total scale value that places it in the highly performing classification and did not meet either the requisite percentage of students in the exceeds category on AIMS necessary for an excelling classification or highly performing Arizona's Accountability System: Volume I Arizona Department of Education 23 classification will be designated as a performing school. Such schools are recognized with the non-statutory designation of performing plus. 7. A school that receives a total scale value that places it in the performing classification will be designated as a performing school, regardless if the school meets the requisite percentage of students in the exceeds category on AIMS necessary for an excelling classification or a highly performing classification. 8. A school that receives a total scale value that places it in the underperforming classification will be designated as an underperforming school, regardless if the school meets the requisite percentage of students in the exceeds category on AIMS necessary for an excelling classification or a highly performing classification. The determination of whether a school met the goals for percent exceeding is based on a z-score calculated in the following manner: 1. For each school, the percent exceeding is calculated by grade across all three subjects for both 2004-06 and 2006. The introduction of a one-year estimation of percent exceeding is to give credit to schools that have shown improvement in the percent of students exceeding the standard. 2. The percent-exceeding is converted into a z-score by subtracting the statewide average for that grade for percent exceeding and dividing by the statewide standard deviation. (These parameters are given in table 8.1) Note: The same statewide parameters are used to calculate both the one-year and threeyear z-score. This is because the purpose of the z-score is to create standard scores comparable across grades, not to create scores comparing a school’s performance to the state norm. The z-scores are fixed parameters so if a school shows improvement over time its z-score will increase regardless of the movement of the state average over the same period. 3. The one- and three-year z-scores across all grades for a school: each school will have two z-scores: one based on its three-year average for percent exceeding, another based on the percent exceeding for the current year. 4. The higher of the one- and three-year averages are taken and compared to the performance thresholds. To be a highly performing school, the average z-score for a school must be greater than or equal to 0.45. To be an excelling school, the average zscore for a school must be greater than or equal to 1.00. Arizona's Accountability System: Volume I Arizona Department of Education 24 Table 8.1. Parameters for Calculating Z-Scores Grade Average Standard Deviation 3 0.124576 0.093475 4 0.111652 0.087267 5 0.092426 0.080033 6 0.095892 0.085301 7 0.06592 0.0666 8 0.052172 0.056997 10 0.058823 0.082827 Example. The following table shows the third grade AIMS scores for Gila Monster Elementary over the past three years. Subject Year 2003 2004 2005 Total Table 8.2. Number of Students Exceeding the Standard—Third Grade Reading Mathematics Writing # Exceeding #Tested # Exceeding #Tested # Exceeding #Tested 25 100 24 100 23 100 24 105 23 105 22 105 26 99 25 99 24 99 75 304 72 304 69 304 The three-year average for percent of students exceeding the standard is: Percent Exceeding = 75 + 72 + 69 = 23.6%. 304 + 304 + 304 The one-year average for percent of students exceeding the standard is: Percent Exceeding = 25 + 25 + 24 99 + 99 + 99 = 25.3%. The three-year z-score for third grade is: z - score = .236 - .125 .093 = 1.19. The one-year z-score for third grade is: z - score = .253 - .125 .093 = 1.38. Gila Monster serves grades K-6. The one- and three-year average z-scores for the entire school are: Arizona's Accountability System: Volume I Arizona Department of Education 25 Grade K-2 3 4 5 6 Average Table 8.3. Average Z-scores One-year NA 1.38 1.02 .89 .95 1.06 Three-year NA 1.19 .86 .72 .80 .89 The one-year average is greater than 1.00, so Gila Monster would earn an excelling—if it has earned sufficient points. Arizona's Accountability System: Volume I Arizona Department of Education 26 9. AZ LEARNS Achievement Profile Appeals Process Procedure and Timeline In accordance with A.R.S. §15-241, school administrators are allowed the opportunity to appeal an achievement profile classification on behalf of the schools for which they are responsible. Step 1: Data Correction. The first step in completing the AZ LEARNS Appeals Process required districts and schools to review their data in order to confirm its accuracy. Data correction took place June 14 through October 6, 2006. It is important to note that districts and charter holders were solely responsible for verifying information for their districts and schools. If a district or charter holder did not verify the information for its district and schools through the correction process, the ADE assumed the schools on file and the data available were correct as listed. Step 2: Appeal Application. Administrators choosing to appeal an AZ LEARNS profile completed the AZ LEARNS Appeal Application, which was accessible via the common logon during the specified appeal window. Appeals were only accepted through the website application. Appeals sent to ADE via email, fax, or mail/delivery were not accepted. Schools were able to appeal AZ LEARNS profiles in two categories: data (statistical) and non-data (substantive) reasons – districts and schools were not limited to one category and were able to appeal in both if necessary. Statistical appeals are appeals of the accuracy of the data used in the AZ LEARNS profile. Given the extensive time allowed to view and correct the data, it is expected that any errors should be corrected by the time preliminary profiles are released. Statistical appeals were not granted unless the underlying data was corrected. Substantive appeals are arguments by districts and schools that that circumstances outside of the district’s/school’s control negatively affected school performance. Administrators that chose to appeal an AZ LEARNS profile must have clearly articulated the issues they believe merited an appeal. Administrators must have submitted evidence that the issues they believe merited an appeal directly resulted in a significant decrease in student academic achievement as demonstrated on the AIMS. The evidence must have been submitted to ADE at the time the appeal was submitted. Failure to provide this evidence resulted in the appeal not being granted. Evidence submitted after the appeal deadline closed was not considered. Once appeals were submitted through the Common Logon, the school/district/charter holder received an email verifying that the appeal was received. NOTE: In order to protect student privacy and the integrity of the appeals process, schools were asked to refer to a specific student only by that student’s SAIS ID. The SAIS ID was required so that ADE staff could verify the contentions in the appeal. Arizona's Accountability System: Volume I Arizona Department of Education 27 The ADE, if necessary, requested that a district or school administrator provide additional information/evidence to assist in the appeals process. Only those requests for additional information that were provided during the specified timeframe allotted were included in the appeals process. Requests submitted after the specified timeframe were excluded from the appeals process. Unsolicited additional information submitted after the appeal deadline was not accepted. Step 3: Appeal Resolution. After all appeals were submitted and the appeals window closed, the ADE began to process the appeals. Appeals were addressed categorically, not necessarily in the order received, so the fact that a district or school submitted its appeal during the first day of the appeal window did not mean it necessarily received a decision first during the resolution process. Statistical appeals were resolved only through recalculation of the AZ LEARNS profile by ADE staff using any corrected data submitted by the school. The purpose of a statistical appeal is principally to advise ADE staff that data was in error and has been corrected. Calculations submitted by schools via an appeal were not taken at face value nor used to alter a profile if the underlying data was not corrected. Schools that were labeled underperforming for a third consecutive year were entitled to a site visit to determine if the label was warranted. These site visits were important as the third year underperforming label merited an alternative designation of failing to meet the academic standards in accordance with statute. After the statistical review of appeals took place, schools labeled underperforming for a third consecutive time were scheduled to receive that visit. Teams from the Research and Evaluation and State Intervention sections of the ADE visited each of the third year underperforming schools to gather additional supporting data for the appeal as well as gather information related to the school’s Arizona school improvement plan (ASIP) as required by statute. All information gathered from the site visits was taken to the substantive appeal committee for use in the third stage of the appeal process. Only third year underperforming schools received site visits as part of the appeals process. Substantive appeals were resolved in a committee process. Committee members represented a diverse background of ADE staff and school administrators to ensure that appeals were considered from multiple perspectives. Appeals were evaluated using an appeals rubric approved by the State Board of Education that evaluated the argument presented and whether or not the evidence provided to support the argument was compelling. The appeals rubric consisted of a three-tiered system for appeal evaluation: 1) Initial review of the appeal to determine its merit. 2) Review of the evidence provided. 3) Committee recommendation. Initial Review. The substantive appeal rubric provided for three categories that apply during the initial review. Each appeal was classified into the categories based on the information provided in the appeals. 1. Mitigating Factors Outside the School’s Control. Appeals of this nature referenced when the school indicated significant issues that affected test scores outside of the school’s control. If a school provided information detailing a significant event that impacted test scores, which was clearly outside the school’s control, the appeal was deemed as passing the initial review. Appeals involving the adverse affect of school or Arizona's Accountability System: Volume I Arizona Department of Education 28 district guidelines; errors made by school or district personnel regarding test administration or data entry; or events whose impact could have been foreseen and mitigated by school or district action were not considered valid appeals. 2. Implementation of the School Improvement Plan. Appeals in this category discussed how the school was actively, consistently, and reliably implemented the school improvement plan; that the priorities of the solutions team had been addressed; the school improvement plan had been revised and updated to address assessed ongoing needs; professional development that supported the targeted goals had been planned and implemented. If a school provided information highlighting their school improvement plan, it may have accentuated the other two components on the rubric. However, per statute, a successful implementation of the school improvement plan alone cannot change a classification of a school. Additionally, only third-year underperforming schools may include information about their ASIP in their appeal. Review of Evidence. Once the appeals progressed through the first tier of the rubric, initial review, the evidence provided to support the appeal was evaluated. In this tier, three determinations were possible: 1. Compelling evidence: in this area, the school or district adequately provided information that led the committee to conclude that, had the circumstance been different, the achievement profile would have been different as well. If a school had a special circumstance that affected a certain grade and was able to demonstrate that the specific grades test scores suffered, the school was deemed to have provided compelling evidence. Again, if a school provided compelling evidence highlighting their school improvement plan, it may have accentuated the other two components on the rubric (data calculations/mitigating factors). However, per statute, a successful implementation of the school improvement plan alone could not change the classification of a school. 2. Not compelling evidence: appeals were categorized in this area when they were able to provide information that a significant issue could have impacted the school’s performance but did not provide detailed, specific information as to specific outcomes that hindered the school’s performance. 3. Not applicable evidence: if an appeal was submitted, made it through the initial review, and presented evidence that was not linked in any way to the performance of the school, the evidence was deemed not applicable. If the evidence did not directly support the claim made in the appeal, it was deemed not applicable. Committee Recommendation. Once the appeal and evidence were reviewed, the committee arrived at a decision as to the outcome of the appeal. There were three possible outcomes: 1. Appeal granted and AZ LEARNS determination changed. In these cases, the appeal successfully made it through the initial review and evidentiary stages. It was determined that the points needed to change classifications would have been earned by the school had the special circumstance/data discrepancy not occurred. Therefore, the classification for the school was changed. Arizona's Accountability System: Volume I Arizona Department of Education 29 2. Appeal granted and AZ LEARNS determination remains the same. In these cases, the appeal successfully made it through the initial review and evidentiary stages. However, it was determined that the criteria needed to change classifications were not earned by the school had the special circumstance/data discrepancy not occurred. For example, a school provided information and evidence that their AYP point was not accurately included in the calculations and the committee determined the school provided information to prove they earned the one point set aside for AYP in the AZ LEARNS formula. However, the school needed 5 points to get from underperforming to performing. Therefore, the one point earned was not enough to change their designation, so their determination remained the same. 3. Appeal Denied. In these cases, the appeal did not successfully make it through the initial review and evidentiary stages. Therefore, the classification for the school remained the same. Step 4. Notification of Result Sent to Schools. Once all appeals were resolved, notifications were sent to the districts and/or schools that had filed appeals. The contact person of record for the district/school received an email from Achieve with directions as to how to access appeal information via the Common Logon when the appeal had been processed. Districts and schools were notified before the final public release of the AZ LEARNS profiles as to the outcome of the appeal process. All appeals were final. Special Appeals for 2006 The following appeals were allowed for special circumstances during the 2006 school year. Appeal of writing scores. Due to anomalies in scoring of the writing portion of the AIMS DPA, schools serving grades three through eight had the option to appeal the status and growth points earned in their profile. Schools could appeal to be held harmless from any adverse impact by requesting that their profile be recalculated using the total points earned in 2005 for status and growth in writing. Example. School A serves grades K-5. Below are the status/growth points earned by the school in writing for 2005 and 2006. Grade 3 4 5 Average 2005 3.3 3 4.7 3.7 2006 2.3 3 3.7 3 In 2005 the school earned 3.7 points from writing. This year the school is earned 3 points in 2006. Upon appeal, the school’s final profile for 2006 would be calculated using 3.7 points rather than 3. • This appeal was only available for grades 3-8. Arizona's Accountability System: Volume I Arizona Department of Education 30 • This appeal was only granted at the level of total points earned, not for individual grades. • This appeal was only for the status/growth portion of the formula. Any impact of scoring on the percent of students exceeding was already been accounted for. Appeal of AYP determination. A school may have appealed its AYP determination if it did not make AYP because of the inclusion of ELL students or the exclusion of students with alternate accommodations, and not making AYP adversely affected its AZ LEARNS profile. • If a school appealed the inclusion of ELL students in its AYP determination, the determination was recalculated with ELL students counting as tested but not included in the AMO calculation. • If a school appealed the exclusion of students tested with alternate accommodations, the determination was recalculated with such students counting as tested but as not proficient in the AMO calculation. Arizona's Accountability System: Volume I Arizona Department of Education 31 10. AZ LEARNS Achievement Profiles for Alternative Schools In 2004, the ADE published profiles for alternative schools for the first time. Alternative schools are defined as schools that meet the Board-approved definition as schools whose sole and clearly-stated mission is to serve specific populations of at-risk students. Alternative school status is granted by application to the ADE. A.R.S. §15-241 makes an allowance for a “parallel” evaluation method for alternative schools. When AZ LEARNS achievement profiles were first issued in 2002 the Board determined that alternative schools would not receive an achievement profile using the conventional AZ LEARNS methodology, and that ADE should develop an alternate method for evaluating these schools. Definition of an Alternative School The following is the definition of an alternative school as approved by the Board of Education in 2002. There are currently 138 schools that have been granted alternative school status. 1. A school operated by a school district must have adopted a mission statement that clearly identifies its purpose and intent to serve a specific student population (please see criterion three) that will benefit from an alternative school setting. A charter school must be expressly chartered to serve a specific student population that will benefit from an alternative school setting. 2. The educational program and related student services of the school must match the mission or charter of the school. 3. The school must intend to serve students exclusively in one or more of the following categories: • Students with behavioral issues (documented history of disruptive behavior) • Students identified as dropouts • Students in poor academic standing who are either severely behind on academic credits (more than one year) or have a demonstrated pattern of failing grades • Pregnant and/or parenting students • Adjudicated youth 4. Any school offering secondary instruction for academic credit used to fulfill Arizona State Board of Education graduation requirements (in part or in full) must offer a diploma of high school graduation. Arizona's Accountability System: Volume I Arizona Department of Education 32 General Process to Produce Achievement Profiles for Alternative Schools The small number and wide variety of alternative schools makes it difficult to develop reliable parameters for measures of school performance. Consequently, a single method of evaluating alternative schools was created that employed performance measures available to the majority of schools. In plainer terms, it would be too inaccurate to develop separate methods for large alternative high schools, small alternative high schools, alternative middle schools, and small alternative middle schools. Given the small number of schools in each category, benchmark performances and expectations would be very unreliable. Consequently, ADE developed a single rubric to evaluate all alternative schools. The method for calculating an achievement profile for alternative schools is as follows: 1. AIMS scale score points are calculated using Status only. Status points were calculated like the regular schools based on percentage of students passing the AIMS test. 2. Added evidence points are not included in the achievement profile for alternative schools. Because most alternative schools below the high school level are quite small and have highly mobile populations, there would be little or no students in the Measure of Academic Progress analysis. Consequently, the MAP analysis would be highly volatile and inaccurate. 3. Points based on an alternative school’s dropout rate are calculated as for all other schools. 4. As with other schools, alternative schools will receive one scale score point if they make AYP for the current year. 5. Alternative schools will only receive labels of performing and underperforming. Because of the uncertainty of measurement associated with small sample sizes and the high stakes of school labels, schools initially determined to be underperforming receive a “second look.” Instead of determining baseline groups based on the mean percent of students passing AIMS, an alternate baseline group for these schools is determined based on the upper bound of a 95 percent confidence interval around the mean. If a school initially determined to be underperforming moves to a higher classification due to the “second look,” that school will receive a performing label. Calculation of an Achievement Profile for an Alternative School In addition to AIMS scale score points, alternative schools also earn scale score points via their dropout rates and their current AYP status. The methods used to calculate scale score points earned by alternative schools for these performance measures are the same as the methods used for other schools. Figures 10.1 and 10.2 give a graphical summary of the method for evaluating alternative schools at the elementary and high school levels. Arizona's Accountability System: Volume I Arizona Department of Education 33 • A school may receive up to six scale score points for each subject/grade combination it serves. This is represented by the far left column in each figure. • A school may receive one point if it has made adequately yearly progress (AYP) under the methodology mandated by the No Child Left Behind Act. This is represented by the column second-from-left in each figure. • If a school is a high school it may receive one point based on its dropout rate. This is the third-from-left column in Figure 10.2. • The total points earned by a school are added up and compared to the school classification scale to determine a school’s preliminary classification—the final column in Figures 10.1 and 10.2. Figure 10.1. Method for Evaluating Alternative Schools (Elementary Schools) Each Subject/Grade Whole School Status Grouping School Classification Scale AYP 6 5 1 4 Made AYP 3 2 = + P U 1 0 0 Missing Data Missed AYP Arizona's Accountability System: Volume I Arizona Department of Education 34 Figure 10.2. Method for Evaluating Alternative Schools (High Schools) Each Subject/Grade Status Grouping Whole School Whole School AYP Graduation Rate/ Dropout Rate 6 5 2 Made Graduation and Dropout Rate Requirements 1 4 3 + Made AYP + 2 1 School Classification Scale 1 Made either Graduation or Dropout Rate Requirement = P U 0 0 Missing Data Missed AYP 0 Did not make both requirements Classification Scales for Alternative Schools The following classification scales are used for alternative schools. Separate classification scales are used for high schools and elementary schools due to extra points being available to high schools from the dropout rate. 358 Schools High Schools Table 10.1. Alternative School Classification Cut Points Subject/Grade Subject/Grade Subject/Grade Combination Combination Combination 1 3 6 Underperforming < 2.0 < 5.0 < 11.0 Performing 2.0 5.0 11.0 Underperforming <2.0 <6.0 <1.0 Performing 2.0 6.0 12.0 Arizona's Accountability System: Volume I Subject/Grade Combination 9 < 16 16 <18.0 18.0 Arizona Department of Education 35 11. AZ LEARNS Achievement Profiles for K-2 Schools When AZ LEARNS achievement profiles were first issued in 2002 the Board determined that K-2 schools would not receive an achievement profile using the conventional AZ LEARNS methodology, and that ADE should develop an alternate method for evaluating these schools. In 2004, the ADE published profiles for K-2 schools for the first time. K-2 schools are schools that serve only grades kindergarten through second grade. Since the AIMS is not administered at any of the grades served by these schools, the AZ LEARNS profiles are based solely on the performance of the schools’ second graders on the state’s norm-referenced test. The method of calculating the profile for these schools is straightforward: 1. The average normal curve equivalents (NCE) on the reading and math portions of the test are calculated for the most current year for a school’s second graders. 2. The average normal curve equivalents for the school are added together, and 3. Compared to a scale to determine the school’s label. Example. In 2006, the average NCE for second graders in Gila Monster Elementary was 52 for math and 48 for reading. The two averages summed together yield 52 + 48 = 100 points. This is sufficient for Gila Monster to be a highly performing school. Table 11.1 provides the performance thresholds for K-2 schools. Table 11.1. AZ LEARNS Scale for K-2 Schools Points Achievement Profile <70 Underperforming 70 Performing 97 Highly Performing 106 Excelling Arizona's Accountability System: Volume I Arizona Department of Education 36 Appendix I: Substantive Appeal Committee Evaluation Rubric for AZ LEARNS Evaluation Criteria Data Calculation Discrepancies i.e., school attempts to compare data details with their data sets and gets different numbers Initial Review (Please check the applicable option) Data does not Not ADE data are match that of applicable accurate and ADE. School calculations are submits correct. evidence of discrepancies and provides additional data. Review of Evidence Provided Comments † Compelling evidence † Not compelling evidence † Not applicable evidence Special Circumstances Outside the Control of School/District Administration or Management i.e., school indicates significant teacher attrition; environmental issues/events; adverse testing conditions; school/community emergency/crisis Not applicable Special circumstances that were outside of the school’s control, were not a substantial cause of the overall school performance. School had a situation that was unavoidable and outside of the school’s control and hindered the test administration or student performance. This situation resulted in adverse data for the year(s) in question. † Compelling evidence † Not compelling evidence † Not applicable evidence Guideline/Methodology Issues The ADE will not accept/review appeals related to guideline/methodology. i.e., school disagrees with use of two year baseline Appendix II: Substantive Appeal Rubric for AZ LEARNS Team Decision AZ LEARNS Substantive Appeal (Results represent group consensus regarding appeal) Reason Reviewed Data Calculation Discrepancies Special Circumstances Guideline/Methodological Issues Initial Review Review of Evidence N/A Correct Data does N/A Compelling data/calculation not match evidence N/A Did not cause overall performance Adverse result based on situation Committee Recommendation: Granted Denied Final Appeal Decision: Granted Denied Appeal Result: Compelling evidence The ADE will not accept/review appeals related to guideline/methodology. Please indicate appropriate response(s) by checking within the box(es) provided. Comments: N/A Not compelling evidence Not compelling evidence