Arizona’s School Accountability System Technical Manual Volume I: Arizona LEARNS Achievement Profiles Prepared by Robert Franciosi and Jennifer Regalado Education Research Associates Research & Policy Arizona Department of Education Ildiko Laczko-Kerr, Deputy Associate Superintendent Tom Horne Superintendent of Public Instruction Arizona Department of Education Published by Arizona Department of Education February 17, 2004. Table of Contents INTRODUCTION........................................................................ 1 DEVELOPMENT OF THE ARIZONA LEARNS SYSTEM ................... 1 WHAT’S NEW FOR THE 2003 AZ LEARNS ACHIEVEMENT PROFILES ..................................................................................... 3 STATE BOARD APPROVAL OF AZ LEARNS ACHIEVEMENT PROFILE METHODOLOGY ..................... 5 OVERVIEW OF THE AZ LEARNS ACHIEVEMENT PROFILE....................................................................................... 7 GENERAL PROCESS TO PRODUCE THE ACHIEVEMENT PROFILES .. 7 THE ELEMENTARY SCHOOL MODEL (GRADES K-8)...................... 8 PRINCIPLES BEHIND THE USE OF TEST SCORES TO MEASURE SCHOOL PERFORMANCE ............................................................... 9 THE HIGH SCHOOL MODEL ........................................................ 11 DATA VERIFICATION AND TIMELINE ............................. 14 TIMELINE ................................................................................... 15 CALCULATION OF AIMS BASELINE AND GROWTH SCALE POINTS ......................................................................... 16 OVERVIEW ................................................................................. 16 IDENTIFYING BASELINE GROUPS ............................................... 16 IDENTIFYING GROWTH GROUPS ................................................. 18 CALCULATION OF SUBJECT/GRADE SCALE POINTS FROM AIMS 22 SPECIAL CASES: NEW SCHOOLS, AND MISSING DATA ............... 23 ADDED EVIDENCE: MEASURE OF ACADEMIC PROGRESS AND EXTENDED WRITING ............................ 25 OVERVIEW ................................................................................. 25 MEASURE OF ACADEMIC PROGRESS .......................................... 25 LONGITUDINAL MATCHING ....................................................... 26 MAKING ONE YEAR’S GROWTH ................................................ 27 EXTENDED WRITING .................................................................. 28 CALCULATING ADDED EVIDENCE POINTS ................................. 28 GRADUATION AND DROPOUT RATES.............................. 31 OVERVIEW ................................................................................. 31 GRADUATION RATES ................................................................. 32 DROPOUT RATES........................................................................ 33 EVALUATING THE TOTAL SCALE SCORE VALUE TO DETERMINE A SCHOOL CLASSIFICATION .................... 36 Arizona’s Accountability System: Volume I Arizona Department of Education i APPLICATION OF THRESHOLD CRITERIA FOR EXCELLING AND HIGHLY PERFORMING SCHOOLS .. 39 AZ LEARNS ACHIEVEMENT PROFILE APPEALS PROCESS .................................................................................... 43 STEP 1: DATA VERIFICATION/DATA APPEAL ............................. 43 STEP 2: APPEAL APPLICATION ................................................... 44 STEP 3: APPEAL RESOLUTION .................................................... 45 SMALL SCHOOL SCORE ADJUSTMENT........................... 50 CRITERIA ................................................................................... 50 METHOD .................................................................................... 50 EXEMPTED SCHOOLS ........................................................... 52 OVERVIEW ................................................................................. 52 EXTREMELY SMALL SCHOOLS ................................................... 52 ALTERNATIVE SCHOOLS ............................................................ 54 ACCOMMODATION SCHOOLS ..................................................... 56 NEW SCHOOLS ........................................................................... 56 K-2 SCHOOLS............................................................................. 56 Arizona’s Accountability System: Volume I Arizona Department of Education ii List of Tables and Figures Table 1.1. Comparison of Arizona’s Accountability Systems Requirements .......................................................................... 1 Figure 3.1: Achievement Profile (Elementary Model) ................... 9 Figure 3.2: Linear Model of Making OYG................................... 10 Figure 3.3: Conceptual Model of Making OYG........................... 11 Figure 3.4: Achievement Profile (Secondary Model).................. 12 Table 5.1. Baseline Groupings..................................................... 18 Table E3.1. Number of Students.................................................. 20 Figure 5.1. Determination of Growth Group Cut Points ............. 21 Table 5.2. Growth Point Groupings............................................. 22 Table 5.3. AIMS Scale- Point Distributions by Baseline Grouping and Growth Point Grouping.................................................. 23 Figure 6.1: Summary of MAP Characteristics.............................. 26 Table 6.1. Determining One Year’s Growth................................ 27 Table 6.2. Distribution of Elementary Added Evidence Points by Subject/Grade Combination.................................................. 29 Table 6.3. Elementary School Scale Permutations ...................... 29 Table E6.1 Added Evidence Points. ............................................. 29 Figure 7.1. Graduation Rate Calculation ..................................... 32 Figure 7.2. Calculating a Two Year Average for Graduation Rate ............................................................................................... 32 Figure 7.3. Calculating an Annual Dropout Rate ........................ 34 Figure 7.4. Calculating a Three-Year Average for Dropout Rate 34 Arizona’s Accountability System: Volume I iii Arizona Department of Education Table 7.1. Target Improvements for Dropout and Graduation Rates in the High School Achievement Profile .................... 35 Table 7.2. Point Value Outcomes ................................................. 34 Table E8.1. Scaled Score Thresholds ........................................... 35 Table E8.2. Scaled Score Thresholds ........................................... 36 Table 8.1. Elementary School Classification Cut Points ............. 36 Table 8.1. Elementary School Classification Cut Points ............. 36 Table 8.2. High School Classification Cut Points........................ 37 Figure 9.1. Chart of Potential Scenarios Resulting from Threshold Marks Set for Excelling and Highly Performing Achievement Profile Classifications ........................................................... 41 Table 9.1. Excelling and Highly Performing Threshold Values by Grades Offered...................................................................... 42 Table E9.1. Number of Students Exceeding the Standard—All Grades ................................................................................... 42 Table E12.1. An Extremely Small Schoola .................................. 53 Table E12.2. An Extremely Small School ................................... 53 Arizona’s Accountability System: Volume I iv Arizona Department of Education Introduction In November of 2001, Arizona voters approved Proposition 301 that among other things provided funds to the Arizona Department of Education (ADE) to develop “a system to measure school performance based on student achievement, including student performance on the AIMS test.” The actual legislative requirements for the accountability system are stated in section 15241 of the Arizona Revised Statutes (ARS § 15-241). The accountability system created to satisfy the statute is referred to as the Arizona LEARNS Achievement Profile. During that same year, the federal government enacted the No Child Left Behind Act (NCLB) that required states to establish an accountability system to evaluate the performance of local public schools and school districts. In response to both laws, the Arizona Department of Education (ADE) has developed an integrated accountability system that evaluates the performance of all public schools, districts, and charter holders in the state. The following table provides an overview of the two parts of the accountability system. Table 1.1. Comparison of Arizona’s Accountability System Requirements NCLB Required by federal law Arizona LEARNS Achievement Profile Required by state law One-year snapshot of student performance Longitudinal examination of student performance Components of evaluation • AIMS scores • Percent students assessed • Attendance/Graduation rates Components of evaluation • AIMS scores • Measure of Academic Progress • Graduation/dropout rates • AYP Labels schools on a yes/no system Labels schools on a graded scale: • • • • • Failing to meet academic standards Underperforming Performing Highly performing Excelling Development of the Arizona LEARNS Achievement Profile System Since the passage of Proposition 301 and NCLB, the Arizona Department of Education (ADE) has consulted with a diverse Arizona’s Accountability System: Volume I Arizona Department of Education 1 group of experts, ranging from measurement experts, curriculum coordinators to classroom teachers. These experts volunteered their time to undertake the difficult task of advising the department on the complex issue of state-level school accountability. During the summer of 2002, the ADE met with district and educational representatives and formed the AZ LEARNS/NCLB Accountability Formula Working Group. Since then, this group has worked to create and update the formula for AZ LEARNS. This group dealt with the fundamental questions regarding school accountability and sought to develop a fair and accurate system to measure student achievement and school performance. The ultimate purpose of the school performance evaluation is to advance student learning at the local level. This purpose guided all decisions in the development process. With that understanding, the ADE along with the AZ LEARNS/NCLB Accountability Formula Working Group developed the AZ LEARNS Achievement Profile with the explicit purpose of identifying schools using a fair and accurate classification system based on a set of academic performance indicators. The following are key components of AZ LEARNS that are necessary to meet the purpose and ensure that school performance evaluations are both valid and reliable. ¾ A spectrum of school “classifications” to identify diverse school outcomes. School performance is based on a continuum, and multiple school classifications are necessary to most accurately reflect that continuum. In addition, a multiple classification system is the optimal method to provide meaningful information to stakeholders. The achievement profile will include classifications to identify diverse states of school performance. ¾ Absolute and contextual achievement indicators to report school performance in the appropriate context. Contextual achievement indicators are established through an empirical process. The results of this process yield cut scores, which are grounded within the context of student achievement in our state. An empirical process is the most accurate approach, because the scores will be based solely on the results of data analysis. Our goal is to classify every school accurately. Any school-level scores that are established without the proper consideration of all necessary data are essentially arbitrary, resulting in the likely misclassification of schools. ¾ An achievement system for elementary and high schools ¾ Longitudinal data to capture student and school trends ¾ Academic achievement by all student groups ¾ Consideration of the effects of student mobility Arizona’s Accountability System: Volume I Arizona Department of Education 2 ¾ Multiple outcome indicators, such as dropout, attendance and graduation rates ¾ Multiple levels of reporting, ranging from the media level to the school/classroom level. Improved student learning is the central goal of the achievement profile. To that end, the results of the achievement profile must be communicated at multiple levels and with varied amounts of detail depending on the target audience. At the media level, results will be “publicly consumable” to allow for the information to be communicated efficiently and easily understood by a diverse audience. As the target audience approaches the classroom level, the level of detail will increase to provide information for programmatic and instructional purposes. The Department of Education will ensure that each level of communication builds on the previous level and provides consistent information. ¾ Development of one comprehensive system to supersede other fragmented accountability measures in state statute and to fulfill federal requirements ¾ A parallel accountability system for unique schools (i.e., accommodation/alternative, extremely small). In order to avoid the pitfalls of a “one size fits all” approach, AZ LEARNS includes a parallel accountability system for schools with unique characteristics such as accommodation schools and extremely small schools. The unique characteristics of this subset of schools preclude them from fitting into a general accountability system with fair treatment. Many states provide a parallel accountability system for these schools with the goal of improving the accountability system for all schools. What’s New for the 2003 AZ LEARNS Achievement Profiles In response to comments on the 2002 achievement profile process and methodology, in order to strengthen the accuracy and fairness of the system, and to integrate AZ LEARNS with the accountability system mandated by the No Child Left Behind Act of 2001 (NCLB) the following changes were made to the AZ LEARNS Achievement Profile school evaluation process for 2003: • New School Classifications. The 2002 classifications of underperforming, maintaining, improving, and excelling have been replaced by underperforming, performing, highly performing, and excelling. • New Definition of Excelling. The definition of excelling that had been defined by state statute has been eliminated. Arizona’s Accountability System: Volume I Arizona Department of Education 3 The standards for an excelling school are now set by the Arizona Board of Education. • Two-Year Baseline. The baseline performance levels are now based on a weighted average of the 2000 and 2001 AIMS results. The baseline for high school math remains based on 2001 AIMS results. • Increase in Minimum Group Size. The minimum group size to be included in the achievement profile calculation was increased from 15 to 30 valid assessment scores. • Omission of mobile students. To be consistent with the requirements of No Child Left Behind, mobile students were omitted from the achievement profile calculation. • Six Baseline Groupings. The number of baseline groupings has been raised from five to six. • New Method for Aggregating Baseline and Growth Performance. The grid used in 2002 for determining subject/grade values has been replaced. Subject/grade values are now determined with by weighted sum of baseline and growth points. • Addition of an Adequate Yearly Progress (AYP) Performance Indicator. A school’s AZ LEARNS achievement profile now takes into consideration the school’s performance on the AYP criteria mandated by the No Child Left Behind Act (NCLB). • Greater Weight Placed on the Added Evidence. The calculation of the AZ LEARNS achievement profile now places a greater weight on the two additional indicators the Extended Writing Score (EWS) for AIMS and the Measure of Academic Progress (MAP). • Reweighting of the MAP in Added Evidence. The MAP now has greater weight when calculating added evidence points. • Added Performance Criteria for Highly Performing and Excelling Schools. For a school to receive a classification of highly performing or excelling a given percentage of its students must have attained the performance level of exceeding the standard. • Data Verification Period. The Arizona Department of Education opened a data verification period to allow schools to verify that all data used in the achievement profile calculations were correct. • Appeals Process. An appeals process has been created to allow schools to appeal their AZ LEARNS achievement profile. Arizona’s Accountability System: Volume I Arizona Department of Education 4 State Board Approval of AZ LEARNS Achievement Profile Methodology Through the spring and summer of 2003, the Arizona State Board of Education reviewed, commented upon, and approved the AZ LEARNS Achievement Profile methodology, or formula, and the subsequent components of the elementary and high school accountability models. The ADE provided the State Board information packets that outlined the decisions regarding the formula that needed to be made. This documentation can be found online via the AZ LEARNS web site, http://www.ade.az.gov/azlearns/boardinfo/. On April 28, 2003, the board approved modifications to the AZ LEARNS Achievement Profile methodology including the use of a two-year average to determine the baseline groupings; the use of baseline and growth group scales that ranged from one to six; a one-zero scale for adequate yearly progress; a two-zero scale for the graduation/dropout rates; the use of a 70-30 float weight when adding growth and baseline scale points; the calculation of the total scale score; the application of added evidence for elementary schools; the exclusion of mobile students; raising the minimum N count from 15 to 30; and the use of a compensatory model. (Appendix I:) On May 19, 2003, the board approved the use of one year of data to calculate the baseline groups for high school mathematics. (Appendix II:) On June 30, 2003, the board approved the use of a three-year rolling average to calculate the growth group for high school mathematics, and approved the thresholds and calculations for the additional indicators used in the adequate yearly progress evaluation. (Appendix III:) On August 25, 2003, the board approved the methodologies to be used to evaluate K-2, K-1, new schools, and small schools under both the AZ LEARNS and NCLB accountability systems. It also approved the policy for schools missing data under AZ LEARNS. (Appendix IV:) On September 16, 2003 the board approved the complete AZ LEARNS methodology, including the cutpoints used for both Arizona’s Accountability System: Volume I Arizona Department of Education 5 baseline and growth point groups, the formula used to calculate added evidence for elementary schools, and the use of a performance threshold for highly performing and excelling schools. (Appendix V:) Arizona’s Accountability System: Volume I Arizona Department of Education 6 Overview of the AZ LEARNS Evaluation System This section provides and overview of the determination of school achievement profiles for AZ LEARNS. More detailed discussions of the methodology used to determine the profiles, including descriptions of equations, algorithms, and data used are given in subsequent chapters. Under AZ LEARNS, there are two separate models for evaluating schools: one for K-8 schools and one for high schools. A school that serves both grades K-8 and high school receives two separate achievement profiles. General Process to Produce the Achievement Profiles The general process to calculate the achievement profile for each school is as follows: A. Identify the baseline group for each subject/grade combination (baseline grouping) and establish associated scale values. B. Calculate total growth points for each subject/grade combination (growth point grouping) and establish associated scale values. C. Complete a determination of adequate yearly progress (AYP) as mandated by the No Child Left Behind Act of 2001 for each public school and establish associated scale values. D. Add additional (non-AYP) indicators of graduation rate and dropout rate and establish associated scale values [secondary schools only]. E. Calculate a total scale score value by adding the baseline group scale values for each subject/ grade combination to the growth point group scale values for each subject/grade combination (giving a 70% weight to the school’s strongest scale value and 30% weight to the other scale value) plus the AYP scale value plus the Arizona’s Accountability System: Volume I Arizona Department of Education 7 additional (non-AYP) indicator scale value [secondary schools only]. F. Evaluate the sum of all scale values (i.e. the total scale score value) in relation to the school classification scale and associated cut points to determine secondary school achievement profile classifications and preliminary (prior to the added evidence indicator being applied) elementary school achievement profile classifications. G. Add “additional evidence” (MAP and EWS) to total scale score value in order to produce elementary school classifications. H. Apply threshold criteria for excelling and highly performing achievement profile school classifications. Schools will be evaluated based on average percentage of students in the “exceeds the standard” category on AIMS as well as the total scale score values. Requisite percentages were set for excelling and highly performing classifications based on the subject/grade combinations assessed at a particular school. The Elementary School Achievement Profile Model Overview (Grades K-8) Figure 3.1 provides an overview of how the achievement profile is calculated for an elementary school. Under AZ LEARNS, elementary schools are evaluated based on four primary indicators: ¾ Baseline performance on AIMS and growth in student performance in AIMS ¾ The measure of academic progress (MAP) and percent of students scoring 24 or more on EWS. ¾ The measure of adequate yearly progress (AYP) required by NCLB, and ¾ The percentage of students exceeding the standard on AIMS. Example 1. Gila Monster Elementary, a K-6 school, has earned 24 points on AIMS, 3.75 points for MAP/EWS, and 1 point for making AYP. This gives it a total of 28.75 scale points. For a school with a K-6 grade configuration, this is sufficient points to classify Gila Monster Elem. as a highly performing school. However, only 15 percent of students at Gila Monster Elem. exceeded the standard in AIMS over the past three years. This is less than the threshold of 30.9 percent needed by schools with K-6 configurations to qualify as a highly performing school. Therefore, Gila Monster Elem. is classified as a performing school. Arizona’s Accountability System: Volume I Arizona Department of Education 8 In order to comply with the mandate of the No Child Left Behind Act of 2001 that the state have an integrated accountability system, a school’s AYP determination was factored into the calculation of its achievement profile. To create an incentive for schools to increase the achievement of average and above-average students, thresholds were established for the highly performing and excelling labels. A school did not earn the highly performing or excelling labels unless over the previous three years the percentage of its students exceeding the standard on AIMS met the thresholds set as well as met the scale score requirements. Figure 3.1: Achievement Profile (Elementary Model) Each Subject/Grade Whole School Baseline Grouping Growth Point Grouping 6 6 5 5 Pre-MAP/EWS Whole School School Classification Scale AYP Additional Points 1 E 4 4 3 3 + 2 2 1 1 0 Missing Data 0 Missing Data Made AYP + = HP P 0 Missed AYP U School Classification Scale MAP/ EWS + Up to 40% of Total subject/ grade points and AYP calculation E = 70/30 Float Weight Principles Behind the Use of Test Scores to Measure School Performance AIMS provides a benchmark to the Arizona Academic Standards, and MAP provides a longitudinal view of individual student progress across all grade levels. These indicators ensure that all grade levels in the elementary school share in the responsibility of moving all students forward. The component that examines baseline and growth in student performance in AIMS is an absolute standard that provides a snapshot of 3rd, 5th and 8th grade performance across multiple years and focuses on reducing achievement gaps between groups of students. MAP is a contextual standard that takes into consideration where individual students begin (on an academic Arizona’s Accountability System: Volume I Arizona Department of Education 9 HP P U level) and ensures all students, regardless of achievement level, are making One Year’s growth (OYG) each academic year. Student groups vary from year to year and their performance will fluctuate from one year to the next, regardless of schooling effects. This phenomenon is known as a cohort effect. Both AIMS and MAP results will be based on a 3-year rolling average to capture trends, rather than aberrations, and to account for differences in student cohorts. A 3-year average mitigates these cohort effects and increases the reliability of the results. Furthermore, multiple years allows for the identification of real trends in school performance. A downward bump may be attributed to cohort effects, but a downward trend (continuous bumps) is an indication of lagging performance. Figure 3.2: Linear Model of Making OYG AIMS AIMS 3-year 3-year 3-year Average Average Average 1 Grade 2 3 4 5 AIMS 6 7 8 MAP (SAT9) The MAP Indicator will ensure that students in all achievement groups are making One Years growth (OYG) based on the results of the Stanford Achievement Test, Ninth Edition (SAT9). MAP also ensures that all grades are represented in the revised Achievement Profile. By requiring that students in all achievement levels make OYG, the elementary model ensures that all students are progressing academically. Schools are evaluated based on the percentage of students that achieve OYG. Arizona’s Accountability System: Volume I Arizona Department of Education 10 Figure 3.3: Conceptual Model of Making OYG A Stanines 1 2 B 3 4 C 5 6 7 8 9 (SAT 9) Below Average Average Above Average Though remaining at the same stanine is an accurate measure of OYG, this standard will not be sufficient for some schools to demonstrate adequate progress on AIMS. Schools with large numbers of students in the lower stanines will be required to advance students into higher stanines in order to meet the absolute standard at the AIMS grade levels. The elementary model is designed to allow MAP to complement AIMS, but the model maintains primary emphasis on achievement of the academic standards via AIMS. The High School Achievement Profile Model Overview Figure 3.4 provides and overview of how an achievement profile is calculated for a high school. Like the elementary model, the high school model incorporates multiple measures to capture the breadth of school performance. The five measures in the high school model are the following: ¾ Baseline performance in AIMS and growth in student performance in AIMS ¾ Graduation rate ¾ Dropout rate ¾ The measure of Adequate Yearly Progress (AYP) required by NCLB, and ¾ The percentage of students exceeding the standard on AIMS. Example. Desert Mountain Cactus Vista High School has earned 12 points on AIMS, 2 points for its graduation/drop out rates, and 1 point for making AYP. This gives it a total of 15 scale points— exactly the number needed to be classified as a highly performing school. In addition, 11 percent of students at DMCVHS have exceeded the standard in AIMS over the past three years—more than the threshold of 9.3 percent needed by high schools to qualify Arizona’s Accountability System: Volume I Arizona Department of Education 11 as highly performing. Thus, DMCVHS is classified as a highly performing school. Figure 3.4: Achievement Profile (Secondary Model) Each Subject/Grade Whole School Baseline Grouping Growth Point Grouping 6 6 5 5 1 4 4 Made AYP 3 + 3 Whole School AYP + 2 2 1 1 0 0 Missing Data 0 Missing Data Missed AYP School Classification Scale Graduation Rate/ Dropout Rate 2 Made Graduation and Dropout Rate Requirements + 1 Made either Graduation or Dropout Rate Requirement 0 Did not make both requirements E = HP P U 70/30 Float Weight Just as the elementary model, three years of AIMS results are included in a rolling average for reading, writing, and mathematics. A two-year average is used for the baseline measures in reading and writing. The results of a single year—2001—are used for the baseline measure in mathematics. Due to changes in state testing the 1999-2000 mathematics results are not comparable to the 200001, 2001-02 and 2002-03 administrations of the high school AIMS mathematics assessment. Currently, the methodology for high schools uses all grades tested (10, 11 and 12) in each of the years provided. The graduation rate used in the AZ LEARNS Achievement Profiles calculation is a five-year, longitudinal measure of how many students graduate from high school. By examining a cohort of students who began high school at the same time, the graduation rate assesses how many students actually complete high school. The dropout rate is an annual measure of how many students drop out of a school. It is expressed as the proportion of students who had the opportunity to drop out, and did, during a 12-month reporting period. The dropout rate provides an annual snapshot Arizona’s Accountability System: Volume I Arizona Department of Education 12 and detects more immediate changes in school attendance than the graduation rate. The graduation and dropout rates are important complements to the test score results in the achievement profiles. Graduation rates indicate the success of students in meeting course requirements and achieving passing grades in subject areas not covered by the AIMS test. Dropout rates are included in the high school model as a measure of student persistence and to ensure that AIMS results reflect the largest percentage of the student population possible. As with elementary schools, AYP determination was factored into the calculation of the achievement profile for high schools in order to comply with No Child Left Behind. Also, thresholds were established for the highly performing and excelling labels. A school did not earn the highly performing or excelling labels unless over the previous three years the percentage of its students exceeding the standard on AIMS met the thresholds set as well as met the scale score requirements. Arizona’s Accountability System: Volume I Arizona Department of Education 13 Data Verification and Timeline Because of the stakes involved and the volume and scope of the data used, the ADE considered it prudent to allow districts and charter holders to review their data before preliminary AZ LEARNS Achievement Profiles calculations were carried out. From September 19, 2003, through October 1, 2003 an application to verify data was made available to districts and charter holders through the common logon on the ADE web site. The application displayed all the data that were to be used for the calculation of the achievement profile. In addition, a link was provided through the common logon that allowed schools to download student-level testing data in order to make any necessary corrections. All schools in districts or under a charter holder that expected to receive an achievement profile were required to verify school data and additional school level information, which included: Elementary Schools • AIMS results disaggregated by subject and grade • Measure of Academic Progress • Extended Writing Scores • AYP determination High Schools • AIMS results disaggregated by subject and grade • Graduation rate • Dropout rate • AYP determination It is important to note that districts and charter holders were solely responsible for verifying information for their schools. If a district or charter holder did not verify the information for its schools through the verification process, the ADE assumed the schools on file and the data available were correct and complete as listed. The ADE used the data and produced an achievement profile for the schools based on that information. Specifically, administrators were expected to: • Provide updated contact information. • Confirm that entities were in fact schools that were open during the years under review. Arizona’s Accountability System: Volume I Arizona Department of Education 14 • Confirm the grade configurations of the schools. • Confirm school type: e.g. Traditional, alternative, or K-2. • Notify ADE of any significant discrepancies in the data: e.g., the test scores of entire classes or grades miscoded to another school. The web site application allowed schools to submit contact information, and information regarding school type and grade configurations. If large-scale corrections were required, schools were asked to download their test score data from the web site and submit the corrected data to the Research and Policy section via its Achieve inbox (achieve@ade.az.gov). IMPORTANT NOTE: The criteria used to select AIMS scores for AZ LEARNS Achievement Profiles differ from the criteria used to select scores for Adequate Yearly Progress. The criteria also differ from the scores provided to schools by the testing contractor, the scores publicly reported by ADE, and the scores available through the ADE’s AIMS Wizard located at www.ade.az.gov/profile/publicview/. Complete verification of the data was a condition for being permitted to appeal a school’s Achievement Profile. That is, schools that did not verify their data were not allowed to appeal their Arizona LEARNS Achievement Profile designation. Timeline The timeline for AZ LEARNS Achievement Profile process was: • September 19, 2003. Opening of data verification process. • October 1, 2003. Closing of data verification process. • October 10, 2003. Preliminary release of AZ LEARNS Achievement Profiles. Opening of window for appeals submissions. • October 15, 2003. Public release of AZ LEARNS Achievement Profiles for schools not appealing profile. • October 20, 2003. Closing of appeals window. • October 21-November 12, 2003. Appeals review process. • November 15, 2003. Final release of AZ LEARNS Achievement Profiles for all schools. Arizona’s Accountability System: Volume I Arizona Department of Education 15 Calculation of AIMS Baseline and Growth Scale Points Overview The Arizona Instrument to Measure Standards (AIMS) is a criterion-referenced test used by the state of Arizona to measure student performance in the areas of math, reading and writing. In each subject area, students are grouped into performance categories based on how they performed relative to the state standard: ¾ Falls Far Below the Standard (FFB) ¾ Approaches the Standard (APP) ¾ Meets the Standard (Meet) ¾ Exceeds the Standard (EXC) The AIMS portion of the achievement profile carries an expectation that students will meet the state standards. For this reason, a year-to-year analysis of the percentage of students that fall far below the standard and meet or exceed the standard is used for determining a school classification. Due to variation in student performance from grade level to grade level, the expectations for improvement, cut-points, for these categories will change with each grade and subject combination. Identifying Baseline Groups For each subject/grade combination there are six baseline groups separated by five baseline cut points. Baseline groups are established using the average percent of students meeting the standard for 2000 and 2001, for all subjects and grades except high school mathematics. High school mathematics will utilize the percent of students meeting the standard for 2001 to calculate baseline scores. Arizona’s Accountability System: Volume I Arizona Department of Education 16 Equation Used to Calculate Baseline Scores The following equation was used to calculate a baseline for every subject/grade combination offered by a school. Baseline Points = # Students Passing AIMS 2000 + # Students Passing AIMS 2001 # Tested 2000 + # Tested 2001 The baseline was rounded to the nearest hundredth e.g. .675 = .68; .672 = .67. Data Used A student’s score was excluded from the baseline calculation if any of the following criteria were met: ¾ Student received no score on the test. ¾ Student was not English proficient. A student was considered not proficient if he/she indicated on the AIMS test sheet that he/she was an English learner (question 8) AND the student was classified as an English learner for three years or less (question 9). ¾ The student received a modification on the test which changed the standardization of the results. ¾ The student tested out of level. ¾ The student did not start the year at the school (Startyr = N). Calculation of the Baseline Group Cut Points Baseline cut points were calculated for each subject/grade combination using the distribution of calculated baselines for schools with a total number of students tested greater than or equal to 60—except for 10th grade mathematics where the threshold for inclusion was 30. These thresholds reflect the consistent application of the minimum group size of 30 used in all accountability calculations for both AZ LEARNS Achievement Profiles and NCLB. Thirty is the sample size conventionally considered large enough to provide statistically meaningful results. The baseline cut points were established at the 90th, 75th, 50th, 25th, and 10th percentiles of the beta distribution with parameters α and β, where α and β are estimated for each subject/grade combination using the following formulas: ⎡ ⎡ x (1 − x ) ⎤ ⎤ 2 ⎥⎦ − 1⎥ ⎣⎣ s ⎦ αˆ = x ⎢ ⎢ Arizona’s Accountability System: Volume I Arizona Department of Education 17 ⎡ ⎡ x (1 − x ) ⎤ ⎤ 2 ⎥⎦ − 1⎥ ⎣⎣ s ⎦ βˆ = (1 − x ) ⎢ ⎢ where x is the sample mean and s 2 is the biased sample variance. Using the beta distribution has two advantages. First, the distribution is bounded by zero and one. This approach is preferred when dealing with a proportion, since it will not generate negative values for baseline cut points. Second, the beta distribution allows for oddly shaped or skewed distributions of baselines. The baseline groups per subject/grade are given in table 5.1. Table 5.1. Baseline Groupings Grade Subject 3 Math Baseline Grouping 1 0% - 26% Baseline Grouping 2 27% - 40% Baseline Grouping 3 41% - 56% Baseline Grouping 4 57% - 71% Baseline Grouping 5 72% - 82% Baseline Grouping 6 83% - 100% 3 Reading 0% - 46% 47% - 59% 60% - 73% 74% - 84% 85% - 91% 92% - 100% 3 Writing 0% - 54% 55% - 67% 68% - 79% 80% - 89% 90% - 94% 95% - 100% 5 Math 0% - 11% 12% - 21% 22% - 36% 37% - 52% 53% - 66% 67% - 100% 5 Reading 0% - 31% 32% - 44% 45% - 60% 61% - 75% 76% - 85% 86% - 100% 5 Writing 0% - 25% 26% - 38% 39% - 53% 54% - 68% 69% - 79% 80% - 100% 8 Math 0% - 1% 2% - 5% 6% - 12% 13% - 22% 23% - 34% 35% - 100% 8 Reading 0% - 25% 26% - 37% 38% - 51% 52% - 66% 67% - 77% 78% - 100% 8 Writing 0% - 18% 19% - 28% 29% - 42% 43% - 56% 57% - 68% 69% - 100% 48% - 100% H.S. Math 0% - 3% 4% - 8% 9% - 19% 20% - 33% 34% - 47% H.S. Reading 0% - 28% 29% - 42% 43% - 58% 59% - 73% 74% - 83% 84% - 100% H.S. Writing 0% - 16% 17% - 25% 26% - 39% 40% - 53% 54% - 66% 67% - 100% Baseline Scale Values A school is awarded baseline scale points for each subject/grade combination it offers that meets the minimum N count. The number of points awarded for each subject grade is equal to that subject/grade’s baseline grouping. Example. In the baseline years, 66 percent of the students in Gila Monster Elementary passed math portion of the third grade AIMS. This value places the subject/grade in baseline grouping 4. Gila Monster Elem. has earned 4 baseline scale points for this particular subject/grade. Identifying Growth Groups Determining a school’s growth points for each subject/grade combination is based on student movement out of the Falls Far Below (FFB) category and student movement into the Meet/Exceeds (M/E) category. The value of the change points is determined by the difference between the school’s three-year average and the baseline percentages. A three-year average is calculated by adding the total number of students in each category Arizona’s Accountability System: Volume I Arizona Department of Education 18 over three years and dividing by the three-year total number of students tested for each subject/grade combination. The ADE considers a school to have made positive change if the three-year average percent of students that fall into the M/E category is higher than the baseline percentage or if the three-year average FFB is lower than the baseline. The ADE considers a school to have made negative change if the three-year average percentage of students in the M/E category is lower than the baseline or if the FFB category is higher than the baseline percentage. The ADE expects schools to increase the percentage of students that meet the standards over time, and decrease the percentage that falls far below the standards over time. Equations Used to Calculate Growth Points Growth points for a subject/grade are calculated in the following seven steps. Equation 1 Baseline FFB = # Students FFB 2000 + # Students FFB 2001 # Tested 2000 + # Tested 2001 Equation 2 Baseline M/E = # Students M/E 2000 + # Students M/E 2001 # Tested 2000 + # Tested 2001 Equation 3 3 - Yr. Avg. FFB = # Students FFB 2001 + # Students FFB 2002 + # Students FFB 2003 # Tested 2001 + # Tested 2002 + # Tested 2003 Equation 4 3 - Yr. Avg. M/E = # Students M/E 2001 + # Students M/E 2002 + # Students M/E 2003 # Tested 2001 + # Tested 2002 + # Tested 2003 Equation 5 Change FFB = 3 - Yr. Avg. FFB - Baseline FFB Equation 6 Change M/E = 3 - Yr. Avg. M/E - Baseline M/E Arizona’s Accountability System: Volume I Arizona Department of Education 19 Equation 7 Growth Points = Change M/E - Change FFB All values were rounded to the nearest ten-thousandth e.g. .67556 = .6756; .67221 = .6722. Data Used A student’s score was excluded from the growth point calculation if any of the following criteria were met: ¾ Student received no score on the test. ¾ Student was not English proficient. A student was considered not proficient if he/she indicated on the AIMS test sheet that he/she was an English learner (question 8) AND the student was classified as an English learner for three years or less (question 9). ¾ The student received a modification on the test. ¾ The student tested out of level. ¾ The student did not start the year at the school (Startyr = N). Example. The following example demonstrates how growth points are calculated. Table 3E.1 shows four years of AIMS scores for a single subject and grade for a hypothetical school. Table E3.1. Number of Students Year FFB A 2000 25 25 2001 20 30 2002 15 35 2003 10 35 M 25 25 30 30 E 25 25 30 35 Total 100 100 110 110 The following equations show the steps used to calculate the growth points given the test scores in the above table. Equation 1 Baseline FFB = 25 + 20 = .2250 100 + 100 Equation 2 Baseline M/E = Arizona’s Accountability System: Volume I 50 + 50 = .5000 100 + 100 Arizona Department of Education 20 Equation 3 3 - Yr. Avg. FFB = 20 + 15 + 10 = .1406 100 + 110 + 110 Equation 4 3 - Yr. Avg. M/E = 50 + 60 + 65 = .5469 100 + 110 + 110 Equation 5 Change FFB = .1406 - .2250 = - .0844 Equation 6 Change M/E = .5469 - .5000 = .0469 Equation 7 Growth Points = .0469 - (-.0844) = .1313 Calculation of the Growth Group Cut Points The growth group cut points were established at the mean and ±0.5 and ±1 standard deviations from the mean, assuming the distribution of growth points for each subject/grade combination are distributed normally. (See figure 5.1). This is a reasonable assumption given that the theoretical range of growth points (±200) is far greater than the actual sample. Figure 5.1. Determination of Growth Group Cut Points Growth Point Groupings 1 2 -1.0 SD 3 - .5 SD 4 X 5 .5 SD 6 1.0 SD This methodology yielded the following cut points for the six growth point groupings. Arizona’s Accountability System: Volume I Arizona Department of Education 21 Table 5.2. Growth Point Groupings Grade Subject Growth Point Grouping 1 3 Math <-1.75% 3 Reading <-5.41% 3 Writing <-9.23% 5 Math <-1.61% 5 Reading <-15.16% 5 Writing <-8.18% 8 Math <-7.99% 8 Reading <-5.86% 8 Writing <-10.24% H.S. Math <-5.81% H.S. Reading <-10.50% H.S. Writing <10.72% Growth Scale Values Growth Point Grouping 2 -1.74% 4.18% -5.40% -0.90% -9.22% -5.00% -1.60% 4.11% -15.15% -10.46% -8.17% -3.44% -7.98% -1.94% -5.85% -0.81% -10.23% -5.92% -5.80% -1.60% -10.49% - 6.10% 10.73% 15.32% Growth Point Grouping 3 4.19% 10.12% -0.89% 3.60% -4.99% -0.77% 4.12% 9.83% -10.45% -5.77% -3.43% 1.29% -1.93% 4.11% -0.80% 4.24% -5.91% -1.61% -1.59% 2.61% -6.09% -1.70% 15.33% 19.92% Growth Point Grouping 4 10.13% 16.05% 3.61% 8.11% -0.76% 3.46% 9.84% 15.56% -5.76% -1.07% 1.30% 6.02% 4.12% 10.17% 4.25% 9.29% -1.60% 2.70% 2.62% 6.83% -1.69% 2.71% 19.93% 24.52% Growth Point Grouping 5 16.06% 21.98% 8.12% 12.61% 3.47% 7.69% 15.57% 21.28% -1.06% 3.62% 6.03% 10.76% 10.18% 16.22% 9.30% 14.34% 2.71% 7.02% 6.84% 11.04% 2.72% 7.11% 24.53% 29.12% Growth Point Grouping 6 21.99% > 12.62% > 7.70% > 21.29% > 3.63% > 10.77% > 16.23% > 14.35% > 7.03% > 11.05% > 7.12% > 29.13% > A school is awarded growth scale points for each subject/grade combination it offers that meets the minimum N count. The number of points awarded for each subject grade is equal to that subject/grade’s growth grouping. Example. Gila Monster Elementary has obtained 5.15 growth points. This is equivalent to growth grouping 3. Gila Monster Elem. has earned 3 growth scale points in 3rd grade mathematics. Calculation of Subject/Grade Scale Points from AIMS In order to calculate a school’s scale classification value (prior to MAP/EWS) for elementary classifications, the baseline group scale values for each subject/grade combination are added to the growth point group scale values for each grade/subject combination. A 70 percent weight is given to the school’s strongest scale value (baseline group or growth point group) and a 30 percent weight to the other scale value. Table 5.3 shows the scale points earned per subject/grade for all combinations of baseline and growth group scale points. Example. In third grade mathematics, Gila Monster Elementary has earned 4 baseline group scale points and 3 growth groups scale points. Because it has earned more scale points for its baseline group, the baseline scale points are given a 70 percent weight and the growth group scale points are given a 30 percent weight. Thus, the total scale points earned for third grade math are (.7 X 4) + (.3 X 3) = 3.7. Arizona’s Accountability System: Volume I Arizona Department of Education 22 Example. In mathematics, Desert Mountain Vista High School has earned 2 baseline group scale points and 5 growth groups scale points. Because it has earned more scale points for its growth group, the growth scale points are given a 70 percent weight and the baseline group scale points are given a 30 percent weight. Thus, the total scale points earned for third grade math are (.3 X 2) + (.7 X 5) = 4.1. Minimum N-Size. If for any school’s subject/grade combination the total number of valid scores in the baseline years was less than 60/ 30 for high school math, that subject/grade combination was not included in the calculation of AIMS scale score points. The thresholds reflect the consistent application of the minimum group size of 30 used in all accountability calculations for both AZ LEARNS and NCLB. Thirty is the sample size conventionally considered large enough to provide statistically meaningful results. The total scale values for all subject/grade combinations for a school are summed and added to the scale score values for other performance measures. Table 5.3. AIMS Scale- Point Distributions by Baseline Grouping and Growth Point Grouping Growth Point Grouping 1 Growth Point Grouping 2 Growth Point Grouping 3 Growth Point Grouping 4 Growth Point Grouping 5 Growth Point Grouping 6 Baseline Grouping 1 1.0 1.7 2.4 3.1 3.8 4.5 Baseline Grouping 2 1.7 2.0 2.7 3.4 4.1 4.8 Baseline Grouping 3 2.4 2.7 3.0 3.7 4.4 5.1 Baseline Grouping 4 3.1 3.4 3.7 4.0 4.7 5.4 Baseline Grouping 5 3.8 4.1 4.4 4.7 5.0 5.7 Baseline Grouping 6 4.5 4.8 5.1 5.4 5.7 6.0 Special Cases: New Schools, and Missing Data If a school is missing AIMS test data for a subject/grade combination if offers, it receives zero scale points for that subject/grade for the achievement profile calculation for the test year in which the data are missing. In the subsequent years, the Arizona’s Accountability System: Volume I Arizona Department of Education 23 following rules approved by the State Board of Education are used for calculating baseline and growth scale points. 1. If a school has one year of missing data for the baseline years, its baseline will be calculated using the single year of baseline data it does have. 2. If a school has two years of missing data for the baseline years it receives zero baseline scale points. Growth is calculated from a base of zero, but growth scale points only receive 30 percent weight. Example Gila Monster Elementary is missing data for fifth grade writing for both 2000 and 2001. Its growth points for that subject/grade combination put it in growth group five. Thus it receives 5 X 0.3 = 1.5 AIMS scale points for fifth grade writing. 3. If a school is missing one year of data used in the threeyear rolling average to calculate growth points, its growth points are calculated using two years of data. 4. If a school is missing two years of data used in the threeyear rolling average to calculate growth points, it receives 0 growth scale points, and its baseline scale points are given a weight of 30 percent. Example Gila Monster Elementary is missing data for fifth grade reading for both 2002 and 2003. Its baseline for that subject/grade combination put it in baseline group two. Thus it receives 2 X 0.3 = 0.6 AIMS scale points for fifth grade writing. New Schools Once a school has been operational for three test administrations, the school will receive an achievement profile utilizing the updated formula changes passed by the Board during 2003, with the exception of a one-year baseline analysis rather than a two-year baseline analysis. The new school will be evaluated based on three years of data; a school must only have one overlapping year of test data to be evaluated for an achievement profile. Example. Desert Mountain Vista High School opened in 2001 and is still currently in operation. Its achievement profile would be calculated with the following methodology: Baseline grouping AIMS data (single year)- 2001 Growth point grouping AIMS data (three year average)- 2001, 2002, 2003. Arizona’s Accountability System: Volume I Arizona Department of Education 24 Added Evidence: Measure of Academic Progress and Extended Writing Overview The Added Evidence portion of an elementary profile relies on the Arizona Measure of Academic Progress (MAP), which uses longitudinal Stanford 9 test scores to provide a measure of student academic growth, and Extended Writing trait scores from the AIMS test. These two items work together in order to provide additional information about the magnitude of a school’s performance; they are used as “Added Evidence” in order to supplement an elementary school’s performance level. In a way, MAP and Extended Writing scores are used as “bonus points” and can only be used to attain a higher classification; a lack of these points cannot affect the profile outcome in a negative fashion. It is important to note, that only students with valid Stanford 9 scores will be included in the MAP analysis for the Achievement Profiles. Additionally, students who have not been instructed for three (3) academic years in English, as well as students testing under nonstandardized conditions are excluded from schools’ Extended Writing data. Measure of Academic Progress Unlike traditional measures of achievement, such as percentile ranks that mark achievement at one point in time, MAP measures growth over time. A measure of the progress made over a school year is obtained through linking individual student test scores from one year to the next. This progress is attributed to the school the student attended, if a student has remained in the same school for two academic years. Traditional indices of achievement, such as a comparison of schools’ percentile ranks calculated at a point in time, are highly correlated to student demographic variables. As a result, the same schools consistently score at the top and bottom of the percentile rank listings. With MAP as a measure of school effectiveness, schools traditionally seen as low performing, by way of a percentile rank, may show remarkable gain with the students they have had an opportunity to teach. Arizona’s Accountability System: Volume I Arizona Department of Education 25 Figure 6.1: Summary of MAP Characteristics MAP Key Features: ¾ Utilizes Stanford 9 stanine scores ¾ Captures individual student growth over time ¾ Accounts for mobility ¾ Includes only those students a school has had an opportunity to teach ¾ Provides meaningful information to teachers ¾ Focuses on all students ¾ Aligns to the Arizona Academic Standards in elementary reading and math Longitudinal Matching In order to determine the number of students that made One Year’s growth (OYG), individual student records were matched from test year to test year. The process of matching Stanford 9 student records between test years involve the use of four pieces of student level data. Perfect student matches were made if all of the following were true: ¾ The student attended the same school during two years of consecutive testing ¾ The student had the same first and last name ¾ The student’s gender was the same for both years ¾ And the birth date of the student was identical for both years. However, due to inconsistencies in some of the student information (e.g. if a student misspelled his or her name, or if a mistake was made on any other piece of necessary information) matching student records that were not picked up in a perfect match involved a multi-stage process. For example, if a student spelled his name “Frank Adams” in 2001 and misspelled it as “Grank Adams” in 2002, Frank’s records would not be matched in the initial pass. If all other pertinent pieces of information were identical, the second phase of the matching process would in all likelihood link his records. In addition to letter changes or letter transposition, the matching routine could also handle transposed birth dates. For example, if Frank Adams had also indicated that he was born 09/10/90 in 2001 and 90/10/09 in 2002 his record would likely be Arizona’s Accountability System: Volume I Arizona Department of Education 26 linked. Typically, close to 90% of student test scores can be matched between any two test years. The multistage approach to matching has limitations, however. If the student’s first or last name varied by more than two letters from one year to the next, the records cannot be matched. Also, students that do not provide their birth dates or indicate their gender in one of the years cannot be matched. In rare instances, false matches can occur between two different students in different test years. This is most likely to occur when students are related, attending the same schools in successive years and have similar first names, identical last names and are a year apart in age. Given the high proportion of test scores that are appropriately matched, the above stated limitations are of minor concern. Making One Year’s Growth MAP is used to measure individual student growth. Student Stanford 9 test scores are linked from one year to the next and growth on the test is calculated. One Year’s growth (OYG) is broadly defined as attaining the same level of absolute achievement from year to year, while learning more difficult material. OYG is determined by examining a student’s stanine score: if one attains the same stanine score or a higher stanine score relative to the previous year, that student has made one year’s growth. There are two exceptions to this approach (see table 6.1): ¾ Students who begin in stanine 9 and move to stanine 8 will make OYG ¾ Those that stay in stanine 1 from one year to the next will not make OYG For example, a student who begins at the 5th stanine (50th percentile) as a 3rd grader and maintains a 5th stanine score as a 4th grader has achieved OYG. This is the minimum growth that is expected for any student who remains at a school for an academic year. When all students achieve OYG, schools ensure that no students are falling behind from one school year to the next. Table 6.1. Determining One Year’s Growth Stanine 2001 5 6 9 5 1 Arizona’s Accountability System: Volume I Stanine 2002 5 7 8 4 1 Result OYG OYG OYG Did not Accomplish OYG Did not Accomplish OYG Arizona Department of Education 27 Extended Writing In addition to using MAP for Added Evidence, a writing score is determined using trait scores on the AIMS test. It was determined that the writing portion of the AIMS assessment could serve a dual purpose: 1. Be used to both provide Added Evidence for writing, since the MAP Added Evidence only applied to math and reading. 2. Provide an emphasis on the writing abilities of Arizona’s students. The AIMS Extended Writing (EW) test assesses students’ writing abilities based on the six traits rubric. A student receives a score from 0-6 on each of the six traits, for a score range from 0-36. The ADE determined that Added Evidence would be based on the number of students receiving a minimum total score of 24 (or an average of 4 on each trait) on the writing portion of the AIMS exam. Calculating Added Evidence Points The following methodology is used to calculate added evidence points. 1. The number of students making OYG and the number in the analysis is determined for the most recent three years for the whole school (reading and mathematics). 2. The number of students scoring 24 or more points on the EWS and the number included in the analysis is determined for the most recent three years. 3. The total number of students to be included in the added evidence points is the weighted sum of the number of students making OYG (reading and mathematics) and the number of students scoring 24 or more on the EWS. The weights are 75 percent for students making OYG and 25 percent for students scoring 24 or more on the EWS. Divide this weighted sum by the weighted sum of the total number included in the analysis for OYG (reading and mathematics) and EWS to determine the percent total added evidence. 4. The following grid is used to determine the points assigned by subject/grade combination (please see table 6.2): Arizona’s Accountability System: Volume I Arizona Department of Education 28 Table 6.2. Distribution of Elementary Added Evidence Points by Subject/Grade Combination Subject/Grade Combinations 4 5 6 7 % Total Added Evidence 1 2 3 8 9 90% + 3 5 8 10 12 15 17 20 22 80%-89% 2.25 3.75 6 7.5 9 11.25 12.75 15 16.5 70%-79% 1.5 2.5 4 5 6 7.5 8.5 10 11 60%-69% 0.75 1.25 2 2.5 3 3.75 4.25 5 5.5 Application of this methodology results in the following scale permutations for the elementary achievement profile (please see table 6.3): Table 6.3. Elementary School Scale Permutations Subject/Grade Combinations 1 2 3 4 5 6 7 8 9 Subject/Grade Total Points 6 12 18 24 30 36 42 48 54 Total Subject/Grade and AYP Points 7 13 19 25 31 37 43 49 55 AYP Total Points 1 1 1 1 1 1 1 1 1 Total Added Evidence Points Up to 3 Up to 5 Up to 8 Up to 10 Up to 12 Up to 15 Up to 17 Up to 20 Up to 22 Final Total Points 10 18 27 35 43 52 60 69 77 Example. The table below shows hypothetical data used to determine added evidence points for the 2003 achievement profiles. Table E6.1. Added Evidence Points Year 2001 2002 2003 Total MAP Math # Making OYG 100 110 105 315 MAP Read # Making OYG 105 100 100 305 Number in MAP Analysis 300 320 310 930 EWS Over 24 45 45 48 138 Number in EWS Analysis 50 52 50 152 The formula for calculating the percent total added evidence (PTAE) is: Arizona’s Accountability System: Volume I Arizona Department of Education 29 PTAE = 0.75 (Total students making OYG reading and math) + 0.25 (Total students with EWS ≥ 24) 0.75 (Total students in MAP analysis) + 0.25 (Total students in EWS analysis) For the numbers in the above table, the percent total added evidence would be: PTAE = 0.75 (315 + 305) + 0.25 (138) 0.75 (930) + 0.25 (152) = .68. Assuming the school is has a grade configuration of K-6 and meets the minimum n-count for evaluation for all subjects and grades, it would have six total subject/grade combinations. Reading from table 6.2, 68 percent total added evidence and six subject/grade combinations would earn the school 3.75 added evidence points. Arizona’s Accountability System: Volume I Arizona Department of Education 30 Graduation and Dropout Rates Overview The graduation and dropout rates are important complements to the high school model used in the revised achievement profiles. Graduation rates indicate the success of students in meeting course requirements and achieving passing grades in subject areas not covered by the AIMS test. Dropout rates are included in the high school model as a measure of student persistence and to ensure that AIMS results reflect the largest percentage of the student population possible. Graduation and dropout rates are used solely in the calculation of a high school profile. High school status was granted to any school that reported data in grade ten for each of the relevant school years (2000-2001, 2001-2002 and 2002-2003). In the calculation of a high school achievement profile, the two rates operate independently of one another; in other words, the point value outcome is a reflection of how the school performs in each of the categories. When one of the pieces of data is missing, a school cannot meet the target for that category. Some educational facilities are required to provide an annual dropout rate, but are not required to provide a graduation rate. Under these circumstances, solely the dropout rate is used to calculate this portion of a profile. Rounding The rounding of dropout and graduation rate percentages worked in such a way that the decimal was only taken out to four places, regardless of the value of the fifth. In other words, the percentages, expressed at this point as decimals, were divided by 1000, then rounded based on one decimal place, such that 0.5% was rounded up and 0.4% was rounded down. Lastly, the number was divided by 10 to produce a percentage and one decimal place. For example, if, in 2001, a school reported a total enrollment of 90 and a dropout count of 25, this translates into a baseline dropout rate of 0.27777778. To round this into a percent, the calculation divided this number by 1000, which equals 277.7, then rounded to 278, then divided by 10: 27.8%. The final percent is compared to the cutoff points in each category. Arizona’s Accountability System: Volume I Arizona Department of Education 31 Graduation Rates Background The Graduation Rate is a five-year, longitudinal measure of how many students graduate from high school. By examining a cohort of students who began high school at the same time, the graduation rate assesses how many students actually complete high school within a five-year period (see figure 7.1). Figure 7.1. Graduation Rate Calculation Number of Cohort members who graduated after five years Graduation Rate = Original Cohort Membership X 100 + Transfers - Transfers - Deceased In Out For more information on Arizona’s Graduation Rate methodology, please see the Graduation Rate Study published by the ADE. For graduation rate data, three years of data were used: 1. Graduation rate for the cohort class of 2000, which represents the baseline rate and is used as a reference point in order to measure increases from year to year -and2. A three-year average of the cohort classes of 2000, 2001 and 2002. The three-year average is calculated by taking the total number of combined five year graduates and dividing by the total number of students in the combined cohort classes (see figure 7.2). Figure 7.2. Calculating a Three Year Average for Graduation Rate 2 Year Average = Cut-points 2000 + 2001 + 2002 5 Year Graduates Combined number of students in three cohorts X 100 Cut-points for the graduation rate portion of the profile were determined by examining a distribution of school graduation rates in Arizona and comparing them to the state mean. After analyzing the state distribution of scores and the guidelines in the achievement profile legislation, the following rates were targeted as cut-point values for graduation rate: Arizona’s Accountability System: Volume I Arizona Department of Education 32 ¾ 73.5%: The state mean when school size is controlled for by averaging the rates of all schools in Arizona. ¾ 89.5%: As stated in A.R.S.§15-241, the highest cut-point for a 5-year graduation rate is at least 90%. Meeting the Target In order for a school to meet the target for their 5-year graduation rate achievements, incremental gains must be made from one year to the next. These gains are evaluated by comparing the threeyear average rate to the baseline rate (see table 7.1). For graduation rate data, due the inclusion of a five-year rate, current year data could not be used. A school can meet the target for graduation rate in multiple ways, depending upon the value of its three-year average rate: ¾ If the three-year average is 89.5% or greater, the target is automatically met. If the three-year average is less than 89.5%, the baseline rate is used as a reference point and incremental gains must be made from year to year. The gains required to meet the target were derived from the statewide distribution of graduation rates. ¾ If the baseline rate is greater than or equal to 73.5%, the difference between the three-year average and the baseline rate must be greater than or equal to 0.5%. ¾ If the baseline rate is less than 73.5%, the difference between the three-year average and the baseline rate must be greater than or equal to 1.5%. Dropout Rates Background The Dropout Rate is an annual measure of how many students drop out of a school during a 12-month reporting period. It is expressed as the proportion of students who dropped out during the year to the total number of students that enrolled in the school over the course of the year (see figure 7.3). The state of Arizona defines a dropout as a student who was enrolled in a school at any point during the year, was not enrolled at the end of the year and did not: ¾ Graduate or complete high school -or¾ Transfer to another qualified educational facility -or¾ Die Arizona’s Accountability System: Volume I Arizona Department of Education 33 Figure 7.3. Calculating an Annual Dropout Rate Dropout Rate Number of students who dropped out Number of students who were enrolled during the school year = X 100 For more information on Arizona’s dropout rate methodology, please see the Annual Dropout Rate Study published by the ADE. For dropout rate data, two values were used: 1. Dropout rate for the 2001 school year, which represents the baseline rate and was used as a reference point in order to measure increases from year to year -and2. A three-year average of the 2001, 2002 and 2003 school years. The three-year average is calculated by taking the total number of combined dropouts and dividing by the total number of combined students served (see figure 7.4). Figure 7.4. Calculating a Three-Year Average for Dropout Rate 3 Year Average Cut-points = Total number of students who dropped out in 3 years Total number of students who were enrolled during 3 years X 100 Cut-points for the drop out rate portion of the profile were determined by looking at a distribution of school dropout rates in Arizona and comparing them to the state mean. After analyzing the state distribution of scores and the guidelines in the legislation, the following rates were targeted as cut-point values for dropout rate: ¾ 9.4%: The state mean when school size is controlled for by averaging the rates of all schools in Arizona. ¾ 6.0%: As stated in A.R.S.§15-241, the lowest cut-point for dropout rate is less than 6% Meeting the Target In order for a school to meet the target for their annual dropout rate achievements, incremental decreases must be made from one year Arizona’s Accountability System: Volume I Arizona Department of Education 34 to the next. These decreases are evaluated by comparing the three -year average rate to the baseline rate (see table 7.1). A school can meet the target for dropout rate in multiple ways, depending upon what the three-year average rate is: ¾ If the three year average for the annual dropout rate is 6.0% or less, the target is automatically met. If the three year average is greater than 6.0%, the baseline rate is used as a reference point and incremental gains must be made from year to year. The gains required to meet the target were derived from the statewide distribution of dropout rates. ¾ If the baseline rate is less than or equal to 9.4%, the difference between the three year average and the baseline rate must be greater than or equal to 0.5%. ¾ If the baseline rate is greater than 9.4%, the difference between the three-year average and the baseline rate must be greater than or equal to 1.5%. Table 7.1. Target Improvements for Dropout and Graduation Rates in the High School Achievement Profile Baseline Dropout Rate* Target** Baseline Graduation Rate* Target** < or = 9.4% 0.5% Decrease > or = 73.5% 0.5% Increase > 9.4% 1.5% Decrease < 73.5% 1.5% Increase *Recall the baseline rates are taken from the 2000-2001 academic year **Meeting the target is met assessed by calculating the difference between the average rate and the baseline rate. Recall that, for the annual dropout rate, a three-year average is used and for the graduation rate, a two-year average is used. Point Value Outcomes A total of two points is awarded for the combination of dropout and graduation rate portion of an achievement profile. If both targets are met, two points are awarded; if one of the targets is met, one point is given. If neither of the targets is met, no points are awarded. The target could not be met if data were missing in a category. Table 7.2. Point Value Outcomes Point Value Graduation and Dropout Targets Met 2 Graduation or Dropout Target Met 1 Neither Graduation or Dropout Target Met 0 Arizona’s Accountability System: Volume I Arizona Department of Education 35 Evaluating the Total Scale Score Value to Determine a School Classification The tables below show the total number of scale score points that schools must earn in order to receive a given classification. A school may receive up to six AIMS scale points for each of its subject/grade combinations that are evaluated; up to 22 added evidence scale points if it is an elementary or middle school; and up to 2 graduation/drop out scale points if it is a high school. A school receives one scale point if it made adequate yearly progress (AYP). For each school, the applicable scale score thresholds for classification labels are calculated by multiplying the number of subject/grade combinations evaluated for AIMS scale score points at that school by the base classification cut points. The base classification cut points for elementary schools are given in the first column of table 8.1 and those for high schools are given in the first column of table 8.2. Table 8.1. Elementary School Classification Cut Points Subject/Grade Combination 1 Subject/Grade Combination 3 Subject/Grade Combination 6 Subject/Grade Combination 9 Underperforming <4 < 12 < 24 < 36 Performing 4 12 24 36 Highly Performing 4.6 13.8 27.6 41.4 Excelling 5.4 16.2 32.4 48.6 Arizona’s Accountability System: Volume I Arizona Department of Education 36 Table 8.2. High School Classification Cut Points Subject/Grade Combination 1 Subject/Grade Combination 3 Underperforming < 3.2 < 9.6 Performing 3.2 9.6 Highly Performing 5 15 Excelling 5.4 16.2 Example. Gila Monster Elementary, a K-6, has six subject/grade combinations (reading, math, and writing for grades 3 and 5) evaluated for AIMS scale score points. Thus, the scale score thresholds for Gila Monster Elem. are: Table E8.1. Scale Score Thresholds #Subject/Grade Combinations Base Cut Points ¾ Underperforming Scale Points <24.0 ¾ Performing 6 4.0 =24.0 ¾ Highly Performing 6 4.6 =27.6 ¾ Excelling 6 5.4 =32.4 Example. Kangaroo Rat Elementary, a K-6, school only has five subject/grade combinations evaluated. One subject/grade combination failed to meet the minimum 30 valid score count for the baseline years. Thus, the scale score thresholds for Kangaroo Rat Elem. are: Arizona’s Accountability System: Volume I Arizona Department of Education 37 Table E8.2. Scale Score Thresholds #Subject/ Grade Combinations Base Cut Points ¾ Underperforming <20.0 ¾ Performing 5 ¾ Highly Performing 5 ¾ Excelling 5 Arizona’s Accountability System: Volume I Scale Points 4.0 4.6 5.4 =20.0 =23.0 =27.0 Arizona Department of Education 38 Application of Threshold Criteria for Excelling and Highly Performing Schools To ensure continued focus on improving the academic achievement of all students as they reach their absolute levels of attainment, including those students currently demonstrating proficiency in Arizona’s academic standards on AIMS, threshold criteria are applied to determine Excelling and Highly Performing schools. These threshold criteria are based on the three-year average percentage of students in the “Exceeds the Standard” category on AIMS (reading, writing, and mathematics) in a particular school. Conceptually, these threshold criteria serve as parameters to establish distinct boundaries around the Excelling and Highly Performing achievement profile classifications. Schools must not only receive a total scale value that places them into either Excelling or Highly Performing, but must also meet the requisite percentage of students in the “Exceeds the Standard” category on AIMS to be designated as either an Excelling or Highly Performing schools. The application of threshold criteria for Excelling and Highly Performing schools results in the following scenarios. (Please see figure 9.1): 1. A school that receives a total scale value that places it in the Excelling classification and meets the requisite percentage of students in the “Exceeds the Standard” category on AIMS necessary for an Excelling classification will be designated an Excelling school. 2. A school that receives a total scale value that places it in the Excelling classification and did not meet the requisite percentage of students in the “Exceeds the Standard” category on AIMS necessary for a Excelling classification, but did met the requisite percentage of students in the “Exceeds the Standard” category on AIMS necessary for a Highly Performing classification will be designated as a Highly Performing school. 3. A school that receives a total scale value that places it in the Excelling classification and did not meet either the requisite percentage of students in the “Exceeds the Standard” category on AIMS necessary for the Excelling Arizona’s Accountability System: Volume I Arizona Department of Education 39 classification or the Highly Performing classification will be designated as a Performing school. 4. A school that receives a total scale value that places it in the Highly Performing classification and meets the requisite percentage of students in the “Exceeds the Standard” category on AIMS necessary for an Excelling classification will be designated as a Highly Performing school. 5. A school that receives a total scale value that places it in the Highly Performing classification and meets the requisite percentage of students in the “Exceeds the Standard” category on AIMS necessary for a Highly Performing classification will be designated as a Highly Performing school. 6. A school that receives a total scale value that places it in the Highly Performing classification and did not meet either the requisite percentage of students in the “Exceeds the Standard” category on AIMS necessary for an Excelling classification or Highly Performing classification will be designated a Performing school. 7. A school that receives a total scale value that places it in the Performing classification will be designated as a Performing school, regardless if the school meets the requisite percentage of students in the “Exceeds the Standard” category on AIMS necessary for an Excelling classification or Highly Performing classification. 8. A school that receives a total scale value that places it in the Underperforming classification will be designated as an Underperforming school, regardless if the school meets the requisite percentage of students in the “Exceeds the Standard” category on AIMS necessary for an Excelling classification or a Highly Performing classification. Arizona’s Accountability System: Volume I Arizona Department of Education 40 Figure 9.1. Chart of Potential Scenarios Resulting from Threshold Marks Set for Excelling and Highly Performing Achievement Profile Classifications Excelling School receives total scale value placing it in the Excelling classification and meets the requisite % of students in “Exceeds the Standard” category on AIMS for the Excelling classification. Highly Performing Highly Performing School receives total scale value placing it in the Excelling classification and meets the requisite % of students in the “Exceeds the Standard” category on AIMS for Highly Performing classification. School receives total scale value placing it in the Highly Performing classification and meets the requisite % of students in the “Exceeds the Standard” category on AIMS for Highly Performing classification. Performing Performing School receives total scale value placing it in the Excelling classification but does not meet the requisite % of students for the Excelling or Highly Performing classification. School receives total scale value placing it in the Highly Performing classification but does not meet the requisite % of students for the Highly Performing classification. Performing School receives total scale value placing it in the Performing classification. Underperforming School receives total scale value placing it in the Underperforming classification. Utilizing a three-year average of the percentage of students in the “Exceeds the Standard” category on AIMS the ADE set the thresholds for Excelling and Highly Performing achievement assessed at a particular school. In order to establish thresholds for excelling and highly performing schools, the ADE rank ordered all schools by the three-year average percentage of students exceeding the standard on AIMS. Then, the threshold for Highly Performing was set at the 75th percentile rank of schools with students exceeding the standard; the threshold for Excelling was set at the Arizona’s Accountability System: Volume I Arizona Department of Education 41 90th percentile rank of schools with students exceeding the standard. (Please see table 9.1). Table 9.1. Excelling and Highly Performing Threshold Values by Grades Offered Subject Grade Combinations 3 Highly Performing 22.6% School Type (Serving grades) 3 or 5 Excelling 28.7% 3 8 6.5% 10.7% 6 3 and 5 30.9% 38.8% 6 5 and 8 19.9% 31.7% 9 3, 5 and 8 19.3% 25.7% 3 High School 9.3% 12.7% Example. The following table shows distribution of AIMS scores for Gila Monster Elementary. The numbers are for all grades in Gila Monster Elem. for which the AIMS is administered. Table E9.1. Number of Students Exceeding the Standard—All Grades Subject Year Reading # Exceeding #Tested Mathematics # Exceeding #Tested Writing # Exceeding #Tested 2001 25 100 24 100 23 100 2002 24 105 23 105 22 105 2003 26 99 25 99 24 99 Total 75 304 72 304 69 304 The percent of students exceeding the standard is then: Percent Exceeding = 75 + 72 + 69 304 + 304 + 304 = 23.6%. Since Gila Monster serves both grades 3 and 5, it must meet thresholds of 30.9 percent of students exceeding the standard to be classified as highly performing, and 38.8 percent to be classified as excelling. Gila Monster Elem. meets neither threshold. Arizona’s Accountability System: Volume I Arizona Department of Education 42 AZ LEARNS Achievement Profile Appeals Process The cornerstone of Arizona’s school accountability system (AZ LEARNS) is the achievement profile. The achievement profile is used to designate all public schools as Excelling, Highly Performing, Performing, Underperforming, or Failing. In accordance with A.R.S. §15-241, beginning with the October 15, 2003 achievement profile, school principals/administrators were allowed the opportunity to appeal an achievement profile classification on behalf of the school(s) for which they were responsible. Step 1: Data Verification/Data Appeal The first step in completing the AZ LEARNS Achievement Profile Appeals Process required all schools to review and verify all data in order to confirm accuracy. The data verification took place utilizing the AZLEARNS/Adequate Yearly Progress (NCLB) Application through the Common Logon located at the ADE’s Web site. Data verification included two distinct types of data, both of which had to have been verified by local administrators: school information and data information. School information All schools in the district or under a charter holder that expected to receive an achievement profile were required to verify school information, which included: ¾ School information ¾ The grades offered at each school during the years evaluated. (Schools open 1999-2000 through 2002-2003 academic years will be evaluated using the traditional achievement profile formula. Schools open 2000-2001 through 2002-2003 academic years will be evaluated using the “New Schools” achievement profile formula based on three years of data. Schools open after the summer of 2001 did not receive an achievement profile in 20032004, but needed to verify data. ¾ New school (open after the summer of 2000) ¾ Updated contact information Arizona’s Accountability System: Volume I Arizona Department of Education 43 Data Information All schools in districts or under a charter holder that expected to receive an achievement profile were required to verify data information, which included: Elementary Schools ¾ AIMS results ¾ Measure of Academic Progress (MAP) ¾ Extended writing scores from AIMS ¾ Adequate Yearly Progress (AYP) High Schools ¾ AIMS results ¾ Graduation rate ¾ Dropout Rate ¾ Adequate Yearly Progress (AYP) It is important to note that districts and charter holders were solely responsible for verifying information for their schools. If a district or charter holder did not verify the information for its schools through the verification process, the ADE assumed the schools on file and the data available were correct and complete as listed and produced an achievement profile for these schools based on that information. Step 2: Appeal Application School principals/administrators choosing to appeal an achievement profile must have completed the appeal application, which was accessible via the Common Logon during the specified appeal timeframe in order to indicate the exact issue(s) of appeal. Appeals were only accepted through the website application. Appeals sent to ADE via email, fax, or mail/delivery were not accepted. Schools were able to appeal achievement profiles in two categories: data (statistical) and non-data (substantive) appeals schools were not limited to one category and were able to appeal in both areas if necessary. Statistical Appeals Substantive Appeals Schools that appealed based on statistical arguments could have argued that their data were inaccurate. This could include (but was not limited to) data that were missing, miscoded, or invalid. Any of the data verified, as listed above, could be included in a statistical argument. Schools that appealed based on substantive arguments could have argued that mitigating circumstances, outside of the school’s Arizona’s Accountability System: Volume I Arizona Department of Education 44 control, negatively impacted the quantity or quality of test data in any of the years in which data were collected. This included circumstances that affected test conditions, test scores, and performance levels. Important Notes for the Appeal Process School principals/administrators that chose to appeal an achievement profile must have clearly articulated the issue(s) they believed merited an appeal through the appeal application. School principals/administrators must have submitted evidence that the issue(s) they believed merited an appeal directly resulted in a significant decrease in student academic achievement as demonstrated on AIMS or SAT-9 and/or an impact on other indicators used in the formula (elementary: MAP score, high school: graduation rate). The evidence must have been submitted to ADE at the time the appeal was submitted. Failure to provide this evidence resulted in the appeal not being granted. Evidence that was submitted after the appeal deadline closed was not considered. Once appeals were submitted through the Common Logon, the school/district/charter holder received an email verifying that the appeal had been received. The ADE, if necessary, requested that a school principal/administrator provide additional information/evidence to assist in the appeals process. Only those requests for additional information that were provided during the specific timeframe were included in the appeals process. Requests submitted after the specified timeframe were excluded from the appeals process. Unsolicited additional information submitted after the appeal deadline was not accepted. Step 3: Appeal Resolution After all appeals were submitted and the appeal window closed, the ADE began to process the appeals. Appeals were addressed categorically, not necessarily in the order received so the fact that a school submitted its appeal during the first day of the appeal window did not mean it would necessarily receive a decision first during the resolution process. The appeal process was implemented in three stages. Stage 1 – Statistical Appeals Processed All statistical appeals needed to be supported with compelling evidence. While some statistical arguments seemed compelling and may even have seemed obvious to the school submitting the appeal, evidence needed to be provided to support all appeals. For example, if the school was disputing the number of test scores used in the analysis, because some scores were excluded due to coding errors, etc., the school had to be explicit as to how the number in the analysis was not accurate. Simply stating, “number of students ADE used in the analysis doesn’t match district’s count” was not Arizona’s Accountability System: Volume I Arizona Department of Education 45 compelling; ADE needed to know why the numbers were different; meaning that particular students needed to be identified as miscoded or missing. Note: Statistical appeals that appealed the formula used to calculate the achievement profile were not accepted/reviewed. Additional Note: Some schools, when providing information in the appeals mentioned specific details about students such as names, id#, ethnicity, and specific student record information which violated guidelines set forth by FERPA. Schools were strongly encouraged to follow the FERPA guidelines in the future. When referring to students in appeals, identifying student information such as name, id#, etc. is not to be submitted with the appeal. Instead, students are to be referred to as student #1, student #2, etc. Stage 2- Substantive Appeals Processed Substantive appeals were resolved in a committee process. All committee members were nominated and represented a diverse background to ensure that appeals were considered from multiple perspectives. Among those perspectives were those of principals, teachers, school administrators, department administrators, researchers, and Title I representatives. Once the committee was assembled, the appeals were evaluated utilizing an appeals rubric approved by the State Board of Education that evaluated the argument presented and whether or not the evidence provided to support the argument was compelling. The appeals rubric consisted of a three-tiered system for appeal evaluation: 1) Initial review of the appeal to determine its merit. 2) Review of the evidence provided. 3) Committee recommendation. (Appendix VI: Substantive Appeal Rubric for AZ LEARNS) Initial Review The substantive appeal rubric provided for three categories that would apply during the initial review. Each appeal was classified into the categories based on the information provided in the appeals. 1. Data Calculation Discrepancies. Appeals of this nature occurred when schools attempted to compare data details provided in the common logon with their data sets and obtained different results. If in the appeal, the school/district provided information of data discrepancies and those data discrepancies were actual and not a result of the school or district’s inability to replicate the achievement profiles formula, the appeal was deemed as passing the initial review. At this stage, it was important for the school or district to be specific in their claim of differences so that the ADE could adequately determine if the data sets were in fact different. Arizona’s Accountability System: Volume I Arizona Department of Education 46 2. Special Circumstances outside the school’s/district’s control. Appeals of this nature occurred when the school indicated significant issues that impacted test scores, administration, etc. such as teacher attrition; environmental issues/events; adverse testing conditions; school/community/emergency/crisis; etc. If a school provided information dealing with a significant event that impacted test scores, which was clearly outside the school’s control the appeal would have been deemed as passing the initial review. 3. Policy/Methodology Issues. Appeals in this category were disagreeing with the way in which the AZ LEARNS Achievement Profile formula was used. Appeals of this nature were not accepted/reviewed beyond the initial review. Review of Evidence Once the appeals had made it through the first tier of the rubric, initial review, the evidence provided to support the appeal was evaluated. In this tier, three determinations were possible: 1. Compelling evidence: in this area, the school or district adequately provided information that led the committee to conclude that had the circumstance been different, the achievement profile would have been different as well. For example, if a school presented an argument that their data was different and was able to provide specific and relevant data supporting their claim, they were deemed as providing compelling evidence. If a school had a special circumstance that affected a certain grade and were able to demonstrate that the specific grades test scores suffered, they were deemed as providing compelling evidence. 2. Non-compelling evidence: appeals would be categorized in this area when they were able to provide information that a significant issue that could have impacted the school’s performance but they did not provided detailed, specific information as to specific outcomes that hindered the school’s performance. For example, if a school had high teacher attrition mentioned as a special circumstance, the committee would have allowed that appeal during the initial review. However, during the evidence stage, the committee would need to know when the teachers left the school, what grades were impacted, and if test scores suffered in that grade (as compared to other grades or prior years). If no specific information was presented, other than there was attrition, that evidence would have been deemed as not compelling. Another example of this type of Arizona’s Accountability System: Volume I Arizona Department of Education 47 outcome is in the case of where a school mentioned that they had high teacher attrition in certain grades during a specific year. However, when researching the test scores, it was found that the grade in question had higher scores in that year than other grades/classes and other test years despite the attrition. In this case, the evidence would not be compelling. 3. Not applicable evidence: if an appeal was submitted, made it through the initial review, and had evidence that was presented that was not linked in any way to the performance of the school, the evidence was deemed not applicable. For example, some appeals had described data discrepancies as a problem yet supported their data with evidence that argued the AZ LEARNS Achievement Profile methodology (which is not eligible for appeal). In cases such as this, the evidence was deemed not applicable. If the evidence did not directly support the claim made in the appeal, it was deemed not applicable. Committee Recommendation Once the appeal and evidence was reviewed, the committee came to a decision as to the outcome of the appeal. There were three possible outcomes: 1. Appeal granted and AZ LEARNS Achievement Profile changed. In these cases, the appeals successfully made it through the initial review and evidentiary stages and it was determined that the points needed to change classifications would have been earned by the school had the special circumstance/data discrepancy not occurred. Therefore, the classification for the school was changed. 2. Appeal granted and AZ LEARNS Achievement Profile remained the same. In these cases, the appeals successfully made it through the initial review and evidentiary stages. However, it was determined that the criteria needed to change classifications were not earned by the school had the special circumstance/data discrepancy not occurred. For example, a few schools provided information and evidence that their AYP points were not accurate and the committee felt the school provided information to prove they earned the 1-point set aside for AYP in the AZ LEARNS Achievement Profile formula. However, the schools were appealing to change from performing to highly performing schools and the one point difference did not impact their 3 year average for percent excelling which would have needed to change in order for Arizona’s Accountability System: Volume I Arizona Department of Education 48 their designation to change to highly performing (or excelling) so their determination remained the same. 3. Appeal Denied. In these cases, the appeals did not successfully make it through the initial review and evidentiary stages. Therefore, the classification for the school remained the same. Appeal Resolution Notes If the school/charter submitted both a statistical and substantive appeal, the statistical appeal was evaluated first. Only after the statistical arguments had been exhausted was the appeal sent to the substantive committee for evaluation. Schools needed to be certain to provide all information/support when submitting the appeal; late information to support the appeal was not accepted (unless ADE specifically asked for additional information as noted above). Stage 3 – Notification of Result Sent to Schools Once all appeals were resolved, notifications went to the schools that filed appeals. The contact person of record for the school received an email from Achieve with directions as to how to access appeal information via the Common Logon when the appeal had been processed. Schools were notified before the final public release of the achievement profiles as to the outcome of the appeal process. All appeals were final. Arizona’s Accountability System: Volume I Arizona Department of Education 49 Small School Score Adjustment Criteria Some schools are eligible for low score outlier removal as a result of small overall size. The following criteria are used to determine which schools should receive this adjustment: The school has an average daily membership (ADM) of ninetynine students or fewer and the school has AIMS scores for ninetynine students or fewer. Or, if the school did not report ADM but has valid AIMS scores for ninety-nine students or fewer. Method Small school adjustments were made to schools that met these criteria. The following steps were taken in adjusting the scores of qualified schools: 1. Low and high score outlier identification was conducted in each of the subjects of reading, writing and math for grades 3, 5, 8 and high school respectively. A decision was reached to remove low scores only. It was reasoned that statistically low outlier scores represented chance occurrences of poor individual level test performance. Removing low outlier scores from a distribution of test scores was particularly important for schools with small size because it negated the impact that a very low score(s) might have on the overall picture of the school’s performance. These scores were identified by employing a bivariate regression model where each school’s mean scale score was regressed on a two-category indicator for the school: Where 1 = the student was tested in that school, and 0 = the student was not tested in the school. 2. A unique equation was estimated for each school in order to produce a statistic that would address the question about the affect that any one student’s score within that school would have on the small school’s overall mean scale score. This equation took the following form: Arizona’s Accountability System: Volume I Arizona Department of Education 50 Mean scale scorei = α + b School Dummyi In this application, the regression coefficient plus or minus the intercept represented the school’s mean scale score. 3. The DFFIT statistic was computed for each score within the school and saved to a separate data file for later analysis. The DFFIT statistic assesses the change that a particular case will have on the predicted value i.e., the small school’s mean scale score when it has been deleted from the regression equation (Belsey et al. 1980)1. The following equation was used: hi 1 − hi DFFITSi = ti where ti = studentized residual and hi = leverage value 4. Once the DFFITS were estimated for each case, the standard deviation of the statistic was computed within each school. A cut-point was set such that any student with a DFFIT score greater or less than two standard deviations above or below the within school mean DFFIT could be considered an outlier relative to the distribution of test scores of that school. 5. Students were assigned an identification number so that their observation could be flagged if it fell above or below this school specific cut-point. Scores that had been identified statistically as outliers and were at the low end of the school’s test score distribution were eliminated from the datasets that were used to compute the final performance label. 1 Belsley, D.A., Kuh, E. and Welsch, R.E. (1980). Regression Diagnostics: Identifying Influential Data and Sources of Collinearity. New York: John Wiley. Arizona’s Accountability System: Volume I Arizona Department of Education 51 Exempted Schools Overview For the 2002-2003 achievement profiles, certain types of schools were exempted from receiving a profile. Due to a wide variety of schools with vastly different characteristics in Arizona, modified formulae will be created in order to evaluate them appropriately. There are five distinct categories of schools that did not receive profiles in the 2002-2003 school year: ¾ Extremely Small Schools ¾ Alternative Schools ¾ Accommodation Schools ¾ New Schools ¾ K-2 Schools Extremely Small Schools The criteria used to classify schools as extremely small in size for the purpose of computing an achievement profile via Arizona’s accountability system differed from the process used to eliminate low test score outliers. Although all schools classified as small in the “low score adjustment” process were eligible for low score removal, there was a subset of schools where the Average Daily Membership (ADM) and the number of students tested were too small to be classified using the standard methodology. Extremely small schools cannot be evaluated using the existing model of performance categories and change scores because of the small number of students tested. Given the architecture of the present accountability system there is a direct relationship between the number of students tested in a school and the overall reliability of the school achievement profile. In some cases, particularly where the number of students tested was less than or equal to 50, the movement of one, two or three students from one performance category to another directly affected the number of change points a school was awarded. Due to the fact that the smallest schools in the state were most likely to be effected by the conventional computation of their achievement profile, the State Board of Education determined that extremely small schools would not receive an achievement profile based on the traditional formula. A discussion of the criteria that were used to determine which schools were small enough to be excluded follows. Arizona’s Accountability System: Volume I Arizona Department of Education 52 A series of steps were taken to determine which small schools could be classified as extremely small. 1. First, a school was examined to determine how many grade/subject combinations it had for each test year. For example, grade school X may have had the following grade/subject combinations for each AIMS test year. Table E12.1. An Extremely Small Schoola AIMS test year 99-00 Grade 3 Grade 5 ADM b AIMS test year 00-01 AIMS test year 01-02 AIMS test year 02-03 Math (25) Math (31) Math (33) Math (31) Reading (24) Reading (30) Reading (34) Reading (32) Writing (25) Writing (29) Writing (34) Writing (32) Math (29) Math (31) Math (35) Math (31) Reading (31) Reading (31) Reading (35) Reading (28) Writing (30) Writing (31) Writing (33) Writing (29) 88 89 95 98 a The number of students tested are shown in parentheses. Average daily membership (ADM) for the school is shown in the last row. ADM and the number of students tested in the school were rarely equivalent. b The AIMS scores for the math test given during the 1999-2000 school year proved to be incompatible with the 20002001 and 2001-2002 versions of the high school math test. Hence, these scores could not be used. 2. School data were examined to determine if the school had an ADM of less than 100 students enrolled in any of the four tests years. If the school met this minimum condition the school data was referenced again to see if there were less than 60 valid scores in the 2000 and 2001 baseline years combined. This process was iteratively executed for each grade and each subject within the specific grade. If both criteria were met in the grade and subject area the school received an “S” (for very small) in the respective cell. Given the numbers in the table shown above, the conversion of table E12.1 would appear as table E12.2. Table E12.2. An Extremely Small School AIMS test year 99-00 Grade 3 Grade 5 ADM b AIMS test year 00-01 AIMS test year 01-02 AIMS test year 02-03 Math (S) Math (S) Math (33) Math (31) Reading (S) Reading (S) Reading (34) Reading (32) Writing (S) Writing (S) Writing (34) Writing (32) Math (29) Math (31) Math (35) Math (31) Reading (31) Reading (31) Reading (35) Reading (28) Writing (30) Writing (31) Writing (33) Writing (29) 88 Arizona’s Accountability System: Volume I 89 95 98 Arizona Department of Education 53 3. The final step to determine if the school would be considered extremely small was based on similar results shown in table E12.2. It was decided that if over 1/3 or 33% of all the grade/subject combinations were identified as “S” then the whole school would be considered too small to accurately compute an achievement profile. This decision was made based on the three (3) subject/grade value scale which resulted in allowing one of the three values to be identified as “S” and still produce an achievement profile, while two of the three values being identified as “S” would not produce a profile. This logic was then carried out into each of the other subject/grade value scales and ultimately produced the greater than 1/3 rule for an “extremely small” school classification. Such schools meeting this requirement would be identified as extremely small. According to this criterion, the school in the example above would be considered too small to receive an achievement profile because 3 of 6 subject/grade combinations received an “S” - 3/6 = .5 or 50 percent, which is greater than 1/3. Note that this is despite the school having 30 or more students in all relevant subject/grade combinations in more recent years. On August 25, 2003. the state board of education approved a policy for evaluation of extremely small schools. Achievement profiles for these schools will be based on a comprehensive site visit. The ADE will develop an evaluation rubric for extremely small schools, develop the capacity within its research and policy section to evaluate each extremely small school, and provide final AZ LEARNS Achievement Profile. The ADE estimates that extremely small schools will receive an AZ LEARNS achievement profile for the first time in 2004. Alternative Schools A public school desiring identification as an alternative school must apply to the Arizona State Board of Education for such status. These schools must be separate entities according to Arizona school finance provisions (funded as a school, reported as a school, etc.). Alternative school status was not granted to a program within a school. Following the passage of A.R.S.§15-241 the Arizona Department of Education (ADE) established a committee of educators with expertise in the field of alternative education to develop the elemental criteria used to define an alternative school. The ADE took great care in soliciting the participation of educators representing a wide variety of schools, Local Education Agencies (LEAs), and communities. Based on the committee’s recommendations, an alternative school is a school that the Arizona’s Accountability System: Volume I Arizona Department of Education 54 Arizona State Board of Education determined met all of the following criteria: 1. A school operated by a school district must have adopted a mission statement that clearly identifies its purpose and intent to serve a specific student population (please see criterion #3) that will benefit from an alternative school setting. A charter school must be expressly chartered to serve a specific student population that will benefit from an alternative school setting. Note: The school’s mission statement or charter must be communicated to the public. 2. The educational program and related student services of the school must match the mission or charter of the school. 3. The school must intend to serve students exclusively in one or more of the following categories: ¾ Students with behavioral issues (documented history of disruptive behavior) ¾ Students identified as dropouts ¾ Students in poor academic standing who are either severely behind on academic credits (more than one year) or have a demonstrated pattern of failing grades ¾ Pregnant and/or parenting students ¾ Adjudicated youth 4. Any school offering secondary instruction for academic credit used to fulfill Arizona State Board of Education graduation requirements (in part or in full) must offer a diploma of high school graduation. Also, no public school district may have more than ten percent (10%) of their total student population attending an alternative school or any combination of alternative schools served by the district at one time. Smaller districts, if they wish, may participate in the development of a “consortium” alternative school. Alternative school status was obtained through an application process that was open in the summer and fall of 2002. No additional schools were designated as alternative schools for the 2003 achievement profiles. Alternative schools did not receive an achievement profile in 2002 or 2003. In future years, alternative schools will receive profiles based on a modified formula, as a result of the unique student composition attending these schools. Arizona’s Accountability System: Volume I Arizona Department of Education 55 Accommodation Schools The ADE met with the county superintendents’ offices in 2002 in order to determine the policy for calculating achievement profiles for accommodation schools. These schools are, by statue, regulated and maintained by the county superintendents. The ADE along with this governing body determined that accommodation schools could be placed into three distinct categories: ¾ Similar to traditional schools ¾ Alternative schools and ¾ Detention centers Accommodation schools that could be identified as similar to traditional schools were given an achievement profile based on the standard methodology outlined for public schools. Schools that could be identified for alternative school status were granted this status and will receive an achievement profile as outlined in the methodology for alternative schools. Lastly, schools identified as detention centers were determined to not require an achievement profile, therefore these particular accommodation schools will not have an achievement profile calculated at this time. New Schools A certain percentage of schools currently in operation were not in operation for three consecutive academic years in order to provide all data necessary to calculate an achievement profile. Schools that do not have three years of data due to their recent opening will not receive a profile until they are in operation for three consecutive years. Their initial year of operation will serve as the baseline year for these schools. For example, if a school’s first year in operation was the 20012002 academic year, the data reported for this year will function as the baseline data for that school. A profile will not be calculated until it is in operation for three consecutive years, for example, in the fall of 2004. K-2 Schools Schools exclusively serving the kindergarten to second grade student populations will not receive an achievement profile at this time, but will in future years. These grades are not currently assessed via our state assessment (AIMS) and the ADE cannot compute a measure of academic progress (MAP) for these schools, both of which are key components to the elementary model. On August 25, 2003 the state board of education adopted a policy for the evaluation of K-2 schools. K-2 schools will receive an achievement profile based on a comprehensive site visit. The ADE Arizona’s Accountability System: Volume I Arizona Department of Education 56 will develop an evaluation rubric for K-2 schools; develop capacity within it research and policy section to evaluate each K-2 school; and provide final AZ LEARNS Achievement Profile. The ADE estimates that K-2 schools will receive an AZ LEARNS Achievement Profile for the first time in 2004. Arizona’s Accountability System: Volume I Arizona Department of Education 57 Appendices APPENDIX I: ARIZONA STATE BOARD OF EDUCATION INFORMATION PACKET FOR APRIL 28, 2003 ................. 59 APPENDIX II: ARIZONA STATE BOARD OF EDUCATION INFORMATION PACKET FOR MAY 19, 2003 ....................................................................................................... 67 APPENDIX III: ARIZONA STATE BOARD OF EDUCATION INFORMATION PACKET FOR JUNE 30, 2003............................................................................................... 69 APPENDIX IV: ARIZONA STATE BOARD OF EDUCATION INFORMATION PACKET FOR AUGUST 25, 2003............................................................................................... 74 APPENDIX V: ARIZONA STATE BOARD OF EDUCATION INFORMATION PACKET FOR SEPTEMBER 16, 2003 ..... 80 APPENDIX VI: SUBSTANTIVE APPEAL RUBRIC FOR AZ LEARNS ...................................................................................... 96 Arizona’s Accountability System: Volume I Arizona Department of Education 58 Appendix I: Arizona State Board of Education Information Packet for April 28, 2003 On April 28, 2003, the Arizona State Board of Education (Board) must adopt modifications made to the Achievement Profile methodology. This methodology will be applied to Arizona public schools, including charter schools, in order to determine school classifications by October 15, 2003 as required in A.R.S. §15-241 (Arizona LEARNS). The purpose of this document is to inform the Board of all necessary decisions required to adopt the modifications made to the Achievement Profile. This document includes an overview of the general process to produce the Achievement Profiles, a summary of the actions before the Board, specific numeric values associated with those actions, and the administrative policies necessary to implement the Achievement Profiles. As mandated by A.R.S. §15-241, the Achievement Profile was developed according to a research-based methodology by the Arizona Department of Education (ADE) and members of the education community. All modifications to the Achievement Profile follow this principle. Upon adoption by the Board, the ADE will produce a technical report detailing the Achievement Profile methodology, including specific formulas and supporting documentation. I. GENERAL PROCESS TO PRODUCE THE ACHIEVEMENT PROFILES According to A.R.S. §15-241, the Achievement Profile is utilized to determine a public school classification. The general process to calculate the Achievement Profile for each school is as follows: A. Identify the Baseline Group for each subject/grade combination (Baseline Grouping) and establish associated scale values. B. Calculate total Growth Points for each subject/grade combination (Growth Point Grouping) and establish associated scale values. C. Complete a determination of Adequate Yearly Progress (AYP) as mandated by the No Child Left Behind Act of 2001 for each public school and establish associated scale values. D. Add all additional (non-AYP) indicators and establish associated scale values [secondary schools only]. Arizona’s Accountability System: Volume I Arizona Department of Education 59 E. Option 1: Calculate a Total Scale Score Value by adding the Baseline Group scale values for each grade/subject combination to the Growth Point Group scale values for each subject grade combination (giving a 70% weight to the school’s strongest scale value and 30% weight to the other scale value) plus the AYP scale value plus the additional (non-AYP) indicator scale value. [Please note that the additional (non-AYP) indicator scale value is applied only to secondary schools. Option 2: Calculate a Total Scale Score Value by adding the Baseline Group scale values for each subject/grade combination or the Growth Point Group scale values for each subject/grade combination (whichever yields the greatest value) to the AYP scale value plus the additional (non-AYP) indicator scale value. [Please note that the additional (non-AYP) indicator scale value is applied only to secondary schools.] F. Evaluate the sum of all scale values (Total Scale Score Value) in relation to the school classification scale and associated cut points to determine final secondary school classifications and preliminary elementary school classifications. G. [For elementary school classifications only] Add “additional evidence” score value (MAP/EWS scores) to Total Scale Score Value (post calculation of preliminary school classification) to produce final elementary school classifications. A. Identifying Baseline Groups There are six (6) baseline groups created by created by five (5) different separation points (please refer to Table A). The percentage of students in the Meets or Exceeds (M/E) performance level on AIMS [% proficient] used to determine each Baseline Group will be established during Spring 2003 and brought to the Board for final approval in June 2003. The ADE will utilize these six (6) baseline groups to establish a six (6) – one (1) scale [six being the highest value and one being the lowest value]. A zero (0) value will be given for missing data. Arizona’s Accountability System: Volume I Arizona Department of Education 60 Table A: Baseline Groups 6 5 4 3 2 Starting Point (“Maintaining Line”) 1 Growth Point Groups 1 2 -1.0 SD 3 -.5 SD 4 X 5 .5 SD 6 1.0 SD Additionally, the ADE will apply a two (2)-year average of 2000 and 2001 AIMS data in order to determine Baseline groups. BOARD ACTION: A.1. The Board must adopt a two (2)-year average of 20002001 AIMS data in order to determine Baseline Groups. A.2. The Board must adopt the six (6) – one (1) Baseline Group scale as illustrated in Attachment One and Attachment Two. B. Calculating Growth Points Total Growth Points for each school and subject/grade combination are calculated by adding the following figures: 1. Elementary Schools [K-8; or any combination of those grades] (Reading, Writing, and Mathematic) a. Arizona’s Accountability System: Volume I The difference between the average percentage of students in the Falls Far Bellow (FFB) performance level on AIMS Arizona Department of Education 61 averaged over the 2001-2003 academic years and the percentage of students in the FFB performance level over a two (2)-year average of 2000 and 2001 AIMS. b. The difference between the average percentage of students in the Meets or Exceeds (M/E) performance levels on AIMS averaged over the 2001-2003 academic years and the percentage of students in the M/E performance level over a two (2)-year average of 2000 and 2001 AIMS. c. Total growth points are calculated by adding (a) and (b). 2. Secondary Schools [Grades 9-12] a. The difference between the average percentage of students in the Falls Far Below (FFB) performance level on AIMS averaged over the 2001-2003 academic years and the percentage of students in the FFB performance level over a two (2)-year average of 2000 and 2001 AIMS. b. The difference between the average percentage in the Meets or Exceeds (M/E) performance levels on AIMS averaged over the 2001-2003 academic years and the percentage of students in the M/E performance level over a two (2)-year average of 2000 and 2001 AIMS. c. Total growth points are calculated by adding (a) and (b). There are six (6) Growth Point Groups created by five (5) different separation points (please see Table A on page 2). The ADE will utilize these six (6) Growth Point groups to establish a six (6) – one (1) scale [six (6) being the highest value and one (1) being the lowest value]. A zero (0) value will be given for missing data. BOARD ACTION: B.1. The Board must adopt the six (6) – one (1) Growth Point Scale illustrated in Attachment One and Attachment Two. C. Adequate Yearly Progress (AYP) In accordance with the No Child Left Behind Act of 2001 (NCLB), the ADE will complete an Adequate Yearly Progress (AYP) determination for each public elementary and secondary school as defined by Section 1111 of Title I (NCLB). This determination simply identifies those schools that have made the federal definition of AYP and those schools that have not made AYP. The ADE will integrate the AYP determination into our accountability system by including it as a component of the Achievement Profile. As such, a scale value must be established for the AYP determination. The ADE will apply a one (1) – zero (0) scale value for the AYP determination. One (1) [given to Arizona’s Accountability System: Volume I Arizona Department of Education 62 schools that made AYP] represents the highest value, while zero (0) (given to schools that did not make AYP] represents the lowest value. BOARD ACTION: C.1. The Board must adopt the one (1) – zero (0) AYP scale illustrated in Attachment One and Attachment Two. D. Add All Additional (non-AYP) Indicators (Secondary Schools Only) The Achievement Profile for secondary schools includes the Graduation Rate and the Annual Dropout Rate (please refer to Table B on page 4, which summarizes the Graduation and Dropout Rate targets). Table B: Baseline and Targets for Annual Graduation and Dropout Rates (Secondary School Achievement Profile) Baseline* Dropout Rate 6-9 % > 9% Target** 1% Decrease 2% Decrease Baseline* Graduation Rate 74-90% < 74% Target** 1% Increase 2% Increase * The baseline is the 2000 academic year. ** The Annual Dropout Rate targets are the difference between the baseline year and the three (3) year average for the 2001-2003 academic years. The Graduation Rate targets are the difference between the baseline year and the three (3) year average for the years 2000-2002. The scale values for the Annual Graduation Rate and Dropout Rate indicators will be distributed based on the following table (please refer to Table C): Table C: School met the target in: Graduation T Dropout Scale Value Yes Yes 2 Yes No 1 No Yes 1 No No 0 Arizona’s Accountability System: Volume I Arizona Department of Education 63 BOARD ACTION: D.1. The Board must adopt the two (2) – zero (0) scale for the Additional (non-AYP) Indicator scale illustrated in Attachment Two. E. Calculating a Total Scale Score Value In calculating a Total Scale Score Value, the ADE presents two possible options to the Board. Option 1: In order to calculate a school’s Total Scale Score Value the ADE will add the Baseline Group scale values for each subject/grade combination to the Growth Point Group scale values for each subject/grade combination. The ADE will apply a 70% weight to the school’s strongest scale value (Baseline Group or Growth Point Group) and a 30% weight to the other scale value. This will result in an adjusted value. These values (for each subject/grade combination) are then added to the AYP scale score value. After the Baseline and Growth Point scale values for each subject/grade combination have been adjusted and added to the AYP scale value, the ADE will add the additional (non-AYP) indicator scale value. [Please note that the additional (non-AYP) indicator scale value is applied to only secondary schools.] Option 2: In order to calculate a school’s Total Scale Score Value the ADE will add the Baseline Group scale values for each subject/grade combination or the Growth point Group scale value for each subject/grade combination (which ever yields the greatest value) to the AYP scale value plus the additional (non-AYP) indicator scale value. [As indicated above, the additional indicator scale is applied only to secondary schools]. Due to the fact that Option 1 maintains systemic focus on student growth and increased academic achievement while allowing added emphasis to be placed on a school’s particular strength, the ADE recommends this option to the Board. BOARD ACTION: E.1.(a) The Board must adopt the calculation of the Total Scale Score Value (Option 1) as illustrated in Attachment One and Attachment Two. Or Arizona’s Accountability System: Volume I Arizona Department of Education 64 E.1.(b) The Board must adopt the calculation of the Total Scale Score Value (Option 2) as illustrated in Attachment Three and Attachment Four. F. Evaluate the Total Scale Score Value to Determine a School Classification At this time, the ADE is working in conjunction with school and community leaders to establish cut points associated with the school classification scale. The location of a school’s Total Scale Score Value when place on the school classification scale (depending on forthcoming cut points) will determine the classification of the school. BOARD ACTION: F.1. The Board must adopt the calculation illustrated in Attachment One and Attachment Two. Please note that cut scores associated with the school classification scale will be established during Spring 2003 and brought to the Board for final approval in June 2003. G. Add Additional Evidence to Produce Final Elementary School Classifications The ADE will award points for additional evidence of student growth and increased academic achievement. These “additional evidence” points will be added to the Total Scale Score Value (applied only to elementary schools). The distribution of additional points will be based on the average percentage of students making One Year’s Growth (OYG) according to the Measure of Academic Progress (MAP) over the 2001-2003 academic years and the average percentage of students with an extended writing trait score (EWS) of 24 or higher on AIMS over the 2001-2003 academic years. MAP will be calculated for each subject (reading and mathematics) as a whole school measure, while EWS will be calculated for each elementary grade assessed with AIMS (grades 3, 5, and 8). BOARD ACTION: G.1. The Board must adopt the application of added evidence as illustrated in Attachment One. Please note that cut scores associated with the school classification scale will be established during Spring 2003 and brought to the Board for final approval in June 2003. II. MODIFICATIONS MADE TO ACHIEVEMENT PROFILE FORMULA Arizona’s Accountability System: Volume I Arizona Department of Education 65 A. Mobility The ADE will not include students that have not been enrolled at a particular school for a full academic year in the Achievement Profile analysis for that school. BOARD ACTION 1.) The Board must adopt the exclusion of mobile students (as defined above) from the Achievement Profile analysis. B. “N” Count The ADE will utilize an “N” count [minimum population size] of thirty (30) for the Achievement Profile analysis. BOARD ACTION: 1.) The Board must adopt an “N” count of thirty (30) students for the Achievement Profile analysis. C. Compensatory model The ADE will utilize the previously adopted compensatory methodology for calculating the modified Achievement Profile. BOARD ACTION: 1.) The Board must adopt the use of the previously approved compensatory methodology for the calculation of the modified Achievement Profile. III. DISCUSSION ITEM Inclusion of special needs students into the accountability system for AYP purposes. Introduction of draft policy regarding the testing of Special needs students detailing the use of standard and non-standard accommodations during the administration of AIMS Arizona’s Accountability System: Volume I Arizona Department of Education 66 Appendix II: Arizona State Board of Education Information Packet for May 19, 2003 On May 19, 2003, the Arizona State Board of Education (Board) must adopt modifications made to the Achievement Profile methodology. This methodology will be applied to Arizona public schools, including charter schools, in order to determine school classifications by October 15, 2003 as required in A.R.S. §15-241 (Arizona LEARNS). The purpose of this document is to inform the Board of all necessary decisions required to adopt the modifications made to the Achievement Profile. This document includes a summary of the discussion items before the Board necessary to implement the Achievement Profiles. As mandated by A.R.S. §15-241, the Achievement Profile was developed according to a research-based methodology by the Arizona Department of Education (ADE) and members of the education community. All modifications to the Achievement Profile follow this principle. Upon adoption by the Board, the ADE will produce a technical report detailing the Achievement Profile methodology, including specific formulas and supporting documentation. I. Calculation of High School Mathematics Baseline The State Board of Education approved the use of a two (2) year average to calculate the AZ LEARNS Achievement Profile baseline groups. The Board was not presented options at that time to address the baseline calculation for high school mathematics, which presents a unique circumstance. a. The Arizona Department of Education will present to the Board a recommendation that the baseline for high school mathematics remain a single year (2000-2001) rather than move to a two (2) year average II. Calculation of Growth Point Groupings for High School Mathematics The Arizona Department of Education will recommend to the State Board of Education that based on their determination for the high school mathematics baseline that the department will utilize the appropriate methodology for calculating the Growth Point Groupings for high school mathematics. III. Baseline Grouping Cut point (6-1) Methodology Arizona’s Accountability System: Volume I Arizona Department of Education 67 The Arizona Department of Education will present to the State Board of Education the methodology setting new cut points for the Baseline Groupings which were established by the Board. The Arizona Department of Education after consultation with the Accountability Workgroup has determined an appropriate methodology to determine the necessary 6-1 cut point values. The Arizona Department of Education will recommend that the State Board of Education adopt the use of similar methodology that is employed to calculate the cut points for the Growth Point Groupings. a. The Arizona Department of Education will calculate the Baseline value for each school (traditional and charter) by subject and grade. b. The state average and standard deviation will be determined based on the Baseline values. c. The cut points for each subject/grade will be set using the ½ and 1 standard deviation as benchmarks. d. These cut points and subsequent values will be presented to the State Board during the June 2003 meeting. IV. Adequate Yearly Progress Additional Indicators The Arizona Department of Education will recommend to the State Board of Education that the threshold and gain expectations required by the No Child Left Behind Act (NCLB) should be consistent with existing expectations. a. The threshold for the elementary indicator, attendance, should be consistent with School Finance expectations for attendance rate. The threshold for the high school indicator, graduation rate, should also be consistent with the expectation articulated in Arizona’s Consolidated Application for the calculation of a graduate. The Arizona Department of Education will utilize the state average graduation rate to set the threshold, based on the definition of a graduate approved by the U.S. Department of Education. b. The Arizona Department of Education will recommend that the expected rate of gain for both the elementary and secondary Adequate Yearly Progress indicators be set at 1% increase annually. Arizona’s Accountability System: Volume I Arizona Department of Education 68 Appendix III: Arizona State Board of Education Information Packet for June 30, 2003 On June 30, 2003, the Arizona Department of Education will present to the Arizona State Board of Education (Board) proposed modifications to the Achievement Profile methodology. This methodology will be applied to Arizona public schools, including charter schools, in order to determine school classifications by October 15, 2003 as required in A.R.S. §15-241 (Arizona LEARNS). The purpose of this document is to inform the Board of all necessary decisions required to adopt the modifications made to the Achievement Profile. This document includes a summary of the action items before the Board necessary to implement the Achievement Profiles. As mandated by A.R.S. §15-241, the Achievement Profile was developed according to a research-based methodology by the Arizona Department of Education (ADE) and members of the education community. All modifications to the Achievement Profile follow this principle. Upon adoption by the Board, the ADE will produce a technical report detailing the Achievement Profile methodology, including specific formulas and supporting documentation. I. Calculation of Growth Point Groupings for high school math The Arizona Department of Education recommends to the State Board of Education that based on their determination for the high school mathematics baseline, utilizing 2001 data only, that the department utilizes the appropriate methodology for calculating the Growth Point Groupings for high school mathematics. The calculation for high school mathematics Growth Point Groupings will result in the use of a threeyear rolling average. Growth Points are calculated for high school mathematics by adding the following figures: a. The difference between the average percentage of students in the Falls Far Below (FFB) performance level on AIMS averaged over the 2001, 2002 and 2003 academic years and the percentage of students in the FFB performance level on the 2001 AIMS. b. The difference between the average percentage of students in the Meets or Exceeds (M/E) performance levels on AIMS averaged over the 2001, 2002 and 2003 academic years and the percentage of students in the M/E performance levels on the 2001 AIMS. Arizona’s Accountability System: Volume I Arizona Department of Education 69 BOARD ACTION: 1.1. The Arizona Department of Education recommends that the Board adopt the Growth Point calculation for high school mathematics as described above. II. Baseline Grouping Cut point (6-1) Methodology The Arizona Department of Education presents to the State Board of Education the methodology for setting new cut points for the Baseline Groupings, which were established by the Board. The Arizona Department of Education after consultation with the Accountability Workgroup has determined the most appropriate methodology to establish the necessary 6-1 cut point values. The following methodology will be used to calculate Baseline Groupings for all subjects and grades: c. The Arizona Department of Education will calculate the Baseline value for each school (traditional and charter) by subject and grade. d. The state average and standard deviation will be determined based on the Baseline values. e. The cut points for each subject/grade will be set using the ½ and 1 standard deviation as benchmarks. f. Attachment One (1) represents the Baseline Grouping separation points calculated using the Standard Deviation Methodology described above. g. Attachment Two (2) represents the number of schools in each Baseline Group; this impact data is based on the new calculation of baseline year and the standard deviation methodology. h. Attachment Three (3) represents a comparison between the number of schools in each Baseline Group utilizing the new methodology and the number of schools in each Baseline Group for the 2003 Achievement Profiles. BOARD ACTION: 2.1. The Arizona Department of Education recommends that the Board adopt the Baseline Grouping methodology as described above. 2.2. The Arizona Department of Education recommends that the Board adopt the Baseline Grouping separation points in Attachment One. Arizona’s Accountability System: Volume I Arizona Department of Education 70 III. Adequate Yearly Progress (AYP) Additional Indicators The Arizona Department of Education recommends to the State Board of Education that the threshold and gain expectations required by the No Child Left Behind Act (NCLB) should be consistent with existing expectations set forth by the Arizona Department of Education’s School Finance division as well as the U.S. Department of Education’s Office of Elementary and Secondary Education. Under NCLB, schools are required to either met the threshold set for these additional indicators or meet the expected gain set for these additional indicators. i. The threshold for the elementary AYP indicator, attendance, should be consistent with School Finance expectations for attendance rate set by A.R.S. § 15-902 A-B. Therefore, the Arizona Department of Education recommends that the threshold be set at 94%. j. The U.S. Department of Education requires Arizona to use a four (4) year graduation rate for all No Child Left Behind calculations. Therefore, the threshold for the high school AYP indicator, graduation rate, will be set at the four (4) year state average graduation rate. The Arizona Department of Education recommends that the threshold rate be set at 71%, which represents the State average graduation rate for 2001. k. The Arizona Department of Education recommends that the expected rate of gain for both the elementary and secondary Adequate Yearly Progress (AYP) indicators be set at a 1% increase annually. BOARD ACTION: 3.A. The Arizona Department of Education recommends that the Board adopt the Adequate Yearly Progress (AYP) additional indicator threshold rate for the elementary schools as described above. 3.B. The Arizona Department of Education recommends that the Board adopt the Adequate Yearly Progress (AYP) additional indicator threshold rate for high schools as described above. 3.C. The Arizona Department of Education recommends that the Board adopt the Adequate Yearly Progress (AYP) additional indicator expected rate of gain for elementary schools and high schools as described above. IV. Inclusion of special needs students into the accountability system for AYP purposes Arizona’s Accountability System: Volume I Arizona Department of Education 71 The No Child Left Behind Act (NCLB) dictates that all students must be assessed against state standards. A workgroup of special educators, assessment coordinators and ADE staff have been working since November to develop a set of recommendations for best practices in assessing students with disabilities. This list of recommendations is presented below for your consideration and appropriate action. The list of recommendations refers to AIMS administration only. The Arizona Department of Education and the workgroup recommended to the Board that the administration of the Stanford Nine (SAT 9) remain intact. Upon adoption by the Board, the ADE will produce a technical report detailing the testing methodology for student with disabilities, including specific examples and supporting documentation. a. The Arizona Department of Education recommends to the State Board of Education that it broaden acceptable accommodations to include standard and non-standard accommodations. Standard accommodations are changes in the routine conditions under which students take AIMS, and do not substantially change the instructional level, the content or the performance criteria. Nonstandard accommodations reflect changes in the test administration or in the way a student responds to test questions. As such, non-standard accommodations may involve substantial changes in what a student is expected to learn and to demonstrate, possibly altering what the test measures. b. The Arizona Department of Education recommends to the State Board of Education that students who have an IEP may be considered for standard and non-standard accommodations as well as alternate assessment. Students with a 504 plan may only be considered for standard accommodations. Given that non-standard accommodations involve substantial changes in what a student is expected to learn and to demonstrate, students considered for this accommodation must receive at least part of their instruction in special education and must have an IEP. c. The Arizona Department of Education recommends to the State Board of Education that AIMS results taken with standard and non-standard accommodations be included in the results with students who took these tests under standard conditions, at the school, district and state level. For reporting purposes the Arizona Department of Education will maintain a record of the number of students in each school and district taking AIMS tests with non-standard accommodations. Given that test scores with non-standard accommodations cannot be Arizona’s Accountability System: Volume I Arizona Department of Education 72 interpreted in the same way, guidance for appropriate interpretations will be included in the technical report. BOARD ACTION: 3.A. The Arizona Department of Education recommends that the Board adopt the use of standard and non-standard accommodations as described above. 3.B. The Arizona Department of Education recommends that the Board adopt the use of regulations regarding the use of standard and non-standard accommodations as described above. 3.C. The Arizona Department of Education recommends that the Board adopt the reporting of test data for standard and non-standard accommodations as described above. Arizona’s Accountability System: Volume I Arizona Department of Education 73 Appendix IV: Arizona State Board of Education Information Packet for August 25, 2003 On August 25, 2003, the Arizona Department of Education will present to the Arizona State Board of Education (Board) proposed modifications to the Achievement Profile methodology. This methodology will be applied to Arizona public schools, including charter schools, in order to determine school classifications by October 15, 2003 as required in A.R.S. §15-241 (Arizona LEARNS). The purpose of this document is to inform the Board of all necessary decisions required to adopt the modifications made to the Achievement Profile. This document includes a summary of the action items before the Board necessary to implement the Achievement Profiles. As mandated by A.R.S. §15-241, the Achievement Profile was developed according to a research-based methodology by the Arizona Department of Education (ADE) and members of the education community. All modifications to the Achievement Profile follow this principle. Upon adoption by the Board, the ADE will produce a technical report detailing the Achievement Profile methodology, including specific formulas and supporting documentation. V. Administrative Policies A. Schools receiving Adequate Yearly Progress designations with alternate methodology B. Schools not receiving an AZ LEARNS Achievement Profile on October 15, 2003 C. Schools receiving an AZ LEARNS Achievement Profile on October 15, 2003 with alternate methodology D. Missing Data Policy A. Schools receiving Adequate Yearly Progress designations with alternate methodology 1. K-2 schools- defined as a school serving grades kindergarten through second grade, or any combination of these grades. According to current data available Arizona has 51 schools educating students from preK-K-2 grade. The ADE recommends to the Board that for the purposes of Adequate Yearly Progress (AYP) designations K-2 schools should be evaluated based on student performance on the SAT 9 Arizona’s Accountability System: Volume I Arizona Department of Education 74 as well as attendance rate. While the No Child Left Behind Act (NCLB) generally prohibits the use of norm-referenced assessments the U.S. Department of Education has made exceptions to this rule when criterion-referenced assessments are not available, as in this case. Therefore, the U.S. Department of Education will allow Arizona to use results from the SAT 9 to evaluate K-2 schools. The ADE recommends that K-2 schools be evaluated based on the percentage of students scoring at or above the 50th percentile on the SAT 9, providing the school meets the minimum number of students for evaluation (30). a. The 50th percentile represents the national average and will serve as the proficiency expectation for AYP determinations. b. The ADE will utilize 2002 SAT 9 scores as a baseline measure for all schools, utilizing the NCLB methodology for setting a proficiency expectation, the ADE will determine whether each school meets the proficiency expectation. c. K-2 schools are expected to increase the percent meeting the 50th percentile threshold by 1 % each year. d. K-2 schools will be evaluated based on aggregate scores for reading and mathematics- disaggregated groups will not be evaluated. Additionally, K-2 schools will be evaluated for the additional indicator required for all elementary schools, attendance. The Arizona Department of Education recommends that the threshold and expected rate of gain for this indicator be set at: a. Threshold: 94% attendance Or b. Expected gain: 1% increase annually BOARD ACTION: A.1. The Arizona Department of Education recommends that the Board adopt the Adequate Yearly Progress (AYP) methodology for K-2 schools as described above. 2. K-1 schools- defined as serving students from grades kindergarten through first, or any combination of these grades must also be evaluated for AYP according to the No Child Left Behind Act. The ADE recommends to the Board that for purposes of AYP designations these schools be evaluated based on a comprehensive site visit. The ADE must still develop an Arizona’s Accountability System: Volume I Arizona Department of Education 75 evaluation rubric for K-1 schools; develop capacity within Research and Policy to evaluate each K-1 school and provide final AYP determinations. The ADE estimates that K-1 schools will receive an AYP determination for the first time in 2004. BOARD ACTION: A.2. The Arizona Department of Education recommends that the Board adopt the Adequate Yearly Progress (AYP) methodology for K-1 schools as described above. 3. Extremely small schools – defined as schools with less than 100 students ADM in the baseline year (2001-2002) and less than 30 students in more than 1/3 of its subject/grade values. The ADE recommends that AYP determinations for extremely small schools be based on aggregate data for the subjects and grades assessed (reading and mathematics). a. For AMO calculations, all of the students in the school will be evaluated at the subject level, rather than grade level to make valid and reliable AYP determinations. b. For AMO calculations, disaggregated groups will not be evaluated. c. The ADE will make AMO determinations based on a 99% confidence level to ensure statistical validity. d. AYP additional indicators will be evaluated based on the following criteria: i. The ADE will not evaluate subjects/grades that do not meet the minimum number required for evaluation. ii. The ADE will not evaluate disaggregated groups. iii. The ADE will evaluate attendance rate and graduation rate for extremely small schools based on the same criteria established by the Board for all other schools. BOARD ACTION: A.3. The Arizona Department of Education recommends that the Board adopt the Adequate Yearly Progress (AYP) methodology for extremely small schools as described above. B. Schools Not Receiving an AZ LEARNS Achievement Profile on October 15, 2003 1. Extremely small schools – defined as schools that average less than 100 Arizona’s Accountability System: Volume I Arizona Department of Education 76 students ADM in the baseline years (2000/2001) and less than 30 students in more than 1/3 of its subject/grade values. The ADE recommends to the Board that for purposes of AZ LEARNS Achievement Profiles these schools be evaluated based on a comprehensive site visit. The ADE must still develop an evaluation rubric for extremely small schools; develop capacity within Research and Policy to evaluate each extremely small school and provide final AZ LEARNS determinations. The ADE estimates that extremely small schools will receive an AZ LEARNS Achievement Profile for the first time in 2004. BOARD ACTION: B.1. The Arizona Department of Education recommends that the Board adopt the AZ LEARNS Achievement Profile methodology for extremely small schools as outlined above. 2. K-2 schools- defined as a school serving grades kindergarten through second grade, or any combination of these grades. According to current data available Arizona has 51 schools educating students from preK-K-2 grade. The ADE recommends to the Board that for purposes of AZ LEARNS Achievement Profile designations these schools be evaluated based on a comprehensive site visit. The ADE must still develop an evaluation rubric for K-2 schools; develop capacity within Research and Policy to evaluate each K-2 school and provide final AZ LEARNS Achievement Profile. The ADE estimates that K-2 schools will receive an AZ LEARNS Achievement Profile for the first time in 2004. BOARD ACTION: B.2. The Arizona Department of Education recommends that the Board adopt the AZ LEARNS Achievement Profile methodology for K-2 schools as outlined above. C. Schools Receiving an AZ LEARNS Achievement Profile on October 15, 2003 with alternate methodology 1. New schools – defined as schools that opened for the first time after Summer 2000. The ADE recommends that the State Board adopt option 1 below. A second option is provided for the Board as an alternative. Arizona’s Accountability System: Volume I Arizona Department of Education 77 Option 1: Once a school has been operational for three (3) test administrations, the school will receive an achievement profile utilizing the updated formula changes passed by the Board during 2003, with the exception of a one-year baseline analysis rather than a two-year baseline analysis. The new school will be evaluated based on three years of data; a school must only have one (1) overlapping year of test data to be evaluated for an Achievement Profile. Example: School A opened in 2001 and is still currently in operation. Baseline grouping AIMS data (single year)- 2001 Growth point grouping AIMS data (three year average)2001, 2002, 2003 Option 2: Once a school has been operational for four (4) test administrations, the school will receive an achievement profile utilizing the updated formula changes passed by the Board during 2003. The new school will be evaluated based on four years of data; a school must only have one (1) overlapping year of test data to be evaluated for an Achievement Profile. Example: School B opened in 2001 and is still currently in operation. Baseline grouping AIMS data (two year average)- 2001 and 2002 Growth point grouping AIMS data (three year average)2002, 2003, 2004 BOARD ACTION: B.1. The Arizona Department of Education recommends that the Board adopt the calculation of the Achievement Profile for new schools (Option 1) as outlined above. Or The Arizona Department of Education recommends that the Board adopt the calculation of the Achievement Profile for new schools (Option 2) as outlined above. D. Missing Data Policy 1. A school that has not provided the necessary data for any subject/grade combination shall receive a subject/grade value of zero (0) for that subject/grade combination the first year it is missing data. In future calculations of the Achievement Profile, which contain the missing year previously counted as 0-MD, the Arizona Department Arizona’s Accountability System: Volume I Arizona Department of Education 78 of Education will apply the following rules by subject/grade combination: If a school has one (1) year of missing baseline data, its baseline will be calculated using the single year of baseline data it does have. If a school has two (2) years of missing baseline data it receives a 0. Growth points are calculated from a baseline of zero, but the school can only receive 30 percent weight for the growth point grouping. The baseline grouping reflecting the 0 must be counted as 70 percent weight. If a school has one (1) year of the required data missing for the calculation of growth points, its growth points are calculated using two years of data (rather than three years of data). If a school is missing two (2) or more years of data required for the calculation of growth points, it receives a 0. BOARD ACTION: D.1. The Arizona Department of Education recommends that the Board adopt the rules for evaluating missing data in the Achievement Profile calculation as outlined above. Arizona’s Accountability System: Volume I Arizona Department of Education 79 Appendix V: Arizona State Board of Education Information Packet for September 16, 2003 On September 16, 2003, the Arizona Department of Education (ADE) will present to the Arizona State Board of Education (Board) proposed modifications to the Achievement Profile methodology. This methodology will be applied to Arizona public elementary and secondary schools, including charter schools, in order to determine school classifications by October 15, 2003 as required by A.R.S. §15-241 (ARIZONA LEARNS). The purpose of this document is to inform the Board of all necessary decisions required to adopt the modifications made to the Achievement Profile. This document includes an overview of the general process to produce the Achievement Profiles, a summary of the actions before the Board, specific numeric values associated with those actions, and the administrative policies necessary to implement the Achievement Profiles. As mandated by A.R.S. §15-241, the ADE in collaboration with members of the education community developed the Achievement Profile according to a research-based methodology. All modifications to the Achievement Profile follow this principle. Upon adoption by the Board, the ADE will produce a technical report detailing the Achievement Profile methodology, including specific formulas and supporting documentation. Arizona’s Accountability System: Volume I Arizona Department of Education 80 Illustration A: Achievement Profile (Elementary Model) Each Subject/Grade Whole School Baseline Grouping Growth Point Grouping 6 6 5 5 Pre-MAP/EWS Whole School School Classification Scale AYP Additional Points 1 E 4 4 3 3 + Made AYP + 2 2 1 1 0 Missing Data 0 Missing Data HP = + P 0 School Classification Scale MAP/ EWS U Missed AYP Up to 40% of Total subject/ grade points and AYP calculation E = P U 70/30 Float Weight Illustration B: Achievement Profile (Secondary Model) Each Subject/Grade Whole School Baseline Grouping Growth Point Grouping 6 6 5 5 1 4 4 Made AYP 3 + 3 Whole School AYP + 2 2 1 1 0 0 Missing Data 0 Missing Data Missed AYP School Classification Scale Graduation Rate/ Dropout Rate 2 Made Graduation and Dropout Rate Requirements E + 1 Made either Graduation or Dropout Rate Requirement 0 Did not make both requirements = HP P U 70/30 Float Weight Arizona’s Accountability System: Volume I HP Arizona Department of Education 81 I. GENERAL PROCESS TO PRODUCE THE ACHIEVEMENT PROFILES According to A.R.S. §15-241, the Achievement Profile is utilized to determine a public school classification. The general process to calculate the Achievement Profile for each school is as follows: A. Identify the Baseline Group for each subject/grade combination (Baseline Grouping) and establish associated scale values. B. Calculate total Growth Points for each subject/grade combination (Growth Point Grouping) and establish associated scale values. C. Complete a determination of Adequate Yearly Progress (AYP) as mandated by the No Child Left Behind Act of 2001 for each public school and establish associated scale values. D. Add additional (non-AYP) indicators of graduation rate and dropout rate and establish associated scale values [secondary schools only]. E. Calculate a total scale score value by adding the Baseline Group scale values for each grade/subject combination to the Growth Point Group scale values for each subject grade combination (giving a 70% weight to the school’s strongest scale value and 30% weight to the other scale value) plus the AYP scale value plus the additional (non-AYP) indicator scale value [secondary schools only]. F. Evaluate the sum of all scale values (i.e. the total scale score value) in relation to the school classification scale and associated cut points to determine secondary school Achievement Profile classifications and preliminary (preadded evidence) elementary school Achievement Profile classifications. G. Add “additional evidence” (MAP and EWS) to total scale score value in order to produce elementary school classifications. H. Apply threshold criteria for Excelling and Highly Performing Achievement Profile school classifications based on average percentage of students in the “Exceeds the Standard” category on AIMS. Requisite percentages will be set for Excelling and Highly Performing classifications based on the subject/grade combinations assessed at a particular school. Arizona’s Accountability System: Volume I Arizona Department of Education 82 A. Identifying Baseline Groupings The Board previously approved six (6) baseline groups created by five (5) different separation points. This modification reflects legislative amendments made to A.R.S. §15-241. The ADE will utilize these five (5) separation points to establish a six (6) – one (1) scale [six being the highest value and one being the lowest value]. A zero (0) value will be assigned for missing data. Additionally, the ADE will apply a two (2)-year average of 2000 and 2001 AIMS data in order to determine Baseline groups for all grades/subjects, except for high school mathematics. 2001 AIMS data will serve to determine Baseline groups for high school mathematics. The ADE recommends that Baseline Groupings be established using the average percent of students meeting the standard for 2000 and 2001, for all subjects and grades except high school mathematics. High school mathematics will utilize the percent of students meeting the standard for 2001 to calculate baseline scores. Utilizing a beta weight distribution the ADE determined the cut points for the six Baseline Groupings based on the following percentile ranks: 90, 75, 50, 25 and 10. The beta weight distribution has two distinct advantages. First, its scores fall within the boundaries of zero and one, resulting in positive values associated with the percentage of students meeting the standard. This approach is preferred when dealing with a proportion, as it isn’t possible to have a negative value in the baseline grouping. Second, a beta weight distribution allows the department to deal with oddly shaped or skewed distributions of data. The application of the beta weight distribution yielded the following cut points for the Baseline Groupings per subject/grade (please see Table 1). Table 1: Baseline Groupings based on the percent of students Meeting/Exceeding the standard Grade Subject Baseline Grouping 1 Baseline Grouping 2 Baseline Grouping 3 Baseline Grouping 4 Baseline Grouping 5 Baseline Grouping 6 3 3 3 5 5 5 8 8 8 H.S. H.S. H.S. Math Reading Writing Math Reading Writing Math Reading Writing Math Reading Writing 0% - 26% 0% - 46% 0% - 54% 0% - 11% 0% - 31% 0% - 25% 0% - 1% 0% - 25% 0% - 18% 0% - 3% 0% - 28% 0% - 16% 27% - 40% 47% - 59% 55% - 67% 12% - 21% 32% - 44% 26% - 38% 2% - 5% 26% - 37% 19% - 28% 4% - 8% 29% - 42% 17% - 25% 41% - 56% 60% - 73% 68% - 79% 22% - 36% 45% - 60% 39% - 53% 6% - 12% 38% - 51% 29% - 42% 9% - 19% 43% - 58% 26% - 39% 57% - 71% 74% - 84% 80% - 89% 37% - 52% 61% - 75% 54% - 68% 13% - 22% 52% - 66% 43% - 56% 20% - 33% 59% - 73% 40% - 53% 72% - 82% 85% - 91% 90% - 94% 53% - 66% 76% - 85% 69% - 79% 23% - 34% 67% - 77% 57% - 68% 34% - 47% 74% - 83% 54% - 66% 83% - 100% 92% - 100% 95% - 100% 67% - 100% 86% - 100% 80% - 100% 35% - 100% 78% - 100% 69% - 100% 48% - 100% 84% - 100% 67% - 100% BOARD ACTION: A1.) The ADE recommends that the Board approve the utilization of the beta distribution to determine cut points for the six (6) Baseline Groupings based on the following percentile ranks: 90, Arizona’s Accountability System: Volume I Arizona Department of Education 83 75, 50, 25 and 10, which yield the cut scores illustrated in Table 1 (on page 3 of this document). B. Calculating Growth Points Total Growth Points for each school and subject/grade combination are calculated by adding the following figures: 1. Elementary Schools [K-8; or any combination of those grades] (Reading, Writing, and Mathematic) a. The difference between the average percentage of students in the Falls Far Below (FFB) performance level on AIMS averaged over the 2001-2003 academic years and the percentage of students in the FFB performance level over a two (2)-year average of 2000 and 2001 AIMS. The difference between the average percentage of students in the Meets or Exceeds (M/E) performance levels on AIMS averaged over the 2001-2003 academic years and the percentage of students in the M/E performance level over a two (2)-year average of 2000 and 2001 AIMS. b. Secondary Schools [Grades 9-12] (Reading and Writing) a. The difference between the average percentage of students in the Falls Far Below (FFB) performance level on AIMS averaged over the 2001-2003 academic years and the percentage of students in the FFB performance level over a two (2)-year average of 2000 and 2001 AIMS. The difference between the average percentage in the Meets or Exceeds (M/E) performance levels on AIMS averaged over the 2001-2003 academic years and the percentage of students in the M/E performance level over a two (2)-year average of 2000 and 2001 AIMS. b. Secondary Schools [Grades 9-12] (Mathematics) a. The difference between the average percentage of students in the Falls Far Below (FFB) performance level on AIMS averaged over 2001-2003 academic years and the percentage of students in the FFB performance level on the 2001 AIMS. b. The difference between the average percentage of students in the Meets or Exceeds (M/E) performance levels on AIMS averaged over the 2001-2003 academic years and the percentage of students in the M/E performance levels on the 2001 AIMS. The ADE recommends that the Board approve the six (6) Growth Point Groups created by five (5) different separation points. The ADE will utilize these six (6) Growth Point groups to establish a six (6) – one (1) scale [six (6) being the highest value and one (1) being the lowest value]. A zero (0) value will be given for missing data. Arizona’s Accountability System: Volume I Arizona Department of Education 84 The ADE recommends that the Board approve the same methodology approved by the Board for the 2002 Achievement Profile (please see Illustration C) to set the cut points for the six Growth Point Groupings. This methodology utilizes the distribution of growth points by subject and grade combination for the state. The separation points are determined by evaluating the state average (mean) and the values associated within ½ standard deviation and 1 standard deviation from the mean. These cut point results can also be represented using percentile rankings as well. The subsequent percentile ranks would be 16%, 31%, 50%, 69%, and 84%. Illustration C: Determination of Growth Group Cut Points Growth Point Groupings 1 2 -1.0 SD 3 - .5 SD 4 X 5 .5 SD 6 1.0 SD The utilization of this methodology will yield the following cut points for the six Growth Point Groupings (please see Table 2). Table 2: Growth Point Groupings 3 Math Growth Point Grouping 1 <-1.75% 3 Reading <-5.41% 3 Writing <-9.23% 5 Math <-1.61% 5 Reading <-15.16% 5 Writing <-8.18% 8 Math <-7.99% 8 Reading <-5.86% 8 Writing <-10.24% H.S. Math <-5.81 H.S. Reading <-10.50% H.S. Writing <10.72% Grade Subject Arizona’s Accountability System: Volume I Growth Point Grouping 2 -1.74% 4.18% -5.40% -0.90% -9.22% -5.00% -1.60% 4.11% -15.15 -10.46% -8.17% -3.44% -7.98% -1.94% -5.85% -0.81 -10.23% -5.92% -5.80% -1.60% -10.49% -6.10% 10.73% 15.32% - Growth Point Grouping 3 4.19% 10.12% -0.89% 3.60% -4.99% -0.77% 4.12% 9.83% -10.45% -5.77% -3.43% 1.29% -1.93% 4.11% -0.80% 4.24% -5.91% -1.61% -1.59% 2.61% -6.09% -1.70% 15.33% 19.92% - Growth Point Grouping 4 10.13% 16.05% 3.61% 8.11% -0.76% 3.46% 9.84% 15.56% -5.76% -1.07% 1.30% 6.02% 4.12% 10.17% 4.25% 9.29% -1.60% 2.70% 2.62% 6.83% -1.69% 2.71% 19.93% 24.52% - Growth Point Grouping 5 16.06% 21.98% 8.12% 12.61% 3.47% 7.69% 15.57% 21.28% -1.06% 3.62% 6.03% 10.76% 10.18% 16.22% 9.30% 14.34% 2.71% 7.02% 6.84% 11.04% 2.72% 7.11% 24.53% 29.12% - Growth Point Grouping 6 21.99% > - 12.62% > - 7.70% > - 21.29% > - 3.63% > - 10.77% > - 16.23% > - 14.35% > - 7.03% > - 11.05% > - 7.12% > - 29.13% > Arizona Department of Education 85 BOARD ACTION: B1.) The ADE recommends that the Board approve the application of the Growth Point Grouping methodology as described above and detailed in Illustration C, yielding the cut points presented in Table 2. C. Adequate Yearly Progress (AYP) The Board has approved the calculation of Adequate Yearly Progress (AYP), in accordance with the No Child Left Behind Act of 2001 (NCLB), the ADE will complete an Adequate Yearly Progress (AYP) determination for each public elementary and secondary school as defined by Section 1111 of Title I (NCLB). This determination simply identifies those schools that have made the federal definition of AYP and those schools that have not made AYP. The ADE will integrate the AYP determination into our accountability system by including it as a component of the Achievement Profile. As such, a scale value must be established for the AYP determination. The ADE will apply a one (1) – zero (0) scale value for the AYP determination. One (1) [given to schools that made AYP] represents the highest value, while zero (0) (given to schools that did not make AYP) represents the lowest value. The Board has adopted the one (1) – zero (0) AYP scale as detailed in Illustration A (6/03). D. Add All Additional (non-AYP) Indicators (Secondary Schools Only) The Achievement Profile for high schools includes the Graduation Rate and the Annual Dropout Rate (please refer to the Table 3, which summarizes the Graduation and Dropout Rate targets). Table 3: Baseline and Targets for Annual Graduation and Dropout Rates (Secondary School Achievement Profile) Baseline* Dropout Rate 6-9 % > 9% Target** 1% Decrease 2% Decrease Baseline* Graduation Rate 74-90% < 74% Target** 1% Increase 2% Increase * The baseline is the 2000 academic year. ** The Annual Dropout Rate targets are the difference between the baseline year and the three (3) year average for the 2001-2003 academic years. The Graduation Rate targets are the difference between the baseline year and the three (3) year average for the years 2000-2002. The scale values for the Annual Graduation Rate and Dropout Rate indicators will be distributed based on the following table: Arizona’s Accountability System: Volume I Arizona Department of Education 86 Table 4: Decision matrix and point values for High School Additional Indicators School met the target in: Graduation Dropout Scale Value Yes Yes 2 Yes No 1 No Yes 1 No No 0 E. Calculating the School Classification Scale In order to calculate a school’s scale classification value (pre MAP/EWS), the ADE will add the Baseline Group scale values for each grade/subject combination to the Growth Point Group scale values for each grade/subject combination. The ADE will apply a 70% weight to the school’s strongest scale value (Baseline Group or Growth Point Group) and a 30% weight to the other scale value as approved by the Board. The Baseline and Growth Point scale values for each grade/subject combination (post float weight) are then added to the AYP scale score value. After the Baseline and Growth Point scale values for each grade/subject combination (post float weight) are added to the AYP scale value, the ADE will add the graduation rate and dropout rate scale value (applied only to secondary schools). This methodology has been approved by the Board (6/03). Application of the 70%/30% float weight methodology and Board approval of action items A1 and B1 will result in the following Baseline and Growth Point values per subject/grade (please see Table 5 and Table 6): Arizona’s Accountability System: Volume I Arizona Department of Education 87 Table 5: Elementary Scale- Point Distributions by Baseline Grouping and Growth Point Grouping Growth Point Grouping 1 Growth Point Grouping 2 Growth Point Grouping 3 Growth Point Grouping 4 Growth Point Grouping 5 Growth Point Grouping 6 Baseline Grouping 1 1 1.7 2.4 3.1 3.8 4.5 Baseline Grouping 2 1.7 2 2.7 3.4 4.1 4.8 Baseline Grouping 3 2.4 2.7 3 3.7 4.4 5.1 Baseline Grouping 4 3.1 3.4 3.7 4 4.7 5.4 Baseline Grouping 5 3.8 4.1 4.4 4.7 5 5.7 Baseline Grouping 6 4.5 4.8 5.1 5.4 5.7 6 Table 6: High School Scale- Point Distributions by Baseline Grouping and Growth Point Grouping Growth Point Grouping1 Growth Point Grouping 2 Growth Point Grouping 3 Growth Point Grouping 4 Growth Point Grouping 5 Growth Point Grouping 6 Baseline Grouping 1 1 1.7 2.4 3.1 3.8 4.5 Baseline Grouping 2 1.7 2 2.7 3.4 4.1 4.8 Baseline Grouping 3 2.4 2.7 3 3.7 4.4 5.1 Baseline Grouping 4 3.1 3.4 3.7 4 4.7 5.4 Baseline Grouping 5 3.8 4.1 4.4 4.7 5 5.7 Baseline Grouping 6 4.5 4.8 5.1 5.4 5.7 6 Arizona’s Accountability System: Volume I Arizona Department of Education 88 Evaluate the Total Scale Score Value to Determine a School Classification F. The location of a school’s total scale score value when placed on the school classification scale will determine the classification of the school. Table 5 and Table 6 represent a visual model of the cut points provided below. Score ranges represented in red are conceptually equivalent to underperforming, yellow are conceptually equivalent to performing, orange are conceptually equivalent to highly performing, and green are conceptually equivalent to excelling. To complete the cut point setting process the ADE recommends to the Board the following cut points for school classification (please see Table 7 and Table 8). Table 7: Elementary School Classification Cut Points Elementary Model Subject/Grade Combination 1 Subject/Grade Combination 3 Subject/Grade Combination 6 Subject/Grade Combination 9 Underperforming <4 < 12 < 24 < 36 Performing 4 12 24 36 Highly Performing 4.6 13.8 27.6 41.4 Excelling 5.4 16.2 32.4 48.6 Table 8: High School Classification Cut Points Secondary Model Subject/Grade Combination 1 Subject/Grade Combination 3 Underperforming < 3.2 < 9.6 Performing 3.2 9.6 Highly Performing 5 15 Excelling 5.4 16.2 Arizona’s Accountability System: Volume I Arizona Department of Education 89 BOARD ACTION: F1.) The ADE recommends that the Board approve the cut points detailed in Table 7 and Table 8 to determine the Achievement Profile school classification. G. Add Additional Evidence to Produce Elementary School Achievement Profile Classifications The ADE recommends to the Board that they award points for additional evidence of student growth and increased academic achievement post calculation of the total scale score value (applied only to elementary schools). The distribution of additional points will be based on the average percentage of students making One Year’s Growth (OYG) according to the Measure of Academic Progress (MAP) over the 20012003 academic years and the average percentage of students with an extended writing trait score (EWS) of 24 or higher on AIMS over the 2001-2003 academic years. MAP will be calculated for each grade/subject (reading and mathematics) combination (whole school evaluation), while EWS will be evaluated for each elementary grade assessed with AIMS (grades 3, 5, and 8). Calculating Added Evidence Points (Elementary Model): In calculating the added evidence points the ADE recommends the following methodology: 1.) 2.) 3.) 4.) Arizona’s Accountability System: Volume I Calculate number of students making OYG and the number in the analysis using a three (3) year average for the whole school (reading and mathematics). Calculate the number of students scoring 24 or more points on the EWS and the number included in the analysis using a three (3) year average. Determine the total number of students to be included in the added evidence points by adding the number of students making OYG (reading and mathematics) and the number of students scoring 24 or more on the EWS. Divide total by the total number included in the analysis for OYG (reading and mathematics) and EWS to determine the percent total added evidence. Use the following grid to determine the points assigned by subject/grade combination (please see Table 9): Arizona Department of Education 90 Table 9: Distribution of Elementary Added Evidence Points by Subject/Grade Combination Subject/Grade Combinations % Total Added Evidence 1 2 3 4 5 6 7 8 9 90% + 3 5 8 10 12 15 17 20 22 80%-89% 2.25 3.75 6 7.5 9 11.25 12.75 15 16.5 70%-79% 1.5 2.5 4 5 6 7.5 8.5 10 11 60%-69% 0.75 1.25 2 2.5 3 3.75 4.25 5 5.5 Application of this methodology results in the following scale permutations for the elementary Achievement Profile (please see Table 10): Table 10: Elementary School Scale Permutations Subject/Grade Combinations Subject/Grade Total Points AYP Total Points 1 2 3 4 5 6 7 8 9 6 12 18 24 30 36 42 48 54 1 1 1 1 1 1 1 1 1 Total Subject/Grade and AYP Points 7 13 19 25 31 37 43 49 55 Total Added Evidence Points Up to 3 Up to 5 Up to 8 Up to 10 Up to 12 Up to 15 Up to 17 Up to 20 Up to 22 Final Total Points 10 18 27 35 43 52 60 69 77 BOARD ACTION: G1.) The ADE recommends that the Board approve the added evidence methodology as detailed in action steps 1-4 on page 9 of this document and all resulting point values as illustrated in Table 9 and Table 10 of this document. H. Application of Threshold Criteria for Excelling and Highly Performing Schools To ensure continued focus on improving the academic achievement of all students as they reach their absolute levels of attainment, including those students currently demonstrating proficiency in Arizona’s Academic Standards on AIMS, the ADE proposes the application of threshold criteria to determine Excelling and Highly Performing schools. These threshold criteria are based on the average percentage of Arizona’s Accountability System: Volume I Arizona Department of Education 91 students in the “Exceeds the Standard” category on AIMS (reading, writing or mathematics) in a particular school. Conceptually, these threshold criteria serve as parameters to establish distinct boundaries around the Excelling and Highly Performing Achievement Profile classifications. Schools must not only receive a total scale value that places them into either Excelling or Highly Performing, but must also meet the requisite percentage of students in the “Exceeds the Standard” category on AIMS to be designated as either an Excelling or Highly Performing schools. The application of threshold criteria for Excelling and Highly Performing schools results in the following scenarios (please see Illustration D): 1.) A school that receives a total scale value that places it in the Excelling classification and meets the requisite percentage of students in the Exceeds the Standard category on AIMS necessary for an Excelling classification will be designated an Excelling school. 2.) A school that receives a total scale value that places it in the Excelling classification and did not meet the requisite percentage of students in the Exceeds the Standard category on AIMS necessary for a Excelling classification, but did met the requisite percentage of students in the Exceeds the Standard category on AIMS necessary for a Highly Performing classification will be designated as a Highly Performing school. 3.) A school that receives a total scale value that places it in the Excelling classification and did not meet either the requisite percentage of students in the Exceeds the Standard category on AIMS necessary for the Excelling classification or the Highly Performing classification will be designated as a Performing school. 4.) A school that receives a total scale value that places it in the Highly Performing classification and meets the requisite percentage of students in the Exceeds the Standard category on AIMS necessary for an Excelling classification will be designated as a Highly Performing school. 5.) A school that receives a total scale value that places it in the Highly Performing classification and meets the requisite percentage of students in the Exceeds the Standard category on AIMS necessary for a Highly Performing classification will be designated as a Highly Performing school. 6.) A school that receives a total scale value that places it in the Highly Performing classification and did not meet either the requisite percentage of students in the Exceeds the Standard category on AIMS necessary for an Excelling classification or Highly Performing classification will be designated a Performing school. 7.) A school that receives a total scale value that places it in the Performing classification will be designated as a Performing Arizona’s Accountability System: Volume I Arizona Department of Education 92 school, regardless if the school meets the requisite percentage of students in the Exceeds the Standard category on AIMS necessary for an Excelling classification or Highly Performing classification. 8.) A school that receives a total scale value that places it in the Underperforming classification will be designated as an Underperforming school, regardless if the school meets the requisite percentage of students in the Exceeds the Standard category on AIMS necessary for an Excelling classification or a Highly Performing classification. Illustration D: Chart of Potential Scenarios Resulting from Threshold Marks Set for Excelling and Highly Performing Achievement Profile Classifications Excelling School receives total scale value placing it in the Excelling classification and meets the requisite % of students in “Exceeds the Standard” category on AIMS for the Excelling classification. Highly Performing Highly Performing School receives total scale value placing it in the Excelling classification and meets the requisite % of students in the “Exceeds the Standard” category on AIMS for Highly Performing classification School receives total scale value placing it in the Highly Performing classification and meets the requisite % of students in the “Exceeds the Standard” category on AIMS for Highly Performing classification Performing School receives total scale value placing it in the Excelling classification but does not meet the requisite % of students for the Excelling or Highly Performing classification Performing School receives total scale value placing it in the Highly Performing classification but does not meet the requisite % of students for the Highly Performing classification. Performing School receives total scale value placing it in the Performing classification. Underperforming School receives total scale value placing it in the Underperforming classification. Arizona’s Accountability System: Volume I Arizona Department of Education 93 Utilizing a three-year average of the percentage of students in the “Exceeds the Standard” category on AIMS the ADE set the thresholds for Excelling and Highly Performing Achievement Profile classifications based on the subject/grade combinations assessed at a particular school. The ADE recommends that Board approve the following threshold marks, based on the fraction of students exceeding the standard on AIMS. In order to establish thresholds for excelling and highly performing schools the ADE rank ordered all schools by the percentage of students exceeding the standard on AIMS. Then the threshold for highly performing was set at the 75th percentile rank of schools with students exceeding the standard; the threshold for excelling was set at the 90th percentile rank of schools with students exceeding the standard (please see Table 11): Table 11: Excelling and Highly Performing Threshold Values by Grades Offered Subject Grade Combinations School Type (Serving grades) Highly Performing Excelling 3 3 6 6 9 3 3 or 5 8 3 and 5 5 and 8 3, 5 and 8 High School 22.6% 6.5% 30.9% 19.9% 19.3% 9.3% 28.7% 10.7% 38.8% 31.7% 25.7% 12.7% BOARD ACTION: H1.) The ADE recommends that the Board approve the application of threshold criteria based on the average percentage of students in the “Exceeds the Standard” category on AIMS (reading, writing or mathematics) to determine the Excelling and Highly Performing Achievement Profile classifications. H2.) The ADE recommends that the Board approve the threshold criteria for the Excelling Achievement Profile classification and the Highly Performing Achievement Profile classification as illustrated in Table 11. I. Impact Data The ADE staff has calculated impact data based on the subset of schools with all data required for the standard version of the AZ LEARNS Achievement Profile methodology. This subset does not include schools with missing data, schools with less than four years of data, extremely small schools, K-2 schools or alternative schools. The current subset totals 1055 schools (elementary and high schools combined). Please note that these are preliminary figures, they may change in the future with the addition of additional school Achievement Profiles. Arizona’s Accountability System: Volume I Arizona Department of Education 94 Table 12: Impact data with Excelling and Highly Performing Thresholds Applied Profile Excelling Highly Performing Performing Underperforming Percent Of Schools 11.7 % 15.7 % 59.1 % 13.5 % Table 13: Impact data without Excelling and Highly Performing Thresholds Applied Profile Excelling Highly Performing Performing Underperforming Percent Of Schools 38.3 % 23.8 % 24.4 % 13.5 % Arizona State Board of Education AZ LEARNS Resolutions Approved September 16, 2003: A1.) The ADE recommends that the Board approve the utilization of the beta distribution to determine cut points for the six (6) Baseline Groupings based on the following percentile ranks: 90, 75, 50, 25 and 10, which yield the cut scores illustrated in Table 1. B1.) The ADE recommends that the Board approve the application of the Growth Point Grouping methodology as described and detailed in Illustration C, yielding the cut points presented in Table 2. F1.) The ADE recommends that the Board approve the cut points detailed in Table 7 and Table 8 to determine the Achievement Profile school classification. G1.) The ADE recommends that the Board approve the added evidence methodology as detailed in action steps 1-4 and all resulting point values as illustrated in Table 9 and Table 10. H1.) The ADE recommends that the Board approve the application of threshold criteria based on the average percentage of students in the “Exceeds the Standard” category on AIMS (reading, writing or mathematics) to determine the Excelling and Highly Performing Achievement Profile classifications. H2.) The ADE recommends that the Board approve the threshold criteria for the Excelling Achievement Profile classification and the Highly Performing Achievement Profile classification as illustrated in Table 11. Arizona’s Accountability System: Volume I Arizona Department of Education 95 School Name: ______________________________ Entity ID: ____________ Reviewer ID #: ___________ Date: _______________ Appendix VI: Substantive Appeal Rubric for AZ LEARNS Evaluation Criteria Data Calculation Discrepancies i.e., school attempts to compare data details with their data sets and gets different numbers Special Circumstances Outside the Control of School/District Administration or Management i.e., school indicates significant teacher attrition; environmental issues/events; adverse testing conditions; school/community emergency/crisis Initial Review (Please check the applicable option) Not ADE data are Data does not applicable accurate and match that of calculations are ADE. School correct. submits evidence of discrepancies and provides additional data. School had a Not Special situation that applicable circumstances that were outside was unavoidable of the school’s control, were not and outside of the school’s a substantial control and cause of the hindered the overall school test performance. administration or student performance. This situation resulted in adverse data for the year(s) in question. Review of Evidence Provided Comments † Compelling evidence † Not compelling evidence † Not applicable evidence † Compelling evidence † Not compelling evidence † Not applicable evidence Policy/Methodology Issues The ADE will not accept/review appeals related to policy/methodology. i.e., school disagrees with use of two year baseline School Name: ______________________________ Entity ID: ____________ Reviewer ID #: ___________ Date: _______________ Team Decision AZ LEARNS Substantive Appeal (Results represent group consensus regarding appeal) Reason Reviewed Data Calculation Discrepancies Special Circumstances Policy/Methodological Issues Initial Review Review of Evidence N/A Correct Data N/A Compelling data/calculation does not evidence match N/A Did not cause Adverse N/A Compelling overall result evidence performance based on situation The ADE will not accept/review appeals related to policy/methodology. Please indicate appropriate response(s) by checking within the box(es) provided. Committee Recommendation: Granted Denied Final Appeal Decision: Granted Denied Comments: Appeal Result: Not compelling evidence Not compelling evidence